Reading position predicts personal highlights better than popularity or similarity

Reading Position Is the Baseline to Beat: A Time-Ordered Evaluation of Personalised Highlight Prediction

Information RetrievalHuman-Computer Interaction

Summary

Finding what parts of a text a reader will highlight next is tricky. The authors show that simply looking near where a reader first highlights on a page predicts their next highlights better than using popular highlights from others or complex similarity methods. They argue that any system trying to personalize highlights should at least do better than this reading position baseline. Their study uses real user data ordered in time to make sure predictions are compared fairly.

What this means in practice

  • For reading app developers: Improve personalized highlight suggestions by ranking sentences near a user's first highlight before considering popularity or similarity data.
  • For content recommendation teams: Create more timely and personal reading suggestions by evaluating models in a time-ordered way, using reading position as a baseline.

Tested on one dataset.

Authors

Kazuki Nakayashiki, Keisuke Watanabe

Abstract

A reader's first highlights on a page are the cheapest personal signal a reading product has. The natural plan is to suggest what similar earlier readers marked, and to judge the result against popularity. We argue that the baseline to beat is reading position. In a time-ordered evaluation on one social highlighting platform (7,343 reader-page pairs on 1,511 pages after one highlight), ranking the sentences just below a reader's first highlight, with no other reader's data, puts the next highlight in the top five 47% of the time, against 26% for popularity and 29% for the better of two similarity methods. The baseline depends on the target: over all later highlights that ranking loses to popularity, while popularity discounted by distance from the latest highlight, at the scale with the best average precision of three tried, beats popularity and both similarity methods on both targets. In a comparison specified in advance, neither similarity method shows a gain over popularity in average precision over all later highlights, from one to five highlights, and a gain of +0.01 is excluded. Nor would a gain by itself show that a method has found a reader's preferences: synthetic readers who share one set of preferences produce one, and an evaluation out of time order shows a method where the reader went. The position results are exploratory and unconfirmed. Personalisation inside a document should be evaluated in time order and against reading position.