An Eye-Tracking Approach to the Analysis of Relevance Judgments on the Web: The Case of Google Search Engine

(Balatsoukas & Ruthven, 2012)

The authors address the topic of relevance judgements in decision making on the web by using eye tracking and present a methodological framework for the analysis of relevance judgements. They suggest that there have been two approaches to studying how people assess information, particularly on the web.

The first of these approaches is a cognitive approach which focused on the relevance criteria people use to make judgements. Their definition of relevance criteria is “A relevance criterion can be defined as the parameter or value by which users determine the relevance of a retrieved object at a certain point in time” (Balatsoukas & Ruthven, 2012, p. 1728).

The second approach is from a behavioural perspective. This approach focuses on the analysis of click-through data and visual behaviour. The authors’ research attempts to amalgamate these two approaches in looking at a Google search engine page using eye tracking technology. They suggest two types of judgement occur in the context of a search engine results page)

1. Predictive judgement (the user predicts the relevance of a document)
2. Evaluative judgement (when the user interacts with the actual document)

The authors then describe the usefulness of using the tracking of eye movements to determine cognitive activity (cognitive load) as well as attention. In terms of search engine results value judgements, they suggest that there are two types:

1. Economic evaluators (who view fewer items, are more experienced and don’t spend much time fixating on results)
2. Exhaustive evaluators (slower, view results more carefully, did not just focus on the top results)

They had 24 subjects in their study. They provide a step-by-step analysis description of the results: Analysis of eye movement data, associating eye movement data with the talk-aloud protocol, association with postsearch interview transcripts, assignment of relevance criteria, assignment of grades of relevance. Their results were presented in terms of means and averages (measures of central tendency)

“Understanding human visual searching behaviour during relevance judgment could provide new ground for experimentation in information retrieval, as data log and click-through analysis did in the past.”

MY NOTES

1. See p 1729 for a justification for the structuring of their article.
2. See their description of the experimental set-up

 

Balatsoukas, P., & Ruthven, I. (2012). An eye-tracking approach to the analysis of relevance judgments on the web: The case of Google search engine. Journal of the American Society for Information Science and Technology, 63(9), 1728–1746.