Thorsten Joachims
ABSTRACT
This paper examines the reliability of implicit feedback generated from clickthrough data in WWW search. Analyzing the users' decision process using eyetracking and comparing implicit feedback against manual relevance judgments, we conclude that clicks are informative but biased. While this makes the interpretation of clicks as absolute relevance judgments difficult, we show that relative preferences derived from clicks are reasonably accurate on average.
- R. Almeida and V. Almeida. A community-aware search engine. In Proceedings of the World Wide Web Conference (WWW), 2004. Google Scholar
Digital Library
- B. Bartell, G. Cottrell, and R. Belew. Automatic combination of multiple ranked retrieval systems. In Annual ACM SIGIR Conf. on Research and Development in Information Retrieval (SIGIR), pages 173--181, 1994. Google ScholarDigital Library
- R. Belew. Finding Out About. Cambridge, 2000.Google Scholar
- J. Boyan, D. Freitag, and T. Joachims. A machine learning architecture for optimizing web search engines. In AAAI Workshop on Internet Based Information Systems, pages 1--8, August 1996.Google Scholar
- A. Broder. A taxonomy of web search. SIGIR Forum, 36(2):3--10, 2002. Google ScholarDigital Library
- M. Claypool, P. Le, M. Waseda, and D. Brown. Implicit interest indicators. In International Conference on Intelligent User Interfaces (IUI), pages 33--40, 2001. Google ScholarDigital Library
- W. Cohen, R. Shapire, and Y. Singer. Learning to order things. Journal of Artificial Intelligence Research, 10:243--270, 1999. Google ScholarCross Ref
- S. Fox, K. Karnawat, M. Mydland, S. Dumais, and T. White. Evaluating implicit measures to improve the search experiences. In Talk presented at SIGIR03 Workshop on Implicit Measures of User Interests and Preferences, 2003.Google Scholar
- N. Fuhr. Optimum polynomial retrieval functions based on the probability ranking principle. ACM Transactions on Information Systems, 7(3):183--204, 1989. Google ScholarDigital Library
- J. Goldberg, M. Stimson, M. Lewenstein, M. Scott, and A. Wichansky. Eye-tracking in web search tasks: design implications. In Proceedings of the Eye tracking Research and Applications Symposium (ETRA), pages 51--58, 2002. Google ScholarDigital Library
- L. Granka, T. Joachims, and G. Gay. Eye-tracking analysis of user behavior in www search. In ACM Conference on Research and Development in Information Retrieval (SIGIR), 2004. Google ScholarDigital Library
- T. Halverson and A. Hornof. Link colors guide a search. In ACM Conference on Computer-Human Interaction (CHI), 2004. Google ScholarDigital Library
- S. Holland, M. Ester, and W. Kieβling. Preference mining: A novel approach on mining user preferences for personalized applications. In Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD), pages 204 -- 216, 2003.Google ScholarCross Ref
- T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD), 2002. Google ScholarDigital Library
- M. Just and P. Carpenter. A theory of reading: From eye fixations to comprehension. Psychological Review, 87:329--354, 1980.Google ScholarCross Ref
- D. Kelly and N. Belkin. Display time as implicit feedback: Understanding task effects. In ACM International Conference on Research and Development in Information Retrieval (SIGIR), pages 377--384, 2004. Google ScholarDigital Library
- D. Kemp and K. Ramamohanarao. Long-term learning for web search engines. In European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD), pages 263--274, 2002. Google ScholarDigital Library
- K. Kloeckner, N. Wirschum, and A. Jameson. Depth- and breadth-first processing of search result lists. In ACM Conference on Computer-Human Interaction, 2004. Google ScholarDigital Library
- C. Lankford. Gazetracker: software designed to facilitate eye movement analysis. In Proceedings of Eye Tracking Research & Applications, pages 51--55, 2000. Google ScholarDigital Library
- M. Morita and Y. Shinoda. Information filtering based on user behavior analysis and best match text retrieval. In ACM Conference on Research and Development in Information Retrieval (SIGIR), pages 272--281, 1994. Google ScholarDigital Library
- B. Pan, H. Hembrooke, G. Gay, L. Granka, M. Feusner, and J. Newman. The determinants of web page viewing behavior: An eye tracking study. In S. Spencer, editor, Proceedings of Eye Tracking Research & Applications. ACM, New York, 2004. Google ScholarDigital Library
- F. Radlinski and T. Joachims. Query chains: Learning to rank from implicit feedback. In ACM SIGKDD International Conference On Knowledge Discovery and Data Mining (KDD), 2005. Google ScholarDigital Library
- K. Rayner. Eye movements in reading and information processing. Psychological Bulletin, 124:372--252, 1998.Google ScholarCross Ref
- E. Russo and F. LeClerc. An eye-fixation analysis of choice processes for consumer nondurables. Journal of Consumer Research, 21(2):274--290, 1994.Google ScholarCross Ref
- J. Salogarvi, I. Kojo, S. Jaana, and S. Kaski. Can relevance be inferred from eye movements in information retrieval? In Proceedings of the Workshop on Self-Organizing Maps, pages 261--266, 2003.Google Scholar
- R. White, J. Jose, and I. Ruthven. Comparing explicit and implicit feedback techniques for web retrieval: Trec-10 interactive track report. In Text Retrieval Conference (TREC), 2001.Google Scholar
Index Terms
- Accurately interpreting clickthrough data as implicit feedback
Recommendations
Accurately Interpreting Clickthrough Data as Implicit Feedback
This paper examines the reliability of implicit feedback generated from clickthrough data in WWW search. Analyzing the users' decision process using eyetracking and comparing implicit feedback against manual relevance judgments, we conclude that clicks ...
Impact of query intent and search context on clickthrough behavior in sponsored search
Implicit feedback techniques may be used for query intent detection, taking advantage of user behavior to understand their interests and preferences. In sponsored search, a primary concern is the user's interest in purchasing or utilizing a commercial ...
Click data as implicit relevance feedback in web search
Search sessions consist of a person presenting a query to a search engine, followed by that person examining the search results, selecting some of those search results for further review, possibly following some series of hyperlinks, and perhaps ...
Comments