Jaime Teevan
Susan T. Dumais
Eric Horvitz
Skip Abstract Section
Abstract
Current Web search tools do a good job of retrieving documents that satisfy the most common intentions associated with a query, but do not do a very good job of discerning different individuals' unique search goals. We explore the variation in what different people consider relevant to the same query by mining three data sources: (1) explicit relevance judgments, (2) clicks on search results (a behavior-based implicit measure of relevance), and (3) the similarity of desktop content to search results (a content-based implicit measure of relevance). We find that people's explicit judgments for the same queries differ greatly. As a result, there is a large gap between how well search engines could perform if they were to tailor results to the individual, and how well they currently perform by returning results designed to satisfy everyone. We call this gap the potential for personalization. The two implicit indicators we studied provide complementary value for approximating this variation in result relevance among people. We discuss several uses of our findings, including a personalized search system that takes advantage of the implicit measures by ranking personally relevant results more highly and improving click-through rates.
- Agichtein, E., Brill, E., Dumais, S., and Ragno, R. 2006. Learning user interaction models for predicting Web search preferences. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 3--10. Google Scholar
Digital Library
- Anick, P. 2003. Using terminological feedback for web search refinement: A log based study. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 88--95. Google ScholarDigital Library
- Claypool, M., Brown, D., Le, P., and Waseda, M. 2001. Inferring user interest. IEEE Intern. Comput. 32--39. Google ScholarDigital Library
- Chi, E. and Pirolli, P. 2006. Social information foraging and collaborative search. 2006 Human Computer Interaction Consortium. http://www2.parc. com/istl/projects/uir/publications/author/Pirolli_ab.htmlGoogle Scholar
- Chirita, P., Firan, C., and Nejdl, W. 2006. Summarizing local context to personalize global Web search. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 287--296. Google ScholarDigital Library
- Dou, Z., Song, R., and Wen, J. R. 2007. A large-scale evaluation and analysis of personalized search strategies. In Proceedings of the International World Wide Web Conference. 581--590. Google ScholarDigital Library
- Dumais, S. T., Cutrell, E., Cadiz, J. J., Jancke, G., Sarin, R. and Robbins, D. 2003. Stuff I've Seen: A system for personal information retrieval and re-use. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 72--79. Google ScholarDigital Library
- Eastman, C. M. and Jansen, B. J. 2003. Coverage, relevance and ranking: The impact of query operators on Web search engine results. ACM Trans. Inform. Syst. 21, 4, 383--411. Google ScholarDigital Library
- Fidel, R. and Crandall, M. 1997. Users' perception of the performance of a filtering system. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 198--205. Google ScholarDigital Library
- Frias-Martinez, E., Chen, S. Y., and Liu, X. 2007. Automatic cognitive style identification of digital library users for personalization. J. Amer. Soc. Inform. Sci. Tech. 58, 2, 237--251. Google ScholarDigital Library
- Fox, S., Karnawat, K., Mydland, M., Dumais, S. T., and White, T. 2005. Evaluating implicit measures to improve Web search. ACM Trans. Inform. Syst. 23, 2, 147--168. Google ScholarDigital Library
- Guan, Z. and Cutrell, E. 2007. An eye-tracking study of the effect of target rank on Web search. In Proceedings of the SIGCHI Conference on Human Factors in Information Systems. 417--420. Google ScholarDigital Library
- Harter, S. P. 1996. Variations in relevance assessments and the measurement of retrieval effectiveness. J. Amer. Soc. Inform. Sci. 47, 1, 37--49. Google ScholarDigital Library
- Hawking, D. 2000. Overview of the TREC-9 Web Track. In Proceedings of the Text Retrieval Conference. 87--102.Google Scholar
- Hawking, D. and Craswell, N. 2001. Overview of the TREC-2001 Web Track. In Proceedings of the Text Retrieval Conference. 61--68.Google Scholar
- Järvelin, K. and Kekäläinen, J. 2000. IR evaluation methods for retrieving highly relevant documents. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 41--48. Google ScholarDigital Library
- Jeh, G. and Widom, J. 2003. Scaling personalized Web search. In Proceedings of the International World Wide Web Conference. 271--279. Google ScholarDigital Library
- Joachims, T., Granka, L., Pang, B., Hembrooke, H., and Gay, G. 2005. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 154--161. Google ScholarDigital Library
- Kelly, D. and Fu, X. 2007. Eliciting better information need descriptions from users of information systems. Inform. Process. Manage. 43, 1, 30--46. Google ScholarDigital Library
- Kelly, D. and Teevan, J. 2003. Implicit feedback for inferring user preference: A bibliography. SIGIR Forum 37, 2, 18--28. Google ScholarDigital Library
- Koenmann, J. and Belkin, N. 1996. A case for interaction: A study of interactive information retrieval behavior and effectiveness. In Proceedings of the SIGCHI Conference on Human Factors in Information Systems. 205--212. Google ScholarDigital Library
- Ma, Z., Pant, G., and Sheng, O. 2007. Interest-based personalized search. ACM Trans. Inform. Syst. 25, 5, Article 5. Google ScholarDigital Library
- Mizzaro, S. 1997. Relevance: The whole history. J. Amer. Soc. Inform. Sci. 48, 9, 810--832. Google ScholarDigital Library
- Morita, M. and Shinoda, Y. 1994. Information filtering based on user behavior analysis and best match text retrieval. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 272--281. Google ScholarDigital Library
- Morris, M. R. and Teevan, J. 2008. Understanding groups' properties as a means of improving collaborative search systems. In Proceedings of the JCDL Workshop on Collaborative Information Retrieval.Google Scholar
- Morris, M. R., Teevan, J., and Bush, S. 2008. Enhancing collaborative Web search with personalization: Groupization, smart splitting, and group hit-highlighting. In Proceedings of the ACM Conference on Computer Supported Cooprative Work. 481--484. Google ScholarDigital Library
- Pitkow, J., Schutze, H., Cass, T., Cooley, R., Turnbull, D., Edmonds, A., Adar, E., and Breuel, T. 2002. Personalized search. Comm. ACM, 45, 9, 50--55. Google ScholarDigital Library
- Radlinski, F. and Dumais. S. 2006. Improving personalized Web search using result diversification. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 691--692. Google ScholarDigital Library
- Russell, D. and Grimes, C. 2007. Assigned and self-chosen tasks are not the same in Web search. In Proceedings of the Annual Hawaii International Conference on System Sciences. Google ScholarDigital Library
- Ruthven, I. 2003. Re-examining the potential effectiveness of interactive query expansion. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 213--220. Google ScholarDigital Library
- Saracevic, T. 1976. Relevance: A review of the literature and a framework for thinking on the notion in information science. Adv. Librarian. 6, 81--139.Google Scholar
- Saracevic, T. 2006. Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II. Adv. Librarian. 30, 3--71.Google ScholarCross Ref
- Schamber, L. 1994. Relevance and information behavior. Ann. Rev. Inform. Sci. Tech. 29, 3--48.Google Scholar
- Shen, X., Tan, B. and Zhai, C. X. 2005. Implicit user modeling for personalized search. In Proceedings of the International Conference on Information and Knowledge Management. 824--831. Google ScholarDigital Library
- Sparck Jones, K., Walker, S., and Robertson, S. A. 1998. Probabilistic model of information retrieval: Development and status. Tech. rep. TR-446, Cambridge University Computer Laboratory.Google Scholar
- Spink, A. and Jansen, B. 2004. Web Search: Public Searching of the Web. Kluwer Academic Publishers. Google ScholarDigital Library
- Sugiyama, K., Hatano, K., and Yoshikawa, M. 2004. Adaptive Web search based on user profile constructed without any effort from user. In Proceedings of the International World Wide Web Conference. 675--684. Google ScholarDigital Library
- Teevan, J., Alvarado, C., Ackerman, M. S., and Karger, D. R. 2004. The perfect search engine is not enough: A study of orienteering behavior in directed search. In Proceedings of the SIGCHI Conference on Human Factors in Information Systems. 415--422. Google ScholarDigital Library
- Teevan, J., Dumais, S. T., and Horvitz, E. 2005. Personalizing search via automated analysis of interests and activities. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 449--456. Google ScholarDigital Library
- Teevan, J., Dumais, S. T., and Horvitz, E. 2007a. Characterizing the value of personalizing search. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 757--756. Google ScholarDigital Library
- Teevan J., Adar, E., Jones, R., and Potts, M. 2007b. Information re-retrieval: Repeat queries in Yahoo's logs. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 151--158. Google ScholarDigital Library
- Teevan, J., Dumais, S. T., and Liebling, D. J. 2008. Personalize or not to personalize: Modeling queries with variation in user intent. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 163--170. Google ScholarDigital Library
- Teevan, J., Morris, M., and Bush, S. 2009. Discovering and using groups to improve personalized search. In Proceedings of the ACM International Conference on Web Search and Data Mining. 15--24. Google ScholarDigital Library
- Voorhees, E. 1998. Variations in relevance judgments and the measurement of retrieval effectiveness. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 315--323. Google ScholarDigital Library
- Voorhees, E. and Harman, D., Eds. 2005. TREC: Experiment and Evaluation in Information Retrieval. The MIT Press. Google ScholarDigital Library
- Wu, M., Turpin, A., and Zobel, J. 2008. An investigation on a community's web search variability. In Proceedings of the Australian Computer Society Conference (ACSC‘08), 117--126. Google ScholarDigital Library
Index Terms
- Potential for personalization
Recommendations
Search personalization through query and page topical analysis
Thousands of users issue keyword queries to the Web search engines to find information on a number of topics. Since the users may have diverse backgrounds and may have different expectations for a given query, some search engines try to personalize ...
Characterizing the value of personalizing search
SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrievalWe investigate the diverse goals that people have when they issue the same query to a search engine, and the ability of current search engines to address such diversity. We quantify the potential value of personalizing search results based on this ...
Personalized Search: Potential and Pitfalls
CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge ManagementTraditionally search engines have returned the same results to everyone who asks the same question. However, using a single ranking for everyone in every context at every point in time limits how well a search engine can do in providing relevant ...
Comments