Mehmet Akcay
ABSTRACT
A recent study on the topic of additivity addresses the task of search result diversification and concludes that while weaker baselines are almost always significantly improved by the evaluated diversification methods, for stronger baselines, just the opposite happens, i.e., no significant improvement can be observed. Due to the importance of the issue in shaping future research directions and evaluation strategies in search results diversification, in this work, we first aim to reproduce the findings reported in the previous study, and then investigate its possible limitations. Our extensive experiments first reveal that under the same experimental setting with that previous study, we can reach similar results. Next, we hypothesize that for stronger baselines, tuning the parameters of some methods (i.e., the trade-off parameter between the relevance and diversity of the results in this particular scenario) should be done in a more fine-grained manner. With trade-off parameters that are specifically determined for each baseline run, we show that the percentage of significant improvements even over the strong baselines can be doubled. As a further issue, we discuss the possible impact of using the same strong baseline retrieval function for the diversity computations of the methods. Our takeaway message is that in the case of a strong baseline, it is more crucial to tune the parameters of the diversification methods to be evaluated; but once this is done, additivity is achievable.
- Rakesh Agrawal, Sreenivas Gollapudi, Alan Halverson, and Samuel Ieong 2009. Diversifying Search Results. In Proceedings of WSDM. 5--14. Google Scholar
Digital Library
- Timothy Armstrong, Alistair Moffat, William Webber, and Justin Zobel 2009. Improvements that don't add up: ad-hoc retrieval results since 1998 Proceedings of CIKM. 601--610. Google ScholarDigital Library
- Van Dang and W. Bruce Croft 2012. Diversity by proportionality: an election-based approach to search result diversification. Proceedings of SIGIR. 65--74. Google ScholarDigital Library
- Elena Demidova, Peter Fankhauser, Xuan Zhou, and Wolfgang Nejdl 2010. DivQ: diversification for keyword search over structured databases Proceedings of SIGIR. 331--338. Google ScholarDigital Library
- Joseph A. Fox and Edward Shaw 1994. Combination of multiple sources: The TREC-2 interactive track matrix experiment Proceedings of SIGIR.Google Scholar
- Sreenivas Gollapudi and Aneesh Sharma 2009. An Axiomatic Approach for Result Diversification. Proceedings of WWW. 381--390. Google ScholarDigital Library
- Sadegh Kharazmi, Falk Scholer, David Vallet, and Mark Sanderson 2016. Examining Additivity and Weak Baselines. Transactions on Information Systems Vol. 34, 4 (2016), 23. Google ScholarDigital Library
- Sadegh Kharazmi, Falk Scholer, David Vallet, and Mark Sanderson 2017. Personal communication. (May 2017).Google Scholar
- Joon Ho Lee. 1997. Analyses of multiple evidence combination. In Proceedings of SIGIR. 267--276. Google ScholarDigital Library
- Enrico Minack, Wolf Siberski, and Wolfgang Nejdl. 2011. Incremental diversification for very large sets: a streaming-based approach Proceedings of SIGIR. 585--594. Google ScholarDigital Library
- Kaweh Djafari Naini, Ismail Sengor Altingovde, and Wolf Siberski 2016. Scalable and Efficient Web Search Result Diversification. Transactions on the Web Vol. 10, 3 (2016), 15:1--15:30. Google ScholarDigital Library
- Kezban Dilek Onal, Ismail Sengor Altingovde, and Pinar Karagoz 2015. Utilizing Word Embeddings for Result Diversification in Tweet Search Proceedings of AIRS. 366--378.Google Scholar
- Ahmet Murat Ozdemiray and Ismail Sengor Altingovde 2014. Query Performance Prediction for Aspect Weighting in Search Result Diversification Proceedings of CIKM. 1871--1874. Google ScholarDigital Library
- Ahmet Murat Ozdemiray and Ismail Sengor Altingovde 2015. Explicit search result diversification using score and rank aggregation methods. JASIST, Vol. 66, 6 (2015), 1212--1228.Google Scholar
- Makbule Gulcin Ozsoy, Kezban Dilek Onal, and Ismail Sengor Altingovde 2014. Result Diversification for Tweet Search. In Proceedings of WISE. 78--89.Google ScholarCross Ref
- Monica Lestari Paramita, Jiayu Tang, and Mark Sanderson. 2009. Generic and Spatial Approaches to Image Search Results Diversification Proceedings of ECIR. 603--610. Google ScholarDigital Library
- Stephen Robertson and Hugo Zaragoza 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, Vol. 3, 4 (2009), 333--389. Google ScholarDigital Library
Index Terms
- On the Additivity and Weak Baselines for Search Result Diversification Research
Recommendations
Examining Additivity and Weak Baselines
We present a study of which baseline to use when testing a new retrieval technique. In contrast to past work, we show that measuring a statistically significant improvement over a weak baseline is not a good predictor of whether a similar improvement ...
Explicit diversification of image search
ICMR '13: Proceedings of the 3rd ACM conference on International conference on multimedia retrievalSearch result diversification can increase user satisfaction in answering a particular information need. There are many ways of diversify search results. In some cases the user has a clear idea of how they would like to see their results diversified. ...
Intent-aware search result diversification
SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information RetrievalSearch result diversification has gained momentum as a way to tackle ambiguous queries. An effective approach to this problem is to explicitly model the possible aspects underlying a query, in order to maximise the estimated relevance of the retrieved ...
Comments