Pinar Donmez
ABSTRACT
A machine learning approach to learning to rank trains a model to optimize a target evaluation measure with repect to training data. Currently, existing information retrieval measures are impossible to optimize directly except for models with a very small number of parameters. The IR community thus faces a major challenge: how to optimize IR measures of interest directly. In this paper, we present a solution. Specifically, we show that LambdaRank, which smoothly approximates the gradient of the target measure, can be adapted to work with four popular IR target evaluation measures using the same underlying gradient construction. It is likely, therefore, that this construction is extendable to other evaluation measures. We empirically show that LambdaRank finds a locally optimal solution for mean NDCG@10, mean NDCG, MAP and MRR with a 99% confidence rate. We also show that the amount of effective training data varies with IR measure and that with a sufficiently large training set size, matching the training optimization measure to the target evaluation measure yields the best accuracy.
- C.J.C. Burges, R. Ragno, and Q.V. Le. Learning to rank with nonsmooth cost functions. In Neural Information Processing Systems (NIPS), 2006. See also MSR Technical Report MSR-TR-2006-60.Google Scholar
- C.J.C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In International Conference on Machine Learning (ICML), Bonn, Germany, 2005. Google ScholarDigital Library
- Z. Cao, T. Qin, T.Y. Liu, M.F. Tsai, and H. Li. Learning to rank: From pairwise to listwise approach. In International Conference on Machine Learning (ICML), pages 129--136, 2007. Google ScholarDigital Library
- K. Crammer and Y. Singer. Pranking with ranking. In Neural Information Processing Systems (NIPS), 2001.Google Scholar
- R. Herbrich, T. Graepel, and K. Obermayer. Large margin rank boundaries for ordinal regression. Advances in Large Margin Classifiers, pages 115--132, 2000.Google Scholar
- T. Qin, T.Y. Liu, and H. Li. A general approximation framework for direct optimization of information retrieval measures. Microsoft Technical Report MSR-TR-2008-164, 2008.Google Scholar
- T. Qin, X.-D. Zhang, M.-F. Tsai, D.-S. Wang, T.-Y. Liu, and H. Li. Query-level loss functions for information retrieval. Information Processing and Management, 44(2):838--855, 2007. Google ScholarDigital Library
- S. Robertson and H. Zaragoza. On rank-based effectiveness measures and optimization. Information Processing and Management, 10:321--339, 2007. Google ScholarDigital Library
- B. Taskar, V. Chatalbashev, D. Koller, and C. Guestrin. Learning structured prediction models: A large margin approach. In International Conference on Machine Learning (ICML), Bonn, Germany, 2005. Google ScholarDigital Library
- I. Tsochantaridis, T. Hofmann, T. Joachims, and Y. Altun. Support vector machine learning for interdependent and structured output spaces. In International Conference on Machine Learning (ICML), 2004. Google ScholarDigital Library
- J. Xu and H. Li. Adarank: A boosting algorithm for information retrieval. In ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 391--398, 2007. Google ScholarDigital Library
- Y. Yue and C.J.C Burges. On using simultaneous perturbation stochastic approximation for ir measures, and the empirical optimality of lambdarank. NIPS Machine Learning for Web Search Workshop, 2007.Google Scholar
- Y. Yue, T. Finley, F. Radlinski, and T. Joachims. A support vector method for optimizing average precision. In ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2007. Google ScholarDigital Library
Index Terms
- On the local optimality of LambdaRank
Recommendations
Quality-biased ranking for queries with commercial intent
WWW '13 Companion: Proceedings of the 22nd International Conference on World Wide WebModern search engines are good enough to answer popular commercial queries with mainly highly relevant documents. However, our experiments show that users behavior on such relevant commercial sites may differ from one to another web-site with the same ...
Incremental learning to rank with partially-labeled data
WSCD '09: Proceedings of the 2009 workshop on Web Search Click DataIn this paper we present a semi-supervised learning method for a problem of learning to rank where we exploit Markov random walks and graph regularization in order to incorporate not only "labeled" web pages but also plenty of "un-labeled" web pages (...
Learning to rank with multiple objective functions
WWW '11: Proceedings of the 20th international conference on World wide webWe investigate the problem of learning to rank with document retrieval from the perspective of learning for multiple objective functions. We present solutions to two open problems in learning to rank: first, we show how multiple measures can be combined ...
Comments