Shachar Kaufman
Claudia Perlich
Abstract
Deemed “one of the top ten data mining mistakes”, leakage is the introduction of information about the data mining target that should not be legitimately available to mine from. In addition to our own industry experience with real-life projects, controversies around several major public data mining competitions held recently such as the INFORMS 2010 Data Mining Challenge and the IJCNN 2011 Social Network Challenge are evidence that this issue is as relevant today as it has ever been. While acknowledging the importance and prevalence of leakage in both synthetic competitions and real-life data mining projects, existing literature has largely left this idea unexplored. What little has been said turns out not to be broad enough to cover more complex cases of leakage, such as those where the classical independently and identically distributed (i.i.d.) assumption is violated, that have been recently documented. In our new approach, these cases and others are explained by explicitly defining modeling goals and analyzing the broader framework of the data mining problem. The resulting definition enables us to derive general methodology for dealing with the issue. We show that it is possible to avoid leakage with a simple specific approach to data management followed by what we call a learn-predict separation, and present several ways of detecting leakage when the modeler has no control over how the data have been collected. We also offer an alternative point of view on leakage that is based on causal graph modeling concepts.
- Buneman, P., Khanna, S., and Wang-Chiew, T. 2001. Why and where: A characterization of data provenance. In Proceedings of the International Conference on Database Theory (ICDT'01). 316--330. Google Scholar
Digital Library
- Engle, R. and Granger, C. 1987. Co-Integration and error correction: Representation, estimation and testing. Econometrica 55, 2, 251--276.Google ScholarCross Ref
- Hastie, T., Tibshirani, R., and Friedman, J. H. 2009. The Elements of Statistical Learning: Data Mining, Inference and Prediction. Springer.Google Scholar
- Kohavi, R., Brodley, C., Frasca, B., Mason, L., and Zheng, Z. 2000. Kdd-Cup 2000 organizers' report: Peeling the onion. ACM SIGKDD Explor. Newslett. 2. Google ScholarDigital Library
- Kohavi, R., Mason, L., Parekh, R., and Zheng, Z. 2004. Lessons and challenges from mining retail e-commerce data. Mach. Learn. 1, 2. Google ScholarDigital Library
- Kohavi, R. and Parekh, R. 2003. Ten supplementary analyses to improve e-commerce web sites. In Proceedings of the 5th WEBKDD Workshop.Google Scholar
- Laiy, S., Xiang, L., Diao, R., Liu, Y., Gu, H., Xu, L., Li, H., Wang, D., Liu, K., Zhao, J., et al. 2011. Hybrid recommendation models for binary user preference prediction problem. http://jmir.csail.mit.edu/proceedings/papers/v18/lai12a/lai12a.pdf.Google Scholar
- Lo, A. and MacKinlay, A. 1990. Data-Snooping biases in tests of financial asset pricing models. Rev. Finan. Stud. 1, 431--467.Google ScholarCross Ref
- Narayanan, A., Shi, E., and Rubinstein, B. 2011. Link prediction by deanonymization: How we won the kaggle social network challenge. In Proceedings of the International Joint Conference on Neural Networks (IJCNN'11).Google Scholar
- Nisbet, R., Elder, J., and Miner, G. 2009. Handbook of Statistical Analysis and Data Mining Applications. Academic Press. Google ScholarDigital Library
- Pearl, J. 1995. Causal diagrams for empirical research. Biometrika 82, 4, 669--688.Google ScholarCross Ref
- Pearl, J. 2009. Causality: Models, Reasoning and Inference 2nd Ed. Cambridge University Press, Cambridge, UK. Google ScholarDigital Library
- Perlich, C., Melville, P., Liu, Y., Swirszcz, G., Lawrence, R., and Rosset, S. 2008. Breast cancer identification: KDD cup winner's report. ACM SIGKDD Explor. Newslett. 2, 39--42. Google ScholarDigital Library
- Pyle, D. 1999. Data Preparation for Data Mining. Morgan Kaufmann Publishers. Google ScholarDigital Library
- Pyle, D. 2003. Business Modeling and Data Mining. Morgan Kaufmann Publishers. Google ScholarDigital Library
- Pyle, D. 2009. Data Mining: Know it All. Morgan Kaufmann Publishers, Chapter 9.Google Scholar
- Robins, J. 1997. Causal inference from complex longitudinal data. Lat. Variab. Model. Appl. Causal. 120, 69--117.Google ScholarCross Ref
- Rosset, S., Perlich, C., and Liu, Y. 2007. Making the most of your data: KDD cup 2007 “how many ratings” winner's report. ACM SIGKDD Explor. Newslett. 2. Google ScholarDigital Library
- Rosset, S., Perlich, C., Swirszcz, G., Liu, Y., and Melville, P. 2010. Medical data mining: Lessons from winning two competitions. Data Min. Knowl. Discov. 3, 439--468. Google ScholarDigital Library
- Smith, A. and Elkan, C. 2007. Making generative classifiers robust to selection bias. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM Press, New York, 657--666. Google ScholarDigital Library
- Tukey, J. 1977. Exploratory Data Analysis. Addison-Wesley.Google Scholar
- Widmer, G. and Kubat, M. 1996. Learning in the presence of concept drift and hidden contexts. Mach. Learn. 1. Google ScholarDigital Library
- Xie, J. and Coggeshall, S. 2010. Prediction of transfers to tertiary care and hospital mortality: A gradient boosting decision tree approach. Stastist. Anal. Data Min. 3, 4, 253-258. Google ScholarDigital Library
Index Terms
- Leakage in data mining: Formulation, detection, and avoidance
Recommendations
Leakage in data mining: formulation, detection, and avoidance
KDD '11: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data miningDeemed "one of the top ten data mining mistakes", leakage is essentially the introduction of information about the data mining target, which should not be legitimately available to mine from. In addition to our own industry experience with real-life ...
Mining fuzzy specific rare itemsets for education data
Association rule mining is an important data analysis method for the discovery of associations within data. There have been many studies focused on finding fuzzy association rules from transaction databases. Unfortunately, in the real world, one may ...
Leakage- and variability-conscious circuit designs for the 0.5-v nanoscale CMOS era
ISLPED '09: Proceedings of the 2009 ACM/IEEE international symposium on Low power electronics and designLow-voltage scaling limitations of memory-rich CMOS LSIs are one of the major problems in the nanoscale era because they cause the evermore-serious power crisis with device scaling. The problems stem from two unscalable device parameters: The first is ...
Comments