Abstract
Generative algorithms for learning classifiers use training data to separately estimate a probability model for each class. New items are then classified by comparing their probabilities under these models. In contrast, discriminative learning algorithms try to find classifiers that perform well on all the training data.
We show that there is a learning problem that can be solved by a discriminative learning algorithm, but not by any generative learning algorithm (given minimal cryptographic assumptions). This statement is formalized using a framework inspired by previous work of Goldberg [3].
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd edn. Wiley, Chichester (2000)
Fischer, P., Pölt, S., Simon, H.U.: Probably almost Bayes decisions. In: Proceedings of the Fourth Annual COLT, pp. 88–94 (1991)
Goldberg, P.W.: When can two unsupervised learners achieve PAC separation? In: Helmbold, D.P., Williamson, B. (eds.) COLT 2001 and EuroCOLT 2001. LNCS, vol. 2111, p. 303. Springer, Heidelberg (2001)
Goldreich, O., Goldwasser, S., Micali, S.: How to construct random functions. Journal of the Association for Computing Machinery 33(4), 792–807 (1986)
Jaakkola, T., Haussler, D.: Exploiting generative models in discriminative classifiers. In: Advances in NIPS 11, pp. 487–493. Morgan Kaufmann, San Francisco (1998)
Jebara, T.: Machine learning: discriminative and generative. Kluwer, Dordrecht (2003)
Long, P., Servedio, R.: Discriminative Learning can Succeed where Generative Learning Fails (full version), available at: http://www.cs.columbia.edu/~rocco/papers/discgen.html
Ng, A.Y., Jordan, M.I.: On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In: NIPS (2001)
Raina, R., Shen, Y., Ng, A.Y., McCallum, A.: Classification with hybrid generative/discriminative models. In: NIPS (2004)
Håstad, J., Impagliazzo, R., Levin, L., Luby, M.: A pseudorandom generator from any one-way function. SIAM Journal on Computing 28(4), 1364–1396 (1999)
Valiant, L.G.: A theory of the learnable. In: Proc. 16th Annual ACM Symposium on Theory of Computing (STOC), pp. 436–445. ACM Press, New York (1984)
Vapnik, V.: Estimations of dependences based on statistical data. Springer, Heidelberg (1982)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Long, P.M., Servedio, R.A. (2006). Discriminative Learning Can Succeed Where Generative Learning Fails. In: Lugosi, G., Simon, H.U. (eds) Learning Theory. COLT 2006. Lecture Notes in Computer Science(), vol 4005. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11776420_25
Download citation
DOI: https://doi.org/10.1007/11776420_25
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-35294-5
Online ISBN: 978-3-540-35296-9
eBook Packages: Computer ScienceComputer Science (R0)