Hugo Larochelle
Yoshua Bengio
ABSTRACT
Recently, several learning algorithms relying on models with deep architectures have been proposed. Though they have demonstrated impressive performance, to date, they have only been evaluated on relatively simple problems such as digit recognition in a controlled environment, for which many machine learning algorithms already report reasonable results. Here, we present a series of experiments which indicate that these models show promise in solving harder learning problems that exhibit many factors of variation. These models are compared with well-established algorithms such as Support Vector Machines and single hidden-layer feed-forward neural networks.
- Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007). Greedy layer-wise training of deep networks. Advances in Neural Information Processing Systems 19. MIT Press.Google Scholar
- Bengio, Y., & Le Cun, Y. (2007). Scaling learning algorithms towards AI. In L. Bottou, O. Chapelle, D. De-Coste and J. Weston (Eds.), Large scale kernel machines. MIT Press.Google Scholar
- Chang, C.-C., & Lin, C.-J. (2001). LIBSVM: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.Google Scholar
- Decoste, D., & Scholkopf, B. (2002). Training invariant support vector machines. Machine Learning, 46, 161--190. Google ScholarDigital Library
- Hinton, G. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14, 1771--1800. Google ScholarDigital Library
- Hinton, G. (2006). To recognize shapes, first learn to generate images (Technical Report UTML TR 2006-003). University of Toronto.Google Scholar
- Hinton, G., & Salakhutdinov, R. (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504--507.Google ScholarCross Ref
- Hinton, G. E., Osindero, S., & Teh, Y. (2006). A fast learning algorithm for deep belief nets. Neural Computation, 18, 1527--1554. Google ScholarDigital Library
- LeCun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., & Jackel, L. (1989). Backpropagation applied to handwritten zip code recognition. Neural Computation, 1, 541--551.Google ScholarDigital Library
- LeCun, Y., Huang, F.-J., & Bottou, L. (2004). Learning methods for generic object recognition with invariance to pose and lighting. Proceedings of CVPR'04. IEEE Press. Google ScholarDigital Library
- Salakhutdinov, R., & Hinton, G. (2007). Learning a nonlinear embedding by preserving class neighbourhood structure. To Appear in Proceedings of AISTATS'2007.Google Scholar
- Welling, M., Rosen-Zvi, M., & Hinton, G. (2005). Exponential family harmoniums with an application to information retrieval. Advances in Neural Information Processing Systems 17. MIT Press.Google Scholar
- An empirical evaluation of deep architectures on problems with many factors of variation
Recommendations
Evaluation of FPGA routing architectures under process variation
GLSVLSI '11: Proceedings of the 21st edition of the great lakes symposium on Great lakes symposium on VLSIUncertainty in performance of FPGAs is becoming an important issue due to increased process variations in nanometer regime. Therefore, it is vital to decrease the impact of variability in these devices. FPGA routing architecture enhancement can be an ...
An empirical exploration of recurrent network architectures
ICML'15: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37The Recurrent Neural Network (RNN) is an extremely powerful sequence model that is often difficult to train. The Long Short-Term Memory (LSTM) is a specific RNN architecture whose design makes it much easier to train. While wildly successful in practice,...
Autoencoders, unsupervised learning and deep architectures
UTLW'11: Proceedings of the 2011 International Conference on Unsupervised and Transfer Learning workshop - Volume 27Autoencoders play a fundamental role in unsupervised learning and in deep architectures for transfer learning and other tasks. In spite of their fundamental role, only linear autoencoders over the real numbers have been solved analytically. Here we ...
Comments