Neural Networks: Tricks of the Trade

1. Front Matter

   Pages I-VI

   PDF

2. Introduction

   1. Introduction

      Pages 1-5

3. Speeding Learning

   1. Front Matter

      Pages 7-8

      PDF

   2. Efficient BackProp

      • Yann LeCun, Leon Bottou, Genevieve B. Orr, Klaus -Robert Müller

      Pages 9-50

4. Regularization Techniques to Improve Generalization

   1. Front Matter

      Pages 51-53

      PDF

   2. Early Stopping - But When?

      • Lutz Prechelt

      Pages 55-69

   3. A Simple Trick for Estimating the Weight Decay Parameter

      • Thorsteinn S. Rognvaldsson

      Pages 71-92

   4. Controlling the hyperparameter search in MacKay’s Bayesian neural network framework

      • Tony Plate

      Pages 93-112

   5. Adaptive Regularization in Neural Network Modeling

      • Jan Larsen, Claus Svarer, Lars Nonboe Andersen, Lars Kai Hansen

      Pages 113-132

   6. Large Ensemble Averaging

      • David Horn, Ury Naftaly, Nathan Intrator

      Pages 133-139

5. Improving Network Models and Algorithmic Tricks

   1. Front Matter

      Pages 141-143

      PDF

   2. Square Unit Augmented Radially Extended Multilayer Perceptrons

      • Gary William Flake

      Pages 145-163

   3. A Dozen Tricks with Multitask Learning

      • Rich Caruana

      Pages 165-191

   4. Solving the Ill-Conditioning in Neural Network Learning

      • Patrick van der Smagt, Gerd Hirzinger

      Pages 193-206

   5. Centering Neural Network Gradient Factors

      • Nicol N. Schraudolph

      Pages 207-226

   6. Avoiding roundoff error in backpropagating derivatives

      • Tony Plate

      Pages 227-233

6. Representing and Incorporating Prior Knowledge in Neural Network Training

   1. Front Matter

      Pages 235-237

      PDF

   2. Transformation Invariance in Pattern Recognition — Tangent Distance and Tangent Propagation

      • Patrice Y. Simard, Yann A. LeCun, John S. Denker, Bernard Victorri

      Pages 239-274

   3. Combining Neural Networks and Context-Driven Search for Online, Printed Handwriting Recognition in the Newton

      • Yaeger Larry S., Webb Brandyn J., Lyon Richard F.

      Pages 275-298

   4. Neural Network Classification and Prior Class Probabilities

      • Steve Lawrence, Ian Burns, Andrew Back, Ah Chung Tsoi, C. Lee Giles

      Pages 299-313

It is our belief that researchers and practitioners acquire, through experience and word-of-mouth, techniques and heuristics that help them successfully apply neural networks to di cult real world problems. Often these \tricks" are theo- tically well motivated. Sometimes they are the result of trial and error. However, their most common link is that they are usually hidden in people’s heads or in the back pages of space-constrained conference papers. As a result newcomers to the eld waste much time wondering why their networks train so slowly and perform so poorly. This book is an outgrowth of a 1996 NIPS workshop called Tricks of the Trade whose goal was to begin the process of gathering and documenting these tricks. The interest that the workshop generated motivated us to expand our collection and compile it into this book. Although we have no doubt that there are many tricks we have missed, we hope that what we have included will prove to be useful, particularly to those who are relatively new to the eld. Each chapter contains one or more tricks presented by a given author (or authors). We have attempted to group related chapters into sections, though we recognize that the di erent sections are far from disjoint. Some of the chapters (e.g., 1, 13, 17) contain entire systems of tricks that are far more general than the category they have been placed in.

• Artificial Neural Networks
• Learning
• Neural Algorithms
• Pattern Recognition
• Time Series Analysis
• algorithms
• neural networks
• complexity

• Department of Computer Science, Willamette University, Salem, USA

  Genevieve B. Orr

• GMD First (Forschungszentrum Informationstechnik), Berlin, Germany

  Klaus-Robert Müller

Policies and ethics