Regularizing the effect of input noise injection in feedforward neural networks training

Abstract

Injecting input noise during feedforward neural network (NN) training can improve generalization performance markedly. Reported works justify this fact arguing that noise injection is equivalent to a smoothing regularization with the input noise variance playing the role of the regularization parameter. The success of this approach depends on the appropriate choice of the input noise variance. However, it is often not known a priori if the degree of smoothness imposed on the FNN mapping is consistent with the unknown function to be approximated. In order to have a better control over this smoothing effect, a loss function putting in balance the smoothed fitting induced by the noise injection and the precision of approximation, is proposed. The second term, which aims at penalizing the undesirable effect of input noise injection or controlling the deviation of the random perturbed loss, was obtained by expressing a certain distance between the original loss function and its random perturbed version. In fact, this term can be derived in general for parametrical models that satisfy the Lipschitz property. An example is included to illustrate the effectiveness of learning with this proposed loss function when noise injection is used.

Appendices

Appendix 1

Proof of Lemma 1

Recall that F _L was defined as the set of FNN function satisfying assumption 2.1, and let $\mathcal{A} = E{\left\{ {\sup _{{F_{L} }} {\left| {J_{{emp_{{NI_{{emp}} }} }} (\theta _{L} ) - E{\left[ {(y - f_{{NN}} (x + \eta ,\theta _{L} ))^{2} } \right]}} \right|}} \right\}}$

$$ \begin{aligned} {\mathop {\sup }\limits_{F_{L} } }{\left| {J_{{emp_{{NI_{{emp}} }} }} (\theta _{L} ) - J(\theta _{L} )} \right|} & \leq {\mathop {\sup }\limits_{F_{L} } }{\left| {J_{{emp_{{NI_{{emp}} }} }} (\theta _{L} ) - E\{ (y - f(x + \eta ,\theta _{L} ))^{2} \} } \right|} \\ & \quad + {\mathop {\sup }\limits_{F_{L} } }{\left| {E\{ (y - f_{{NN}} (x + \eta ,\theta _{L} ))^{2} \} - J(\theta _{L} ),} \right|} \\ \end{aligned} $$

where, z _i =x _i +η _i , z=x+η and h(z,y)=(y−f _NN (z,θ _L ))². From Theorems 1 and 2 of [15]

$$ \begin{aligned} \mathcal{A} & = E{\left\{ {{\mathop {\sup }\limits_{F_{L} } }{\left| {\frac{1} {{N.r}}{\sum\limits_{j = 1}^r {{\sum\limits_{i = 1}^N {{\left( {y_{i} - f_{{NN}} (x_{i} + \eta _{{j,i}} ,\theta _{L} )} \right)}^{2} } }} } - \frac{1} {r}{\sum\limits_{j = 1}^r {E{\left[ {(y - f_{{NN}} (x + \eta ,\theta _{L} ))^{2} } \right]}} }} \right|}} \right\}} \cr & \leq \frac{1} {r}{\sum\limits_{j = 1}^r {E{\left\{ {{\mathop {\sup }\limits_{F_{L} } }{\left| {\frac{1} {N}{\sum\limits_{i = 1}^N {{\left( {y_{i} - f_{{NN}} (x_{i} + \eta _{{j,i}} ,\theta _{L} )} \right)}^{2} } } - E{\left[ {(y - f_{{NN}} (x + \eta ,\theta _{L} ))^{2} } \right]}} \right|}} \right\}}} } \cr & = E{\left\{ {{\mathop {\sup }\limits_{F_{L} } }{\left| {\frac{1} {N}{\sum\limits_{i = 1}^N {{\left( {y_{i} - f_{{NN}} (x_{i} + \eta _{{j,i}} ,\theta _{L} )} \right)}^{2} } } - E{\left[ {(y - f_{{NN}} (x + \eta ,\theta _{L} ))^{2} } \right]}} \right|}} \right\}} \cr & = E{\left\{ {{\mathop {\sup }\limits_{F_{L} } }{\left| {\frac{1} {N}{\sum\limits_{i = 1}^N {h(x_{i} ,y_{i} )} } - E{\left[ {h(z,y)} \right]}} \right|}} \right\}} \cr \end{aligned} $$

where the set of functions H _L is defined by

$$ {\mathop {\sup }\limits_{H_{L} } }{\left| {\frac{1} {N}{\sum\limits_{i = 1}^N {h(z_{i} ,y_{i} ) - E\{ h(z,y)\} } }} \right|} \to 0,\quad a.s. $$

Therefore,

$$ H_{L} = {\left\{ {h(x,y) = (y - f_{{NN}} (x,\theta _{L} ))^{2} ;{\sum\limits_{i = 1}^L {|\alpha _{l} | \leq \omega _{L} } }} \right\}}. $$

which concludes the proof.

Appendix 2

Proof of Proposition 1

It is easily shown that the sigmoid function ${\left( {\psi (x) = \frac{1}{{1 + e^{{ - x}} }}} \right)}$ has a derivative ψ’ such that, $\left| {\psi '(x) \leq \frac{1} {4}} \right|.$ Hense ψ is a Lipschitz continuous, with a Lipschitz constant $k = \frac{1} {4}.$. Using the expression of f _NN (.), we have

$$ E{\left\{ {{\mathop {\sup }\limits_{H_{L} } }{\left| {\frac{1} {N}{\sum\limits_{i = 1}^N {h(z_{i} ,y_{i} )} } - E\{ h(z,y)\} } \right|}} \right\}} \to 0,\quad a.s., $$

Now, considering the Lipschitz continuity of ψ(.), we obtain:

$$ \begin{aligned} {\left| {f_{{NN}} (x_{1} ) - f_{{NN}} (x_{2} )} \right|} & = {\left| {{\sum\limits_{l = 1}^L {\alpha _{l} \psi (\beta ^{T}_{l} .x_{1} + \gamma _{l} ) - } }{\sum\limits_{l = 1}^L {\alpha _{l} \psi (\beta ^{T}_{l} .x_{2} + \gamma _{l} )} }} \right|} \cr & = {\left| {{\sum\limits_{l = 1}^L {\alpha _{l} [\psi (\beta ^{T}_{l} .x_{1} + \gamma _{l} ) - \psi (\beta ^{T}_{l} .x_{2} + \gamma _{l} )]} }} \right|} \cr & \leq {\sum\limits_{l = 1}^L {{\left| {\alpha _{l} [\psi (\beta ^{T}_{l} .x_{1} + \gamma _{l} ) - \psi (\beta ^{T}_{l} .x_{2} + \gamma _{l} )]} \right|}} } \cr & = {\sum\limits_{l = 1}^L {|\alpha _{l} |.|[\psi (\beta ^{T}_{l} .x_{1} + \gamma _{l} ) - \psi (\beta ^{T}_{l} .x_{2} + \gamma _{l} )]|.} } \cr \end{aligned} $$

where |α| and |β| represent, respectively, the L ₁-norm of the vector of linear parameters and and the L ₁-norm of the input to the hidden layer parameters matrix (which is taken as the sum of the absolutes values of the elements of the vector or the matrix). Therefore, the Lipschitz constant of f _NN (.,θ) is $K = \frac{1} {4}.|{\mathbf{\alpha }}|.|{\mathbf{\beta }}|.$

Proof of Lemma 2

The previous result is used for the demonstration of this lemma. After some simple manipulation, we can write

$$ \begin{aligned} {\left| {f_{{NN}} (x_{1} ) - f_{{NN}} (x_{2} )} \right|} & \leq k{\sum\limits_{l = 1}^L {|\alpha _{l} |.|\beta ^{T}_{l} .x_{1} - \beta ^{T}_{l} .x_{2} |} } \cr & \leq k{\sum\limits_{l = 1}^L {|\alpha _{l} ||\beta ^{T}_{l} ||x_{1} - x_{2} |} } \cr & \leq k|{\mathbf{\alpha }}|.|{\mathbf{\beta }}|.|x_{1} - x_{2} |, \cr \end{aligned} $$

Under the asymptotic assumption the cross term E{ (y−f _NN (x,θ _L )).(f _NN (x,θ _L )−f _NN (x+η, θ _L ))} vanishes [11, 12]. Hence:

$$ \begin{aligned} E\{ (y - f_{{NN}} (x + \eta ,\theta _{L} ))^{2} \} & = E\{ (y - f_{{NN}} (x,\theta _{L} ))^{2} \} \cr & \quad + 2E\{ (y - f_{{NN}} (x,\theta _{L} )).(f_{{NN}} (x,\theta _{L} ) - f_{{NN}} (x + \eta ,\theta _{L} ))\} \cr & \quad + E\{ (f_{{NN}} (x,\theta _{L} ) - f_{{NN}} (x + \eta ,\theta _{L} ))^{2} \} \cr \end{aligned} $$

where |I|=d is the L ₁-norm of the identity matrix and K is the Lipschitz constant of the FNN function.