Accurate on-line support vector regression incorporated with compensated prior knowledge

Abstract

When the training data required by the data-driven model is insufficient or difficult to cover the sample space completely, incorporating the prior knowledge and prior knowledge compensation module into the support vector regression (PESVR) can significantly improve the accuracy and generalization performance of the model. However, the optimization problem to be solved is very complex, resulting long training time, and it must be retrained all the data from scratch every time the training set is modified. Comparing to standard support vector regression (SVR), PESVR has multiple input datasets and more complex objective function and constraints, including several coupling constraints, the existing methods cannot effectively solve accurate on-line learning of this nested (i.e. fully coupled) model. In this paper, an accurate on-line support vector regression incorporated with prior knowledge and error compensation is proposed. Under the constraint of Karush–Kuhn–Tucker conditions, the model parameters are updated recursively through the sequential adiabatic incremental adjustments. The error compensation model and the prediction model are updated simultaneously when a real measured sample or prior knowledge sample is added to or removed from the training set. The updated model is identical to the model produced by the batch learning algorithm. Experiments on an artificial dataset and several benchmark datasets show encouraged results for online learning and prediction.

Appendix 1: Proof to the partition results of the three datasets

In this appendix, the KKT conditions are utilized to derive the partition results of the sets $Set_r,Set_e,Set_p$. According to convex optimization theory [6], the solution of the minimization problem (5) is obtained by minimizing the following Lagrange formulation

$$\begin{aligned} \begin{aligned} L=&\frac{{\widetilde{Q}}_{rr}}{2}+\frac{{\widetilde{Q}}_{pp}}{2}+{\widetilde{Q}}_{rp}+\frac{{\widetilde{Q}}_{rr}^e}{2\lambda }\\&+\frac{{\widetilde{Q}}_{pp}^e}{2\lambda }+\frac{{\widetilde{Q}}_{rp}^e}{\lambda }\\&+\varepsilon _r\sum _t^{N_r}(\alpha _{t}^*+\alpha _{t})+\varepsilon _e\sum _t^{N_r}(\beta _{t}^*+\beta _{t})\\&+\varepsilon _p\sum _k^{N_p}(\eta _{k}^*+\eta _{k})\\&-\sum _t^{N_r}y_t^r(\alpha _{t}^*-\alpha _{t})-\sum _t^{N_r}(z_t^r-y_t^r)(\beta _{t}^*-\beta _{t})\\&-\sum _k^{N_p}z_k^p(\eta _{k}^*-\eta _{k})+{\widetilde{A}}_1+{\widetilde{A}}_2+{\widetilde{A}}_3 \end{aligned} \end{aligned}$$

(57)

where ${\widetilde{A}}_1, {\widetilde{A}}_2, {\widetilde{A}}_3$ are the dual form of the constraints in (5):

$$\begin{aligned} \begin{aligned} {\widetilde{A}}_1=&-\sum _t^{N_r}(\chi _t^*\alpha _t^*+\chi _t\alpha _t)-\sum _t^{N_r}(\upsilon _t^*\beta _t^*\\&+\upsilon _t\beta _t)-\sum _k^{N_p}(\zeta _k^*\eta _K^*+\zeta _k\eta _K)\\ {\widetilde{A}}_2=&-C_r\sum _t^{N_r}(\iota _t^*+\iota _t)+\sum _t^{N_r}(\iota _t^*\alpha _t^*+\iota _t\alpha _t)\\&-C_e\sum _t^{N_r}(\varsigma _t^*+\varsigma _t)\\&+\sum _t^{N_r}(\varsigma _t^*\beta _t^*+\varsigma _t\beta _t)-C_p\sum _k^{N_p}(\tau _k^*+\tau _k)\\&+\sum _k^{N_p}(\tau _k^*\eta _k^*+\tau _k\eta _k)\\ {\widetilde{A}}_3=&\rho \sum _t^{N_r}(\alpha _t^*-\alpha _t)+\kappa \sum _t^{N_r}(\beta _t^*-\beta _t)\\&+(\rho +\kappa )\sum _k^{N_p}(\eta _k^*-\eta _k) \end{aligned} \end{aligned}$$

$\chi ^{(*)}$,$\upsilon ^{(*)}$, $\zeta ^{(*)}$, $\iota ^{(*)}$, $\varsigma ^{(*)}$, $\tau ^{(*)}$, $\rho$, $\kappa$ are Lagrangian multipliers. Then by the KKT theorem [21], the following KKT conditions are obtained

$$\begin{aligned} \begin{aligned} \chi _t^{(*)}&,\iota _t^{(*)},\upsilon _t^{(*)},\varsigma _t^{(*)},\zeta _k^{(*)},\tau _k^{(*)}\ge 0\\ \chi _t^{(*)}&\alpha _t^{(*)}=0,\quad \iota _t^{(*)}(C_r-\alpha _t^{(*)})=0\\ \upsilon _t^{(*)}&\beta _t^{(*)}=0,\quad \varsigma _t^{(*)}(C_e-\beta _t^{(*)})=0\\ \zeta _k^{(*)}&\eta _k^{(*)}=0,\quad \tau _k^{(*)}(C_p-\eta _k^{(*)})=0 \end{aligned} \end{aligned}$$

(58)

Note that $\rho$ and $\kappa$ are, respectively, equal to the optimal b and $b_e$ in (4) [9]. According to (9)–(11) and (57), the partial derivative of Lagrangian function L are written as:

$$\begin{aligned} \frac{\partial L}{\partial \alpha _t}= & {} -h_r(x_t^r)+\varepsilon _r-\chi _t+\iota _t=0 \nonumber \\ \frac{\partial L}{\partial \alpha _t^*}= & {} h_r(x_t^r)+\varepsilon _r-\chi _t^*+\iota _t^*=0 \end{aligned}$$

(59)

$$\begin{aligned} \frac{\partial L}{\partial \beta _t}= & {} -h_e(x_t^r)+\varepsilon _e-\upsilon _t+\varsigma _t=0\nonumber \\ \frac{\partial L}{\partial \beta _t^*}= & {} h_e(x_t^r)+\varepsilon _e-\upsilon _t^*+\varsigma _t^*=0 \end{aligned}$$

(60)

$$\begin{aligned} \frac{\partial L}{\partial \eta _k}= & {} -h_p(x_k^p)+\varepsilon _p+z_k^p-\zeta _k+\tau _k=0 \nonumber \\ \frac{\partial L}{\partial \eta _k^*}= & {} h_p(x_k^p)+\varepsilon _p-z_k^p-\zeta _k^*+\tau _k^*=0 \end{aligned}$$

(61)

Combining (8), (58)–(61), the partition results (12)–(14) of the datasets $Set_r,Set_e$ and $Set_p$ are obtained.