Ensemble of surrogates with recursive arithmetic average

Abstract

Surrogate models are often used to replace expensive simulations of engineering problems. The common approach is to construct a series of metamodels based on a training set, and then, from these surrogates, pick out the best one with the highest accuracy as an approximation of the computationally intensive simulation. However, because the choice of approximate model depends on design of experiments (DOEs), the traditional strategy thus increases the risk of adopting an inappropriate model. Furthermore, in the design of complex product system, because of its feature of one-of-a-kind production, acquiring more samples is very expensive and intensively time-consuming, and sometimes even impossible. Therefore, in order to save sampling cost, it is a reasonable strategy to take full advantage of all the stand-alone surrogates and then combine them into an ensemble model. Ensemble technique is an effective way to make up for the shortfalls of traditional strategy. Motivated by the previous research on ensemble of surrogates, a new technique for constructing of a more accurate ensemble of surrogates is proposed in this paper. The weights are obtained using a recursive process, in which the values of these weights are updated in each iteration until the last ensemble achieves a desirable prediction accuracy. This technique has been evaluated using five benchmark problems and one reality problem. The results show that the proposed ensemble of surrogates with recursive arithmetic average provides more ideal prediction accuracy than the stand-alone surrogates and for most problems even exceeds the previously presented ensemble techniques. Finally, we should point out that the advantages of combination over selection are still difficult to illuminate. We are still using an “insurance policy” mode rather than offering significant improvements.

Appendices

Appendix A: Several metamodeling techniques

Here, there are four metamodeling techniques (PRS, RBF, Kriging, SVR) are considered.

1.1 A.1 PRS

For PRS, the highest order is allowed to be 4 in this paper, but the used order in a specific problem is determined by the selected sample set. When the highest order of a polynomial model is 4, it can be expressed as:

$$ \begin{array}{rll} \widetilde{F}(x)&=&a_0 +\sum\limits_{i=1}^N {b_i x_i } +\sum\limits_{i=1}^N {c_{ii} x_{_{i_i } }^2 } +\sum\limits_{ij(i<j)}^ {c_{ij} x_i x_j }\\ &&+\,\sum\limits_{i=1}^N {d_i x_{_{i_i } }^3 } +\sum\limits_{i=1}^N {e_i x_{_{i_i } }^4 } \end{array} $$

(21)

where $\widetilde{F}$ is the response surface approximation of the actual response function, N is the number of variables in the input vector x, and a,b,c,d,e are the unknown coefficients to be determined by the least squares technique.

Notice that 3rd and 4th order models in polynomial model do not have any mixed polynomial terms (interactions) of order 3 and 4. Only pure cubic and quadratic terms are included to reduce the amount of data required for model construction. A lower order model (Linear, Quadratic, and Cubic) includes only lower order polynomial terms (only linear, quadratic, or cubic terms correspondingly).

1.2 A.2 RBF

The general form of the RBF approximation can be expressed as:

$$ f(x)=\sum\limits_{i=1}^m {\beta _i \varphi \big(\big\| {x-x_i } \big\|\big)} $$

(22)

Powell (1987) considers several forms for the basis function φ(·):

• 1. $\varphi (r) = {e^{\left( {{{{ - {r^2}}} \left/ {{{c^2}}} \right.}} \right)}}{\text{Gaussian}}$

• 2. $\varphi (r)=(r^2+c^2)^{\frac{1}{2}}$ Multiquadrics

• 3. $\varphi (r)=(r^2+c^2)^{-\frac{1}{2}}$ Reciprocal Multiquadrics

• 4. $\varphi (r)=\left( { r} \left/ {c^2} \right. \right)\log \left( {{ {r} \left/ {{{c}}} \right.}} \right)$ Thin-Plate Spline

• 5. $\varphi (r)=\frac{1}{1+e^{{r} \!\mathord/{c}}}$ Logistic

where c ≥ 0. Particularly, the multi-quadratic RBF form has been applied by Meckesheimer et al. (2001, 2002) to construct an approximation after Hardy (1971), who used linear combinations of a radically symmetric function based on the Euclidean distance of the form:

$$ \varphi (x)=\beta _0 +\sum\limits_{i=1}^n {\beta \big\| {{\rm {\bf x}}-{\rm {\bf x}}_i } \big\|} $$

(23)

where $\left\| \cdot \right\|$ represents the Euclidean norm. Replacing φ(x) with the vector of response observations, y yields a linear system of n equations and n variables, which is used to solve β. As described above, this technique can be viewed as an interpolating process. RBF surrogates have produced good fits to arbitrary contours of both deterministic and stochastic responses (Powell 1987). Different RBF forms were compared by McDonald et al. (2000) on a hydro code simulation, and the author found that the Gaussian and the multi-quadratic RBF forms performed best generally.

1.3 A.3 Kriging

For computer experiments, kriging is viewed from a Bayesian perspective where the response is regarded as a realization of a stationary random process. The general form of this model is expressed as:

$$ Y(x)=\sum\limits_{j=1}^k {\beta _j f_j (x)} +Z(x) $$

(24)

Where f _j ,j = 1,....,k is assumed as a known vector of function, β _j is an unknown constant needed to estimated, and Z(·) is a stochastic process, commonly assumed to be Gaussian, with mean zero and covariance

$$ \begin{array}{rll} Cov(Z(w),Z(u))&=&\sigma ^2R(w,u)\\ &=&\sigma ^2\exp \left\{-\theta \sum\limits_{i=1}^d{(w_i -u_i )^2} \right\} \end{array} $$

where σ ² is the process variance. In practice, the linear model component in (20) is often reduced to only an intercept b since the inclusion of a more complex linear model does not necessarily yield a better prediction.

1.4 ε-SVR

Given the data set {(x ₁ ,y ₁ ),......,(x _l ,y _l )}(where l denotes the number of samples) and the kernel matrix K _ij  = K(x _i ,x _j ), and if the loss function in SVR is ε-insensitive loss function

$$ L_\varepsilon \left( {f\left( {\rm {\bf x}} \right)-y} \right)=\left\{ {{\begin{array}{@{}l} {0, \quad \left| {f\left( {\rm {\bf x}} \right)-y} \right|<\varepsilon } \hfill \\ {\left| {f\left( {\rm {\bf x}} \right)-y} \right|-\varepsilon , \quad \mathit{other}} \hfill \\ \end{array} }} \right., $$

(25)

then the ε-SVR is written as:

$$ \min \mbox{ }\Phi \left( {{\rm {\bf w}},{\rm {\bf \xi }}} \right)=\frac{1}{2}{\rm {\bf w}}^T{\rm {\bf w}}+C\sum\limits_{i=1}^l {\big( {\xi _i^- +\xi _i^+ } \big)} $$

(26)

$$ s.t.\left\{ {{\begin{array}{@{}c} {f\big( {{\rm {\bf x}}_i } \big)-y_i \le \varepsilon +\xi _i^+ } \hfill \\ {y_i -f\big( {{\rm {\bf x}}_i } \big)\le \varepsilon +\xi _i^- } \hfill \\ {\xi _i^- ,\xi _i^+ \ge 0,} \hfill \\ \end{array} }} \right.,\mbox{ }i=1,\cdots l. $$

The Lagrange dual model of the above model is expressed as:

$$ \begin{array}{rll} \mathop {\min }\limits_{{\rm {\bf \alpha }}^{\left( \ast \right)}} &\mbox{}\frac{1}{2}\sum\limits_{i,j=1}^l \left( {\alpha _i -\alpha _i^\ast } \right)\left( {\alpha _j -\alpha _j^\ast } \right)K\left( {{\rm {\bf x}}_i ,{\rm {\bf x}}_j } \right)\\ &-\sum\limits_{i=1}^l {\left( {\alpha _i -\alpha _i^\ast } \right)y_i } +\varepsilon \sum\limits_{i=1}^l {\left( {\alpha _i +\alpha _i^\ast } \right)} \end{array} $$

(27)

$$ s.t.\left\{ {{\begin{array}{@{}c} {0\le \alpha _i ,\alpha _i^\ast \le C\mbox{ },i=1,\cdots ,l,} \hfill \\ {\sum\limits_{i=1}^l {\left( {\alpha _i -\alpha _i^\ast } \right)=0.} } \hfill \\ \end{array} }} \right., $$

where $K\left( {\cdot ,\cdot } \right)$ is kernel function. After being worked out the parameter α ^( ∗ ), the regression function f(x) can be gotten.

Appendix B: Box plots

In a box plot, the box is composed of lower quartile (25%), median (50%), and upper quartile (75%) values. Besides the box, there are two lines extended from each end of the box, whose upper limit and lower limit are defined as follows:

$$ {{\rm low\_limit = max}}\left\{ {{{\rm Q}}_1}{{\rm - 1}}{{\rm .5IQR, X_{minimum}}}\right\} $$

(28)

$$ {{\rm up\_limit = min}}\left\{ {{{\rm Q}}_3}{{\rm + 1}}{{\rm .5IQR, X_{maximum}}}\right\} $$

(29)

where Q₁ is the value of the line at lower quartile, Q₃ is the value of the line at upper quartile, IQR = Q₃ − Q ₁, X _minimum and X _maximum are the minimum and maximum value of the data. Outliers are data with values beyond the ends of the lines by placing a “+” sign for each point.