Skip to main content
SpringerLink
Log in

Design of computer experiments: space filling and beyond

  • Published:
Statistics and Computing Aims and scope Submit manuscript

Abstract

When setting up a computer experiment, it has become a standard practice to select the inputs spread out uniformly across the available space. These so-called space-filling designs are now ubiquitous in corresponding publications and conferences. The statistical folklore is that such designs have superior properties when it comes to prediction and estimation of emulator functions. In this paper we want to review the circumstances under which this superiority holds, provide some new arguments and clarify the motives to go beyond space-filling. An overview over the state of the art of space-filling is introducing and complementing these results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abt, M.: Estimating the prediction mean squared error in Gaussian stochastic processes with exponential correlation structure. Scand. J. Stat. 26(4), 563–578 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  • Angelis, L., Senta, E.B., Moyssiadis, C.: Optimal exact experimental designs with correlated errors through a simulated annealing algorithm. Comput. Stat. Data Anal. 37(3), 275–296 (2001)

    Article  MATH  Google Scholar 

  • Ash, R.: Information Theory. Wiley, New York (1965) (republished by Dover, New York, 1990)

    MATH  Google Scholar 

  • Audze, P., Eglais, V.: New approach for planning out experiments. Probl. Dyn. Strengths 35, 104–107 (1977)

    Google Scholar 

  • Ball, K.: Eigenvalues of Euclidean distance matrices. J. Approx. Theory 68, 74–82 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  • Bates, R.A., Buck, R.J., Riccomagno, E., Wynn, H.P.: Experimental design and observation for large systems. J. R. Stat. Soc. B 58(1), 77–94 (1996)

    MathSciNet  MATH  Google Scholar 

  • Beirlant, J., Dudewicz, E., Györfi, L., van der Meulen, E.: Nonparametric entropy estimation; an overview. Int. J. Math. Stat. Sci. 6(1), 17–39 (1997)

    MathSciNet  MATH  Google Scholar 

  • Bellhouse, D.R., Herzberg, A.M.: Equally spaced design points in polynomial regression: a comparison of systematic sampling methods with the optimal design of experiments. Can. J. Stat. 12(2), 77–90 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  • Bettinger, R., Duchêne, P., Pronzato, L., Thierry, E.: Design of experiments for response diversity. In: Proc. 6th International Conference on Inverse Problems in Engineering (ICIPE). J. Phys. Conf. Ser., Dourdan (Paris) (2008)

  • Bettinger, R., Duchêne, P., Pronzato, L.: A sequential design method for the inversion of an unknown system. In: Proc. 15th IFAC Symposium on System Identification, Saint-Malo, France, pp. 1298–1303 (2009)

    Google Scholar 

  • Bischoff, W., Miller, F.: Optimal designs which are efficient for lack of fit tests. Ann. Stat. 34(4), 2015–2025 (2006)

    Article  MathSciNet  Google Scholar 

  • Boissonnat, J.D., Yvinec, M.: Algorithmic Geometry. Cambridge University Press, Cambridge (1998)

    MATH  Google Scholar 

  • Bursztyn, D., Steinberg, D.: Comparison of designs for computer experiments. J. Stat. Plan. Inference 136(3), 1103–1119 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  • Carnell, R.: lhs: latin hypercube samples. R package version 0.5 (2009)

  • Chen, V.C.P., Tsui, K.L., Barton, R.R., Meckesheimer, M.: A review on design, modeling and applications of computer experiments. AIIE Trans. 38(4), 273–291 (2006)

    Google Scholar 

  • Cignoni, P., Montani, C., Scopigno, R.: DeWall: a fast divide and conquer Delaunay triangulation algorithm in E d. Comput. Aided Des. 30(5), 333–341 (1998)

    Article  MATH  Google Scholar 

  • Cortés, J., Bullo, F.: Nonsmooth coordination and geometric optimization via distributed dynamical systems. SIAM Rev. 51(1), 163–189 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  • Cressie, N.: Statistics for Spatial Data. Wiley Series in Probability and Statistics. Wiley-Interscience, New York (1993). Rev. Sub. Ed.

    Google Scholar 

  • den Hertog, D., Kleijnen, J.P.C., Siem, A.Y.D.: The correct kriging variance estimated by bootstrapping. J. Oper. Res. Soc. 57(4), 400–409 (2006)

    Article  MATH  Google Scholar 

  • Dette, H., Pepelyshev, A.: Generalized Latin hypercube design for computer experiments. Technometrics 52(4), 421–429 (2010)

    Article  MathSciNet  Google Scholar 

  • Dette, H., Kunert, J., Pepelyshev, A.: Exact optimal designs for weighted least squares analysis with correlated errors. Stat. Sin. 18(1), 135–154 (2008)

    MathSciNet  MATH  Google Scholar 

  • Fang, K.T.: The uniform design: application of number theoretic methods in experimental design. Acta Math. Appl. Sin. 3, 363–372 (1980)

    MATH  Google Scholar 

  • Fang, K.T., Li, R.: Uniform design for computer experiments and its optimal properties. Int. J. Mater. Prod. Technol. 25(1), 198–210 (2006)

    Google Scholar 

  • Fang, K.T., Wang, Y.: Number-Theoretic Methods in Statistics, 1st edn. Chapman & Hall/CRC Monographs on Statistics & Applied Probability. Chapman & Hall/CRC, London/Boca Raton (1993)

    Google Scholar 

  • Fang, K.T., Lin, D.K.J., Winker, P., Zhang, Y.: Uniform design: Theory and application. Technometrics 42(3), 237–248 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  • Fang, K.T., Li, R., Sudjianto, A.: Design and Modeling for Computer Experiments. Chapman & Hall/CRC, London/Boca Raton (2005)

    Book  Google Scholar 

  • Fedorov, V.: Theory of Optimal Experiments. Academic Press, New York (1972)

    Google Scholar 

  • Fedorov, V.V., Hackl, P.: Model-Oriented Design of Experiments. Lecture Notes in Statistics, vol. 125. Springer, Berlin (1997)

    Book  MATH  Google Scholar 

  • Franco, J.: Planification d’expériences numériques en phase exploratoire pour la simulation de phénomènes complexes. Ph.D. thesis, École Nationale Supérieure des Mines de Saint Etienne (2008)

  • Franco, J., Bay, X., Corre, B., Dupuy, D.: Planification d’expériences numériques à partir du processus ponctuel de Strauss. Preprint, Département 3MI, École Nationale Supérieure des Mines de Saint-Etienne. http://hal.archives-ouvertes.fr/hal-00260701/fr/ (2008)

  • Franco, J., Vasseur, O., Corre, B., Sergent, M.: Minimum Spanning Tree: a new approach to assess the quality of the design of computer experiments. Chemom. Intell. Lab. Syst. 97, 164–169 (2009)

    Article  Google Scholar 

  • Gensane, T.: Dense packings of equal spheres in a cube. Electron. J. Comb. 11 (2004)

  • Glover, F., Kelly, J., Laguna, M.: Genetic algorithms and tabu search: hybrids for optimization. Comput. Oper. Res. 22(1), 111–134 (1995)

    Article  MATH  Google Scholar 

  • Gramacy, R., Lee, H.: Cases for the nugget in modeling computer experiments. Tech. rep., http://arxiv.org/abs/1007.4580 (2010)

  • Gramacy, R.B., Lee, H.K.: Adaptive design and analysis of supercomputer experiments. Technometrics 51(2), 130–144 (2009)

    Article  MathSciNet  Google Scholar 

  • Griffith, D.: Spatial Autocorrelation and Spatial Filtering: Gaining Understanding through Theory and Scientific Visualization. Springer, Berlin (2003)

    Google Scholar 

  • Hall, P., Morton, S.: On the estimation of entropy. Ann. Inst. Stat. Math. 45(1), 69–88 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  • Harville, D.A., Jeske, D.R.: Mean squared error of estimation or prediction under a general linear model. J. Am. Stat. Assoc. 87(419), 724–731 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  • Havrda, M., Charvát, F.: Quantification method of classification processes: concept of structural α-entropy. Kybernetika 3, 30–35 (1967)

    MathSciNet  MATH  Google Scholar 

  • Herzberg, A.M., Huda, S.: A comparison of equally spaced designs with different correlation structures in one and more dimensions. Can. J. Stat. 9(2), 203–208 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  • Hoeting, J.A., Davis, R.A., Merton, A.A., Thompson, S.E.: Model selection for geostatistical models. Ecol. Appl. 16(1), 87–98 (2006)

    Article  Google Scholar 

  • Husslage, B., Rennen, G., van Dam, E., den Hertog, D.: Space-filling Latin hypercube designs for computer experiments. Discussion paper 2006-18, Tilburg University, Center for Economic Research (2006)

  • Iooss, B., Boussouf, L., Feuillard, V., Marrel, A.: Numerical studies of the metamodel fitting and validation processes. International Journal on Advances in Systems and Measurements 3(1–2), 11–21 (2010)

    Google Scholar 

  • Irvine, K., Gitelman, A., Hoeting, J.: Spatial designs and properties of spatial correlation: effects on covariance estimation. J. Agric. Biol. Environ. Stat. 12(4), 450–469 (2007)

    Article  MathSciNet  Google Scholar 

  • Jin, R., Chen, W., Sudjianto, A.: An efficient algorithm for constructing optimal design of computer experiments. J. Stat. Plan. Inference 134(1), 268–287 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  • Johnson, M., Moore, L., Ylvisaker, D.: Minimax and maximin distance designs. J. Stat. Plan. Inference 26, 131–148 (1990)

    Article  MathSciNet  Google Scholar 

  • Johnson, R.T., Montgomery, D.C., Jones, B., Fowler, J.W.: Comparing designs for computer simulation experiments. In: WSC ’08: Proceedings of the 40th Conference on Winter Simulation, pp. 463–470 (2008)

    Google Scholar 

  • Joseph, V.: Limit kriging. Technometrics 48(4), 458–466 (2006)

    Article  MathSciNet  Google Scholar 

  • Jourdan, A., Franco, J.: Optimal Latin hypercube designs for the Kullback-Leibler criterion. AStA Adv. Stat. Anal. 94, 341–351 (2010)

    Article  MathSciNet  Google Scholar 

  • Kiefer, J., Wolfowitz, J.: The equivalence of two extremum problems. Can. J. Math. 12, 363–366 (1960)

    Article  MathSciNet  MATH  Google Scholar 

  • Kiseľák, J., Stehlík, M.: Equidistant and d-optimal designs for parameters of Ornstein–Uhlenbeck process. Stat. Probab. Lett. 78(12), 1388–1396 (2008)

    Article  MATH  Google Scholar 

  • Kleijnen, J.P.C.: Design and Analysis of Simulation Experiments. Springer, New York (2009)

    Google Scholar 

  • Koehler, J., Owen, A.: Computer experiments. In: Ghosh, S., Rao, C.R. (eds.) Handbook of Statistics: Design and Analysis of Experiments, vol. 13, pp. 261–308. North-Holland, Amsterdam (1996)

    Google Scholar 

  • Kozachenko, L., Leonenko, N.: On statistical estimation of entropy of random vector. Probl. Inf. Transm. 23(2), 95–101 (1987) (translated from Problemy Peredachi Informatsii, in Russian, vol. 23, No. 2, pp. 9–16, 1987)

    MathSciNet  MATH  Google Scholar 

  • Leary, S., Bhaskar, A., Keane, A.: Optimal orthogonal-array-based Latin hypercubes. J. Appl. Stat. 30(5), 585–598 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  • Leonenko, N., Pronzato, L., Savani, V.: A class of Rényi information estimators for multidimensional densities. Ann. Stat. 36(5), 2153–2182 (2008) (correction in Ann. Stat. 38(6), 3837–3838, 2010)

    Article  MathSciNet  MATH  Google Scholar 

  • Li, X.S., Fang, S.C.: On the entropic regularization method for solving min-max problems with applications. Math. Methods Oper. Res. 46, 119–130 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  • McKay, M., Beckman, R., Conover, W.: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2), 239–245 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  • Melissen, H.: Packing and covering with circles. Ph.D. thesis, University of Utrecht (1997)

  • Mitchell, T.: An algorithm for the construction of “D-optimal” experimental designs. Technometrics 16, 203–210 (1974)

    Article  MathSciNet  MATH  Google Scholar 

  • Morris, M., Mitchell, T.: Exploratory designs for computational experiments. J. Stat. Plan. Inference 43, 381–402 (1995)

    Article  MATH  Google Scholar 

  • Müller, W.G.: Collecting Spatial Data: Optimum Design of Experiments for Random Fields, 3rd edn. Springer, Heidelberg (2007)

    MATH  Google Scholar 

  • Müller, W.G., Stehlík, M.: Compound optimal spatial designs. Environmetrics 21(3–4), 354–364 (2010)

    MathSciNet  Google Scholar 

  • Müller, W.G., Pronzato, L., Waldl, H.: Relations between designs for prediction and estimation in random fields: an illustrative case. In: E. Porcu, J.M. Montero, M. Schlather (eds.) Advances and Challenges in Space-Time Modelling of natural Events. Springer Lecture Notes in Statistics (2011, to appear)

  • Nagy, B., Loeppky, J.L., Welch, W.J.: Fast Bayesian inference for Gaussian process models. Tech. rep., The University of British Columbia, Department of Statistics (2007)

  • Narcowich, F.: Norms of inverses and condition numbers for matrices associated with scattered data. J. Approx. Theory 64, 69–94 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  • Niederreiter, H.: Random Number Generation and Quasi-Monte Carlo Methods (CBMS-NSF Regional Conference Series in Applied Mathematics). SIAM, Philadelphia (1992)

    Book  Google Scholar 

  • Okabe, A., Books, B., Sugihama, K.: Spatial Tessellations. Concepts and Applications of Voronoi Diagrams. Wiley, New York (1992)

    MATH  Google Scholar 

  • Oler, N.: A finite packing problem. Can. Math. Bull. 4, 153–155 (1961)

    Article  MathSciNet  MATH  Google Scholar 

  • Pebesma, E.J., Heuvelink, G.B.M.: Latin hypercube sampling of Gaussian random fields. Technometrics 41(4), 303–312 (1999)

    Article  Google Scholar 

  • Penrose, M., Yukich, J.: Laws of large numbers and nearest neighbor distances. In: Wells, M., Sengupta, A. (eds.) Advances in Directional and Linear Statistics. A Festschrift for Sreenivasa Rao Jammalamadaka, Chap. 13, pp. 189–199 (2011). arXiv:0911.0331v1

  • Petelet, M., Iooss, B., Asserin, O., Loredo, A.: Latin hypercube sampling with inequality constraints. AStA Adv. Stat. Anal. 94, 325–339 (2010)

    Article  MathSciNet  Google Scholar 

  • Picheny, V., Ginsbourger, D., Roustant, O., Haftka, R.T., Kim, N.H.: Adaptive designs of experiments for accurate approximation of a target region. J. Mech. Des. 132(7), 071,008 (2010)

    Article  Google Scholar 

  • Pistone, G., Vicario, G.: Comparing and generating Latin hypercube designs in Kriging models. AStA Adv. Stat. Anal. 94, 353–366 (2010)

    Article  MathSciNet  Google Scholar 

  • Pronzato, L.: Optimal experimental design and some related control problems. Automatica 44(2), 303–325 (2008)

    Article  MathSciNet  Google Scholar 

  • Putter, H., Young, A.: On the effect of covariance function estimation on the accuracy of kriging predictors. Bernoulli 7(3), 421–438 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  • Qian, P.Z.G., Ai, M., Wu, C.F.J.: Construction of nested space-filling designs. Ann. Stat. 37(6A), 3616–3643 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  • Redmond, C., Yukich, J.: Asymptotics for Euclidian functionals with power-weighted edges. Stoch. Process. Appl. 61, 289–304 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  • Rennen, G., Husslage, B., van Dam, E., den Hertog, D.: Nested maximin Latin hypercube designs. Struct Multidisc Optiml 41, 371–395 (2010)

    Article  Google Scholar 

  • Rényi, A.: On measures of entropy and information. In: Proc. 4th Berkeley Symp. on Math. Statist. and Prob., pp. 547–561 (1961)

    Google Scholar 

  • Riccomagno, E., Schwabe, R., Wynn, H.P.: Lattice-based D-optimum design for Fourier regression. Ann. Stat. 25(6), 2313–2327 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  • Royle, J., Nychka, D.: An algorithm for the construction of spatial coverage designs with implementation in SPLUS. Comput. Geosci. 24(5), 479–488 (1998)

    Article  Google Scholar 

  • Sacks, J., Welch, W., Mitchell, T., Wynn, H.: Design and analysis of computer experiments. Stat. Sci. 4(4), 409–435 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  • Santner, T., Williams, B., Notz, W.: The Design and Analysis of Computer Experiments. Springer, Heidelberg (2003)

    MATH  Google Scholar 

  • Schaback, R.: Lower bounds for norms of inverses of interpolation matrices for radial basis functions. J. Approx. Theory 79, 287–306 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  • Schilling, M.F.: Spatial designs when the observations are correlated. Commun. Stat., Simul. Comput. 21(1), 243–267 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  • Scott, D.: Multivariate Density Estimation. Wiley, New York (1992)

    Book  MATH  Google Scholar 

  • Shewry, M., Wynn, H.: Maximum entropy sampling. Appl. Stat. 14, 165–170 (1987)

    Article  Google Scholar 

  • Sjöstedt-De-Luna, S., Young, A.: The bootstrap and kriging prediction intervals. Scand. J. Stat. 30(1), 175–192 (2003)

    Article  MATH  Google Scholar 

  • Stein, M.: Interpolation of Spatial Data: Some Theory for Kriging. Springer, Heidelberg (1999)

    Book  MATH  Google Scholar 

  • Stinstra, E., den Hertog, D., Stehouwer, P., Vestjens, A.: Constrained maximin designs for computer experiments. Technometrics 45(4), 340–346 (2003)

    Article  MathSciNet  Google Scholar 

  • Sun, X.: Norm estimates for inverses of Euclidean distance matrices. J. Approx. Theory 70, 339–347 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  • Tang, B.: Orthogonal array-based latin hypercubes. J. Am. Stat. Assoc. 88(424), 1392–1397 (1993)

    Article  MATH  Google Scholar 

  • Tsallis, C.: Possible generalization of Boltzmann-Gibbs statistics. J. Stat. Phys. 52(1/2), 479–487 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  • van Dam, E.: Two-dimensional minimax Latin hypercube designs. Discrete Appl. Math. 156(18), 3483–3493 (2007)

    Google Scholar 

  • van Dam, E., Hussage, B., den Hertog, D., Melissen, H.: Maximin Latin hypercube designs in two dimensions. Oper. Res. 55(1), 158–169 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  • van Dam, E., Rennen, G., Husslage, B.: Bounds for maximin Latin hypercube designs. Oper. Res. 57(3), 595–608 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  • van Groenigen, J.: The influence of variogram parameters on optimal sampling schemes for mapping by kriging. Geoderma 97(3–4), 223–236 (2000)

    Article  Google Scholar 

  • Walvoort, D.J.J., Brus, D.J., de Gruijter, J.J.: An R package for spatial coverage sampling and random sampling from compact geographical strata by k-means. Comput. Geosci. 36, 1261–1267 (2010)

    Article  Google Scholar 

  • Wendland, H.: Scattered Data Approximation. Cambridge University Press, Cambridge (2005)

    MATH  Google Scholar 

  • Wolkowicz, H., Styan, G.: Bounds for eigenvalues using traces. Linear Algebra Appl. 29, 471–506 (1980)

    Article  MathSciNet  MATH  Google Scholar 

  • Wynn, H.: Maximum entropy sampling and general equivalence theory. In: Di Bucchianico, A., Läuter, H., Wynn, H. (eds.) mODa’7—Advances in Model–Oriented Design and Analysis, Proceedings of the 7th Int. Workshop, Heeze, Netherlands, pp. 211–218. Physica Verlag, Heidelberg (2004)

    Google Scholar 

  • Yfantis, E., Flatman, G., Behar, J.: Efficiency of kriging estimation for square, triangular, and hexagonal grids. Math. Geol. 19(3), 183–205 (1987)

    Article  Google Scholar 

  • Yukich, J.: Probability Theory of Classical Euclidean Optimization Problems. Springer, Berlin (1998)

    MATH  Google Scholar 

  • Zagoraiou, M., Antognini, A.B.: Optimal designs for parameter estimation of the Ornstein-Uhlenbeck process. Appl. Stoch. Models Bus. Ind. 25(5), 583–600 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  • Zhang, H., Zimmerman, D.: Towards reconciling two asymptotic frameworks in spatial statistics. Biometrika 92(4), 921–936 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  • Zhu, Z., Stein, M.: Spatial sampling design for parameter estimation of the covariance function. J. Stat. Plan. Inference 134(2), 583–603 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  • Zhu, Z., Zhang, H.: Spatial sampling design under the infill asymptotic framework. Environmetrics 17(4), 323–337 (2006)

    Article  MathSciNet  Google Scholar 

  • Zimmerman, D.L.: Optimal network design for spatial prediction, covariance parameter estimation, and empirical prediction. Environmetrics 17(6), 635–652 (2006)

    Article  MathSciNet  Google Scholar 

  • Zimmerman, D., Cressie, N.: Mean squared prediction error in the spatial linear model with estimated covariance parameters. Ann. Inst. Stat. Math. 44(1), 27–43 (1992)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratoire I3S, bâtiment Euclide, les Algorithmes, Université de Nice-Sophia Antipolis/CNRS, 2000 route des Lucioles, BP 121, 06903, Sophia Antipolis cedex, France

    Luc Pronzato

  2. Department of Applied Statistics, Johannes-Kepler-University, Linz Freistädter Straße 315, 4040, Linz, Austria

    Werner G. Müller

Authors
  1. Luc Pronzato
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Werner G. Müller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luc Pronzato.

Additional information

This work was partially supported by a PHC Amadeus/OEAD Amadée grant FR11/2010.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Pronzato, L., Müller, W.G. Design of computer experiments: space filling and beyond. Stat Comput 22, 681–701 (2012). https://doi.org/10.1007/s11222-011-9242-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11222-011-9242-3

Keywords

Search

Navigation