Skip to main content
SpringerLink
Log in

The Algorithm Selection Problem on the Continuous Optimization Domain

  • Conference paper
Computational Intelligence in Intelligent Data Analysis

Part of the book series: Studies in Computational Intelligence ((SCI,volume 445))

Abstract

The problem of algorithm selection, that is identifying the most efficient algorithm for a given computational task, is non-trivial. Meta-learning techniques have been used successfully for this problem in particular domains, including pattern recognition and constraint satisfaction. However, there has been a paucity of studies focused specifically on algorithm selection for continuous optimization problems. This may be attributed to some extent to the difficulties associated with quantifying problem “hardness” in terms of the underlying cost function. In this paper, we provide a survey of the related literature in the continuous optimization domain. We discuss alternative approaches for landscape analysis, algorithm modeling and portfolio development. Finally, we propose a meta-learning framework for the algorithm selection problem in the continuous optimization domain.

This work has been partially funded through a 2012-2013 DAAD/Go8 Grant.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Achiloptas, D., Naor, A., Peres, Y.: Rigorous location of phase transitions in hard optimization problems. Nature 435, 759–764 (2005)

    Article  Google Scholar 

  2. Anderson, E.: Markov chain modelling of the solution surface in local search. J. Oper. Res. Soc. 53(6), 630–636 (2002)

    Article  MATH  Google Scholar 

  3. Angel, E., Zissimopoulos, V.: On the hardness of the quadratic assignment problem with metaheuristics. Heuristics 8(4), 399–414 (2002)

    Article  Google Scholar 

  4. Bartz-Beielstein, T., Markon, S.: Tuning search algorithms for real-world applications: a regression tree based approach. In: CEC 2004, vol. 1, pp. 1111–1118 (2004)

    Google Scholar 

  5. Bartz-Beielstein, T., Parsopoulos, K., Vrahatis, M.: Analysis of particle swarm optimization using computational statistics. In: ICNAAM 2004, pp. 34–37 (2004)

    Google Scholar 

  6. Beck, J., Watson, J.: Adaptive search algorithms and Fitness-Distance correlation. In: Proc. 5th Metaheuristics Int. Conf. (2003)

    Google Scholar 

  7. Borenstein, Y., Poli, R.: Fitness Distributions and GA Hardness. In: Yao, X., Burke, E.K., Lozano, J.A., Smith, J., Merelo-Guervós, J.J., Bullinaria, J.A., Rowe, J.E., Tiňo, P., Kabán, A., Schwefel, H.-P. (eds.) PPSN 2004. LNCS, vol. 3242, pp. 11–20. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  8. Borenstein, Y., Poli, R.: Information landscapes. In: GECCO 2005, pp. 1515–1522. ACM (2005)

    Google Scholar 

  9. Borenstein, Y., Poli, R.: Kolmogorov complexity, optimization and hardness. In: CEC 2006, pp. 112–119 (2006)

    Google Scholar 

  10. Boukeas, G., Halatsis, C., Zissimopoulos, V., Stamatopoulos, P.: Measures of Intrinsic Hardness for Constraint Satisfaction Problem Instances. In: Van Emde Boas, P., Pokorný, J., Bieliková, M., Štuller, J. (eds.) SOFSEM 2004. LNCS, vol. 2932, pp. 184–195. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  11. Branke, J., Schmidt, C.: Faster convergence by means of fitness estimation. Soft Comput. 9(1), 13–20 (2005)

    Article  Google Scholar 

  12. Brooks, C., Durfee, E.: Using Landscape Theory to Measure Learning Difficulty for Adaptive Agents. In: Alonso, E., Kudenko, D., Kazakov, D. (eds.) AAMAS 2000 and AAMAS 2002. LNCS (LNAI), vol. 2636, pp. 291–305. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  13. Carchrae, T., Beck, J.: Low knowledge algorithm control. In: AAAI 2004, pp. 49–54 (2004)

    Google Scholar 

  14. Carchrae, T., Beck, J.: Applying machine learning to low-knowledge control of optimization algorihms. Comput. Intell. 21(4), 372–387 (2005)

    Article  MathSciNet  Google Scholar 

  15. Davidor, Y.: Epistasis variance: A viewpoint on GA-hardness. In: Foundations of Genetic Algorithms. Morgan Kaufmann (1991)

    Google Scholar 

  16. Eiben, A., Hinterding, R., Michalewicz, Z.: Parameter control in evolutionary algorithms. IEEE Trans. Evol. Comput. 3(2), 124–141 (1999)

    Article  Google Scholar 

  17. Eremeev, A.V., Reeves, C.R.: Non-parametric Estimation of Properties of Combinatorial Landscapes. In: Cagnoni, S., Gottlieb, J., Hart, E., Middendorf, M., Raidl, G.R. (eds.) EvoIASP 2002, EvoWorkshops 2002, EvoSTIM 2002, EvoCOP 2002, and EvoPlan 2002. LNCS, vol. 2279, pp. 31–40. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  18. Eremeev, A.V., Reeves, C.R.: On Confidence Intervals for the Number of Local Optima. In: Raidl, G.R., Cagnoni, S., Cardalda, J.J.R., Corne, D.W., Gottlieb, J., Guillot, A., Hart, E., Johnson, C.G., Marchiori, E., Meyer, J.-A., Middendorf, M. (eds.) EvoIASP 2003, EvoWorkshops 2003, EvoSTIM 2003, EvoROB/EvoRobot 2003, EvoCOP 2003, EvoBIO 2003, and EvoMUSART 2003. LNCS, vol. 2611, pp. 224–235. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  19. Fonlupt, C., Robilliard, D., Preux, P.: A Bit-Wise Epistasis Measure for Binary Search Spaces. In: Eiben, A.E., Bäck, T., Schoenauer, M., Schwefel, H.-P. (eds.) PPSN 1998. LNCS, vol. 1498, pp. 47–56. Springer, Heidelberg (1998)

    Chapter  Google Scholar 

  20. Francois, O., Lavergne, C.: Design of evolutionary algorithms-A statistical perspective. IEEE Trans. Evol. Comput. 5(2), 129–148 (2001)

    Article  Google Scholar 

  21. Galván-López, E., McDermott, J., O’Neill, M., Brabazon, A.: Defining locality as a problem difficulty measure in genetic programming. Genet. Program Evolvable Mach. 12(4), 365–401 (2011)

    Article  Google Scholar 

  22. Garnier, J., Kallel, L.: Efficiency of local search with multiple local optima. SIAM J. Discrete Math. 15(1), 122–141 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  23. Gomes, C., Selman, B.: Algorithm portfolios. Artif. Intell. 126(1-2), 43–62 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  24. Graff, M., Poli, R.: Practical performance models of algorithms in evolutionary program induction and other domains. Artif. Intell. 174, 1254–1276 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  25. Grobler, J., Engelbrecht, A., Kendall, G., Yadavalli, V.: Alternative hyper-heuristic strategies for multi-method global optimization. In: CEC 2010, pp. 1–8 (2010)

    Google Scholar 

  26. He, J., Reeves, C., Witt, C., Yao, X.: A note on problem difficulty measures in black-box optimization: Classification, realizations and predictability. Evol. Comput. 15(4), 435–443 (2007)

    Article  Google Scholar 

  27. Heckendorn, R., Whitley, D.: Predicting epistasis from mathematical models. Evol. Comput. 7(1), 69–101 (1999)

    Article  Google Scholar 

  28. Hilario, M., Kalousis, A., Nguyen, P., Woznica, A.: A data mining ontology for algorithm selection and meta-mining. In: SoKD 2009, pp. 76–87 (2009)

    Google Scholar 

  29. Hough, P., Williams, P.: Modern machine learning for automatic optimization algorithm selection. In: INFORMS AI/DM Workshop (2006)

    Google Scholar 

  30. Hutter, F., Hamadi, Y., Hoos, H.H., Leyton-Brown, K.: Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms. In: Benhamou, F. (ed.) CP 2006. LNCS, vol. 4204, pp. 213–228. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  31. Hutter, F., Hamadi, Y., Hoos, H., Leyton-Brown, K.: Performance prediction and automated tuning of randomized and parametric algorithms: An initial investigation. In: The AAAI Workshop on Learning for Search: Schedule (2006)

    Google Scholar 

  32. Hutter, F., Hoos, H., Leyton-Brown, K.: Tradeoffs in the empirical evaluation of competing algorithm designs. Tech. Rep. TR-2009-21, The University of British Columbia (2009)

    Google Scholar 

  33. Jansen, T.: On classifications of fitness functions. Tech. Rep. CI-76/99, University of Dortmund (1999)

    Google Scholar 

  34. Jones, T., Forrest, S.: Fitness distance correlation as a measure of problem difficulty for genetic algorithms. In: GECCO 1995, pp. 184–192. Morgan Kaufmann Publishers Inc. (1995)

    Google Scholar 

  35. Jong, K.D.: Parameter setting in EAs: a 30 year perspective. In: Parameter Setting in Evolutionary Algorithms, Stud. Comput Intell., vol. 54, pp. 1–18. Springer (2005)

    Google Scholar 

  36. Leyton-Brown, K., Nudelman, E., Shoham, Y.: Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions. In: Van Hentenryck, P. (ed.) CP 2002. LNCS, vol. 2470, pp. 556–572. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  37. Leyton-Brown, K., Nudelman, E., Shoham, Y.: Empirical hardness models: Methodology and a case study on combinatorial auctions. J. ACM 56(4), 22:1–22:52 (2009)

    Google Scholar 

  38. Liu, L., Abbass, H., Green, D., Zhong, W.: Motif difficulty (MD): a predictive measure of problem difficulty for evolutionary algorithms using network motifs. Evol. Comput. 20(3), 321–347 (2012)

    Google Scholar 

  39. Lunacek, M., Whitley, D.: The dispersion metric and the CMA evolution strategy. In: GECCO 2006, pp. 477–484. ACM, New York (2006)

    Chapter  Google Scholar 

  40. Merkuryeva, G., Bolshakovs, V.: Structural analysis of benchmarking fitness landscapes. Scientific Journal of Riga Technical University: Computer Sciences 42, 81–86 (2010)

    Article  Google Scholar 

  41. Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C., Rudolph, G.: Exploratory landscape analysis. In: GECCO 2011, pp. 829–836. ACM (2011)

    Google Scholar 

  42. Messelis, T., Haspeslagh, S., Bilgin, B., Causmaecker, P.D., Berghe, G.V.: Towards prediction of algorithm performance in real world optimisation problems. In: BNAIC 2009, pp. 177–183 (2009)

    Google Scholar 

  43. Molina, D., Herrera, F., Lozano, M.: Adaptive local search parameters for real-coded memetic algorithms. In: CEC 2005, vol. 1, pp. 888–895 (2005)

    Google Scholar 

  44. Muñoz, M., Kirley, M., Halgamuge, S.: Landscape characterization of numerical optimization problems using biased scattered data. In: CEC 2012, pp. 2162–2169 (2012)

    Google Scholar 

  45. Muñoz, M.A., Kirley, M., Halgamuge, S.K.: A Meta-learning Prediction Model of Algorithm Performance for Continuous Optimization Problems. In: Coello Coello, C.A., Cutello, V., Deb, K., Forrest, S., Nicosia, G., Pavone, M. (eds.) PPSN 2012, Part I. LNCS, vol. 7491, pp. 226–235. Springer, Heidelberg (2012)

    Google Scholar 

  46. Müller, C.L., Sbalzarini, I.F.: Global Characterization of the CEC 2005 Fitness Landscapes Using Fitness-Distance Analysis. In: Di Chio, C., Cagnoni, S., Cotta, C., Ebner, M., Ekárt, A., Esparcia-Alcázar, A.I., Merelo, J.J., Neri, F., Preuss, M., Richter, H., Togelius, J., Yannakakis, G.N. (eds.) EvoApplications 2011, Part I. LNCS, vol. 6624, pp. 294–303. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  47. Naudts, B., Kallel, L.: A comparison of predictive measures of problem difficulty in evolutionary algorithms. IEEE Trans. Evol. Comput. 4(1), 1–15 (2000)

    Article  Google Scholar 

  48. Naudts, B., Suys, D., Verschoren, A.: Epistasis as a basic concept in formal landscape analysis. In: GECCO 1997, pp. 65–72. Morgan Kaufmann (1997)

    Google Scholar 

  49. Özcan, E., Bilgin, B., Korkmaz, E.E.: Hill Climbers and Mutational Heuristics in Hyperheuristics. In: Runarsson, T.P., Beyer, H.-G., Burke, E.K., Merelo-Guervós, J.J., Whitley, L.D., Yao, X. (eds.) PPSN 2006. LNCS, vol. 4193, pp. 202–211. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  50. Pedersen, M.: Tuning & simplifying heuristical optimization. PhD thesis, University of Southampton (2009)

    Google Scholar 

  51. Peng, F., Tang, K., Chen, G., Yao, X.: Population-Based algorithm portfolios for numerical optimization. IEEE Trans. Evol. Comput. 14(5), 782–800 (2010)

    Article  Google Scholar 

  52. Petrovic, S., Epstein, S., Wallace, R.: Learning a mixture of search heuristics. In: CP 2007 (2007)

    Google Scholar 

  53. Reeves, C.: Fitness landscapes. In: Search Methodologies, pp. 587–610. Springer (2005)

    Google Scholar 

  54. Reeves, C., Eremeev, A.: Statistical analysis of local search landscapes. J. Oper. Res. Soc. 55(7), 687–693 (2004)

    Article  MATH  Google Scholar 

  55. Reeves, C., Wright, C.: An experimental design perspective on genetic algorithms. In: Foundations of Genetic Algorithms, vol. 3, pp. 7–22. Morgan Kaufmann (1995)

    Google Scholar 

  56. Rice, J.: The algorithm selection problem. In: Adv. Comput., vol. 15, pp. 65–118. Elsevier (1976)

    Google Scholar 

  57. Rice, J.: Methodology for the algorithm selection problem. In: Proc. IFIP TC 2.5 Working Conference on Performance Evaluation of Numerical Software (1979)

    Google Scholar 

  58. Rochet, S., Venturini, G., Slimane, M., El Kharoubi, E.M.: A Critical and Empirical Study of Epistasis Measures for Predicting GA Performances: A Summary. In: Hao, J.-K., Lutton, E., Ronald, E., Schoenauer, M., Snyers, D. (eds.) AE 1997. LNCS, vol. 1363, pp. 275–285. Springer, Heidelberg (1998)

    Chapter  Google Scholar 

  59. Rockmore, D., Kostelec, P., Hordijk, W., Stadler, P.: Fast fourier transform for fitness landscapes. Appl. Comput. Harmon Anal. 12(1), 57–76 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  60. Seo, D., Moon, B.: An Information-Theoretic analysis on the interactions of variables in combinatorial optimization problems. Evol. Comput. 15(2), 169–198 (2007)

    Article  Google Scholar 

  61. Smith, T., Husbands, P., Layzell, P., O’Shea, M.: Fitness landscapes and evolvability. Evol. Comput. 10(1), 1–34 (2002)

    Article  Google Scholar 

  62. Smith-Miles, K.: Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Comput. Surv. 41(1), 6:1–6:25 (2009)

    Google Scholar 

  63. Smith-Miles, K., Hemert, J.v.: Discovering the suitability of optimisation algorithms by learning from evolved instances. Ann. Math. Artif. Intel. 61(2), 87–104 (2011)

    Article  MATH  Google Scholar 

  64. Stadler, P.: Landscapes and their correlation functions. J. Math. Chem. 20(1), 1–45 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  65. Streeter, M., Golovin, D., Smith, S.: Combining multiple heuristics online. In: AAAI 2007, vol. 2, pp. 1197–1203. AAAI Press (2007)

    Google Scholar 

  66. Tomassini, M., Vanneschi, L., Collard, P., Clergue, M.: A study of fitness distance correlation as a difficulty measure in genetic programming. Evolutionary Computation 13(2), 213–239 (2005)

    Article  Google Scholar 

  67. Vanneschi, L.: Investigating problem hardness of real life applications. In: Genetic Programming Theory and Practice V, Genetic and Evolutionary Computation, pp. 107–124. Springer, US (2008)

    Chapter  Google Scholar 

  68. Vanneschi, L., Clergue, M., Collard, P., Tomassini, M., Vérel, S.: Fitness Clouds and Problem Hardness in Genetic Programming. In: Deb, K., Tari, Z. (eds.) GECCO 2004. LNCS, vol. 3103, pp. 690–701. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  69. Vanneschi, L., Tomassini, M., Collard, P., Vérel, S.: Negative Slope Coefficient: A Measure to Characterize Genetic Programming Fitness Landscapes. In: Collet, P., Tomassini, M., Ebner, M., Gustafson, S., Ekárt, A. (eds.) EuroGP 2006. LNCS, vol. 3905, pp. 178–189. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  70. Vanneschi, L., Valsecchi, A., Poli, R.: Limitations of the fitness-proportional negative slope coefficient as a difficulty measure. In: GECCO 2009, pp. 1877–1878. ACM, New York (2009)

    Chapter  Google Scholar 

  71. Vassilev, V., Fogarty, T., Miller, J.: Information characteristics and the structure of landscapes. Evol. Comput. 8(1), 31–60 (2000)

    Article  Google Scholar 

  72. Vassilev, V., Fogarty, T., Miller, J.: Smoothness, ruggedness and neutrality of fitness landscapes: from theory to application. In: Advances in Evolutionary Computing, pp. 3–44. Springer, New York (2003)

    Chapter  Google Scholar 

  73. Vassilevska, V., Williams, R., Woo, S.: Confronting hardness using a hybrid approach. In: Proc. 17th Ann. ACM-SIAM Symp. Discrete Algorithms, pp. 1–10. ACM, New York (2006)

    Google Scholar 

  74. Vrugt, J., Robinson, B., Hyman, J.: Self-Adaptive multimethod search for global optimization in Real-Parameter spaces. IEEE Trans. Evol. Comput. 13(2), 243–259 (2009)

    Article  Google Scholar 

  75. Watson, J., Beck, J., Howe, A., Whitley, L.: Problem difficulty for tabu search in job-shop scheduling. Artif. Intell. 143(2), 189–217 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  76. Weinberger, E.: Correlated and uncorrelated fitness landscapes and how to tell the difference. Biol. Cybern. 63(5), 325–336 (1990)

    Article  MATH  Google Scholar 

  77. Weinberger, E., Stadler, P.: Why some fitness landscapes are fractal. J. Theor. Biol. 163(2), 255–275 (1993)

    Article  Google Scholar 

  78. Wolpert, D., Macready, W.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997)

    Article  Google Scholar 

  79. Yeguas, E., Joan-Arinyo, R., Luzón, M.V.: Modeling the performance of evolutionary algorithms on the root identification problem: A case study with PBIL and CHC algorithms. Evol. Comput. 19(1), 107–135 (2011)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering, The University of Melbourne, Parkville, Victoria, 3010, Australia

    Mario A. Muñoz & Saman K. Halgamuge

  2. Department of Computing and Information Systems, The University of Melbourne, Parkville, Victoria, 3010, Australia

    Michael Kirley

Authors
  1. Mario A. Muñoz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Kirley
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Saman K. Halgamuge
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mario A. Muñoz .

Editor information

Editors and Affiliations

  1. Faculty of Computer Science, Otto-von-Guericke-University Magdeburg, Universitätsplatz 2, Magdeburg, 39106, Germany

    Christian Moewes

  2. , Faculty of Computer Science, Otto-von-Guericke University Magdeburg, Universitätsplatz 2, Magdeburg, 39106, Germany

    Andreas Nürnberger

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Muñoz, M.A., Kirley, M., Halgamuge, S.K. (2013). The Algorithm Selection Problem on the Continuous Optimization Domain. In: Moewes, C., Nürnberger, A. (eds) Computational Intelligence in Intelligent Data Analysis. Studies in Computational Intelligence, vol 445. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-32378-2_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-32378-2_6

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-32377-5

  • Online ISBN: 978-3-642-32378-2

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation