Skip to main content
SpringerLink
Log in

Mapping Learning Algorithms on Data, a Promising Novel Methodology to Compare Learning Algorithms

  • Conference paper
  • First Online:
Computer and Communication Engineering (CCCE 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1823))

Included in the following conference series:

Abstract

The paper describes a novel methodology to compare learning algorithms by exploiting their performance maps. A performance map enhances the comparison of a learner across learning contexts and it also provides insights in the distribution of a learners’ performances across its parameter space. Also some initial empirical findings are commented.

In order to explain the novel comparison methodology, this study introduces the notions of learning context, performance map, and high performance function. These concepts are then applied to a variety of learning contexts to show how the methodology can be applied.

Finally, we will use meta-optimization as an instrument to improve the efficiency of the parameter space search with respect to its complete enumeration. But, note that meta-optimization is neither an essential part of our methodology nor the focus of our study.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    We projected two parameters on the X axis for creating the 3-d graphs. In particular, we projected ‘min impurity’ and ‘min samples’ on the X axis and ‘max depth’ on the Y axis for DT. Then the label ‘0.1 - 20’ on the X axis has to be interpreted as ‘min impurity’ = 0.2 and ‘min samples = 20’. Instead, for SVM, we projected ‘gamma’ and ‘C value’ on the X axis and ‘kernel’ on the Y axis.

References

  1. Bergmeir, C., Benítez, J.M.: On the use of cross-validation for time series predictor evaluation. Inf. Sci. 191, 192–213 (2012)

    Article  Google Scholar 

  2. Blum, C., Roli, A.: Metaheuristics in combinatorial optimization: overview and conceptual comparison. ACM Comput. Surv. 35(3), 268–308 (2003)

    Article  Google Scholar 

  3. Bradley, A.P.: The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern Recogn. 30(7), 1145–1159 (1997)

    Article  Google Scholar 

  4. Camilleri, M., Neri, F.: Parameter optimization in decision tree learning by using simple genetic algorithms. WSEAS Trans. Comput. 13, 582–591 (2014)

    Google Scholar 

  5. Camilleri, M., Neri, F., Papoutsidakis, M.: An algorithmic approach to parameter selection in machine learning using meta-optimization techniques. WSEAS Trans. Syst. 13(1), 203–212 (2014)

    Google Scholar 

  6. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)

    Google Scholar 

  7. Eiben, A., Hinterding, R., Michalewicz, Z.: Parameter control in evolutionary algorithms. IEEE Trans. Evol. Comput. 3(2), 124–141 (1999)

    Article  Google Scholar 

  8. Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning. TSSCML, pp. 3–33. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05318-5_1

    Chapter  Google Scholar 

  9. García-Margariño, I., Plaza, I., Neri, F.: ABS-mindburnout: an agent-based simulator of the effects of mindfulness-based interventions on job burnout. J. Comput. Sci. 36, 101012 (2019)

    Google Scholar 

  10. Goldberg, D.: Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, Reading, Ma (1989)

    MATH  Google Scholar 

  11. Grefenstette, J.J.: Optimization of control parameters for genetic algorithms. IEEE Trans. Syst. Man Cybern. 16(1), 122–128 (1986)

    Article  Google Scholar 

  12. Lorenzo, P.R., Nalepa, J., Kawulok, M., Ramos, L.S., Pastor, J.R.: Particle swarm optimization for hyper-parameter selection in deep neural networks. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 481–488. GECCO 2017. ACM (2017)

    Google Scholar 

  13. Marino, A., Neri, F.: PID tuning with neural networks. In: Nguyen, N.T., Gaol, F.L., Hong, T.-P., Trawiński, B. (eds.) ACIIDS 2019. LNCS (LNAI), vol. 11431, pp. 476–487. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-14799-0_41

    Chapter  Google Scholar 

  14. Neri, F.: Traffic packet based intrusion detection: decision trees and genetic based learning evaluation. WSEAS Trans. Comput. 4(9), 1017–1024 (2005)

    Google Scholar 

  15. Neri, F.: PIRR: a methodology for distributed network management in mobile networks. WSEAS Trans. Inf. Sci. Appl. 5(3), 306–311 (2008)

    Google Scholar 

  16. Neri, F.: Learning and predicting financial time series by combining natural computation and agent simulation. In: Di Chio, C., et al. (eds.) EvoApplications 2011. LNCS, vol. 6625, pp. 111–119. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20520-0_12

    Chapter  Google Scholar 

  17. Neri, F.: Agent-based modeling under partial and full knowledge learning settings to simulate financial markets. AI Commun. 25(4), 295–304 (2012)

    Article  Google Scholar 

  18. Neri, F.: A comparative study of a financial agent based simulator across learning scenarios. In: Cao, L., Bazzan, A.L.C., Symeonidis, A.L., Gorodetsky, V.I., Weiss, G., Yu, P.S. (eds.) ADMI 2011. LNCS (LNAI), vol. 7103, pp. 86–97. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-27609-5_7

    Chapter  Google Scholar 

  19. Neri, F.: Learning predictive models for financial time series by using agent based simulations. In: Nguyen, N.T. (ed.) Transactions on Computational Collective Intelligence VI. LNCS, vol. 7190, pp. 202–221. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29356-6_10

    Chapter  Google Scholar 

  20. Neri, F.: Coevolution and learning symbolic concepts: statistical validation: empirical statistical validation of co-evolutive machine learning systems. In: Neri, F. (ed.) 7th International Conference on Machine Learning Technologies ICMLT 2022, pp. 244–248. ACM (2022)

    Google Scholar 

  21. Neri, F.: Explainability and interpretability in agent based modelling to approximate market indexes. In: Neri, F. (ed.) 8th International Conference on Machine Learning Technologies ICMLT 2023, in press. ACM (2023)

    Google Scholar 

  22. Neri, F.: Explainability and interpretability in decision trees and agent based modelling when approximating financial time series, a matter of balance with performance. In: Neri, F. (ed.) 2023 8th International Conference on Computational Intelligence and Applications ICCIA 2023, in press. Springer press (2023)

    Google Scholar 

  23. Neri, F., Margariño, I.: Simulating and modeling the DAX index and the USO Etf financial time series by using a simple agent-based learning architecture. Expert Syst. 37(4), 12516 (2020)

    Google Scholar 

  24. Neri, F.: Software agents as a versatile simulation tool to model complex systems. WSEAS Trans. Info. Sci. and App. 7(5), 609–618 (2010)

    Google Scholar 

  25. Neri, F.: Case study on modeling the silver and nasdaq financial time series with simulated annealing. In: Rocha, Á., Adeli, H., Reis, L.P., Costanzo, S. (eds.) WorldCIST’18 2018. AISC, vol. 746, pp. 755–763. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77712-2_71

    Chapter  Google Scholar 

  26. Neri, F.: Combining machine learning and agent based modeling for gold price prediction. In: Cagnoni, S., Mordonini, M., Pecori, R., Roli, A., Villani, M. (eds.) WIVACE 2018. CCIS, vol. 900, pp. 91–100. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-21733-4_7

    Chapter  Google Scholar 

  27. Neri, F.: How to identify investor’s types in real financial markets by means of agent based simulation. In: 6th International Conference on Machine Learning Technologies ICMLT 2021, pp. 144–149. ACM (2021)

    Google Scholar 

  28. Neri, F.: Unpublished result: Domain specific concept drift detectors for predicting financial time series. https://arxiv.org/abs/2103.14079. Arxiv.org (UP)

  29. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  30. Quinlan, J.R.: C4.5: programs for machine learning. Morgan Kaufmann, California (1993)

    Google Scholar 

  31. Racine, J.: Consistent cross-validatory model-selection for dependent data: hv-block cross-validation. J. Econometr. 99(1), 39–61 (2000)

    Article  MATH  Google Scholar 

  32. Refaeilzadeh, P., Tang, L., Liu, H.: Cross-Validation, pp. 532–538. Springer, US, Boston, MA (2009). https://doi.org/10.1007/978-0-387-39940-9_565

  33. Reif, M., Shafait, F., Dengel, A.: Meta-learning for evolutionary parameter optimization of classifiers. Mach. Learn. 87(3), 357–380 (2012)

    Article  MathSciNet  Google Scholar 

  34. Schlimmer, J.C.: Concept acquisition through representational adjustment. Doctoral dissertation, Department of Information and Computer Science, University of California, Irvine, CA (1987)

    Google Scholar 

  35. Stone, M.: Cross-validatory choice and assessment of statistical predictions. discussion. J. Royal Statist. Soc. Ser. B 36, 111–147 (1974)

    Google Scholar 

  36. Wolpert, D., Macready, W.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. DIETI, University of Naples, via Claudio 21, 80100, Naples, Italy

    Filippo Neri

Authors
  1. Filippo Neri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Filippo Neri .

Editor information

Editors and Affiliations

  1. University of Naples Federico II, Naples, Italy

    Filippo Neri

  2. Concordia University, Montreal, QC, Canada

    Ke-Lin Du

  3. The University of New South Wales, Sydney, NSW, Australia

    Vijayakumar Varadarajan

  4. Miguel Hernández University of Elche, Elche, Spain

    Angel-Antonio San-Blas

  5. Northwestern Polytechnical University, Xi'an, China

    Zhiyu Jiang

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Neri, F. (2023). Mapping Learning Algorithms on Data, a Promising Novel Methodology to Compare Learning Algorithms. In: Neri, F., Du, KL., Varadarajan, V., San-Blas, AA., Jiang, Z. (eds) Computer and Communication Engineering. CCCE 2023. Communications in Computer and Information Science, vol 1823. Springer, Cham. https://doi.org/10.1007/978-3-031-35299-7_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-35299-7_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-35298-0

  • Online ISBN: 978-3-031-35299-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation