Skip to main content
SpringerLink
Log in

A Survey and Comparison of Performance Evaluation in Intrusion Detection Systems

  • Chapter
  • First Online:
Computer and Network Security Essentials

Abstract

Performance evaluation is an important aspect when designing a system. However, with intrusion detection systems (IDS), there are many other factors to consider. What are the metrics which are being used to compare the systems? Which attacks do particular approaches detect? Is the solution able to adapt and recognize new attacks, or is it limited to a set of attacks which were known at the time the system was designed? This chapter provides an overview of some of these concerns and tries to highlight in each surveyed IDS which metrics are used for performance evaluation, whether or not the solution is flexible, and which attacks the IDS is able to detect. This will provide the reader with a good basis for choosing the type of approach to use to guard against attacks, or as a basis to dig deeper into a particular aspect of intrusion detection.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Cup, K. (1999). Dataset. Available at the following website http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html

  2. Sharma, V., & Nema, A. (2013). Innovative genetic approach for intrusion detection by using decision tree. In 2013 international conference on communication systems and network technologies (pp. 418–422).

    Google Scholar 

  3. Lippmann, R., Haines, J. W., Fried, D. J., Korba, J., & Das, K. (2000). The 1999 DARPA off-line intrusion detection evaluation. Computer Networks, 34(4), 579–595.

    Article  Google Scholar 

  4. J. G. Elevate Communications (2016). Terabit-scale multi-vector DDoS attacks to become the new normal in 2017, Predict DDoS Experts, Business Wire.

    Google Scholar 

  5. García, S., Grill, M., Stiborek, J., & Zunino, A. (2014). An empirical comparison of botnet detection methods. Computers & Security, 45, 100–123.

    Article  Google Scholar 

  6. Małowidzki, M., Berezinski, P., & Mazur, M. (2015). Network intrusion detection: half a kingdom for a good dataset. In Proceedings of NATO STO SAS-139 Workshop. Portugal.

    Google Scholar 

  7. Scully, P. (2016). Where can I get the latest dataset for a network intrusion detection system?. Quora [Online]. Available: https://www.quora.com/Where-can-I-get-the-latest-dataset-for-a-network-intrusion-detection-system . Accessed January 12, 2017.

  8. ubershmekel (2012). Precision, recall, sensitivity and specificity. Ubershmekel’s Uberpython Pythonlog [Online]. Available: https://uberpython.wordpress.com/2012/01/01/precision-recall-sensitivity-and-specificity/ . Accessed February 09, 2017.

  9. Natesan, P., Balasubramanie, P., & Gowrison, G. (2012). Improving the attack detection rate in network intrusion detection using adaboost algorithm. Journal of Computer Science, 8(7), 1041–1048.

    Article  Google Scholar 

  10. Mo, Y., Ma, Y., & Xu, L. (2008). Design and implementation of intrusion detection based on mobile agents. In 2008 IEEE international symposium on IT in medicine and education (pp. 278–281).

    Google Scholar 

  11. Uppuluri, P., & Sekar, R. (2001). Experiences with specification-based intrusion detection. In Recent advances in intrusion detection (pp. 172–189).

    Google Scholar 

  12. Sekar, R. et al. (2002). Specification-based anomaly detection: A new approach for detecting network intrusions. In Proceedings of the 9th ACM conference on computer and communications security (pp. 265–274). Washington, DC, USA.

    Google Scholar 

  13. Shon, T., & Moon, J. (2007). A hybrid machine learning approach to network anomaly detection. Information Science, 177(18), 3799–3821.

    Article  Google Scholar 

  14. MeeraGandhi, G., Appavoo, K., & Srivasta, S. (2010). Effective network intrusion detection using classifiers decision trees and decision rules. International Journal Advanced network and Application, 2(3), 686–692.

    Google Scholar 

  15. Trinius, P., Willems, C., Holz, T., & Rieck, K. (2009). A malware instruction set for behavior-based analysis. Tech. Rep. TR-2009-07, University of Mannheim.

    Google Scholar 

  16. Xu, J., & Wu, S. (2010). Intrusion detection model of mobile agent based on Aglets. In 2010 international conference on computer application and system modeling (ICCASM 2010) (Vol. 4, pp. V4-347–V4-350).

    Google Scholar 

  17. Gong, Y., Mabu, S., Chen, C., Wang, Y., & Hirasawa, K. (2009). Intrusion detection system combining misuse detection and anomaly detection using Genetic Network Programming. ICCAS-SICE, 2009.

    Google Scholar 

  18. Yang, W., Wan, W., Guo, L., & Zhang L. J. (2007). An efficient intrusion detection model based on fast inductive learning. In 2007 international conference on machine learning and cybernetics (Vol. 6, pp. 3249–3254).

    Google Scholar 

  19. Lan, F., Chunlei, W., & Guoqing, M. (2010). A framework for network security situation awareness based on knowledge discovery. In 2nd international conference on computer engineering and technology (Vol. 1, pp. V1-226–V1-231).

    Google Scholar 

  20. Jaiganesh, V., Sumathi, P., & Mangayarkarasi, S. (2013). An analysis of intrusion detection system using back propagation neural network. In 2013 international conference on information communication and embedded systems (ICICES) (pp. 232–236).

    Google Scholar 

  21. Shanmugavadivu, R., & Nagarajan, N. (2011). Network intrusion detection system using fuzzy logic. Indian Journal of Computer Science and Engineering (IJCSE), 2(1), 101–111.

    Google Scholar 

  22. Sen, J. (2010). Efficient routing anomaly detection in wireless mesh networks. In 2010 first international conference on integrated intelligent computing (pp. 302–307).

    Google Scholar 

  23. Aggarwal, P., & Sharma, S. K. (2015). An empirical comparison of classifiers to analyze intrusion detection. In 2015 fifth international conference on advanced computing communication technologies (pp. 446–450).

    Google Scholar 

  24. Vyas, T., Prajapati, P., & Gadhwal, S. (2015). A survey and evaluation of supervised machine learning techniques for spam e-mail filtering. In 2015 IEEE international conference on electrical, computer and communication technologies (ICECCT) (pp. 1–7).

    Google Scholar 

  25. Rieck, K., Schwenk, G., Limmer, T., Holz, T., & Laskov, P. (2010). Botzilla: Detecting the phoning home of malicious software. In Proceedings of the 2010 ACM symposium on applied computing (pp. 1978–1984).

    Google Scholar 

  26. Lane, T. (2006). A decision-theoretic, semi-supervised model for intrusion detection. In M. A. Maloof (Ed.), Machine learning and data mining for computer security (pp. 157–177). London: Springer.

    Chapter  Google Scholar 

  27. Warrender, C., Forrest, S., & Pearlmutter, B. (1999). Detecting intrusions using system calls: alternative data models. In Proceedings of the 1999 IEEE symposium on security and privacy (Cat. No.99CB36344) (pp. 133–145).

    Google Scholar 

  28. Joo, D., Hong, T., & Han, I. (2003). The neural network models for IDS based on the asymmetric costs of false negative errors and false positive errors. Expert System with Applications, 25(1), 69–75.

    Article  Google Scholar 

  29. Kolias, C., Kambourakis, G., Stavrou, A., & Gritzalis, S. (2016). Intrusion detection in 802.11 networks: Empirical evaluation of threats and a public dataset. IEEE Communications Surveys Tutorials, 18(1), 184–208.

    Article  Google Scholar 

  30. Subramanian, U., & Ong, H. S. (2014). Analysis of the effect of clustering the training data in Naive Bayes classifier for anomaly network intrusion detection. Journal of Advances in Computer Networks, 2(1), 91–94.

    Article  Google Scholar 

  31. Casas, P., Mazel, J., & Owezarski, P. (2012). Unsupervised network intrusion detection systems: Detecting the unknown without knowledge. Computer Communications, 35(7), 772–783.

    Article  Google Scholar 

  32. Muzammil, M. J., Qazi, S., & Ali, T. (2013). Comparative analysis of classification algorithms performance for statistical based intrusion detection system. In 3rd IEEE international conference on computer, control and communication (IC4) (pp. 1–6).

    Google Scholar 

  33. Tan, Z., Jamdagni, A., He, X., Nanda, P., Liu, R. P., & Hu, J. (2015). Detection of denial-of-service attacks based on computer vision techniques. IEEE Transactions Computers, 64(9), 2519–2533.

    Article  MathSciNet  MATH  Google Scholar 

  34. Bhuse, V., & Gupta, A. (2006). Anomaly intrusion detection in wireless sensor networks. Journal of High Speed Networks, 15(1), 33–51.

    Google Scholar 

  35. Zhao, Y. J., Wei, M. J., & Wang, J. (2013). Realization of intrusion detection system based on the improved data mining technology. In 8th international conference on Computer Science and Education. Colombo, Sri Lanka.

    Google Scholar 

  36. Mahoney, M. V., & Chan, P. K. (2001). PHAD: Packet header anomaly detection for identifying hostile network traffic (Tech. Rep. CS-2001-4). Melbourne, FL: Florida Institute of Technology.

    Google Scholar 

  37. Sedjelmaci, H., & Senouci, S. M. (2015). An accurate and efficient collaborative intrusion detection framework to secure vehicular networks. Computers and Electrical Engineering, 43, 33–47.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Guelph, Guelph, ON, N1G 2W1, Canada

    Jason Ernst, Tarfa Hamed & Stefan Kremer

Authors
  1. Jason Ernst
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tarfa Hamed
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Kremer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jason Ernst .

Editor information

Editors and Affiliations

  1. University of Detroit Mercy, Detroit, Michigan, USA

    Kevin Daimi

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this chapter

Cite this chapter

Ernst, J., Hamed, T., Kremer, S. (2018). A Survey and Comparison of Performance Evaluation in Intrusion Detection Systems. In: Daimi, K. (eds) Computer and Network Security Essentials. Springer, Cham. https://doi.org/10.1007/978-3-319-58424-9_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-58424-9_32

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-58423-2

  • Online ISBN: 978-3-319-58424-9

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation