Skip to main content
SpringerLink
Log in

Artificial neural networks as multi-networks automated test oracle

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

One of the important issues in software testing is to provide an automated test oracle. Test oracles are reliable sources of how the software under test must operate. In particular, they are used to evaluate the actual results produced by the software. However, in order to generate an automated test oracle, it is necessary to map the input domain to the output domain automatically. In this paper, Multi-Networks Oracles based on Artificial Neural Networks are introduced to handle the mapping automatically. They are an enhanced version of previous ANN-Based Oracles. The proposed model was evaluated by a framework provided by mutation testing and applied to test two industry-sized case studies. In particular, a mutated version of each case study was provided and injected with some faults. Then, a fault-free version of it was developed as a Golden Version to evaluate the capability of the proposed oracle finding the injected faults. Meanwhile, the quality of the proposed oracle is measured by assessing its accuracy, precision, misclassification error and recall. Furthermore, the results of the proposed oracle are compared with former ANN-based Oracles. Accuracy of the proposed oracle was up to 98.93%, and the oracle detected up to 98% of the injected faults. The results of the study show the proposed oracle has better quality and applicability than the previous model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aggarwal, K.K., Singh, Y., Kaur, A., Sangwan, O.P.: A neural net based approach to test oracle. ACM SIGSOFT Softw. Eng. Notes 29(3), 1–6 (2004)

    Article  Google Scholar 

  • Ammann, P., Offutt, J.: Introduction to Software Testing, 1st edn. Cambridge University Press, New York (2008)

    MATH  Google Scholar 

  • Bourlard, H.A., Morgan, N.: Connectionist Speech Recognition: A Hybrid Approach. Kluwer Academic, Norwel (1994)

    Google Scholar 

  • Briand, L.C., Labiche, Y., He, S.: Automating regression test selection based on UML designs. Inf. Softw. Technol. 51(1), 16–30 (2009). doi:10.1016/j.infsof.2008.09.010

    Article  Google Scholar 

  • Hall, R.J.: A method and tools for large scale scenarios. Autom. Softw. Eng. 15(2), 113–148 (2008). doi:10.1007/s10515-008-0026-8

    Article  Google Scholar 

  • Heiat, A.: Comparison of artificial neural network and regression models for estimating software development effort. Inf. Softw. Technol. 44(15), 911–922 (2002). doi:10.1016/s0950-5849(02)00128-3

    Article  Google Scholar 

  • Jin, H., Wang, Y., Chen, N.-W., Gou, Z.-J., Wang, S.: Artificial neural network for automatic test oracles generation. Paper presented at the Proceedings of the 2008 International Conference on Computer Science and Software Engineering, Wuhan, Hubei, 12–14 Dec. (2008)

  • Jorgensen, P.C.: Software Testing: A Craftsman’s Approach, 2nd edn. CRC Press LLC, Boca Raton (2002)

    Google Scholar 

  • Kessentini, M.K.M., Sahraoui, H., Boukadoum, M.: Example-based model-transformation testing. Autom. Softw. Eng. 18(2), 199–224 (2011a). doi:10.1007/s10515-010-0079-3

    Article  Google Scholar 

  • Kessentini, M., Sahraoui, H., Boukadoum, M.: Example-based model-transformation testing. Autom. Softw. Eng. 18(2), 199–224 (2011b). doi:10.1007/s10515-010-0079-3

    Article  Google Scholar 

  • Khoshgoftaar, T.M., Pandya, A.S., More, H.B.: A neural network approach for predicting software development faults. In: Pandya, A.S. (ed.) Third International Symposium on Software Reliability Engineering, Research Triangle Park, NC, pp. 83–89 (1992)

    Chapter  Google Scholar 

  • Khoshgoftaar, T.M., Szabo, R.M., Guasti, P.J.: Exploring the behavior of neural network software quality models. Softw. Eng. J. 10(3), 89–96 (1995)

    Article  Google Scholar 

  • Khoshgoftaar, T.M., Allen, E.B., Xu, Z.: Predicting testability of program modules using a neural network. In: Allen, E.B. (ed.) 3rd IEEE Symposium on Application-Specific Systems and Software Engineering Technology, Richardson, TX, USA, pp. 57–62 (2000)

    Chapter  Google Scholar 

  • Last, M., Freidman, M.: Black-box testing with info-fuzzy networks. In: Last, M., Kandel, A., Bunke, H. (eds.) Artificial Intelligence Methods in Software Testing. Series in Machine Perceprion and Artificial Intelligence, vol. 56, pp. 21–50. World Scientific, Singapore (2004)

    Chapter  Google Scholar 

  • Last, M., Friendman, M., Kandel, A.: Using data mining for automated software testing. Int. J. Softw. Eng. Knowl. Eng. 14(4), 369–393 (2004)

    Article  Google Scholar 

  • Lu, Y., Ye, M.: Oracle model based on RBF neural networks for automated software testing. Inf. Technol. J. 6(3), 469–474 (2007)

    Article  Google Scholar 

  • Manolache, L.I., Kourie, D.G.: Software testing using model programs. Softw. Pract. Exp. 31(13), 1211–1236 (2001)

    Article  MATH  Google Scholar 

  • Mao, Y., Boqin, F., Li, Z., Yao, L.: Neural networks based automated test oracle for software testing. In: Proceedings of the 13th International Conference Neural Information Processing, Hong Kong, China, 2006. Lecture Notes in Computer Science, pp. 498–507. Springer, Heidelberg (2006)

    Google Scholar 

  • McCaffrey, J.D.: Software Testing: Fundamental Principles and Essential Knowledge. BookSurge Publishing, Charleston (2009)

    Google Scholar 

  • Memon, A.M.: Automated GUI regression testing using AI planning. In: Last, M., Kandel, A., Bunke, H. (eds.) Atificial Intelligence Methods in Software Testing. Series in Machine Perception and Artificial Intelligence, vol. 56, pp. 51–99. World Scientific, Singapore (2004)

    Chapter  Google Scholar 

  • Memon, A.M., Pollack, M.E., Soffa, M.L.: Automated test oracles for GUIs. SIGSOFT Softw. Eng. Notes 25(6), 30–39 (2000). http://doi.acm.org/10.1145/357474.355050

    Article  Google Scholar 

  • Memon, A., Nagarajan, A., Xie, Q.: Automating regression testing for evolving GUI software. J. Softw. Maint. Evol. 17(1 Spec Iss.), 27–64 (2005)

    Article  Google Scholar 

  • Menhaj, M.B.: Basics of Neural Networks, 1st edn. Amirkabir Technology University, Tehran (2001)

    Google Scholar 

  • Michael, C., McGraw, G.: Automated software test data generation for complex programs. Paper presented at the Proceedings of the 13th IEEE International Conference on Automated Software Engineering, Honolulu, Hawaii, October (1998)

  • Michael, C.C., McGraw, G., Schatz, M.A.: Generating software test data by evolution. IEEE Trans. Softw. Eng. 27(12), 1085–1110 (2001)

    Article  Google Scholar 

  • Myers, G.J.: The Art of Software Testing, 2nd edn. Wiley, New York (2004)

    Google Scholar 

  • Ntafos, S.C.: On comparisons of random, partition, and proportional partition testing. IEEE Trans. Softw. Eng. 27(10), 949–960 (2001). doi:10.1109/32.962563

    Article  Google Scholar 

  • Patton, R.: Software Testing, 2nd edn. Sams Publishing, Indianapolis (2005)

    Google Scholar 

  • Peters, D., Parnas, D.L.: Generating a test oracle from program documentation. In: Proceedings of the 1994 ACM SIGSOFT International Symposium on Software Testing and Analysis, Seattle, WA, USA, p. 58. ACM, New York (1994)

    Google Scholar 

  • Peters, D.K., Parnas, D.L.: Using test oracles generated from program documentation. IEEE Trans. Softw. Eng. 24(3), 161–173 (1998)

    Article  Google Scholar 

  • Pfleeger, S.L., Hatton, L.: Investigating the influence of formal methods. Computer 30(2), 33–43 (1997). doi:10.1109/2.566148

    Article  Google Scholar 

  • Ran, L., Dyreson, C., Andrews, A.A., Bryce, R., Mallery, C.: Building test cases and oracles to automate the testing of web database applications. Inf. Softw. Technol. 51(2), 460–477 (2009). doi:10.1016/j.infsof.2008.05.016

    Article  Google Scholar 

  • Richardson, D.J., Aha, S.L., O’Malley, T.O.: Specification-based test oracles for reactive systems. In: Proceedings of the 14th International Conference on Software Engineering, Melbourne, Australia, pp. 105–118. ACM, New York (1992)

    Google Scholar 

  • Saraph, P., Last, M., Kandell, A.: Test case generation and reduction by automated input-output analysis. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Washington DC, United States, pp. 768–773. IEEE, New York (2003)

    Google Scholar 

  • Schalkoff, R.J.: Artificial Neural Networks. McGraw-Hill, New York (1997)

    MATH  Google Scholar 

  • Schroeder, P.J., Korel, B.: Black-box test reduction using input-output analysis. SIGSOFT Softw. Eng. Notes 25(5), 173–177 (2000). http://doi.acm.org/10.1145/347636.349042

    Article  Google Scholar 

  • Schroeder, P.J., Faherty, P., Korel, B.: Generating expected results for automated black-box testing. In: Proceedings 17th IEEE International Conference on Automated Software Engineering ASE, Edinburgh, UK, pp. 139–148 (2002)

    Chapter  Google Scholar 

  • Shahamiri, S.R., Wan, M.N.W.K.: Intelligent and automated software testing methods classification. In: 4th Postgraduate Annual Research Seminar (PARS’08), UTM Skudai, Malaysia, 1–3 July 2008 (2008)

    Google Scholar 

  • Shahamiri, S.R., Kadir, W.M.N.W., Mohd-Hashim, S.Z.: A comparative study on automated software test oracle methods. In: Proceedings of the 2009 Fourth International Conference on Software Engineering Advances, Porto, Portugal, September 20–25, 2009, pp. 140–145. IEEE Computer Society, Los Alamitos (2009)

    Chapter  Google Scholar 

  • Shahamiri, S.R., Kadir, W.M.N.W., Ibrahim, S.: An automated oracle approach to test decision-making structures. In: Proceedings of the 2010 3rd IEEE International Conference on Computer Science and Information Technology, Chengdu, China, July 9–11, 2010, pp. 30–34. IEEE, New York (2010a)

    Chapter  Google Scholar 

  • Shahamiri, S.R., Kadir, W.M.N.W., Ibrahim, S.: A single-network ANN-based oracle to verify logical software modules. In: Proceedings of the 2010 2nd International Conference on Software Technology and Engineering, Puerto Rico, USA, October 3–5, 2010, pp. 272–276. IEEE, New York (2010b)

    Google Scholar 

  • Shahamiri, S.R., Kadir, W.M.N.W., Ibrahim, S., Hashim, S.Z.M.: An automated framework for software test oracle. Inf. Softw. Technol. 53(7), 774–788 (2011). doi:10.1016/j.infsof.2011.02.006

    Article  Google Scholar 

  • Sofokleous, A.A., Andreou, A.S.: Automatic evolutionary test data generation for dynamic software testing. J. Syst. Softw. 81(11), 1883–1898 (2008). doi:10.1016/j.jss.2007.12.809

    Article  Google Scholar 

  • Spillner, A., Linz, T., Schaefer, H.: Software Testing Foundations, 2nd edn. Rocky Nook Inc, Santa Barbara (2007)

    Google Scholar 

  • Stocks, P., Carrington, D.: A framework for specification-based testing. IEEE Trans. Softw. Eng. 22(11), 777–793 (1996). doi:10.1109/32.553698

    Article  Google Scholar 

  • Su, Y., Huang, C.: Neural-network-based approaches for software reliability estimation using dynamic weighted combinational models. J. Syst. Softw. 80(4), 606–615 (2007)

    Article  MathSciNet  Google Scholar 

  • Sun, Y., Bosch, L.T., Boves, L.: Hybrid HMM/BLSTM-RNN for robust speech recognition. Paper presented at the Proceedings of the 13th International Conference on Text, Speech and Dialogue, Brno, Czech Republic (2010)

  • Vanmali, M., Last, M., Kandel, A.: Using a neural network in the software testing process. Int. J. Intell. Syst. 17(1), 45–62 (2002)

    Article  MATH  Google Scholar 

  • Whittaker, J.A.: What is software testing? And why is it so hard? IEEE Softw. 17(1), 70–79 (2000)

    Article  Google Scholar 

  • Woodward, M.R.: Mutation testing—its origin and evolution. Inf. Softw. Technol. 35(3), 163–169 (1993). doi:10.1016/0950-5849(93)90053-6

    Article  Google Scholar 

  • Woodward, M.R., Hennell, M.A.: On the relationship between two control-flow coverage criteria: all JJ-paths and MCDC. Inf. Softw. Technol. 48(7), 433–440 (2006). doi:10.1016/j.infsof.2005.05.003

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Software Engineering, Faculty of Computer Science and Information Technology, University of Malaya, 50603, Kuala Lumpur, Malaysia

    Seyed Reza Shahamiri

  2. Department of Software Engineering, Faculty of Computer Science and Information Systems, Universiti Teknologi Malaysia, 81310, UTM Skudai, Johor, Malaysia

    Wan M. N. Wan-Kadir, Suhaimi Ibrahim & Siti Zaiton Mohd Hashim

Authors
  1. Seyed Reza Shahamiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wan M. N. Wan-Kadir
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Suhaimi Ibrahim
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Siti Zaiton Mohd Hashim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seyed Reza Shahamiri.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Shahamiri, S.R., Wan-Kadir, W.M.N., Ibrahim, S. et al. Artificial neural networks as multi-networks automated test oracle. Autom Softw Eng 19, 303–334 (2012). https://doi.org/10.1007/s10515-011-0094-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10515-011-0094-z

Keywords

Search

Navigation