Pablo Pedemonte
ABSTRACT
As applications are developed, functional tests ensure they continue to function as expected. Nowadays, functional testing is mostly done manually, with human testers verifying a system's functionality themselves, following hand-written instructions. While there exist tools supporting functional test automation, in practice they are hard to use, require programming skills, and do not provide good support for test maintenance. In this paper, we take an alternative approach: we semi-automatically convert hand-written instructions into automated tests. Our approach consists of two stages: first, employing machine learning and natural language processing to compute an intermediate representation from test steps; and second, interactively disambiguating that representation to create a fully automated test. These two stages comprise a complete system for converting hand-written functional tests into automated tests. We also present a quantitative study analyzing the effectiveness of our approach. Our results show that 70% of manual test steps can be automatically converted to automated test steps with no user intervention.
- Abney, S. P. Parsing by chunks. In Principle-Based Parsing: Computation and Psycholinguistics (1991), 257--278.Google Scholar
Digital Library
- Branavan, S. R. K., Zettlemoyer, L. S., and Barzilay, R. Reading between the lines: learning to map high-level instructions to commands. In Proc. of the 48th Annual Meeting of the Association for Computational Linguistics, ACL '10 (2010), 1268--1277. Google ScholarDigital Library
- Brill, E. A simple rule-based part of speech tagger. In Proc. of the third conference on Applied natural language processing, ANLC '92 (1992), 152--155. Google ScholarDigital Library
- Chen, D. L., and Mooney, R. J. Learning to interpret natural language navigation instructions from observations. In Proc. of the Twenty-Fifth AAAI Conference on Artificial Intelligence (2011).Google ScholarDigital Library
- Cortes, C., and Vapnik, V. Support-vector networks. Mach. Learn. 20 (1995), 273--297. Google ScholarDigital Library
- Dustin, E., Rashka, J., and Paul, J. Automated software testing: introduction, management, and performance. Addison-Wesley Longman Publishing Co., Inc., 1999. Google ScholarDigital Library
- Fantechi, A., Gnesi, S., Lami, G., and Maccari, A. Application of linguistic techniques for use case analysis. In Proc. of the 10th Anniversary IEEE Joint intl. conf. on Requirements engineering, RE '02 (2002), 157--164. Google ScholarDigital Library
- Gouveia, D., Davis, C., Saracevic, F., Bocarsly, J., Chirillo, D., and Quesada, L. Software Test Engineering with IBM Rational Functional Tester: The Definitive Resource. IBM Press, 2009. Google ScholarDigital Library
- Kit, E., and Finzi, S. Software testing in the real world: improving the process. ACM Press/Addison-Wesley Publishing Co., 1995. Google ScholarDigital Library
- Kollar, T., Tellex, S., Roy, D., and Roy, N. Toward understanding natural language directions. In Proc. of the 5th ACM/IEEE intl. conf. on Human-robot interaction, HRI '10 (2010), 259--266. Google ScholarDigital Library
- Lafferty, J. D., McCallum, A., and Pereira, F. C. N. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proc. of the 18th intl. conf. on Machine learning, ICML '01 (2001), 282--289. Google ScholarDigital Library
- Lau, T., Drews, C., and Nichols, J. Interpreting written how-to instructions. In Proc. of the 21st intl. joint conf. on Artificial intelligence (2009), 1433--1438. Google ScholarDigital Library
- Leshed, G., Haber, E. M., Matthews, T., and Lau, T. CoScripter: automating & sharing how-to knowledge in the enterprise. In Proc. of the 25th annual SIGCHI conf. on Human factors in computing systems, CHI '08 (2008), 1719--1728. Google ScholarDigital Library
- Manber, U., and Myers, G. Suffix arrays: a new method for on-line string searches. SIAM J. Comput. 22 (1993), 935--948. Google ScholarDigital Library
- Ramshaw, L. A., and Marcus, M. P. Text chunking using transformation-based learning. In Proc. of the Third Annual Workshop on Very Large Corpora (1995), 82--94.Google Scholar
- Shimizu, N. Semantic discourse segmentation and labeling for route instructions. In Proc. of the 21st intl. conf. on Computational linguistics, COLING ACL '06 (2006), 31--36. Google ScholarDigital Library
- Shimizu, N., and Haas, A. Learning to follow navigational route instructions. In Proc. of the 21st intl. joint conf. on Artificial intelligence (2009), 1488--1493. Google ScholarDigital Library
- Sinha, A., Jr., S. M. S., and Paradkar, A. Text2Test: Automated inspection of natural language use cases. In Proc. of the 2010 3rd intl. conf. on Software Testing, Verification and Validation, ICST '10 (2010), 155--164. Google ScholarDigital Library
- Sinha, A., Paradkar, A. M., Kumanan, P., and Boguraev, B. A linguistic analysis engine for natural language use case description and its application to dependability analysis in industrial use cases. In DSN (2009), 327--336.Google Scholar
- Thummalapenta, S., Sinha, S., Mukherjee, D., and Chandra, S. Automating test automation. Tech. Rep. RI11015, IBM Research, 2011.Google Scholar
- Tichy, W. F., and Koerner, S. J. Text to software: developing tools to close the gaps in software engineering. In Proc. of the FSE/SDP workshop on Future of software engineering research, FoSER '10 (2010), 379--384. Google ScholarDigital Library
- Tjong Kim Sang, E. F., and Buchholz, S. Introduction to the CoNLL-2000 shared task: chunking. In Proc. of the 2nd workshop on Learning language in logic and the 4th conf. on Computational natural language learning Volume 7, ConLL '00 (2000), 127--132. Google ScholarDigital Library
Index Terms
- Towards automatic functional test execution
Recommendations
An Automatic Execution System for Web Functional Test Base on Modelling User's Behaviour
ISISE '08: Proceedings of the 2008 International Symposium on Information Science and Engieering - Volume 02Web applications are widely used. Generally, they are built by the integration of many loosely coupled and reusable services using open standards. It has become more and more difficult to testing web applications automatically. This paper presents a ...
Model based approach to assist test case creation, execution, and maintenance for test automation
ETSE '11: Proceedings of the First International Workshop on End-to-End Test Script EngineeringApplications, once developed, need to be maintained and tested as they undergo frequent changes. Test automation plays a significant role in testing activity, as it saves time and provides better utilization of resources. Test automation itself comes ...
Comments