ABSTRACT
Classical Design of Experiment (DOE) techniques have been in use for many years to aid in the performance testing of systems. In particular fractional factorial designs have been used in cases with many numerical factors to reduce the number of experimental runs necessary. For experiments involving categorical factors, this is not the case; experimenters regularly resort to exhaustive (full factorial) experiments. Recently, D-optimal designs have been used to reduce numbers of tests for experiments involving categorical factors because of their flexibility, but not necessarily because they can closely approximate full factorial results. In commonly used statistical packages, the only generic alternative for reduced experiments involving categorical factors is afforded by optimal designs. The extent to which D-optimal designs succeed in estimating exhaustive results has not been evaluated, and it is natural to determine this. An alternative design based on covering arrays may offer a better approximation of full factorial data. Covering arrays are used in software testing for accurate coverage of interactions, while D-optimal and factorial designs measure the amount of interaction. Initial work involved exhaustive generation of designs in order to compare covering arrays and D-optimal designs in approximating full factorial designs. In that setting, covering arrays provided better approximations of full factorial analysis, while ensuring coverage of all small interactions. Here we examine commercially viable covering array and D-optimal design generators to compare designs. Commercial covering array generators, while not as good as exhaustively generated designs, remain competitive with D-optimal design generators.
- Alphaworks:IBM. Combinatorial test services tool. http://www.alphaworks.ibm.com/tech/cts.]]Google Scholar
- D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton. The AETG system: an approach to testing based on combinatorial design. IEEE Transactions on Software Engineering, 23(7):437--44, July 1997.]] Google ScholarDigital Library
- A. Hartman and L. Raskin. Problems and algorithms for covering arrays. Discrete Math, 284(3):149--156, Dec 2003.]]Google Scholar
- D. Hoskins, R. C. Turban, and C. J. Colbourn. Experimental designs in software engineering: D-optimal designs and covering arrays. In Proc. SIGSOFT 2004/Foundations on Software Engineering (FSE-12): Workshop on Interdisciplinary Software Engineering Research (WISER), pages 55--66. ACM, November 2004.]] Google ScholarDigital Library
- R. Meyer and C. Nachtsheim. The coordinate-exchange algorithm for constructing exact optimal experimental designs. Technometrics, 37(1):60--69, Feb 1995.]]Google ScholarCross Ref
- T. Mitchell. An algorithm for the construction of 'd-optimal' experimental designs. Technometrics, 42(1):48--54, Jan. 1974.]] Google ScholarDigital Library
- D. C. Montgomery. Design and Analysis of Experiments (Fifth Edition). John Wiley and Sons, New York NY, 2001.]] Google ScholarDigital Library
- R. C. Turban, C. J. Colbourn, and M. B. Cohen. A framework of greedy methods for constructing interaction test suites. In Proc. 27th International Conference on Software Engineering (ICSE2005), page to appear, May 2005.]] Google ScholarDigital Library
- K. Vadde and V. R. Syrotiuk. Factor interaction on service delivery in mobile ad hoc networks. IEEE Journal on Selected Areas in Communications, 22(7):1335-- 1346, Sept. 2004.]] Google ScholarDigital Library
Index Terms
- Software performance testing using covering arrays: efficient screening designs with categorical factors
Recommendations
Experimental designs in software engineering: d-optimal designs and covering arrays
WISER '04: Proceedings of the 2004 ACM workshop on Interdisciplinary software engineering researchFor over a century, Design of Experiment (DOE) techniques have been applied to testing in large problem domains such as agriculture, chemistry, medicine, and industrial design. Recently, the application of DOE has appeared in component-based software ...
Tower of covering arrays
Covering arrays are combinatorial objects that have several practical applications, specially in the design of experiments for software and hardware testing. A covering array of strength t and order v is an N í k array over Z v with the property that ...
A greedy-metaheuristic 3-stage approach to construct covering arrays
Highlights- A very effective 3-stage approach to construct covering arrays.
- A mixture of ...
AbstractCovering arrays are combinatorial designs used as test-suites in software and hardware testing. Because of their practical applications, the construction of covering arrays with a smaller number of rows is desirable. In this work we ...
Comments