Skip to main content
SpringerLink
Log in

A two-stage algorithm for combinatorial testing

  • Original Paper
  • Published:
Optimization Letters Aims and scope Submit manuscript

Abstract

Covering arrays are combinatorial structures which have applications in fields like software testing and hardware Trojan detection. In this paper we proposed a two-stage simulated annealing algorithm to construct covering arrays. The proposed algorithm is instanced in this paper through the construction of ternary covering arrays of strength three. We were able to get 579 new upper bounds. In order to show the generality of our proposal, we defined a new benchmark composed of 25 instances of MCAs taken from the literature, all instances were improved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. For a complete specification of these parameters, see: http://developer.android.com/reference/android/content/res/Configuration.html.

References

  1. Ansótegui, C., Izquierdo-Marquez, I., Manya, F., Torres-Jimenez, J.: A Max-SAT-Based approach to constructing optimal covering arrays. Artificial intelligence research and development. CCIA 2013, 51–59 (2013)

    Google Scholar 

  2. Avila-George, H., Torres-Jimenez, J., Gonzalez-Hernandez, L., Hernández, V.: Metaheuristic approach for constructing functional test-suites. IET Softw. 7(2), 104–117 (2013)

    Article  Google Scholar 

  3. Avila-George, H., Torres-Jimenez, J., Hernández, V., Gonzalez-Hernandez, L.: Simulated annealing for constructing mixed covering arrays. In: Distributed computing and artificial intelligence, pp. 657–664 (2012)

  4. Bush, K.A.: Orthogonal arrays of index unity. Ann. Math. Stat. 23(3), 426–434 (1952)

    Article  MathSciNet  MATH  Google Scholar 

  5. Calvagna, A., Gargantini, A.: T-wise combinatorial interaction test suites construction based on coverage inheritance. Softw. Test. Verif. Reliab. 22, 507–526 (2012)

    Article  Google Scholar 

  6. Cawse, J.N.: Experimental design for combinatorial and high throughput materials development. Wiley, New York (2003)

    Google Scholar 

  7. Chateauneuf, M., Kreher, D.L.: On the state of strength-three covering arrays. J. Comb. Des. 10(4), 217–238 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  8. Cohen, D.M., Dalal, S.R., Parelius, J., Patton, G.C.: The combinatorial design approach to automatic test generation. IEEE Soft. 13(5), 83–88 (1996)

    Article  Google Scholar 

  9. Cohen, M.B., Colbourn, C.J., Ling, A.C.H.: Augmenting simulated annealing to build interaction test suites. In: Software reliability engineering, 2003. ISSRE 2003. 14th international symposium on IEEE, pp. 394–405. (2003)

  10. Cohen, M.B., Colbourn, C.J., Ling, A.C.H.: Constructing strength three covering arrays with augmented annealing. Discret. Math. 308(13), 2709–2722 (2008). (Combinatorial Designs: A tribute to Jennifer Seberry on her 60th Birthday)

    Article  MathSciNet  MATH  Google Scholar 

  11. Colbourn, C.J.: Covering arrays from cyclotomy. Des. Codes Cryptogr. 55(2–3), 201–219 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  12. Colbourn, C.J. (2015). Covering array tables. http://www.public.asu.edu/~ccolbou/src/tabby/3-3-ca.html. Accessed on 11 Mar 2015

  13. Colbourn, C.J., Martirosyan, S.S., Trung, T., Walker II, R.A.: Roux-type constructions for covering arrays of strengths three and four. Des. Codes Cryptogr. 41(1), 33–57 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  14. Duan, F., Lei, Y., Yu, L., Kacker, R.N., Kuhn, D.R.: Improving IPOG’s vertical growth based on a graph coloring scheme. In: 2015 IEEE Eighth international conference on software testing, verification and validation workshops (ICSTW), pp. 1–8 (2015)

  15. Flottes, M.-L., Dupuis, S., Ba, P.-S., Rouzeyre, B.: On the limitations of logic testing for detecting hardware trojans horses. In: Design technology of integrated systems in nanoscale era (DTIS), 2015 10th international conference on, pp. 1–5 (2015)

  16. Forbes, M., Lawrence, J., Lei, Y., Kacker, R.N., Kuhn, D.R.: Refining the in-parameter-order strategy for constructing covering arrays. J. Res. Natl. Inst. Stand. Technol. 113(5), 287–297 (2008)

    Article  Google Scholar 

  17. Hartman, A.: Software and hardware testing using combinatorial covering suites. In: Graph theory, combinatorics and algorithms, vol. 34 of operations research/computer science interfaces series, pp. 237–266 (2005)

  18. Kitsos, P., Simos, D.E., Torres-Jimenez, J., Voyiatzis, A.G.: Exciting FPGA cryptographic trojans using combinatorial testing. In: The 26th IEEE international symposium on software reliability engineering (2015) (To appear)

  19. Kuhn, D.R., Kacker, R.N., Lei, Y.: Practical combinatorial testing. Technical report, National Institute of Standards and Technology (2010)

  20. Martinez-Pena, J., Torres-Jimenez, J.: A branch and bound algorithm for ternary covering arrays construction using trinomial coefficients. Res. Comput. Sci. 49, 61–71 (2010)

    Google Scholar 

  21. Meagher, K., Stevens, B.: Group construction of covering arrays. J. Comb. Des. 13(1), 70–77 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  22. Shasha, D.E., Kouranov, A.Y., Lejay, L.V., Chou, M.F., Coruzzi, G.M.: Using combinatorial design to study regulation by multiple input signals: A tool for parsimony in the post-genomics era. Plant Physiol. 127(4), 1590–1594 (2001)

    Article  Google Scholar 

  23. Sherwood, G. (2015). On the construction of orthogonal arrays and covering arrays using permutation groups. http://testcover.com/pub/background/cover.htm. Accessed 20 Mar 2015

  24. Shiba, T., Tsuchiya, T., Kikuno, T.: Using artificial life techniques to generate test cases for combinatorial testing. In: Proceedings of the 28th annual international computer software and applications Conference, IEEE Computer Society, pp. 72–77 (2004)

  25. Sloane, N.J.A.: Covering arrays and intersecting codes. J. Comb. Des. 1(1), 51–63 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  26. Torres-Jimenez, J., Avila-George, H., Rangel-Valdez, N., Gonzalez-Hernandez, L.: Construction of orthogonal arrays of index unity using logarithm tables for galois fields, chapter 4. InTech, pp. 71–90 (2012)

  27. Torres-Jimenez, J., Rodriguez-Tello, E.: New bounds for binary covering arrays using simulated annealing. Inf. Sci. 185(1), 137–152 (2012)

    Article  Google Scholar 

  28. Walker II, R.A., Colbourn, C.J.: Tabu search for covering arrays using permutation vectors. J. Stat. Plan. Inference 139(1), 69–80 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  29. Younis, M.I., Zamli, K.Z.: MIPOG - An Efficient t-Way Minimization Strategy for Combinatorial Testing. Int. J. Comput. Theory Eng. 3(3), 388–397 (2011)

    Article  Google Scholar 

Download references

Acknowledgments

The authors acknowledge to ABACUS-CINVESTAV, CONACYT grant EDOMEX-2011-COI-165873 for providing access of high performance computing. The authors acknowledge General Coordination of Information and Communications Technologies (CGSTIC) at CINVESTAV for providing HPC resources on the Hybrid Cluster Supercomputer “Xiuhcoatl”, that have contributed to the research results reported. The following projects have funded the research reported in this paper: 148784—FOMIX—Unidad de Transferencia Tecnologica CICESE—Nayarit; 238469—CONACyT Métodos Exactos para Construir Covering Arrays Óptimos; 2143—Cátedras CONACyT—Fortalecimiento de las capacidades de TICs en Nayarit.

Author information

Authors and Affiliations

  1. Information Technology Laboratory, CINVESTAV-Tamaulipas, Ciudad Victoria, Tamaulipas, Mexico

    Jose Torres-Jimenez & Idelfonso Izquierdo-Marquez

  2. HARAMARA TIC-LAB, CICESE-UT³, Andador 10, Ciudad del conocimiento, Tepic, Nayarit, Mexico

    Himer Avila-George

Authors
  1. Jose Torres-Jimenez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Himer Avila-George
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Idelfonso Izquierdo-Marquez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jose Torres-Jimenez.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Torres-Jimenez, J., Avila-George, H. & Izquierdo-Marquez, I. A two-stage algorithm for combinatorial testing. Optim Lett 11, 457–469 (2017). https://doi.org/10.1007/s11590-016-1012-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11590-016-1012-x

Keywords

Search

Navigation