Skip to main content
SpringerLink
Log in

Comparing Performances of Multiple Comparison Methods in Commonly Used 2 × C Contingency Tables

  • Original Research Article
  • Published:
Interdisciplinary Sciences: Computational Life Sciences Aims and scope Submit manuscript

Abstract

This study aims at mentioning briefly multiple comparison methods such as Bonferroni, Holm–Bonferroni, Hochberg, Hommel, Marascuilo, Tukey, Benjamini–Hochberg and Gavrilov–Benjamini–Sarkar for contingency tables, through the data obtained from a medical research and examining their performances by simulation study which was constructed as the total 36 scenarios to 2 × 4 contingency table. As results of simulation, it was observed that when the sample size is more than 100, the methods which can preserve the nominal alpha level are Gavrilov–Benjamini–Sarkar, Holm–Bonferroni and Bonferroni. Marascuilo method was found to be a more conservative than Bonferroni. It was found that Type I error rate for Hommel method is around 2 % in all scenarios. Moreover, when the proportions of the three populations are equal and the proportion value of the fourth population is far at a level of ±3 standard deviation from the other populations, the power value for Unadjusted All-Pairwise Comparison approach is at least a bit higher than the ones obtained by Gavrilov–Benjamini–Sarkar, Holm–Bonferroni and Bonferroni. Consequently, Gavrilov–Benjamini–Sarkar and Holm–Bonferroni methods have the best performance according to simulation. Hommel and Marascuilo methods are not recommended to be used because they have medium or lower performance. In addition, we have written a Minitab macro about multiple comparisons for use in scientific research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Zar JH (1999) Biostatistical analysis, 4th edn. Prentice-Hall, Upper Saddle River, NJ

    Google Scholar 

  2. Elliott AC, Reisch JS (2006) Implementing a multiple comparison test for proportions in a 2×c crosstabulation in SAS®. Proceedings of the 31st annual SAS users group international conference. SAS Institute Inc, Cary, NC

  3. Gavrilov Y, Benjamini Y, Sarkar SK (2009) An adaptive step-down procedure with proven FDR control under independence. Ann Stat 37:619–629

    Article  Google Scholar 

  4. Blakesley RE, Mazumdar S, Dew MA, Houck PR, Tang G, Reynolds CF 3rd et al (2009) Comparisons of methods for multiple hypothesis testing in neuropsychological research. Neuropsychology 23:255–264. doi:10.1037/a0012850

    Article  PubMed  PubMed Central  Google Scholar 

  5. Oden N, VanVeldhuisen PC, Ingrid US, Ip MS, SCORE Study Investigator Group (2010) SCORE Study report 8: closed tests for all pair-wise comparisons of means. Drug Inf J 44:405–420

  6. Michael GA (2007) A significance test of interaction in 2 × K designs with proportions. Tutor Quant Methods Psychol 3:1–7

    Article  Google Scholar 

  7. Ryan TA (1960) Significance tests for multiple comparison of proportions, variances and other statistics. Psychol Bull 57:318–328

    Article  CAS  PubMed  Google Scholar 

  8. Marascuilo L (1966) Large-sample multiple comparisons. Psychol Bull 65:280–290

    Article  CAS  PubMed  Google Scholar 

  9. Holland BS, Copenhaver MD (1988) Improved Bonferroni-type multiple testing procedures. Psychol Bull 104:145–149

    Article  Google Scholar 

  10. Westfall PH, Young SS (1989) P value adjustments for multiple tests in multivariate binomial models. J Am Stat Assoc 84:780–786

    Google Scholar 

  11. Kim SB, Tsui KL, Borodovsky M (2006) Multiple hypothesis testing in large-scale contingency tables: inferring patterns of pair-wise amino acid association in β-sheets. Int J Bioinform Res Appl 2:193–217

    Article  CAS  PubMed  Google Scholar 

  12. Horne J, Plaehn D (2007) Multiple comparisons on 2×c proportions. SAS conference proceedings: Pacific Northwest SAS Users Group 2007. Pacific Northwest SAS Users Group, Seattle, WA. www.pnwsug.org/content/multiple-comparisons-2xc-proportions. Accessed March 2014

  13. Lancaster MB (1949) The derivation and partition of χ2 in certain discrete distributions. Biometrika 36:117–129

    Article  CAS  PubMed  Google Scholar 

  14. Haberman SJ (1973) The analysis of residuals in cross-classified tables. Biometrics 29:205–220

    Article  Google Scholar 

  15. Hsu JC (1996) Multiple comparisons: theory and methods. CRC Press, Boca Raton

    Book  Google Scholar 

  16. Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc B 57:289–300

    Google Scholar 

  17. Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6:65–70

    Google Scholar 

  18. Hochberg Y (1988) A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75:800–802

    Article  Google Scholar 

  19. Hommel G (1988) A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75:383–386

    Article  Google Scholar 

  20. Cangur S, Ankarali H (2014) Examining the probabilities of type i error for unadjusted all pairwise comparisons and Bonferroni adjustment approaches in hypothesis testing for proportions. Int J Stat Med Res 3(4):404–411. doi:10.6000/1929-6029.2014.03.04.9

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Biostatistics and Medical Informatics, Faculty of Medicine, Duzce University, Duzce, Turkey

    Sengul Cangur, Handan Ankarali & Ozge Pasin

Authors
  1. Sengul Cangur
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Handan Ankarali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ozge Pasin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sengul Cangur.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 72 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cangur, S., Ankarali, H. & Pasin, O. Comparing Performances of Multiple Comparison Methods in Commonly Used 2 × C Contingency Tables. Interdiscip Sci Comput Life Sci 8, 337–345 (2016). https://doi.org/10.1007/s12539-015-0128-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12539-015-0128-5

Keywords

Search

Navigation