Skip to main content
SpringerLink
Log in

Dynamic Bayesian Combination of Multiple Imperfect Classifiers

  • Chapter
Decision Making and Imperfection

Part of the book series: Studies in Computational Intelligence ((SCI,volume 474))

Abstract

Classifier combination methods need to make best use of the outputs of multiple, imperfect classifiers to enable higher accuracy classifications. In many situations, such as when human decisions need to be combined, the base decisions can vary enormously in reliability. A Bayesian approach to such uncertain combination allows us to infer the differences in performance between individuals and to incorporate any available prior knowledge about their abilities when training data is sparse. In this chapter we explore Bayesian classifier combination, using the computationally efficient framework of variational Bayesian inference. We apply the approach to real data from a large citizen science project, Galaxy Zoo Supernovae, and show that our method far outperforms other established approaches to imperfect decision combination.We go on to analyse the putative community structure of the decision makers, based on their inferred decision making strategies, and show that natural groupings are formed. Finally we present a dynamic Bayesian classifier combination approach and investigate the changes in base classifier performance over time.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Abbeel, P., Ng, A.Y.: Apprenticeship learning via inverse reinforcement learning. In: Proceedings of the Twenty-First International Conference on Machine Learning, ICML 2004, p. 1. ACM, New York (2004), http://doi.acm.org/10.1145/1015330.1015430

    Google Scholar 

  2. Abramowitz, M., Stegun, I.A. (eds.): Handbook of Mathematical Functions: with Formulas, Graphs, and Mathematical Tables. Dover Publications (1965)

    Google Scholar 

  3. Attias, H.: A Variational Bayesian Framework for Graphical Models. In: Advances in Neural Information Processing Systems 12, pp. 209–215 (2000)

    Google Scholar 

  4. Bierman, G.J.: Fixed interval smoothing with discrete measurements. International Journal of Control 18(1), 65–75 (1973), http://www.tandfonline.com/doi/abs/10.1080/00207177308932487

    Google Scholar 

  5. Bishop, C.M.: Pattern recognition and machine learning, 4th edn. Information Science and Statistics. Springer Science+Business Media, LLC (2006)

    Google Scholar 

  6. Bloodgood, M., Callison-Burch, C.: Using Mechanical Turk to build machine translation evaluation sets. In: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazons Mechanical Turk, Los Angeles, pp. 208–211 (2010)

    Google Scholar 

  7. Choudrey, R., Roberts, S.: Variational mixture of Bayesian independent component analysers. Neural Computation 15(1) (2003)

    Google Scholar 

  8. Dash, R.K., Jennings, N.R., Parkes, D.C.: Computational-mechanism design: a call to arms. IEEE Intelligent Systems 18(6), 40–47 (2003)

    Article  Google Scholar 

  9. Dawid, A.P., Skene, A.M.: Maximum likelihood estimation of observer Error-Rates using the EM algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics) 28(1), 20–28 (1979), http://www.jstor.org/stable/2346806

    Google Scholar 

  10. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological) 39(1), 1–38 (1977)

    MathSciNet  MATH  Google Scholar 

  11. Fawcett, T.: An introduction to ROC analysis. Pattern Recogn. Lett. 27(8), 861–874 (2006), http://dx.doi.org/10.1016/j.patrec.2005.10.010 , doi:10.1016/j.patrec.2005.10.010

    Article  MathSciNet  Google Scholar 

  12. Fox, C., Roberts, S.: A tutorial on variational Bayesian inference. Artificial Intelligence Review 38(2), 85–95 (2011), http://www.springerlink.com/content/j5617xn756j27671/abstract/ , doi:10.1007/s10462-011-9236-8

    Article  Google Scholar 

  13. Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-6(6), 721–741 (1984)

    Google Scholar 

  14. Ghahramani, Z., Kim, H.C.: Bayesian classifier combination. Gatsby Computational Neuroscience Unit Technical Report GCNU-T, London, UK (2003)

    Google Scholar 

  15. Gilks, W.R., Wild, P.: Adaptive rejection sampling for gibbs sampling. Journal of the Royal Statistical Society. Series C (Applied Statistics) 41(2), 337–348 (1992)

    MATH  Google Scholar 

  16. Girvan, M., Newman, M.E.J.: Community structure in social and biological networks. Proceedings of the National Academy of Sciences 99(12), 7821–7826 (2002), http://www.pnas.org/content/99/12/7821 , doi:10.1073/pnas.122653799

    Article  MathSciNet  MATH  Google Scholar 

  17. Kullback, S., Leibler, R.A.: On information and sufficiency. The Annals of Mathematical Statistics 22(1), 79–86 (1951), http://www.jstor.org/stable/2236703

    Article  MathSciNet  MATH  Google Scholar 

  18. Law, N.M., Kulkarni, S.R., Dekany, R.G., Ofek, E.O., Quimby, R.M., Nugent, P.E., Surace, J., Grillmair, C.C., Bloom, J.S., Kasliwal, M.M., et al.: The Palomar transient factory: System overview, performance, and first results. Publications of the Astronomical Society of the Pacific 121(886), 1395–1408 (2009)

    Article  Google Scholar 

  19. Lee, S.M., Roberts, S.J.: Sequential dynamic classification using latent variable models. The Computer Journal 53(9), 1415–1429 (2010), http://comjnl.oxfordjournals.org/content/53/9/1415 , doi:10.1093/comjnl/bxp127

    Google Scholar 

  20. Lefkimmiatis, S., Maragos, P., Papandreou, G.: Bayesian inference on multiscale models for Poisson intensity estimation: Applications to Photon-Limited image denoising. IEEE Transactions on Image Processing 18(8), 1724–1741 (2009), doi:10.1109/TIP.2009.2022008

    Article  MathSciNet  Google Scholar 

  21. Littlestone, N., Warmuth, M.K.: The weighted majority algorithm. In: 30th Annual Symposium on Foundations of Computer Science 1989, pp. 256–261 (1989)

    Google Scholar 

  22. Monteith, K., Carroll, J.L., Seppi, K., Martinez, T.: Turning Bayesian model averaging into Bayesian model combination. In: The 2011 International Joint Conference on Neural Networks (IJCNN), pp. 2657–2663. IEEE (2011), doi:10.1109/IJCNN.2011.6033566

    Google Scholar 

  23. Parisi, G., Shankar, R.: Statistical field theory. Physics Today 41, 110 (1988)

    Article  Google Scholar 

  24. Penny, W.D., Roberts, S.J.: Dynamic logistic regression. In: International Joint Conference on Neural Networks, IJCNN 1999, vol. 3, pp. 1562–1567 (2002)

    Google Scholar 

  25. Psorakis, I., Roberts, S.J., Ebden, M., Sheldon, B.: Overlapping community detection using Bayesian non-negative matrix factorization. Physical Review E 83(6) (2011), http://www.orchid.ac.uk/eprints/9/ , doi:10.1103/PhysRevE.83.066114

  26. Quinn, A.J., Bederson, B.B., Yeh, T., Lin, J.: Crowdflow: Integrating machine learning with Mechanical Turk for speed-cost-quality flexibility. Tech. rep., HCIL-2010-09, University of Maryland, College Park (2010)

    Google Scholar 

  27. Raykar, V.C., Yu, S., Zhao, L.H., Valadez, G.H., Florin, C., Bogoni, L., Moy, L.: Learning from crowds. J. Mach. Learn. Res. 11, 1297–1322 (2010), http://dl.acm.org/citation.cfm?id=1756006.1859894

    MathSciNet  Google Scholar 

  28. Smith, A.M., Lynn, S., Sullivan, M., Lintott, C.J., Nugent, P.E., Botyanszki, J., Kasliwal, M., Quimby, R., Bamford, S.P., Fortson, L.F., et al.: Galaxy Zoo Supernovae. Monthly Notices of the Royal Astronomical Society (2010)

    Google Scholar 

  29. Tan, V.Y.F., Fvotte, C.: Automatic relevance determination in nonnegative matrix factorization. In: SPARS 2009 - Signal Processing with Adaptive Sparse Structured Representations, Inria Rennes-Bretagne Atlantique, Saint Malo, Royaume-Uni (2009), http://hal.inria.fr/view_by_stamp.php?&halsid=oerba5b5pkfqikvfb5e1ideks4&label=SPARS09&langue=en&action_todo=view&id=inria-00369376&version=1

  30. Tulyakov, S., Jaeger, S., Govindaraju, V., Doermann, D.: Review of classifier combination methods. In: Machine Learning in Document Analysis and Recognition, pp. 361–386 (2008)

    Google Scholar 

  31. Weatherburn, C.E.: A First Course in Mathematical Statistics. CUP Archive (1949)

    Google Scholar 

  32. West, M., Harrison, J.: Bayesian forecasting and dynamic models. Springer (1997)

    Google Scholar 

  33. West, M., Harrison, P.J., Migon, H.S.: Dynamic generalized linear models and Bayesian forecasting. Journal of the American Statistical Association 80(389), 73–83 (1985), http://www.jstor.org/stable/2288042 , doi:10.2307/2288042

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Engineering Science, University of Oxford, Oxford, UK

    Edwin Simpson, Stephen Roberts & Ioannis Psorakis

  2. Zooniverse, Cambridge, UK

    Arfon Smith

Authors
  1. Edwin Simpson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephen Roberts
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ioannis Psorakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Arfon Smith
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Edwin Simpson .

Editor information

Editors and Affiliations

  1. Theory and Automation, Academy of Sciences, Institute of Information, Pod vodarenskou vezi 4, Prague, 182 08, Czech Republic

    Tatiana V. Guy

  2. and Automation, Academy of Sciences, Institute of Information Theory, Pod vodarenskou vezi 4, Prague, Czech Republic

    Miroslav Karny

  3. Los Alamos National Laboratory, MS B256, Los Alamos, 87545, USA

    David Wolpert

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Simpson, E., Roberts, S., Psorakis, I., Smith, A. (2013). Dynamic Bayesian Combination of Multiple Imperfect Classifiers. In: Guy, T., Karny, M., Wolpert, D. (eds) Decision Making and Imperfection. Studies in Computational Intelligence, vol 474. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-36406-8_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-36406-8_1

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-36405-1

  • Online ISBN: 978-3-642-36406-8

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation