Leveraging Wisdom of the Crowd for Decision Support

While Decision Support Systems (DSS) have a long history, their usefulness for non-experts outside specific organisations has not lived up to the promise. A key reason for this is the high cost associated with populating the underlying knowledge bases. In this article, we describe how DSSs can leverage crowds and their wisdom in constructing knowledge bases to overcome this challenge. We also demonstrate how to construct DSSs on-the-fly using the collected data. Our user-driven laboratory studies focus on user perceptions of the concept itself, and motives for contributing and using such a DSS. To the best of our knowledge, our approach and implementation are the first to demonstrate such use of crowdsourcing in building DSSs.

Content

Author and article information

Contributors

Simo Hosio

Jorge Goncalves

Theodoros Anagnostopoulos

Vassilis Kostakos

Conference

Publication date: July 2016

Publication date (Print): July 2016

Pages: 1-12

Affiliations

[0001]University of Oulu

Pentti Kaiteran katu 1

90570 Oulu, Finland

Article

DOI: 10.14236/ewic/HCI2016.38

SO-VID: 76d4e220-f3dc-4968-bd51-def0dea0bc8b

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: Proceedings of the 30th International BCS Human Computer Interaction Conference

Conference acronym: HCI

Conference number: 30

Conference location: Bournemouth University, Poole, UK

Conference date: 11 - 15 July 2016

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Fusion

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2016.38

Self URI (journal page): https://ewic.bcs.org/

REFERENCES

S. Alter 1982 Decision support systems: Current practice and continuing challenges. Reading Massachusetts: Addison-Wesley Publishing Co 1980 316 Behavioral Science 27 1 91 2
A. De AngeliA. SutcliffeJ. Hartmann 2006 Interaction, Usability and Aesthetics: What Influences Users’ Preferences? Proceedings of the 6th Conference on Designing Interactive Systems ACM 271 80
D. ArnottG. Pervan 2005 A critical analysis of decision support systems research Journal of information technology 20 2 67 87
Ask Dan! about DSS - How does sensitivity analysis differ from “What if? ” analysis? Retrieved May 21 2015 http://dssresources.com/faq/index.php?action=artikel&id=121
A. BangorP.T. KortumJ.T. Miller 2008 An Empirical Evaluation of the System Usability Scale Intl. Journal of Human-Computer Interaction 24 6 574 94
G.O. BarnettJ.J. CiminoJ.A. HuppE.P. Hoffer 1987 DXplain. An evolving diagnostic decisionsupport system JAMA 258 1 67 74
M.S. Bernstein et al 2010 Soylent: A Word Processor with a Crowd Inside Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology ACM 313 22
C. ChiuT. LiangE. Turban 2014 What can crowdsourcing do for decision support? Decision Support Systems 65 40 9
J.S. DownsM.B. HolbrookS. ShengL.F. Cranor 2010 Are Your Participants Gaming the System?: Screening Mechanical Turk Workers Proceedings of the SIGCHI Conference on Human Factors in Computing Systems ACM 2399 402
M.J. DruzdzelR.R. Flynn 1999 Decision support systems. Encyclopedia of library and information science Marcel Dekker, Inc. Last Login
M.C. Er 1988 Decision Support Systems: A Summary, Problems, and Future Trends Decis. Support Syst. 4 3 355 63
P.N. Finlay 1994 Introducing decision support systems NCC Blackwell ; Cambridge, Mass., USA : Blackwell Publishers Oxford, UK
F. Galton 1907 Vox populi (the wisdom of crowds) Nature 75 450 1
M.D. Geurts 1994 Data problems in decision support systems Hawaii International Conference on System Sciences IEEE 155 8
J. GoncalvesS. HosioD. FerreiraV. Kostakos 2014a Game of Words: Tagging Places through Crowdsourcing on Public Displays Designing Interactive Systems Vancouver, BC, Canada 705 14
J. GoncalvesS. HosioY. LiuV. Kostakos 2014b Eliciting Situated Feedback: A Comparison of Paper, Web Forms and Public Displays Displays 35 1 27 37
J. Goncalves et al 2013 Crowdsourcing on the spot: altruistic use of public displays, feasibility, performance, and behaviours International Joint Conference on Pervasive and Ubiquitous Computing ACM 753 62
S. HosioJ. GoncalvesV. KostakosJ. Riekki 2014 Exploring Civic Engagement on Public Displays, in S Saeed (ed) User-Centric Technology Design for Nonprofit and Civic Engagements Springer International Publishing 91 111
S. HosioJ. GoncalvesV. KostakosJ. Riekki 2015 Crowdsourcing Public Opinion using Urban Pervasive Technologies: Lessons from Real-Life Experiments in Oulu Policy & Internet 7 2 203 22
S. Hosio et al 2012 From school food to skate parks in a few clicks: using public displays to bootstrap civic engagement of the young International Conference on Pervasive Computing Springer 425 42
S. Hosio et al 2014 Situated Crowdsourcing Using a Market Model User Interface Software and Technology ACM 55 64
D. Huffaker 2010 Dimensions of Leadership and Social Influence in Online Communities Human Communication Research 36 4 593 617
P.G. IpeirotisE. Gabrilovich 2014 Quizz: Targeted crowdsourcing with a billion (potential) users Proceedings of the 23rd international conference on World wide web 143 54
D. JannachM. ZankerM. GeM. Gröning 2012 Recommender Systems in Computer Science and Information Systems – A Landscape of Research E-Commerce and Web Technologies Springer Berlin Heidelberg 76 87
K. JärvelinJ. Kekäläinen 2000 IR Evaluation Methods for Retrieving Highly Relevant Documents Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval ACM 41 8
N. KaufmannT. SchulzeD. Veit 2011 More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk AMCIS
P.G.W. Keen 1981 Value Analysis: Justifying Decision Support Systems MIS Q. 5 1 1 15
A. Kittur et al 2013 The future of crowd work Proceedings of the 2013 conference on Computer supported cooperative work ACM 1301 18
V. Kostakos 2009 Is the Crowd’s Wisdom Biased? A Quantitative Analysis of Three Online Communities Computational Science and Engineering 251 5
N. LandwehrM. HallE. Frank 2005 Logistic Model Trees Machine Learning 59 1-2 161 205
J. LorenzH. RauhutF. SchweitzerD. Helbing 2011 How social influence can undermine the wisdom of crowd effect Proceedings of the National Academy of Sciences 108 22 9020 5
R. McCarneyJ. WarnerS. IliffeR. van HaselenM. GriffinP. Fisher 2007 The Hawthorne Effect: a randomised, controlled trial BMC medical research methodology 7 30
B. NonneckeJ. Preece 2000 Lurker demographics: counting the silent Proceedings of the SIGCHI conference on Human Factors in Computing Systems ACM 73 80
J. NoronhaE. HysenH. ZhangK.Z. Gajos 2011 Platemate crowdsourcing nutritional analysis from food photographs Proceedings of the 24th annual ACM symposium on User interface software and technology 1 12
S.E. Page 2008 The difference : how the power of diversity creates better groups, firms, schools, and societies Princeton University Press Princeton, N.J. Woodstock
M.K. PoetzM. Schreier 2012 The Value of Crowdsourcing: Can Users Really Compete with Professionals in Generating New Product Ideas? Journal of Product Innovation Management 29 2 245 56
D.J. Power 2002 Decision support systems : concepts and resources for managers Quorum Books, Westport, Conn
D.J. PowerR. Sharda 2007 Model-driven decision support systems: Concepts and research directions Decision Support Systems 43 3 1044 61
J. PreeceB. NonneckeD. Andrews 2004 The top five reasons for lurking: improving community experiences for everyone Computers in Human Behavior 20 2 201 223
P. ResnickH.R. Varian 1997 Recommender Systems Commun. ACM 40 3 56 8
F. RicciL. RokachB. Shapira 2011 Introduction to Recommender Systems Handbook Recommender Systems Handbook Springer US 1 35
J. Rogstadius et al 2011 An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets International AAAI Conference on Web and Social Media Barcelona, Spain 321 8
MJ SalganikPS DoddsDJ Watts 2006 Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market Science February 2006 854 6
Sauro JMeasuring U, Retrieved September 28 2014 http://www.measuringu.com/sus.php
N. Shambaugh 2009 Personalized Decision Support Systems JR RabunalJ DoradoA Pazos Sierra Encyclopedia of artificial intelligence Information Science Reference Hershey, PA
J.P. ShimM. WarkentinJ.F. CourtneyD.J. PowerR. ShardaC. Carlsson 2002 Past, present, and future of decision support technology Decision Support Systems 33 2 111 26
J. Surowiecki 2005 The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations 1st Anchor, New York
VotingAid VotingAid Retrieved September 16 2015 http://zef.fi/votingaid/en/home/
Y. WangY. WangS. PatelD. Patel 2006 A layered reference model of the brain (LRMB) Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on 36 2 124 33
I. YanivM. Milyavsky 2007 Using advice from multiple sources to revise and improve judgments Organizational Behavior and Human Decision Processes 103 1 104 20

Comments

Comment on this article

[1] S. Alter 1982 Decision support systems: Current practice and continuing challenges. Reading Massachusetts: Addison-Wesley Publishing Co 1980 316 Behavioral Science 27 1 91 2

[2] A. De AngeliA. SutcliffeJ. Hartmann 2006 Interaction, Usability and Aesthetics: What Influences Users’ Preferences? Proceedings of the 6th Conference on Designing Interactive Systems ACM 271 80

[3] D. ArnottG. Pervan 2005 A critical analysis of decision support systems research Journal of information technology 20 2 67 87

[4] Ask Dan! about DSS - How does sensitivity analysis differ from “What if? ” analysis? Retrieved May 21 2015 http://dssresources.com/faq/index.php?action=artikel&id=121

[5] A. BangorP.T. KortumJ.T. Miller 2008 An Empirical Evaluation of the System Usability Scale Intl. Journal of Human-Computer Interaction 24 6 574 94

[6] G.O. BarnettJ.J. CiminoJ.A. HuppE.P. Hoffer 1987 DXplain. An evolving diagnostic decisionsupport system JAMA 258 1 67 74

[7] M.S. Bernstein et al 2010 Soylent: A Word Processor with a Crowd Inside Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology ACM 313 22

[8] C. ChiuT. LiangE. Turban 2014 What can crowdsourcing do for decision support? Decision Support Systems 65 40 9

[9] J.S. DownsM.B. HolbrookS. ShengL.F. Cranor 2010 Are Your Participants Gaming the System?: Screening Mechanical Turk Workers Proceedings of the SIGCHI Conference on Human Factors in Computing Systems ACM 2399 402

[10] M.J. DruzdzelR.R. Flynn 1999 Decision support systems. Encyclopedia of library and information science Marcel Dekker, Inc. Last Login

[11] M.C. Er 1988 Decision Support Systems: A Summary, Problems, and Future Trends Decis. Support Syst. 4 3 355 63

[12] P.N. Finlay 1994 Introducing decision support systems NCC Blackwell ; Cambridge, Mass., USA : Blackwell Publishers Oxford, UK

[13] F. Galton 1907 Vox populi (the wisdom of crowds) Nature 75 450 1

[14] M.D. Geurts 1994 Data problems in decision support systems Hawaii International Conference on System Sciences IEEE 155 8

[15] J. GoncalvesS. HosioD. FerreiraV. Kostakos 2014a Game of Words: Tagging Places through Crowdsourcing on Public Displays Designing Interactive Systems Vancouver, BC, Canada 705 14

[16] J. GoncalvesS. HosioY. LiuV. Kostakos 2014b Eliciting Situated Feedback: A Comparison of Paper, Web Forms and Public Displays Displays 35 1 27 37

[17] J. Goncalves et al 2013 Crowdsourcing on the spot: altruistic use of public displays, feasibility, performance, and behaviours International Joint Conference on Pervasive and Ubiquitous Computing ACM 753 62

[18] S. HosioJ. GoncalvesV. KostakosJ. Riekki 2014 Exploring Civic Engagement on Public Displays, in S Saeed (ed) User-Centric Technology Design for Nonprofit and Civic Engagements Springer International Publishing 91 111

[19] S. HosioJ. GoncalvesV. KostakosJ. Riekki 2015 Crowdsourcing Public Opinion using Urban Pervasive Technologies: Lessons from Real-Life Experiments in Oulu Policy & Internet 7 2 203 22

[20] S. Hosio et al 2012 From school food to skate parks in a few clicks: using public displays to bootstrap civic engagement of the young International Conference on Pervasive Computing Springer 425 42

[21] S. Hosio et al 2014 Situated Crowdsourcing Using a Market Model User Interface Software and Technology ACM 55 64

[22] D. Huffaker 2010 Dimensions of Leadership and Social Influence in Online Communities Human Communication Research 36 4 593 617

[23] P.G. IpeirotisE. Gabrilovich 2014 Quizz: Targeted crowdsourcing with a billion (potential) users Proceedings of the 23rd international conference on World wide web 143 54

[24] D. JannachM. ZankerM. GeM. Gröning 2012 Recommender Systems in Computer Science and Information Systems – A Landscape of Research E-Commerce and Web Technologies Springer Berlin Heidelberg 76 87

[25] K. JärvelinJ. Kekäläinen 2000 IR Evaluation Methods for Retrieving Highly Relevant Documents Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval ACM 41 8

[26] N. KaufmannT. SchulzeD. Veit 2011 More than fun and money. Worker Motivation in Crowdsourcing-A Study on Mechanical Turk AMCIS

[27] P.G.W. Keen 1981 Value Analysis: Justifying Decision Support Systems MIS Q. 5 1 1 15

[28] A. Kittur et al 2013 The future of crowd work Proceedings of the 2013 conference on Computer supported cooperative work ACM 1301 18

[29] V. Kostakos 2009 Is the Crowd’s Wisdom Biased? A Quantitative Analysis of Three Online Communities Computational Science and Engineering 251 5

[30] N. LandwehrM. HallE. Frank 2005 Logistic Model Trees Machine Learning 59 1-2 161 205

[31] J. LorenzH. RauhutF. SchweitzerD. Helbing 2011 How social influence can undermine the wisdom of crowd effect Proceedings of the National Academy of Sciences 108 22 9020 5

[32] R. McCarneyJ. WarnerS. IliffeR. van HaselenM. GriffinP. Fisher 2007 The Hawthorne Effect: a randomised, controlled trial BMC medical research methodology 7 30

[33] B. NonneckeJ. Preece 2000 Lurker demographics: counting the silent Proceedings of the SIGCHI conference on Human Factors in Computing Systems ACM 73 80

[34] J. NoronhaE. HysenH. ZhangK.Z. Gajos 2011 Platemate crowdsourcing nutritional analysis from food photographs Proceedings of the 24th annual ACM symposium on User interface software and technology 1 12

[35] S.E. Page 2008 The difference : how the power of diversity creates better groups, firms, schools, and societies Princeton University Press Princeton, N.J. Woodstock

[36] M.K. PoetzM. Schreier 2012 The Value of Crowdsourcing: Can Users Really Compete with Professionals in Generating New Product Ideas? Journal of Product Innovation Management 29 2 245 56

[37] D.J. Power 2002 Decision support systems : concepts and resources for managers Quorum Books, Westport, Conn

[38] D.J. PowerR. Sharda 2007 Model-driven decision support systems: Concepts and research directions Decision Support Systems 43 3 1044 61

[39] J. PreeceB. NonneckeD. Andrews 2004 The top five reasons for lurking: improving community experiences for everyone Computers in Human Behavior 20 2 201 223

[40] P. ResnickH.R. Varian 1997 Recommender Systems Commun. ACM 40 3 56 8

[41] F. RicciL. RokachB. Shapira 2011 Introduction to Recommender Systems Handbook Recommender Systems Handbook Springer US 1 35

[42] J. Rogstadius et al 2011 An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets International AAAI Conference on Web and Social Media Barcelona, Spain 321 8

[43] MJ SalganikPS DoddsDJ Watts 2006 Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market Science February 2006 854 6

[44] Sauro JMeasuring U, Retrieved September 28 2014 http://www.measuringu.com/sus.php

[45] N. Shambaugh 2009 Personalized Decision Support Systems JR RabunalJ DoradoA Pazos Sierra Encyclopedia of artificial intelligence Information Science Reference Hershey, PA

[46] J.P. ShimM. WarkentinJ.F. CourtneyD.J. PowerR. ShardaC. Carlsson 2002 Past, present, and future of decision support technology Decision Support Systems 33 2 111 26

[47] J. Surowiecki 2005 The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations 1st Anchor, New York

[48] VotingAid VotingAid Retrieved September 16 2015 http://zef.fi/votingaid/en/home/

[49] Y. WangY. WangS. PatelD. Patel 2006 A layered reference model of the brain (LRMB) Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on 36 2 124 33

[50] I. YanivM. Milyavsky 2007 Using advice from multiple sources to revise and improve judgments Organizational Behavior and Human Decision Processes 103 1 104 20

Celebrating 65 years of The Computer Journal - free-to-read perspectives - bcs.org/tcj65