Niloufar Salehi
Michael Bernstein
ABSTRACT
Distributed, parallel crowd workers can accomplish simple tasks through workflows, but teams of collaborating crowd workers are necessary for complex goals. Unfortunately, a fundamental condition for effective teams -- familiarity with other members -- stands in contrast to crowd work's flexible, on-demand nature. We enable effective crowd teams with Huddler, a system for workers to assemble familiar teams even under unpredictable availability and strict time constraints. Huddler utilizes a dynamic programming algorithm to optimize for highly familiar teammates when individual availability is unknown. We first present a field experiment that demonstrates the value of familiarity for crowd teams: familiar crowd teams doubled the performance of ad-hoc (unfamiliar) teams on a collaborative task. We then report a two-week field deployment wherein Huddler enabled crowd workers to convene highly familiar teams in 18 minutes on average. This research advances the goal of supporting long-term, team-based collaborations without sacrificing the flexibility of crowd work.
- Shoshana Altschuller, and Raquel Benbunan-Fich. "Trust, Performance, and the Communication Process in Ad Hoc Decision-Making Virtual Teams." Journal of Computer-Mediated Communication 16, no. 1 (2010): 27--47.Google Scholar
Cross Ref
- Linda Argote, Paul Ingram, John M. Levine, and Richard L. Moreland. "Knowledge transfer in organizations: Learning from the experience of others." Organizational behavior and human decision processes 82, no. 1 (2000): 1--8.Google Scholar
- Linda Argote. "Input uncertainty and organizational coordination in hospital emergency units." Administrative science quarterly (1982): 420--434.Google Scholar
- Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. "Soylent: a word processor with a crowd inside." In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pp. 313--322. ACM, 2010. Google ScholarDigital Library
- Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller et al. "VizWiz: nearly real-time answers to visual questions." In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pp. 333--342. ACM, 2010. Google ScholarDigital Library
- Jeffrey P. Bigham, Michael S. Bernstein, and Eytan Adar. "Human-Computer Interaction and Collective Intelligence." Handbook of Collective Intelligence (2015): 57.Google Scholar
- Kathleen Carley. "Organizational learning and personnel turnover."Organization Science 3.1 (1992): 20--46.Google ScholarDigital Library
- Tom Finholt, Lee Sproull, and Sara Kiesler. "Communication and performance in ad hoc task groups." Intellectual teamwork: Social and technological foundations of cooperative work (1990): 291--325. Google ScholarDigital Library
- Susan G. Cohen, and Diane E. Bailey. "What makes teams work: Group effectiveness research from the shop floor to the executive suite." Journal of management 23, no. 3 (1997): 239--290.Google ScholarCross Ref
- Lincoln Dahlberg. "The Internet and democratic discourse: Exploring the prospects of online deliberative forums extending the public sphere." Information, Communication & Society 4, no. 4 (2001): 615--633.Google ScholarCross Ref
- Steven P. Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel L. Schwartz, and Scott R. Klemmer. "Parallel prototyping leads to better design results, more divergence, and increased self-efficacy." ACM Transactions on Computer-Human Interaction (TOCHI) 17, no. 4 (2010): 18. Google ScholarDigital Library
- Scott B. Droege, and Jenny M. Hoobler. "Employee turnover and tacit knowledge diffusion: A network perspective." Journal of Managerial Issues(2003): 5064.Google Scholar
- Amy C. Edmondson Teaming: How organizations learn, innovate, and compete in the knowledge economy. John Wiley & Sons, 2012.Google Scholar
- Amy Edmondson. "Psychological safety and learning behavior in work teams." Administrative science quarterly 44, no. 2 (1999): 350--383.Google ScholarCross Ref
- J. Alberto Espinosa, Sandra A. Slaughter, Robert E. Kraut, and James D. Herbsleb. "Team knowledge and coordination in geographically distributed software development." Journal of Management Information Systems 24, no. 1 (2007): 135--169. Google ScholarDigital Library
- John E. Ettlie "The impact of interorganizational manpower flows on the innovation process." Management Science 31, no. 9 (1985): 10551071.Google ScholarDigital Library
- Mary L. Gray, Siddharth Suri, Syed S. Ali, and Deepti Kulkarni. "The Crowd is a Collaborative Network." Proceedings of Computer-Supported Cooperative Work (2016). Google ScholarDigital Library
- Deborah H. Gruenfeld, Elizabeth A. Mannix, Katherine Y. Williams, and Margaret A. Neale. "Group composition and decision making: How member familiarity and information distribution affect process and performance." Organizational behavior and human decision processes 67, no. 1 (1996): 1--15.Google Scholar
- Rebecca M. Guidice, Joyce Thompson Heames, and Sheng Wang. "The indirect relationship between organizational-level knowledge worker turnover and innovation: An integrated application of related literature." The Learning Organization 16, no. 2 (2009): 143--167.Google ScholarCross Ref
- Daniel Haas, Jason Ansel, Lydia Gu, and Adam Marcus. "Argonaut: macrotask crowdsourcing for complex data processing." Proceedings of the VLDB Endowment 8, no. 12 (2015): 1642--1653. Google ScholarDigital Library
- J. Richard. Hackman, Leading teams: Setting the stage for great performances. Harvard Business Press, 2002.Google Scholar
- D.A. Harrison, Mohammed, S., McGrath, J.E., Florey, A.T. and Vanderstoep, S.W., 2003. Time matters in team performance: Effects of member familiarity, entrainment, and task discontinuity on speed and quality. Personnel Psychology, 56(3), pp.633--669.Google ScholarCross Ref
- Mark Hefke, and Ljiljana Stojanovic. "An ontologybased approach for competence bundling and composition of ad-hoc teams in an organization." Proc. I-KNOW'04 (2004): 126--134.Google Scholar
- Pamela J. Hinds, and Mark Mortensen. "Understanding conflict in geographically distributed teams: The moderating effects of shared identity, shared context, and spontaneous communication." Organization science 16, no. 3 (2005): 290--307. Google ScholarDigital Library
- Robert S. Huckman, and Bradley R. Staats. "Fluid tasks and fluid teams: The impact of diversity in experience and team familiarity on team performance." Manufacturing & Service Operations Management 13, no. 3 (2011): 310--328. Google ScholarDigital Library
- Robert S. Huckman, Bradley R. Staats, and David M. Upton. "Team familiarity, role experience, and performance: Evidence from Indian software services." Management science 55, no. 1 (2009): 85--100. Google ScholarDigital Library
- Aniket Kittur, Bongwon Suh, Bryan A. Pendleton, and Ed H. Chi. "He says, she says: conflict and coordination in Wikipedia." In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 453--462. ACM, 2007. Google ScholarDigital Library
- Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. "Crowdforge: Crowdsourcing complex work." In Proceedings of the 24th annual ACM symposium on User interface software and technology, pp. 43--52. ACM, 2011. Google ScholarDigital Library
- Aniket Kittur. "Crowdsourcing, collaboration and creativity." ACM Crossroads 17, no. 2 (2010): 22--26. Google ScholarDigital Library
- David A. Kravitz, and Barbara Martin. "Ringelmann rediscovered: The original article." (1986): 936.Google Scholar
- Karim Lakhani, David A. Garvin, and Eric Lonstein. "Topcoder (a): Developing software through crowdsourcing." Harvard Business School General Management Unit Case 610-032 (2010).Google Scholar
- Walter S. Lasecki, Juho Kim, Nick Rafter, Onkur Sen, Jeffrey P. Bigham, and Michael S. Bernstein. "Apparition: Crowdsourced user interfaces that come To life as you sketch them." In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1925--1934. ACM, 2015. Google ScholarDigital Library
- Diane Wei Liang, Richard Moreland, and Linda Argote. "Group versus individual training and group performance: The mediating role of transactive memory." Personality and Social Psychology Bulletin 21, no. 4 (1995): 384--393.Google ScholarCross Ref
- Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. "Turkit: tools for iterative tasks on mechanical turk." In Proceedings of the ACM SIGKDD workshop on human computation, pp. 29--30. ACM, 2009. Google ScholarDigital Library
- Ioanna Lykourentzou, Shannon Wang, Robert E. Kraut, and Steven P. Dow. "Team Dating: A SelfOrganized Team Formation Strategy for Collaborative Crowdsourcing." In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pp. 1243--1249. ACM, 2016. Google ScholarDigital Library
- Ioanna Lykourentzou, Angeliki Antoniou, Yannick Naudet, and Steven P. Dow. "Personality Matters: Balancing for Personality Types Leads to Better Outcomes for Crowd Teams." (2016).Google Scholar
- Andrew Mao, Winter Mason, Siddharth Suri, and Duncan J. Watts. "An experimental study of team size and performance on a complex task." PloS one 11, no. 4 (2016): e0153048.Google ScholarCross Ref
- David Martin, Benjamin V. Hanrahan, Jacki O'Neill, and Neha Gupta. "Being a turker." In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, pp. 224--235. ACM, 2014. Google ScholarDigital Library
- Winter Mason, and Siddharth Suri. "Conducting behavioral research on Amazon's Mechanical Turk." Behavior research methods 44.1 (2012): 1--23.Google ScholarCross Ref
- Joseph E. McGrath, Holly Arrow, and Jennifer L. Berdahl. "The study of groups: past, present, and future." Personality and Social Psychology Review 4, no. 1 (2000): 95--105.Google ScholarCross Ref
- Mitzi M. Montoya-Weiss, Anne P. Massey, and Michael Song. "Getting it together: Temporal coordination and conflict management in global virtual teams." Academy of management Journal 44, no. 6 (2001): 1251--1262.Google Scholar
- M. Mortensen 2014 "Constructing the Team: The Antecedents and Effects of Membership Model Divergence." Organization Science, 25: 909--931. Google ScholarDigital Library
- Ingrid Mulder, Janine Swaak, and Joseph Kessels. "In search of reflective behavior and shared understanding in ad hoc expert teams." CyberPsychology & Behavior 7, no. 2 (2004): 141--154.Google ScholarCross Ref
- Gary M. Olson, and Judith S. Olson. "Distance matters." Human-computer interaction 15, no. 2 (2000): 139--178. Google ScholarDigital Library
- Ray Reagans, Linda Argote, and Daria Brooks. "Individual experience and experience working together: Predicting learning rates from knowing who knows what and knowing how to work together." Management science 51.6 (2005): 869--881. Google ScholarDigital Library
- Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. "Expert crowdsourcing with flash teams." In Proceedings of the 27th annual ACM symposium on User interface software and technology, pp. 75--85. ACM, 2014. Google ScholarDigital Library
- Floor Rink, Aimée A. Kane, Naomi Ellemers, and Gerben Van der Vegt. "Team receptivity to newcomers: Five decades of evidence and future research themes." The Academy of Management Annals 7, no. 1 (2013): 247--293.Google ScholarCross Ref
- Elena Rocco, Thomas Finholt, Erik C. Hofer, and James Herbsleb. "Designing as if trust mattered." Collaboratory for Research on Electronic Work (CREW) Technical Report (2000).Google Scholar
- Kate Starbird and Leysia Palen. "Voluntweeters: Selforganizing by digital volunteers in times of crisis." In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1071--1080. ACM, 2011. Google ScholarDigital Library
- Andrew H. Van de Ven, Andre L. Delbecq, and Richard Koenig Jr. "Determinants of coordination modes within organizations." American sociological review (1976): 322--338.Google Scholar
- Anita Williams Woolley, Christopher F. Chabris, Alex Pentland, Nada Hashmi, and Thomas W. Malone. "Evidence for a collective intelligence factor in the performance of human groups." science 330, no. 6004 (2010): 686--688.Google Scholar
- Galen Pickard, Wei Pan, Iyad Rahwan, Manuel Cebrian, Riley Crane, Anmol Madan, and Alex Pentland. "Time-critical social mobilization." Science 334, no. 6055 (2011): 509--512.Google ScholarCross Ref
- Niloufar Salehi, Lilly C. Irani, Michael S. Bernstein, Ali Alkhatib, Eva Ogbe, and Kristy Milland. "We are dynamo: Overcoming stalling and friction in collective action for crowd workers." In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1621--1630. ACM, 2015. Google ScholarDigital Library
- Haiyi Zhu, Steven P. Dow, Robert E. Kraut, and Aniket Kittur. "Reviewing versus doing: Learning and performance in crowd assessment." In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, pp. 1445--1455. ACM, 2014. Google ScholarDigital Library
- http://www.pewinternet.org/2016/07/11/research-inthe-crowdsourcing-age-a-casestudy/http://www.pewinternet.org/2016/07/11/research-in-the-crowdsourcing-age-a-case-study/Google Scholar
Index Terms
- Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability
Recommendations
The future of crowd work
CSCW '13: Proceedings of the 2013 conference on Computer supported cooperative workPaid crowd work offers remarkable opportunities for improving productivity, social mobility, and the global economy by engaging a geographically distributed workforce to complete complex tasks on demand and at scale. But it is also possible that crowd ...
Human Collaboration on Crowdsourcing Platforms – a Content Analysis
HCI in Business, Government and Organizations. Information Systems and AnalyticsAbstractThe crowdsourcing phenomenon offers the opportunity to address an open call to the crowd. Crowd workers may work together to find a solution that satisfies the open call. One of the major benefits for a crowdsourcer is the pool of crowd workers ...
Designing a Crowdsourcing System for the Elderly: A Gamified Approach to Speech Collection
CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing SystemsDespite the need for representation of older adults in crowdsourced data, crowd work is generally not designed for older adults and participation by older adults is low. In this paper, we demonstrate a process for designing crowd work for older adults; ...
Comments