Ryo Suzuki
Niloufar Salehi
Michelle S. Lam
Juan C. Marroquin
Michael S. Bernstein
ABSTRACT
Expert crowdsourcing marketplaces have untapped potential to empower workers' career and skill development. Currently, many workers cannot afford to invest the time and sacrifice the earnings required to learn a new skill, and a lack of experience makes it difficult to get job offers even if they do. In this paper, we seek to lower the threshold to skill development by repurposing existing tasks on the marketplace as mentored, paid, real-world work experiences, which we refer to as micro-internships. We instantiate this idea in Atelier, a micro-internship platform that connects crowd interns with crowd mentors. Atelier guides mentor-intern pairs to break down expert crowdsourcing tasks into milestones, review intermediate output, and problem-solve together. We conducted a field experiment comparing Atelier's mentorship model to a non-mentored alternative on a real-world programming crowdsourcing task, finding that Atelier helped interns maintain forward progress and absorb best practices.
- Susan A Ambrose, Michael W Bridges, Michele DiPietro, Marsha C Lovett, and Marie K Norman. 2010. How learning works: Seven research-based principles for smart teaching. John Wiley & Sons.Google Scholar
- Judd Antin and Aaron Shaw. 2012. Social desirability bias and self-reports of motivation: a study of Amazon Mechanical Turk in the US and India. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2925--2934. Google ScholarDigital Library
- Alan S Blinder. 2006. Offshoring: the next industrial revolution? FOREIGN AFFAIRS-NEW YORK 85, 2 (2006), 113.Google ScholarCross Ref
- Benjamin S Bloom. 1984. The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational researcher (1984), 4--16.Google Scholar
- Andrew Bosworth. 2009. Facebook Engineering Bootcamp. https://www.facebook.com/notes/177577963919/. (2009).Google Scholar
- John D Bransford, Ann L Brown, and Rodney R Cocking. 1999. How people learn: Brain, mind, experience, and school. National Academy Press.Google Scholar
- John Seely Brown, Allan Collins, and Paul Duguid. 1989. Situated cognition and the culture of learning. Educational researcher 18, 1 (1989), 32--42.Google Scholar
- Allan Collins, John Seely Brown, and Susan E Newman. 1989. Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. Knowing, learning, and instruction: Essays in honor of Robert Glaser 18 (1989), 32--42.Google Scholar
- Elizabeth A Davis and Naomi Miyake. 2004. Explorations of scaffolding in complex classroom systems. The Journal of the Learning Sciences 13, 3 (2004), 265--272.Google ScholarCross Ref
- Mira Dontcheva, Robert R Morris, Joel R Brandt, and Elizabeth M Gerber. 2014. Combining crowdsourcing and learning to improve engagement and performance. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 3379--3388. Google ScholarDigital Library
- Steven Dow, Anand Kulkarni, Scott Klemmer, and Bjorn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 1013--1022. Google ScholarDigital Library
- George W. Furnas, Thomas K. Landauer, Louis M. Gomez, and Susan T. Dumais. 1987. The vocabulary problem in human-system communication. Commun. ACM 30, 11 (1987), 964--971. Google ScholarDigital Library
- Philip J Guo. 2015. Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming (To appear). In Proceedings of the 28th annual ACM symposium on User interface software and technology. ACM. Google ScholarDigital Library
- Neha Gupta, David Martin, Benjamin V Hanrahan, and Jacki O'Neill. 2014. Turk-Life in India. In Proceedings of the 18th International Conference on Supporting Group Work. ACM, 1--11. Google ScholarDigital Library
- Daniel Haas, Jason Ansel, Lydia Gu, and Adam Marcus. 2015. Argonaut: Macrotask Crowdsourcing for Complex Data Processing. Proceedings of the VLDB Endowment 8, 12 (2015). Google ScholarDigital Library
- Pamela Hinds. 2002. Distributed work. MIT Press. Google ScholarDigital Library
- Jim Hollan and Scott Stornetta. 1992. Beyond being there. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 119--125. Google ScholarDigital Library
- Panagiotis G Ipeirotis. 2010. Demographics of Mechanical Turk. Working paper no CEDER-10-01. Available at: http://ssrn.com/abstract=1585030 (2010).Google Scholar
- Nicolas Kaufmann and Thimo Schulze. 2011. Worker motivation in crowdsourcing and human computation. Education 17, 2009 (2011), 6.Google Scholar
- Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 1301--1318. Google ScholarDigital Library
- Marios Kokkodis and Panos Ipeirotis. 2014. The utility of skills in online labor markets. Thirty-Fifth International Conference on Information Systems, Auckland 2014 (2014).Google Scholar
- Marios Kokkodis and Panagiotis G Ipeirotis. 2015. Reputation Transferability in Online Labor Markets. Management Science (2015).Google Scholar
- Anand Kulkarni, Philipp Gutheim, Prayag Narula, David Rolnitzky, Tapan Parikh, and Bjorn Hartmann. 2012. Mobileworks: Designing for quality in a managed crowdsourcing architecture. Internet Computing, IEEE 16, 5 (2012), 28--35. Google ScholarDigital Library
- Chinmay Kulkarni, Michael S Bernstein, and Scott Klemmer. 2015. PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings from The Second (2015) ACM Conference on Learning@ Scale. 75--84. Google ScholarDigital Library
- Ben Lafreniere, Andrea Bunt, Matthew Lount, and Michael A Terry. 2013. Understanding the Roles and Uses of Web Tutorials.. In ICWSM.Google Scholar
- Min Liu. 1998. A study of engaging high-school students as multimedia designers in a cognitive apprenticeship-style learning environment. Computers in Human Behavior 14, 3 (1998), 387--415.Google ScholarCross Ref
- Edwin A Locke. 1980. Goal Setting. Technical Report. DTIC Document.Google Scholar
- Kurt Luther, Jari-Lee Tolentino, Wei Wu, Amy Pavel, Brian P Bailey, Maneesh Agrawala, Bjorn Hartmann, and Steven P Dow. 2015. Structuring, Aggregating, and Evaluating Crowdsourced Design Critique. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 473--485. Google ScholarDigital Library
- David Martin, Benjamin V Hanrahan, Jacki O'Neill, and Neha Gupta. 2014. Being a Turker. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 224--235. Google ScholarDigital Library
- Christine Nash and Dave Collins. 2006. Tacit knowledge in expert coaching: Science or art? Quest 58, 4 (2006), 465--477.Google ScholarCross Ref
- Michael I Norton, Daniel Mochon, and Dan Ariely. 2012. The 'IKEA effect': When labor leads to love. J Consum Psychol. 22: 453460. doi: 10.1016/j.jcps.2011.08.002 (2012).Google ScholarCross Ref
- Seymour Papert. 1980. Mindstorms: Children, computers, and powerful ideas. Basic Books, Inc. Google ScholarDigital Library
- Daniel Rees Lewis, Emily Harburg, Elizabeth Gerber, and Matthew Easterday. 2015. Building Support Tools to Connect Novice Designers with Professional Coaches. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition. ACM, 43--52. Google ScholarDigital Library
- Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S Bernstein. 2014. Expert crowdsourcing with flkash teams. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 75--85. Google ScholarDigital Library
- Roger C Schank, Tamara R Berman, and Kimberli A Macpherson. 1999. Learning by doing. Instructional-design theories and models: A new paradigm of instructional theory 2 (1999), 161--181.Google Scholar
- Daniel L Schwartz and John D Bransford. 1998. A time for telling. Cognition and instruction 16, 4 (1998), 475--5223.Google Scholar
- C Addison Stone. 1998. The metaphor of scaffolding its utility for the field of learning disabilities. Journal of learning disabilities 31, 4 (1998), 344--364.Google ScholarCross Ref
- Upwork. 2015. Freelancing in America: 2015. https://www.upwork.com/i/freelancinginamerica2015/. (2015).Google Scholar
- Lev Vygotsky. 1978. Interaction between learning and development. Readings on the development of children 23, 3 (1978), 34--41.Google Scholar
- Rob Wass and Clinton Golding. 2014. Sharpening a tool for teaching: the zone of proximal development. Teaching in Higher Education 19, 6 (2014), 671--684.Google ScholarCross Ref
- David Wood, Jerome S Bruner, Gail Ross, and others. 1976. The role of tutoring in problem solving. Journal of child psychology and psychiatry 17, 2 (1976), 89--100.Google ScholarCross Ref
Index Terms
- Atelier: Repurposing Expert Crowdsourcing Tasks as Micro-internships
Recommendations
The future of crowd work
CSCW '13: Proceedings of the 2013 conference on Computer supported cooperative workPaid crowd work offers remarkable opportunities for improving productivity, social mobility, and the global economy by engaging a geographically distributed workforce to complete complex tasks on demand and at scale. But it is also possible that crowd ...
Human Collaboration on Crowdsourcing Platforms – a Content Analysis
HCI in Business, Government and Organizations. Information Systems and AnalyticsAbstractThe crowdsourcing phenomenon offers the opportunity to address an open call to the crowd. Crowd workers may work together to find a solution that satisfies the open call. One of the major benefits for a crowdsourcer is the pool of crowd workers ...
Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability
CSCW '17: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social ComputingDistributed, parallel crowd workers can accomplish simple tasks through workflows, but teams of collaborating crowd workers are necessary for complex goals. Unfortunately, a fundamental condition for effective teams -- familiarity with other members -- ...
Comments