skip to main content
research-article
Public Access

Ink: Increasing Worker Agency to Reduce Friction in Hiring Crowd Workers

Published:11 April 2018Publication History
Skip Abstract Section

Abstract

The web affords connections by which end-users can receive paid, expert help—such as programming, design, and writing—to reach their goals. While a number of online marketplaces have emerged to facilitate such connections, most end-users do not approach a market to hire an expert when faced with a challenge. To reduce friction in hiring from peer-to-peer expert crowd work markets, we propose Ink, a system that crowd workers can use to showcase their services by embedding tasks inside web tutorials—a common destination for users with information needs. Workers have agency to define and manage tasks, through which users can request their help to review or execute each step of the tutorial, for example, to give feedback on a paper outline, perform a statistical analysis, or host a practice programming interview. In a public deployment, over 25,000 pageviews led 168 tutorial readers to pay crowd workers for their services, most of whom had not previously hired from crowdsourcing marketplaces. A field experiment showed that users were more likely to hire crowd experts when the task was embedded inside the tutorial rather than when they were redirected to the same worker’s Upwork profile to hire them. Qualitative analysis of interviews showed that Ink framed hiring expert crowd workers within users’ well-established information seeking habits and gave workers more control over their work.

References

  1. Lada A. Adamic, Jun Zhang, Eytan Bakshy, and Mark S. Ackerman. 2008. Knowledge sharing and Yahoo answers: Everyone knows something. In Proceedings of the 17th International Conference on World Wide Web. ACM, 665--674. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. 2010. Soylent: A word processor with a crowd inside. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology. ACM, 313--322. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. 2010. VizWiz: Nearly real-time answers to visual questions. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology. ACM, 333--342. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Joel Brandt, Philip J. Guo, Joel Lewenstein, Mira Dontcheva, and Scott R. Klemmer. 2009. Two studies of opportunistic programming: Interleaving web foraging, learning, and writing code. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1589--1598. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Andrea Bunt, Patrick Dubois, Ben Lafreniere, Michael A. Terry, and David T. Cormack. 2014. TaggedComments: Promoting and integrating user comments in online application tutorials. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems. ACM, 4037--4046. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Yan Chen, Teck-Hua Ho, and Yong-Mi Kim. 2010. Knowledge market design: A field experiment at google answers. Journal of Public Economic Theory 12, 4 (2010), 641--664.Google ScholarGoogle ScholarCross RefCross Ref
  7. Snehalkumar Neil S. Gaikwad, Durim Morina, Adam Ginzberg, Catherine Mullings, Shirish Goyal, Dilrukshi Gamage, Christopher Diemert, Mathias Burton, Sharon Zhou, Mark Whiting, and others. 2016. Boomerang: Rebounding the consequences of reputation feedback on crowdsourcing platforms. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 625--637. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Mary L. Gray, Siddharth Suri, Syed Shoaib Ali, and Deepti Kulkarni. 2016. The crowd is a collaborative network. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work 8 Social Computing. ACM, 134--147. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Lei Guo, Enhua Tan, Songqing Chen, Xiaodong Zhang, and Yihong Eric Zhao. 2009. Analyzing patterns of user content generation in online social networks. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 369--378. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Lilly C. Irani and M. Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 611--620. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Lilly C. Irani and M. Silberman. 2016. Stories we tell about labor: Turkopticon and the trouble with design. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4573--4586. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Sheena S. Iyengar and Mark R. Lepper. 2000. When choice is demotivating: Can one desire too much of a good thing?Journal of Personality and Social Psychology 79, 6 (2000), 995.Google ScholarGoogle Scholar
  13. Daniel Kahneman. 2011. Thinking, Fast and Slow. Macmillan.Google ScholarGoogle Scholar
  14. Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work. ACM, 1301--1318. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Ben Lafreniere, Andrea Bunt, Matthew Lount, and Michael A. Terry. 2013. Understanding the roles and uses of web tutorials. In ICWSM.Google ScholarGoogle Scholar
  16. Min Kyung Lee, Daniel Kusbit, Evan Metsky, and Laura Dabbish. 2015. Working with machines: The impact of algorithmic and data-driven management on human workers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1603--1612. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Lena Mamykina, Bella Manoim, Manas Mittal, George Hripcsak, and Björn Hartmann. 2011. Design lessons from the fastest Q&A site in the west. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2857--2866. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. David Martin, Benjamin V. Hanrahan, Jacki O’Neill, and Neha Gupta. 2014. Being a turker. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work 8 Social Computing. ACM, 224--235. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Meredith Ringel Morris, Jeffrey P. Bigham, Robin Brewer, Jonathan Bragg, Anand Kulkarni, Jessie Li, and Saiph Savage. 2017. Subcontracting microwork. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. 2014. Expert crowdsourcing with flash teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology. ACM, 75--85. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Niloufar Salehi, Lilly C. Irani, Michael S. Bernstein, Ali Alkhatib, Eva Ogbe, Kristy Milland, and others. 2015. We are dynamo: Overcoming stalling and friction in collective action for crowd workers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1621--1630. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Niloufar Salehi, Jaime Teevan, Shamsi Iqbal, and Ece Kamar. 2017. Communicating context to the crowd for complex writing tasks. In Proceedings of the ACM 2017 Conference on Computer Supported Cooperative Work. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Ryo Suzuki, Niloufar Salehi, Michelle S. Lam, Juan C. Marroquin, and Michael S. Bernstein. 2016. Atelier: Repurposing expert crowdsourcing tasks as micro-internships. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2645--2656. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Amos Tversky and Daniel Kahneman. 1992. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and Uncertainty 5, 4 (1992), 297--323.Google ScholarGoogle ScholarCross RefCross Ref
  25. Benjamin O. Weeks. 2004. The paradox of choice: Why more is less. The Academy of Management Executive 18, 4 (2004), 170--171.Google ScholarGoogle Scholar
  26. Howard M. Weiss and Daniel R. Ilgen. 1985. Routinized behavior in organizations. Journal of Behavioral Economics 14, 1 (1985), 57--67.Google ScholarGoogle ScholarCross RefCross Ref
  27. Mark E. Whiting, Dilrukshi Gamage, Snehalkumar Neil S. Gaikwad, Aaron Gilbee, Shirish Goyal, Alipta Ballav, Dinesh Majeti, Nalin Chhibber, Angela Richmond-Fuller, Freddie Vargus, Tejas Seshadri Sarma, Varshine Chandrakanthan, Teogenes Moura, Mohamed Hashim Salih, Gabriel Bayomi Tinoco Kalejaiye, Adam Ginzberg, Catherine A. Mullings, Yoni Dayan, Kristy Milland, Henrique Orefice, Jeff Regino, Sayna Parsi, Kunz Mainali, Vibhor Sehgal, Sekandar Matin, Akshansh Sinha, Rajan Vaish, and Michael S. Bernstein. 2017. Crowd guilds: Worker-led reputation and feedback on crowdsourcing platforms. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 1902–1913. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Ink: Increasing Worker Agency to Reduce Friction in Hiring Crowd Workers

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Computer-Human Interaction
      ACM Transactions on Computer-Human Interaction  Volume 25, Issue 2
      April 2018
      188 pages
      ISSN:1073-0516
      EISSN:1557-7325
      DOI:10.1145/3200181
      Issue’s Table of Contents

      Copyright © 2018 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 11 April 2018
      • Accepted: 1 December 2017
      • Revised: 1 November 2017
      • Received: 1 February 2017
      Published in tochi Volume 25, Issue 2

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Research
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader