Ujwal Gadiraju
Gianluca Demartini
ABSTRACT
Requesters on crowdsourcing platforms typically exercise the power to decide the fate of tasks completed by crowd workers. Rejecting work has a direct impact on workers; (i) they may not be rewarded for the work completed and for their effort that has been exerted, and (ii) rejection affects worker reputation and may limit their access to future work opportunities. This paper presents a comprehensive study that aims to understand worker moods and how workers react to rejections in microtask crowdsourcing. We experimentally investigate the effect of the mood of workers on their performance, and the interaction of their moods with their reactions to rejection. Finally, we explore techniques such as presenting social comparative explanations to foster positive reactions to rejection. We found that workers in pleasant moods significantly outperform those in unpleasant moods. Workers whose work is rejected due to narrowly failing pre-screening tests exhibited the most negative emotional responses.
- Christopher Beedie, Peter Terry, and Andrew Lane. 2005. Distinctions between emotion and mood. Cognition & Emotion , Vol. 19, 6 (2005), 847--878.Google Scholar
Cross Ref
- Kathy R Berenson, Geraldine Downey, Eshkol Rafaeli, Karin G Coifman, and Nina Leventhal Paquin. 2011. The rejection--rage contingency in borderline personality disorder. Journal of abnormal psychology , Vol. 120, 3 (2011), 681.Google ScholarCross Ref
- Kathy R Berenson, Anett Gyurak, Özlem Ayduk, Geraldine Downey, Matthew J Garner, Karin Mogg, Brendan P Bradley, and Daniel S Pine. 2009. Rejection sensitivity and disruption of attention by social threat cues. Journal of research in personality , Vol. 43, 6 (2009), 1064--1072.Google ScholarCross Ref
- Margaret M Bradley, Mark K Greenwald, Margaret C Petry, and Peter J Lang. 1992. Remembering pictures: pleasure and arousal in memory. Journal of experimental psychology: Learning, Memory, and Cognition , Vol. 18, 2 (1992), 379.Google ScholarCross Ref
- Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry , Vol. 25, 1 (1994), 49--59.Google ScholarCross Ref
- Ruth MJ Byrne. 2002. Mental models and counterfactual thoughts about what might have been. Trends in cognitive sciences , Vol. 6, 10 (2002), 426--431.Google Scholar
- Carrie J Cai, Shamsi T Iqbal, and Jaime Teevan. 2016. Chain reactions: The impact of order on microtask chains. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3143--3154. Google ScholarDigital Library
- Phillip Cassey and Tim M Blackburn. 2003. Publication rejection among ecologists. Trends in Ecology & Evolution , Vol. 18, 8 (2003), 375--376.Google ScholarCross Ref
- Richard J Davidson. 1994. On emotion, mood, and related affective constructs. The nature of emotion: Fundamental questions (1994), 51--55.Google Scholar
- Pieter MA Desmet, Vastenburg, and Natalia Romero. 2016a. Pick-A-Mood manual: Pictorial self-report scale for measuring mood states. (2016).Google Scholar
- Pieter MA Desmet, Martijn H Vastenburg, and Natalia Romero. 2016b. Mood measurement with Pick-A-Mood: review of current methods and design of a pictorial self-report scale. Journal of Design Research , Vol. 14, 3 (2016), 241--279.Google ScholarCross Ref
- Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G Ipeirotis, and Philippe Cudré-Mauroux. 2015. The dynamics of micro-task crowdsourcing: The case of amazon mturk. In Proceedings of the 24th International Conference on World Wide Web. 238--247. Google ScholarDigital Library
- Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 1013--1022. Google ScholarDigital Library
- Geraldine Downey and Scott I Feldman. 1996. Implications of rejection sensitivity for intimate relationships. Journal of personality and social psychology , Vol. 70, 6 (1996).Google ScholarCross Ref
- Julie S Downs, Mandy B Holbrook, Steve Sheng, and Lorrie Faith Cranor. 2010. Are your participants gaming the system?: screening mechanical turk workers. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 2399--2402. Google ScholarDigital Library
- Matthew T Feldner, Michael J Zvolensky, Georg H Eifert, and Adam P Spira. 2003. Emotional avoidance: An experimental test of individual differences and response suppression using biological challenge. Behaviour research and therapy , Vol. 41, 4 (2003), 403--411.Google Scholar
- Nico H Frijda et almbox. 1994. Varieties of affect: Emotions and episodes, moods, and sentiments. (1994).Google Scholar
- Ujwal Gadiraju, Gianluca Demartini, Ricardo Kawase, and Stefan Dietze. 2018. Crowd Anatomy Beyond the Good and Bad: Behavioral Traces for Crowd Worker Modeling and Pre-selection. Computer Supported Cooperative Work (CSCW) (2018), 1--27.Google Scholar
- Ujwal Gadiraju, Besnik Fetahu, and Ricardo Kawase. 2015. Training workers for improving performance in crowdsourcing microtasks. In Design for Teaching and Learning in a Networked World. Springer, 100--114.Google Scholar
- Ujwal Gadiraju, Besnik Fetahu, Ricardo Kawase, Patrick Siehndel, and Stefan Dietze. 2017a. Using worker self-assessments for competence-based pre-selection in crowdsourcing microtasks. ACM Transactions on Computer-Human Interaction (TOCHI) , Vol. 24, 4 (2017), 30. Google ScholarDigital Library
- Ujwal Gadiraju, Sebastian Möller, Martin Nöllenburg, Dietmar Saupe, Sebastian Egger-Lampl, Daniel Archambault, and Brian Fisher. 2017b. Crowdsourcing versus the laboratory: towards human-centered experiments using the crowd. In Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Springer, 6--26.Google Scholar
- Snehalkumar Neil S Gaikwad et almbox. 2016. Boomerang: Rebounding the consequences of reputation feedback on crowdsourcing platforms. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 625--637. Google ScholarDigital Library
- Stephen W Gilliland, Markus Groth, Robert C Baker IV, Angela E Dew, Lisa M Polly, and Jay C Langdon. 2001. Improving Applicants' Reactions to Rejection Letters: An Application of Fairness Theory. Personnel Psychology , Vol. 54, 3 (2001), 669--703.Google ScholarCross Ref
- Thorvald Hærem, Brian T Pentland, and Kent D Miller. 2015. Task complexity: Extending a core concept. Academy of Management Review , Vol. 40, 3 (2015), 446--460.Google ScholarCross Ref
- Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena, and Gianluca Demartini. 2019. All those wasted hours: On task abandonment in crowdsourcing. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. ACM, 321--329. Google ScholarDigital Library
- Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics (1979), 65--70.Google Scholar
- Ting-Hao Kenneth Huang, Walter S Lasecki, and Jeffrey P Bigham. 2015. Guardian: A crowd-powered spoken dialog system for web apis. In Third AAAI conference on human computation and crowdsourcing .Google Scholar
- Panagiotis G Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation. ACM, 64--67. Google ScholarDigital Library
- Lilly C Irani and M Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 611--620. Google ScholarDigital Library
- Fredric M Jablin and Kathleen Krone. 1984. Characteristics of rejection letters and their effects on job applicants. Written communication , Vol. 1, 4 (1984), 387--406.Google Scholar
- Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 Conference on Computer Supported Cooperative work. ACM, 1301--1318. Google ScholarDigital Library
- Chinmay E Kulkarni, Michael S Bernstein, and Scott R Klemmer. 2015. PeerStudio: rapid peer feedback emphasizes revision and improves performance. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale. ACM, 75--84. Google ScholarDigital Library
- Peter J Lang. 1985. The cognitive psychophysiology of emotion: Anxiety and the anxiety disorders. Lawrence Eribaum, Hillsdale (1985).Google Scholar
- Richard Lazarus. 1994. The stable and the unstable in emotion. The nature of emotion: Fundamental questions (1994), 79--85.Google Scholar
- Brian McInnis, Dan Cosley, Chaebong Nam, and Gilly Leshed. 2016. Taking a HIT: Designing around rejection, mistrust, risk, and workers' experiences in Amazon Mechanical Turk. In Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, 2271--2282. Google ScholarDigital Library
- Mary W Meagher, Randolph C Arnau, and Jamie L Rhudy. 2001. Pain and emotion: effects of affective picture modulation. Psychosomatic medicine , Vol. 63, 1 (2001), 79--90.Google Scholar
- Victoria Husted Medvec, Scott F Madey, and Thomas Gilovich. 1995. When less is more: counterfactual thinking and satisfaction among Olympic medalists. Journal of personality and social psychology , Vol. 69, 4 (1995), 603.Google ScholarCross Ref
- Jon D Morris. 1995. Observations: SAM: the Self-Assessment Manikin; an efficient cross-cultural measurement of emotional response. Journal of advertising research , Vol. 35, 6 (1995), 63--68.Google Scholar
- William N Morris. 2012. Mood: The frame of mind .Springer Sci. & Business Media.Google Scholar
- David Oleson, Alexander Sorokin, Greg P Laughlin, Vaughn Hester, John Le, and Lukas Biewald. 2011. Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing. Human computation , Vol. 11, 11 (2011). Google ScholarDigital Library
- James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psychometric properties of LIWC2015 . Technical Report.Google Scholar
- Jesse J Prinz. 2004. Gut reactions: A perceptual theory of emotion .Oxford UP.Google Scholar
- Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . 2931--2937.Google ScholarCross Ref
- Neal J Roese. 1997. Counterfactual thinking. Psychological bulletin , Vol. 121, 1 (1997), 133.Google Scholar
- Neal J Roese and James M Olson. 2014. What might have been: The social psychology of counterfactual thinking .Psychology Press.Google Scholar
- Niloufar Salehi, Lilly C Irani, Michael S Bernstein, Ali Alkhatib, Eva Ogbe, Kristy Milland, et almbox. 2015. We are dynamo: Overcoming stalling and friction in collective action for crowd workers. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. ACM, 1621--1630. Google ScholarDigital Library
- Yla R Tausczik and James W Pennebaker. 2010. The psychological meaning of words: LIWC and computerized text analysis methods. Journal of language and social psychology , Vol. 29, 1 (2010), 24--54.Google ScholarCross Ref
- Cesar Vandevelde, Francis Wyffels, Maria-Cristina Ciocci, Bram Vanderborght, and Jelle Saldien. 2016. Design and evaluation of a DIY construction system for educational robot kits. International Journal of Technology and Design Education , Vol. 26, 4 (2016), 521--540.Google ScholarCross Ref
- Philippe Verduyn, Iven Van Mechelen, and Francis Tuerlinckx. 2011. The relation between event processing and the duration of emotional experience. Emotion , Vol. 11, 1 (2011), 20.Google ScholarCross Ref
- Bjørn J Villa, Katrien De Moor, Poul E Heegaard, and Anders Instefjord. 2013. Investigating Quality of Experience in the context of adaptive video streaming: findings from an experimental user study. Akademika forlag Stavanger, Norway (2013).Google Scholar
- Svitlana Volkova, Kyle Shaffer, Jin Yea Jang, and Nathan Hodas. 2017. Separating facts from fiction: Linguistic models to classify suspicious and trusted news posts on twitter. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , Vol. 2. 647--653.Google ScholarCross Ref
- David Watson and Auke Tellegen. 1985. Toward a consensual structure of mood. Psychological bulletin , Vol. 98, 2 (1985), 219.Google Scholar
- Robert E Wood. 1986. Task complexity: Definition of the construct. Organizational behavior and human decision processes , Vol. 37, 1 (1986), 60--82.Google Scholar
- Karen L Woolley and J Patrick Barron. 2009. Handling manuscript rejection: insights from evidence and experience. Chest , Vol. 135, 2 (2009), 573--577.Google ScholarCross Ref
- Thomas A Wright and Russell Cropanzano. 2007. The happy/productive worker thesis revisited. In Research in personnel and human resources management. Emerald Group Publishing Limited, 269--307.Google Scholar
- Jie Yang, Judith Redi, Gianluca Demartini, and Alessandro Bozzon. 2016. Modeling task complexity in crowdsourcing. In Fourth AAAI Conference on Human Computation and Crowdsourcing .Google Scholar
- John M Zelenski, Steven A Murphy, and David A Jenkins. 2008. The happy-productive worker thesis revisited. Journal of Happiness Studies , Vol. 9, 4 (2008), 521--537.Google ScholarCross Ref
- Mengdie Zhuang and Ujwal Gadiraju. 2019. In What Mood Are You Today?: An Analysis of Crowd Workers' Mood, Performance and Engagement. In Proceedings of the 10th ACM Conference on Web Science. ACM, 373--382. Google ScholarDigital Library
Index Terms
- Understanding Worker Moods and Reactions to Rejection in Crowdsourcing
Recommendations
Improving Reactions to Rejection in Crowdsourcing Through Self-Reflection
WebSci '21: Proceedings of the 13th ACM Web Science Conference 2021In popular crowdsourcing marketplaces like Amazon Mechanical Turk, crowd workers complete tasks posted by requesters in return for monetary rewards. Task requesters are solely responsible for deciding whether to accept or reject submitted work. ...
Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments
The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing SystemsCrowdsourcing is increasingly being used as a means to tackle problems requiring human intelligence. With the ever-growing worker base that aims to complete microtasks on crowdsourcing platforms in exchange for financial gains, there is a need for ...
Comments