Eunice Jun
Nigini Oliveira
Skip Abstract Section
Abstract
Online experiments allow researchers to collect data from large, demographically diverse global populations. Unlike in-lab studies, however, online experiments often fail to inform participants about the research to which they contribute. This paper is the first to investigate barriers that prevent researchers from providing such science communication in online experiments. We found that the main obstacles preventing researchers from including such information are assumptions about participant disinterest, limited time, concerns about losing anonymity, and concerns about experimental bias. Researchers also noted the dearth of tools to help them close the information loop with their study participants. Based on these findings, we formulated design requirements and implemented Digestif, a new web-based tool that supports researchers in providing their participants with science communication pages. Our evaluation shows that Digestif's scaffolding, examples, and nudges to focus on participants make researchers more aware of their participants' curiosity about research and more likely to disclose pertinent research information.
- David Armstrong, Ann Gosling, John Weinman, and Theresa Marteau. 1997. The place of inter-rater reliability in qualitative research: an empirical study. Sociology , Vol. 31, 3 (1997), 597--606.Google Scholar
Cross Ref
- American Psychological Association. 2017. Ethical Principles of Psychologists and Code of Conduct. (2017).Google Scholar
- Kimberly A Barchard and John Williams. 2008. Practical advice for conducting ethical online experiments and questionnaires for United States psychologists. Behavior Research Methods , Vol. 40, 4 (2008), 1111--1128.Google ScholarCross Ref
- Benjamin B Bederson and Alexander J Quinn. 2011. Web workers unite! addressing challenges of online laborers. In Extended Abstracts of ACM Conference on Human Factors in Computing Systems. ACM, 97--106. Google ScholarDigital Library
- Nicholas Behm, Sherry Rankins-Robertson, and Duane Roen. 2014. The case for academics as public intellectuals. Academe , Vol. 100, 1 (2014), 13.Google Scholar
- Adam J Berinsky, Gregory A Huber, and Gabriel S Lenz. 2012. Using Mechanical Turk as a subject recruitment tool for experimental research . Political Analysis , Vol. 20 (2012), 351--368.Google ScholarCross Ref
- Nathan Bos, Ann Zimmerman, Judith Olson, Jude Yew, Jason Yerkie, Erik Dahl, and Gary Olson. 2007. From shared databases to communities of practice: A taxonomy of collaboratories. Journal of Computer-Mediated Communication , Vol. 12, 2 (2007), 652--672.Google ScholarCross Ref
- Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology , Vol. 3, 2 (2006), 77--101.Google ScholarCross Ref
- Amy Bruckman. 2014. Research ethics and HCI. In Ways of Knowing in HCI . Springer New York, 449--468.Google Scholar
- Amy S Bruckman, Casey Fiesler, Jeff Hancock, and Cosmin Munteanu. 2017. CSCW research ethics town hall: Working towards community norms. In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing . ACM, 113--115. Google ScholarDigital Library
- Elizabeth A Buchanan. 1999. An overview of information ethics issues in a world-wide context. Ethics and Information Technology , Vol. 1, 3 (1999), 193--201. Google ScholarDigital Library
- Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. 2011. Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science , Vol. 6, 1 (2011), 3--5.Google ScholarCross Ref
- Marissa Burgermaster, Krzysztof Z Gajos, Patricia Davidson, and Lena Mamykina. 2017. The role of explanations in casual observational learning about nutrition. In Proceedings of ACM Conference on Human Factors in Computing Systems. ACM, 4097--4145. Google ScholarDigital Library
- Terry W Burns, D John O'Connor, and Susan M Stocklmayer. 2003. Science communication: a contemporary definition. Public Understanding of Science , Vol. 12, 2 (2003), 183--202.Google ScholarCross Ref
- Matthew JC Crump, John V McDonnell, and Todd M Gureckis. 2013. Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research. PloS one , Vol. 8, 3 (2013), e57410.Google ScholarCross Ref
- Django. 2018. https://www.djangoproject.com/, last accessed April 16, 2018.Google Scholar
- Jessica Ficler and Yoav Goldberg. 2017. Controlling linguistic style aspects in neural language generation. arXiv preprint arXiv:1707.02633 (2017).Google Scholar
- National Commission for the Protection of Human Subjects of Biome Beha Resea and Kenneth John Pres Ryan. 1978. The Belmont report: Ethical principles and guidelines for the protection of Human Subjects of Research-the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research .US Government Printing Office.Google Scholar
- Christopher Frauenberger, Amy S Bruckman, Cosmin Munteanu, Melissa Densmore, and Jenny Waycott. 2017. Research Ethics in HCI: A town hall meeting. In Extended Abstracts of Conference on Human Factors in Computing Systems. ACM, 1295--1299. Google ScholarDigital Library
- Snehal Neil Gaikwad, Durim Morina, Rohit Nistala, Megha Agarwal, Alison Cossette, Radhika Bhanu, Saiph Savage, Vishwajeet Narwal, Karan Rajpal, Jeff Regino, et almbox. 2015. Daemo: A self-governed crowdsourcing marketplace. In Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 101--102. Google ScholarDigital Library
- Laura Germine, Ken Nakayama, Bradley C Duchaine, Christopher F Chabris, Garga Chatterjee, and Jeremy B Wilmer. 2012. Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychonomic Bulletin & Review , Vol. 19, 5 (2012), 847--857.Google ScholarCross Ref
- Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. Hafez: an Interactive Poetry Generation System. Proceedings of ACL 2017, System Demonstrations (2017), 43--48.Google ScholarCross Ref
- Ivor Goodson. 1999. The educational researcher as a public intellectual. British Educational Research Journal , Vol. 25, 3 (1999), 277--297.Google ScholarCross Ref
- Samuel D Gosling, Simine Vazire, Sanjay Srivastava, and Oliver P John. 2004. Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist , Vol. 59, 2 (2004), 93.Google ScholarCross Ref
- Michael D Greenberg, Matthew W Easterday, and Elizabeth M Gerber. 2015. Critiki: A scaffolded approach to gathering design feedback from paid crowdworkers. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition. ACM, 235--244. Google ScholarDigital Library
- James C Hamilton. 1999. The ethics of conducting social-science research on the Internet. The Chronicle of Higher Education , Vol. 46, 15 (1999), B6--7.Google Scholar
- David R Hodge and David F Gillespie. 2007. Phrase completion scales: a better measurement approach than Likert scales? Journal of Social Service Research , Vol. 33, 4 (2007), 1--12.Google ScholarCross Ref
- J J Horton, D G Rand, and R J Zeckhauser. 2011. The online laboratory: Conducting experiments in a real labor market . Experimental Economics (2011).Google Scholar
- Panagiotis G Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality management on Amazon Mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation. ACM, New York, NY, USA, 64--67. Google ScholarDigital Library
- Lilly C Irani and M Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proc. ACM Conference on Human Factors in Computing Systems. ACM, 611--620. Google ScholarDigital Library
- jsPsych. 2017. http://www.jspsych.com, last accessed August 20, 2017.Google Scholar
- Eunice Jun, Morelle Arian, and Katharina Reinecke. 2018. The potential for scientific outreach and learning in mechanical turk experiments. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale. ACM. Google ScholarDigital Library
- Eunice Jun, Gary Hsieh, and Katharina Reinecke. 2017. Types of motivation affect study selection, attention, and dropouts in online experiments. ACM Human-Computer Interaction, Computer Supported Cooperative Work and Social Computing , Vol. 1, 2 (2017), 56. Google ScholarDigital Library
- Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work and Social Computing. ACM, 1301--1318. Google ScholarDigital Library
- Robert Kraut, Judith Olson, Mahzarin Banaji, Amy Bruckman, Jeffrey Cohen, and Mick Couper. 2004. Psychological research online: Report of Board of Scientific Affairs' Advisory Group on the conduct of research on the internet. American psychologist , Vol. 59, 2 (2004), 105.Google Scholar
- LabintheWild. 2017. http://www.labinthewild.org, last accessed September 13, 2017.Google Scholar
- Edith Law, Krzysztof Z Gajos, Andrea Wiggins, Mary L Gray, and Alex C Williams. 2017. Crowdsourcing as a Tool for Research: Implications of Uncertainty.. In Computer Supported Cooperative Work and Social Computing. 1544--1561. Google ScholarDigital Library
- Kurt Luther, Jari-Lee Tolentino, Wei Wu, Amy Pavel, Brian P Bailey, Maneesh Agrawala, Björn Hartmann, and Steven P Dow. 2015. Structuring, aggregating, and evaluating crowdsourced design critique. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 473--485. Google ScholarDigital Library
- Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods , Vol. 44, 1 (2012), 1--23.Google ScholarCross Ref
- MySocialBrain. 2017. http://www.mysocialbrain.org, last accessed August 20, 2017.Google Scholar
- Social Psychology Network. 2017. http://www.socialpsychology.org, last accessed August 20, 2017.Google Scholar
- Nigini Oliveira, Eunice Jun, and Katharina Reinecke. 2017. Citizen science opportunities in volunteer-based online experiments. In Proceedings of ACM Conference on Human Factors in Computing Systems. ACM, 6800--6812. Google ScholarDigital Library
- Gabriele Paolacci, Jesse Chandler, and Panagiotis G Ipeirotis. 2010. Running experiments on amazon mechanical turk. Judgment and Decision Making , Vol. 5, 5 (2010).Google Scholar
- Katharina Reinecke and Krzysztof Z Gajos. 2015. LabintheWild: Conducting large-scale online experiments with uncompensated samples. In Proceedings of ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 1364--1378. Google ScholarDigital Library
- Ulf-Dietrich Reips. 2000. The Web experiment method: Advantages, disadvantages, and solutions. Psychological Experiments on the Internet (2000), 89--117.Google Scholar
- Joel Ross, Lilly Irani, M Silberman, Andrew Zaldivar, and Bill Tomlinson. 2010. Who are the crowdworkers?: shifting demographics in mechanical turk. In Extended abstracts of ACM Conference on Human factors in Computing Systems. ACM, 2863--2872. Google ScholarDigital Library
- SciStarter. 2017. http://www.scistarter.com, last accessed August 20, 2017.Google Scholar
- Priya Sharma and Michael J Hannafin. 2007. Scaffolding in technology-enhanced learning environments. Interactive Learning Environments , Vol. 15, 1 (2007), 27--46.Google ScholarCross Ref
- M Silberman, Joel Ross, Lilly Irani, and Bill Tomlinson. 2010b. Sellers' problems in human computation markets. In Proceedings of the ACM SIGKDD Workshop on Human Computation. ACM, 18--21. Google ScholarDigital Library
- M Six Silberman, Lilly Irani, and R Ross. 2010a. Ethics and tactics of professional crowdwork. XRDS , Vol. 17, 2 (2010), 39--43. Google ScholarDigital Library
- British Psychological Society. 2007. British Psychological Society Report of the working party on conducting research on the Internet: Guidelines for ethical practice in psychological research online. http://www.bps.org.uk/sites/default/files/documents/conducting_research_on_the_internet-guidelines_for_ethical_practice_in_psychological_research_online.pdf.Google Scholar
- Elliot Soloway, Mark Guzdial, and Kenneth E Hay. 1994. Learner-centered design: The challenge for HCI in the 21st century. interactions , Vol. 1, 2 (1994), 36--48. Google ScholarDigital Library
- Richard H Thaler and Cass R Sunstein. 1999. Nudge: Improving decisions about health, wealth, and happiness .HeinOnline.Google Scholar
- Joseph B Walther. 2002. Research ethics in Internet-enabled research: Human subjects issues and methodological myopia. Ethics and Information Technology , Vol. 4, 3 (2002), 205--216. Google ScholarDigital Library
- Andrea Wiggins and Kevin Crowston. 2011. From conservation to crowdsourcing: A typology of citizen science. In Hawaii International Conference on System Sciences. IEEE, 1--10. Google ScholarDigital Library
- Madeline Wishart and Marion Kostanski. 2009. First Do No Harm: Valuing and Respecting the'Person'in Psychological Research Online. Counselling, Psychotherapy, and Health , Vol. 5, 1 (2009), 300--328.Google Scholar
- Games With Words. 2017. http://www.gameswithwords.org, last accessed August 15, 2017.Google Scholar
Index Terms
- Digestif: Promoting Science Communication in Online Experiments
Recommendations
Older adults' perceptions and experiences of online social support
This paper reports an investigation of older adults' needs and preferences concerning online social support. We focused our analysis on seven different aspects of online support: Self disclosure, Deep support, Light support, Community building, ...
Recruiting Older Adults in the Wild: Reflections on Challenges and Lessons Learned from Research Experience
PervasiveHealth '18: Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for HealthcareIt is important to understand the older adults prior to the design process. The understanding can better facilitate design conversations between the researchers and the older adults. In this paper, we discussed our experiences of building a relationship ...
What's going on? Age, distraction, and multitasking during online survey taking
Participants in an online survey revealed what other activities they engaged in while taking it.Younger people were more likely than older people to multitask.The relationship between age, multitasking, and sense of distraction was curvilinear.Most ...
Comments