Skip to main content
SpringerLink
Log in

Comparing In-Person, Sona, and Mechanical Turk Measurements of Three Prejudice-Relevant Constructs

  • Published:
Current Psychology Aims and scope Submit manuscript

Abstract

Electronic data collection and participant pool management tools give researchers new ways to conduct research. The current study investigated the equivalency of in-person and online administrations of the Right-Wing Authoritarianism, Social Dominance Orientation, and Modern Racism scales across three modalities (administration in person, online through Sona Systems, and online through Mechanical Turk). Results indicate that in-person administration was largely equivalent to the randomly assigned online sample (Sona Systems) in terms of their intercorrelations, mean scores, variability, and reliability. However, the Sona sample consistently responded with strong attitudes for all measures, suggesting that social desirability may be decreased when completing these measures online. In addition, the Mechanical Turk sample differed in many ways from both in-person and Sona administration; although this nonequivalence is at least partially explained by sample demographic differences, other considerations may have exacerbated nonequivalence, including prior exposure to the measures and a desire to respond correctly.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Altemeyer, B. (1998). The other “Authoritarian” personality. Advances in Experimental Social Psychology, 30, 47–91.

    Article  Google Scholar 

  • Bäckström, M., & Björklund, F. (2007). Structural modeling of generalized prejudice: The role of social dominance, authoritarianism, and empathy. Journal of Individual Differences, 28, 10–17.

    Article  Google Scholar 

  • Batson, C. D., Naifeh, S. J., & Pate, S. (1978). Social desirability, religious orientation, and prejudice. Journal for the Scientific Study of Religion, 17, 31–41.

    Article  Google Scholar 

  • Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800–813.

    Article  PubMed  Google Scholar 

  • Berinsky, A. J., Huber, G. A., & Lenz, G. A. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20, 351–368.

    Article  Google Scholar 

  • Birnbaum, M. H. (2004). Human research and data collection via the internet. Annual Review of Psychology, 55, 803–832.

    Article  PubMed  Google Scholar 

  • Bowen, A. M., Daniel, C. M., Williams, M. L., & Baird, G. L. (2008). Identifying multiple submissions in internet research: preserving data integrity. AIDS Behavior, 12, 964–973.

    Article  PubMed  PubMed Central  Google Scholar 

  • Brock, T. C., & Becker, L. A. (1966). “Debriefing” and susceptibility to subsequent experimental manipulations. Journal of Experimental Social Psychology, 2, 314–323.

    Article  Google Scholar 

  • Buchanan, T. (2002). Online assessment: desirable or dangerous? Professional Psychology: Research and Practice, 33, 148–154.

    Article  Google Scholar 

  • Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5.

    Article  PubMed  Google Scholar 

  • Chambers, J. R., Schlenker, B. R., & Collisson, B. (2013). Ideology and prejudice: the role of value conflicts. Psychological Science, 24, 140–149.

    Article  PubMed  Google Scholar 

  • Chandler, J., Mueller, P., & Paolacci, G. (2014). Nonnaivete among Amazon Mechanical Turk workers: consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130.

    Article  PubMed  Google Scholar 

  • Crawford, J. T. (2012). The ideologically objectionable premise model: predicting biased political judgments on the left and right. Journal of Experimental Social Psychology, 48, 138–151.

    Article  Google Scholar 

  • Crawford, J. T., & Pilanski, J. M. (2014). The differential effects of right-wing authoritarianism and social dominance orientation on political intolerance. Political Psychology, 35, 557–576.

    Article  Google Scholar 

  • Duriez, B., & Van Hiel, A. (2002). The March of modern fascism: A comparison of social dominance orientation and authoritarianism. Personality and Individual Differences, 32, 1199–1213.

    Article  Google Scholar 

  • Ekehammar, B., Akrami, N., Gylje, M., & Zakrisson, I. (2004). What matters most to prejudice: Big Five personality, social dominance orientation, or right-wing authoritarianism? European Journal of Personality, 18, 463–482.

    Article  Google Scholar 

  • Evans, D., Garcia, D., Garcia, D., & Baron, R. (2003). In the privacy of their own homes: using the Internet to assess racial bias. Personality and Social Psychology Bulletin, 29, 273–284.

    Article  PubMed  Google Scholar 

  • Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26, 213–224.

    Article  Google Scholar 

  • Gosling, S., Vazire, S., Srivastava, S., & John, O. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59, 93–104.

    Article  PubMed  Google Scholar 

  • Ho, A. K., Sidanius, J., Pratto, F., Levin, S., Thomsen, L., Kteily, N., & Sheehy-Skeffington, J. (2012). Social dominance orientation: revisiting the structure and function of a variable predicting social and political attitudes. Personality and Social Psychology Bulletin, 38, 583–606.

    Article  PubMed  Google Scholar 

  • Krysan, M. (1998). Privacy and the expression of White racial attitudes. Public Opinion Quarterly, 62, 506–544.

    Article  Google Scholar 

  • Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44, 1–23.

    Article  PubMed  Google Scholar 

  • McConahay, J. (1986). Modern racism, ambivalence, and the Modern Racism Scale. In Prejudice, discrimination, and racism (pp. 91-125). San Diego: Academic Press.

  • Nederhof, A. J. (1985). Methods of coping with social desirability bias: a review. European Journal of Social Psychology, 15, 263–280.

    Article  Google Scholar 

  • Oliver, J. E., & Wong, J. (2003). Intergroup prejudice in multiethnic settings. American Journal of Political Science, 47, 567–582.

    Article  Google Scholar 

  • Paolacci, G., & Chandler, J. (2014). Inside the Turk: understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184–188.

    Article  Google Scholar 

  • Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.

    Google Scholar 

  • Pratto, F., Sidanius, J., Stallworth, L., & Malle, B. (1994). Social dominance orientation: a personality variable predicting social and political attitudes. Journal of Personality and Social Psychology, 67, 741–763.

    Article  Google Scholar 

  • Richman, W., Kiesler, S., Weisband, S., & Drasgow, F. (1999). A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews. Journal of Applied Psychology, 84, 754–775.

    Article  Google Scholar 

  • Roberts, L. D. (2007). Equivalence of electronic and off-line measures. In R. A. Reynolds, R. Woods, & J. D. Baker (Eds.), Handbook of research on electronic surveys and measurements (pp. 97–103). Hershey: Idea Group Reference/IGI Global.

    Chapter  Google Scholar 

  • Robins, R. W., Trzesniewski, K. H., Tracy, J. L., Gosling, S. D., & Potter, J. (2002). Global self-esteem across the life span. Psychology and Aging, 17, 423–434.

    Article  PubMed  Google Scholar 

  • Roese, N. J., & Jamieson, D. W. (1993). Twenty years of bogus pipeline research: a critical review and meta-analysis. Psychological Bulletin, 114, 363–375.

    Article  Google Scholar 

  • Schlachter, A., & Duckitt, J. (2002). Psychopathology, authoritarian attitudes, and prejudice. South African Journal of Psychology, 32, 1–8.

    Article  Google Scholar 

  • Sibley, C. G., & Duckitt, J. (2008). Personality and prejudice: a meta-analysis and theoretical review. Personality and Social Psychology Review, 12, 248–279.

    Article  PubMed  Google Scholar 

  • Sidanius, J., Levin, S., Liu, J., & Pratto, F. (2000). Social dominance orientation, anti-egalitarianism and the political psychology of gender: an extension and cross-cultural replication. European Journal of Social Psychology, 30, 41–67.

    Article  Google Scholar 

  • Srivastava, S., John, O. P., Gosling, S. D., & Potter, J. (2003). Development of personality in early and middle adulthood: Set like plaster or persistent change? Journal of Personality and Social Psychology, 84, 1041--1053.

  • Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133, 859–883.

    Article  PubMed  Google Scholar 

  • Trawalter, S., Hoffman, K. M., & Waytz, A. (2012). Racial bias in perceptions of others’ pain. PloS One, 7(11), 1–8.

    Article  Google Scholar 

  • Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453–458.

    Article  PubMed  Google Scholar 

  • von Hippel, W., Silver, L. A., & Lynch, M. E. (2000). Stereotyping against your will: the role of inhibitory ability in stereotyping and prejudice among the elderly. Personality and Social Psychology Bulletin, 26, 523–532.

    Article  Google Scholar 

  • Watson, P. J., Grisham, S. O., Trotter, M. V., & Biderman, M. D. (1984). Narcissism and empathy: validity evidence for the narcissistic personality inventory. Journal of Personality Assessment, 48, 301–305.

    Article  PubMed  Google Scholar 

  • Whitley, B. E. (1999). Right-wing authoritarianism, social dominance orientation, and prejudice. Journal of Personality and Social Psychology, 77, 126–134.

    Article  Google Scholar 

  • Wilson, M. S., & Sibley, C. G. (2013). Social dominance orientation and right-wing authoritarianism: additive and interactive effects on political conservatism. Political Psychology, 34, 277–284.

    Article  Google Scholar 

Download references

Acknowledgments

Thanks to Meredith Wells for reviewing a draft of the manuscript.

Author information

Authors and Affiliations

  1. Psychology Department, University of North Dakota, Grand Forks, ND, USA

    Bradlee W. Gamblin & Andre Kehn

  2. Psychology Department, Eastern Kentucky University, Richmond, KY, USA

    Matthew P. Winslow & Benjamin Lindsay

  3. Marquette University, Milwaukee, WI, USA

    Andrew W. Newsom

Authors
  1. Bradlee W. Gamblin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matthew P. Winslow
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Benjamin Lindsay
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andrew W. Newsom
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andre Kehn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bradlee W. Gamblin.

Ethics declarations

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Conflict of Interest

Bradlee W. Gamblin declares that he has no conflict of interest. Matthew P. Winslow declares that he has no conflict of interest. Benjamin Lindsay declares that he has no conflict of interest. Andrew W. Newsom declares that he has no conflict of interest. Andre Kehn declares that he has no conflict of interest.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gamblin, B.W., Winslow, M.P., Lindsay, B. et al. Comparing In-Person, Sona, and Mechanical Turk Measurements of Three Prejudice-Relevant Constructs. Curr Psychol 36, 217–224 (2017). https://doi.org/10.1007/s12144-015-9403-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12144-015-9403-1

Keywords

Search

Navigation