Kostantinos Papadamou
Jeremy Blackburn
Emiliano De Cristofaro
Skip Abstract Section
Abstract
YouTube is by far the largest host of user-generated video content worldwide. Alas, the platform has also come under fire for hosting inappropriate, toxic, and hateful content. One community that has often been linked to sharing and publishing hateful and misogynistic content are the Involuntary Celibates (Incels), a loosely defined movement ostensibly focusing on men's issues. In this paper, we set out to analyze the Incel community on YouTube by focusing on this community's evolution over the last decade and understanding whether YouTube's recommendation algorithm steers users towards Incel-related videos. We collect videos shared on Incel communities within Reddit and perform a data-driven characterization of the content posted on YouTube.
Among other things, we find that the Incel community on YouTube is getting traction and that, during the last decade, the number of Incel-related videos and comments rose substantially. We also find that users have a 6.3% chance of being suggested an Incel-related video by YouTube's recommendation algorithm within five hops when starting from a non Incel-related video. Overall, our findings paint an alarming picture of online radicalization: not only Incel activity is increasing over time, but platforms may also play an active role in steering users towards such extreme content.
- Swati Agarwal and Ashish Sureka. 2014. A Focused Crawler for Mining Hate and Extremism Promoting Videos on YouTube. In Proceedings of the 25th ACM conference on Hypertext and social media.Google Scholar
Digital Library
- Stephane J. Baele, Lewys Brace, and Travis G. Coan. 2019. From "Incel" to "Saint": Analyzing the violent worldview behind the 2018 Toronto attack. In Terrorism and Political Violence. Routledge.Google Scholar
- Nadia Bashir. 2018. Doing Research in Peoples' Homes: Fieldwork, Ethics and Safety--On the Practical Challenges of Researching and Representing Life on the Margins. In Qualitative Research.Google Scholar
- Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The Pushshift Reddit Dataset. In Proceedings of the International AAAI Conference on Web and Social Media.Google ScholarCross Ref
- BBC. 2018. How rampage killer became misogynist “hero”. https://www.bbc.com/news/world-us-canada-43892189.Google Scholar
- Barbara Beale, Rose Cole, Sharon Hillege, Rose McMaster, and Sue Nagy. 2004. Impact of In-depth Interviews on the Interviewer: Roller Coaster Ride. In Nursing & health sciences.Google Scholar
- Zack Beauchamp. 2018. Incel, the misogynist ideology that inspired the deadly Toronto attack, explained. https://www.vox.com/world/2018/4/25/17277496/incel-toronto-attack-alek-minassian.Google Scholar
- Nicholas Blagden and Sarah Pemberton. 2010. The Challenge in Conducting Qualitative Research With Convicted Sex Offenders. In The Howard Journal of Criminal Justice.Google Scholar
- Mélissa Blais and Francis Dupuis-Déri. 2012. Masculinism and the Antifeminist Countermovement. In Social Movement Studies. Taylor & Francis.Google Scholar
- Jack Bratich and Sarah Banet-Weiser. 2019. From Pick-Up Artists to Incels: Con(fidence) Games, Networked Misogyny, and the Failure of Neoliberalism. In International Journal of Communication.Google Scholar
- Leyland Cecco. 2019. Toronto van attack suspect says he was 'radicalized' online by 'incels'. https://www.theguardian.com/world/2019/sep/27/alek-minassian-toronto-van-attack-interview-incels.Google Scholar
- Southern Poverty Law Center. 2019. Male Supremacy. https://www.splcenter.org/fighting-hate/extremist-files/ideology/male-supremacy.Google Scholar
- Eshwar Chandrasekharan, Umashanthi Pavalanathan, Anirudh Srinivasan, Adam Glynn, Jacob Eisenstein, and Eric Gilbert. 2017. You Can't Stay Here: The Efficacy of Reddit's 2015 Ban Examined Through Hate Speech. In Proceedings of the ACM on Human-Computer Interaction. ACM New York, NY, USA.Google ScholarDigital Library
- Matteo Cinelli, Gianmarco De Francisci Morales, Alessandro Galeazzi, Walter Quattrociocchi, and Michele Starnini. 2021. The Echo Chamber Effect on Social Media. In Proceedings of the National Academy of Sciences.Google Scholar
- Allie Conti. 2018. Learn to Decode the Secret Language of the Incel Subculture. https://www.vice.com/en/article/7xmaze/learn-to-decode-the-secret-language-of-the-incel-subculture.Google Scholar
- Jesselyn Cook. 2018. Inside Incels' Looksmaxing Obsession: Penis Stretching, Skull Implants And Rage. https://www.huffpost.com/entry/incels-looksmaxing-obsession_n_5b50e56ee4b0de86f48b0a4f.Google Scholar
- Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM conference on recommender systems.Google ScholarDigital Library
- David Dittrich, Erin Kenneally, et al. 2012. The Menlo Report: Ethical Principles Guiding Information and Communication Technology Research. Technical Report. US Department of Homeland Security.Google Scholar
- Nicola Döring and M Rohangis Mohseni. 2019. Male Dominance and Sexism on YouTube: Results of Three Content Analyses. In Feminist Media Studies. Taylor & Francis.Google Scholar
- The Fifth Estate. 2019. Why incels are a 'real and present threat' for Canadians. https://www.cbc.ca/news/canada/incel-threat-canadians-fifth-estate-1.4992184.Google Scholar
- Tracie Farrell, Miriam Fernandez, Jakub Novotny, and Harith Alani. 2019. Exploring Misogyny Across the Manosphere in Reddit. In Proceedings of the 10th ACM Conference on Web Science.Google ScholarDigital Library
- Warren Farrell. 1996. The Myth of Male Power. Berkeley Publishing Group.Google Scholar
- Ronald A Fisher. 1922. On the Interpretation of χ 2 From Contingency Tables, and the Calculation of P. Journal of the Royal Statistical Society (1922).Google Scholar
- Joseph L Fleiss. 1971. Measuring Nominal Scale Agreement Among Many Raters. In Psychological bulletin.Google Scholar
- Theodoros Giannakopoulos, Aggelos Pikrakis, and Sergios Theodoridis. 2010. A Multimodal Approach to Violence Detection in Video Sharing Sites. In 2010 20th International Conference on Pattern Recognition. IEEE.Google ScholarDigital Library
- Debbie Ging. 2019. Alphas, Betas, and Incels: Theorizing the Masculinities of the Manosphere. In Men and Masculinities. SAGE Publications Sage CA: Los Angeles, CA.Google Scholar
- Google Developers. 2020. YouTube Data API. https://developers.google.com/youtube/v3/.Google Scholar
- Lise Gotell and Emily Dutton. 2016. Sexual Violence in the “Manosphere”: Antifeminist Men's Rights Discourses on Rape. In International Journal for Crime, Justice and Social Democracy. Queensland University of Technology.Google Scholar
- Christine Hauser. 2017. Reddit bans "Incel" group for inciting violence against women. https://www.nytimes.com/2017/11/09/technology/incels-reddit-banned.html.Google Scholar
- Bruce Hoffman, Jacob Ware, and Ezra Shapiro. 2020. Assessing the Threat of Incel Violence. In Studies in Conflict & Terrorism. Taylor & Francis.Google Scholar
- Zhané Hunte and Kerstin Engström. 2019. “Female Nature, Cucks, and Simps”: Understanding Men Going Their Own Way as Part of the Manosphere. Master's thesis.Google Scholar
- Eslam Hussein, Prerna Juneja, and Tanushree Mitra. 2020. Measuring Misinformation in Video Search Platforms: An Audit Study on YouTube. Proceedings of the ACM on Human-Computer Interaction CSCW (2020).Google ScholarDigital Library
- Incels Wiki. 2019 a. Blackpill. https://incels.wiki/w/Blackpill.Google Scholar
- Incels Wiki. 2019 b. Incel Forums Term Glossary. https://incels.wiki/w/Incel_Forums_Term_Glossary.Google Scholar
- Incels Wiki. 2019 c. The Incel Wiki. https://incels.wiki.Google Scholar
- Sylvia Jaki, Tom De Smedt, Maja Gwó'zd'z, Rudresh Panchal, Alexander Rossa, and Guy De Pauw. 2019. Online Hatred of Women in the Incels.me Forum: Linguistic Analysis and Automatic Detection. In Journal of Language Aggression and Conflict.Google Scholar
- Shan Jiang, Ronald E Robertson, and Christo Wilson. 2019. Bias Misperceived: The Role of Partisanship and Misinformation in YouTube Comment Moderation. In Proceedings of the International AAAI Conference on Web and Social Media.Google Scholar
- Jonas Kaiser and Adrian Rauchfleisch. 2018. Unite the right? How YouTube's recommendation algorithm connects the US Far-Right. In D&S Media Manipulation.Google Scholar
- Kathleen E Kiernan. 1988. Who Remains Celibate?. In Journal of Biosocial Science. Cambridge University Press.Google Scholar
- J Richard Landis and Gary G Koch. 1977. The Measurement of Observer Agreement for Categorical Data. In Biometrics. JSTOR.Google Scholar
- Jie Liang Lin. 2017. Antifeminism Online: MGTOW (Men Going Their Own Way). JSTOR.Google Scholar
- Digital Media Literacy. 2021. What is an echo chamber? https://edu.gcfglobal.org/en/digital-media-literacy/what-is-an-echo-chamber/1/.Google Scholar
- Enrico Mariconti, Guillermo Suarez-Tangil, Jeremy Blackburn, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Jordi Luque Serrano, and Gianluca Stringhini. 2019. “You Know What to Do”: Proactive Detection of YouTube Videos Targeted by Coordinated Hate Attacks. In Proceedings of the ACM on Human-Computer Interaction. ACM New York, NY, USA.Google ScholarDigital Library
- Paris Martineau. 2019. YouTube Is Banning Extremist Videos. Will It Work? https://www.wired.com/story/how-effective-youtube-latest-ban-extremism/.Google Scholar
- Alice Marwick and Rebecca Lewis. 2017. Media Manipulation and Disinformation Online. New York: Data & Society Research Institute.Google Scholar
- Adrienne Massanari. 2017. # Gamergate and The Fappening: How Reddit's Algorithm, Governance, and Culture Support Toxic Technocultures. In New Media & Society. Sage Publications Sage UK: London, England.Google Scholar
- Frank J Massey Jr. 1951. The Kolmogorov-Smirnov Test for Goodness of Fit. In Journal of the American statistical Association. Taylor & Francis.Google Scholar
- December Maxwell, Sarah R Robinson, Jessica R Williams, and Craig Keaton. 2020. "A Short Story of a Lonely Guy": A Qualitative Thematic Analysis of Involuntary Celibacy Using Reddit. In Sexuality & Culture. Springer.Google Scholar
- Lauren Menzie. 2020. Stacys, Beckys, and Chads: The Construction of Femininity and Hegemonic Masculinity Within Incel Rhetoric. In Psychology & Sexuality. Taylor & Francis.Google Scholar
- Michael A Messner. 1998. The Limits of "The Male Sex Role" An Analysis of the Men's Liberation and Men's Rights Movements' Discourse. In Gender & Society. SAGE Publications, Inc.Google Scholar
- Mozila Foundation. 2019. Got YouTube Regrets? Thousands do! https://foundation.mozilla.org/en/campaigns/youtube-regrets/.Google Scholar
- Angela Nagle. 2015. An Investigation into Contemporary Online Anti-feminist Movements. Ph.D. Dissertation. Dublin City University.Google Scholar
- Edward Newell, David Jurgens, Haji Saleem, Hardik Vala, Jad Sassine, Caitrin Armstrong, and Derek Ruths. 2016. User Migration in Online Social Networks: A case study on Reddit during a period of Community Unrest. In Proceedings of the International AAAI Conference on Web and Social Media.Google Scholar
- Caroline O'Donovan, Charlie Warzel, Logan McDonald, Brian Clifton, and Max Woolf. 2019. We Followed YouTube's Recommendation Algorithm Down The Rabbit Hole. https://www.buzzfeednews.com/article/carolineodonovan/down-youtubes-recommendation-rabbithole.Google Scholar
- Association of Internet Research. 2019. Internet Research: Ethical Guidelines 3.0. https://aoir.org/reports/ethics3.pdf.Google Scholar
- Abby Ohlheiser. 2018. Inside the online world of "incels," the dark corner of the Internet linked to the Toronto suspect. https://tinyurl.com/incels-link-toronto-suspect.Google Scholar
- Raphael Ottoni, Evandro Cunha, Gabriel Magno, Pedro Bernardina, Wagner Meira Jr, and Virg'ilio Almeida. 2018. Analyzing Right-wing YouTube Channels: Hate, Violence and Discrimination. In Proceedings of the 10th ACM Conference on Web Science.Google ScholarDigital Library
- Kostantinos Papadamou, Antonis Papasavva, Savvas Zannettou, Jeremy Blackburn, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, and Michael Sirivianos. 2020. Disturbed YouTube for Kids: Characterizing and Detecting Inappropriate Videos Targeting Young Children. In Proceedings of the International AAAI Conference on Web and Social Media.Google ScholarCross Ref
- Eli Pariser. 2011. The Filter Bubble: How the New Personalized Web is Changing What We Read and How We Think. Penguin.Google ScholarDigital Library
- Rational Wiki. 2019. Incel. https://rationalwiki.org/wiki/Incel.Google Scholar
- Manoel Horta Ribeiro, Jeremy Blackburn, Barry Bradlyn, Emiliano De Cristofaro, Gianluca Stringhini, Summer Long, Stephanie Greenberg, and Savvas Zannettou. 2021. The Evolution of the Manosphere Across the Web. In Proceedings of the International AAAI Conference on Web and Social Media.Google Scholar
- Manoel Horta Ribeiro, Shagun Jhaver, Savvas Zannettou, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, and Robert West. 2020 a. Does Platform Migration Compromise Content Moderation? Evidence from r/The_Donald and r/Incels. In arXiv preprint 2010.10397.Google Scholar
- Manoel Horta Ribeiro, Raphael Ottoni, Robert West, Virg'ilio AF Almeida, and Wagner Meira Jr. 2020 b. Auditing Radicalization Pathways on YouTube. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency.Google ScholarDigital Library
- Caitlin M Rivers and Bryan L Lewis. 2014. Ethical Research Standards in a World of Big Data. In F1000Research, Vol. 3.Google Scholar
- Adi Robertson. 2019. Reddit has broadened its anti-harassment rules and banned a major incel forum. https://www.theverge.com/2019/9/30/20891920/reddit-harassment-bullying-threats-new-policy-change-rules-subreddits.Google Scholar
- Kevin Roose. 2019. The Making of a YouTube Radical. https://tinyurl.com/nytimes-youtube-radical.Google Scholar
- Sdner Sara, Kyung Lah, Steve Almasy, and Ralph Ellis. 2015. Oregon Shooting: Gunman a Student at Umpqua Community College. https://tinyurl.com/umpqua-community-college-shoot.Google Scholar
- Christian Stöcker and Mike Preuss. 2020. Riding the Wave of Misclassification: How We End up with Extreme YouTube Content. In International Conference on Human-Computer Interaction. Springer.Google ScholarDigital Library
- Jonathan Stray. 2020. Aligning AI Optimization to Community Well-Being. International Journal of Community Well-Being, Vol. 3, 4 (2020), 443--463.Google ScholarCross Ref
- Ashish Sureka, Ponnurangam Kumaraguru, Atul Goyal, and Sidharth Chhabra. 2010. Mining YouTube to Discover Extremist Videos, Users and Hidden Communities. In Asia Information Retrieval Symposium. Springer.Google Scholar
- Rashid Tahir, Faizan Ahmed, Hammas Saeed, Shiza Ali, Fareed Zaffar, and Christo Wilson. 2019. Bringing the Kid Back into YouTube Kids : Detecting Inappropriate Content on Video Streaming Platforms. In IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). IEEE.Google ScholarDigital Library
- The YouTube Team. 2019. Our ongoing work to tackle hate. https://blog.youtube/news-and-events/our-ongoing-work-to-tackle-hate.Google Scholar
- Zeynep Tufekci. 2018. YouTube, The Great Radicalizer. In The New York Times.Google Scholar
- MK Vijaymeena and K Kavitha. 2016. A Survey on Similarity Measures in Text Mining. In Machine Learning and Applications: An International Journal (MLAIJ).Google Scholar
- Matthew Weaver and Steven Morris. 2021. Plymouth gunman: a hate-filled misogynist and `Incel'. https://www.theguardian.com/uk-news/2021/aug/13/plymouth-shooting-suspect-what-we-know-jake-davison.Google Scholar
- Wikipedia. 2019. 2014 Isla Vista Killings. https://en.wikipedia.org/wiki/2014_Isla_Vista_killings.Google Scholar
- Lindsey Wotanis and Laurie McMillan. 2014. Performing Gender on YouTube: How Jenna Marbles Negotiates a Hostile Online Environment. Feminist Media Studies.Google Scholar
- Savvas Zannettou, Sotirios Chatzis, Kostantinos Papadamou, and Michael Sirivianos. 2018. The Good, the Bad and the Bait: Detecting and Characterizing Clickbait on YouTube. In 2018 IEEE Security and Privacy Workshops (SPW). IEEE.Google ScholarCross Ref
- Zhe Zhao, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamoorthy, Xinyang Yi, and Ed Chi. 2019. Recommending What Video to Watch Next: A Multitask Ranking System. In Proceedings of the 13th ACM Conference on Recommender Systems.Google ScholarDigital Library
- Shannon Zimmerman, Luisa Ryan, and David Duriesmith. 2018. Recognizing the violent extremist ideology of 'Incels'. In Women in International Security Policy.Google Scholar
Index Terms
- "How over is it?" Understanding the Incel Community on YouTube
Recommendations
Are Anti-Feminist Communities Gateways to the Far Right? Evidence from Reddit and YouTube
WebSci '21: Proceedings of the 13th ACM Web Science Conference 2021Researchers have suggested that “the Manosphere,” a conglomerate of men-centered online communities, may serve as a gateway to far right movements. In that context, this paper quantitatively studies the migratory patterns between a variety of groups ...
Analyzing Hate Speech with Incel-Hunters’ Critiques
SMSociety'20: International Conference on Social Media and SocietyThe ubiquity of online media services helps to promote free speech but also provides opportunities for the spread of problematic content such as hate speech. A group of individuals known as “incels” (involuntary celibates) sometimes use online media ...
Disinformation Warfare: Understanding State-Sponsored Trolls on Twitter and Their Influence on the Web
WWW '19: Companion Proceedings of The 2019 World Wide Web ConferenceOver the past couple of years, anecdotal evidence has emerged linking coordinated campaigns by state-sponsored actors with efforts to manipulate public opinion on the Web, often around major political events, through dedicated accounts, or “trolls.” ...
Comments