Abstract
Metrics like the number of tweets or Mendeley readers are currently discussed as an alternative to evaluate research. These alternative metrics (altmetrics) still need to be evaluated in order to fully understand their meaning, their benefits and limitations. While several preceding studies concentrate on correlations of altmetrics with classical measures like citations, this study aims at investigating metric-compatibility within altmetrics. For this purpose, 5000 journal articles from six disciplines have been analyzed regarding their metrics with the help of the aggregators PlumX and Altmetric.com. For this set, the highest numbers of events have been recognized regarding Mendeley readers, followed by Twitter and Facebook mentions. Thereby variations considering the aggregators could be observed. Intra-correlations between the metrics across one aggregator have been calculated, as well as inter-correlations for the corresponding metrics across the aggregators. For both aggregators, low to medium intra-correlations could be calculated which shows the diversity of the different metrics. Regarding inter-correlations, PlumX and Altmetric.com are highly consistent concerning Mendeley readers (\(r=0.97\)) and Wikipedia mentions (\(r=0.82\)), whereas the consistency concerning Twitter (\(r=0.49\)), blogs (\(r=0.46\)) and Reddit (\(r=0.41\)) is on a moderate level. The sources Facebook (\(r=0.29\)), Google+ (\(r=0.28\)) and News (\(r=0.11\)) show only low correlations.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Allen, P. (2017). World-wide news coverage in PlumX. https://plumanalytics.com/world-wide-news-coverage-plumx/. Accessed 2 Dec 2017.
Almind, T. C., & Ingwersen, P. (1997). Informetric analyses on the world wide web: Methodological approaches to ‘webometrics’. Journal of Documentation, 53(4), 404–426. https://doi.org/10.1108/EUM0000000007205.
Altmetric. (2015). Sources of attention. https://www.altmetric.com/about-our-data/our-sources/. Accessed 30 Jan 2018.
Altmetric. (2016a). How is the altmetric attention score calculated? https://help.altmetric.com/support/solutions/articles/6000060969-how-is-the-altmetric-score-calculated-. Accessed 30 Jan 2018.
Altmetric. (2016b). Mainstream media outlets. https://www.altmetric.com/about-our-data/our-sources/news/. Accessed 2 Dec 2017.
Altmetric. (2017). When did altmetric start tracking attention to each attention source? https://help.altmetric.com/support/solutions/articles/6000136884-when-did-altmetric-start-tracking-attention-to-each-attention-source-. Accessed 30 Jan 2018.
Bornmann, L. (2015). Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics. Scientometrics, 103(3), 1123–1144. https://doi.org/10.1007/s11192-015-1565-y.
Erdt, M., Nagarajan, A., Sin, S.-C. J., & Theng, Y.-L. (2016). Altmetrics: An analysis of the state-of-the-art in measuring research impact on social media. Scientometrics, 109(2), 1117–1166. https://doi.org/10.1007/s11192-016-2077-0.
Garfield, E. (1955). Citation indexes for science. Science, 122(3159), 108–111. https://doi.org/10.1126/science.123.3185.61-a.
Haustein, S. (2016). Grand challenges in altmetrics: Heterogeneity, data quality and dependencies. Scientometrics, 108(1), 413–423. https://doi.org/10.1007/s11192-016-1910-9.
Haustein, S., Bowman, T. D., & Costas, R. (2015). Interpreting “altmetrics”: Viewing acts on social media through the lens of citation and social theories. In C. R. Sugimoto (Ed.), Theories of informetrics and scholarly communication (pp. 372–405). Berlin/Boston: De Gruyter.
Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R., & Larivière, V. (2016). Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. Journal of the Association for Information Science and Technology, 67(1), 232–238. https://doi.org/10.1002/asi.23456.
Haustein, S., Peters, I., Bar-Ilan, J., Priem, J., Shema, H., & Terliesner, J. (2014). Coverage and adoption of altmetrics sources in the bibliometric community. Scientometrics, 101(2), 1145–1163. https://doi.org/10.1007/s11192-013-1221-3.
Haustein, S., & Siebenlist, T. (2011). Applying social bookmarking data to evaluate journal usage. Journal of Informetrics, 5(3), 446–457. https://doi.org/10.1016/j.joi.2011.04.002.
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). Bibliometrics: The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431. https://doi.org/10.1038/520429a.
Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572.
Holmberg, K., & Thelwall, M. (2014). Disciplinary differences in Twitter scholarly communication. Scientometrics, 101(2), 1027–1042. https://doi.org/10.1007/s11192-014-1229-3.
Jobmann, A., Hoffmann, C. P., Künne, S., Peters, I., Schmitz, J., & Wollnik-Korn, G. (2014). Altmetrics for large, multidisciplinary research groups: Comparison of current tools. Bibliometrie-Praxis und Forschung, 3(1), 1–19.
Lohr, C. (2017). New in captures: CiteULike readership. https://plumanalytics.com/new-in-captures-citeulike-readership/. Accessed 12 Dec 2017.
Martin, J. (2015). Responsible metrics. In J. Wilsdon (Ed.), The metric tide. Report of the independent review of the role of metrics in research assessment and management (pp. 134–147). HEFCE. https://doi.org/10.13140/RG.2.1.4929.1363.
Merton, R. K. (1988). The Matthew effect in science, II: Cumulative advantage and the symbolism of intellectual property. ISIS, 79(4), 606–623. https://doi.org/10.1086/354848.
Moed, H. F. (2009). New developments in the use of citation analysis in research evaluation. Archivum Immunologiae et Therapiae Experimentalis, 57(1), 13–18. https://doi.org/10.1007/s00005-009-0001-5.
Mukherjee, B., Subotić, S., & Chaubey, A. K. (2018). And now for something completely different: The congruence of the altmetric attention score’s structure between different article groups. Scientometrics, 114(1), 253–275. https://doi.org/10.1007/s11192-017-2559-8.
Nicholas, D., & Rowlands, I. (2011). Social media use in the research workflow. Information Services and Use, 31(1–2), 61–83. https://doi.org/10.3233/ISU-2011-0623.
Plum Analytics. (2016). Mention metrics. https://plumanalytics.com/learn/about-metrics/mention-metrics/. Accessed 2 Dec 2017.
Plum Analytics. (2017). Leadership. http://plumanalytics.com/about/leadership/. Accessed 30 Jan 2018.
Priem, J. (2010). I like the term #articlelevelmetrics, but it fails to imply *diversity* of measures. Lately, I’m liking #altmetrics. https://twitter.com/jasonpriem/status/25844968813. Accessed 30 Jan 2018.
Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Altmetrics: A manifesto. http://altmetrics.org/manifesto. Accessed 30 Jan 2018.
Shema, H., Bar-Ilan, J., & Thelwall, M. (2012). Research blogs and the discussion of scholarly information. PLoS ONE, 7(5), 1–8. https://doi.org/10.1371/journal.pone.0035869.
Spearman, C. (1904). The proof and measurement of association between two things. The American Journal of Psychology, 15(1), 72–101. https://doi.org/10.2307/1412159.
Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98(2), 1131–1143. https://doi.org/10.1007/s11192-013-1117-2.
Thelwall, M. (2016). Webometrics and altmetrics: Home birth vs. hospital birth. In C. R. Sugimoto (Ed.), Theories of informetrics and scholarly communication (pp. 337–346). Berlin/Boston: De Gruyter.
Torres-Salinas, D., Robinson-Garcia, N., & Jiménez-Contreras, E. (2016). Can we use altmetrics at the institutional level? A case study analysing the coverage by research areas of four Spanish Universities. In Proceedings of the 21st international conference on science and technology indicators, València, Spain. http://dx.doi.org/10.4995/STI2016.2016.xxxx.
Weller, K. (2015). Social media and altmetrics: An overview of current alternative approaches to measuring scholarly impact. In I. M. Welpe, J. Wollersheim, S. Ringelhan & M. Osterloh (Eds.), Incentives and performance: governance of research organizations (pp. 261–276). Cham, Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-319-09785-5_16.
Zahedi, Z., Costas, R., & Wouters, P. (2014). How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics, 101(2), 1491–1513. https://doi.org/10.1007/s11192-014-1264-0.
Zahedi, Z., Fenner, M., & Costas, R. (2014). How consistent are altmetrics providers? Study of 1000 PLOS ONE publications using the PLOS ALM, Mendeley and Altmetric. com APIs. altmetrics 14. In Workshop at the Web Science Conference Bloomington, USA, pp. 4–6. https://doi.org/10.6084/m9.figshare.1041821.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Meschede, C., Siebenlist, T. Cross-metric compatability and inconsistencies of altmetrics. Scientometrics 115, 283–297 (2018). https://doi.org/10.1007/s11192-018-2674-1
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-018-2674-1