Skip to main content
SpringerLink
Log in

An effectiveness analysis of altmetrics indices for different levels of artificial intelligence publications

  • Published:
Scientometrics Aims and scope Submit manuscript

A Correction to this article was published on 08 September 2021

This article has been updated

Abstract

Altmetrics indices are increasingly applied to measure scholarly influence in recent years because they can reflect the influence of research outputs more timely comparing with traditional measurements. Simultaneously, artificial intelligence (AI), as an emerging interdiscipline, has a rapid development in these years. Traditional indices can’t reflect the influence of the AI research outputs quickly, thus more timely altmetrics indices are needed. In this paper, we conduct four studies about altmetrics indices and AI research outputs based on the datasets collected from Altmetric.com and Scopus database. First, we provide a review of the research status in the AI field. Second, we show the AI researches that attracted the most attention. Third, we demonstrate the general effectiveness of altmetrics indices in the AI field. Last, we examine the effectiveness of altmetrics indices for different levels of AI journal papers and AI conference papers. Our results indicate that there is a rapid increase of AI publications and the public has paid more attention to AI research outputs since 2011. It is found that altmetrics indices are effective to discriminate highly cited publications and publications whose citation counts increase quickly. Among all Altmetric sub-indicators, Number of Mendeley readers is the most effective. Moreover, the results indicate that altmetrics indices are more effective in high levels of AI journal papers and AI conference papers. The main contribution of this paper is investigating the effectiveness of altmetrics indices from the perspective of different levels of publications. This study lays the foundation for further investigations about effectiveness of altmetrics indices from new perspectives, and it has important implication for the studies about the impact of social media on the scientific community.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Change history

Notes

  1. https://www.altmetric.com/.

  2. https://jcr.incites.thomsonreuters.com.

  3. http://www.scimagojr.com.

  4. https://www.elsevier.com/solutions/scopus/features/metrics.

  5. https://journalmetrics.scopus.com.

  6. http://www.ccf.org.cn/xspj/gyml.

  7. http://www.core.edu.au/team.

  8. The information about the journals and conferences in Dataset 2 can be found at CCF’s official website, http://www.ccf.org.cn/xspj/gyml.

  9. https://www.ibm.com/watson/.

  10. http://www.nytimes-se.com/.

  11. https://deepmind.com/research/alphago/.

  12. https://www.altmetric.com/blog/announcing-sina-weibo-support/.

References

  • Adie, E., & Roe, W. (2013). Altmetric: Enriching scholarly content with article-level discussion and metrics. Learned Publishing, 26(1), 11–17.

    Google Scholar 

  • Altmetric.com. (2017). Sources of attention-altmetric track a unique range of online sources to capture the conversations relating to research outputs. https://www.altmetric.com/about-our%20-data/our-sources/.

  • Bando, K. (2014). Importance of Mendeley readership on altmetrics based on the Altmetric score. Journal of Molecular Structure, 435(2), 123–132.

    Google Scholar 

  • Bench-Capon, T. J., & Dunne, P. E. (2007). Argumentation in artificial intelligence. Artificial Intelligence, 171(10–15), 619–641.

    MathSciNet  MATH  Google Scholar 

  • Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8(4), 895–903.

    Google Scholar 

  • Boyd, S., & Vandenberghe, L. (2004). Convex optimization. Cambridge: Cambridge University Press.

    MATH  Google Scholar 

  • Colledge, L., Moya-Anegón, F. D., Guerrero-Bote, V., López-Illescas, C., Aisati, M. H. E., & Moed, H. (2010). SJR and SNIP: Two new journal metrics in Elsevier’s Scopus. Serials the Journal for the Serials Community, 23(3), 215–221.

    Google Scholar 

  • Costas, R., Zahedi, Z., & Wouters, P. (2015). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science & Technology, 66(10), 2003–2019.

    Google Scholar 

  • Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In Paper presented at the IEEE computer society conference on computer vision and pattern recognition.

  • Daud, A., Ahmad, M., Malik, M. S., & Che, D. (2015). Using machine learning techniques for rising star prediction in co-author network. Scientometrics, 102(2), 1687–1711.

    Google Scholar 

  • González-Pereira, B., Guerrero-Bote, V. P., & Moya-Anegón, F. (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of Informetrics, 4(3), 379–391.

    Google Scholar 

  • Gunther, E. (2011). Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. Journal of Medical Internet Research, 13(4), e123.

    Google Scholar 

  • Hartley, R., & Zisserman, A. (2003). Multiple view geometry in computer vision. Cambridge: Cambridge University Press.

    MATH  Google Scholar 

  • Harvey, C., Morris, H., Kelly, A., & Rowlinson, M. (2010). Academic journal quality guide. London: The Association of Business Schools.

    Google Scholar 

  • Hassan, S. U., & Gillani, U. A. (2016). Altmetrics of “altmetrics” using Google Scholar, Twitter, Mendeley, Facebook, Google-plus, CiteULike, Blogs and Wiki.

  • Hassan, S. U., & Haddawy, P. (2013). Measuring international knowledge flows and scholarly impact of scientific research. Scientometrics, 94(1), 163–179.

    Google Scholar 

  • Hassan, S. U., Imran, M., Gillani, U., Aljohani, N. R., Bowman, T. D., & Didegah, F. (2017). Measuring social media activity of scientific literature: An exhaustive comparison of scopus and novel altmetrics big data. Scientometrics, 113(2), 1037–1057.

    Google Scholar 

  • Haykin, S. (1994). Neural networks: A comprehensive foundation. Upper Saddle River: Prentice Hall.

    MATH  Google Scholar 

  • Heagerty, P. J., & Zheng, Y. (2005). Survival model predictive accuracy and ROC curves. Biometrics, 61(1), 92.

    MathSciNet  MATH  Google Scholar 

  • Holmberg, K., & Thelwall, M. (2014). Disciplinary differences in Twitter scholarly communication. Scientometrics, 101(2), 1027–1042.

    Google Scholar 

  • Hu, B., Lu, Z., Hang, L., & Chen, Q. (2014). Convolutional neural network architectures for matching natural language sentences. In Paper presented at the international conference on neural information processing systems.

  • Kemp, C., Tenenbaum, J. B., Griffiths, T. L., Yamada, T., & Ueda, N. (2006). Learning systems of concepts with an infinite relational model. In Paper presented at the national conference on artificial intelligence.

  • Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. In International conference on learning representations. arXiv:1412.6980.

  • Kousha, K., & Thelwall, M. (2017). Are wikipedia citations important evidence of the impact of scholarly articles and books? Journal of the Association for Information Science & Technology, 68(3), 762–779.

    Google Scholar 

  • Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Paper presented at the Advances in neural information processing systems.

  • Lavin, A., & Gray, S. (2016). Fast algorithms for convolutional neural networks. In Paper presented at the computer vision and pattern recognition.

  • Lee, Y. G., Lee, J. D., Song, Y. I., & Lee, S. J. (2007). An in-depth empirical analysis of patent citation counts using zero-inflated count data model: The case of KIST. Scientometrics, 70(1), 27–39.

    Google Scholar 

  • Ling, C. X., Huang, J., & Zhang, H. (2003). AUC: A better measure than accuracy in comparing learning algorithms. In Paper presented at the Canadian society for computational studies of intelligence conference on advances in artificial intelligence.

  • Liu, X. Z., & Fang, H. (2017). What we can learn from tweets linking to research papers. Scientometrics, 111, 1–21.

    Google Scholar 

  • Lyder, A. (2008). Mechanical design of Odin, an extendable heterogeneous deformable modular robot: Intelligent robots and systems. In Paper presented at the IEEE/RSJ international conference on intelligent robots and systems, 2008.

  • Metz, C. E. (1978). Basic principles of ROC analysis. Seminars in Nuclear Medicine, 8(4), 283.

    Google Scholar 

  • Ming, J., Han, J., & Danilevsky, M. (2011). Ranking-based classification of heterogeneous information networks. In Paper presented at the Acm Sigkdd international conference on knowledge discovery and data mining.

  • Moed, H. F. (2009). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265–277.

    Google Scholar 

  • Moed, H. F., & Halevi, G. (2015). Multidimensional assessment of scholarly research impact. Journal of the Association for Information Science & Technology, 66(10), 1988–2002.

    Google Scholar 

  • Nguyen, V., Tomatis, N., Tomatis, N., & Siegwart, R. (2007). A comparison of line extraction algorithms using 2D range data for indoor mobile robotics. Autonomous Robots, 23(2), 97–111.

    Google Scholar 

  • Noorden, R. V. (2014). Scientists and the social network. Nature, 512, 126–129.

    Google Scholar 

  • Peoples, B. K., Midway, S. R., Sackett, D., Lynch, A., & Cooney, P. B. (2016). Twitter predicts citation rates of ecological research. PLoS ONE, 11(11), e0166570.

    Google Scholar 

  • Piwowar, H. (2013). Altmetrics: Value all research products. Nature, 493(7431), 159.

    Google Scholar 

  • Pooladian, A., & Borrego, Á. (2017). Methodological issues in measuring citations in Wikipedia: A case study in library and information science. Scientometrics, 113(3), 1–10.

    Google Scholar 

  • Priem, J., & Hemminger, B. H. (2010). Scientometrics 2.0: New metrics of scholarly impact on the social Web. First Monday, 15(7). Available at: https://firstmonday.org/article/view/2874/2570/.

  • Priem, J., Piwowar, H. A., & Hemminger, B. M. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. arXiv:1203.4745.

  • Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Altmetrics: A manifesto. http://altmetrics.org/manifesto/.

  • Qian, Y., Rong, W., Jiang, N., Tang, J., & Xiong, Z. (2017). Citation regression analysis of computer science publications in different ranking categories and subfields. Scientometrics, 110(3), 1–24.

    Google Scholar 

  • Rendell, L. (1983). A new basis for state-space learning systems and a successful implementation. Artificial Intelligence, 20(4), 369–392.

    Google Scholar 

  • Ringelhan, S., Wollersheim, J., & Welpe, I. M. (2015). I like, i cite? Do Facebook likes predict the impact of scientific work? PLoS ONE, 10(8), e0134389.

    Google Scholar 

  • Russell, S. J., & Norvig, P. (2002). Artificial intelligence: A modern approach. Petaling Jaya: Pearson Education Limited.

    MATH  Google Scholar 

  • Salvatori, H. (2014). Coverage and adoption of altmetrics sources in the bibliometric community. Scientometrics. Scientometrics, 101(2), 1145–1163.

    Google Scholar 

  • Sause, M. G. R., Gribov, A., Unwin, A. R., & Horn, S. (2012). Pattern recognition approach to identify natural clusters of acoustic emission signals. Pattern Recognition Letters, 33(1), 17–23.

    Google Scholar 

  • Shu, F., & Haustein, S. (2017). On the citation advantage of tweeted papers at the journal level. Proceedings of the Association for Information Science & Technology, 54(1), 366–372.

    Google Scholar 

  • Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98(2), 1131–1143.

    Google Scholar 

  • Sugimoto, C. R., Work, S., Larivière, V., & Haustein, S. (2017). Scholarly use of social media and altmetrics: A review of the literature. Journal of the Association for Information Science & Technology, 68(9), 2037–2062.

    Google Scholar 

  • Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Paper presented at the proceedings of the IEEE conference on computer vision and pattern recognition.

  • Teixeira, D. S., Jaime, A., & Memon, A. R. (2017). CiteScore: A cite for sore eyes, or a valuable, transparent metric? Scientometrics, 111(1), 1–4.

    Google Scholar 

  • Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. R. (2013). Do altmetrics work? Twitter and ten other social web services. PLoS ONE, 8(5), e64841.

    Google Scholar 

  • Thelwall, M., & Wilson, P. (2016). Mendeley readership altmetrics for medical articles: An analysis of 45 fields. Journal of the Association for Information Science & Technology, 67(8), 1962–1972.

    Google Scholar 

  • Winter, J. C. F. D. (2015). The relationship between tweets, citations, and article views for PLOS ONE articles. Scientometrics, 102(2), 1773–1779.

    Google Scholar 

  • Wouters, P., & Costas, R. (2012). Users, narcissism and control—Tracking the impact of scholarly publications in the 21st century. Available at: https://apo.org.au/node/28603/.

  • Xia, F., Su, X., Wang, W., Zhang, C., Ning, Z., & Lee, I. (2016). Bibliographic analysis of nature based on Twitter and Facebook altmetrics data. PLoS ONE, 11(12), e0165997.

    Google Scholar 

  • Xin, S., Alberto, P., & Johan, B. (2012). How the scientific community reacts to newly submitted preprints: Article downloads, Twitter mentions, and citations. PLoS ONE, 7(11), e47523.

    Google Scholar 

  • Yu, H. (2017). Context of altmetrics data matters: An investigation of count type and user category. Scientometrics, 111(1), 1–17.

    Google Scholar 

  • Zahedi, Z., Costas, R., & Wouters, P. (2013). What is the impact of the publications read by the different Mendeley users? Could they help to identify alternative types of impact. In Plos Alm workshop.

Download references

Acknowledgements

The study is supported by funds from National Natural Science Foundation of China (Nos: 71722005 and 71571133 and 71790594 and 71790590). And from Natural Science Foundation of Tianjin (No. 18JCJQJC45900), the Humanities and Social Sciences Foundation of the Ministry of Education, China (Project No. 16YJC870011). We are grateful to Altmetric.com for providing the data.

Author information

Authors and Affiliations

  1. College of Management and Economics, Tianjin University, Tianjin, 300072, China

    Xi Zhang, Xianhai Wang & Hongke Zhao

  2. Department of Business Administration, University of Oviedo, Oviedo, Spain

    Patricia Ordóñez de Pablos

  3. School of Information Management, Wuhan University, Wuhan, 430072, China

    Yongqiang Sun

  4. Rutgers Business School - Newark and New Brunswick, Rutgers University, Newark, NJ, USA

    Hui Xiong

Authors
  1. Xi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xianhai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongke Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Patricia Ordóñez de Pablos
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yongqiang Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hui Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xi Zhang.

Additional information

The original online version of this article was revised: In the original publication of the the article, the reference Hassan et al. (2017) was published with incorrect details.

Appendices

Appendix A

See Table 11.

Table 11 Leading Journals and Leading Conferences in Dataset 1
Full size table

Appendix B

See Figs. 14 and 15.

Fig. 14
figure 14

The ROC curves of Altmetric attention score for discriminating highly cited publications in different levels of AI journal papers and AI conference papers

Full size image
Fig. 15
figure 15

The ROC curves of Altmetric attention score for discriminating publications whose citation counts increase quickly in different levels of AI journal papers and AI conference papers

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, X., Wang, X., Zhao, H. et al. An effectiveness analysis of altmetrics indices for different levels of artificial intelligence publications. Scientometrics 119, 1311–1344 (2019). https://doi.org/10.1007/s11192-019-03088-x

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-019-03088-x

Keywords

Search

Navigation