Skip to main content
SpringerLink
Log in

Evaluation and comparison of influence in international Open Access journals between China and USA

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

This paper makes an evaluation and comparison of influence on international Open Access (OA) between China and USA. First, Chinese and American OA journals are taken as an example. Second, combining traditional citation indicators with altmetrics indicators, a process of influence evaluation and comparison model on Chinese and American OA journals is constructed. Third, an evaluation indicator system of OA journals from two dimensions of academic influence and societal influence is constructed too. Lastly, through normalization, correlation analysis, reliability analysis, validity analysis, factor analysis and two-dimensional analysis, the academic and societal influence of OA journals are evaluated and compared between China and USA. It is found that in terms of academic influence and societal influence, the evaluation scores of international OA journals are quite different between China and USA. The influence of the USA is far greater than that of China. The results show a strong positive correlation among the 7 traditional citation indicators based on JCR, and a high positive correlation among the 8 altmetrics indicators based on Altmetrics Explorer, and the indicators have a significant consistency both overall and internal. There is a highly positive correlation between academic influence and societal influence scores for the Chinese and American OA journals. Overall, the results confirm that the evaluation of societal influence of OA journals is used as supplement to the evaluation of academic influence. Altmetrics is also a powerful tool to measure the societal influence of OA journals, but it cannot replace the traditional citation indicators at present.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Akers, K. G. (2017). Introducing altmetrics to the Journal of the Medical Library Association. Journal of the Medical Library Association Jmla, 105(3), 213–215. https://doi.org/10.5195/jmla.2017.250.

    Article  Google Scholar 

  • Amath, A., Ambacher, K., Leddy, J. J., Wood, T. J., & Ramnanan, C. J. (2017). Comparing alternative and traditional dissemination metrics in medical education. Medical Education, 51(9), 935.

    Google Scholar 

  • Bai, X., Xia, F., Ivan, L., Zhang, J., & Ning, Z. (2016). Identifying anomalous citations for objective evaluation of scholarly article impact. PLoS ONE, 11(9), e162364.

    Google Scholar 

  • Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8(4), 895–903.

    Google Scholar 

  • Bornmann, L. (2015). Usefulness of altmetrics for measuring the broader impact of research: A case study using data from PLOS and F1000Prime. Aslib Journal of Information Management, 67(3), 305–319.

    Google Scholar 

  • Bornmann, L., & Haunschild, R. (2018). Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data. PLoS ONE, 13(5), e0197133.

    Google Scholar 

  • Chang, Y. (2017). Comparative study of characteristics of authors between open access and non-open access journals in library and information science. Library & Information Science Research, 39(1), 8–15. https://doi.org/10.1016/j.lisr.2017.01.002.

    Article  Google Scholar 

  • Costas, R., Zahedi, Z., & Wouters, P. (2014). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science & Technology, 66(10), 2003–2019.

    Google Scholar 

  • Davis, P. M. (2011). Open access, readership, citations: A randomized controlled trial of scientific journal publishing. FASEB Journal, 25(7), 2129–2134. https://doi.org/10.1096/fj.11-183988.

    Article  Google Scholar 

  • Ezema, I. J., & Onyancha, O. B. (2017). Citation impact of health and medical journals in Africa: Does open accessibility matter? Electronic Library, 35(5), 934–952. https://doi.org/10.1108/EL-11-2016-0245.

    Article  Google Scholar 

  • Garfield, E. (1955). Citation indexes for science; a new dimension in documentation through association of ideas. Science, 122(3159), 108–111.

    Google Scholar 

  • Garfield, E. (1970). Citation indexing for studying science. Nature, 227(5260), 870.

    Google Scholar 

  • Guo, F., Xue, J., & Li, R. (2014). Open access in China: A study of social science journals. Journal of Scholarly Publishing, 45(4), 336–352. https://doi.org/10.3138/jsp.45.4.02.

    Article  Google Scholar 

  • Hallo, M., Lujan-Mora, S., & Mate, A. (2017). Evaluating open access journals using Semantic Web technologies and scorecards. Journal of Information Science, 43(1), 3–16. https://doi.org/10.1177/0165551515624353.

    Article  Google Scholar 

  • Harnad, S. (2009). Open access scientometrics and the UK Research Assessment Exercise. Scientometrics, 79(1), 147–156. https://doi.org/10.1007/s11192-009-0409-z.

    Article  Google Scholar 

  • Holmberg, K., & Park, H. W. (2018). An altmetric investigation of the online visibility of South Korea-based scientific journals. Scientometrics, 117(1), 603–613. https://doi.org/10.1007/s11192-018-2874-8.

    Article  Google Scholar 

  • Hu, D., Huang, B., & Zhou, W. (2012). Open access journals in China: The current situation and development strategies. Serials Review, 38(2), 86–92. https://doi.org/10.1016/j.serrev.2012.03.001.

    Article  Google Scholar 

  • Hua, F., Sun, H., Walsh, T., Glenny, A. M., & Worthington, H. (2017). Open access to journal articles in oncology: Current situation and citation impact. Annals of Oncology, 28(10), 2612–2617. https://doi.org/10.1093/annonc/mdx398.

    Article  Google Scholar 

  • Huang, W., Wang, P., & Wu, Q. (2018). A correlation comparison between Altmetric Attention Scores and citations for six PLOS journals. PLoS ONE, 13(4), e194962.

    Google Scholar 

  • Jabaley, C. S., Groff, R. F., Stentz, M. J., Moll, V., Lynde, G. C., Blum, J. M., et al. (2018). Highly visible sepsis publications from 2012 to 2017: Analysis and comparison of altmetrics and bibliometrics. Journal of Critical Care, 48, 357–371. https://doi.org/10.1016/j.jcrc.2018.09.033.

    Article  Google Scholar 

  • Jamali, H. R., & Alimohammadi, D. (2015). Blog citations as indicators of the societal impact of research: Content analysis of social sciences blogs. International Journal of Knowledge Content Development & Technology, 5(1), 15–32.

    Google Scholar 

  • Jan, V. (2003). Should scholarly societies embrace open access (or is it the kiss of death)? Learned Publishing, 16(3), 167–169.

    Google Scholar 

  • Jokic, M., Mervar, A., & Mateljan, S. (2018). Scientific potential of European fully open access journals. Scientometrics, 114(3), 1373–1394. https://doi.org/10.1007/s11192-017-2629-y.

    Article  Google Scholar 

  • Koler-Povh, T., Juznic, P., & Turk, G. (2014). Impact of open access on citation of scholarly publications in the field of civil engineering. Scientometrics, 98(2), 1033–1045. https://doi.org/10.1007/s11192-013-1101-x.

    Article  Google Scholar 

  • Lawrence, S. (2001). Free online availability substantially increases a paper’s impact. Nature, 411(6837), 521.

    Google Scholar 

  • Liu, C. L., Xu, Y. Q., Wu, H., Chen, S. S., & Guo, J. J. (2013). Correlation and interaction visualization of altmetric indicators extracted from scholarly social network activities: Dimensions and structure. Journal of Medical Internet Research, 15(11), e259. https://doi.org/10.2196/jmir.27.

    Article  Google Scholar 

  • Liu, W., & Li, Y. (2018). Open access publications in sciences and social sciences: A comparative analysis. Learned Publishing, 31(2), 107–119. https://doi.org/10.1002/leap.1114.

    Article  Google Scholar 

  • Luis Ortega, J. (2017). The presence of academic journals on Twitter and its relationship with dissemination (tweets) and research impact (citations). Aslib Journal of Information Management, 69(6), 674–687. https://doi.org/10.1108/AJIM-02-2017-0055.

    Article  Google Scholar 

  • Maflahi, N., & Thelwall, M. (2018). How quickly do publications get read? The evolution of Mendeley reader counts for new articles. Journal of the Association for Information Science and Technology, 69(1), 158–167. https://doi.org/10.1002/asi.23909.

    Article  Google Scholar 

  • Maggio, L. A., Leroux, T. C., Meyer, H. S., & Artino, A. R. (2018). #meded: Exploring the relationship between altmetrics and traditional measures of dissemination in health professions education. Perspectives on Medical Education, 7(4), 239–247.

    Google Scholar 

  • Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations/Robert K. Merton; edited and with an introd. by Norman W. Storer. Contemporary Sociology, 5(5), 52.

  • Merton, R. K. (1988). The Matthew effect in science, II: Cumulative Advantage and the symbolism of intellectual property. ISIS, 79(4), 606–623.

    Google Scholar 

  • Moed, H. F. (2009). New developments in the use of citation analysis in research evaluation. Archivum Immunologiae et Therapiae Experimentalis, 57(1), 13–18. https://doi.org/10.1007/s00005-009-0001-5.

    Article  Google Scholar 

  • Moed, H. F., & Halevi, G. (2015). Multidimensional assessment of scholarly research impact. Journal of the Association for Information Science & Technology, 66(10), 1988–2002.

    Google Scholar 

  • Moksness, L., & Olsen, S. O. (2017). Understanding researchers’ intention to publish in open access journals. Journal of Documentation, 73(6), 1149–1166. https://doi.org/10.1108/JD-02-2017-0019.

    Article  Google Scholar 

  • Pisoschi, A. M., & Pisoschi, C. G. (2016). Is open access the solution to increase the impact of scientific journals? Scientometrics, 109(2), 1075–1095. https://doi.org/10.1007/s11192-016-2088-x.

    Article  Google Scholar 

  • Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). altmetrics: A manifesto. Retrieved March 1, 2019, from http://altmetrics.org/manifesto/.

  • Shen, C. (2017). Open access scholarly journal publishing in Chinese. Publications. https://doi.org/10.3390/publications5040022.

    Article  Google Scholar 

  • Sotudeh, H., & Estakhr, Z. (2018). Sustainability of open access citation advantage: The case of Elsevier’s author-pays hybrid open access journals. Scientometrics, 115(1), 563–576. https://doi.org/10.1007/s11192-018-2663-4.

    Article  Google Scholar 

  • Spearman, C. (1987). The proof and measurement of association between two things. American Journal of Psychology, 100(3/4), 441–471.

    Google Scholar 

  • Sugimoto, C. R., Work, S., Larivière, V., & Haustein, S. (2017). Scholarly use of social media and altmetrics: A review of the literature. Journal of the Association for Information Science & Technology, 68(9), 2037–2062.

    Google Scholar 

  • Thelwall, M. (2017). Are Mendeley reader counts high enough for research evaluations when articles are published? Aslib Journal of Information Management, 69(2), 174–183. https://doi.org/10.1108/AJIM-01-2017-0028.

    Article  Google Scholar 

  • Thelwall, M., & Nevill, T. (2018). Could scientists use altmetric.com scores to predict longer term citation counts? Journal of Informetrics, 12(1), 237–248.

    Google Scholar 

  • Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365–391.

    MathSciNet  Google Scholar 

  • Werner, R. (2015). The focus on bibliometrics makes papers less useful. Nature, 517(7534), 245.

    Google Scholar 

  • Xia, J. (2010). A longitudinal study of scholars attitudes and behaviors toward open-access journal publishing. Journal of the Association for Information Science & Technology, 61(3), 615–624. https://doi.org/10.1002/asi.21283.

    Article  Google Scholar 

  • Yan, E., & Li, K. (2018). Which domains do open-access journals do best in? A 5-year longitudinal study. Journal of the Association for Information Science and Technology, 69(6), 844–856. https://doi.org/10.1002/asi.24002.

    Article  Google Scholar 

  • Yang, S., Xing, X., & Wolfram, D. (2018). Difference in the impact of open-access papers published by China and the USA. Scientometrics, 115(2), 1017–1037. https://doi.org/10.1007/s11192-018-2697-7.

    Article  Google Scholar 

  • Yu, H. (2017). Context of altmetrics data matters: An investigation of count type and user category. Scientometrics, 111(1), 1–17.

    Google Scholar 

  • Yu, L., Pan, Y., & Wu, Y. (2009). Research on data normalization methods in multi-attribute evaluation. Library and Information Service, 53(12), 136–139.

    Google Scholar 

  • Yuan, S., & Hua, W. (2011). Scholarly impact measurements of LIS open access journals: Based on citations and links. Electronic Library, 29(5), 682–697. https://doi.org/10.1108/02640471111177107.

    Article  MathSciNet  Google Scholar 

  • Zhang, D. X. (2014). Development of open access in China: Strategies, practices, challenges. Insights, 27(1), 45–50.

    Google Scholar 

  • Zhang, L., & Watson, E. M. (2017). Measuring the impact of gold and green open access. Journal of Academic Librarianship, 43(4), 337–345. https://doi.org/10.1016/j.acalib.2017.06.004.

    Article  Google Scholar 

  • Zygmont, C., & Smith, M. R. (2014). Robust factor analysis in the presence of normality violations, missing data, and outliers: Empirical questions and possible solutions. The Quantitative Methods for Psychology, 10(1), 40–55.

    Google Scholar 

Download references

Acknowledgements

Thank Altmetric.com for providing the dataset and anonymous reviewers for their useful comments. This research is funded by the National Social Science Found Major Project of China (18ZDA325), the National Social Science Found Project of China (16BTQ055) and the Fundamental Research Funds for the Central Universities (Independent Research Projects of Wuhan University (Humanities and Social Sciences)).

Author information

Authors and Affiliations

  1. Research Center for Chinese Science Evaluation, Wuhan University, Wuhan, 430072, China

    Rongying Zhao & Xu Wang

  2. Center for Studies of Information Resources, Wuhan University, Wuhan, 430072, China

    Rongying Zhao

  3. School of Information Management, Wuhan University, Wuhan, 430072, China

    Rongying Zhao & Xu Wang

Authors
  1. Rongying Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xu Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, R., Wang, X. Evaluation and comparison of influence in international Open Access journals between China and USA. Scientometrics 120, 1091–1110 (2019). https://doi.org/10.1007/s11192-019-03159-z

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-019-03159-z

Keywords

Search

Navigation