Skip to main content
SpringerLink
Log in

Citation time window choice for research impact evaluation

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

This paper aims to inform choice of citation time window for research evaluation, by answering three questions: (1) How accurate is it to use citation counts in short time windows to approximate total citations? (2) How does citation ageing vary by research fields, document types, publication months, and total citations? (3) Can field normalization improve the accuracy of using short citation time windows? We investigate the 31-year life time non-self-citation processes of all Thomson Reuters Web of Science journal papers published in 1980. The correlation between non-self-citation counts in each time window and total non-self-citations in all 31 years is calculated, and it is lower for more highly cited papers than less highly cited ones. There are significant differences in citation ageing between different research fields, document types, total citation counts, and publication months. However, the within group differences are more striking; many papers in the slowest ageing field may still age faster than many papers in the fastest ageing field. Furthermore, field normalization cannot improve the accuracy of using short citation time windows. Implications and recommendations for choosing adequate citation time windows are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Abbott, A. (2009). Italy introduces performance-related funding. Nature, 460(7255), 559. doi:10.1038/460559a.

    Article  Google Scholar 

  • Abramo, G., Cicero, T., & D’Angelo, C. A. (2011). Assessing the varying level of impact measurement accuracy as a function of the citation window length. Journal of Informetrics, 5(4), 659–667. doi:10.1016/j.joi.2011.06.004.

    Article  Google Scholar 

  • Abramo, G., Cicero, T., & D’Angelo, C. A. (2012a). A sensitivity analysis of research institutions’ productivity rankings to the time of citation observation. Journal of Informetrics, 6(2), 298–306. doi:10.1016/j.joi.2011.11.005.

    Article  Google Scholar 

  • Abramo, G., Cicero, T., & D’Angelo, C. A. (2012b). A sensitivity analysis of researchers’ productivity rankings to the time of citation observation. Journal of Informetrics, 6(2), 192–201. doi:10.1016/j.joi.2011.12.003.

    Article  Google Scholar 

  • Adams, J. (2005). Early citation counts correlate with accumulated impact. Scientometrics, 63(3), 567–581.

    Article  Google Scholar 

  • Aksnes, D. W. (2003a). Characteristics of highly cited papers. Research Evaluation, 12(3), 159–170.

    Article  Google Scholar 

  • Aksnes, D. W. (2003b). A macro study of self-citation. Scientometrics, 56(2), 235–246. doi:10.1023/a:1021919228368.

    Article  Google Scholar 

  • Aversa, E. S. (1985). Citation patterns of highly cited papers and their relationship to literature aging: A study of the working literature. Scientometrics, 7(3), 383–389.

    Article  MathSciNet  Google Scholar 

  • Costas, R., Van Leeuwen, T. N., & van Raan, A. F. J. (2010). Is scientific literature subject to a ‘Sell By Date’? A general methodology to analyze the ‘durability’of scientific documents. Journal of the American Society for Information Science and Technology, 61(2), 329–339.

    Google Scholar 

  • Costas, R., van Leeuwen, T. N., & van Raan, A. F. J. (2011). The “Mendel syndrome” in science: durability of scientific literature and its effects on bibliometric analysis of individual scientists. Scientometrics, 89(1), 177–205.

    Article  Google Scholar 

  • De Bellis, N. (2009). Bibliometrics and citation analysis: from the Science citation index to cybermetrics. Lanham, MD: Scarecrow Press.

    Google Scholar 

  • Garfield, E. (1980). Premature discovery or delayed recognition—why. Current Contents, 21, 5–10.

    Google Scholar 

  • Garfield, E. (1985a). The articles most cited in the SCI from 1961 to 1982. 7. Another 100 citation-classics—the Watson-Crick double helix has its turn. Current Contents, 20, 3–12.

    Google Scholar 

  • Garfield, E. (1985b). The articles most cited in the SCI from 1961 to 1982. 8. Ninety-eight more classic papers from unimolecular reaction velocities to natural opiates-the changing frontiers of science. Current Contents, 33, 3–11.

    Google Scholar 

  • Garfield, E. (1986). Letter to editor. Information Processing and Management, 22(5), 445.

    Article  Google Scholar 

  • Glänzel, W. (2008). Seven myths in bibliometrics. About facts and fiction in quantitative science studies. In H. Kretschmer & F. Havemann (Eds.), Proceedings of WIS 2008 (pp. 1–10). Germany: Berlin.

  • Glänzel, W., Debackere, K., Thijs, B., & Schubert, A. (2006). A concise review on the role of author self-citations in information science, bibliometrics and science policy. Scientometrics, 67(2), 263–277. doi:10.1007/s11192-006-0098-9.

    Article  Google Scholar 

  • Glänzel, W., Schlemmer, B., & Thijs, B. (2003). Better late than never? On the chance to become highly cited only beyond the standard bibliometric time horizon. Scientometrics, 58(3), 571–586.

    Article  Google Scholar 

  • Glänzel, W., & Schoepflin, U. (1995). A bibliometric study on ageing and reception processes of scientific literature. Journal of information Science, 21(1), 37–53.

    Article  Google Scholar 

  • King, D. A. (2004). The scientific impact of nations. Nature, 430(6997), 311–316. doi:10.1038/430311a.

    Article  Google Scholar 

  • Levitt, J. M., & Thelwall, M. (2008). Patterns of annual citation of highly cited articles and the prediction of their citation ranking: A comparison across subjects. Scientometrics, 77(1), 41–60.

    Article  Google Scholar 

  • Leydesdorff, L. (2008). Caveats for the use of citation indicators in research and journal evaluations. Journal of the American Society for Information Science and Technology, 59(2), 278–287. doi:10.1002/Asi.20743.

    Article  Google Scholar 

  • Leydesdorff, L. (2009). How are new citation-based journal indicators adding to the bibliometric toolbox? Journal of the American Society for Information Science and Technology, 60(7), 1327–1336. doi:10.1002/asi.21024.

    Article  Google Scholar 

  • Leydesdorff, L., & Opthof, T. (2010). Normalization at the field level: Fractional counting of citations. Journal of Informetrics, 4(4), 644–646. doi:10.1016/j.joi.2010.05.003.

    Article  Google Scholar 

  • Line, M. B. (1993). Changes in the use of literature with time: Obsolescence revisited. Library Trends, 41(4), 665–683.

    Google Scholar 

  • Moed, H. F., Burger, W., Frankfort, J., & van Raan, A. (1985). The application of bibliometric indicators: Important field- and time-dependent factors to be considered. Scientometrics, 8(3), 177–203. doi:10.1007/bf02016935.

    Article  Google Scholar 

  • Moed, H. F., van Leeuwen, T. N., & Reedijk, J. (1998). A new classification system to describe the ageing of scientific journals and their impact factors. Journal of Documentation, 54(4), 387–419.

    Article  Google Scholar 

  • Porter, A. L. (1977). Citation analysis: Queries and caveats. Social Studies of Science, 7(2), 257–267. doi:10.1177/030631277700700207.

    Article  Google Scholar 

  • Radicchi, F., & Castellano, C. (2011). Rescaling citations of publications in physics. Physical Review E, 83(4), 046116.

    Article  Google Scholar 

  • Radicchi, F., Fortunato, S., & Castellano, C. (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National academy of Sciences of the United States of America, 105(45), 17268–17272. doi:10.1073/pnas.0806977105.

    Article  Google Scholar 

  • Rogers, J. D. (2010). Citation analysis of nanotechnology at the field level: Implications of RD evaluation. Research Evaluation, 19(4), 281–290.

    Article  Google Scholar 

  • Schubert, A., & Braun, T. (1996). Cross-field normalization of scientometric indicators. Scientometrics, 36(3), 311–324. doi:10.1007/bf02129597.

    Article  Google Scholar 

  • Stent, G. (1972). Prematurity and uniqueness in scientific discovery. Scientific American, 227(6), 84–93.

    Article  Google Scholar 

  • Van Raan, A. F. J. (2004). Sleeping beauties in science. Scientometrics, 59(3), 467–472.

    Article  Google Scholar 

  • Walters, G. D. (2011). The citation life cycle of articles published in 13 American Psychological Association journals: A 25-year longitudinal analysis. Journal of the American Society for Information Science and Technology, 62(8), 1629–1636. doi:10.1002/asi.21560.

    Article  Google Scholar 

Download references

Acknowledgments

The author would like to thank Stefan Hornbostel, Sybille Hinze, and William Dinkel for their efforts in the early stage of project initiation and research design, Diana Hicks and Daniel Sirtes for their suggestions which were most helpful in improving the paper, Jasmin Schmitz, Haiko Lietz, Marion Schmidt, Pei-Shan Chi, and Jana Schütze for their many helpful ideas and collegial support, and two anonymous reviewers for their critical and constructive comments. The research underlying this paper was supported by the German Federal Ministry for Education and Research (BMBF, project number 01PQ08004A). The data used in this paper are from a bibliometrics database developed and maintained by the Competence Center for Bibliometrics for the German Science System (KB) and derived from the 1980 to 2011 Science Citation Index Expanded (SCIE), Social Sciences Citation Index (SSCI), and Arts and Humanities Citation Index (AHCI) prepared by Thomson Reuters (Scientific) Inc. (TR®), Philadelphia, Pennsylvania, USA: ©Copyright Thomson Reuters (Scientific) 2012. The author thanks the KB team for its collective effort in the development of the KB database.

Author information

Authors and Affiliations

  1. Institute for Research Information and Quality Assurance (iFQ), Schuetzenstrasse 6a, 10117, Berlin, Germany

    Jian Wang

Authors
  1. Jian Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jian Wang.

Appendix

Appendix

See Tables 2 and 3.

Table 2 Accuracy of using short citation time windows (based on dataset 1)
Full size table
Table 3 Spearman correlation with total citations by field (based on dataset 1)
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wang, J. Citation time window choice for research impact evaluation. Scientometrics 94, 851–872 (2013). https://doi.org/10.1007/s11192-012-0775-9

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-012-0775-9

Keywords

Search

Navigation