Skip to main content
SpringerLink
Log in

Sources of Google Scholar citations outside the Science Citation Index: A comparison between four science disciplines

  • Special-Session Papers Presented at the 9th International Conference on Science and Technology Indicators
  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

For practical reasons, bibliographic databases can only contain a subset of the scientific literature. The ISI citation databases are designed to cover the highest impact scientific research journals as well as a few other sources chosen by the Institute for Scientific Information (ISI). Google Scholar also contains citation information, but includes a less quality controlled collection of publications from different types of web documents. We define Google Scholar unique citations as those retrieved by Google Scholar which are not in the ISI database. We took a sample of 882 articles from 39 open access ISI-indexed journals in 2001 from biology, chemistry, physics and computing and classified the type, language, publication year and accessibility of the Google Scholar unique citing sources. The majority of Google Scholar unique citations (70%) were from full-text sources and there were large disciplinary differences between types of citing documents, suggesting that a wide range of non-ISI citing sources, especially from non-journal documents, are accessible by Google Scholar. This might be considered to be an advantage of Google Scholar, since it could be useful for citation tracking in a wider range of open access scholarly documents and to give a broader type of citation impact. An important corollary from our study is that Google Scholar’s wider coverage of Open Access (OA) web documents is likely to give a boost to the impact of OA research and the OA movement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Antelman, K. (2004), Do Open-Access articles have a greater research impact? College & Research Libraries, 65(5): 372–382. Retrieved May 4, 2006, from http://eprints.rclis.org/archive/00002309/01/do_open_access_CRL.pdf

    Google Scholar 

  • Bauer, K., Bakkalbasi, N. (2005), An examination of citation counts in a new scholarly communication environment. D-Lib Magazine, 11(9), Retrieved December 23, 2005, from http://www.dlib.org/dlib/september05/bauer/09bauer.html

  • Bawden, D., Holtham, C., Courtney, N. (1999), Perspectives on information overload. Aslib Proceedings, 51(8): 249–255.

    Article  Google Scholar 

  • Belew, R. (2005), Scientific impact quantity and quality: Analysis of two sources of bibliographic data. Retrieved May 3, 2006, from http://arxiv.org/abs/cs.IR/0504036

  • Bollacker, K. D., Lawrence, S., Lee, C. (1998), CiteSeer: An autonomous web agent for automatic retrieval and identification of interesting publications. In: Proceedings of 2nd International ACM Conference on Autonomous Agents, ACM Press, 1998: 116–123, Retrieved May 10, 2006, from http://maya.cs.depaul.edu/:_classes/ds575/papers/citeseer.pdf

  • Brody, T., Carr, L., Harnad, S. (2002), Evidence of hypertext in the scholarly archive. Proceedings of ACM Hypertext 2002: 74–75.

  • Brody, T., Stamerjohanns, H., Harnad, S. Gingras, Y., Vallieres, F., Oppenheim, C. (2004), The effect of open access on citation impact. Presented at: National Policies on Open Access (OA) Provision for University Research Output: An International Meeting. Southampton University, Southampton UK. 19 February 2004. Retrieved May 2, 2006, from http://opcit.eprints.org/feb19oa/brody-impact.pdf

  • Brown, C. (2003). The role of electronic preprints in chemical communication: analysis of citation, acceptance in the journal literature. Journal of the American Society for Information Science and Technology, 54(5): 362–371.

    Article  Google Scholar 

  • Cameron, R.D. (1997), A universal citation catalyst for reform in scholarly communication. First Monday, 2(4), Retrieved May 9, 2006, from http://www.firstmonday.dk/issues/issue2_4/cameron/index.html

  • Davis, P. M., Fromerth, M. J. (2006), Does the arXiv Lead to Higher Citations and Reduced Publisher Downloads for Mathematics Articles? Retrieved May 5, 2006, from http://arxiv.org/ftp/cs/papers/0603/0603056.pdf

  • DOAJ (Directory of Open Access Journals) (2006), Retrieved May 3, 2006, from http://www.doaj.org

  • Friend, F. (2006), Google Scholar: Potentially good for users of academic information. The Journal of Electronic Publishing, 9(1), Retrieved April 28, 2006, from http://hdl.handle.net/2027/spo.3336451.0009.105

  • Fry, J., Talja, S. (2004), The cultural shaping of scholarly communication: Explaining e-journal use within and across academic fields. In: ASIST 2004: Proceedings of the 67th ASIST Annual Meeting (Vol. 41, pp. 20–30): Medford, NJ.: Information Today.

    Google Scholar 

  • Garfield, E. (1955), Citation indexes for science: A new dimension in documentation through association of ideas. Science, 3159(122): 108–111. Retrieved May 3, 2006, from http://www.garfield.library.upenn.edu/essays/v6p468y1983.pdf

    Article  Google Scholar 

  • Garfield, E. (1965), Can citation indexing be automated? National Bureau of Standards, Miscellaneous Publication, 269(114): 189–192, Retrieved May 8, from http://www.garfield.library.upenn.edu/essays/V1p084y1962-73.pdf

    Google Scholar 

  • Glänzel, W., Schoepflin, U. (1994), Little Scientometrics — Big Scientometrics ... and Beyond. Scientometrics, 30(2–3): 375–384.

    Article  Google Scholar 

  • Goodrum, A.A., McCain, K.W., Lawrence, S., Giles, C.L. (2001), Scholarly publishing in the Internet age: a citation analysis of computer science literature. Information Processing & Management, 37(5): 661–676.

    Article  MATH  Google Scholar 

  • Google Scholar (2006), Retrieved Feb 12, 2006, from http://scholar.google.com/scholar/about.html

  • Hajjem, C., Harnad, S., Gingras, Y. (2005), Ten-year cross-disciplinary comparison of the growth of open access and how it increases research citation impact. IEEE Data Engineering Bulletin, 28(4): 39–47. Retrieved May 5, 2006, from http://sites.computer.org/debull/A05dec/hajjem.pdf

    Google Scholar 

  • Harnad, S. (1991), Post-Gutenberg Galaxy: The fourth revolution in the means of production of knowledge. Public-Access Computer Systems Review, 2(1): 39–53. Retrieved November 12, 2004 from http://www.cogsci.soton.ac.uk/:_harnad/Papers/Harnad/harnad91.postgutenberg.html

    MathSciNet  Google Scholar 

  • Harnad, S., Carr, L. (2000), Integrating, navigating, and analysing open eprint archives through open citation linking (the OpCit project). Current Science, 79(5): 629–638.

    Google Scholar 

  • Harnad, S., Brody, T. (2004), Comparing the Impact of Open Access (OA) vs. non-OA articles in the same journals. D-Lib Magazine, 10(6). Retrieved May 2, 2006, from http://www.dlib.org/dlib/june04/harnad/06harnad.html

  • Harnad, S., Brody, T., Vallieres, F., Carr, L., Hitchcock, S., Gingras, Y., Oppenheim, C., Stamerjohanns, H., Hilf, E. (2004), The access/impact problem and the green and gold roads to open access. Serials Review, 30(4). Retrieved May, 5, 2006, from http://eprints.ecs.soton.ac.uk/10209/01/impact.html

  • Harter, S., Ford, C. (2000), Web-based analysis of E-journal impact: Approaches, problems, and issues, Journal of the American Society for Information Science, 51(13): 1159–1176.

    Article  Google Scholar 

  • Herring, S.D. (2002), Use of electronic resources in scholarly electronic journals: A citation analysis. College and Research Libraries, 63(4): 334–340.

    Google Scholar 

  • Hitchcock, S., Bergmark, D., Brody, T., Gutteridge, C., Carr, L., Hall, W., Lagoze, C., Harnad, S. (2002), Open Citation Linking: The way forward. D Lib Magazine, 8(10), Retrieved Jan 10, 2006, from http://www.dlib.org/dlib/october02/hitchcock/10hitchcock.html

  • ISI Press Release Essay on the Impact of Open Access Journals: A Citation Study from Thomson ISI. Retrieved November 13, 2004, from http://www.isinet.com/oaj

  • ISI journal selection process (2004), Retrieved May 10, 2006, from http://scientific.thomson.com/free/essays/selectionofmaterial/journalselection

  • Jacso, P. (2004), Google Scholar Beta. Péter’s Digital Reference Shelf, Retrieved Jan 10, 2006, from http://snipurl.com/dwco

  • Jacso, P. (2005a), Google Scholar: the pros and the cons. Online Information Review, 29(2): 208–214.

    Article  Google Scholar 

  • Jacso, P. (2005b), As we may search: Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Current Science, 89(9): 1537–1547. Retrieved April 28, 2006, from http://www.ias.ac.in/currsci/nov102005/1537.pdf

    Google Scholar 

  • Kim, H.J. (2000), Motivations for hyperlinking in scholarly electronic articles: A qualitative study. Journal of the American Society for Information Science, 51(10): 887–899.

    Article  Google Scholar 

  • Kling, R., McKim, G. (1999), Scholarly communication and the continuum of electronic publishing. Journal of American Society for Information Science, 50(10): 890–906.

    Article  Google Scholar 

  • Kousha, K., Thelwall, M. (2006). Motivations for URL citations to open access library and information science articles. Scientometrics, 68(3): 501–517.

    Article  Google Scholar 

  • Kousha, K., Thelwall, M. (to appear, 2007). Google Scholar citations and Google Web/URL citations: A multi-discipline exploratory analysis, Journal of the American Society for Information Science and Technology, 57(6): 1055–1065.

    Article  Google Scholar 

  • Kurtz, M.J. (2004), Restrictive access policies cut readership of electronic research journal articles by a factor of two, Harvard-Smithsonian Centre for Astrophysics, Cambridge, MA. Retrieved November 13, 2006, from http://opcit.eprints.org/feb19oa/kurtz.pdf

  • Kurtz, M. J., Eichhorn, G., Accomazzi, A., Grant, C., Demleitner, M., Murray, S. S. (2005), Worldwide use and impact of the NASA Astrophysics Data System digital library. Journal of the American Society for Information Science & Technology, 56(1): 36–45.

    Article  Google Scholar 

  • Lancaster, F.W., Warner, A. (2001), Intelligent Technologies in Library and Information Service Applications, Information Today, Medford, NJ.

  • Lancaster, F. W. (2003), Indexing and Abstracting in Theory and Practice. 3rd ed. University of Illinois, Graduate School of Library and Information Science, Champaign, IL.

    Google Scholar 

  • Lawrence, S. (2001), Free online availability substantially increases a paper’s impact. Nature, 411, 521. Retrieved November 13, 2001, from http://www.nature.com/nature/debates/e-access/Articles/lawrence.html

    Article  Google Scholar 

  • Lawrence, S., Giles, C. L., Bollacker, K. (1999), Digital libraries and autonomous citation indexing. IEEE Computer, 32(6): 67–71, Retrieved April 1, 2006, from http://csdl.computer.org/dl/mags/co/1999/06/r6067.pdf

    Google Scholar 

  • Lawson, M., Kemp, N., Lynch, M., Chowdhury, G. (1996), Automatic extraction of citations from the text of English-language patents: an example of template mining. Journal of Information Science, 22(6): 423–436.

    Article  Google Scholar 

  • MacRoberts, M. H., MacRoberts, B. R. (1989), Problems of citation analysis: A critical review. Journal of the American Society for Information Science, 40(5): 342–349.

    Article  Google Scholar 

  • MacRoberts, M. H., MacRoberts, B. R. (1996), Problems of citation analysis. Scientometrics, 36(3): 435–444.

    Article  Google Scholar 

  • Martello, A. (2006), Selection of Content for the Web Citation Index: Institutional Repositories and Subject Specific Archives, Thomson Scientific essay. Retrieved May 11, 2006, from http://scientific.thomson.com/free/essays/selectionofmaterial/wci-selection

  • Moed, H., F. (2005), Citation Analysis in Research Evaluation. Springer, New York.

    Google Scholar 

  • Mowshowitz, A., Kawaguchi, A. (2005), Measuring search engine bias. Information Processing and Management. 41(5):1193–1205.

    Article  Google Scholar 

  • Notess, G. R. (2005), Scholarly Web searching: Google Scholar and Scirus. Online, 29(4). Retrieved May 6, 2006, from http://www.infotoday.com/Online/jul05/OnTheNet.shtml

  • Oppenheim, C. (2000), Do patent citations count? In: B. Cronin, H. B. Atkins (Eds), The Web of Knowledge: A Festscrift in Honor of Eugene Garfield. Information Today Inc ASS Monograph Series, Metford, NJ, pp. 405–432.

  • Pauly, D., Stergiou, K. (2005), Equivalence of results from two citation Thomson ISI’s Citation Index and Google’s Scholar service. Ethics in Science and Environmental Politics, December: 33–35. Retrieved April 25, 2006, from http://www.int-res.com/articles/esep/2005/E65.pdf

  • Salton, G. (1963), Associative document retrieval techniques using bibliographic information. Journal of the ACM, 10(4):440–457, Retrieved May 9, 2006, from http://portal.acm.org/citation.cfm?id=321186.321188

    Article  MATH  Google Scholar 

  • Salton, G. (1971), Automatic indexing using bibliographic citations. Journal of Documentation, 27(2):98–110.

    Article  Google Scholar 

  • Schwarz, G., Kennicutt, R. (2004), Demographic and citation trends in astrophysical journal papers and preprints. Bulletin of the American Astronomical Society, 36:1654–1663. Retrieved May 6, 2006, from http://arxiv.org/PS_cache/astro-ph/pdf/0411/0411275.pdf

    Google Scholar 

  • Shin, E.-J. (2003), Do Impact Factors change with a change of medium? A comparison of Impact Factors when publication is by paper and through parallel publishing. Journal of Information Science, 29(6), 527–533.

    Article  Google Scholar 

  • Smith, A. G. (1999), A tale of two Web spaces: Comparing sites using Web impact factors. Journal of Documentation, 55(5):577–592.

    Google Scholar 

  • Swan, A., Brown, S. (2004), Report of the JISC/OSI Open Access Journal Authors Survey, 1–76. Retrieved April 20, 2006, from http://www.jisc.ac.uk/uploaded_documents/JISCOAreport1.pdf

  • Swan, A., Brown, S. (2005), Open Access Self-archiving: An Author Study, 1–97. Retrieved April 20, 2006, from http://eprints.ecs.soton.ac.uk/10999/01/jisc2.pdf

  • Thelwall, M., Vaughan, L., Björneborn, L. (2005), Webometrics. Annual Review of Information Science and Technology, 39, Information Today Inc., Medford, NJ. 81–135.

    Google Scholar 

  • Thelwall, M. (2006), Interpreting social science link analysis research: A theoretical framework. Journal of the American Society for Information Science and Technology. 57(1): 57 (1):60–68.

    Article  Google Scholar 

  • Vaughan, L., Hysen, K. (2002), Relationship between links to journal Web sites and Impact Factors. Aslib Proceedings: New Information Perspectives, 54(6):356–361.

    Google Scholar 

  • Vaughan, L., Shaw, D. (2003), Bibliographic and Web citations: What is the difference? Journal of the American Society for Information Science and Technology, 54(4): 54 (4):1313–1324.

    Article  Google Scholar 

  • Vaughan, L., Thelwall, M. (2003), Scholarly use of the Web: What are the key inducers of links to journal Web sites? Journal of the American Society for Information Science and Technology, 54(1):29–38.

    Article  Google Scholar 

  • Vaughan, L., Shaw, D. (2005), Web citation data for impact assessment: A comparison of four science disciplines. Journal of the American Society for Information Science and Technology, 56(10):1075–1087.

    Article  Google Scholar 

  • Wouters, P., Vries, R. (2004), Formally citing the Web. Journal of the American Society for Information Science and Technology, 55(14):1250–1260.

    Article  Google Scholar 

  • Zhao, D., Logan, E. (2002), Citation analysis using scientific publications on the Web as data source: A case study in the XML research area. Scientometrics, 54(3):449–472.

    Article  Google Scholar 

  • Zhao, D. (2005), Challenges of scholarly publications on the Web to the evaluation of science — A comparison of author visibility on the Web and in print journals. Information Processing and Management, 41(6):1403–1418

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing and Information Technology, University of Wolverhampton, Wolverhampton, UK

    Kayvan Kousha & Mike Thelwall

  2. Department of Library and Information Science, University of Tehran, Jalal-Al-e-Ahmed Ave., P.O. Box 11455/6456, Tehran, Iran

    Kayvan Kousha

Authors
  1. Kayvan Kousha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mike Thelwall
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kayvan Kousha.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kousha, K., Thelwall, M. Sources of Google Scholar citations outside the Science Citation Index: A comparison between four science disciplines. Scientometrics 74, 273–294 (2008). https://doi.org/10.1007/s11192-008-0217-x

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-008-0217-x

Keywords

Search

Navigation