Preserving the Integrity of Citations and References by All Stakeholders of Science Communication

Scientific citations are often viewed as tools for guiding readers across myriads of published sources, distinguishing innovations and preserving the integrity of bibliographic records (1, 2). All contributors of scholarly articles are currently encouraged to upgrade their skills in citing, analyzing relevance, and managing references to ensure the accuracy of citations and completeness of references lists (3).

Technically correct and thoroughly validated citations add to the quality of reference lists and allow stakeholders of science communication to judge manuscripts and published articles fairly. As a good example, journal editors often pick potential reviewers from the reference lists, while reviewers scan the timelines of the references to judge the novelty and scope of the manuscripts(4, 5).

In the era of 'big science' and expansion of online bibliographic databases, the implications of retrieving relevant sources and mapping citations are critical for interlinking large amounts of scholarly items, finding influential (highly cited) articles, and promoting periodicals, individual authors and academic institutions (Fig. 1). Tracking related sources and citations helps information facilitators build up networks of likeminded scholars and arrange professional interactions on online platforms such as ResearchGate® (6).

Fig. 1
The expanding role of citations and references of scholarly articles.

• Click for larger image
• Download as PowerPoint slide

One of the basic principles of scholarly writing stands on distinguishing one's own ideas, words, and graphics from those adopted from published sources. Skilled authors always credit each adopted statement, known scientific fact and methods by citing related publications, giving preference to sources visible in reputed databases, libraries, or archiving platforms. Providing access to primary sources for reading, comprehending main points, and citing them in the proper context is the accepted norm for building up scholarly information (7). The availability of subscription or open-access channels ensures that authors are offered the basic opportunity for comprehending the context of cited sources and correctly organizing their references (3). Reviewers and editors with access to the currently available hubs of scholarly information are capable of suggesting pertinent changes and improving the quality and readability of the manuscripts by analyzing the relevance, ethical and technical correctness of bibliographies (8, 9).

Prestigious citation-tracking databases, such as Scopus and Web of Science, rely heavily on the correctness of reference lists in the indexed items, which, in turn, influence the functionality of the bibliographic records and links between peer-reviewed sources (10). Indexers of these databases bear their share of responsibility, which may encompass regularly analyzing the correctness of references and delisting periodicals with massive technical mistakes and unethical referencing. By accepting for or continuing coverage of periodicals with poorly edited reference lists, indexers skew the citation-based impact indicators and distort researchers' and their institutions' profiles. The implications of such mistakes are far-reaching since current global ranking systems such as the QS World University Rankings, the Academic Ranking of World University and the Times Higher Education World University Rankings are all anchored on Scopus, the largest database of peer-reviewed journals, and powered by related reference and citation tracking (11).

Moreover, bibliographies have become a subject of semantic analyses over the past few decades, helping readers, researchers, and information scientists systematize the relatedness of scientific facts and explore new directions for research (12, 13, 14).

The US National Library of Medicine recognized the growing importance of research in the field by introducing "bibliography as topic" to the Medical Subject Headings (MeSH) thesaurus in 2008. As a major term it has extensive links to other terms in the MeSH hierarchy, such as "documentation", "information science", "publications", and "bibliometrics". The term has been tagged on 15,620 items indexed by PubMed, with the highest number of 823 items published in 2013 (as of August 18, 2015).

Considering the growing importance of relevant and correct reference lists of scholarly items, we aimed to analyze the role of stakeholders of science communication in preserving the integrity of citations and references.

Scientific authors are primary users of published items and ultimately responsible for selecting relevant and ethical citations (Table 1). They are engaged in science communication from the stage of retrieving scholarly items, publishing their own research, and adding new records to bibliographic databases. Although ideal citation practices are not defined, authors are advised to avoid excessive citations to their own works or other forms of manipulation, read full-texts of papers and cite them in the proper context, and give credit when credit is due (15). The authors, who strictly follow the instructions and adhere to the referencing style of the target journals, increase their chances of getting published (16).

Table 1
Examples of recommended and unacceptable references for scholarly articles

• Click for larger image
• Click for full table
• Download as Excel file

Authors should realize that bibliographic databases rely heavily on the quantity and quality of the reference lists. The sustainability of citation-based databases depends on technical correctness and relevance of these lists. Despite the growth of databases and the availability of advanced search engines, several analyses across scientific disciplines have identified that 20%-25% of references cited by authors are erroneous and do not support their quotes or other statements (17, 18, 19, 20, 21).

The implications of online bibliographic searches on the relevance and technical correctness of references have not been fully explored. However, it is likely that improved access to quality sources and familiarity with abstracts, keywords, and main points in the primary literature may improve the referencing and shape the whole publishing landscape (7, 21, 22). There is evidence suggesting that training on bibliographic searches, librarian-guided workshops, and peer collaboration at the stage of undergraduate medical education improve the ongoing authors' referencing and essay writing skills in an ongoing fashion (23). Although the role of referencing courses in many other fields is unclear, it is likely that advanced searches through multidisciplinary citation databases, such as Scopus and Web of Science, will increase chances of retrieving highly relevant primary sources, which, in turn, will add to the overall quality of multidisciplinary journals.

The analysis of reference lists and records of citations on the databases may help authors deepen their searches for evidence-based and highly-cited items, which are usually endorsed by the wider scientific community. Highly-cited items (>100 times) are often methodological articles or those representing 'hot' topics and high level of evidence (e.g., large studies, trials, systematic reviews) (24, 25, 26, 27).

Some, even highly-cited sources may contain erroneous, flawed, or otherwise unethical information, which is linked to the correction or retraction notices. The number of retracted publications is constantly growing while citations validating these items are still accumulating and influencing the scientific discourse (28, 29). In a landmark analysis of 5,000 citations to retracted biomedical articles, 93% of these citations were research-related, supporting the validity of predominantly fraudulent publications (30). Unfortunately, the majority of authors, who cite primary sources, overlook the related retraction notices and continue citing flawed and unethical items, thus adding inconsistency to the whole system of citation analyses (31).

The information explosion and expansion of interdisciplinary and cross-country research set common standards for writing, referencing, and indexing. Such standards, however, overlook the differences in the authors' citation behavior, which is compounded by the access to relevant bibliographic databases, professional, geographic, and language backgrounds (32, 33, 34, 35). While citations have traditionally been viewed as credits to generators of new knowledge, related publications, or background reading, citing motives have diversified enormously in the past decades, and partly because of nonscientific reasons (36). For example, an analysis of 3,813 references from articles published in 2011 by three Brazilian orthopedics journals found that roughly 8% of them were to local sources and 41% of the analyzed articles did not contain any Brazilian reference (37). On the other extreme, preferential citations to local/national sources are commonplace in other countries, which skew quantitative and qualitative citation analyses further (38). Paradoxical citation patterns were documented for Korean physicists and mechanical engineers, who preferentially cited domestic sources in the articles published in Korean journals and mainstream science sources - in the articles published in the U.S., U.K., Netherlands, and Germany (39).

We analyzed the quantitative growth of references over the past decade across several countries (Table 2) and scientific disciplines (Table 3) based on the SCImago Journal & Country Rank data. The results indicate the exponential increase of indexed items and related references in the era of open access (from 2002 onward). Interestingly, the average number of references per document (Refs/Doc) has increased in most countries, but it is still the lowest in some non-Anglophone countries (e.g., Russia). Among biomedical disciplines, immunology has one of the highest value of Refs/Doc for 2013 (40.3) while nursing - the lowest (25.1).

Table 2
Quantitative analysis of the growth of indexed items and related references across several countries over the period of 2002-2013

• Click for larger image
• Click for full table
• Download as Excel file

Table 3
Quantitative analysis of the growth of indexed items and related references across several scientific disciplines over the period of 2002-2013

• Click for larger image
• Click for full table
• Download as Excel file

Reviewers are expected to assess the manuscripts' validity, adherence to research reporting guidelines, clarity and consistency of writing, and correctness of references. The latter includes the adherence to the journal's style and limits of referencing, as well as the novelty and relevance of the listed references (40).

Currently used editorial management systems such as Aries systems® and ScholarOne® are equipped with tools for online bibliographic searches, which help reviewers analyze novelty and completeness of reference lists, overlaps with published bibliographies, and pick additional references to support their own statements. The editorial management systems refer the reviewers to the reference validation tools linked to CrossRef and PubMed.

Recommending additional references to support new ideas or debatable statements are helpful for substantiating the writing and avoiding plagiarism (41). Reviewers' analyses of the sequence and semantic links between in-text citations may reveal forms of plagiarisms, which are otherwise undetectable (42).

Reviewers may track statements and general knowledge, linked to irrational and multiple references, which artificially inflate the journals' and individuals' impact profiles. Suggesting replacements of irrelevant, old, nonpeer-reviewed, and secondary sources with more appropriate, evidence-based, and widely-visible ones are generally accepted and encouraged. Furthermore, reviewers may also perform searches through online databases and recommend relevant replacements for retracted, illegitimate (predatory), hardly accessible, secondary, or tertiary sources.

Attention should be paid to the verification of Web sites as references, which may change their contents and become inaccessible over the time. An analysis of reference lists of 2,822 articles, published in 2006-2013, revealed that mainstream general medical journals such as The Lancet and BMJ contain significantly less inaccessible Web references than their peripheral counterparts (43), pointing to more stringent verification practices in the formers.

Online items with permanent presence on the Internet, and particularly those with Digital Object Identifiers (DOIs), can be preferentially recommended for citation. Print and online items, including audio and video presentations, which do not pass through the traditional peer review, should be processed with caution since these may contain promotional, inappropriate, and potentially harmful information (44, 45).

Examples of tertiary sources, which are increasingly, but not always justifiably, cited in the indexed literature, are easily accessible encyclopedic articles from Wikipedia, and particularly those containing definitions and descriptions (46). Although Wikipedia pages are regularly edited, linked to a large number of primary references, and recommended as didactic resources for some disciplines (47), they may contain biased and inappropriately edited information (48). Other examples of tertiary and secondary references, which can be replaced by primary evidence-based sources, include monographs, textbooks, and dissertations (Table 1). These are abundantly cited by authors from nonmainstream science countries, encountering difficulties with accessing high-quality periodicals, which are accessible through the subscription paywalls.

An opportunity to recommend replacements and additions brought about an unethical practice of coercive citations. The practice involves both reviewers and editors, who coerce their authors to add citations to their articles and journals and make their decisions based on the willingness of the authors to cite additional references. An analysis of 6,672 responses in a multidisciplinary survey of authors demonstrated that 20% of the surveyees were coerced to cite and more than 40% were aware of that practice (49). The respondents reported 175 coercer journals in economics, sociology, psychology, and business.

Another recent analysis of 616 reviewer comments for the Journal of Psychosomatic Research found that one-third of recommended additional citations were to sources authored by the reviewers with 21% of these references turned irrelevant. Self-cites were twice more often in comments suggesting revision or acceptance than in those suggesting rejection (33% vs. 15%, P<0.001) (50). On the other hand, a report on 927 referee comments submitted to the Journal of the American Society of Information Science and Technology found little evidence that reviewers abuse the peer review system to play the 'citation games' (51), suggesting that there are differences in the journals' strategies for managing citations and references.

For decades, scholars educated at and affiliated to academic institutions of mainstream science countries have been considered as the best reviewers and editors with advanced skills of literature searches and referencing (52). Experts even advise to hire seasoned editors of reputable journals to curb the problem of inaccurate and unethical citations in nonmainstream science journals (53).

Some journal editors have raised concerns over the editorial biases, which are increasingly affecting the integrity of peer review, author-editor relationships, and authors' citation behavior (54, 55, 56). Such biases have affected numerous disciplines and turned out to be related not to the professional credentials of the editors, but to their desire to attract 'easy' citations (57). The 'citation games' have damaged reputation of some low-impact journals and their desperate editors while the most reputable international journals and their editors distanced themselves from such unethical practices (53, 58).

Journal editors are in a position to detect and avoid irrelevant or coercive citations, and particularly those added during the peer review. By ensuring relevance of citations and proper credits to publicized facts and ideas, editors ensure the quality of their authors' writings (59). As a final resort, editors may monitor cited and citing sources post-publication to trace coercion or other manipulations in their journals and raise the issue of retracting items, which undermine the trustworthiness of citation networks (60, 61, 62). Irrelevant or coercive citations are relatively new causes of retractions, which are often linked to more conspicuous causes such as fraud, plagiarism, authorship disputes, or honest errors (63).

Publishers are capable of implementing strategies of proper citing and referencing by providing modes of citations to their own journal articles and upgrading their instructions for authors (64). Most journal instructions contain sections on technical accuracy, style (Vancouver, Harvard, or mixed systems), and limits of referencing. But it is increasingly important to highlight ethical principles of selecting and citing primary sources, which are missing in the majority of the instructions. As a precedent, an exemplary list of pointers to ethical citations is now available as a section of the editorial policies of BioMedCentral publisher (65).

An excellent initiative, which is aimed at improving the quality and reuse of reference lists, is set by Informa publishing company. The expert opinion and review article series of the publisher contain bibliographies where sources of considerable interest to readers are specifically marked. Elsevier and Springer went further and launched automatic alert services, informing their authors about citations to their articles appearing in other journals and tracked by CrossRef. Such an alert delves into the relatedness of cited and citing sources and facilitates networking of authors in the same and allied disciplines.

Enhancing visibility of bibliographies is yet another tool offered by publishers that may increase authors' responsibility over the reference lists and draw readers' attention to this section of scholarly articles. By appreciating the autonomous role of bibliographies in the distribution of scholarly information, large subscription publishers such as Springer, Elsevier and Emerald have opened access to their reference lists, leaving the rest of the subscription articles behind the paywalls. Such an initiative is supported further by the Open Citation Corpus, a repository of freely available citation data, which was recently launched to enhance the reuse of correctly recorded reference lists without copyright restrictions (66). The project started providing access to biomedical citations from PubMed Central, but its ultimate goal is to integrate with CrossRef and aggregators of reference lists such as CiteSeerX (citations in computer science; citeseerx.ist.psu.edu) and CitEc (Citations in Economics; citec.repec.org), cover most fields of science, and overcome limitations of databases and aggregators processing citation data (67). The Open Citation Corpus joined the Directory of Open Access Journals under the umbrella of the Infrastructure Services for Open Access in 2014 (68).

Only few learned associations have raised concerns about irrelevant or unethical citations, and formulated primary guidance in their recommendations and position papers. The most updated document containing points on the integrity of references is the "Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals" issued by the International Committee of Medical Journal Editors (ICMJE) in December, 2014 (69). The recommendations are of considerable interest to the multidisciplinary scientific community. They bring a balanced view on preferentially citing primary sources, keeping self-cites and bibliographies within justifiable limits, avoiding conference abstracts, "personal communications", and retracted items as references. Verifying technical correctness of each reference and adhering to the Vancouver style were also emphasized. The Council of Science Editors (CSE) included a section on citation manipulation, coercion and unethical boosting of the impact factors in the latest version of the "White Paper on Promoting Integrity in Scientific Journal Publications" (2012) (70). Finally, the American Society of Cells Biology released the widely endorsed San Francisco Declaration on Research Integrity (DORA, December 2012) that strongly encouraged citing primary scientific literature (71).

Bibliographies of scholarly articles reflect the overall quality and integrity of writing, editing, and publishing. With the exponential growth of scholarly articles and references across numerous disciplines, it is essential to adopt and enforce comprehensive strategies aimed at retrieving relevant references while writing, validating citations at the peer review, and checking their technical correctness at the copyediting and proofreading. Currently available digital tools for interlinking millions of scholarly items, particularly through the platform of CrossRef, help publishers manage the reference lists and avoid technical mistakes, which were common in the pre-digital era (72, 73).

One of the latest accounts of the inaccuracy rates (7.6% inaccurate quotations out of 3,840 scanned ones in ten orthopedic journals) suggest that the irrelevance of citations is still unacceptably high and is a much bigger issue than technical mistakes (74). The open access movement that facilitates easy access to both full-texts and bibliographies may improve the relevance of citing and draw attention of readers to the quality of reference lists. Selectively opening access to bibliographies by some subscription publishers is an interim measure that may add to the quality of bibliographies and form the basis for reference aggregation by specifically designed platforms.

Global and regional editorial associations fill the gaps in the quality assurance of citations and references by (re)drafting their recommendations and paying attention to the emerging issues of access, relevance, and ethics rather than technical mistakes. Defining responsibilities of all stakeholders of science communication in curbing these issues is an emerging task for editorial associations. Their recommendations and position statements are warranted for all scholarly journals, but it seems that small and nonmainstream science journals, struggling to get indexed, may benefit from the instructive recommendations more.

Finally, indexing services may play a decisive role in improving the quality of citations and references by tightening indexing criteria and discontinuing coverage of periodicals, which devalue the citation analyses. Global indexing services such as Scopus and Web of Science have traditionally prioritized the quantity, language, technical correctness, and visibility of references. The time has come to incorporate ethics and relevance of references in the expanding list of the indexing criteria.