Rewarding Peer Reviewers: Maintaining the Integrity of Science Communication

Comprehensive evaluation of journal submissions and prioritization of selected manuscripts for publishing are pillars of science communication. Although pre-publication peer review has been criticized for its slowness, expense, bias, proneness to abuse (1) and lack of strong evidence of its effectiveness (2), the absolute majority of scholarly journals still employ various models of peer review to choose innovative and citable items, reveal shortcomings, and recommend amendments that increase the scientific merit, graphical representation and readability of the published articles. Reliance on expedient peer review is essential for journal indexing in most prestigious bibliographic databases; selected journals that manage to establish and maintain their banks of reviewers expand their indexing chances and global visibility (3).

Over the past few decades, the concept of 'best reviewers' has emerged to signify the importance of professional contributions of the reviewers, who should be skilled in statistical analyses, research reporting, and publishing (4). The involvement of the best reviewers in the review facilitates publication of trustworthy and influential articles that increase the journal's scientific impact. While most high-ranking journals published by large publishers and professional societies have already established their pool of devoted reviewers, many others struggle to find cooperating experts. As a result, the global question arises as to how attract, credit and incentivize best reviewers to facilitate good publishing practice.

In our times, when new journals are being launched on an unprecedented scale and established journals are competing for citations and higher ranks, the problem of recruiting able reviewers and incentivizing them for quality comments is becoming perplexing. The reviewers should be active researchers and good authors, who passed trainings in publication ethics (yet another criterion of the best reviewers). They should be able to distinguish ethically sound practices from those jeopardizing the integrity of science communication.

The need for comprehensively disclosing financial and personal competing interests of all responsible parties (authors, reviewers, editors) has come to the fore in recent decades (5). Editorial policies overlooking the importance of disclosing the reviewers' financial and personal (academic) competing interests put the journals' staff at risk of being cheated and manipulated (6, 7). Editors, who fail to evaluate the referee profiles in the context of author-reviewer relationships may open the floodgates for unethical publications authored and reviewed by the same or closely related individuals (8). Reviewer profiles in various databases should always be screened to choose the most relevant referees. Reviewers, in turn, primarily respond to the assignments that fall into their narrow field of research interests, and may be tempted to push their own publications as additional references or recommend citations of sources from 'friendly' journals. The editors' duty is to detect any violation of peer review ethics, guide the reviewers, and advise them against coercive self-citations (9).

Over the past decades, if not centuries, peer review has been cultivated as "a culture of service to the profession" (10). As such, repetitive reviewer invitations should be the main incentive for the best reviewers (4) that leave no room for instances of unethical review in an ideal world. Nonmainstream science countries, however, still confront the omnipresent substandard review, which is rooted in a poor research environment and lack of appreciation of the reviewers' efforts. The reviewer contributions are largely disregarded as an integral part of academic activities, and more often so in the developing world. The Open Researcher and Contributor ID (ORCID) initiative, which was launched in 2012 and integrated in the editorial management systems of large and small publishers, may provide a solution to this problem by tracking all academic contributions and archiving records of potential reviewers (11).

Despite some limitations of the current peer review system and concerns over the unaddressed and lost communication at pre-publication stage (12), available positive data suggest that most authors of indexed journals (95%) accept and highly value reviewer critiques, and the majority of the reviewers (70%) consider their contributions as responsible academic work that improves the quality of research output in the field (13). The workload depends on the subject of the journal submissions and varies in rigor and details, often calling for researcher skills in professional and ethical research reporting (14). This is why qualified authors of top scholarly journals are expected to more frequently take part in the peer review and receive favorable feedback for their voluntary contribution (i.e., privileged track of their own submissions) (14).

Individual reviewers can be offered privileged access to subscription databases and research platforms to help them judge the novelty and originality of the processed manuscripts. Searches through the databases may also yield in arguments for more justified and evidence-based reviewer comments. Although university subscriptions to bibliographic databases and digital libraries are currently widely available, researchers and reviewers from nonacademic backgrounds may find the privileged individual access throughout the peer review facilitative. Elsevier, for example, grants its reviewers free one-month access to Scopus and ScienceDirect. SAGE opens 60-day complimentary access to all its journals along with a 25% discount on any SAGE book for those who successfully accomplish a reviewer assignment. Publishers operating with the ScholarOne editorial management system direct their evaluators to external searches through Thomson Reuters' Web of Science platform. Such a reviewing scheme, backed by systematic searches through highly informative databases, keeps the referees abreast of the developments in their field of research and adds to their academic knowledge. This is perhaps the most important stimulus for the arduous and time-consuming activity (15).

Interestingly, an "Elsevier Reviewer Badges and Rewards" scheme was recently proposed to incentivize frequent contributors further (16). The differential scheme of "Badges" implies a range of discounts on Elsevier products. For example, those with more than 20 reviews, or holders of "Senior Reviewer Badge", can be eligible for 25% discount on any Elsevier book.

Optimal time spent on reading, verifying contents and commenting on an average journal submission is about 3 hr, or about 15 hr of weekly work, given the willing expert accepts and fulfills at least five reviewer assignments weekly (17). An empirically estimated financial remuneration for prospective reviewers would be US $50 per hour of their work or $200 per reviewed item (18). Globally, there are some high-profit publishers that incentivize their ad hoc reviewers by paying differentially (more or less of that sum) for professional and statistical evaluations, though such a rewarding scheme is an exception rather than a norm.

A unique financial scheme was proposed by Rubriq, a US-based company that started experiencing its 'portable peer review' with some large publishers (e.g., PLOS, Wiley, Karger) and around 500 paid reviewers (19). The model implies rewarding its anonymous reviewers for their quality and timely evaluations that follow the manuscripts across multiple journals from their initial submission and revisions until the final publication. By charging its clients (authors) $500-700 and paying $100 from that sum for each reviewer contribution, the company aims to save the qualified reviewers' time and efforts and improve the efficiency of the whole system of peer review.

A landmark survey of 551 reviewers, arranged by the BMJ Publishing Group, revealed that financial rewards are not effective incentives for busy experts, who appreciate more offers of free access to the journal, updates on the process and results of the review, annual official acknowledgments on the journal website, and appointment to the journal's editorial board (20). Notably, 194 (35%) surveyees believed that financial incentives could even compromise the quality of reviews (21).

Additionally, many seasoned editors have voiced their concerns, warning that financial incentives alone would not work for the global community of reviewers (22, 23, 24, 25). Best reviewers are often affiliated to reputable universities, where credits for noncurricular academic activities count more than limited financial rewards. Top medical journals, such as the New England Journal of Medicine, the Journal of the American Medical Association and Annals of Internal Medicine, set a good example by granting Continuing Medical Education (CME) credits, which can be claimed by US-based reviewers for certain hours of contribution or a number of quality and timely reviews (26). Obviously, peer review is an effective learning process that updates knowledge of busy physicians and medical educators without incurring financial expenses, often abundantly spent on other forms of CME (e.g., conferences, seminars, trainings). Journals in other fields of science can implement similar models of Continuing Professional Development (CPD) credits, considering the discipline specifics and rating the reviewers' quality of work.

In 2013, in recognition of an 'outstanding contribution', Elsevier piloted awarding certificates of excellence to approximately 25 best reviewers per each title of 35 journals. Timeliness, quality and frequency of contributions were rated for issuing these certificates (27). The publisher's formal appreciation signified the enduring value of the reviewer contributions. Obviously, listing such awards in one's CV or academic ID (ORCID) would add weight to the profile and allow successfully competing with counterparts for editorial posts.

A bulk of pre-publication reviews remains behind journals' in-house bars and is discarded at some point (12). Consequently, innovative ideas, proposals, thought-provoking discussions on the one hand and unfair critiques or reviewer misconduct on the other go unnoticed for readers and vanish (28). At the same time, reviewers contributing substantively, at a level deserving an authorship or contributorship credit, lack options to claim their input.

In the era of open access, it has become possible to publicize an unlimited number of reviews by uploading these critically important, yet non-citable items on the journals' websites along with related articles. As a prime example, large open-access publishers such as BioMedCentral embraced the policy of open (public) reviews to encourage responsible commenting. It also allowed unmasking reviewer identities and publishing their contributions.

In 2012, Andrew Preston and Daniel Johnston went further and launched Publons, a New Zealand-based academic networking platform for opening access to peer review and crediting reviewers for posting their reviews on the platform. By partnering with PeerJ and GigaScience, two large open-access journals, they uploaded and validated an initial set of reviewer notes. Contributions, highly rated by Publons members, were assigned Digital Object Identifiers (DOI), allowing the best reviewers to expand their lists of published citable items (29). Following the integration of Publons with Altmetric in 2013, a new scoring system was introduced for widening exposure to social networking media and measuring alternative impact of the reviews (30). So far, 66,224 reviews from 3,675 journals have been archived on Publons. All reviews have been processed after an agreement between reviewers, editors, and journals. Reviewers can now get credits for posted items and export these records to their IDs for editorial and academic promotion and research funding applications (31).

With the rapid development of the digital editorial management and the integration with individual authors' and researchers' IDs, such as ORCID IDs, more reviews can go open and aggregate on specifically designed hubs to serve their main purpose - improve the quality of science communication (32).

Over the past decade, the advent of digital technologies has changed most features of peer review. It has become much easier to pick referees by searching relevant publications through online databases and to communicate with all contributors of the process. Reviewers have been equipped with online bibliographic tools for comprehensive evaluation of the integrity, originality and ethical value of journal submissions. Positive technological changes have shortened timelines of the review to approximately two weeks for an average reviewer evaluation and to five weeks for an editorial decision (33). The accelerated mechanics allowed publishers to consume the exponentially increasing journal submissions and to meet the new demanding criteria of online indexing services. However, no comprehensive strategy has been proposed so far to reward reviewers for donating their precious time and contributing to the quality and integrity of scholarly publications.

Contributions of peers from different professional backgrounds are increasingly important in the current times, when big data, multidisciplinary and cross-country research studies are gaining momentum. The best reviewers invest their expertise in the correctness of research methodologies, statistical analyses, graphic representation and interpretation of the results that may transform into a 'creative' input and bring about a deserved authorship or contributorship credit (34, 35, 36).

The open access movement and available digital technologies gave birth to public peer review and, as a consequence, storing reviews on hubs, such as Publons, where reviewers get online publication credits tagged with DOIs. Such reviewer credits can be counted for distinguishing the best reviewers and fully benefitting from their expertise by repeat invitations to serve as referees and accepting as editorial board members.

In conclusion, all stakeholders of science communication should be aware of the available set of rewards for best reviewers and incentivize them for the benefit of the journals and the scientific community at large (Table 1). None of the currently known reviewer rewards and incentives is sufficient and effective on its own. Combining these incentives will allow engagement of more willing and qualified experts in the quality and timely evaluations of scholarly journal submissions.

Table 1
Main rewards and incentives for peer reviewers of scholarly journals

• Click for larger image
• Click for full table
• Download as Excel file