By Laura Harvey, Head of Relations, Publons

Published: 17 September 2019

The theme for this year’s Peer Review Week is quality. In advance of a report publishing this October on the global state of grant peer review, Laura Harvey, Head of Relations at Publons, teases some early findings of the report on the question of quality.

Last year Publons, part of the Web of Science Group, published its inaugural Global State of Peer Review Report, the largest ever study of publication peer review.

This year we’re pleased to announce Global State of Peer Review: Grant Review In Focus due to be published at the beginning of October.  This report sprang from feedback on our inaugural report highlighting a substantial gap in data and insights into the peer review of research grant and funding applications. In this special Peer Review Week blog for the AMRC, we’d like to preview some early findings on the question of quality in the grant review process.

Grant Review in Focus - how did we do it?

Publons is a researcher profile site and peer review solutions provider. With over 1.8 million researchers on Publons using their profiles to get recognition for over 4 million reviews, we’re in a unique position to offer data and insights into the world of peer review.

So how did we go about collecting the data we needed for Grant Review In Focus? We surveyed over 4,500 Publons users about grant review - the how, when and why they do it, and importantly, asked them what they thought about it.

We then combined this unique survey data with the full power of the Web of Science datasets and interviews with a range of research funders, including the AMRC, MRC and the UKRI - making Grant Review In Focus the most extensive researcher survey on grant peer review ever conducted.

What did we find out?

In short - a lot! Our surveyed researchers have reviewed or applied for grants for over 800 unique funders, spread across 95 countries. You can read a preview and sign up for a copy of the full report here, but here are some early highlights as regards peer review quality, this year’s theme for Peer Review Week.   

Grant peer review as a bulwark of quality grant funding

We found the use of experts continues to be viewed as a bulwark of quality decision-making in the funding process, ensuring continued trust and accountability. 78% of respondents agree or strongly agree with the statement that peer review of grant applications is the best method we have for ensuring we fund the best research (see Figure 1). The researcher perspective resonated with our funder interviewees. Matthias Egger, President of the National Research Council of the Swiss National Science Foundation (SNSF), maintained that “peer reviewing is essential to the quality of our evaluation process.” This was endorsed by the Royal Society Te Apārangi of New Zealand, who declared that “the reviewer’s judgement of quality play(s) the most significant factor in determining who gets funded.”

However, researchers also identified perceived failings of the peer review process, including fair treatment of junior researchers, the promotion of innovative or risky research, and bias. Researchers’ perspectives on these issues is explored in full in the final report.

Grant peer review quality and trust in the process

Beyond acting as a quality control for a funder’s grant decisions, we found grant peer review quality was also essential in maintaining trust in the grant decision-making process. As Arailym Akbolat from the Kazakhstan National Center of Science and Technology Evaluation (NCSTE) noted, “One cannot underestimate the importance of the expert reviewer, and therefore the quality of evaluation, to supporting researcher trust in the funding process as a whole.”

Grant peer review quality at risk

However, alongside these positive affirmations, the report also found that grant peer review quality is at risk. The report looks in detail at issues such as reviewer workload, rejection-rates and reviewer incentives. Uniting all these issues was a struggle to allocate the best reviewers to the proposals at hand. As Eleanor Riley, Director of the Roslin Institute and experienced grant panel member, observed, “every reviewer that declines to review gets you further away from the most expert reviewers for the application in hand. There is a real danger we lose quality in the process if we do not address this.”

New ways of sustaining and advancing grant review quality

However, the report identifies several ways to sustain and even advance grant review quality, including training, transparency and incentives. To look just at incentives here, we found that 89% of reviewers believe that greater recognition of grant peer review work would improve the grant peer review process, whilst nearly half of reviewers are dissatisfied with the recognition they currently get for doing grant peer review. Efforts from funders in this area will clearly be met with enthusiasm by reviewers and may go some way to re-energizing the grant reviewer community – drawing previously reluctant reviewers into what is currently an overworked pool, and reminding established reviewers of the huge contribution they make to the quality of the funding process through effective and high quality review.

“OFFERING GREATER RECOGNITION TO REVIEWERS IS DEFINITELY SOMETHING WE SEE CHANGING IN THE FUTURE AND SUPPORTING INCREASED TRANSPARENCY.”

AMRC Interviewee, Grant Review in Focus

Interest piqued? Find out more...

Please register here for the full report, which will be published at the beginning of October.

In the meantime, please drop us a line with any questions or just share any comments you may have on these initial findings. You can also read more about Publons offering for research Funders here or drop us a line with any questions ([email protected] or [email protected])