Peer review is currently the preferred way for health and medical research charities to decide what research to fund. Done properly, it should allow charities to support the best research and the best researchers.
However, peer review does not come without its pitfalls. From being time intensive to potentially obstructing innovative or high-risk research, charities and researchers face many hurdles when it comes to peer review.
This thought-piece is for charities to reflect on the future of peer review and consider changes that could be made. You can navigate by clicking one of the sections listed in the below diagram.
Introduction
Peer review is the way in which all AMRC members make funding decisions on grant applications. In peer review, the research proposal is read and commented on by experts with knowledge and interest in the subject area. This is usually done anonymously to enable open and constructive criticism.
It is currently the preferred way for health and medical research charities to decide what research to fund. Done properly, peer review should allow charities to support the best research and the best researchers. This, in turn, can help charities maximise the impact of their funding and deliver changes that really matter to their supporters and stakeholders, so that patients benefit from the fruits of research. [To further understand the reasoning and processes behind peer review, please see AMRC’s Principles of Peer Review]
Peer review can look different across our membership and may vary depending on the type of research being undertaken, who is doing it and for how long. The peer review process must be practical and adaptable – there isn’t a ‘one size fits all’ model and what has worked in the past may not work in the future.
Peer review, however, does not come without its pitfalls. From being time intensive to potentially obstructing innovative or high-risk research, charities and researchers face many hurdles when it comes to peer review.
Our goal is for charities to support the best research and the best researchers, but is this not something that is continually changing? As novel research methods emerge, young researchers arise, and charities increasingly develop new funding approaches, we want to ensure that the peer review process will continue to pick out the applications that will deliver the best possible outcomes, without tradition dominating in a continually evolving environment.
Alongside this, charities should seek to constantly improve the process to ease the burden for themselves, their advisory committees, their reviewers and grant applicants.
So, whilst the principles of peer review (accountability, independence, balance, rotation, impartiality) remain the same, we wanted to set out the current challenges of the processes involved, and how charities might seek to address them.
This document is a thought-piece for charities to reflect on the future of peer review and begin to consider ways of:
THE PEOPLE INVOLVED
Burden
Peer review is time consuming, taking 9 to 12 months from the receipt of applications to the ultimate funding decisions.
Burden on the funders and reviewers:
Funders report difficulties with identifying reviewers, reviewers declining invitations to review, late submission of review reports and high administrative and financial costs.
Burden on the applicants:
While efforts to reduce the burden are often focussed on funders and reviewers, data shows that approximately 75% of the burden falls on the applicants (Guthrie et al., 2017).
Recognition
Traditionally there has been little recognition or incentive for researchers to perform peer review. Peer review is generally undertaken because it is ‘the right thing to do’ and part of an academic’s duty.
However, many are questioning whether this argument still stands, as there is increasing pressure on academics to bring in funding and deliver impact.
This means that the traditional lack of incentivisation or recognition may no longer be viable. Proving recognition to reviewers could help to offset some of the burden associated with peer review and could strengthen the relationship between the funder and reviewers.
THE DECISION-MAKING PROCESS
Bias
The peer review process can be impacted by different types of bias by all players involved. This can in turn prevent new, innovative research from being funded.
Biases in reviewer selection:
Funders may display biases in their selection of reviewers. Some rely on a small pool they know are dependable, which reduces diversity. Others favour UK reviewers as they believe international reviewers may not be aware of the nuances and funding policies of the UK system.
Biases towards applicants:
Reviewers may choose to assess applicants according to the number of papers they’ve published, or the funding they’ve received in the past. This may disadvantage early career researchers and those who have taken career breaks, especially when they are submitting innovative applications.
Funders and reviewers may, often unintentionally, introduce biases such as ageism, sexism, racism, cronyism (appointment of friends and associates) and institutional affiliation into eligibility and selection criteria.
Biases towards the proposed research:
Reviewers may favour proposals that align with their own ideas or applications in areas they are completely unfamiliar with (Wang & Sandstrom, 2015). However, others have suggested reviewers may be more critical of those within their own expertise (Gallo et al., 2016).
Reviewers often favour traditional approaches to research and may view more ground-breaking research as ‘reckless speculation’ with insufficient preceding work. This is likely contributing to the observed drop in the number of innovation or high-risk applications.
Applications which combine multiple threads of research, such as interdisciplinary and translational research may be unfairly disadvantaged. It may be difficult to find reviewers with the necessary expertise to be able to review an entire application in detail. In combination with reviewer’s reliance on funding success as an important assessment, a negative cycle may develop for researchers attempting to undertake research in these areas.
Quality
Peer review may not necessarily fund the best quality research.
Ranking applications
Reviewers can typically identify the top 20-30% of applications, but struggle with identifying the top 10% (Fang & Casadevall, 2012).
The duration of a reviewer committee meeting may affect ratings (e.g. they may be harsher to those applications reviewed later in the meeting).
Assessing the research
Assessing applications based on publication history, which is common, may not necessarily translate into the best quality research being funded. This is particularly true for early career researchers and for applicants who have performed high-quality research that has not been published (i.e. replication studies, negative results).
In addition, reviewers’ ratings may vary considerably due to genuine difference of opinion or differing interpretations or understanding – it is difficult to ascertain which. The risk of misinterpretation of applications may be further exacerbated by the fact that reviewers have no real opportunity to ask applicants questions to inform their decision.
On topics where they are less knowledgeable, reviewers may defer to the perceived wisdom of a few specific experts, reducing the chances of critical discussion and quality review.
Limitations
The transparency and anonymity of a reviewer’s identity may have an effect e.g. if reviewers’ identities are known to the applicant, it may lead to less honest reviews whereas if they remain anonymous it may lead to harsh or poor-quality reviews.
The funder’s role in research is often limited to the application stage meaning that projects that are perhaps under-performing will still be funded until completion.
Indeed, some research has shown that peer review is a weak predictor of commercial success of early-stage technologies in small businesses (Galbraith et al., 2010).
Are any of these issues impacting your funding process and decision? Use these questions to reflect on the efficiency and effectiveness of your current peer review system and identify areas for improvement.
Read more