Fantastic impact tools and where to find them By Charlotte Pelekanou, Research Intern, AMRC Published: 13 April 2017 There is an increasing demand for funders to assess their research impact more thoroughly. In March we held an Impact and Evaluation Masterclass to learn from medical research funders and research institutions about how best to collect and disseminate research impact. In this blog we have summarised the key points of the day. Why should we measure impact? Traditionally research impact has been judged by assessing academic publications. However, funding research leads to a wide range of outputs which can be categorised into: Generating knowledge and resources Translating research ideas into products, protocols or treatments Influencing policy and engaging a wider audience Leveraging further funding and partnerships Supporting researcher development As we start to capture these different forms of impact we will discover how charities can use impact data to achieve their strategic objectives. Figure 1. 4A’s model of research assessment How should we measure impact? When thinking about research impact the first thing we need to decide is what questions we want to ask. During the masterclass, Andrew Knowles from Cancer Research UK stressed that a lot of problems in impact evaluation stem from a lack of clarity on what is being evaluated and why. In this ‘golden age’ of data it’s easy to collect large quantities of data but quantity of data doesn’t necessarily equal insight. Conversely as research has so many possible outputs, there isn’t going to be a single magical bullet to measure all research outcomes. Therefore defining your objectives and how to achieve them prior to impact evaluation will save yourself from data-induced headaches. Luis Tojo from the MS Society also highlighted that the impact data you collect needs to mean something to your stakeholders like trustees, patients and donors. If it doesn’t mean anything to someone outside of research, is it a valuable data point to collect? He also emphasised that stakeholders should be involved when planning to collect impact data, allowing them to contribute what information they would find interesting/important to have. Once you have decided what data you are going to collect you then need to decide how. A number of different companies attended our masterclass to talk to our members about their research impact tools and how they could help charities. Figure 2. Where to find research impact tools: Clarivate Analytics, Researchfish, Altmetric, Über Research, F1000, Vertigo Ventures What are the challenges of impact measurement? As discussed above, measuring impact isn’t as simple as counting the number of publications resulting from a grant. A lot of impact information is qualitative rather than quantitative making it difficult to measure. To add to this, multi-funded researchers, further funding and collaboration means that the point at which an outcome can be attributed to a specific grant is often blurry. There is also a long time delay between grant initiation and impact generation making tracking outcomes tricky. In addition to time delays, there are also challenges surrounding collecting impact data from researchers. As highlighted by a number of our speakers, a lack of understanding of the need to report impact leads to frustration from researchers when submitting annual reports or Researchfish data. This can be alleviated by collaborating with researchers and showing the outcomes of their data contribution. As mentioned by Julie Bayley at the University of Coventry “engage not enrage” is the key to collecting impact data. Julie also highlighted that in order to improve impact researchers must become impact literate. Impact literacy involves understanding the what (demonstrable benefit), how (process by which impact can be achieved e.g. engagement activities) and who (individuals who facilitate the achievement) of impact. Not having the right combination of these factors can lead to underachieving impact goals. How do you communicate impact? Once you’ve successfully collected impact data, the data needs to be translated into a story with the use of impact case studies. Creating a story around the data will make it more engaging and useful for other members of your charity such as the fundraising or policy teams in addition to stakeholders. It will also aid achieving the 4A’s of impact research, especially demonstrating to donors how money was spent and proving the worth of medical research to government. In addition to creating their own case studies, Arthritis Research UK utilised the REF2014 impact case study database to find and adapt other impact stories from the researchers they funded. Conclusion Although there are a lot of challenges to collecting and analysing impact data, collecting this data is vital for a charity to understand whether their funding is making a difference. There are lots of helpful resources available to promote impact and aid reporting it. The key to successful impact is to define what questions you want to answer about the impact of grants funded and consult stakeholders about what impact means to them. Going forward AMRC is continuing to support charities to find the best practice in collecting and analysing impact data. Increased sharing of impact data between charities could also help to benchmark and identify the best methods of funding for a charity to achieve its strategic aims. Manage Cookie Preferences
Why should we measure impact? Traditionally research impact has been judged by assessing academic publications. However, funding research leads to a wide range of outputs which can be categorised into: Generating knowledge and resources Translating research ideas into products, protocols or treatments Influencing policy and engaging a wider audience Leveraging further funding and partnerships Supporting researcher development As we start to capture these different forms of impact we will discover how charities can use impact data to achieve their strategic objectives. Figure 1. 4A’s model of research assessment How should we measure impact? When thinking about research impact the first thing we need to decide is what questions we want to ask. During the masterclass, Andrew Knowles from Cancer Research UK stressed that a lot of problems in impact evaluation stem from a lack of clarity on what is being evaluated and why. In this ‘golden age’ of data it’s easy to collect large quantities of data but quantity of data doesn’t necessarily equal insight. Conversely as research has so many possible outputs, there isn’t going to be a single magical bullet to measure all research outcomes. Therefore defining your objectives and how to achieve them prior to impact evaluation will save yourself from data-induced headaches. Luis Tojo from the MS Society also highlighted that the impact data you collect needs to mean something to your stakeholders like trustees, patients and donors. If it doesn’t mean anything to someone outside of research, is it a valuable data point to collect? He also emphasised that stakeholders should be involved when planning to collect impact data, allowing them to contribute what information they would find interesting/important to have. Once you have decided what data you are going to collect you then need to decide how. A number of different companies attended our masterclass to talk to our members about their research impact tools and how they could help charities. Figure 2. Where to find research impact tools: Clarivate Analytics, Researchfish, Altmetric, Über Research, F1000, Vertigo Ventures What are the challenges of impact measurement? As discussed above, measuring impact isn’t as simple as counting the number of publications resulting from a grant. A lot of impact information is qualitative rather than quantitative making it difficult to measure. To add to this, multi-funded researchers, further funding and collaboration means that the point at which an outcome can be attributed to a specific grant is often blurry. There is also a long time delay between grant initiation and impact generation making tracking outcomes tricky. In addition to time delays, there are also challenges surrounding collecting impact data from researchers. As highlighted by a number of our speakers, a lack of understanding of the need to report impact leads to frustration from researchers when submitting annual reports or Researchfish data. This can be alleviated by collaborating with researchers and showing the outcomes of their data contribution. As mentioned by Julie Bayley at the University of Coventry “engage not enrage” is the key to collecting impact data. Julie also highlighted that in order to improve impact researchers must become impact literate. Impact literacy involves understanding the what (demonstrable benefit), how (process by which impact can be achieved e.g. engagement activities) and who (individuals who facilitate the achievement) of impact. Not having the right combination of these factors can lead to underachieving impact goals. How do you communicate impact? Once you’ve successfully collected impact data, the data needs to be translated into a story with the use of impact case studies. Creating a story around the data will make it more engaging and useful for other members of your charity such as the fundraising or policy teams in addition to stakeholders. It will also aid achieving the 4A’s of impact research, especially demonstrating to donors how money was spent and proving the worth of medical research to government. In addition to creating their own case studies, Arthritis Research UK utilised the REF2014 impact case study database to find and adapt other impact stories from the researchers they funded. Conclusion Although there are a lot of challenges to collecting and analysing impact data, collecting this data is vital for a charity to understand whether their funding is making a difference. There are lots of helpful resources available to promote impact and aid reporting it. The key to successful impact is to define what questions you want to answer about the impact of grants funded and consult stakeholders about what impact means to them. Going forward AMRC is continuing to support charities to find the best practice in collecting and analysing impact data. Increased sharing of impact data between charities could also help to benchmark and identify the best methods of funding for a charity to achieve its strategic aims.