Research funding organisations are well positioned to be drivers of change towards more responsible research assessment practices. They are in a unique position to shape research culture, determining the allocation of resources, shaping research priorities, and influencing who receives funding and for what projects. Funders around the world are experimenting with formats and processes, like the narrative CV, that move beyond the use of journal impact factor or journal prestige to assess research quality. These new formats aim to better assess research on its own merits, address unconscious biases, and value a wider range of research outputs. The design and supporting processes that go into creating funding calls or programmes influence factors like: who has access to funding, applicant population, the formatting of proposal materials, the types of work outputs that applicants highlight, and how proposals are assessed by reviewers. 

DORA’s Asia-Pacific (A-P) and Africa, Americas, Europe (AAE) funder discussion groups were created in 2020 to support communication between funding organisations about research assessment reform, and to accelerate the development of new policies and practices that lead to positive changes in research culture.

During their September 2022 meetings, these groups participated in a mapping exercise to identify existing research assessment interventions and areas to align on for future work. One of the key themes that emerged from this exercise was the importance of considering all steps in the funding process as part of responsible research assessment. This included the processes that take place before research is submitted for funding (pre-award processes), such as the timing of proposal calls and deadlines, transparency and guidance, phrasing and language use, selection and training of reviewers, and planning for “evaluating the evaluators” or “evaluating the evaluation process”.

In January 2023, the Elizabeth Blackwell Institute for Health Research at the University of Bristol, in collaboration with the MoreBrains Cooperative, organised a symposium of researchers to analyse how pre-award processes can present obstacles to researchers and unintentionally reinforce biases like the Matthew effect, in which resources flow to those who have them. The findings from this symposium, and from subsequent discussions with research funders, resulted in a report with general recommendations on how the scholarly community can support transparency and equality*, diversity, and inclusion (EDI) in pre-award processes. 

A natural next step for this report was to identify which recommendations were most practically applicable. Given the interest of the DORA research funders discussion groups in this topic, DORA partnered with Elizabeth Blackwell Institute and MoreBrains to identify which of the report recommendations were most actionable for research funders to undertake. In September and October 2023, DORA, Elizabeth Blackwell Institute, and MoreBrains organised a set of parallel symposia and workshops for each funder discussion group. The groups used the Elizabeth Blackwell Institute report recommendations as a tool to develop and prioritise areas for intervention.

Key takeaways

Feasibility and impact

During the virtual symposia held in September 2023, Ian Penton-Voak (University of Bristol), Josh Brown and Alice Meadows (MoreBrains) presented the report rationale and findings to the groups. This was followed by a presentation from Gearoid Maguire (Wellcome) on the Equitable Funding Practice Library to prime the subsequent breakout discussions. Members of the research funder groups then conducted a deep dive into select recommended actions from the Elizabeth Blackwell Institute report and identified the 1) feasibility and 2) anticipated impact(s) of each recommendation. Here, the term “impact” was used in the context of the anticipated benefit that researchers might have from a particular recommendation, in the form of reduced barriers to funding applications and/or greater transparency. Funders in both groups discussed several mechanisms by which this recommendation could be implemented. The actions and potential ways to address them practically included: 

• Avoid training becoming yet another burden. 
• Diversify the idea of research careers, and break down cultural silos. 
• Explore and experiment fairly and transparently. 
• Consider the impact of how funding opportunities are designed, structured, and shared. 
• Leverage providing applicants with reviewer feedback to support reapplication and reviewer integrity. 

During the large group discussion, several important points of consideration were highlighted by the group: 

1. Communication is a key facet of pre-award processes that can either encourage or discourage applicants. 
2. Funders should work to focus on inclusivity and reach (e.g., gender neutral language, opportunities to apply in other languages, broaden communications channels used to share funding calls, target smaller organisations and early career researchers). Of note, the value of protected or targeted opportunities in various forms was mentioned in the majority of Asia-Pacific breakout sessions. 
3. Co-creation itself was highlighted as a potential barrier to more inclusive call design. For example, if a call is co-created with researchers, the biases and priorities of those academics influence call design and limit flexibility on the part of the funder. 

Reform in action

The work from the September symposia informed the design of the October workshops, which featured recorded lightning talks from members of each discussion group. These lightning talks highlighted practical examples of how funders have worked to reduce barriers for applicants and support transparency and/or EDI in their funding calls. 

Watch the lightning talk recordings

Identifying realistic and transformative actions

Following the lightning talks, the attendees reviewed the list of actions that were discussed during the symposia and prioritised which actions they would like to focus on. During the subsequent breakout sessions, participants drafted brief action plans on how best to put each proposed action into practice. Examples include:

“Our group focused on making call structure simpler, clearer, and more inclusive. The activities needed to implement this idea are working with the communications department (including editing and creating plain language), and community consultation on what is needed. The resources needed to implement this idea are time and staff, for inter- and intra-organisational communications, access to experts, and community members. The people or groups who would need to be involved in this are community members, communications department, policy staff, EDI advisors, IT department. We think we should all take this idea forward because it’ll increase applications.”

Funders then voted on the top actions that would be 1) most realistic for funders to implement and 2) most transformative to the pre-award ecosystem. The areas of activity that were most impactful or transformative were:

• Making assessment more holistic
• Making call structures simpler, clearer, and more inclusive 
• Improving training for reviewers and assessors 

The areas of activity that were most feasible and realistically achieved were:

• Improving training for reviewers and assessors
• Making call structures simpler, clearer, and more inclusive

Across the two workshops, which represent different geographic regions, the participants felt that making call structures simpler, clearer, and more inclusive was one of the most realistically achievable for funders and potentially most transformative for applicants and the pre-award landscape.

*  In the report, Elizabeth Blackwell Institute and MoreBrains team used the term ‘equality’ as in the acronym EDI, for equality, diversity, and inclusion to refer to approaches and values that may lead to equitable outcomes, noting that “the terms ‘equality’ and ‘equity’ are different and [that they] felt…  the focus on equality of rights and opportunities was highly pertinent in the context of the pre-award process.”

This story was republished from DORA.

Visit DORA to read about next steps

Find out about our Open and transparent research project