Improving how non-traditionally submitted research outputs are evaluated in the Research Excellence Framework

Non-traditionally submitted outputs (NTOs) are research outputs such as artifacts, databases, software, and exhibitions. They are vital to most research and often ex press essential methods and technologies. The Research Excellence Framework (REF) recognises their importance and actively encourages the submission of a wide range of NTOs. However, traditional outputs like journal articles are prioritised for submission to the REF, as Higher Education Institutions (HEIs) are sceptical that NTOs will be evaluated fairly and competitively. This risk avoidance has resulted in 97.6% of outputs submitted to REF2021 being scholarly publications.

About the research

Our research took place during the Hidden REF Festival 2025, tasking research stakeholders with evaluating NTOs using the REF criteria and ratings. We examined which aspects participants considered when evaluating and rating NTOs. The experiment was based on a ‘think aloud’ method which enabled participants to verbalise and record their thought processes in real time. Sixty eight observations were recorded.

The study sheds light on the urgent need to embed trust in NTO evaluation by adopting a two‑pronged approach to guiding both HEIs and REF panels on how these outputs be evaluated.

Policy recommendations

  • NTO submission guidance should incorporate a standardised template outlining specific required elements. To enable this level of detail, the current 300‑word limit could be increased to 500 words.
  • Provide panels with stakeholder‑co‑created subcriteria that adapt the concepts of significance, originality and rigour for NTOs; for example, originality could incorporate considerations of creative innovation.
  • REF assessments should, where appropriate, draw on user‑evaluators (e.g., educators, policymakers, research support roles, technicians). This would complement existing expertise and bring a broader range of relevant perspectives into the evaluation.
  • REF panels should consider evaluating NTOs in type based groups rather than folded into Unit of Assessment (UoA) based peer review. This approach would enable panels to draw on a majority of expert reviewers, supported by UoA experts to maintain disciplinary consistency.

Key findings

Evaluation of NTOs is challenging because:

  • Evaluation relies heavily on the submission description of the NTO which is often inconsistent and lacks standardisation across the different NTO types and submissions;

Evaluators do not always have direct expertise in the output types and it can take longer to assess these diverse formats;

  • The articulation of the research process in NTO submissions is insufficiently clear, contributing to lower evaluation scores;
  • The boundaries between research, artistic practice and professional practice blur, distracting evaluators away from focusing on research excellence. The REF criteria are important for NTOs, but:
  • REF criteria were not applied uniformly across NTOs. Instead, evaluators treated the criteria as flexible tools and used them to the extent that the output itself allowed;
  • Evaluators used multiple, sometimes inconsistent, cues to judge the NTO’s quality, including institutional prestige, ties to traditional outputs, perceived social impact, download figures, and similar imperfect proxies;
  • High‑quality NTOs prompted fuller, more detailed and frequent engagement with the criteria, whereas weaker outputs led to more limited use. In effect, the richness of the NTO shaped the richness of the evaluation;
  • This underscores the importance of providing structured, detailed submission guidance and developing NTO‑specific subcriteria to support fair, consistent and meaningful evaluation.

Further information

The Research Excellence Framework

Ref submitted outputs' details

www.hidden‑ref.org  

The Hidden REF White Paper

Embedding Trust in Evaluation (E-TIE) is the funded project behind the Hidden REF Competition and Hidden REF Festival. It is funded by Research England from 2025 to 2030.

The researchers

Dr Ola Thomson, University of Bristol;

Professor Gemma Derrick ,University of Bristol;

Professor James Baker, University of Southampton

Lyndsey Ballan tyne, University of Southampton

Professor Simon Hettrick, University of Southampton

Dr Ben Thomas, University of Southampton