Responsible Metrics

University of Bristol signs San Francisco Declaration on Research Assessment (DORA)

The University of Bristol has signed the San Francisco Declaration on Research Assessment (DORA) on the recommendation of the University Research Committee. DORA is a set of recommendations designed to ensure that “…the quality and impact of scientific outputs…is measured accurately and evaluated wisely.” To date, signatories include seven other Russell Group universities, such as Imperial College London, University of Manchester and UCL.

The University is committed to the critical role that peer review and expert judgement plays in the assessment of research, but also recognises the value that quantitative metrics can play in complementing and supporting decision-making. We will therefore be developing a series of institutional principles for research evaluation and assessment, based on the recommendations presented in the Declaration, through the establishment of a Task and Finish Group of members from the academic community and Professional Services.

DORA’s key tenet is to “not use journal-based metrics, such as Journal Impact Factors (JIFs), as surrogate measures of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions”. It includes a series of recommendations for funding agencies, institutions, publishers, organisations that supply metrics and for researchers.

Recommendations for research institutions are to:

  1. Be explicit about the criteria used to reach hiring, tenure, and promotion decisions, clearly highlighting, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.

  2. For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.

Recommendations for researchers are to:

  1. When involved in committees making decisions about funding, hiring, tenure, or promotion, make assessments based on scientific content rather than publication metrics

  2. Wherever appropriate, cite primary literature in which observations are first reported rather than reviews in order to give credit where credit is due

  3. Use a range of article metrics and indicators on personal/supporting statements, as evidence of the impact of individual published articles and other research outputs

  4. Challenge research assessment practices that rely inappropriately on Journal Impact Factors and promote and teach best practice that focuses on the value and influence of specific research outputs

A number of recent reports have emphasised that a commitment to DORA is a way for the sector to address the inappropriate use of journal-based metrics, including The Metric Tide 2015 report on the use of metrics in research assessment. This Independent Review of the Role of Metrics in Research Assessment and Management was set up in April 2014 to investigate the current and potential future roles that quantitative indicators can play in the assessment and management of research.

The Metric Tide introduces the notion of responsible metrics as a way of “framing appropriate uses of quantitative indicators in the governance, management and assessment of research".

Responsible metrics can be understood in terms of the following dimensions:

  • Robustness: basing metrics on the best possible data in terms of accuracy and scope;

  • Humility: recognising that quantitative evaluation should support – but not supplant – qualitative, expert assessment;

  • Transparency: keeping data collection and analytical processes open and transparent, so that those being evaluated can test and verify the results;

  • Diversity: accounting for variation by field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system;

  • Reflexivity: recognising and anticipating the systemic and potential effects of indicators, and updating them in response.

In addition, the Forum for Responsible Metrics has been tasked with developing advice to the sector on practical ways to implement a culture of responsible research metrics using the principles/frameworks outlined in the DORA. The Forum is a partnership between the Higher Education Funding Council for England (HEFCE), Research Councils UK, the Wellcome Trust, Universities UK and Jisc, and it will be considering whether to develop an agreement with similar ambitions to DORA, utilising The Metric Tide report, which aligns with the UK research base.

Edit this page