Case study: Self and peer assessment

Origin

School of Physiology, Pharmacology and Neuroscience, Faculty of Biomedical Sciences, University of Bristol

Tools used

Contact

Dr Phil Langton,

Objective

Reduce the administrative burden of conducting peer and self assessment exercises

Background

In the final year of our programmes in the Faculty Biomedical Sciences summative assessments are almost exclusively free text with emphasis on critical evaluative skills and Scientific Method. Although didactic sessions can provide advice on effective strategies and demonstrate using anonymised examples, the benefit of timely and detailed feedback on formative work should be evident.

Occurring in the first four weeks of the year and jointly taught by the School of Physiology, Pharmacology and Neiroscience and Centre for Applied Anatomy, Concepts and Skills (C&S) is a mandatory element of final year teaching for all (~140) students registered on five degree programmes. In addition to advanced library skills, numeracy, problem solving and statistics, Concept & Skills also provides guidance and training in research skills including critical (paper) review, abstracting (summarisation) and the construction of scientific argument. 

‘Paper Review’ is a key research skill, highly valued by employers.  Well developed skills of paper review enable one to make outwardly subjective assessments of published scientific work that is supported by a series of careful objective evaluations of such things as the:

  • Scope and bias of the background information
  • Definition of the outstanding questions or paradoxes that remain
  • Clarity and falsibility of the hypotheses
  • Suitability of the experimental approach
  • Rigor of the methodology employed
  • Data presentation and associated analysis
  • Interpretation – does the data support the interpretation(s)?
  • Conclusions – does the extant literature permit/support the author’s conclusion(s)?

Such a complex skill must be practiced and we [C&S coordinators] have chosen peer assessment to provide students with a formative opportunity to practice the skills of paper review.

What was done

The exercise we constructed required students to critically review two related scientific articles which were chosen to have conflicting (irreconcilable) conclusions.  The overall aim required the students to be able to defend their choice of which (perhaps neither) of the papers they would cite as relevant evidence in a piece of course work.

Students were obliged to demonstrate understanding of the process, including tallying the time allocated to the submission and evaluation phases, by making access to the materials contingent (via ‘adaptive release’) on achieving a minimum score in a quiz, which contained MCQs such as “When do you need to complete the first 'submission phase' of the exercise?”. Emails were also used to flag approaching deadlines and to encourage self-evaluation.

Students submitted their reviews during the submission window. After the deadline, assessment questions & marking criteria were provided. Students evaluated their own and 3 of their peers’ reviews anonymously against the criteria. Evaluation involved assigning marks and feedback comments. Once all evaluations had been done, the Instructor made the feedback available to students via the Blackboard Grade Centre.

Also a summary, produced by the academic responsible for the two seminars that preceded the on-line assessment, was released at the end of the evaluation process.  This outlined the key strengths and/or weakness of each paper – a legitimate version of the intended outcome.

Outcomes

The majority of students complied with the exercise although both submission and evaluation deadlines were extended to allow technical queries to be resolved.

A review of work submitted reveals the expected spectrum, from poor to excellent.  The evaluations are similar although there is evidence from the feedback comments that students were reflecting on their own attempts. 

The real benefits were the ease of administration (staff) and timeliness of feedback (student).

Despite some shortcomings described, the Blackboard Self and Peer Assessment tool does a good job of structuring a self- and peer-assessed exercise and is certainly very effective at managing the administration that could otherwise be very time consuming. 

There are a number of glitches in the software, particularly around timings. However there are ways to design the exercise so as to avoid these.