Planning online tests

Writing questions and developing assessments involves a range of skills and competences. In the first instance we recommend that you familiarize yourself with:

• the online tutorial writing effective MCQs type questions and the question types of the system you intend to use.
• the broader guidance on the system you intend to use, available online for:
  • Blackboard: Creating tests and surveys and Blackboard tests: Uses, ideas and hands on practice
  • Questionmark

Question types suitable for online delivery

You will need to be aware of what styles of questions will work online. To help with this you could look at other people's questions and get some training on the system you intend to use.

We recommend talking to colleagues in your School of Faculty who are already running online assessment and have experience writing questions for online delivery. This is particularly important if the assessment requires the questions to be written in a specific format (eg 'single best answer' in medicine) or there are other criteria that should be followed such as in the number of distractors or use of negative marking. Contact DEO if you would like to know more about online assessment activities in your School.

The following resources can also help:

• Blackboard question types
• Example questions from Bristol
• Constructing written test questions for the clinical sciences (PDF, 1.2 MB), Susan M. Case, David B. Swanson, 2002
• e-assessment by design: using multiple-choice tests to good effect (PDF, 269 kB), D Nicols, 2007

Creating questions

The first step in the creation of the content involves creating a set of questions or a pool. Depending on the system you intend to use questions can be;

• created directly on the system.
• imported using excel file or a third party tool. This Question import template is an example of an excel pre-formatted file that you can use to upload your questions onto Blackboard or Questionmark. If you need further training or would like to discuss ways to import an existing bank of questions please contact the Digital Education Office.

Organizing questions

Even if you are starting with a small number of questions it is worth thinking about how you will organize them. To help with this you can use:

• folders /topics /pools - Questions can be stored together grouped by topic, or by the assessment they are used in, or type of question.
• naming convention - Be consistent naming your questions. Item name should include information about what the item tests or reference to a specific topic. Depending on how long the name can be (some system allows up to 20 characters) you may want to include other information such as question type, author, or standard setting tag.
• tags/metatags / labels - Some systems allow you to pre-define tags that can be applied to questions. Examples of useful tags include: the year in which it has been used (eg 2022), the part of the curriculum it tests (eg Grammar), or the standard settings (eg EE, EI for Ebel method).

Marking schemes

Before designing questions you need to be clear what is possible within the system you intend to use. For example: it may not be possible to give 0.5 of a mark; some mark schemes that work on paper are prohibitively difficult to code online. You may also want to talk about what marking scheme has been adopted in your School, eg if and how negative marking is being used.

Including feedback

Most systems will allow you to add feedback to questions, which can be shown to students after they complete an assessment. The feedback can then be hidden for particular assessments (eg exams).

When designing questions, even if they are intended for use in exams, it is a good idea to consider adding feedback. This will enable a single set of questions to be used for multiple purposes, for example in formative or diagnostic assessments.

Peer review of questions

Questions should be finalized and validated before they are made available in an assessment. This involves making sure that the question wording, the answers and the feedback are clear and unambiguous and don't contain any spelling mistakes, and that the questions are technically faultless.

Peer review of questions is always useful. It might be informal, a colleague having a quick look through and providing comments, or more rigorous in the case of exams.

Here are some possible peer review procedures:

• Review using a Word document. The designer writes questions in Word, circulates them to the peer reviewers, edits and finalizes the questions in Word and then passes them to the administrator, who enters them into the system. In this case the designer is responsible for making all the necessary changes before the questions are entered into the system. The disadvantage of this method is that questions may need to be altered slightly after being finalized, in order to make them work in the system.
• Review within the system. Questions are entered into the system, tested and published electronically for peer review (for example by making them available as a test). The academic lead is responsible for deciding which comments /changes should be incorporated into the questions, though it may be someone else who actually implements the changes.
• Multiple authors, single moderator. Questions are entered into the system by multiple designers, then one person audits all the questions. In this case it may be the authors or the auditor who finalizes the questions.
• External examiner review. The easiest way to provide access to the external examiner is to make the assessment available through the system (create a schedule), if this is not possible you may be able to export the assessment in HTML format and save it in pdf. Please contact DEO if you need help with this.

Developing assessments

Building assessments involves selecting the questions from the question folders and defining the assessment settings. In the assessment settings, you can define options for scheduling, which is when the assessment is available to students, the duration of the assessment, whether students will get feedback on submission, and how the questions will be presented, eg one per page or all in one page.

For summative assessment we recommend specific settings;

• Questionmark recommended settings for high stakes assessments
• Blackboard recommended settings for high stakes assessments

Accessibility in assessment

Accessibility in assessment must be considered at the start of the planning process to ensure that the online content complies with the law and that students are offered an alternative exam format if required. Online assessments enable students to use assistive technology such as a screen reader, and assistive functionalities such as changing the font size or the background colour, however, it is also important to consider the needs of students who may have visual problems (colour blindness) or hearing loss as well as cognitive problems which includes a range of conditions such as dyslexia.

Both Blackboard and Questionmark provide specific functionalities to support diverse student' needs, for example, it is possible to change the layout of the screen, the size of the font or to change the contrast, however, it is up to the person developing the content to ensure that this meets accessibility requirements. The following guidance may be considered;

General principles:

• Use accessible styles, colour and fonts (more information can be found on the University of Bristol web accessibility page), avoid using different colours to convey meaning
• Include an alternative text to images (find more information on the Alternative text page) and avoid text over images or patterned backgrounds
• Add HTML markup to tables (find out more on Tables concept page )
• Use left aligned text, 1.5 line spacing and a suitable font size no smaller than 12pt. Avoid using all capital letters, cursive or script fonts
• Use paragraph with headings or subheadings (using heading styles), or bullet points to break up information. Avoid using an extra large font, bright red colours in instructions or introductions
• If using hyperlinks, makes then descriptive and meaningful, use a URL short link if required, avoid 'click here'
• Allow student control and navigation of media content (audio and video) and provide a text alternative, if available disable the automatic start of any audio or video content
• Avoid using too much animation in the same page, interactive question styles such as 'hotspot and 'drag and drop' must not be used in high stakes assessments or exams.
• Ensure that questions are numbered and appear in a logic sequence, if the randomisation option is required to ensure that each question is separate and doesn't refer or link to another question in the same assessment
• Contact DEO to find out how the online assessment system can be customised

Blackboard:

• Blackboard Ally is a system for checking accessibility of your course. Find out more about Blackboard Ally.

Questionmark:

• Use the recommended UoB template. You can find out more about the templates in Authoring questions and assessments in Questionmark

If the online format is not suitable for a student contact the Examination Office or look into alternative exam arrangements

Depending on the nature of the assessment and the system you intend to use, before scheduling the assessment you may need to carry out some administrative tasks to ensure that students know where to access the assessment and where to find their results.

For formative, if no communication is sent to SITS eVision students will not be able to see the assessment information in My Bristol, you will need to provide the assessment information through other channels (eg Blackboard).

• Blackboard; set up a new test in the desired content area, the system will automatically create a column in the Gradecentre where you will be able to monitor progress and check results, student will access their results from My Grades once they have been released.
• Questionmark, scores and feedback can be provided on submission of the assessment, if you wish to import the result in the Gradecentre you will need to export them from Questionmark and import them into the Blackboard Gradecentre using an excel spreadsheet.

For non invigilated summative assessment

• Blackboard or Questionmark; open a ticket with SSIO to declare the assessment to SITS eVision as BB Test submission type. This will then create a column in the Grade Centre and a blank test.

On the BEAM sharepoint siteyou can find further information about set up of submission points and assessment guidance.

For invigilated exams, visit the Running and invigilating online exams information page.

For asynchronous events: More and more students are using their own mobile devices to take assessments from home or around the campus in open access spaces. Formative assessment are generally left open for the duration of the course, students can attempt them multiple times and feedback is provided at the end of the assessment. Summative non invigilated assessments are usually treated as 'take home exams' and you may need to apply certain restrictions; eg availability of assessment, feedback, etc. Generally students prefer paper based tests to online tests because of the anxiety associated with the technology, it is therefore vital that students familiarize themselves with the online system before taking the online exam, familiarity with the system over time is likely to reduce students stress levels. The following options may be considered;

• Ensure that students use their UoB credentials to authenticate into the system
• Include questions that require problem solving or analysis of case scenarios.
• Run a practice test for the students to ensure that their laptops are correctly set up, especially if the tests contains an audio or a video file. You may also recommend that any other applications running on the device should be closed to avoid interference with the browser/assessment application.
• Provide students with information and instructions before they open the assessment, to allow them to arrange a time to take the test and ensure they have a stable internet connection. Also, include instructions about how to troubleshoot technical issues.
• Consider using a locked down browser (eg Questionmark’s secure browser) and make recommendations about compatible browsers.
• Set a time for the test, allow time for the students to read and answer all the questions, review their answers and submit.
• To avoid collusion, use a question by question layout, randomize the questions within the test and shuffle the answers in MCQs type questions. If you have a large pool of questions consider using 'randomly select questions from a bank': in this way students will get different questions in the assessments.
• You can allow only one attempt or multiple attempts but tell students only the first attempt will count.

For invigilated exams please visit the Running and invigilating online assessment page.

Managing results involves reviewing the scores, finalising the results and making them available to the students. This involves:

• Marking any open-ended questions
• In the unlikely event that a problem is found with any questions, re-mark as necessary
• Reviewing the marks within the system or exporting them in an excel format
• The online systems can generate different types of reports, for example a student report which can be useful for student tutorial, or an assessment overview which can be useful for staff or external examiner
• Providing sample results to external examiner
• Archiving the results locally

If this is the first time you introduce an online assessment in your Unit/Programme it is important to gather both student and staff feedback, this will help organize the next iteration of the online assessment. The following is recommended:

• Collect feedback from students and/or staff, as appropriate
• Produce statistical reports on questions, cohort. Online assessment systems provide options for item analysis reports which generates detailed and statistical analysis of individual questions. This can be used to identify the validity and effectiveness of individual question.
• Review meeting for all involved

More information on Item analysis for Blackboard and Questionmark

Before implementing an online assessment it is worth considering some of the key aspects of the implementation process, these include:

• ROLES AND RESPONSIBILITIES – Identify what level of support you will need and who is best placed to do this work within your School; eg scheduling assessment and running reports.
• INFRASTRUCTURE AND TECHNOLOGIES – Gather as much information as you need about the assessment system you want to use and look into the availability and suitability of PC labs, and any IT support you may need.
• CRITICAL DATES – Identify and communicate any critical dates to all the staff involved to ensure you get the support you need; creating and quality assuring the assessment, scheduling the assessment, exporting results.
• PRACTICE TEST – To comply with the Regulation and code of practice for taught programmes (section 11.26-11.33), before online exams, students must familiarize themselves both with the assessment system and with the question types that will be used in the exam. Practice test can be scheduled for students to take from their own locations or in a PC lab as in the exam.
• CONTINGENCY PLAN - Online systems are generally very robust and reliable, however, it is good practice to have a contingency plan. This can be providing students with a paper copy of the assessment or rescheduling the assessment.
• RESULTS – Identify the format in which you want the results to be made available to staff and students, if required. This can be through the Gradecentre on Blackboard or exported in a word format and sent by email.