Examination and assessment materials
Assessment materials gather evidence of learning and competence for decisions about progress, certification or selection. Their design connects objectives and scoring with response formats, item quality and a usable document or interface.
Purpose and planning
Healthcare assessments should test application of knowledge in practice as well as recall. Their structure should serve objectives defined before questions are written. Diagnostic assessment occurs early in a course, formative assessment during instruction and summative assessment after it. Results can support selection, grading, certification or continuous assessment. Self-assessment helps candidates identify their learning needs. Measurement gathers performance information; evaluation judges marks against a standard or other candidates.
A recommended development checklist is:
- Identify the assessment’s objectives and plan the writing in advance.
- Plan the duration, type of assessment and scoring system before writing questions.
- Give particular attention to document layout.
- Prepare a model answer and scoring plan while writing each question.
- Give the questions to a colleague for an external perspective.
- Use simple, clear questions to avoid ambiguity in interpretation and answers.
The target audience should be defined in advance and the terminology match its level. For example, a first-year pharmacy question asks for examples of diuretic classes and comparisons of their indications, actions and adverse effects; a third-year question asks when diuretics are needed in heart failure and for examples, doses and expected adverse effects. Comprehensive assessments should cover interrelated subjects, whereas a test of one topic needs specific questions.
The paper must be practical within its time, cost and candidate constraints, with enough time to complete every section. Words such as “briefly” or “in summary”, or keywords, can indicate the expected scope of a short answer. Marks should reflect the expected answer and item difficulty; equal-mark questions should be of similar difficulty as far as possible. In qualifying or competitive assessments, allocating marks to specific related tasks rather than one large question provides more control and a fairer measure of performance.
Reliability and validity
An assessment instrument must be practical, feasible, reliable and valid. Reliability concerns reproducibility: a candidate should obtain the same result in similar circumstances, while assessor inconsistency, language interpretation and layout can introduce variation. Simple wording and a good marking scheme help produce consistent results. Reliability can be investigated through inter-rater or test–retest methods and internal consistency.
Validity concerns whether the assessment measures what it intends and yields meaningful, appropriate decisions. A test can be reliable yet invalid if its results depend on terminology rather than the knowledge of drug classes specified in its objectives. Where statistical validity testing is impractical, adherence to objectives and review by a colleague uninvolved in writing the questions help check validity.
Combining methods can balance their errors: written and practical assessments, or an end-of-course paper and continuous assessment of clinical or experimental practice, provide triangulation.
Response formats
Each question type has advantages and disadvantages for writing, assessment and scoring. Created-response questions require candidates to compose a brief answer or extended essay and reveal writing, organisation and communication skills. They are relatively easy to prepare and harder to guess successfully, but take time to answer and score, limit coverage and can suffer scoring bias from handwriting, grammar, style, length and spelling penalties. A model answer and marking criteria should be prepared before scoring, with the question revised if marks cannot readily be allocated. In one four-mark pharmacy example, the scheme assigns 0.5 marks to drug choice, 1.5 to three monitoring requirements, 1 to a precaution and 1 to two adverse effects. A 2007 recommendation favoured created responses for formative assessment and advised avoiding them in comprehensive certifying assessments or where rapid, detailed feedback is needed.
Selected-response items ask candidates to choose an answer or give a word or short phrase. They take less time to complete and mark, permit broader coverage and reduce assessor scoring bias, but can be more difficult and time-consuming to write and may need repeated revision. Clear, simple questions can still require higher-order thinking or mathematical skills and test application of knowledge rather than report writing.
Selected-response formats differ in what they test:
| Format | Characteristics |
|---|---|
| One correct answer from alternatives | Realistic incorrect alternatives are difficult to prepare; the format may be used as a warm-up at the start of a selected-response test. |
| Matching | Tests relationships, including abbreviations and drug characteristics. |
| Multiple correct statements selected through a key | Tests depth of knowledge and is difficult to guess. |
| True–false | Very easy to prepare, but memorisation and guessing can compromise reliability. |
| Combined statements with a true–false key | Easy to prepare, tests understanding and application, and is difficult to guess. |
Multiple-choice objectives and anatomy
Multiple-choice questions are typically preferred in continuing education, and research links high scores with superior clinical performance. Their components are:
- Stem: the problem, often a clinical scenario.
- Lead-in: the question prompting a response.
- Options: the correct answer and distractors.
- Rationale: why each option is right or wrong.
- References: published evidence supporting the rationales.
The recommended general approach is to identify the audience and what it needs to know, focus on meaningful content rather than trivia, and emphasise common presentations of frequent diseases rather than rare disorders. The objective should focus on one concept — a recent advance, best practice or clinical-management principle — and begin with a measurable verb, such as “Identify the mammographic findings of suspicious masses”. Questions should be challenging and aligned with learning objectives, with tangential questions revised.
Clinical decision making needs application of knowledge, requiring higher cognitive objectives than recall. In 2016, one CME educator estimated that more than 90% of items were written at a low cognitive level. An analysis of 200 questions from twenty CME units in four German journals found too few questions addressing practical clinical use and actual decisions in every journal.
Stems, options and rationales
A clinical stem should follow its objective and present a common clinical situation clearly and succinctly. Wordiness and teaching a concept instead of asking a question can make it unfocused. The lead-in should be a direct question, define the problem, connect to the scenario and address one task such as diagnosis, testing or management. It should satisfy the cover-the-options rule: the candidate can answer without seeing the options.
Recommended answer-option guidelines are:
- Use simple, clear options.
- Order them logically.
- Make distractors plausible.
- Provide one best answer.
- Vary the correct answer’s position.
- Make options mutually exclusive.
- Keep their grammar parallel.
All options should belong to the same category: if one is a drug, the others should be drugs. Grammar errors can identify distractors; repeated wording from the lead-in or a longer, more detailed correct option can reveal the answer.
Negative wording adds cognitive burden: one analysis estimated twice the working-memory demand of a positive form, and potentially four times when negatives appear in both stem and options. Confusing chains of negations are a reason to revise an item.
The 2016 Deutsches Ärzteblatt style and format rules specify:
- Keep the stem to three sentences or fewer and answers short.
- Avoid negative words in stems and answers.
- Put the main idea in a clear, understandable stem rather than the answers.
- Avoid answer clues such as “always”, “never”, “only”, “exclusively” and “alone”.
- Provide only one correct answer.
- Do not use “all of the above”.
- Keep options homogeneous in content and grammar.
- Make options the same length.
- Avoid verbal associations between the stem and correct answer, such as “test” appearing in both when no other option uses it.
- Ensure one question does not supply information needed to answer another.
Advice differs on catch-all choices: the journal prohibits “all of the above”, whereas 2013 CME guidance recommended using both that option and “none of the above” sparingly.
Rationales are essential in self-assessment modules, though not always included in other activities. Client length requirements should be checked before extensive research and drafting. The recommended rationale checklist is:
- Explain every option.
- Focus on what the lead-in asked.
- Follow the options’ order.
- Include at least three published references supporting the data.
Quality control and evidence
High correct-answer rates can coexist with flawed items: almost 95% of Deutsches Ärzteblatt CME questions were answered correctly in 2008, yet an analysis found formal faults in 60% of its questions, the best performance among three journals. Research cited by a CME educator in 2016 indicated that almost half of test items had flaws.
By 2016, Deutsches Ärzteblatt required at least two questions in each set to address specific clinical situations, decisions or practical use, and each article opened with three learning goals reflected in the questions. Editors checked every question against the identified criticisms and revised it where necessary; a multispecialty team then tested quality, alignment with learning goals and answerability from the article alone, with further modification if needed. The questions were intended to show that the article had been read, rather than serve as an examination in the strict sense, and to be answerable by doctors outside its specialty. Participant enquiries usually exposed problems within a week, by which time the editorial office had usually logged over 1,500 participants. These early data and the proportion of correct answers revealed ambiguity or content errors.
A 2013 study comparing 2012 questions with 2006 data found formally correct Deutsches Ärzteblatt questions increased from 39% to 67% across the introduction of its quality rules. Its editors reported the best results among the three journals studied and attributed improvement to guidelines and better editorial procedures. A separate 2013 study compared surgical, non-surgical medical and interdisciplinary journals; the interdisciplinary journal performed best on all study items and was identified as Deutsches Ärzteblatt in personal communication reported by its editors.
Published questions can become templates for readers and commissioning clients who write new ones, giving item quality consequences beyond the immediate test.
Layout and computer delivery
A paper’s layout should help candidates concentrate on the problems and begin with familiar elements before progressing to questions requiring more thought. It should be accessible across linguistic and cultural backgrounds and prepared early enough to revise the typeset paper beyond proofreading. Bold emphasis is clearer than italics or handwritten script.
A 2007 recommendation called for five or six tasks per case-based question, never more than ten, paragraphs no longer than 240 words, and preferably no question needing more than fifteen minutes. Tasks should be brief, varied and reflect real practice, using familiar language, subjects, drugs and devices and few abbreviations. Humour and a game-like challenge can be appropriate even in advanced assessments, provided candidates remain interested without excessive time pressure. Fantasy cases may have a place in formative learning, but the same guidance opposed them in summative examinations, favouring topics and language from daily practice.
Computer assessment can use multimedia or interactive scenarios that reveal investigation results before the next question, while retaining practical, feasible layout. Interactive media should be used only when they improve the assessment, without inadvertently testing computer skills.