Testing with real readers
Testing with intended readers shows whether information can be found, understood and used. It combines observation of performance with readers' feedback, followed by revision and further testing.
Performance and feedback
Even expert writers need input from members of the public to identify problems in information intended for them. User testing can help writers learn more about intended audiences and create engaging documents. It examines how information performs, rather than assessing its content alone, by asking potential users to find and explain key information in individual interviews. It can apply to documents, media and other communication intended for patients or nonmedical and nonscientific audiences.
Questions about finding and understanding come first; participants then describe what they liked, disliked or would improve. Their opinions are therefore informed by having just used the document, and the combined quantitative and qualitative findings can reveal weaknesses expert writers did not anticipate.
Testing alone identifies problems; resolving them requires expertise in writing and information design. Time should be budgeted for feedback on an early draft, revision and a second round to find new problems introduced by changes. Feedback can be formal, as in focus groups, or informal, as in individual discussions.
Participants and interview conditions
Participants should be potential users with varied reading abilities and ages who do not already know the topic well enough to answer without using the information. Expert patients can contribute during early development but may be unsuitable for testing whether finished information works for people encountering a treatment or health issue for the first time. One user-test group included a retired cleaner, an unemployed person, a stand-up comedian and a bus driver.
Broader patient-material review guidance also recommends audience representatives or patient advocacy groups to assess whether key information can be found and understood. Individual interviews can suit sensitive topics or people whose health, location or circumstances make group participation difficult. Ethical, logistical and budget requirements need to be planned for the material and therapeutic area.
At the beginning of every interview, participants should be told that the information is being tested, not them, and that finding weaknesses is the purpose. Informal home checks have included reading text to relatives or children to see whether they understand it.
Iterative procedures
An eight-step process for a typical health leaflet is:
- Identify the document's key points, usually 12–15 for an average health leaflet.
- Select potential users with varied reading abilities and ages.
- Write a questionnaire testing finding and understanding of each point and gathering general views.
- Pilot it with two or three participants.
- Administer it individually to a round of ten participants.
- Analyse quantitative and qualitative data for strengths and weaknesses.
- Revise problematic parts using good information-writing and design practice.
- Test again with a new round of ten participants.
A medicine-leaflet procedure described in 2012 has nine steps:
- Select fifteen key points relevant to safe and effective medicine use.
- Design and pilot a questionnaire testing finding each point and understanding it through the participant's own words or a scenario response.
- Recruit ten or twenty people from the target patient group.
- Interview each individually using the leaflet; for each point, the target is that 90% find it and 90% of those demonstrate understanding.
- Finish with open questions about likes and dislikes; some authorities, including MHRA, place as much weight on qualitative as quantitative results.
- Analyse difficult-to-find or misunderstood answers and general comments.
- Revise using research evidence and good writing and design practice.
- Test with another ten or twenty people.
- Analyse again and repeat if problems remain.
Not all feedback can be adopted, and some may contradict other feedback. A writer receiving a sample document or template can ask whether it has been user tested.
Diagnostic sample sizes
Small user-test groups can identify problems in written medicines information. In Luto's reported experience of more than 20,000 individual interviews by 2015, a significant problem generally appeared within the first two or three interviews. This is diagnostic testing to find weaknesses, rather than a requirement for a large representative sample.
A separate field-testing recommendation is that samples as small as thirty can often be adequate even for a nationally distributed patient document; serious problems almost always appeared in the first ten readers' responses in that experience.
Formats and pictorial information
User testing can be used for large-print leaflets, audio and video, screen-based information, medical-device instructions and health-service materials as well as manufacturers' leaflets.
Pictograms should be tested because readers can take a meaning different from the intended one, not merely fail to understand. A medicine-pack pictogram intended to mean “Keep out of the reach of children” was interpreted as “Do not give to children”, “Do not use if you are pregnant” and “This medicine is a contraceptive”.
User testing and usability testing can complement each other. Usability testing examines how usable instructions, websites or apps are; a device with a package leaflet and instructions for use can have a hybrid test combining the techniques.
Criteria-based assessment
A suitability checklist and testing with intended patients are both desirable; reviewing a criteria list is more convenient and less time-consuming. A checklist can examine several aspects and pinpoint improvements, while feedback from representative readers is the gold standard for what they find understandable, useful and relevant.
The Suitability Assessment of Materials (SAM) has twenty-two criteria-based factors under six headings:
- Content.
- Literacy demand.
- Graphics.
- Layout and typography.
- Learning stimulation and motivation.
- Cultural suitability.
In 2015, CDC's Clear Communication Index supported development and assessment of public-facing material and referenced the Federal Plain Language Guidelines. It covered twenty items scored zero or one, converted to an overall score from zero to one hundred, with higher scores indicating greater clarity. Its scope included behaviour recommendations, numbers and explanations of risk.
Feedback after publication
Organisations should offer lay audiences a way to evaluate published materials. Open comments sections can produce varied responses that are difficult to assess, whereas surveys, interviews and easy-to-use feedback systems can gather measurable information about understandability and may help address mismatches with readers' literacy.