Testing a lay summary with its readers
Whether a lay summary of trial results works is shown by testing it with the people it is written for. Testing means choosing which patients suit which task and running a patient panel review, whose feedback shows where a summary needs improving, and it costs time and money and has pitfalls of its own.
Why test with readers
Audience testing ensures that trial summary materials are effective.
A lay summary is written for the general public or for trial participants, who are not familiar with medical terminology, so it needs non-technical terms. To avoid boring its readers, it must be neither too long nor too simple, and that balance can best be achieved by having patients, members of the general public or patient advocacy groups take part in preparing the summary through user testing. Getting feedback from the intended audience is essential for a writer, and input from all stakeholders, especially patients and the public, on how information about medicine and clinical research is communicated is crucial.
Reaching out to the audience, through user testing or through engagement with patient advocacy groups, is crucial to understanding what readers really want and need. Because writing a lay summary differs from traditional medical and regulatory writing and has a different audience, it is important to take into account the perspectives and opinions of the public and the patients it is written for.
Incorporating patient input is essential to a summary's relevance and usability, and to its meeting the informational needs and preferences of diverse patient populations. Numerous studies have highlighted the benefits of patient input into healthcare materials, the advice of regulatory agencies and government bodies echoes them, and a systematic review underscored the positive effect of patient engagement on healthcare communication and decision-making. Patients bring perspectives and insights that can enrich a summary's content and readability, making it more relatable and user-friendly. Involving them in its development can help identify the key concepts, terminology and formatting preferences that resonate with its intended readers. Involving them in reviewing and validating it can improve its accuracy, its relevance and its overall effect on patients' decision-making. By prioritising patient input, stakeholders can foster a culture of patient-centred communication and help patients make informed choices about their healthcare.
Lay summary development cannot progress without the patient, and patients need to be considered throughout as standard operating procedures and templates are developed. Patients are the summary's primary end users, and involving them is key to the summary becoming less of a box-ticking exercise and more one of producing good-quality, relevant health information that can help people make better decisions about their healthcare.
Having its end users assess a lay summary's readability is an obvious choice, is feasible and can be highly successful, as it was for the lay summaries of the Newcastle Cognitive Function after Stroke Cohort Study. At one organisation that writes plain language deliverables, almost every deliverable is user tested by a review panel of patients, patient advocates and members of the public. The testing gives its writers confidence that the deliverables are of high quality, and it has built up a body of knowledge and best practice in communicating about trials clearly and humanely.
The EU Clinical Trials Regulation's guidelines encourage involving patients, patient representatives, advocates and members of the public in developing and reviewing the summary, to assess comprehension and the value of the information it gives. The guidelines recommend including the target audience in the review of the summary, but the regulations do not mandate patient involvement at any stage.
Which patients to involve, and for which task
The lay summary is an opportunity to involve patients and patient organisations, if not in the routine process then in developing an appropriate template. In a position paper, the European Patients' Forum suggested ensuring the layness of results summaries by including patients or patient representatives in their review, in a process yet to be defined. Implementing this would be logistically challenging, because appropriate patient representatives would be hard to find for all the diseases studied in clinical trials. Many guidance documents for lay summaries suggest involving patients in producing them but do not set out clearly how, which shows a need for best practice to be shared and a standard to be set. The Roadmap Initiative to Good Lay Summary Practices could be a step towards one.
Patients are not all the same: each group of patients can bring different value and expertise to the process, and so may suit involvement only at certain points. Patients by experience and expert patients are terms now widely used in healthcare settings, and beside them the so-called pro-patient has emerged: a patient who looks at overarching and systemic issues that cross patient communities, is often well connected with stakeholders and has a high level of health literacy, even outside their own disease area. Patients could potentially be involved by reviewing a summary for the relevance of its content and for its readability. A patient by experience could review it for readability and for how well a lay reader would understand it. Expert patients and pro-patients are better suited to assess whether its content is relevant, because that needs a much higher level of health literacy and possibly the ability to understand the trial documents the summary was produced from. Involving patients in summaries outside their own disease area can also help, because patients may become too expert or too comfortable in their own condition to read the document from a lay perspective.
Running a patient panel review
Because a lay summary is written for the general public and patients, a patient panel review is strongly recommended to make sure the summary is clear and understood by its readers, and the EU Clinical Trials Regulation's guidelines recommend one. The review is also called user testing, readability testing or patient advocacy review. It is the best way to test a summary's effectiveness, and it solicits feedback from people familiar with the condition the summary is written for.
The criteria for being on a panel are to be familiar with the condition the summary is written for, whether as a patient, caregiver, immediate family member or close friend; to be enthusiastic about research, though a panelist need not know current research or understand complex scientific terminology; and to be able to listen to others and express one's own views in discussion. Panelists are recruited in the country the master summary is written for, so that, for instance, panelists for a US English summary are from the United States, and they are found through online forums, patient advocacy groups and other networks for the condition.
Once panelists agree to take part, they are given a written confidentiality agreement and a guidance document with instructions on how to join the discussion, and they are told that the purpose is feedback on the clarity and readability of the summary, not promotion of any drug.
The review is interview-style, a structured two-way discussion that draws out solicited and unsolicited feedback. Panelists are asked for their opinion on points of specific interest to the clinical review team, are encouraged to give general feedback on every section, especially any part that is not easy to understand, and are asked to explain the results in their own words to confirm that the intended message comes through. Afterwards the writer compiles a feedback report for the clinical review team, with recommendations on how the feedback could be addressed, and the summary is updated once the recommendations are discussed and agreed with that team.
What the panels find
Over three years, patient panel feedback on one team's summaries identified these improvement areas for a summary's readability and comprehension:
- Find the jargon. The EU guidance suggests that the summary be written in everyday language throughout, and without testing it with its readers a writer can only assume it is understandable. Simplify or define the technical terms the panel identifies, so that a non-scientific reader can understand them, as in replacing "safety assessments" with "health check-up".
- Make the first page clear. It sets the tone, helping readers understand what the summary is about and what they will get from reading it. Give a clear, informative title that describes the condition simply, as in "A clinical trial to find out if study drug can improve presbyopia, the inability to see objects up close", rather than one that names presbyopia without explaining it. Add an introduction that says what the document contains and why it was written, as in one saying that the summary shares the trial's results with the public in simple language and describes why the study was needed, how it was done and the results. Tell the reader that the summary shows the results of one trial only and that broad interpretations of efficacy and safety, or health decisions, should not rest on it.
- Say why a measure matters. When explaining technical concepts such as the mechanism of action, define terms that may not be well known outside the industry. Where the drug affects a protein linked to the disease, or an enzyme or protein is measured because its levels reflect whether the drug works, add text that connects the dots on why the measure is important and what it implies. Readers find this informative and helpful, and prefer it to a summary simplified so far that the rationale behind the study and its assessments is unclear. Biomarker assessments interest patients too, but their results are technical, so lay out in simple terms what each assessment means, letting readers draw conclusions from the data themselves rather than be told them.
- Use visuals. Diagrams, graphs and infographics have been shown to greatly support the understanding of complex data, and panels confirmed that visuals are preferred to an all-text format. Present symptom-severity scales and categories of information as images rather than text. Make results and figures stand alone, restating whether higher or lower values on a graph mean a better or worse outcome for patients: although readers could piece this together from the introduction and the study design, standalone results and figures have been found crucial to readability. Use the same colour for each treatment group throughout, which panelists feel helps them connect the study design with the efficacy and safety results. Balance infographics that aid understanding with text. Chosen carelessly or relied on too heavily, visuals can confuse, because they are open to interpretation.
- Describe the study design, which gives the results their context, specifically enough that readers understand what assessments were done, how the drug was given, any effect on other medicines being taken and the time commitment required. Panel discussion suggests that readers are interested in the route of administration, the effect on other medicines and the timelines of each period of the trial, and all of these can go into the design description and a simple design figure.
- Think beyond layout and word choice, and customise the summary to the condition or therapeutic area it is written for, so that its information is easy to access. For studies of eyesight loss, use larger type and, where possible, an image of what the loss is like for someone with the condition. For children, add infographics and images, and keep more white space, as children can be easily overwhelmed by paragraphs of text. Keep in mind that cultural, geographical and individual experience may affect how readers interpret the information, and that rescue medicines and standard-of-care drugs may be marketed under different names in different countries, as where a summary named a rescue inhaler both "salbutamol" and "albuterol" after panel feedback.
- Be respectful. Panels regularly appreciate language that respects patients, so choose terminology that sets the summary apart from documents that focus on the experimental nature of trials, and use empowering language so that participants feel respected: "participants" rather than "patients" or "subjects", "condition" rather than "disease", and "treatment did not benefit the participants" rather than "participants failed the treatment". Take particular care with conditions that carry social stigma, such as mental illness. Include a thank-you acknowledging the time and effort of participants, without whom clinical trials would not be possible, as in thanking them for taking part and saying that their time and commitment has helped bring better treatments to patients a step closer. Panels often appreciate it, and the EU guidelines also recommend it.
The feedback from these panels agreed with the recommendations of the EU guidelines and with health literacy principles.
Cost, time and the pitfalls of testing
Cost and the possible need to train external collaborators can at first put organisations off involving patients. Available toolkits can make the training easier. One, co-produced by Envision Pharma and multiple stakeholders, gives patient reviewers a template of the key information they need to put the summary in context, which makes the review easier. The cost of involving patients should be set against the value they add to a high-quality, fit-for-purpose document. In future, that cost needs to be treated as integral and necessary to the budget, as the cost of involving a medical writer is.
Writers experienced in user testing hold that it has challenges of its own. One is over-correcting: where one reviewer dislikes how a concept is explained while nine like it as it is and say nothing, what is not broken should not be fixed. Another is feedback too general to help, such as a comment that says only "this is good". The best feedback takes time and active engagement from reviewers, and finding reviewers willing to give their time to feedback that improves the deliverable is no easy task.
Testing also takes extra time. Its feedback can be valuable, but plain-language results summaries can have strict posting deadlines, and fitting meaningful testing into such a timeline can be a real challenge. One strategy is to start the project as early as possible: to keep time for testing, the Center for Information and Study on Clinical Research Participation (CISCRP) has started drafting summaries from the tables, figures and listings rather than from the clinical study report, which sponsors can often take much longer to finalise. Where the project cannot start earlier, recruiting the testers far in advance, with reviewers in place and clear deadlines and expectations communicated, is very helpful.