Quality by design in a trial protocol
Quality in a clinical trial is the absence of the errors that matter, not the absence of all error. A protocol built for quality therefore starts by deciding what quality means for its study: which data to prioritise, which factors are critical to quality and how the risks to them are rated, and how much movement each element can accept, stated in the protocol as a design space. It also settles when the quality management plan and the other plans around the protocol are written, and where the medical writer stands in the quality cycle between the protocol and the report.
What quality means in a clinical trial
No clinical trial is perfect, and therefore no data set is perfect. Accurate records alone do not make a quality trial, because a useful study design, well-defined endpoints, assurance of participant safety and assurance of unbiased observations must also be affirmed. For example, nobody would argue that a trial was adequate if it accurately recorded ingestion of an investigational product in an appropriate healthy population and the ingestion then led to the death of every participant. So quality does not mean freedom from error, nor the absence of audit findings.
Reference organisations define quality without requiring error-free data. In 1999 the Institute of Medicine's roundtable on research and development of drugs, biologics and medical devices defined quality as data strong enough to support conclusions and interpretations equivalent to those derived from error-free data, without requiring error-free data themselves. The Clinical Trials Transformation Initiative, set up by the FDA and Duke University in 2007 to increase the quality and efficiency of clinical trials, characterises quality as the ability to effectively answer the intended question about the benefits and the risks of a medical product or procedure while assuring protection of human subjects. It also defines quality in clinical trials as the absence of errors that matter. The European Medicines Agency's 2013 reflection paper on risk-based quality management defines quality as fitness for purpose. Clinical research generates information to support decision-making, so the information should be sufficient to support good decisions. The adequacy of that quality can be characterised by whether the decisions made would have been no different had the data and information been perfect.
Quality is not a section of the protocol, and it is not a department that starts to operate once the trial has started. Thinking about quality from the beginning of a protocol gives a better chance of ensuring it in what follows. Each project has to define what quality means in its own context, and nobody can say what it means in a given protocol until the writing of that protocol begins.
The protocol as the blueprint for quality
At the trial level the protocol is the blueprint for quality, because it is where the sponsor should prospectively identify the important risks to participant safety and data reliability. These risks may come from a variety of sources, so the protocol and its delivery should be tailored to eliminate or mitigate them.
One recent paper states that risk assessments are done after the project is finalised, but that the most value is gained by beginning them at the initial session before protocol development, though they may be revised and repeated during and after protocol completion. In one practitioner's view, this will ultimately affect how protocols are developed.
Data priorities
In large simple trials and in medical device registries, sufficiently large numbers of participants are more important in providing the power necessary to answer the question than careful attention to the quality of data. This sets a priority; it does not dismiss attention to data quality. No study is better than the quality of its data. The problems include missing data, erroneous, false and fabricated data, large variability, and long delays in data submission. Poor-quality data have many causes and avoiding them all is difficult, so the goal should be to limit their amount and thus their impact on the trial's findings. Many steps that optimise the collection of high-quality data can be taken during planning, which is where the protocol comes in.
Fastidious attention to all data is neither possible nor necessary. One approach is to decide in advance the degree of error to be tolerated for each type of data. Key data, and certain process information such as informed consent, should be as close to error-free as possible, while greater error rates may be tolerated for other data. Each data element should be examined for its importance in answering the question the trial poses. No template can decide which data elements are key, so the protocol writing process should include prioritising the key elements against the less important ones.
Trialists cannot include every outcome that might be nice to know. Each data element requires collection, processing and quality control and adds to the cost and overall burden of the trial.
Critical-to-quality factors and their risks
Defining quality as the absence of errors that matter calls for examining the objectives of the trial prospectively, defining the factors critical to meeting them, focusing effort on the errors that matter to the study's success, and acting to prevent important risks to those critical factors from negatively affecting outcomes.
Critical-to-quality factors are the elements of a trial design that have to go right for the study to succeed, where perhaps no errors, or only minimal variation from what was requested, can be accepted. More briefly, they are what is essential to the study. Identifying them means prioritising the elements that are critical to answering the question the trial poses. The factors then state what quality means in reaching that answer, for the different people involved and at the different phases and stages of the process.
From the critical-to-quality factors follow the critical data and the processes needed to obtain them. These are used to put adequate risk controls and mitigation strategies in place, which in turn influence the quality tolerance limits and the key risk indicators. For each factor, the risks involved are explored and rated, together with how to prevent them, how to monitor and detect them and how to correct them if they materialise, and the parameters for acceptable quality are then set.
The Clinical Trials Transformation Initiative's approach first identifies the critical-to-quality factors of a specific trial by looking at areas such as the protocol and feasibility. That look at feasibility belongs to protocol planning but needs to continue into study conduct. The initiative's second step discusses the potential risks related to each critical-to-quality factor and their impact on study quality. That requires knowing what quality means for each person and body in the trial process, including the hospital, the participant, the regulatory bodies and the ethics committees or institutional review boards. Once the risks are identified, ways to mitigate them are sought, along with how to identify an issue and react to it rapidly. So risk management cannot be a document that clinical operations or the quality department prepares at the start of the study; it has to be continually under review.
Risk-based monitoring and quality by design are two sides of the same coin. Organising monitoring around risk, as the EMA and the FDA were both encouraging sponsors to do, requires the risks to have been identified first, and quality by design supplies that prioritisation of risks. The initiative holds that quality by design helps organisations become prospectively and fully aware, throughout the life cycle, of the important areas that could jeopardise their ability to protect patients and to obtain reliable results.
The initiative's website offers resources on quality by design: slide decks available for public use, which it encourages others to use to push the idea forward, a principles document, and videos explaining quality by design and how to implement it in an organisation. The principles document was prepared because quality by design is a very different way of thinking. Rather than copying a template, the initiative calls for thinking more thoroughly and comprehensively about what quality means in each individual trial. The principles document aims to facilitate critical thinking at each point and each element of a trial, and so help planning for quality, by looking at events that can place participants at unnecessary risk or hinder efforts to use the resulting data to answer the scientific questions being addressed.
One forecast, made in 2022, was that the maturing understanding of risk and risk mitigation under ICH E6(R2), together with the fundamental shift toward quality by design in ICH E8(R1) and its likely reflection in ICH E6(R3), would have a material impact on medical writers' work. In it, more sophisticated risk management strategies, targeted enrolment and forecasting, and quality-focused study design would change the way studies are designed and analysed, and protocol design would combine a greater understanding of risk management and mitigation language with new methods for constructing studies around critical-to-quality factors.
The design space
In a protocol, quality by design takes the form of a design space. ICH defines a design space as a multidimensional combination and interaction of input variables and process parameters that have been demonstrated to provide assurance of quality. Applied to a protocol, a design space states what quality means for the trial and how much movement each element can accept, with zero errors in some variables and a defined tolerance for movement in others. Protocols already apply the idea in visit windows, and adaptive designs have used similar concepts for many years, but the idea needs to go further, to just about everything in the protocol that is not core. Once a design space has been authorised, movements within it are not considered a change requiring additional regulatory approval.
Many sponsors are anxious that regulators will not accept data produced within a design space. The FDA encourages sponsors to submit a quality management plan with the protocol, so that it can see how the design space has been defined and approve it prospectively. The FDA does not want to inspect a study three or four years after everything has happened; it wants to know that quality was adequately addressed from the very beginning.
In one protocol writer's experience, acceptance of design spaces by ethics committees and competent authorities in Europe has been patchy. Ethics committees were the harder case, because they are not always aware of, or up to date with, what is happening even in their own areas, and that is a risk. Even so, as long as the approach could be justified in the reply, the writer met no problem, and none at all with the regulatory authorities. Regulatory authorities are, in the writer's view, far more willing to innovate and to enter into dialogue than sponsors tend to think, so a sponsor in doubt should ask, and ask before submitting.
Tolerances stated in the protocol
Once the priorities and the processes for mitigating their risks are established, the acceptable variation and tolerance limits for the trial procedures should be set, bearing in mind the statistical design of the trial and the potential impact of different levels of variability on its power. The protocol should therefore go some way towards defining protocol deviations. A design space changes the idea of a deviation slightly, because movements within the design space are accepted. Thinking about what quality is in the trial also means thinking about what missing data are acceptable and what will not be queried. Many writers may be more comfortable leaving what will not be queried to the data management plan, and that is acceptable. But ethics committees and regulatory agencies neither review nor approve the data management plan, which leaves doubt about whether the approach really is acceptable. There is no real reason not to plan it in the protocol, and doing so already means thinking about what is core and what is not.
Stating acceptable missing data raises a legitimate concern, to which one protocol writer had no answer: a site told that some data may be missing may simply not bother collecting it. Identifying acceptable missing data should also raise the question of why those data are being collected at all.
For one protocol writer, concomitant medications are the clearest example of where a design space applies, though others' experience may differ. In that writer's years in clinical operations, collecting and documenting every medication a participant takes needed a disproportionate amount of time, both for data entry at hospitals and for monitoring by the clinical research associates. By the time the clinical study report is written, most of these medications turn out to be irrelevant because they were not related to adverse events, which questions the usefulness of collecting so much information. A team may decide that it needs to know what the participant is taking but not the same level of detail about doses; in some studies that detail will matter, and which ones cannot be said in general. Applied to concomitant medications, a design space starts from the fact that some matter more than others, and the critical element is whether a medication is related to an adverse event. The remaining risk, that an adverse event goes unrecognised, is then addressed as a risk, and the design space still applies.
Spelling is increasingly an issue in multinational trials, where English spelling can be difficult. Where the meaning is understood, a self-evident correction can be made instead of a query, because querying is one of the big wastes of resources in clinical trials.
The quality management plan
Planning for quality means focusing on what matters and defining quality at the outset, developing a quality management plan, and possibly submitting it with the protocol for regulatory review, for the added certainty that the authorities approve of how risks are categorised and handled. It also means assessing the protocol continually through its lifetime against the set parameters, training and analysing procedures continually, and finally reporting the quality management approach. The FDA and the EMA are not looking for error-free data and are not trying to trip sponsors up; they are more interested in what went wrong and how the sponsor reacted to it.
How the quality management plan corresponds to the monitoring plan, the query management plan and medical review depends a lot on the organisation. They will overlap a good deal, which can create confusion, because the work needs to be interdepartmental. Clinical operations needs to work very closely with the statistics department and the quality management department already at this stage. The quality management plan should be top-level enough for a regulatory body to see and approve it, with the details left to the monitoring plan. Depending on the size of the trial the two can be the same document, but the quality management plan needs to be ready more or less at the same time as the protocol, and where that is not possible they have to be separate documents. The quality management plan is yet another document, and people may despair at that, but done properly it makes the documents that follow easier. Asking regulators before submitting is a good reason to wait until the quality management plan is finalised before finalising the protocol, and to submit both at the same time.
In a modular arrangement in which contract research organisations and freelancers take part in trial planning, one protocol writer holds that the sponsor should be responsible for the quality management plan, because a CRO usually arrives too late in the process to be involved in that planning. Many CROs could do the work, but the plan is too critical to subcontract or to leave in someone else's hands. The sponsor also needs to be active in monitoring how the CRO and the sites are working, and it is probably the only body logically present through the whole life cycle of the study.
One innovation that has long appealed to one protocol writer is involving participants in quality control, in which a participant looks at the data recorded about them and says whether they have anything to add. The writer grants that there are a dozen reasons why this would be unfeasible and complicated, but holds that there must be at least one or two good reasons for it, since quality happens at every stage of a trial.
Planning time, templates and checklists
In one protocol writer's experience, documents such as the statistical analysis plan and the monitoring plan often rely, perhaps too heavily, on templates. By the time the monitoring plan is written, its author is too far from the protocol design and not familiar enough with the objectives of the trial to go beyond the standard, so they put in all the usual content without prioritising what really matters.
That writer holds that efforts are needed to involve the people responsible for those plans earlier, though in practice the timings are likely to make that very difficult. Protocol writing may then take longer, which is not necessarily bad, because the days a protocol amendment adds to a trial are better spent before the trial starts, honing plans such as the monitoring plan and the statistical plan instead of rushing the first patient in. This needs a change of mentality, so that the success or good conduct of a trial is not judged on the first patient in. Starting a trial with the first dosing is understandably important to sponsors, but a sponsor may do better to look for other factors that indicate a trial's quality or success, to avoid metrics such as first patient in, and to define in the quality management plan what quality will mean. Rather than rushing to finish the protocol, it is worth waiting for a better protocol and a better quality management plan.
Standard operating procedures are essential and the value of checklists is not in question; the problem is relying on them too much and without thought. Checklists and templates are a starting point that has to be adapted, and the person who adapts them is often too far into the planning phase to see that adaptation is needed, so the whole template goes in without prioritising. Templates are not only company ones: different groups are doing a lot of work on example protocol content. The key is to move away from unthinking use of such templates and checklists and to analyse, during project planning, what is appropriate for answering each trial's question, accepting that everything on a checklist is important but some things are more important than others.
When a company builds a learning curve from its own previous studies to prepare protocols, checklists, metrics, their implementation in standard operating procedures and a quality-by-design toolkit can all play a part, depending on the sponsor and its history. In one protocol writer's view, an oncology sponsor has certain advantages here: oncology studies are very similar with little variation, early studies turn on dose-limiting toxicities where the priority is clearly to find the right dose, and much of the work rests on precedent, so checklists are important there. Metrics are important but have to be the right ones.
The medical writer in the quality cycle
The Clinical Trials Transformation Initiative emphasises the plan-do-check-act cycle, a typical quality flow chart found in basically all project-management guides. In one protocol writer's view, the critical point for medical writers is to see themselves inside that cycle, not only at its beginning and its end, the protocol and the clinical study report. Without stepping into the responsibilities of clinical operations or the quality department, writers can have a role through the lifetime of every trial, at least observing as the cycle continues.
Writers may still have a role in study conduct, because they should know whether the protocol they wrote turned out to be feasible as planned. Staying in contact with clinical operations shows where the deviations are happening, where data are missing and what the recruitment problems are, and that feeds back into the protocol design of future studies.
Contract research organisations and freelancers are often invited too late to take full part in the planning, but they can still help by talking about the approach. In one practitioner's observation, many companies are not aware of the Clinical Trials Transformation Initiative and still include a lot of non-core content in their protocols. The literature helps show that the approach is what the major regulatory bodies globally are pushing for, not one contractor's opinion. From there a contractor can start with very small steps, such as bringing a design space into small areas of what is provided to sites and monitors. Where the protocol itself is out of reach, the approach can at least go into a plan; this carries a small risk, because the plan has not been submitted to the ethics committee and the regulatory authorities, but the intention is documented.