Quality tolerance limits
A quality tolerance limit is a bound a sponsor sets before a trial starts on how far the trial's conduct may depart from what it expects. ICH E6 good clinical practice asks for such limits. For a medical writer they raise four questions: where the limits are set and on what, which breaches section 9.6 of the clinical study report summarises and how much it says about them, what root cause analysis stands behind that summary, and how the limits across a submission support its good clinical practice statement.
What a quality tolerance limit is
A quality tolerance limit is predefined: the sponsor states up front how much deviation it can accept on a measurable factor of a trial, and the trial is then judged on whether it stayed within its limits rather than on whether it was perfect.
The total number of errors in a trial, such as inclusion and exclusion protocol violations, splits into two components: random errors and systematic errors. The regulators define the systematic ones as the errors that matter. A protocol violation can be significant, in that it may include or exclude a patient, without being systematic, because it can be random. The sponsor defines the expectation, the number of random errors it expects, and draws upper and lower quality tolerance limits around that expectation on a bell curve.
Limits drawn that way give the sponsor some of the systematic errors for free. Over time, systematic errors are expected to decrease while random errors persist, since trials involve patients, sites and processes in which individuals make mistakes. Each study is its own plan-do-study-act cycle, and quality improves when the next study does not repeat the same error.
The regulators see a constant level of issues across all companies, sometimes a little higher and sometimes a little lower, and they want to start eliminating part of it. They cannot eliminate all of it, because the random part will remain. The industry has relied on the randomised trial alone: it compares active against placebo or against another active compound, accepts the efficacy difference, and takes no real ownership of quality. The regulators want sponsors to start predicting what their quality will be, and one experienced practitioner holds that this can be done and has been.
Deming's principle is to accept defects and not waste effort on zero-tolerance goals, and that wasted effort is what the industry has been making. Its plans are written with perfection in mind. No plan says the sponsor will do 88% of something; 100% is always assumed, which paints a target on the sponsor's back for inspectors' findings. The purpose of introducing quality tolerance limits in ICH E6(R2) was to give sponsors a gift: they could now start to make errors. ICH E8 states that perfection is not achievable in all things. That does not mean not trying to be good, but working on what is actually done. The whole purpose of ICH E8 is to move away from perfection, and quality tolerance limits are the basis of that move.
The relationship with contract research organisations shows the same habit. The contract research organisation may write its plan to indicate perfection, and the sponsor expects that perfection, when both parties should know perfection is not achievable. Quality should rest on defined and measured quality instead, and the limits should be used throughout the organisation rather than as a tick box.
Where the method comes from
Tolerance limits are far older than their place in good clinical practice. Walter Shewhart of Bell Laboratories proposed the concept internally on 17 May 1924 and published it in the 1930s, and it is generally known as statistical quality control. Every methodology that followed, up to total quality management in 1985 and Six Sigma in 1986, uses tolerance limits, but the pharmaceutical industry diverged and never used them.
Deming was working on his own proposals alongside Shewhart, dropped them when he saw Shewhart's method, and refined that method instead. One experienced practitioner prefers Deming's methods, holding them simple enough for every person in an organisation to use. Joseph Juran, regarded as the father of quality by design, worked for Western Electric, a Bell Laboratories subsidiary, and was trained by Shewhart in his early days. Juran's method and Deming's plan-do-study-act cycle lead to the same point. So the methods the industry struggles to move to are not new.
Applied to clinical development, each stage of the plan-do-study-act cycle has its own work. Plan is design diligence and a good protocol, and also understanding what has gone wrong before. In Do, the industry is not always as good as it should be. Study is where analytics should be used to review the quality tolerance limits and to examine the true root cause, rather than simply saying the cause was training. Act is responding to what was found by adapting the tolerance limits and changing the protocols, so that the same problems do not keep coming up, instead of writing the next protocol without considering any of it.
Why the regulators asked for them
At a Clinical Trials Transformation Initiative meeting in August 2011, the regulatory bodies set out what they wanted the industry to do about risk-based monitoring, and with it their concerns about the industry's quality.
One concern was a lack of trust in the good clinical practice statement in submissions. A submission typically says that the sponsor conducted 30 audits and therefore concludes that all studies were run to ICH good clinical practice. That conclusion is odd when more than 95% of audits produce good clinical practice findings, and when inspecting any two sites always produces findings too. The regulators were upset at this lack of transparency, and wanted defined quality and transparency about it.
Another was that quality was arriving by accident. An inspector at the BfArM who is also the EMA's lead inspector for ICH good clinical practice wanted quality by design rather than quality by accident. All too often she found one team in a company doing a good job and another not, supposedly under the same standard operating procedure. She wanted quality throughout the organisation, and felt that the industry was siloed.
Further concerns were oversight and root cause analysis. Oversight was lacking not only over contract research organisations but inside sponsors' own organisations. Root cause analysis was poor: if more than about 30% of root causes come back as a lack of training, with retraining the site or internal staff as the remedy, the analysis is probably not being done correctly. The head of the EMA's inspection branch called treating 100% source data verification as a quality solution stupidity. Overall, the regulators thought the industry was wasting its resources and not bringing enough new medicines to market. One experienced practitioner from the industry came to hold that the regulators were right. Already in 2011 the head of the EMA's inspection branch had pointed the way, in a presentation on learning from quality by design in manufacturing that included quality tolerance limits as an essential part of quality by design.
ICH E6 was revised in November 2016, and the revision, ICH E6(R2), was about risk-based quality management and introduced quality tolerance limits into good clinical practice. Almost as soon as that revision was done, the good clinical practice renovation was announced, in January 2017, and the principles of ICH E6(R3) followed. ICH E8 came out in draft in May 2019 and brought quality by design. ICH E6 governs the conduct of trials, and ICH E8 their design: the protocol and what is to be done in the study. The full impact of quality tolerance limits can only be realised with ICH E8. The whole renovation is about quality by design, and quality by design depends completely on risk-based quality management and on quality tolerance limits, so the guidances are stepping stones to one goal.
What ICH E6 asks for
Section 5.0.4 of ICH E6(R2) states the requirement briefly: predefined quality tolerance limits should be established, taking into consideration the medical and statistical characteristics of the variables as well as the statistical design of the trial. Predefined means telling the regulators up front, and the medical characteristics count as well as the statistical ones.
The design matters because a trial's design can change what the study reports. Of two identical trials, one run in China and one in the United States, the one in the United States will show more protocol violations. The reason is not that sites in the United States complain more. In China each serious adverse event has to be reported on five different forms, to the company, to the regulatory body and to the hospital, so the reports tend to get lost and the doctors do not bother to report them. That is a systematic issue, not a sign that the Chinese sites perform better.
The statistical design also bears on the trial's power. Once the priorities and the mitigation of risks are in place, the acceptable variation or tolerance limits for the trial's procedures should be established bearing in mind the statistical design of the trial and the potential impact of different levels of variability on its power.
The guidance directs attention to patient safety, the reliability of trial results and data integrity, and says that detecting a deviation from the predefined limits should trigger an evaluation to determine whether action is needed. One experienced practitioner calls this a very generous interpretation of tolerance limits. In a manufacturing plant, exceeding a tolerance limit means rejecting the product, whereas here all that is required is root cause analysis and following through on it.
What a limit is set on
A limit is set on something measurable, at the end of a chain that starts at the target product profile, which should record what the patient wants from the product. In practice most of what sits in a target product profile is about manufacturing, such as that the tablet should be white, of a certain size and contain so much drug, and few entries record what the patient wants, such as fewer side effects or once-daily rather than twice-daily administration. The profile should lead to the critical quality attributes, and not all attributes are critical to the quality of the product. From those come the critical-to-quality factors, the term used both by the Clinical Trials Transformation Initiative and by ICH E8(R1) for quality by design. A critical-to-quality factor is the measurable part of a critical quality attribute, and the quality tolerance limit sits on top of the factor, stating how much deviation from it can be accepted.
In a worked example, the target product profile records that patients want extended dosing from a product currently given twice a day, so the critical quality attribute is once-a-day dosing. The critical-to-quality factor is what the sponsor measures internally to know whether it is succeeding: drug levels at 24 hours in early phase studies, and demonstrated efficacy in later phases. The quality tolerance limit is how far the study may deviate from those acceptable limits.
ICH E8 and the Clinical Trials Transformation Initiative call everything they list a critical-to-quality factor, when the list is a mixture of attributes and factors. One experienced practitioner does not expect them ever to give the specification, but their critical quality attributes can be taken as the specification, and quality tolerance limits still have to be defined against them. The critical-to-quality groupings in ICH E8 should be broken up on that basis, since some entries are critical quality attributes but not critical-to-quality factors. ICH E8 rightly calls the types of controls critical, but the type of controls is an attribute of the product and the outcome of a process for deciding it, not a measurable factor. Endpoints are measured at reporting, but the process of selecting them is not a measurable part. Wherever there is a measurable critical-to-quality factor there should be a quality tolerance limit to show whether it is acceptable, which is probably a good deal more than companies implement.
Other critical quality attributes should come from the sponsor's other customers, so deriving them maps out the customers: besides the patient, the regulatory body and perhaps health economic bodies and, in the United States, insurance companies, any of which may have its own requirements on what it is given.
Setting the limits
There is no requirement to put quality tolerance limits in the protocol, and one experienced practitioner knew of only one company that did so. He holds that putting them there is the gold standard, because it shows that the quality of the study was thought about before it started. Many companies put the limits in their monitoring documentation instead, and the monitoring plan should then go to the regulators with the protocol, although in practice it sometimes follows later. Judging a trial by whether it stayed within its limits rather than by perfection puts much more emphasis on the monitoring plan. The limits should also go to the ethics committees or institutional review boards, which have a responsibility for monitoring and should know what levels a sponsor that is not seeking perfection is aiming at.
Very few companies do what the regulators say and tell them beforehand what their quality tolerance limits are. The rest are not predefining their limits, and that puts them at risk.
Who leads the definition varies, among statistics, data management and the centralised monitoring group, but it should be a whole-team activity, with the medical monitors and the on-site monitors involved. Every part of the team also has to understand what not expecting perfection means downstream. If not every query will be followed up, some queries stay unanswered, unanswered queries become missing data, and someone has to know what that does to the tables and listings and how it will be explained.
Most companies use the limits for inclusion and exclusion protocol violations, for investigational medicinal product violations, and for loss to follow-up. Two published starting points extend them. A paper by Arthur Hecht in German Medical Science applies them to the trial master file, since nobody has a perfect trial master file. The TransCelerate paper on source data verification as a quality measure supports limits on transcription errors from source data to the case report form. The regulators would also want limits on adverse events, and probably on ten other areas.
Adverse events are a gap. One experienced practitioner knew of only one company with a critical quality factor on the number of adverse events it expected, yet the design of a study can affect adverse event reporting significantly. Seeing participants every 12 weeks rather than every 2 loses around 60% of the non-serious events, a figure found in both respiratory and cardiovascular programmes. Sponsors do not explain this to the regulators, so a quality tolerance limit is needed to state what is measured and how good it will be. A trial measured that way is not perfect, and that is acceptable, since it can be fit for purpose without being perfect.
One experienced practitioner holds that limits should not be changed once set. TransCelerate's documentation allows them to be revisited and changed during a study, but the industry's reputation is poor enough without changing its quality part way through. Exceeding a limit costs only an explanation of the result, which is little punishment. Revising the quality mid-study is likened to an airline announcing over the tannoy, halfway through a flight, that 20% of its flights now crash.
What measuring quality buys
Quality tolerance limits give more than the defined quality reported in section 9.6. They give continuous quality improvement, because quality is now measured. They give compliance at entry rather than at some later point in the study. They reduce the burden on the company and on the site, and they will reduce inspection issues, because a limit that has not been breached leaves no reason for an inspection finding.
Companies have struggled for years with knowledge management. It has two parts: the statistical and medical knowledge a study needs, and the structural or management knowledge of what to do in a given situation. Both can be built into tolerance limits. Trial simulation can be helped too, as in the work Roche and Amgen were doing with the FDA on historical controls for phase 1 studies, from which trial simulation is done. Bringing in disparate data sources, such as real-world data, is also helped by quality tolerance limits and the modelling of data.
Which deviations the clinical study report summarises
ICH says the sponsor should describe the quality management approach implemented in the trial, and summarise important deviations from the predefined quality tolerance limits and the remedial actions taken, in the clinical study report. The operative word is important. Not every deviation has to be reported, and the regulators are not expecting to see everything: they want to know the important ones and what is being done about them, so that they know the sponsor's process is right.
Importance is decided by what the root cause analysis found. A breach that turns out to be random error, with no more than about one issue per site and no overall problem with the protocol, can be judged not important: the deviation was simply higher than expected, the expectation was wrong, and it does not need writing up. In future a study may carry 15 to 20 quality tolerance limits, and with limits at two standard deviations one of them will exceed its limit by chance. A limit that was not breached is not reported.
Breaches belong in section 9.6, while what the sponsor planned to do can go in the monitoring plan, the central monitoring plan or the protocol. The detail of the deviations can go in the report body or in a dedicated appendix, and an account too long for the body goes in its own appendix. The source data behind the limits need not go into an appendix, since an inspection can ask for them if the regulators are concerned.
Reporting the audits in section 9.6 is not enough to comply with ICH E6(R2). Auditors are not consistent: what one audit finds and what another finds do not agree, the outcome depends heavily on the individual auditor, and repeated attempts to make audits consistent have failed. ICH E8 says sponsors should be less reliant on audits, inspections, retrospective document checking and monitoring, which are the traditional methods of quality measurement.
Journals may in future ask, before publishing a paper, whether quality tolerance limits were used and what the results were, because the limits are part of good clinical practice.
The root cause analysis behind the report
A writer reporting the limits should ask whether the full root cause analysis has been undertaken. The writer has to describe that analysis, so the writer has to ensure it was done sufficiently, and insufficient root cause analysis is one of the biggest failure points affecting the industry. The analysis is the writer's source data: the writer should be given the quality tolerance limits and the detailed root cause analysis behind them.
All too often root cause analysis is pushed down to individual cases, with the on-site monitor asked to find the cause of each non-compliance. That is not sufficient. The analysis should happen at the end of the process, once all the non-compliances are in. It asks which protocol violations breached a tolerance limit, then analyses those again for patterns, including whether several fall in the same sites. That is where random and systematic processes separate.
On one study there were around 190 protocol violations, and at every site the clinical research associate had recorded the cause as training or as use of the wrong scale. Once the wrong scale was identified, the sites were asked to use the correct one. The end-of-study analysis then found that the data changes all moved in one direction: the sites had been pushing patients into the study, which was not a random effect. So rather than accepting isolated case-by-case explanations, the analysis should wait for the end of the study and look for the pattern.
Companies often have a committee that reviews every non-compliance, and it misplaces the same effort. On a large study, non-compliances should be weighed as a percentage, because trials involve humans, and the effort belongs on the systematic problems.
Reporting fraud and misconduct
Some of the systematic effects a report describes are fraud and misconduct, and these are difficult to explain, but the report should take the bull by the horns and wade into them. On one very large study, quality tolerance limits were run and the report carried a section explaining all the problems found: there was fraud, there was misconduct, and around a third of the sites had rounded blood pressure readings.
Because the sponsor had been totally honest, there was no inspection at all. It had shown everything: the fraud, which was minor, five cases among many thousands, the rounded blood pressures in over a third of the patients, and what was being examined and how. Being totally transparent and honest about what is found is not always bad for the sponsor.
The fraud and misconduct account ran to 90 pages, so it went into a separate appendix. Section 9.6 described the action and referred to the appendix, including the decision not to use the blood pressure data for any detailed examination because it was not robust enough.
The limits across a submission
At submission level, the quality tolerance limits from each study can be normalised, which statisticians can explain, and aggregated into a cumulative sum chart, or CUSUM chart. One chart is made for each type of limit, covering every study in the submission, and the charts support the submission's good clinical practice statement. Companies already using the limits within a quality management system will probably know such charts.
In an example from a dummy large company with about 34 trials over six months, a chart of normalised protocol deviations per trial showed variation and a couple of minor breaches, but no pattern to worry about. That overall picture is what is shown at an inspection. Normalising the same data by project separated five or six projects, one of which trended downwards over time, and that project is one the good clinical practice statement might need to explain. The question is no longer whether the trials were perfect but whether they stayed within their tolerance limits.
How widely the limits are used, and for what
In an audience poll, 46% of respondents said the limits were implemented on their trials but not yet reported, 36% were not using or reporting them at all, and 18% had already reported them in clinical study reports. The 36% were not following good clinical practice although their protocols stated that they would. Where a protocol states that ICH good clinical practice will be followed, the lack of quality tolerance limits can itself be reported as a protocol violation.
Asked what else they used the limits for, apart from reporting in the clinical study report, with multiple answers allowed, 85% named quality monitoring within the quality management system, 78% continuous quality improvement and quality by design, 41% the submission's good clinical practice statement, 19% replacing other monitoring activities, and 11% none of these. Using the limits across the organisation matters because, when quality by design arrived, there was a risk that companies would develop something else rather than use the existing tools the regulators had given them.