What protocol complexity costs a trial
Protocols have grown more complex for decades, and the more a protocol asks of a trial, the worse the trial performs. The cost falls on recruitment and retention, on participants and sites, on the quality of the data and on the number of amendments, so a study team limits complexity by deciding together what is critical to the study.
What makes a protocol complex
Protocol complexity has quantitative and qualitative elements. The quantitative elements can be counted, such as the number of objectives, endpoints and activities, and the qualitative ones include whether the study uses adaptive designs or decentralised features such as home visits, and the overall ambition of the study. Protocols range from a very simple study of five site visits to one with adaptive design features, several central endpoints and sub-protocols under a master protocol, with different conditions and points that may each implement a new component.
The Tufts Center for the Study of Drug Development defined a set of complexity variables: total endpoints, total eligibility criteria, procedures performed, the percentage of non-core procedures, procedures per patient visit, total countries and total investigative sites. Among these variables, a non-core procedure is one that supports neither a primary nor a key secondary endpoint.
A Tufts analysis of almost 3,500 protocols sorted their procedures into four classes. Core procedures are related to the study endpoints, required procedures are required by the Declaration of Helsinki or by legislation, standard procedures are measurements such as weight, height and body mass index, and non-core procedures are related to exploratory endpoints. The core, required and standard procedures could not really be changed, and the non-core procedures are the ones to look at. Non-core procedures accounted for just under 20% of the procedure cost of phase 2 and phase 3 studies, and that counts only what the hospital charges the sponsor, not the cost of collecting the data through monitoring and data entry.
How protocol content has grown
Protocol content, as distinct from the structure of the document, has evolved over time, and the content and complexity of protocols have increased over the decades. Before the 1990s the number of assessments, sites and participants increased, particularly in late-stage protocols. In the 1990s cost containment and outsourcing to contract research organisations raised the number of endpoints and assessments in phase 2 protocols. In the 2000s wider patient access and more global protocols increased both the content and the reach of protocols, and phase 1 and phase 2 protocols in particular gained safety procedures and data volume. In the 2010s the pursuit of flexibility and efficiency brought adaptive designs, master protocols and a focus on rare diseases and subpopulations, which increased endpoints, assessments, data volume and complexity, so that protocols as a whole grew bigger.
A Tufts comparison of just over 116 trials from 2002 and 2012 found that the number of endpoints almost doubled, from 7 to 13, and the number of procedures rose from just over 100 to 167. Eligibility criteria, countries and investigative sites all increased, and the number of patients randomised was the only figure that fell, from 729 to below 600. Every one of the Tufts complexity variables increased, total eligibility criteria by 10% and total countries by 69%, so the process is additive.
Protocols once had fewer moving parts. As they have become more complex, the volume of information has grown and the way its parts connect has become more complex, so getting a whole study team on the same page has become increasingly challenging.
What complexity does to trial performance
Complexity is a balancing act against the performance of the trial: the more complex the protocol, for example the more endpoints it has, the poorer the performance. A Tufts study of 187 protocols and their trial data from 20 pharmaceutical companies and contract research organisations compared protocol content with performance variables such as recruitment rate, dropout rate, trial duration, treatment duration and deviations. Performance correlated negatively with the number of endpoints, eligibility criteria, procedures per visit, countries and sites per country. A second study, on a much bigger and more harmonised set of around 2,051 datasets from about 300 drugs, compared the same protocol content with performance and found a negative correlation with the number of endpoints, eligibility criteria, countries, patient visits and patients.
Scientific and operational design characteristics are therefore significant predictors of trial performance. So the effect of the content put into a protocol has to be considered not only when the study outline sets out the scientific concept but also during protocol development, when the details and the operational elements are added. Decisions taken during protocol development have a long-standing effect on trial performance.
By one statistic, 86% of completed trials do not achieve their recruitment targets on time, which lengthens the study and takes more resources, financial ones and also patients who could have been eligible for other studies and are recruited to them later. Of all trials, 19% close prematurely because of poor enrolment, so the data collected from their participants is not used for decision-making. Of the participants who drop out, 30% do so because of overly complex protocols. According to the FDA, only 5% of COVID-19 trials conducted at least partly in the United States achieved actionable information, meaning that they were randomised and adequately powered. A study with an insufficient participant population is underpowered and does not yield actionable information.
A recruitment target set unrealistically at the outset will fail, and even a realistic target is less likely to be met when the protocol is overly complex.
One view holds that the growth in protocol content and the decrease in performance it brings have reached a point that will yield a generational change in protocols.
The burden on participants and sites
Trials show increases in burden, in the amount of collected information that is non-core, not central to the trial's purpose, and in the time participants spend at sites. In 2002 a participant typically had nine visits and spent 187 days on trial, and by 2012 that had risen to 11 visits and 230 days, almost two months longer. Asking participants to be evaluated for many additional procedures whose data will not be used in the clinical study report, the publication or the drug application is a waste of resources and of the clinical sites' time, and one practitioner asks whether it is even unethical.
Sites suffer from highly complex protocols that are very difficult to implement because of the number of assessments or activities to be done and the size of the protocol. Some master protocols reach 400 or 500 pages, and a site implementing one has to digest that amount of information, and presumably to operate under several different conditions.
What complexity does to the data
A review of the literature concludes that the high volume of data collected in protocols distracts research scientists, compromises the data analysis process and ultimately harms data quality. Studies have found that error rates increase as more data are collected during protocol administration, that more protocol procedures are associated with a higher incidence of unused data in new drug application submissions, and that unused clinical data compromises the data analysis process. Initial data from a Tufts study suggested that study monitors perform 100% source data verification on the clinical data from non-core procedures, and that one third of all queries per case report form are associated with these less essential procedures.
Case report forms grew from an average of 55 pages in the late 1990s to almost 200 pages, almost four times as many, between 2008 and 2009. A case report form of that size would explain why so many queries relate to non-core procedures.
The harm is not only more work. Asking investigative sites to spend time on non-core data that will not be relevant to the publication or the new drug application distracts them, and distracts sponsors, from the core procedures and the core data that affect the study endpoints.
Amendments
Protocol complexity is associated with a higher incidence of costly protocol amendments. The number of amendments is increasing because of complexity. A problem left to be fixed mid-flight means more protocol amendments, which lead to more protocol deviations, so stopping it at the beginning benefits everyone.
Limiting complexity
Complex protocols and amendments should be avoided. One reading of the Tufts data holds that intensely ambitious and demanding protocols are at the heart of the problem, and that an important first step in reducing amendments rests with simplifying clinical research studies. The same reading holds that efforts to reduce and prevent certain amendments through better protocol designs and more informed decision-making will improve study feasibility and efficiency and yield substantial cycle-time and budget improvements.
Managing complexity aims to tailor the trial as far as possible to its participants and to its sites. Guidance states that efforts should focus on the activities essential to the study outcomes.
Much of what a protocol contains comes from templates or legacy protocols, and medical writers are sometimes not ruthless enough in asking what is really needed to answer the objectives of the study. One practitioner holds that protocols are frequently a burden of superfluous items, with case report forms full of superfluous questions, and that it is sometimes very hard to make that argument, especially when colleagues are inexperienced and insecure.
Agreeing on what is critical
Not all content is equal: some of it has a higher priority than the rest. So scoping and critically reviewing the material include identifying what is critical to quality and what is not. What is critical is subjective and depends on perspective, so it cannot be programmed and is identified by working with the study team. When one practitioner takes a team through a draft protocol to sort what is critical from what is not, mostly everyone agrees on what is critical, a couple of items are disputed, and some people disagree on the non-critical parts too. Making sure that everyone understands and agrees can help the team share one perspective, keeps it from going off in different directions and brings together competing views of what the study will look like. Helping the team focus first on the critical objectives and the risks associated with them helps focus the discussions and the changes.
Monitoring non-critical items during protocol development is good, but the team should stay focused on what is critical, and summarising the changes after each review round in terms of what is critical and what is not can help it do so.
In one study, sorting critical from non-critical content revealed an exploratory endpoint that was critical for the study but held no position of priority, not being key secondary or anything like it. Going through each step of the sampling for that endpoint showed that it could be a risky and expensive procedure, and to maintain the critical samples the team reduced the sampling frequency and increased risk monitoring. Focusing the team on the specifics helped it work through each step and put risk mitigations in place.
Simplicity in the emergency department
Emergency departments are increasingly crowded, the number and complexity of their patients are rising, the time from arrival to the need for treatment is tight, and caregivers have neither the time nor the resources to help with research, which makes the department a challenging place to conduct research alongside effective and timely care. Given the complexities of implementing a protocol there, investigators designing a trial need to make sure that its intervention and flowchart are not overly complex, and the design must include only key intervention and data points while adhering to the regulatory framework of clinical research. To do that, investigators should work early in protocol development with clinical research staff, such as clinical project managers, study coordinators, biostatisticians, pharmacists and medical writers, who help guide the design to maximise study efficiency in the emergency department.