Setting out the design of a device study
A device study's design is set out in the sections of its clinical investigation plan that state the objectives, hypotheses and claims, the study type, the endpoints, the eligibility of subjects, the point of enrolment and the analysis populations, the study's duration and size, the procedures and the comparator, and the statistical analysis.
Objectives, hypotheses and claims
The plan's required content lists separately the objectives, the hypotheses to be accepted or rejected by the statistical data, and the claims and intended performance of the device to be verified, such as late lumen loss below 0.5 mm at six months. An objective describes what the study intends. Most studies have only one objective, such as assessing the safety and performance of the device, and there can also be secondary objectives, such as assessing the device in subgroups like elderly or diabetic patients. The objectives are mostly supplied by the company, so the writer need not worry much about them.
One experienced writer finds the three-way listing confusing and summarises it in one or two sentences: the objective is to assess safety and performance, the hypothesis is this, and it is confirmed by that claim.
The item asking for the anticipated adverse device effects to be assessed can basically be skipped in a study with clinical endpoints such as death or myocardial infarction, since the endpoint section already mentions them.
Study type
The design section describes the type of investigation, such as comparative or double-blind, with the rationale for the choice. Study types include randomised or not, blinded, prospective or retrospective, and single-centre or multicentre, and also first-in-man, first-in-human, pre-market, post-market and investigational device exemption studies.
Pharmaceutical studies are clearly classified into phases 1 to 4, but device studies are not, and a variety of terms with similar meanings exist instead. These terms are not regulated, so everybody uses their own definition: some speak of a feasibility study, some of a first-in-man study, some of a first-in-human study, and some even call a pre-market study a first-in-man study. The result is a wild mix of nomenclature, and the writer adheres to that of the company they work for. The only terms defined by regulation are clinical investigation, or clinical study, defined in ISO 14155:2011, and device registry, defined in MEDDEV 2.12 revision 2.
Trends in investigation design
Device investigations were once fairly standard, and there is a trend towards new designs adapted from those used for medicinal products. Master protocols are one example, with a core protocol and sub-protocols: basket trials involve different patient populations but the same product, and umbrella trials one patient population but different products. Their aim is to ease the creation of the documents and their review by ethics committees and, where applicable, by competent authorities. Another strategy combines different investigation stages, such as the pilot, pivotal and post-market stages, into one master protocol.
Piggy-backing randomised controlled trials on registries is a novel and efficient way of conducting them. Adaptive designs are increasingly used to make investigations more flexible and more efficient, and decentralised trials are another new design.
Above all, as for medicinal products, patient centricity is becoming more important. The FDA has released a statement encouraging patient engagement in device investigations and has issued principles for patient-reported outcome instruments for use in device evaluations.
Endpoints
The plan gives the primary and secondary endpoints with a rationale for their selection and measurement, and describes the measures taken to minimise or avoid bias, such as blinding or masking. It is very important that the endpoints are defined clearly and unambiguously. There are numerous definitions of myocardial infarction in coronary stent trials, for instance, so the plan makes very clear which definition applies and uses it consistently. When an endpoint is defined one way in the body of the plan and another way in the attached definitions, the sites cannot know which definition applies.
New approaches to endpoints are being implemented. Composite endpoints were routinely used in the past, and hierarchical composite endpoints are a new category, combining disparate endpoints that are neither equivalent in severity nor assessed on the same scale.
It is also important not to have twenty endpoints, as some companies tend to add, because every endpoint then has to be reported. With twenty endpoints the risk is also high that the study team does not focus on the relevant things, and the result can be a quality-of-life questionnaire with only 20% compliance, which is hard to publish. A post hoc analysis declared as such in the paper, by contrast, is usually not a problem.
Eligibility, withdrawal and replacement of subjects
The plan gives the inclusion and exclusion criteria for subjects. ClinicalTrials.gov, publications in the area and guidelines in the disease area give a sense of the common criteria. Companies tend to be over-cautious, especially in first-in-man and pre-market studies, and add far too many criteria, which makes the study unfeasible and prone to error. Someone at the centre has to memorise, say, thirty criteria for one study, and likely those of ten other studies as well, which is three hundred criteria alongside their other work. In an ideal world they are well organised and have a process for checking the criteria, but experience shows that the investigator often memorises the key criteria and simply includes the patient.
That is very dangerous. In an FDA study, too many protocol violations endanger the approval. They also show up when the trial is published. A protocol admitting reference vessel diameters of 2.5 to 4 mm and a population with many 2 mm vessels is a contradiction in the paper that gets questioned, and the explanation, that these were protocol violations, does not look good. Sometimes the minimum and maximum criteria are even left out of a publication, to avoid emphasising that the study enrolled too many patients outside the protocol. The situation is different with a per-protocol analysis.
The plan also gives the criteria and procedures for subject withdrawal or discontinuation, for a non-compliant patient for instance. It gives any procedure for replacing subjects: whether a subject who withdraws consent or is lost to follow-up will be replaced. Mostly the plan says they will not be, and a dropout rate is instead already calculated into the sample size, but this needs to be verified.
Point of enrolment and analysis populations
The point of enrolment is very important, and a medical writer knows when it has been got wrong, because explaining it in the manuscript is a great deal of work. Formally a patient is enrolled when they sign informed consent, and that is also when certain considerations apply, such as insurance and safety reporting. Depending on the area, though, it may make sense to define several phases of enrolment.
Device trials often comprise different analysis groups, particularly where implants are concerned, and it is important to define them clearly upfront. Intention-to-treat can mean the patients who signed informed consent or the patients in whom an implant was attempted, and while the former is the more common definition, the latter might be more suitable for implants. In coronary stent trials, for instance, a patient's final eligibility is usually determined after informed consent, during angiography, and the term "implant attempted" avoids contaminating the intention-to-treat group. For randomised trials, the terms "patients per allocated treatment group" and "patients per treatment received" are comparable to those of the pharmaceutical industry.
An intention-to-treat population can also be defined so that a patient counts in it only once the procedure has begun. In one study where this was not done, patients in the UK sometimes saw the hospital doctor, were sent home, and had their surgery two weeks later, and one patient who died in between disrupted the whole study. The death can be described in a manuscript, but people usually just copy the table and quote the mortality rate. Sometimes an additional analysis population is defined, such as an as-treated population of the patients in whom the device was implanted.
Early clinical studies may include roll-in patients, the first patients treated at a particular centre with a new technique in which complications might be expected as part of the learning curve. They are commonly not counted as part of the primary analysis group. Where complex procedures such as implanting a heart valve are involved, outcomes are also related to the skills and experience of the operator. Analysis per centre might be advisable for clinical study oversight, but is commonly not reported. As in pharmaceutical reports, all post hoc analyses should be appropriately labelled as such in any resulting manuscript or summary.
Study duration and sample size
The total expected study duration is the enrolment period plus the duration of subject participation. It should be estimated generously, because enrolment is usually delayed and nobody objects to enrolment finishing earlier than expected. The number of patients should usually be a statistically calculated sample size, though there may be a special reason not to have one. As a very rough estimate, a trial with fewer than 100 patients would never be approved, because the number of patients has to be calculated as a statistical sample size.
Procedures, comparator and other products in use
The plan describes all the investigation-related procedures the subjects undergo, such as a quality-of-life questionnaire, an echocardiographic examination, a six-minute walk test or an ECG. It describes the methods and timing for assessing, recording and analysing the variables, usually with a table of the assessments and a description of each. It also describes the equipment used to assess the variables, with the arrangements for its monitoring, maintenance and calibration. The assessments should be kept as close as possible to standard of care, because the more complex the plan, the lower the compliance and the more likely the study is to fail.
The exposure to the investigational device, or to the comparator if one is used, is described as a procedure, separately from the description of the device itself, with a justification of the choice of comparator. Comparators that were somewhat outdated were once often chosen, to make sure the investigational product performed well. This item of the plan, together with the Declaration of Helsinki, states that comparators should always be state-of-the-art devices, which should prevent that malpractice.
Any other medical device or medication to be used during the investigation is listed, such as a commercially available introducer sheath in a transcatheter heart valve trial, or a few sentences on a recommended dual antiplatelet regimen in a stent trial. So is the number of investigational devices to be used, with a justification. This sounds obvious, but in a coronary stent trial in multi-vessel disease several stents may be placed.
The activities performed by sponsor representatives are described, and not only monitoring: a sponsor representative present in the operating room to prepare the device is described too. So is any known or foreseeable factor that might compromise the outcome of the investigation or the interpretation of its results, such as subject baseline characteristics or concomitant medications, and the plan may also address how that bias is to be prevented.
MEDDEV 2.7/2 revision 2 recommends that, when complex or unfamiliar devices are handled, risks be mitigated by adequate training and support during the first cases. Such training should be featured in the clinical investigation plan.
Statistical analysis
The input for the statistical section usually comes directly from the statistician, and the writer makes sure that it is complete and perhaps even that a non-statistician understands it. The section covers:
- the statistical design, methods and analytical procedures
- the sample size and its calculation, where applicable
- the significance level and the power
- the expected dropout rates
- the pass and fail criteria
- the provision for an interim analysis, where applicable
- the criteria for terminating the investigation on statistical grounds
- any deviations from the original statistical plan
- the specification of subgroup analyses, if any
- the procedures that take account of all the data
- the treatment of missing, unused or spurious data, including dropouts and withdrawals
- the exclusion of particular information from the testing, if relevant
- in a multicentre investigation, the minimum and maximum number of subjects for each centre
For the number of subjects per centre, a minimum should never be specified, because performance will always be low: sites promise everything and in the end enrol one patient. One experienced writer always gives an "up to" number instead, usually that no centre will enrol more than 15% of the total patient population, with a minimum of basically one.