Establishing that a treatment works
That a treatment works is shown first under the controlled conditions of a trial and then in everyday practice. One patient's improvement proves little, so a trial that regulators rely on is built from features that each close a gap. The disease has to be represented by a measurable feature and turned into an endpoint, the trial population is restricted, and results are judged at group level, for their size and clinical meaning as well as their significance, and against the risks. Where a trial in patients is not feasible, efficacy can be shown another way, and a drug that shows efficacy in trials can work less well in practice.
One patient's improvement
Suppose that over a few weeks someone's elbows and knees develop red, flaky, itchy patches, an internet search suggests psoriasis, and after an oatmeal bath the itching eases and over the next few days the patches shrink and look less red and scaly. Most people would probably conclude that the bath worked, but evaluating how well a treatment works is not as straightforward as it might seem. It is human nature to infer cause and effect from a sequence of events in time, here that the treatment caused the improvement. The bath may really be an effective treatment for psoriasis, assuming the self-diagnosis is accurate, but there are several other possible explanations for the improvement. Many conditions, psoriasis among them, have a natural course of flare-ups followed by periods of less severe symptoms, so the skin might have improved anyway, and other changes in lifestyle, such as less stress or a different diet, might be responsible.
The trials regulators rely on
A regulator, such as one at the US Food and Drug Administration, deciding whether a drug can be approved for potentially millions of patients needs more than one patient's subjective impression, however important that impression is. It needs to be convinced that the drug works well and reliably in a large, representative group of people with the disease, and that the evidence is free from systematic error, or bias.
The strongest evidence comes from large, double-blind, randomised controlled trials, which provide a very controlled setting and limit the potential for bias. Such a trial has to be designed to allow firm conclusions about whether a drug can induce improvements in a group of patients with a given disease, and it has these features:
- Controlled: the trial must include a control arm, either placebo or another drug, an active comparator, and the two groups must undergo the same procedures and differ only in the treatment they receive.
- Parallel-group: the test and control groups must be treated at the same time under one protocol, so that they are truly comparable.
- Randomised: every patient needs the same chance of either group, which avoids systematic bias in allocation and should make the groups comparable at baseline in all important characteristics, whether assessed or not, so that an improvement in one group can be attributed to treatment rather than to fluctuation of the disease or chance.
- Double-blind: neither patient nor investigator may know the treatment, so the measurements and the interpretation of symptoms are not biased.
Defining the disease
Efficacy only means something in relation to the disease a drug is meant to treat, and diseases are often complex, with many symptoms and physiological manifestations.
Modern drug development relies on the biomedical model, in which disease arises because one or more molecules become dysfunctional, and the malfunction is assumed to be fixable with the right molecular tool, a medicine, that stops or corrects it. So developing efficacious treatments depends on understanding the disease biochemically. The biomedical model is not the only way of understanding disease, and holistic approaches focus instead on the whole person, body, mind and spirit, and on balance. Some cultures attribute disease to spirits, imbalances of yin and yang or a blocked flow of qi, and their remedies often try to address imbalances and restore harmony within the individual and with the environment.
What counts as a disease is itself the subject of intense philosophical debate, and historically some states have been regarded as diseases on the basis of societal beliefs not grounded in biomedical science. Not long ago, medical textbooks described homosexuality as a mental illness. With cultural change, diagnoses such as hysteria and drapetomania, a 19th-century "disease" attributed to slaves who tended to run away from their masters, came into fashion and then disappeared. Some current diagnoses, such as female sexual arousal disorder and attention-deficit hyperactivity disorder, raise questions about where the boundary lies between normal and a dysfunction that qualifies as an illness.
A feature that represents the disease
Before efficacy can be determined, a feature has to be identified that is highly relevant to the patient's health and so can represent the disease burden. The feature has to change with severity and be measurable, and the measurement should be fairly simple and robust. It must not be significantly influenced by who takes the measurement or where, and repeat measurements shortly after the first should give very similar results, so the feature needs some stability. For many diseases, regulators and medical societies have agreed which features best represent the disease and can be used to determine whether a treatment is efficacious.
Sometimes the feature measures the disease only indirectly, through a surrogate marker such as a blood biomarker whose concentration varies in proportion to disease severity. Glycosylated haemoglobin (HbA1c) in diabetes is a well-known example. A high level, 8% of haemoglobin or above, is a key physiological consequence of diabetes, but it is neither the disease in all its complexity nor a symptom that troubles patients. HbA1c is straightforward to determine from a blood sample, and it is a reliable marker of average blood glucose over the previous 8 to 12 weeks, whereas blood glucose fluctuates considerably through the day. It also correlates with the frequency of serious consequences of diabetes such as stroke, heart attack, retinopathy and nephropathy. So a diabetes medicine's efficacy can be assessed by its ability to lower HbA1c, which is quicker and much less costly than measuring clinical outcomes such as strokes, heart attacks, diabetic eye disease or diabetic kidney disease. A disease map for type 2 diabetes sets genes, behaviour and environment, inflammation and metabolic stress, reduced glucose uptake in muscle, increased glucose production in the liver, dysfunction of the β-cells in the pancreas and insulin resistance beside hyperglycaemia, high levels of blood glucose and HbA1c, which it labels the characterising feature or marker. The map also shows retinopathy, nephropathy, neuropathy, coronary and peripheral artery disease and stroke.
In some diseases no agreed biomarker can serve as a surrogate for disease severity, and clinical outcomes are the preferred measure for evaluating treatments. In psoriasis, the clinical symptoms are measured, with an instrument developed for the purpose, the Psoriasis Area Severity Index (PASI), a detailed questionnaire that physicians complete on examining a patient. Its answers yield a single score for the average redness, thickness and scaliness of the lesions, weighted by the area of disease involvement, and a lower score means less severe psoriasis.
Endpoints
An endpoint is a variable whose value captures the benefit hoped for from a treatment, as defined in a trial objective.
Endpoints are usually derived by calculation from the measured variables, they need to be precisely defined, and they often capture change between two specified time points, usually the start and end of treatment. A variable measured on a numerical scale, such as HbA1c or PASI, can be analysed as the average change over time or as the proportion of patients achieving a response, such as an improvement in PASI of at least 75% or an HbA1c below 7%. The second is a responder analysis.
The two examples run from disease to endpoint as follows:
| Term | Definition | Diabetes | Psoriasis |
|---|---|---|---|
| Disease | The medical entity considered abnormal | Diabetes | Psoriasis |
| Characteristic | A measurable feature that represents an important aspect of the disease | Average blood sugar over the past 8 to 12 weeks, as HbA1c | The redness, thickness and scaliness of the skin, as captured by PASI |
| Measurement | The process of determining and recording the value of a variable that represents that characteristic | HbA1c (%) recorded in blood | The answers to the PASI questionnaire |
| Variable | A measurable attribute, phenomenon or event whose value is expected to vary over time and between subjects, either measured directly or derived from measured variables | ||
| Directly recorded variables | HbA1c at baseline and week 26 | PASI at baseline and week 16 | |
| Derived variables, patient level | Change from baseline to week 26; responder status as HbA1c below 7% at week 26 | Change from baseline to week 16; responder status as PASI 75, a reduction of at least 75% from baseline, at week 16 | |
| Derived variables, treatment-group level, the endpoints | Mean change from baseline to week 26; response rate as HbA1c below 7% at week 26 | Mean change from baseline to week 16; PASI 75 rate at week 16 |
A trial generally uses several efficacy endpoints to give a comprehensive picture of efficacy. Regulators require the endpoints to be prespecified in the protocol and the statistical analysis plan, which prevents results that merely look positive through random variation or the natural course of the disease from being cherry-picked afterwards.
The trial population
To get clear results for the chosen endpoints, the population of a randomised trial has to be clearly defined, and patients are screened against the inclusion and exclusion criteria in the protocol. The more heterogeneous the population, the harder it is to demonstrate a treatment effect, because a difference between treatments is hard to detect against the background noise of variation due to other factors. So it is important to control the variability in a trial population to ensure a valid test of the treatment, which is called internal validity.
But people vary in myriad ways, and testing a drug in an extremely narrow group does not help when a broader population needs it. The results have to apply to a target population in the real world, which is called external validity. So eligibility criteria attempt to balance a reasonably homogeneous trial population against one broadly representative of those likely to receive the drug once approved.
Judging the treatment effect
Because people differ biologically in many ways, responses to a treatment vary even in a relatively homogeneous group with the same disease, and they are not fully predictable: some patients may improve greatly, while others may notice no difference or even get worse. So treatment effects are analysed at group level, by aggregating all patients' data and testing whether there is a true difference in average outcomes between patients randomly assigned to different treatments. Regulators require the results of these tests to be statistically significant, that is, to show a reasonable probability that the observed difference was caused by the treatments and not by chance.
Once a statistically significant difference between the test and control groups is achieved, regulators are also very interested in the size of the treatment effect. They need to assess whether the benefit of a new treatment is clinically meaningful.
Benefit and risk
To decide whether a drug can be approved, regulators weigh the benefit shown in trials, the treatment effect on the main efficacy endpoints, against the risks of using the drug. All drugs have adverse effects, and some may be experienced by a large proportion of patients while others are very rare but serious. In some settings, such as terminal cancer, patients may be willing to tolerate severe adverse effects if the drug offers a significant extension of life or even a cure. The key question is whether the benefits outweigh the risks for most patients, that is, whether the benefit–risk ratio is positive. Provided a drug consistently demonstrates efficacy in several clinical studies and appears safe, regulators approve it and permit its marketing for the treatment of a disease.
Efficacy where a trial in patients is not feasible
Eurneffy is an adrenaline nasal spray for the emergency treatment of allergic reactions (anaphylaxis). For ethical and practical reasons, controlled clinical trials of its effectiveness in people experiencing a severe allergic reaction were not feasible. There is, however, extensive information on the use of adrenaline to treat severe allergy, and it is the standard treatment for anaphylaxis. The efficacy and safety of Eurneffy were evaluated in 537 healthy people aged 19 to 55, enrolled in fourteen clinical studies. The studies compared it with products in which adrenaline is injected intramuscularly, on blood pressure and heart rate (pharmacodynamics) and on how the medicine is absorbed, modified and removed from the body (pharmacokinetics). The results showed that the effects in the body of nasally administered adrenaline are comparable to those of products given by intramuscular injection.
Efficacy and effectiveness
Efficacy is generally understood as the ability of a drug or other intervention to reliably produce a positive effect in patients with a defined disease under ideal, controlled conditions. Clinical trials are good tools to evaluate it, because they provide a well-controlled framework of clearly defined eligibility criteria, random allocation to treatment groups, blinding of participants and study doctors, and endpoint measures defined before the study starts. Randomised controlled trials are a valuable source of evidence on benefits and risks, but they have limitations: their tightly controlled conditions, a clearly defined population treated under close observation, may differ substantially from the conditions of clinical practice. Evidence of efficacy is no guarantee of effectiveness under real-world conditions. Once on the market, a drug is used across the entire spectrum of patients and under circumstances the trials might not have tested, for example in patients with several comorbidities in addition to the approved indication. Evaluating a drug's effects under everyday conditions is called effectiveness research. Both efficacy and effectiveness are determined in groups rather than individuals: efficacy pertains to the participants in trials, and effectiveness to everyone with the disease who could be treated with the drug.
What limits effectiveness in practice
There are many reasons why a drug's effects in everyday practice differ from those seen in trials, and the entire societal–medical context contributes to its effectiveness.
Above all, to be effective a drug has to be available to patients, at best to all patients with the disease it was developed to treat, and availability has national, regional and local aspects. Approval permits marketing and selling in a certain country, and a drug approved in the United States, for example, may be unavailable in Canada or Mexico. Before a new drug reaches a patient, obstacles in the supply chain have to be overcome, the first of which is reimbursement, the negotiation of price with government agencies, insurers and wholesalers. Discussions on price may drastically affect availability, and a company that considers the proposed price too low ever to recover its investment may decide not to provide the drug at all. Regional wholesalers may decide not to stock a drug in locations where they do not believe it will be widely used, and even where a drug is available and the physician is willing to prescribe it, a patient's insurance plan may not cover its entire cost. The company that runs a hospital may not see a drug's benefits and so leave it out of its treatment plans and offer an alternative, and providers who do not believe in a drug's benefits may prescribe treatments they consider to have superior therapeutic effects instead.
Medical tradition is a second factor. Prescribers undergo many years of training and apply it in their practice, maintaining a tenacious medical tradition, and it may take a long time before new therapeutic options are widely accepted and offered to all patients. Providers and hospitals may continue to offer treatments long shown to be less effective because they are established practice, offer financial advantages or are expected by patients. Practitioners who have the impression that a medicine works for their patients continue to prescribe it, and they may not use a new drug with which they have little experience, particularly when it has been tested only against placebo and not against established treatments. A new drug may also not be given because the physician has not heard of it or does not believe it superior to the drugs she usually prescribes.
Adherence is a third. Trial participants are closely monitored for how, when and at which dose they take their medication, and are reminded, for example, to take the dose at the same time every day, with or without food. In ordinary life patients might occasionally forget doses, may be inclined to increase the dose on days when the disease is particularly troublesome, and may forget to renew a prescription in time and go without medication for a while. So even when the drug is available, its effects may be smaller than in the trials because of limited adherence. Patients may also be taking other drugs for other conditions that influence the effects of the new one.