Designing a PMCF survey
Once a survey is the chosen method, everything that decides whether its data can be analysed is settled before any response arrives: the stages and their timings, the feasibility check, the protocol, the endpoints and their acceptance criteria, the sample size, and the rationale each question carries.
Stages and timings
A survey runs through a sequence of stages, which in practice should form a cycle that feeds back into the initial planning. It starts from a device on the European market with its CE mark, a completed clinical evaluation report and a PMCF plan that states how the data will be collected and justifies a survey as the method. A survey protocol follows, setting the methodology, the sample size and the criteria. The survey design comes next: the inclusion criteria that identify a relevant respondent, and questions that align with the endpoints the protocol documents. The survey is then programmed, most commonly online on survey software, with the routing and logic that make one question's appearance depend on the answer to an earlier one. It is then sent to the end users, and the data is quality-checked as it comes in. The data is then tabulated and analysed in line with the data analysis the survey protocol sets, and the deliverables are handed back to the manufacturer.
Timings are very approximate. Planning and design take one to two months. Programming through data collection is the variable stage, and it varies a great deal with the sample size and with how hard the respondents are to find. Reporting and updating the documentation take another one to two months. The whole runs from three to six months, sometimes longer. Where a survey provider takes part, it is very often brought in at the survey plan or survey design stage and stays through to reporting.
CROs often do much of the planning of a survey, especially a level four survey, before bringing it to a survey provider.
Feasibility
Planning a survey starts from the device: its risk class, which bears on the level of survey, the lifetime of the device and the duration of its use, and whether it is implantable. A level four survey may have the respondent look back at patient charts, so planning also settles how that information will be accessed and what can be obtained. Planning then turns to the users: their specialties, whether they are healthcare professionals or patients or both, and how they will be reached and targeted.
Sales volume decides whether the sample a calculation asks for can actually be gathered, so feasibility should be weighed alongside the statistical justification. A sample size of 400 is not realistic for a device of which 100 units have been sold in the last five years.
Sales also bear on which countries to target, and the choice is made with cost in mind, since sending the survey to every country is not cost-effective.
In a worked case, the device is a Class III legacy catheter for peripheral vascular access, supplied in several French sizes. It is used by healthcare practitioners trained in placing such catheters, including nurses, physicians and physician assistants, in adult and paediatric patients needing short- or long-term access. It sells at varying volumes in Germany, France, the UK, the US and China. Class III legacy devices must meet a minimum level of evidence, so the survey has to be a high-quality, case-specific one. Because the regulation is European, European data is the gold standard. France was dropped because its sales in the last year were too low to justify translating the survey and unlikely to yield enough completed responses. China was left out for a different reason: the patient characteristics and the healthcare system are dissimilar enough that data collected there may not be representative of European use. The US was retained as a possible supplement, because its patient characteristics and care systems are close enough that device usage should not differ greatly, especially on safety and performance, and such supplementation has been accepted in practice. So the focus was Germany and the UK, with the US as a fallback if the sample fell short.
The survey protocol
The critical parts of a survey protocol are these:
- Describe and identify the device or devices in question, including their variations and sizes, the intended purpose and the indications.
- State the objective and the endpoints: the primary and secondary endpoints, the safety objectives, and the acceptance criteria that align with them.
- Justify the sample size and give its rationale. Guidance pushes for a robust justification.
- Define the study population, down to the specialty and to whether the respondents are healthcare professionals, nurses or patients.
- Say how bias will be minimised, including whether the manufacturer or a third party will distribute the survey.
- Give the qualifying or inclusion criteria and any exclusion criteria.
- Set out the questions at a high level, and check that they cover the objectives and endpoints.
- Plan how the survey will be carried out, not only what it will do: its timelines, and the backup options if the sample or the timelines are not met.
Expanding to further markets if the sample falls short, as with the US in the worked case, is one such backup. The usual headings of a protocol may leave no place to state when the data will be collected. A survey or execution plan heading can hold it, laying out the anticipated timeline and how any partnering on the sample will work.
Endpoints and acceptance criteria
Setting endpoints and their criteria is a big difficulty, especially for manufacturers. The primary endpoint is usually about performance, such as procedural or technical success or long-term success, and its question asks how many times the device performed successfully. Its acceptance criterion is set on the lower bound of the observed success rate: the lower bound must be greater than or equal to the acceptable level, and if it falls below, the result cannot be deemed comparable or acceptable.
The safety endpoint mirrors it. It could be an adverse event rate, asked as a question about the frequency of adverse events with follow-up questions on causality and severity. Its acceptance criterion is set on the upper bound: the upper bound of the observed rate must fall below the criterion.
The plan can also define exploratory endpoints, such as other performance or safety questions, off-label usage, and user preference and satisfaction, collected through questions or rating scales. They need no acceptance criteria.
The acceptance criteria should be determined through a review of the clinical evidence, a clinical evaluation that assesses the state of the art, competitor and benchmark devices, and the subject device itself. The worked case begins by reviewing the available data on the subject device and the state of the art and identifying appropriate criteria to align with the endpoints. It then designs the survey questionnaire, collects the data, writes the report and uses the findings to update the relevant clinical documentation. Its endpoints were defined from that review together with the intended purpose and indications. The performance endpoint is the device's ability to be placed successfully in the peripheral vascular system. The safety endpoint assesses the device's adverse events, to ensure they are in line with or below the state of the art, and identifies previously unknown adverse complications. The minimum performance criterion was set at the minimum success rate reported in the clinical literature for benchmark devices, 90%, so the acceptance criterion is a success rate greater than or equal to 90%. It is the minimum performance, here the average rate of successful placement, at which the subject device can be considered comparable to the state of the art.
Sample size
The sample size should let the endpoints be statistically verified at analysis, so it is derived from the primary endpoint's acceptance criterion and the expected rate. The link is the width of the confidence interval: if the criterion is a success rate greater than 80% and the survey observes 85%, a sample of twenty gives an interval wide enough for the lower bound to fall below 80%.
Options for arriving at a sample size include a power calculation from the predefined acceptance criteria, sizing at maximum variability with a desired confidence interval, and calculating confidence bounds from the expected event rate.
The worked case calculates confidence bounds from the expected rate, using the Wilson score method, one of several, with a critical value of 1.96. With the 90% criterion and an expected success rate of 95% for the subject device, the calculation returns a sample size at which the lower bound of the primary endpoint comes out at 90.2%, above the 90% criterion.
The questions
Each question in the worked case carries its rationale: the endpoint it measures, the clinical gap it fills or the claim it tests. A review of the device's clinical documentation showed two gaps: little data on the paediatric population, and little on the smallest and largest French sizes.
The gaps generate questions that are not endpoints. A device-information question asks which French size was used, with the device's sizes of four, six, eight and ten French as the response options, so that the data covers the whole size range, including the two gap sizes. A patient-information question asks the patient's age in ranges rather than as a specific age. The ranges cover the paediatric population, and they mitigate the risk of collecting patient-identifiable information, which a case-specific survey must take particular care to avoid. Neither question needs an acceptance criterion, because neither is an endpoint, but a minimum quota should be set on the gap sizes and on the paediatric population, so that each gap is represented and filled.
The endpoint questions take a yes-or-no answer. The primary endpoint question asks whether the device could be placed in the peripheral vascular system without complication and in accordance with the instructions for use. A no is followed by a question on what issues were encountered, and the acceptance criterion is a success rate greater than or equal to 90%. The secondary safety endpoint question asks whether any adverse events occurred while the device was in use, with follow-ups to select which events occurred and on their severity.
A question that tests a clinical claim follows the exploratory pattern. It asks how easy or difficult the procedure was to perform with the device, on a scale of one (extremely difficult) to seven (extremely easy), and an answer at the low end of the scale, a one or a two, may be followed by a question on what happened and why. It is neither a primary nor a secondary endpoint and carries no acceptance criterion.