Regulating AI in medical devices
A medical device that uses artificial intelligence falls under the AI rules as well as the device rules. In the EU the AI Act adds requirements to those the device regulations already set, above all human oversight, data governance and cybersecurity, and one conformity assessment covers both. In the United States the FDA treats machine-learning software as a device and regulates it across its product lifecycle, including the changes an adaptive device may make after it is marketed.
Devices that use AI
AI-based systems are typically implemented as software in medical devices or as software as a medical device. By late 2019 the AI boom had generated platforms and applications that translated into new regulatory frameworks in both the US and the EU. Some AI- and machine-learning-based technologies were categorised as software as a medical device, and so were subject to strict evaluation to ensure patient safety. Medical devices that use machine learning include automated insulin dose management, medical image segmentation and smartphone-powered diagnostic tests. Automated insulin dose management benefits some people greatly, but with AI it carries much risk, such as supplying too much insulin. A smartphone-powered test can be an in vitro diagnostic that reads the test with the phone's camera and gives the answer, or a test for renal disease. AI-based medical devices are mostly software, which as part of a medical device is covered by the CE marking of the whole device and as stand-alone software is a medical device with its own CE marking. The EU Medical Device Regulation (MDR) and In Vitro Diagnostic Regulation (IVDR) cover many but not all applications of AI in healthcare; they do not cover, for instance, an AI system that makes and manages appointments. The intended purpose of an AI-based device determines which regulation applies, and the GDPR serves data protection.
Risk class of AI devices
Almost all software used in medicine is class IIa or higher and must undergo a conformity assessment before a notified body, so AI medical devices are almost invariably regarded as high-risk, as the digital expert of BVMed, the German medical technology association, pointed out in 2022, while the AI Act was still a proposal. She considered this classification too general and advised giving more weight to the context of use, such as whether a device merely supports medical staff or completely replaces them. Under the MDR, she observed, CE-certified devices such as algorithm-based solutions already had a very high level of safety and quality, and manufacturers were unsettled by the multiple regulations, unsure whether the Act would change the risk assessment of their product because it contains AI.
In May 2024 a clinical systems and medical data lead at a device manufacturer thought that in the draft Act, Rule 11 of the MDR quite literally made every medical device high-risk, a tolerance that was too tight. The released final version, on her reading, loosened the tolerance a little, so that the risk is assessed in its context.
One conformity assessment for both regimes
A device that uses machine learning and is a medical device falls under both the MDR or IVDR and the AI Act, and the Act imposes obligations on top of those already there.
Much of the Act overlaps with what the device regulations already require, because they are already strict for high-risk devices. Risk management systems, quality management systems, document retention periods, post-market surveillance and monitoring plans, instructions for use and technical documentation all exist already. These documents are not new, though they may need tweaks and some new elements to meet the specifics of the AI Act. Retention periods may need checking against the AI Act's minimum of ten years. A single conformity assessment by a third party will show conformity with both the AI Act and the device regulation, and it normally assesses the AI component and the device component together as one system. Notified bodies were not oblivious to AI's risks before the Act either: guidelines existed to help them assess the quality of the AI and the risks averted in the technical documentation. So much of what the Act asks will already have been required, though there may be some new notions.
What the AI Act adds
Some requirements get much more attention under the Act. Human oversight is a design requirement: a device with AI must include parts designed so that a human can interpret how it is functioning, override or correct it where necessary, or stop it if it starts behaving abnormally or in a biased way. The Act puts much pressure on providers and deployers of AI systems to include measures that let humans monitor its performance and act when it deviates from expected behaviour. When a system behaves abnormally or in a biased way, automatic logs must also be kept, so that what went wrong can be traced and corrected, for example through post-market surveillance. A high-risk system should keep its automatic logs for at least six months, or longer where its intended use requires a longer retention period. Data governance, and accuracy, robustness and cybersecurity, are the other two big areas, and both are very specific to the AI part of the device. Data governance is vital because these devices all use machine learning, which depends heavily on data. In the opinion of one speaker on the Act in May 2024, data governance and cybersecurity are the most notable changes in it. It is up to medical writers, he supposes, to show in the documents for a device submission what was done to prevent sources of bias and what data governance strategies were followed to avoid creating an unfair and unethical model.
Surveillance, documentation and data under the proposal
BVMed's digital expert also objected in 2022 to doubled post-market surveillance. Article 65 of the proposal, with Article 67(1), would provide for additional post-market surveillance of medical devices by the market surveillance authorities competent under the MDR, with powers up to and including a recall request, for example where a product presents a health risk. The MDR, she pointed out, already specifies when manufacturers or authorities must take corrective measures, if necessary withdrawals and recalls, for non-compliance or health risks (Article 10(12) and Articles 95 onwards), and its Chapter VII already imposes comprehensive post-market surveillance and vigilance duties on economic operators and close market surveillance by the competent authorities. Given the MDR's already very tight post-marketing control of health-related risks, she held that the added control seemed superfluous and not justified, and warned that extensive retesting of certified devices and the ambiguity of conflicting rules would delay patients' access to innovative, affordable AI medical products in Germany and the EU. Manufacturers would face additional vigilance requirements, such as for incidents the MDR does not cover, and an expanded technical documentation for AI. BVMed objected that requiring common technical documentation for high-risk AI under Article 11(2) could complicate cooperation between companies, create intellectual property issues and even oblige the AI maker and the device maker to merge two technical documentations. It held that a clarification in Article 11(2) would be useful that exceptions are possible where the AI and the related product have different manufacturers.
The proposal demanded that training, validation and test data sets for high-risk AI be relevant, representative, free of errors and complete (Article 10(3)). To meet this, BVMed's digital expert suggested that manufacturers be able to obtain comprehensive training data, so that they could develop AI without bias, such as statutory health insurance data held by the research data centre then being set up at the German Federal Institute for Drugs and Medical Devices. That was not yet possible, and it would require harmonising the legal framework, for example across the Act, the GDPR and local data protection rules for research.
Standards and notified bodies
In 2022 there were few industry-specific guidelines or standards on meeting both the MDR and the AI Act for medical devices, and action was needed so that manufacturers would not have to identify the state of the art for proving conformity themselves, as an expert at the notified body TÜV SÜD Product Service observed. The designation of notified bodies also still needed specifying in detail. TÜV SÜD had set up an expert team and a task force to track the draft and assess how its requirements would be implemented in customers' quality management systems and products, and how to implement the authorisation requirements for conducting the AI conformity assessment and the processes and work instructions for AI product assessment. BVMed recommended that the Act regulate only basic safety and performance requirements, to avoid a jungle of standards that would make the MDR's and IVDR's requirement to observe the generally acknowledged state of the art very burdensome. Regulatory experts recommended awaiting and considering the International Medical Device Regulators Forum's harmonised approach to regulating AI-enabled medical devices.
Liability for self-learning systems
In 2022 BVMed believed that a gradual harmonisation of liability rules may be needed for self-learning systems, because their performance changes independently during operation; erroneous, incomplete or discriminatory clinical data, for example, may degrade the software's security and performance. Its law expert argued that, unlike in the convenience or lifestyle sectors, AI systems in medicine do not usually act fully autonomously but are supervised by doctors or researchers, and that the solutions to date support qualified staff and do not replace them. Even an autonomous system, she argued, can be assigned to the responsibility of a manufacturer or operator: its autonomy rests on the manufacturer's design and programming, and its operation, updates and the plausibility check of its results can always be attributed to a responsible person who is liable in case of damage.
AI devices under the FDA
In late 2019 the MDR did not address AI and machine-learning technologies, while the FDA had recently proposed, for discussion, a regulatory framework for modifications to AI- and machine-learning-based software as a medical device. The proposal addressed a difficulty inherent in the technology: software that constantly adapts may over time become significantly different from the version that was approved, which would warrant a new premarket review to make sure its performance and safety are maintained. The proposal aimed to overcome this with a total lifecycle approach, including closer interaction with developers, identification of pre-determined types of change that would be acceptable, and periodic update reports on such changes. Since 2018 the FDA had already approved the marketing of several AI-based medical devices.
The first AI-based medical device the FDA approved, in April 2018, was IDx-DR, software that used an AI algorithm to analyse retinal camera images to detect diabetic retinopathy in adults with diabetes. By 2023 the use of AI- and machine-learning-based devices had grown exponentially, with built-in algorithms that learn from real-world experience and adapt, in sensors, stimulators, glucose monitors, enhanced imaging systems and wearable devices among others. The number of approvals was steadily increasing, with 41 in 2022 and 15 by 30 April 2023. Recent approvals included an implanted spinal cord stimulation system for chronic pain and a system that continuously measures glucose to manage type 1 diabetes in adults and children.
In late 2022 the FDA considered software that contains machine learning to be a device, and it consistently updated its guidelines for AI and machine-learning applications. The main concerns of funders and the FDA are set out in the Good Machine Learning Practice for Medical Device Development: Guiding Principles. In one presenter's experience, some application developers tend to neglect the following of them. The multidisciplinary team should work together throughout the total product life cycle so that the AI remains relevant. Clinical study participants and data sets should be representative of the population, so that the models developed on them are generalisable. The performance of the human–AI team should be emphasised. Models degrade over time and need to be retrained, and deployed models should be capable of being monitored with a focus on improved safety and performance, with controls in place to manage retraining risks. It is important to be aware of the FDA and funder guidelines.
The FDA increasingly collaborated with other regulatory agencies in the years to 2023, to mitigate the potential adverse consequences of algorithm changes for patients' safety and wellbeing and to support a favourable benefit–risk balance. The complexities are enormous, given that machine learning continuously modifies its own rules. The FDA, Health Canada and the MHRA jointly identified ten guiding principles for developing safe, effective and high-quality devices that use AI and machine learning. Although they were developed for devices, many of them, such as multidisciplinary collaboration, data quality assurance, software engineering and good security practices, and representative study participants and data sets, also apply to drug development in a regulatory writer's view. In April 2023 the FDA issued draft guidance on marketing submission recommendations for a predetermined change control plan for AI- and machine-learning-enabled device software functions. The guidance recognises their development as iterative and describes an approach that would support modifying such a function while continuing to provide a reasonable assurance of safety and effectiveness across relevant patient populations. It also gives medical writers helpful tools for preparing quality documents for submission: it sets out the goals and contents of the device modification protocols and the documentation required on the assessment of the benefits and risks of implementing the plan and on their mitigations. The plan can be submitted as part of the marketing submission to ensure the device's continued safety and effectiveness.
The EU and US approaches compared
By late 2025, in a clinical pharmacist's account, the regulation of AI in healthcare was increasingly a global priority. She read the EU Artificial Intelligence Act and the FDA's 2021 action plan for AI- and machine-learning-based software as a medical device as both stressing transparency, accountability and validation, and as both advocating risk-based governance that requires clear documentation of intended use and limitations, rigorous pre-market evaluation, real-world performance monitoring, and mitigation of bias with adherence to good machine learning practices. Where they differ, the Act classifies most healthcare AI as high-risk, subject to strict pre-market conformity assessment and mandated human oversight, while the FDA takes a total product lifecycle approach that supports innovation through a predetermined change control plan allowing adaptive updates after approval. On her reading, both aim to make healthcare AI innovative but also safe, reliable and aligned with clinical and ethical standards, and together they represent a shift toward globally harmonised, patient-centred AI regulation.