Ethics of artificial intelligence in research and medical writing
AI ethics applies moral values and principles to the choices made in designing, developing and deploying artificial intelligence. In medical research and medical writing it asks who is accountable when a system causes harm, how far a system's decisions must be explained, how its bias is handled, what wider effects its use has, who gains from the data that trains it, how an organisation builds ethics into its use of AI, and what an ethics committee asks of a study with an AI component.
What AI ethics is
Ethics is the application of moral values and principles to decision making and action, and the moral evaluation of decisions and acts. Ethical deliberation, thinking those decisions and acts through, is typically based on some shared understanding, sometimes cultural and sometimes universal, of concepts such as rightness or goodness. AI is a powerful set of technologies that humans build to carry out human-defined goals and objectives. Against the idea that anything so complicated must be magic, one AI ethics researcher holds that AI is not magic: magic is simply the expression of human will and human technology taken to an extraordinary level. AI ethics is then a set of tools for thinking about and guiding responsible choices in the design, development and deployment of digital technology: it tries to boil the large concepts of ethics down to how they can be applied to AI.
A key challenge is that AI is no single thing: it is probably best thought of as a research discipline or a marketing term, a plastic concept that includes a number of computational tools and approaches. So it is hard to apply a single set of rules, concepts and thinking tools to it, and the thinking that guides its use has to stay flexible and keep evolving.
An AI model is a system that, for a given input, can provide a useful output. Generative AI uses the same underlying methods as the AI before it to assemble language, media or other content into plausible content. Training teaches a system to produce output that seems plausible to a human audience and meets a human objective. The many decisions involved in designing and developing these tools are the source of most of the ethical questions and challenges in AI. Generative systems are probability engines: they are trained to find a path from an input to an output, and the goal is not to understand why the input maps to the output but only that it very probably does. So what they are trained to produce is plausible, believable, perhaps persuasive output, and they can be thought of as massively complex personalised autocompletes.
AI was not developed for the purposes medicine, medical research and medical writing now put it to, and the lineage of its development is crucial to understanding it. It first became mainstream not with the release of ChatGPT in November 2022 but with targeted advertising on social media and recommender algorithms on YouTube, and almost everything since is built on those concepts. Those concepts may not be fit for purpose when AI is applied to matters more important than targeting advertising or directing people to vote a particular way.
Principles of AI ethics
Various institutions promote sets of ethical principles for AI. An AI ethics team at the Alan Turing Institute, the UK's national institute for artificial intelligence and data science, tries to put the broad concepts of ethics into a practical framework for developing and deploying AI, and uses the SAFE-D principles: sustainability, accountability, fairness, explainability and data stewardship. One of its researchers holds accountability and explainability to be particularly important when AI is applied in sensitive domains such as human health and wellbeing.
By one count in May 2024, close to a hundred sets of AI ethics principles had come from trade associations, voluntary organisations and pharmaceutical companies; their fundamentals are the same, but each has its specificities.
Principles across cultures
Philosophy and ethics are in many respects culturally situated, and the perspective an ethics group brings to its work limits it in ways that are often hard to face or even to understand. Positionality reflection is reflecting on who one is, what one brings to the discourse and to an analysis, and what the limitations of that are. The Alan Turing Institute engages in it in much of its work and promotes it as part of the ethical frameworks it offers others. Its project Advancing Data Justice Research and Practice engaged communities across the globe to understand what was meant by some of the core terms used in ethics and how far there was agreement on them, and found not much agreement. In one of its researchers' view, some ethical concepts, such as trust, accountability and responsibility, have a universal appeal, though they may not be understood entirely the same way in every context.
A single universal set of AI ethics principles would be easier, but someone in China, for example, would want their own language and nuances brought out before putting their hand to such a principle, and even fairness, while it means the same, carries a different emphasis. Bringing in diversity when groups first deliberate on the principles, and checking the principles with global stakeholders, could bring them closer to the goal, though whether they will ever fully hit the mark remains to be seen.
Accountability
Accountability, in its philosophical definition, means that if a system causes harm or a bad outcome, some person or entity is responsible, and someone ought to take or be assigned responsibility. AI systems are products: companies produce them, people decide they should be used in a certain application, and users use them, and even this simplified chain is a huge chain of decision making and potential responsibility. So blame is hard to assign when systems are opaque, or when they are procured from companies that do not show everything that needs to be known about them. Explainability is closely tied to accountability, because it is unclear who can take responsibility for a decision or action that cannot be explained: the AI company, the person or entity that decides to apply AI in a given domain, or the user, who may simply be tasked with using it. In 2023 one writer held that nobody, not even ChatGPT's inventors and keepers, knows how it does what it does, so its writing cannot be questioned or held accountable.
When things go wrong, and they do, the blame cannot be put on the technology, so human accountability has to be ensured. A speaker for a medicines regulator advised medical writers in May 2024 that where AI generates material, the writer looks at it and owns each paragraph of it: it is not just a suggestion but something a person has to take responsibility for, because AI cannot take responsibility. Companies take full responsibility for their submissions: even where a contract research organisation wrote the material, the company that submitted the data is held responsible, and even if AI is used, the marketing authorisation applicant or holder is fully responsible for the information.
Explainability
A patient trusts a doctor's information without needing to know or be able to explain how the doctor reached it, which raises the question whether the issue with AI is trust in the systems rather than explainability. One AI ethics researcher holds that explainability remains the more important principle, because a person using AI is not in a one-to-one exchange with an artificial agent as equals, or even as professional and patient, but is interacting with an entire infrastructure, whose systems are sold to a customer, such as a healthcare system or a physician, and shaped around that customer's agenda. Some trust may come from interacting with a system over time and gaining confidence in its responses, but deeper trust in a system differs from trust in an individual physician, because it means trusting several layers of people and decisions completely out of view.
Explainability does not require knowing everything, but at least the essence of how AI reached a conclusion. What a patient expects of a physician depends on the procedure: some patients do not want much explanation, but a major high-risk operation calls for questions answered and risks explained, and a routine headache only for the medication. Extrapolated to AI, the end user, patient or consumer has the right to ask for an explanation, and perhaps the emphasis on explainability should be proportionate to the risk, less for a marker in preclinical research than for a decision closer to the patient, such as which formulation goes into a phase III study. Understanding the decision-making process is critical depending on the stage of development and the purpose AI was used for, because the risks are higher close to patient care and product selection than in preclinical testing. In some cases explainability is very critical, and it may not be sufficient to say that the result is right. Reproducibility goes with explainability: when a system gives the same thing consistently, confidence and trust in it can increase. Different circumstances may therefore require different levels of explainability, with no one size fitting all.
In work on whether these tools are compliant for medical communications and writing agencies, explainability is always at the top of the list: the agencies' pharmaceutical clients avoid risk and want their vendors to avoid it, so an inability to explain is a challenge.
Large language models draw on a corpus of knowledge that already carries human bias. Asked in May 2024 whether an AI tool would amplify that bias, a patient held that it is something to pay attention to, and that a grey-box model whose reasoning can be understood would already do a lot. Such a model would also contribute to a person's own assessment, since in an individual decision a person compares the options as best they know before deciding to act, as people do when they seek second opinions from three, five or six doctors. In his view, when such tools reach patients, the general public needs to be made aware of what a medical device label, such as class I or IIa, signifies, because a black box cannot be delivered to patients directly without a supervised model.
Bias
In one AI ethics researcher's view, bias cannot be removed from AI. It is there especially because AI is built out of the histories of people and societies, and bias is shaped by the fact that people and their societies are not uniform but complex and full of irregularities, which is what is meant by culture and by society. People like having opinions and preferences, so expecting technologies that are extensions of human society to be other than biased is, in his view, an absurdity. Bias is in part a product of cultural values and norms, so removing it raises the questions for whom, according to what definition and in what context.
From an engineering perspective, the work therefore starts from the bias being there: acknowledging it, admitting to it and stating the limitations, how well a system works in which situations and where it probably does not work well. Thinking that perfection will ever be achieved is the fallacy to be aware of. On this view eliminating bias is a myth, and the aim is to minimise bias to a level acceptable in the context at hand. Introducing diversity and getting diverse opinions should help people become consciously more aware of bias, and so see where it can be minimised. An AI without bias is like a human being without bias, which humanity has not achieved. So a demand to an AI developer such as OpenAI that it reduce bias would perhaps meet the questions how, for whom and in what circumstances.
Humans also have bias, and several problems in the pharmaceutical industry, such as the selective reporting of studies, meta-analyses and how algorithms are fed, existed before AI, as did misinformation and sensationalism about vaccine hesitancy. This does not discount AI bias, but presenting AI as causing bias while humans have none oversimplifies the problem, because people feed the algorithms and their bias translates into the AI's output. What goes into the algorithms is within human control and can be examined.
Models such as ChatGPT are trained on real-world content scraped from the public domain, much of which is structured by an inbuilt text and image bias that obscures heterogeneity of gender, race, ethnicity and other characteristics. AI researchers note that the data large language models encode are more likely to represent the perspectives of people who already occupy considerable internet real estate than those of women and people of colour, who spend less time online and have less access to determining online content. So idea starters and writing prompts that seem appealing as efficiency levers may already be culturally and racially biased, or at least lack cultural sensitivity and specificity. In one writer's 2023 view, medical writers who do not use these tools deliberately through an equity and inequality framework will unwittingly reproduce bias.
Ecosystem effects
Accountability and explainability do not embrace all the challenges generative AI raises, and the rest are its ecosystem effects. AI is a product, a service and a discipline, often sold as a way of becoming more efficient and working faster. That raises the question of what will be expected in one, five or ten years of people whom a given tool makes faster today, and what they will have gained from the gain. So caution is needed about the claims made for AI, because they may lead down somewhat dark paths. The companies that make AI have serious questions to answer about their ethics and the way they operate: labour issues over who trains and annotates AI, the huge environmental costs of AI tools, and the concentration of power in the handful of companies able to make the most powerful AI. All of these need to be considered in choosing how and how far to use AI.
Generative AI is very powerful and impressive, but it has to be understood holistically. It is not enough that the app in front of a person is useful: AI's human and biospheric effects are significant enough to require a careful, sober evaluation of the benefits, the costs and their distribution across society. Today's killer app may offer short-term advantages, but the safety, health and wellbeing of all people, all organisms and the world have to be ensured.
Who gains from the data that trains AI
Copyright is in a way a trade and a bargain between the copyright holder and those who wish to use the intellectual property for the greater good. Users of AI are, as a community, training its models for the companies' financial gain, and medical writers who write very detailed prompts would seem to be doing even more of that work for the benefit of a commercial entity, which raises the question how that can be equitable and how better equity with AI companies could be achieved.
In one intellectual property lawyer's view, transparency will go a long way, as the least common denominator of accountability. Developers may initially, perhaps with the best intentions, have treated the reuse of data or of copyright content as using the byproduct of another process, of no use to its original owner, like recyclable refuse. In his view societies had by May 2024 come to see a data trail not as refuse but as part of a person's increasingly digital persona, from which adverse inferences can be drawn, as in insurance contracts, so some accountability has to follow.
One adviser to AI startups compares the exchange to social media platforms, which their users made the most powerful in the world while some of them benefited tremendously, a trade-off people are not always conscious of. The freemium business model gives something of high value for free, which twenty years earlier few things were, but the name is a misnomer, because the exchange is data for value. This puts the user in a difficult, precarious position, because keeping their data from being used in an abstract way to train a future model means turning down something that makes life or workflows easier. There is probably work to be done within the startup community to raise these issues, but he sees no fundamental change coming, so the question is more one of what users are willing to accept.
Ethics by design
In one pharmaceutical bioethicist's picture of the industry's position, patients are at the centre of everything, with a legitimate pressure to get medicines, products and devices to market sooner; ethics is foundational, with some areas of zero tolerance and no compromise; and trust is easy to break and difficult to build and sustain. Innovation with technology is part of the same picture, and it is not using AI, and not getting used to using it the right way, that would inevitably lead to collapse. Cost is another factor, in an industry usually associated with profit, and the balance among these factors is always delicate, none of them worse or better than another. Ethics at the centre means doing the right things, communicating the right things at the right time, and building and sustaining trust.
For the industry, AI's benefits include accelerating innovation for the benefit of patients, against historical submission timelines that COVID-19 showed to be unacceptable; efficiency in cost and resources, for the benefit not only of the industry but ultimately of the community, giving more to the people who need it most; and the possibility of reusing data. Its risks and challenges include accountability, explainability, data privacy and integrity, which has reached unthinkable dimensions, and bias. Informed consent, the gold standard for autonomy, raises the questions how it can be reconciled with data privacy in big data and data reuse, and how a participant who entrusted their data can be given valid reassurance that it will be used responsibly without necessarily going through the informed consent process, which is sometimes not feasible.
Responsible use of AI means practising AI ethics pragmatically, without a textbook of prescriptive rules or controls that hamper innovation. In ethics by design, ethics is not an afterthought, a checklist or a tick box completed without thirty seconds' thought, but is built in from the proof of concept, with controls to check in, at the forefront and the centre. Ethics by design accepts that sometimes the right thing is to stop the project or pause the innovation and go back to the drawing table. It is a deliberation and a collective decision with all the relevant stakeholders across the table, including the users of the technology, patients, patient organisations and the people whose data is used, because a single person does not know or have all the answers. In the bioethicist's view, understanding an ethical dilemma means spending 90% of the time understanding the question or the problem rather than jumping to a solution. On receiving a consultancy question the instinct is to listen in order to respond, but asking the person to elaborate often shows the issue to be very different from the one first understood.
Ethics by design takes smart risks, which are acceptable risks with mitigations in place, not street-smart risks taken in the belief that nobody will find out. It is human-centric and human-accountable, with AI enabling, catalysing and supporting, and it rests on people, process, governance and communities. Its culture is the mindset that a rule book might not answer all the questions and that ethical decision making exists to navigate the grey zones, together with recognising missing capabilities and investing in building them, with learning built into every day. Many in the industry would say they have too many processes, written standards, operating procedures and instructions, and the question is how far ethics is embedded in them and whether what was developed ten years ago is still fit for purpose. Governance means a framework, mechanisms and oversight, a place to go when something goes wrong, the resolution of issues, and learning and sharing so that others do not make the same mistakes. Communities such as trade associations are where learning and best practices are shared and cross-fertilised. Decision making in ethics by design is values-based and collective, recognising the risks and challenges and that the work is in progress.
Trade associations have published material on data ethics and AI ethics, meant as directional instruments that trigger reflection and help ask the right questions, not as a spoon-fed answer to all of them.
Ethics review of studies with an AI component
A research ethics committee reviews a study with an AI component as it reviews every study, but such studies bring certain problems and challenges of their own. In the year to May 2024 the studies with an AI component reaching one research ethics committee generally involved monitoring, prediction and diagnosis, as in digital apps for health monitoring in neurodegenerative disorders, platforms using AI to predict which patients are at risk of sepsis after surgery, and systems for autonomous screening of diabetic retinopathy. Such screening lets a physician decide whether a patient needs immediate treatment or can be put on a monitoring scheme of perhaps 12 months.
Data sharing raises the big issues, because much of what these studies involve is the collection of data and what happens to it. Patient data is a hugely precious resource, and most patients are positive about sharing it as long as they understand exactly what is happening. So the relationship between the developers and the participants or organisations needs to be clear, transparent and clearly defined, because patient trust is easily lost if patients feel their data is being taken from them and sold to companies for profit. A committee therefore asks who owns the intellectual property, what will happen to the tool created from data the participants provide for free, what data is necessary and proportionate for the model, whether more is collected than needed, and what will be done with it once collected. It asks whether the data will be stored and transmitted securely, without breaches, and whether it will be used only for the study, sold on, or stored for future studies. These questions have no right or wrong answer: what matters is whether it has been made clear to the participants and organisations involved exactly what is happening to their data and that it will be handled securely.
Autonomy is the capacity to make an informed, uncoerced decision. Most people are happy with aids that help them drive more safely but not with taking the human out of driving completely. In one committee member's view in May 2024, the same applies perhaps even more in healthcare, where human autonomy should supersede machine autonomy, though the situation might be different in five years. Decision-making tools should assist humans, helping doctors identify people who may need additional help or who have certain diseases at an earlier stage, with the medical staff and the patients both involved in making the informed decisions. Autonomy also covers participants' rights to privacy and confidentiality, so the data used in these systems must comply with the current legal framework, including the GDPR, and participants' consent must be full and valid throughout.
Transparency, explainability and intelligibility require that the purpose and use of the AI system and how the data is used be clear and open to all the participants and organisations involved. Some of it is too complex for the minutiae of how it works to be explained, but the general concepts need to be explained in a way each audience can understand. Patients need an informed consent form and a patient information leaflet that make it clear, and developers and regulators need an explanation of how the system works at a level high enough for them to make decisions. In the committee member's view, the committee needs such an explanation too, showing the researchers' protocols, their process for data governance and how they will control the systems.
The risk–benefit ratio is the big thing a committee looks at in every study, and in a study with AI it looks less at adverse events than it would in an ordinary drug study and more at what is happening with the data. Researchers consistently say there is no possibility of harm because they are only collecting data and asking people to fill in questionnaires, but this is not true in all cases. For patients with mental disorders, or on sensitive matters such as miscarriage, abortion or suicide risk, a questionnaire raises the risks, which need to be acknowledged, with something put in place to help participants deal with them. AI should not harm people, and the harm to guard against is mental as well as physical, physical harm being perhaps the less likely, so safeguards need to be put in place.