Deciding what confidential material may go into an AI tool
Whether a document, a data set or a single passage may be entered into an AI tool the writer does not control is a confidentiality decision, and it is taken before the tool is used. It is settled with whoever the material belongs to, and it turns on what is entered, what the vendor says happens to the data and where the system runs, for each of the tools writers reach for: the general chat assistants, the closed systems vendors build for clinical documents, and Copilot inside Microsoft 365. Reviewers, editors and publishers are held to the same constraint, and a ban on the tools raises the question whether it will be enforced.
Who decides what may be entered
The advice of a March 2024 webinar on AI in systematic literature research, asked about material that is not in the public domain, is to put it into no model without the client's permission, because the providers use what is submitted to train their models, which the speaker did not consider a big deal in itself. The point should be looked into before the models are used at all, to avoid issues with clients. Even where the interaction is known to be fully confidential, the client should be told: if the client does not want the model used, it should not be used, and a client who wants it used will say so at the outset.
A lawyer at EMWA's May 2024 symposium in Valencia warned that confidentiality may restrict the use of open tools: whoever handed over a project would not be happy to see a whole block of its confidential text put into ChatGPT. Where the material is patentable, putting the subject matter into the public domain invalidates the patent in virtually every country, a high-stakes step he compared to a permanent delete button. The risk put to him for writers was in uploading intellectual property on behalf of pharmaceutical clients and contravening compliance agreements. Whether to put a patent application into the commercial edition of ChatGPT on the provider's undertaking that it will never be shared is a decision he would rather the client took than take for them.
At an EMWA webinar in 2026, a questioner reported that what blocked people from using the tools was compliance saying that data is not secure if it goes into an AI tool that is learning from other clients, a constraint the questioner saw as very applicable to freelancers and writers at contract research organisations, though not to the speaker from a large pharmaceutical company. The answer, from that speaker, starts from what her company's data is: it is generated from patient data, it is sensitive, and so it cannot simply be uploaded outside the firewall and used for AI. She encouraged everyone, especially anyone in a small organisation, to look properly into the legal aspects. Patients' trust is a second reason: the industry has had problems with patients not trusting it for many years, and sharing personally sensitive data could cost it that trust, so it has to be very careful. This may mean the industry takes longer to adopt some aspects, but in the end it has to make sure that it shares no patient data that could then become available to people who should not have access to it.
Stripping and limiting what is entered
Privacy has to be addressed whenever a text-generation tool is used: careful consideration should be given to including only safe input data, so that confidential or proprietary information is not shared inadvertently.
Asked about the security of add-ins, with an attendee's comment that confidential information on writers' computers should not be accessible to AI tools, the presenter of a December 2023 webinar on AI tools for the medical writing workflow agreed, noting that the session had set out only to explore the tools. To use a tool for paraphrasing, the numbers and the like should be removed first, since the tool can read them, and custom GPTs should be used where needed. Confidential information should not go in at all: no clinical content, no regulatory content and so on.
One medical writing team, described at an EMWA webinar in June 2023, used free proofreading and AI-detection tools only on review papers, and not on original research whose confidential data would have to be uploaded. Asked how safe such tools are, when much of the data for a publication is confidential and using the tools on it would create a lot of fuss, the presenter had no complete answer and was still finding one, as the presenter thought everyone was.
What vendors say happens to the data
In September 2023 writers at TFS HealthScience, writing in EMWA's Medical Writing, put privacy, confidentiality and accuracy at the top of the challenges that might arise with AI tools, noting that confidentiality is particularly sensitive because many AI services use input data to keep training the model, which becomes a problem when working with confidential material. The June 2023 presenter's tip that paid versions might be safer had to be taken with a pinch of salt, because an AI tool learns from what is put into it, which the presenter called a catch-22.
The September 2023 writers reported OpenAI's option to stop input being used for training and a planned paid business version giving users more control over their data. As of March 2024, the webinar's advice relied on two vendor controls: ChatGPT version 4 offered an option to erase the chat history, which the speaker said meant that none of the data entered would be used and the conversation would be fully private, and its Teams Pro version was clearly described as completely private, with nothing trained on its data, which left the speaker entirely comfortable putting any sensitive information into it. The question put to the lawyer at the May 2024 Valencia symposium turned on the difference between the enterprise edition of ChatGPT, whose terms state that submitted material will not be reused to train the model, and the public-facing version, whose users may not realise without reading its terms that they are giving the material away. Asked what could be done if the provider broke the undertaking in its enterprise terms, which a lack of transparency might keep users from knowing, he answered that it depends partly on how paranoid one is. Thinking aloud, he added that if he had to bring a case he would probably consider France, a very good place to take on the large platforms, where terms of Facebook and Google claiming rights over anything uploaded had been invalidated and astronomical daily fines had run until the terms were changed.
A panellist at the same symposium reported that in the preceding two weeks OpenAI had added a setting that turns off its ability to learn from a user's input, which seemed to be available to the public as well, while saying that the panellist had not read the terms behind it, which writers should read, that no legal or practical advice was being given, and that the change might have come from public pressure over privacy or from the provider no longer needing more data for its next version.
For Copilot, the September 2023 writers relayed Microsoft's statements that Copilot is guided by its AI principles and Responsible AI Standard, with its privacy statement as a further guide, and that its AI tools would run locally on the user's device, so that no document data could leak to other users.
The operations director of a supplier of contract clinical and regulatory medical writing, whose company had chosen Copilot as its preferred tool, answered the privacy objection for that tool at the May 2024 Valencia symposium from Microsoft's published policy: prompts, responses and data are not used to train the language model; the use can be managed at user level, so that an IT team can lock down what each user reaches, including by the clients they work on, and someone at a large company cannot wander into finance material in the same SharePoint environment; and the data stays within the Microsoft 365 environment. In ChatGPT, by contrast, which has a switch to turn off its learning, a writer can paste private, confidential information from a browser that then goes into the language model. He had no doubt that Microsoft used the data in some way, but would not get too bogged down in it, because a company already using the hosted Microsoft 365 has its documents sitting on Microsoft's servers anyway. Asked on the panel for the quick wins that give the most benefit for the least effort, he suggested turning off the ChatGPT setting that sends information back into the language model, taking protocols, statistical analysis plans and the like from public sources such as ClinicalTrials.gov, practising on those, and working out which use saves time.
A system administrator who deployed Copilot for a medical writer gave the same reading of Microsoft's privacy statement in the Summer 2024 AMWA Journal: Copilot for Microsoft 365 follows the company's standing privacy, security and compliance commitments to commercial customers; prompts, responses and the data Copilot accesses are not used to train foundation models, and Microsoft does not monitor the use; a prompt draws only on the emails, chats and documents the user is authorised to access; internet searches can be switched off for a query, so that only data inside the user's own environment are used, which can offer added protection when sensitive data are used in a prompt; and further restrictions apply to data inside the European Union boundary.
Where the system runs
In the June 2023 account, a tool inside a secure environment can be used as any other technology is: some large organisations were preparing their own platforms with such tools integrated, and some companies had tools for their internal teams only.
In September 2023 two writers from a company that builds a sector-specific AI tool for clinical trial data thought that AI in medical writing may come not as stand-alone tools but within platforms, and that such a walled garden makes sense because of reluctance to upload intellectual property, personal data or other sensitive information to open platforms while data ownership and protection were being debated. They argued that data privacy is an inherent risk of any technology rather than one specific to AI, that some AI platforms carry the risk of being internet-based, that open systems carry risk even without AI, and that some emerging options let developers build a language model in a secure environment.
A software vendor at EMWA's May 2024 expert seminar on using AI tools in practice drew the line at the internet connection. A generative system that is connected to the internet and writing a clinical study report is a huge ethical problem, while a closed-loop system using models to improve automation is a different proposition, in which what is uploaded should come back fully accurate. ChatGPT, Microsoft Copilot, Alexa and Siri all need the internet, and at the regulatory standard of this work there is no room for them: sponsors will not allow it, and for everybody in the room it is 98% a non-starter. To put a software provider's system in place, it has to be behind the customer's firewall and on premise, and the vendor spends three to six months, sometimes a year, going through IT security and privacy documents to ensure that it is closed, with no access or data breach.
The chief commercial officer of a medical writing automation vendor, speaking at the May 2024 Valencia symposium, gave seven reasons why a general chat assistant cannot write a whole clinical study report, and presented them as the case for a closed-loop system. The question comes up on the vendor's calls, and one pilot client expected exactly that. First, a contract research organisation working for twenty or fifty sponsors has to keep each one separated in a closed loop that pulls in no data from the public domain, whereas ChatGPT draws on sources across the internet. Right behind it come data privacy, proprietary intellectual property and data breaches. Third is keeping up with regulatory writing standards: many sponsors have their own templates and tables to customise, and customising a template can sometimes take two to four weeks. ChatGPT would not work well with each sponsor's own writing style and template, because the machine learning needs a fresh set of models not trained across the public domain, which the vendor installs as a clean slate that learns on one sponsor alone, without sharing that information. No software company can get past the privacy and IT reviews even to try sharing it. Sixth is each writer's style, down to one writer's preference for could and another's for would, which the clean slate learns over time, whereas ChatGPT will not tune its models to a writer's style behind a firewall. Seventh is human interpretation of data analysis and oversight: writers, QC and QA look at what the AI has produced and often edit it, overriding what it puts out. Data privacy and proprietary information he called probably the number one reason for a closed-loop system. Asked by the vendor why AI cannot be used to write clinical study reports, ChatGPT itself gave proprietary information, lack of understanding and context, regulatory requirements, limited ability to provide critical analysis, lack of human input and oversight, and ethical and legal concerns.
He did not tell writers to stop using the general tools, calling them helpful across the process for things such as a synopsis, only to be careful with them. An executive he knew, who oversees one of the executives overseeing Microsoft Copilot, reported that Microsoft was gathering more use cases and more human training on its models for specific use cases, aiming eventually to go down that route, which would take a while. He added that ChatGPT also wants to go more private and offer application-specific versions in certain industries, which will take a lot of human training, human input and use cases, and that bringing a public-domain tool into the private sector, with its data-breach and privacy concerns, is a difficult task, though he expected they would manage it.
Reviewers, editors and publishers
In the December 2023 account, the constraint does not stop with the author: a publisher or reviewer who put a freshly submitted paper through a public AI detector would cause a data breach of the same kind, so the reviewer and the editor have to be as careful as the author not to put the paper and its data into such a tool and compromise the results.
Asked at a 2024 EMWA webinar on publisher perspectives on AI whether they use the general tools or have built more secure ones of their own, Elsevier answered that it uses its own proprietary AI system and is not allowed to load any content into external systems. Taylor & Francis has an internal platform for editors and staff and a rule its speaker called very well known: no work product and no submitted content goes into a third-party platform at all, and out of an abundance of caution the same instruction goes to its academic editors, editorial board members and peer reviewers. Springer Nature likewise currently allows no third-party platform and has built its own language model, through which it processes its own and open-access manuscripts to generate summaries. JAMA asks its peer reviewers not to put confidential manuscripts into AI models, because they are confidential.
At Taylor & Francis, people were not using AI in their work beyond language improvement, because they saw no other purpose for it, which its speaker called sometimes a challenge. A senior editor there found an everyday use the hard thing to find, because as an editor she cannot feed content into a tool. The best practice she gave is to use only company-approved AI platforms and to feed no confidential, sensitive or proprietary information into third-party platforms. At the author's end, one article's acknowledgements disclosed the use of ChatGPT-3.5 for translation and grammatical correction, and its editor reported conversations with authors who had already put parts of manuscripts into ChatGPT or other third-party platforms before submission, in small sections rather than whole manuscripts or large sections, which she called a discrepancy from best practice and an area for better education.
Bans and their enforcement
In December 2023 many policies on the use of AI were being drafted and implemented in companies.
In the June 2023 account, where companies had a complete no-ChatGPT policy that kept the tools off company laptops, people normally used them on their phones or personal laptops instead. A panellist at the May 2024 Valencia symposium reported the same of friends at law firms with blanket bans on uploading their content into these tools: all of them open a personal laptop and do it anyway. In the panellist's view, people will still use the tools whatever the rules say, and the rule has a perverse effect: once it exists, the people breaking it take more care to do it correctly and not to be caught, so the practice is hidden, carried out more responsibly, and unknown to the organisation. The panellist's advice, also put to academic publishers drafting AI policies, is to decide before writing a rule whether it will be policed and enforced, and if not, to have best practices rather than a policy that cannot be enforced.