Judging whether an AI tool is worth adopting
An AI tool is worth adopting when it does well on the properties writers judge it by, when a short evaluation shows that it helps with a task the writer actually has, and when the gain it is claimed to deliver still stands once the path to a submission-ready document and the checks the tool adds are counted.
The properties a tool is judged on
In a live poll in 2026, medical writers were asked which of five properties of an AI tool matters most: accuracy, transparency, interactivity, speed or user-friendliness. Accuracy came far and away first, as it had when the same poll was put to an earlier audience of medical people, among whom it reached 80 to 90%. The writers' results still differed from the medical people's, and many writers chose transparency, which the poll's presenter read as showing how close medical writers are to the data and its quality control: they need to know where things come from. Interactivity, speed and user-friendliness each drew under 10%, although the presenter holds all three to be very useful, interactivity meaning knowing what the tool is doing and why.
In the same presenter's view, a good tool does not depend on a lot of training, and he hopes that any decent medical writer can pick it up and use it after a little experience of what the tool wants to know and why. Some tools are said to need a whole in-house training programme, and he judges such a tool not a useful system, because people will not want to do the training and will fight against the tool; beyond a small amount of training to change the mindset, using a tool should be fairly straightforward. Another presenter holds that people are increasingly impatient and do not want trainings, and that user-friendly tools are needed: even really good tools tend not to be used if they are not user-friendly. In her view, which tool fits also depends on where the writer works: whether in a big pharma company, whether the company makes submissions regularly or one every fifth year, and for which type of document.
What a tool shows the writer
No one can promise 100% accuracy from generation with large language models, by design, but such a model can still be a useful assistant if the end user is in full control of the validation process. 100% reliability cannot be guaranteed, but the end user should at least have tools that make it easier to evaluate what has been transformed and written. What mattered most to one vendor in 2024 was that the medical writer can see the sources, which are also tracked in the audit trail, the type of transformation applied and the result, with some controls on the quality evaluation of the generated text.
Evaluating a tool
Taking up a tool can follow four steps, and the first is to identify the need: a writing activity may have no relevant tool, and a task that takes a couple of minutes by hand does not need a tool to complicate the process. A tool is not worth integrating just because it is an AI tool, only if it helps with or reduces a task the writer currently has. The second step is to research and select the right tool for that particular process. The third is to work out how to implement the tool to enhance the current work and integrate it into the current work process, for example into Microsoft Office or Google Docs if that is what the writer works in. The fourth is continuous training and education, because without handholding the adoption will never be a success.
One medical writing team's evaluation, offered in 2023 as personal experience that was not yet in any guideline and not a recommendation, began with a small internal group of about four people interested in the technology, who identified the activities where tools might help. The exercise took about a month and continued as an ongoing activity. Knowing whether a single tool works for the writer may take about two to three weeks, no more, and a couple of weeks should be enough to get the initial hang of one.
The team selected tools on four parameters. The first is convenience, because writers face short timelines and want a tool that does not increase their work. The second is accuracy: a tool cannot give one answer now and a different one next time, so the prompts put to it are very important. The third is confidentiality, which the team was never sure about, so it had to be very careful what information it put into a tool. The fourth is reproducibility: whether the same activity repeated a day later gives the same results.
The evaluation sorted the tools that worked from those that did not, and in 2023 it was still work in progress. The team judged then that the tools could not be taken up completely or integrated wholly into the current process, but that it was important to talk openly about them and to use them and see how they work, so as to be ready when the time comes for the tools to replace some part of the work.
Where the gain is measured
In one presenter's experience, everybody who thinks of AI thinks of speed, and with the drafting system he is used to, a first draft takes a couple of days against the one to two weeks it used to take, which he notes is what appeals to CEOs and the people who hold the money and the timelines; he also sees dangers in focusing on speed alone. For another presenter, speed is important, because the aim is to shorten the timelines for developing drugs and getting them to market, and it is something AI should be used for, but time to first draft is not something to care that much about, although it is what she sees everyone talking about. What matters to her is the state of that first draft: whether it moves quickly to a final, submission-ready draft, or needs a lot of extra work because the AI was not set up in the way that particular document needs. So the question she would ask instead is how AI can support getting to an approved, submission-ready document. The first presenter agrees, and holds that AI can really help with a first draft but is not really all that helpful with review processes, except indirectly. In her 20 years in the industry, it is not the drafting that is the issue or takes the time but the reviewers.
Reported gains and their costs
AI companies make many astonishing claims of time saved and efficiency gained, and in one experienced writer's view each should be looked at through the extra checks and balances the tool will need, the changes to inputs it needs to function and the outputs it will generate. In her view critical thinking has never been more necessary than when appraising an AI tool. She holds that it is important to explore where it is appropriate to apply AI, and what an experienced medical writer should look for in evaluating technology to make sure it truly helps with the work. Knowing which tool is the right one for the job takes, in her view, critical appraisal and medical writing experience, and it is crucial to evaluate each tool critically for its strengths, its limitations and the specific tasks it can enhance, and, in so fast-moving a field, for how future-proof it is and what pipeline it offers. Humans also make mistakes, and they too need checks and balances to identify and correct errors.
One firm building generative AI systems for clients held in 2024 that the business case has to make sense once the cost of the AI, the technology behind it and the people running it are set against the effort saved. Its proof of concept tests whether quality is the same or better, because good-quality output is a must. For example, its AI translation system, live for four years, improved its BLEU scores year on year while the manual translation effort fell, with 100% quality control kept throughout. The same firm reported efficiency of up to 30 to 35% from generative AI document authoring and reporting.
A company presenting its own drafting tool in 2024 reported a first draft in two to four days against the 10 to 15 days it then took, with the tool taking roughly 20 to 30 minutes to produce its output and the first draft depending on the size and complexity of the study. It also claimed that the tool can remove the need for a CSR shell or mock report, because the team can go straight to a first draft, and that it makes data review meetings more efficient.
Another vendor reported in 2024 that some of its clients cut reviewing time by 50% and QC time by about 20% just by generating the first draft with its solution, and that some CSR automation use cases may gain four to six weeks.
An internal case study a firm presented in 2025 reported that AI for medical device post-market surveillance improved staff efficiency in data collection by 50%, meaning half as many people were needed and those experts could work elsewhere, and in analysis, corrective and preventive action and regulatory follow-up by 50 to 60%. It put the financial impact at around 10 to 15% of the device's revenue saved in that part of the work.