Evaluating AI output in medical writing
Whether the output of an AI tool can be used depends on the errors such tools make, from hallucination to failures of reasoning, and on what the model was trained on and which work that puts out of its reach. It also depends on how the tool performs on a real task, on whether its errors and the route to its output can be found, on whether the person checking it can judge it at all, and on how much checking the output needs as a starting point rather than a finished product.
Hallucination
Language models generate incorrect statements and even fabricate false information, and these fabrications, called hallucinations, are unintended text generations not supported by the input data or the context, yet stated with utmost confidence. Evaluating and mitigating them is challenging, because evaluating a hallucination is subjective and based on the user's expectations, and mitigating hallucinations could come at the cost of fluency or naturalness in the generated text, which can harm the user experience.
In one study, four of five ChatGPT-generated articles were significantly inaccurate, though all were written in language convincing and coherent enough to make the errors difficult for non-expert readers to spot; in another, ChatGPT answered questions within a wide range of medical fields accurately. So language models can generate accurate content, but there is also a high risk of inaccuracies making their way into the text; they are language models, not knowledge models. In 2023 it was up to users to keep hallucination in mind and check that generated content is factual, though this was likely to improve with further technological developments.
A publisher that generates article summaries with AI found that in its earlier days the AI was hallucinating most of the time, putting points not presented in the paper into summaries, interpreting the results and exaggerating them, so all generated content had to be fact-checked by a human. Hallucinations still arise when a large language model is trained on data whose input is controlled. Before the publisher fine-tuned its models on its own data, where such studies had already been published elsewhere, a plain language summary of an article could be 99% accurate with 1% taken from that published data that the article never mentioned, such as a mouse model. The publisher's subject-matter experts then checked word by word whether everything in the summary was present in the original article and, where something was not, where it had come from. Sometimes it came from the original article's references, since the model took them into account along with the text sent for summarising. By 2024 such hallucination had been reduced a lot with newer model versions and the mistakes were very minute, but exaggeration and hype terms remained, such as research called revolutionary or a finding said to be proven by a clinical study when the work was still preclinical, which is why a medical writer has to intervene and read everything thoroughly.
For a fake reference, reference managers can check, though they might require manual checking against PubMed or PubMed Central, and for AI-generated text there is a tortured-phrase detector, with many other tools coming out in 2024. One journal editor had seen no tool that is foolproof at catching everything that is fake, and the concern is that as soon as such tools exist the models get better and smarter.
Reasoning and statistics
Posed the Monty Hall problem, with three doors, a car behind one and the host opening another to show a goat, ChatGPT gives the correct answer, to switch, most likely because its training data included a written account of how the problem is solved. Posed a "dumb" version in which the doors are clear glass, so that the car is visible and anyone would stick with their choice, it always suggests switching. Two writers took this in 2023 as a sign of the model's inability to reason as a human does, making deductions from premises rather than probabilistic inferences from word frequencies, which is exactly the challenge of interpreting statistical data from new drugs, where the margins for error are significantly smaller, so language models alone cannot be relied on. Many generic language models can create authentic content but do not always perform well when the content is novel or its frame of reference is new, because language models can only produce content related to the data they have been trained on. ChatGPT can "solve" maths problems it was not specifically designed for but cannot interpret statistics, and a language model can produce results that are grammatically correct but factually incorrect. Generic language models are known to make up information or cite references when they have no information.
On one researcher's account in 2024, ChatGPT could not solve a fairly simple question that most people could answer after a moment's thought. On the same account, generative systems fluctuate in their ability to solve maths equations and do far more poorly around the Christmas holiday than at other times of the year, for reasons no one is sure of.
What the model was trained on
Communicators using generative AI should understand the tools, as every good craftsperson knows their tools, and in medical writing and editing that largely means understanding GPTs. Consumer GPTs, which hit the market with DALL·E and then ChatGPT in 2022, are trained on the internet and ingest online content good and bad. They have no access to paywalled peer-reviewed content, and their training cut-offs may leave the most recent content months or years old. High-quality medical communication requires accurate, detailed and current sources, so most content consumer GPTs generate may not meet that standard, especially on more complicated or technical topics. By 2024 more consumer GPTs had added real-time web search, which for many casual users made the training date a distinction without a difference. The human quality of their language can lead users to trust their content more than they should, and they provide inaccurate and biased information, plagiarise their sources and have a limited ability to generate meaningful technical language.
Proprietary or enterprise models are trained on industry-appropriate information, including paywalled articles, and then on the organisation's own data, which only its members can reach. They are often sparse expert models of under 100 billion parameters, against ChatGPT 3.5's 175 billion, and can be more accurate than larger general models because the data they ingest are more specific to the users' needs. Like consumer GPTs, though, they tend to prevaricate, hallucinate and plagiarise.
The accuracy and reliability of AI-generated text depend on the quality and quantity of the data used to train the model, and a model not trained on sufficient, relevant and diverse data may generate inaccurate or misleading information. OpenAI is not transparent about how it ingested its information. A reverse-engineering exercise by the Washington Post found that its sources included an open-access peer-reviewed publisher, the New York Times and Reddit. One speaker from the author-services industry calls the Reddit content regurgitated Reddit, not where a scientist or medical writer wants to be, and the mixture a polluted data set.
Because of the problems with where a general model's content comes from, the same speaker recommends that writers consider more specific tools for a purpose such as a literature review, which give a summary with real sourced information that the writer then checks against the article. Faced with 300 articles that might be relevant, a writer might summarise them with AI, use the summaries to pick the 15 or 20 worth going into, and read those papers. What matters, in her view, is how the tools are used, more than a binary judgement of good or bad use.
In a recent test among patent lawyers, by one intellectual property lawyer's recollection, a tool did the 2018 European patent examination perfectly and failed the 2019 paper completely; where it had the exam paper results, it did brilliantly, and where it had not, its answer was like a complete fabrication. He takes such a test as a good way of working back to what a tool has ingested. What he calls a serious bug is that the tool gives an answer with total confidence irrespective of the quality of the answer, and he expected that within six months to two years the problem might be sorted out, or a warning would say it cannot answer with any degree of confidence.
Med-PaLM, built on Google's large language models, can answer a range of medical queries with fairly good accuracy, and in 2024 one researcher understood its latest iteration to meet a fairly high standard for accuracy. Critics point out that even at its best its information is likely to be incomplete and may be missing important detail, so that, on the same researcher's account, while promising it cannot be relied on without close human supervision, and this plays out across the entire range of AI products.
Work the general tools do not reach
In one writer's opinion in December 2023, general AI systems will not help much with analysing data or with writing clinical and regulatory documentation unless they are specifically trained or made for it. Writers in regulated industries such as healthcare may not gain as much benefit from AI writers because of the requirement for accurate and complete information, and organisations should develop processes for evaluating the accuracy and completeness of AI-generated content.
As one practitioner put it in December 2023, hallucinations are really common, and an AI tool that makes things up is not acceptable from a regulatory point of view: it is a known issue that regulators have a real problem with, for good reason.
Performance on publication tasks
In a test reported in May 2023, ChatGPT produced from a sample poster a target journal recommendation, a manuscript title, an abstract, keywords and a lay summary. Three blinded reviewers working in the publications team of a contract research organisation rated the results on a scale from 1 (worst) to 5 (best), along with compliance with instructions, language accuracy and language style. The median scores were 2.5 for target journal selection, 3.5 for the title, 2 for the abstract, 4 for the keywords and 3.75 for the lay summary, against 4.5 for following instructions and 4 each for language accuracy and language style. The reviewers' comments named awkward wording, questionable journal selection, and poor content and language in the lay summary and abstract, especially poor reporting of results; one reviewer's overall verdict was neutral and two were negative. The authors concluded that ChatGPT performed poorly at producing most of the content requested, although it performed well at following instructions and at limited linguistic tasks that did not require an understanding of the content.
A second test reported in May 2023 asked whether ChatGPT could produce an abstract meeting the CONSORT for Abstracts requirements. The model first shortened each section of a randomised controlled trial publication by two-thirds to fit its data entry field. The shortened sections were combined, enriched with any missing CONSORT-related information and entered again with the instruction to produce a CONSORT-compliant structured abstract under 250 words and a title, and the output was assessed against the CONSORT for Abstracts checklist and qualitatively. The abstract ran to 277 words. Of the checklist's 17 items it reported 10 (59%) and did not report 5: participants, blinding, numbers randomised, numbers analysed and trial registration, while 2 (12%) were considered not applicable. Its text did not always match its sections, was repetitive and was at times hard to comprehend. The authors concluded that ChatGPT did not comply with the checklist, generated a poorly written abstract, and could not be considered effective at producing scientific abstracts of sufficient quality.
Errors the usual checks miss
One writer gave a general model a three-page document whose data she knew very well and asked for a summary. It returned a one-page version that rearranged things and at first glance looked like the data she wanted to present, but it had moved some of the numbers around, and had she not paid attention she would have presented wrong information. Data shared with and submitted to health authorities need to be right. She agrees that humans also make errors and has no comparison of how much data a model gets wrong against a human writing, say, a clinical study report. But she holds that the errors seen with these models might be more difficult to identify, because everything looks right while the numbers may have been turned around or be completely different.
By one regulatory writer's account in 2026, a client's AI summary of congress notes from medical experts got a lot of things wrong and was unusable, and her team made a table in a couple of hours that better summarised the information the client wanted.
Large language models and other generative tools make kinds of mistakes that humans are not used to finding, while the quality-control systems in place were designed to find human errors. Those systems check a five in the document against the source to make sure it is a five, and do not go through the original source material looking for signals the writer missed. On the account of a writer who works on an AI tool, that is where an AI tool, assuming it knows how to analyse the data, can be a big advantage. He and a junior writer had spent three days going line by line through a client's 180-page laboratory safety table and settled on what they thought were the four clinically relevant findings in it. A year and a half later an early version of his tool took five minutes over the same table, confirmed three of the four, showed that the fourth did not meet the criteria he had wanted to use, and found a fifth that he had missed.
In May 2024 a medical writing company that had been developing its own AI tool for five years ran it, at a well-respected pharmaceutical company's suggestion, over a clinical study report that the pharmaceutical company had recently submitted to a regulator. The tool found two clinically relevant differences that the report did not mention: one of over 5% between groups in treatment-related adverse events of NCI grade three or four, and one of over 10% between groups in QTc elongation on the ECG, where the submitted report said there were no clinically meaningful findings in the ECG results related to safety. A writer from the tool's company describes the pharmaceutical company's writers as very good medical writers who are human and make mistakes, and holds that AI can at least give the security of not missing any safety signals, and speed.
Tracing how the output was reached
A writer who had used an AI system for a clinical study report disagreed with some of what it had done, did not know where other parts of it had come from, tried to work that out, and in the end binned it after wasting two days and started again from scratch. On the account of a writer who works on an AI tool, a system that gives no transparency can in the end be less valuable than starting from scratch.
In 2024 one speaker from the author-services industry expected the next generation of tools to bring chaining: instead of an output to be trusted, a trace of the steps the model took to reach it, so that the process can be checked one step at a time. Anyone who has run a data analysis through ChatGPT can already see the Python code it wrote to reach its conclusion, and can check whether that is the process they would have used and iterate on it if not.
Whether the checker can judge the output
Asking a model to improve the syntax, grammar and readability of a text leaves the user to decide whether the result is good, which they can do if they have the knowledge, and asking it for three alternatives leaves them to decide which are good. In a language they are not proficient in, or about a domain they are not familiar with, they will not be able to judge the output, and the trap one translator names is using AI-generated output that a human has not checked, especially in languages the human is not familiar with.
Having translated the website of the largest cancer centre in her state from Spanish into English, she was asked by the client to do German and said no. Asked whether she could not use AI, she said she could not, because she had no way of knowing whether the result would be good. She could have said yes and made a lot of money, and the client would not have known either; the client's misconception, as she puts it, is that having the tool means being able to do the work.
The output as a starting point
A writer using these tools is still responsible for the output. The output is never an end product, and the danger lies in not knowing that; once the writer knows it is a starting point, to read over, change, add to and organise differently, it is really helpful.
Generative models in 2023 were primarily sophisticated prediction engines whose output lacks true comprehension, critical thinking or consciousness. So the texts they generate tend to be lengthy, articulate replies that could include plausible but inaccurate or biased information, and users must review the output critically and always check its accuracy, appropriateness, context and actual usefulness before accepting any result. Especially in medical writing, text-generating tools should be regarded as assistants or a tool in the toolbox, providing a starting base or helping to overcome writer's block, and the human ability to comprehend context, strategise, critically evaluate and convey nuanced emotions remains irreplaceable. Those abilities play a crucial role in moderating and refining AI-generated output.
A starting point helps where a writer has to describe again something already written up many times, such as an operational or surgical technique, a laboratory protocol, or the same assay used across ten papers within one research scope. Completeness requires including it, but after writing it up ten times the writer runs out of ways to phrase it, and the model can be asked to rephrase it or make it shorter. A model also helps in finding titles, and can be asked for alternative ones. Asked where the weak points are and what has not been discussed, it sometimes gives quite amazing answers, though the work on top of it still has to be done.
Asking a model to rephrase a passage and give three alternatives can be a great starting point for breaking a writer's or a translator's block, though the alternatives are not necessarily ready to submit. One translator finds three the sweet spot, since ten would take too long to pick from. Asked to combine its three alternatives into one, the model gave a sentence that had given up and got confused, saying the same thing twice. Human intelligence needs to be in the driver's seat, and whatever comes out of the AI cannot be trusted blindly.
How much checking the output needs
Writers are responsible for the content they generate with these tools, as they were for the content they created before them, and must quality-control every single step of the process.
One reviewer running systematic reviews with a language model started by checking everything and moved to the sampling method used before AI: 100% of the first hundred screening results, and if those hundred justifications satisfied him, 80% of the next two hundred, then 50% of the next, and so on down to 10%. After a task has been done enough times it can be trusted for its reliability, and out of a great many tasks he had stopped checking only some three particular easy ones, with everything else still checked. His checking in 2024 was significantly less than a year earlier, though he was still not fully confident in everything and still checked a lot. In screening, a record wrongly included is picked up at full-text review and excluded later, but whether all the excludes are real excludes can never be certain.
A study from 2015, as he recalls it, assessed the title-and-abstract screening decisions of 86 human reviewers, over 300,000 decisions, and found an error rate of about 10%; he is confident that the model's error rate, if there is one, is not 10% but more likely a fraction of a per cent. Asked whether he had ever missed an important article with ChatGPT that would have changed his conclusions, he answered not that he knew of: nobody had told him of a miss in a review published about a year earlier or in any client report he had delivered. He has made mistakes in systematic reviews, as everyone has, though none that changed the conclusions, and so he spends much more time checking than generating. Screening, extraction and the other steps are completed quickly with the model, and the checking then takes far longer, so that he can be certain of the quality. For the published review he and two junior analysts each checked everything independently, three checks that took about two to two and a half weeks against about a week to write the report, and he puts the checking at about twice, perhaps three times, as long as the writing. He expects the checking to shrink as trust in the model grows and the share checked comes down.