AI-assisted literature review
A language model can take part in most steps of a systematic review: building the search string, screening titles and abstracts, extracting data, assessing risk of bias and drafting the study summaries, with the synthesis left to the reviewer. Each step has its own prompt and its own check. The commercial review tools of 2019 to 2024 automated some steps far further than others, and few reviewers used them.
The steps AI can support
In 2019 specialists in systematic reviews were beginning to adopt AI tools that aim to optimise searches and streamline the review process, and machine learning and natural language processing were held to promise a way to generate, update and maintain an up-to-date synthesis of clinical data. An AI-assisted review has the same steps as a conventional one, namely protocol, search, screening, extraction, appraisal and analysis, and differs in how far each step can be semi-automated: machine learning increases the speed and efficiency of a task rather than executing it autonomously, in a human-in-the-loop workflow in which AI supports the people involved rather than replacing them. Most of the labour-intensive tasks, namely screening, data extraction and to some extent critical appraisal, could be supported by AI-based tools, and most of the example tools of the time supported only one discrete task.
In May 2024 the state of the vendors' tools could be described in four stages: query generation and retrieval, screening, appraisal and extraction, and analysis and report generation.
Why reviews are being automated
Individual trials often give limited insight, which is why meta-analyses are done, and in an outbreak such as COVID-19 systematic reviews are crucial because people's lives depend on them. The unprecedented rate of evidence generation makes such reviews demanding in resources and time.
The adoption of the European device and in vitro diagnostics regulations greatly increased the demand for systematic literature reviews just as the volume of literature kept growing, and adopting software that makes retrieving, identifying, analysing and synthesising the literature more efficient is an important strategy for dealing with the workload.
One firm that builds AI tools for clients counts generative AI-based literature screening as another big area of efficiency to be unlocked for aggregate report writers, who all go through the same exercise of reading the literature and summarising the results.
Prompting a general model through a review
The prompting method below is one experienced systematic reviewer's, demonstrated in March 2024 on a review he had published after executing it with ChatGPT. Speed is the obvious gain, since a systematic review is a very time-consuming process. In the reviewer's view AI will in time also help improve reproducibility, which is a problem as the process stands, and quality. The gains come with potential traps and limitations that need to be taken into consideration.
The worked question was picked to be fairly straightforward: whether proton pump inhibitors are associated with the risk of major adverse cardiovascular events. It admits little ambiguity in the inclusion and exclusion criteria and has few variables. Only meta-analyses were wanted, making the review an umbrella review, to keep the numbers down and the experiment manageable.
The published version of that review was produced with ChatGPT 3.5, before version 4 was out, and two steps had to be overridden by hand: the risk-of-bias assessment, which the model could not be got to perform, and some of the search-string keywords, which were not up to par. The risk-of-bias step failed because the model was given more than it could handle, and its assessments were unreliable when compared with the human ones; with a lot of patience and trial and error the step can be made to work. Every step can be performed with ChatGPT 3.5, though the prompts may need adjusting quite a bit to give it the appropriate context and background information.
Search strategy
The search strategy is assembled in three prompts inside one chat: ask for an exhaustive list of the intervention's generic names, ask the model to define the outcome, then give it the review question as context and ask for a PubMed search string. The word exhaustive is the one the reviewer uses to get a complete list, though ChatGPT is still more than likely to give only the top 20 when the list runs longer, so he asks again whether the previous response was exhaustive. The first prompt ends with the restriction that no other output is required, so that only the list of generic names comes back.
The definition supplies the keywords for the outcome. Asked to define major adverse cardiovascular events, the model returned the three components of the most widely accepted definition, myocardial infarction, stroke and cardiovascular death, and further elements that the definition sometimes includes, such as hospitalisation for unstable angina and coronary revascularisation.
The syntax can be specified rather than accepted. Controlled-vocabulary terms, title-and-abstract terms, phrase index terms, free terms, wildcards, publication types or date limits can be asked for, and the model constructs the string accordingly. Changing the database named in the prompt from PubMed to Embase still produces a string. Where the database returns an error message, pasting it back with a request for a fix is quite reliable and ends in a string the reviewer is happy with.
The request for a broader or narrower string is the instruction the reviewer likes most. When a string returns nothing, or returns a hundred results none of which are relevant, requesting a broader string works, in his words, 99 times out of 100. It is handy for any PubMed search, not only a systematic review, and he finds it particularly useful when the information sought is known beforehand to be unlikely to exist.
In the reviewer's view a generated search strategy always needs human oversight. In the live rebuild, PubMed did not recognise one phrase-index term in the model's string. More seriously, the string did not retrieve one of the seven studies the published review had included. The screening itself was effective; the issue lay in the search string. For the published review the term mortality had been added by hand to expand the search; it is not specific to the cardiovascular system, but it was found potentially to include more relevant studies. So a generated string should not be dropped into PubMed and taken as done: someone who knows what they are doing and has done it before should evaluate it, because limitations of this kind will be present and need to be identified and addressed.
The reviewer does not trust PubMed's publication-type filter, finding its indexing terrible as far as he can tell, and used it in the demonstration only for the demonstration's purposes. Limiting to meta-analyses, whether by the sidebar filter or within the string, means that studies may be missed, so he would screen the 2,300 or so results of the unfiltered string. He always goes broad with screening, even though it takes more time, because it is the only way he can be certain nothing is missed.
Many vendors' tools reach the databases through an application programming interface, which leaves them heavily dependent on the source database's indexing. Indexing is done by people and is not real time, so there is a lag. The alternative is to store one's own database with one's own retrieval indexing, which should be validated on precision, recall and the other performance metrics. Query translation was improving in May 2024, with Embase releasing a better version of its translator, which uses AI to translate PubMed's MeSH terms into its own. What was wanted and still some way off is a platform where a person writes down what they are looking for and the terms are translated into the correct ontology for each database searched. Elsevier requires a specific licence for running an AI on its database, and was having that discussion with every client using its system.
The volume of literature keeps growing: on recent bibliometric figures, the number of published scientific papers has grown by eight to nine per cent a year over the past several decades, and in biomedicine alone more than a million papers a year enter PubMed, about two a minute. This overload, called info-obesity, comes together with the clients' fear of missing something, so there is a greater need for a tool that extracts information deeper in scientific papers, and news and market research need more granular sifting too.
At one large pharmaceutical company, a researcher who asks the library for a literature search on a topic, run through PubMed and Embase, gets back between 40 and 60 articles depending on the subject. The library's goal is no more than ten, using good AI tools, and in May 2024 it was nowhere near that. It had licensed a literature review accelerator to tighten its searching so that eight or ten articles come back rather than 40. It was also collaborating with data colleagues to build an automated data extraction tool for systematic reviews that anyone in the company will be able to use; anyone who has written a systematic review knows, in the librarian's words, how horrible extracting the data is.
Screening
Screening is done by pasting formatted citations into the model in fixed batches, and the batch size is a matter of tokens. On the subscription tier used, which gave 8,000 tokens per interaction and 32,000 for the whole chat, the reviewer screened ten titles and abstracts at a time. On the regular version 4.0 he recommends three or perhaps four and no more, because beyond that the mistakes start. In a later answer he put the batch for version 4 and the free version 3.5 at two or three, because their context windows are much smaller, and said the prompts would not be as effective on them. In his view the free version and the $20-a-month version can be about equally effective, though he would recommend version 4.0. When the final batch is short, the number stated in the prompt is changed to match it. Eighty-four records at ten a time means running the same prompt nine or ten times. Running out of tokens happens sometimes and stops the output.
The screening prompt sets the model up as an expert systematic reviewer, which establishes context in as few words as possible. It states the task and the eligibility criteria as a PICOS. It describes the structure of the text being pasted in, so the model is not left to work it out on its own: for EndNote's copied citations, that they begin with the author list, then the year, the title, the citation information and the abstract, and that some citations may be missing an abstract. And it fixes the output as each study title, the relevance determination and a brief justification for the decision.
The exclusions are stated with their exception: studies of the intervention co-administered with antiplatelets must be excluded, but studies with mixed populations must be included where they report the outcome for patients on the intervention as monotherapy. The outcomes are worded as myocardial infarction and stroke rather than non-fatal myocardial infarction and non-fatal stroke, because non-fatal myocardial infarction will not be found reported in any study, and working out that a non-fatal myocardial infarction is a myocardial infarction would take up processing capacity the reviewer wants kept for the screening itself.
The reference manager is not a constraint. Where a copy of formatted references is unavailable, the text file of results saved from PubMed works just as well, pasted or uploaded; only the description of the text structure in the prompt changes, naming the number, citation information, title, authors, affiliations and abstract in the order that file gives them.
Each decision comes back with a reason attached, such as that a record compares the intervention with H2 receptor antagonists in patients with coronary artery disease on dual antiplatelet therapy and so is not of interest. In the reviewer's view the log those reasons make is much better than the typical PRISMA flow chart: much more detailed, and much easier to come back to years later to see why a record was included or excluded.
In the demonstration, 84 abstracts gave six includes, and every justification still needs to be reviewed by someone. The reviewer has checked the justifications extensively and found them reliable, and where one is not reliable, that is quite evident in the justification itself.
In a test that put all six included studies in one batch with four irrelevant ones, the included studies came back as relevant, two of them as relevant with caution, one of these the study whose population took the intervention alone or with antiplatelets. The records do not compete with each other: a batch can hold ten relevant or ten irrelevant records without it mattering, and each response is independent of the previous one.
In 2019 screening was one of the most time-consuming steps and an area where AI research had concentrated, with some tools mature enough to use. After deduplication a screener may face several hundred abstracts. The tools learnt from each include-or-exclude decision and continuously re-prioritised the remaining records by their likelihood of inclusion, which can focus the screener on the likeliest ones, and some also grouped related abstracts automatically by topic modelling. The potential time savings had been shown in a user study of one such tool, RobotAnalyst. An AI screening tool could also serve as the second screener. DistillerSR, a web-based screening, extraction and reporting package familiar to some device writers, had used machine learning and natural language processing to support screening since 2018, and Rayyan and SWIFT-Review were further examples.
In May 2024 screening was held to be the most resolved stage of the vendors' tools. Ranking and re-ranking bring the likely-relevant records to the top and push the rest down. Some tools offer proactive termination, telling the reviewer when to stop on a stated confidence that some proportion of the relevant articles, 95% in the example, is already in, and leaving it to the reviewer whether to finish the rest anyway. And some classify outright and leave a human in the loop to accept or reject, readjusting from that feedback as an active learning process.
Data extraction
Data extraction runs as two prompts rather than one. The first asks for a bulleted extraction form built from the PICOS, and the second turns the completed form into a table with named column headings. The same bulleted form is then reused as the input to the study summaries.
Two instructions guard against the models' tendency to deduce or extrapolate information that is not there. Fields that may genuinely be absent are marked "if available", since a systematic review will typically report the number of included studies but not necessarily the total number of patients across them. The prompt's restrictions tell the model to omit any requested information the text does not contain and not to extrapolate or deduce it, which is the reviewer's way of ensuring there are no hallucinations. In his view writers should all keep in mind which use cases are prone to hallucinations and address them.
The prompt also asks for all relevant numerical values, because a model summarising text usually reports no numbers. Without that instruction a hazard ratio with its confidence interval and p value comes back as "significantly increased risk".
Given a PDF, the model stops looking as soon as it finds relevant information, though the same or additional information may be elsewhere in the document, which is a trap for missing vital information. So pretty much every prompt of the reviewer's that involves a PDF ends with the instruction to scan the entire document, including all tables and figures, before generating a response, and with it the model should be fine in the vast majority of cases. Tables are read as reliably as text. Some images the model has trouble reading, though it reads them reliably perhaps nine times out of ten, so the images in a PDF are worth going back to during checking. The reviewer had never tried a PDF whose text is held as images; there he would supervise and double-check the output, since the model is prone to some kind of mistake, and advises being very cautious.
The extracted numbers are then checked against the source. The patient total is among the first things to verify in the PDF, and the assembled table is a very good starting point that still needs human oversight and some editorial work.
In 2019 the extraction tools ranged from highlighting the sentences most likely to hold relevant information to extracting a specific data element. RobotReviewer, which then had a free online demonstration, highlighted the text describing the population, intervention, control and outcomes of a randomised trial report and generated a summary of the study's characteristics and main findings. It captured intervention and control together under a single intervention label, so a human reviewer was still needed to tell the treatment under study from its comparator. ExaCT was a second prototype, and both presented extracted elements such as PICO information and the sample size for the reviewer to verify or change. A systematic review of extraction methods had found many of them aimed at relatively straightforward elements, the most studied being participant characteristics, interventions and outcomes, with others such as the duration of follow-up or the incidence of adverse events in each group studied less or not at all. Most tools were limited to randomised trials, which was a barrier for device writers, since a substantial amount of the data in a clinical evaluation comes from observational studies; some research groups were working to close that gap.
In May 2024 most vendors' appraisal and extraction focused on the abstract, because a full text is a great many tokens and computationally demanding, even though an abstract does not always capture everything a reviewer wants; splitting documents into chunks is one technique being worked on. The entities extracted are mainly the PICO elements, whereas a full text yields a great deal more.
Risk-of-bias appraisal
A report can be uploaded with a request for the risk-of-bias assessment tool that suits it, and the answer can be quite reliable. Every such tool comes with a guidance document, except the Cochrane tools, which come as books, and that document is what the prompt is built from: upload it and ask the model for a prompt that evaluates reports against one domain only, then repeat for each remaining domain. For an umbrella review the applicable tool is AMSTAR 2, which has 16 domains, so the process yields 16 prompts. All 16 can be attempted in one prompt, but in the reviewer's view one domain at a time is the most appropriate way.
Run against a real report, the domain prompt gives its justification before its decision. For the first domain, whether the review's PICO elements were clearly described, the reviewer knew the answer to be a full yes rather than a partial yes or a no, and the model's decision was yes.
For the reviewer, the case for automating this step rests on reproducibility and quality rather than on time. Risk-of-bias assessments are not the most reproducible part of a review, because several elements are subjective and domains are often left to the reviewer's discretion or experience: reviewers vary, and the same reviewer may not be consistent from one day to the next. A model has biases of its own, but there are a handful of language models and billions of people, so in a hypothetical where one model did all risk-of-bias assessments its biases would be known and could be taken into account when evaluating its assessments, unlike those of many reviewers each with their own.
In 2019 RobotReviewer attempted to detect risk of bias in randomised trials: it analysed the full text for randomisation, allocation concealment and blinding of participants and outcome assessment, produced a bias assessment in the terms of the Cochrane tool with the relevant text highlighted, and left the human reviewer to confirm its initial assessment.
Study summaries and synthesis
The last step of the process converts each completed extraction form into a narrative study summary. The prompt casts the model as an expert medical writer and states the conversion task in two lines, and the rest of it is rules and restrictions, because ChatGPT tends not to sound fully scientific without them: no metaphorical language, storytelling or subjective description, a focus on the study's objectives, design, methods and results, and the formal tone of scientific papers, with an emphasis on clarity, conciseness and objectivity. The two rules from the extraction prompt reappear: include all relevant numerical values, and state that information is not provided rather than deduce it.
Fed one extraction form, the model usually gives two paragraphs where one was wanted, and fed two forms it would give one paragraph, so getting the shape wanted is a matter of balancing the input against the output.
Summarisation by natural language generation models comes in two kinds. Extractive summarisation selects important sentences from the original text whole and assembles them, which suits an author who means to rewrite the material in their own words afterwards. Abstractive summarisation interprets the text and reproduces its important content in a new way, simulating how humans review literature, which suits an author who wants an abstract of the selected recommendations; GPT-3, T5, BERT and BART are examples of abstractive models. Given three sentences on psoriasis vulgaris, the extractive version keeps two of them verbatim, and the abstractive version compresses them into one new sentence describing a chronic inflammatory skin disease with erythematous scaly plaques and an overlying silvery scale. Summarisation models were first trained on news articles, which supply human-written texts and summaries, were then being trained on the published medical literature, and still needed refining for specific diseases and drugs.
What the prompting method produces is a starting point for evidence synthesis, not a synthesis. Search strategy, screened results with justifications, extracted data, risk-of-bias assessments and study summaries arrive within a very short time; reading that output and writing the introduction, methods, results and discussion is where human intelligence is essential. In March 2024, combining the studies needed a human brain.
In May 2024 the vendors were working on the analysis and reporting stage: dynamic updating of the PRISMA diagram and of the analysis, and augmented authoring, in which the accumulated screening and extraction feed a content-authoring tool that notifies the reviewer when a new article appears, screens it, extracts the data wanted, and updates the quantitative, qualitative and narrative parts of the publication for a human to review. A good deal of this work is going on at PubMed and Cochrane, where dedicated teams are bringing out AI tools, the majority of them free to access.
Generating hypotheses from the literature
AI can also be used to generate research hypotheses from the literature. It can mine the research already published in a domain to find an area that has not really been developed, find connections in a dataset the researcher holds and the gaps in it, and suggest what further research would test them. The result is down to the prompts to a large degree, and a little creativity in how they are used helps generate new potential hypotheses.
Time saved and the checking it costs
The time saved is hard to quantify, since no two systematic reviews are alike. The industry standard for manual title-and-abstract screening is, on the reviewer's estimate, about 500 to 600 abstracts in an eight-hour shift, and he screens 1,200 to 1,500 with the model in the same shift. The method involves a lot of copying and pasting: 5,000 abstracts, a normal volume, means repeating the batch 500 times and reading 5,000 justifications, and he does this in about half the time.
An automation layer removes the copying, and in March 2024 such tools were expected to appear within about a year. They use the same prompts, but process records as fast as calls can be made to the model, around 10,000 abstracts an hour against 60 to 70 at best for a person, depending on the topic.
The time saved goes into checking. The last AI-generated review the reviewer delivered took longer than quoted, and he asked for no more hours, taking the extra time to make sure everything was right; he was then experiencing little time saving, and what the method gave him was quality. The reviewer knows of no article missed through the model that would have changed his conclusions, in a review published for about a year or in any client report, and puts his certainty down to the endless hours he spends checking whenever he uses AI.
Review platforms
Tools for systematic literature review and for the technical documentation of the device life cycle had been released since 2008. In one device writer's account their surge came with the European device regulation's implementation in May 2021, driven primarily by the burden of managing and updating clinical evidence and of the stringent conformity assessment, which left manufacturers facing unprecedented challenges in costs and time with limited ways of coping. DistillerSR lets the user set how far the AI's inclusion and exclusion recommendations are automated and for which exclusion reasons, with duplication, language and missing abstracts common reasons for high automation. It re-ranks references continuously against the records already screened, double-checks inclusion and exclusion decisions, and deduplicates through a smart quarantine whose confidence threshold the user sets. It lets a project leader assign portions of the review to named reviewers and track their progress, traces every action and search to a user, a date and time and a query, and adds a module for curating and reusing evidence across teams. Fern.ai, formerly Giotto Compliance, starts each review from a PICOS research strategy that its AI uses as the basis for its recommendations, which it delivers during title and abstract screening and extraction from models trained by medical writers in a human-in-the-loop approach. It extracts data through natural language processing and customisable templates, each extracted item linked back to its place in the article for one-click review. Nested Knowledge splits the work into AutoLit and Synthesis. AutoLit covers search, screening and extraction, with AI assistance at each step but expert review required to confirm accuracy, and in screening the AI acts as a third reviewer that learns from the user's decisions. Synthesis provides AI-assisted living reviews, a network meta-analysis environment, forest and funnel plots, and heterogeneity and risk-of-bias displays, in interactive visualisations that change as the data are updated. MedBoard's systematic-review module adds automated updates, customisable appraisal criteria, automated PRISMA diagrams, identification of similar reviews and an AI reviewer assistant on top of its curated regulatory, market and clinical databases. avasis builds on the Polarion application lifecycle software to turn document content into reusable digital datasets, including a module that manages IMDRF adverse event codes centrally rather than in Excel files.
In the same writer's assessment, no one review platform suits every use, and users should invest time in finding the one that suits their use cases. The tools share a goal but prioritise different features, content authoring and analytical synthesis in some, reusable data, automated extraction and curated global intelligence in others. Not all of them incorporate AI, and those that do use it as an additional feature that keeps the user in the driver's seat. Platforms built for professional use differ notably from publicly available AI in how they are trained, tested and validated. A new tool is best understood through several methods and sources, and discussions with long-term users and adequate testing are essential.
Validating a tool
The first thing the reviewer would ask of an automation tool built on screening prompts is how it was validated: how many abstracts were screened, and what accuracy, sensitivity and specificity were achieved.
In May 2024 none of the vendors' tools was necessarily validated. The questions to put to a vendor before paying a large amount of money are what validation was done, and what recall, precision, F score, sensitivity and specificity it achieved. The choice turns on what the reviewer wants to achieve rather than on the model: handed a fork, a knife and a spoon, the right question is what to eat.
Uptake among reviewers
In 2019 AI tools for systematic review had not matured enough for widespread use by researchers or industry. Seventeen machine-learning systematic review tools in healthcare, in prototype or early development, could be found in the SR Toolbox, with clear limits on the data elements they could extract and little research on observational studies or designs other than randomised trials. The barriers named were scepticism about the reliability of AI-assisted reviews, a general distrust of handing an assessment to a computer, the logistics of fitting a new tool into an established process, and, especially for a freelance device writer, the possibly prohibitive cost of some commercial packages, though most of the example tools were free or had free versions. A quick informal survey of EMWA's medical device special interest group found few members with experience of such tools: DistillerSR and Rayyan were mentioned but not used regularly, and most relied on spreadsheets and their preferred reference manager. Early adopters could potentially help develop the tools by contributing data sets of screened and appraised literature to train them, though whether device writers would adopt the tools remained to be seen. It seemed clear that AI was not about to replace the human systematic reviewer, and a validated, reliable, easy-to-use tool supporting several steps for several study designs was hoped to be not too far away.
In 2023 an anonymous online survey of medical writers and others conducting systematic literature reviews for devices and in vitro diagnostics ran from 3 April to 9 May among EMWA members and on LinkedIn. Its 125 respondents came from 31 countries, Germany the largest group at 22 (17.6%), and Germany, France, Belgium, the United States, the UK and Canada together gave more than half the responses (55.2%). Of the respondents, 71.2% were women, and 62.4% were employees, 23.2% freelancers, 8.8% hybrid and 5.6% small business owners. 58.4% did reviews as medical writers, 22.4% as clinical or regulatory affairs managers, 10.4% as researchers and 8.8% from another role, and experience was spread quite evenly across the bands from under three to over ten years. Only 6% ran reviews through an automated process, 35% partially automated and 59% fully manually without dedicated software. Of the 40 (32%) who used specific tools for at least part of the process, 47.5% used commercial desktop or self-hosted software, 37.5% a software-as-a-service platform and 15.0% a custom or self-made tool. About a third worked in duplicate: 38.1% for screening, 37.1% for appraisal and 27.8% for data extraction. Among the 82 who answered the familiarity question, on average 4.6% currently used one of the five named platforms, 3.4% had used one in the past, 2.2% planned to, 12.7% knew it existed and 77.1% were unfamiliar with it. DistillerSR was the most widely used and known, current use running from 1.2% for Giotto Compliance and the Systematic Review Accelerator to 8.5% for DistillerSR and Rayyan, and unfamiliarity from 61.0% for DistillerSR to 84.2% for Rayyan and the Systematic Review Accelerator.
Among the 78 respondents who gave a valid figure, a new review averaged 84.8 hours (SD 71.2, IQR 37.5–105.0) and an update 45.9 hours (SD 70.4, IQR 13.75–50). Eight reported at least 200 hours for a new review, the maximum 500, and fourteen under eight hours, which the survey reads as either domains with no or very limited data or misreported units. The question gave no volume of retrieved literature, which 10 respondents (8%) pointed out, and 35 (28%) skipped it. Review time did not differ significantly by employment type, role or experience, and new-review time tended to be lower with automated processes without reaching significance (P=0.072). Respondents used a median of five other general-purpose packages: word processors (89.4%), spreadsheets (86.9%), PDF software (79.9%), reference managers (73.2%), graphical software (61%) and databases (18.3%). Microsoft Office led every category it appears in and Acrobat dominated PDF handling. Three respondents drew PRISMA charts in Word and three others had the function built into their review software. Respondents who did not work manually, or who used a software-as-a-service package, used significantly fewer tools in total (P=0.03), and those with custom or self-made applications used more database applications (P=0.009).
The challenges named were keeping up with large volumes of information, reducing errors, managing the process efficiently, defining the scope and searches, traceability and finishing on time, with obtaining full texts and meeting client or employer expectations while maintaining quality added under other. Data extraction was both the most time-consuming and the most error-prone step. Data extraction, screening and analysis were the three places where software could most reduce errors and improve traceability, and process improvement, better extraction and time saving the most valuable contributions.