Judging a search tool or a clinical resource before you rely on it
How far a search service, a scholarly index or a secondary clinical resource can be relied on depends on whether its stated scope holds against what it returns, whether its ranking and term mapping are known, whether its statements can be traced to something citable, and whether what it holds and how it can be reached fit the work.
Medical metasearch engines
OmniMedicalSearch billed itself in 2007 as a metasearch engine for the medical search engines: it sent each query at once to 32 databases, among them PubMed, MEDLINE and the Centers for Disease Control and Prevention, instead of searching the whole web as Google or Yahoo do, and by its own description it was intended for consumers. A search for hypertension from its main box returned 29 results, the first ten from a mix of fairly reputable sources, including healthline.com, the National Institutes of Health, FamilyDoctor.com and the Mayo Clinic. The same search under its news tab, across all news sources rather than one chosen source such as AMA News, BBC Health or Physician's Weekly, returned 173 results, of which at least the first ten, the only ones checked, appeared mostly relevant and current. A search restricted to noncommercial sites in the .org, .gov, .ac.uk and .edu domains returned 74 results, the second of them a "page not found" link to a rather dubious British site offering cottages for rent.
Every result gave a link, a one- or two-sentence excerpt and its source, and the results were surrounded by pitches to download the site's search bar or visit its shopping subsidiary, and by an animated advertisement for a consumer web magazine. There was no way to tell or set how results were sorted, by date, relevance or source, nor how many appeared on a page, and clicking a result opened a new window. Choosing the news tab cleared the search term, and in the 2007 reviewer's judgement the image search was not nearly as useful as Google's. The reviewer found that the service offered nothing unique and presented what it offered poorly, and would not use it again, mainly for its navigation but also for want of confidence that its results were comprehensive and trustworthy.
PubMed and Embase
PubMed, made available by the National Center for Biotechnology Information within the US National Library of Medicine at the National Institutes of Health, is often the first database a medical writer goes to on a new biomedical topic, and it is free. With more than 23 million records in 2016, it can be a comprehensive source for writers without subscription access, such as freelances and those at smaller agencies. Embase, for those with access, then held more than 30 million records and, unlike PubMed, about 2 million conference abstracts as well as journal articles. Individual subscriptions to Embase were advertised at $1,500 a month, and many university libraries offered public access.
According to a speaker from Embase's publisher at a 2018 conference, Embase is a commercially available tool for searching the peer-reviewed literature that the European Medicines Agency, the Cochrane Collaboration, the World Health Organization and other institutions recommend, and the European Commission recommends it for the literature searches behind a clinical evaluation report. In the same account, MEDLINE is a good starting point but insufficient for systematic searching, because its coverage of European journals is incomplete, so other sources, among them Embase and the Cochrane Central Database, are recommended for a systematic review. Embase includes MEDLINE's content and about 3,000 further non-US journals, mainly European but also from Asia and the rest of the world, and draws on conference abstracts as well as peer-reviewed journals.
PubMed indexes with Medical Subject Headings (MeSH), and Embase with Emtree, which contains every MeSH term and terms of its own. Embase records therefore map to broader and narrower search terms, and the more extensive indexing can give more, and perhaps more relevant, results. Natural-language recognition embedded in Emtree also makes it easier than in PubMed to use plain-language search terms and find relevant information. Embase is particularly suited to researching drugs: subject headings such as "drug comparison", "route of administration" and "drug development" build focused searches. Most Emtree and MeSH terms are continuously updated, but the terms for individual drugs, such as laboratory codes, chemical names and trade and generic names, are updated more often in Embase than in PubMed, three times a year against once, and older records are back-posted. PubMed, for its part, links a growing number of records to free full text in PubMed Central.
Emtree centres on drugs, diseases and devices, and Embase offers two search forms for devices: one that limits a search by manufacturer or trade name, among other fields, and one built on PICO. PICO splits the clinical question into patient, intervention, control and outcome, and the Commission's guidance says a literature review should use it. For each PICO component the most specific Emtree term should be used first and then supplemented with as many synonyms as possible. Embase proposes the taxonomy's synonyms, trade names among them, and records the term and every synonym, so that the full search strategy can be documented. Results export to a reference manager, and email alerts set on any relevant query deliver regular updates.
No single database, subscription or free, offers everything. Each has its own strengths and weaknesses of indexing, structure and coverage. PubMed and Embase used together can give complementary results, but research had shown by 2016 that neither alone is sufficient for the searches behind a systematic review. The most thorough search may therefore require several databases. Because each database is created differently, a searcher needs to know how those differences affect searching and adjust the strategy accordingly.
Google Scholar
What it retrieves
Medical writers search Google Scholar, though some may hesitate to admit it, and searched effectively it can supplement the other academic databases, which depends on knowing what it can and cannot do. A 2010 survey found PubMed, Web of Science and Google Scholar to be the three most popular databases among science researchers. Scholar may lack the scholarly reputation of the other two, but it requires no subscription, is easy to use and can yield results not available in PubMed. It retrieves published and unpublished literature, conference papers, technical reports, dissertations, pre-prints, drafts, patents, abstracts and many other document types from publisher sites, online repositories, professional societies and universities, and it tracks article citations and links to the citing articles. A searcher with college or university library privileges can add the institution under the library links setting, after which articles that would otherwise need a fee are reachable through the library's subscriptions. Scholar does not search the deep web: sites that require logins, sit behind pay walls or are buried deep in university repositories or websites.
Its features against PubMed and Web of Science
As compared in 2014, Google Scholar has no controlled vocabulary, searches full text, supports Boolean searching, provides citation counts and alerts, exports one record at a time to reference software, recognises no truncation symbols and offers only limited search limits, such as date and journal name. Google says Scholar updates several times a week, but updates to some existing records take more than six months. PubMed indexes with the MeSH controlled vocabulary, covers about 5,600 journals from 1946 with some older material, updates daily and offers extensive limits, by age group, humans or animals and type of article such as guidelines, randomised controlled trials and meta-analyses. Web of Science covers 12,000 journals and 150,000 conference proceedings from 1900, updates weekly and carries citation counts.
Ranking and term mapping
Scholar lists results by date or by relevance, which its own algorithm determines; Google does not disclose the algorithm's exact nature, but it draws on the full text of the article, author and journal information and the number of times the article has been cited. Databases such as PubMed instead build their relevance lists from indexing systems that use controlled vocabulary terms, together with author and journal names, article titles and other article information. The lack of information about Scholar's algorithm is one of its downsides, because an effective search is easier to create when the searcher knows how a database retrieves articles.
Scholar does not index items with a controlled vocabulary: it finds an item only if the specified terms appear in its full text or abstract, and it adds no subject terms to increase retrieval. Searchers and authors tend to use the terms of their professions or of everyday language rather than indexing terms, so searching in familiar language is easy and can yield good results, but it limits the ability to find articles that use alternative spellings or synonyms. PubMed, by contrast, maps popular keywords to MeSH index terms: it translates a search for breast cancer into the subject heading breast neoplasms together with each word and phrase searched in all fields, and so retrieves articles indexed under breast neoplasms. To retrieve all the relevant articles on breast cancer, a Scholar searcher may also need to add tumor and neoplasm. The difference can matter to a medical communicator in many situations, such as preparing systematic reviews or meta-analyses or identifying gaps in the research literature.
Search syntax
Scholar recognises no truncation symbols and instead uses automatic stemming, adding letters to the end of a word, so that jump also retrieves jumps, jumped and jumping. Stemming does not get every variant, since surge will not retrieve surgery, surgeries or surgeon, so a searcher must include possible variants and synonyms in the search strategy. Scholar adds AND automatically between search terms or sets of terms in parentheses, but OR has to be typed deliberately and in capital letters or it is ignored, so a cancer search reads cancer OR neoplasm OR tumor. Parentheses force terms to combine in a specified way, as in colon AND (neoplasm OR cancer OR tumor); without parentheses and capitalised operators, a search for colon cancer can also return articles about the colon and articles on other types of cancer. A minus sign excludes a term, as in colon (neoplasm OR cancer OR tumor) -rectal, at the risk of losing an article that mentions the excluded word briefly and otherwise meets the searcher's needs. The advanced search finds specific authors, publications, titles, date ranges and subject areas, and any search result can be turned into an automated email alert for new papers.
Errors in its records and citation counts
Scholar holds no full text or abstracts of its own: it data-mines websites and displays results through its algorithms, so incorrect information that it finds gives incorrect search results. Its crawling has misidentified many authors of articles as "subscribe" and, for one Lancet article, fabricated the first initials and surname of the first author from the subtitle of another paper and the second author's name from the title of a third. Some of these indexing problems appear to have been addressed, but in February 2013 a blogger reported Scholar listing an author as "MA Lib" instead of the correct author, K. Fraser. Because the number of times a paper has been cited feeds the relevance ranking, such ghost authors may affect search results.
Citation searching through the "cited by" link supplements a subject search but can be manipulated artificially: researchers raised articles' display rank by uploading fake papers for Google's crawlers to discover, an experiment that added 774 citations to 129 papers. No case had been documented of anyone manipulating citations to raise their own article's ranking, but PubMed, Web of Science and Scopus are less vulnerable to that kind of manipulation.
How much relevant literature it retrieves
A Canadian study by Shariff and colleagues, reported as recent in 2013, compared Scholar with PubMed in nephrology. From 100 systematic reviews the researchers derived 100 research questions, practising nephrologists gave the search terms they would use to answer each, and those terms were run in both tools, with relevant articles defined as the 1,574 articles cited in total across the 100 reviews.
Across whole result sets the two returned similar numbers of relevant articles, but Scholar returned more articles overall, so its proportion of relevant articles was lower; it also returned more articles whose full text was free. Within the first 40 results, the most that 80% of nephrologists had reported scanning for relevance in an earlier study, Scholar retrieved twice as many relevant articles as PubMed and three times as many relevant articles with free full text. The two covered the relevant articles similarly, PubMed holding 80% of them and Scholar 83%. In one commentator's view, which tool is better depends on factors such as how many citations a searcher is willing to trawl through and how much obtaining free content matters; earlier comparisons in other fields had reached no consensus.
Scholar can be quite effective for quick clinical research but may not be as effective for other search queries, and an article in Systematic Reviews found its article retrieval too low for it to serve as the single database for a systematic review.
Science lecture collections
Scitalks, launched in June 2007, collected science lectures in video or audio format across more than 30 specialty areas, from updates on HIV/AIDS treatment and in genetics to interviews with scientists such as Richard Feynman and Max Perutz. It worked as a kind of targeted search engine that linked the user to existing online sources, so that universities and other copyright holders controlled the content; scientists and lecturers could upload their own videos, and linked sites covered lectures in the humanities, government and business. One of its few disadvantages, in a 2008 review, was that finding a lecture meant scrolling through each of the 33 listed specialties, although a search box led to specific institutions or speakers. Each video's length and a rating were given, and video quality varied widely, from classrooms less than perfectly lit to full professional broadcasting lighting.
Clinical reference services
eMedicine
Most medical writers feel both enjoyment and apprehension when writing about an unfamiliar medical condition, in the view of a 2008 website review. eMedicine, then a WebMD-hosted site carrying continuing education material on more than 7,000 diseases and disorders, gave such a writer clinical background and presentation, differential diagnoses, work-up and evaluation, current therapies such as medications and surgical procedures, potential complications, inpatient and outpatient care, and ethical, legal and practice implications. Its content was written by academic physicians, peer reviewed by other physicians and a pharmacist and described as evidence-based, with the time of last update stated for each article. The site also carried news headlines, recalls, alerts, recent journal updates, patient education, practice guidelines, multimedia images and industry promotional articles.
Continuing education for health care professionals was the site's forte. Viewers reached it and the promotional material through different tabs, and each article was labelled as continuing education or promotional in small text at its top. The courses cost $7.50 per 1.5 hours, could be completed entirely online and were accredited by the University of Nebraska Medical Center's Center for Continuing Education, the University of Nebraska College of Nursing and the University of Missouri St. Louis College of Optometry.
The review found several shortcomings. Statements were not individually referenced to original articles, so verifying or citing one meant working out its source among all the items in the article's bibliography. The MEDLINE link in a reference list led to the Medscape summary of the article rather than to the article. And the site excluded original scientific research publications and material by non-physician experts, such as biostatisticians, dentists, epidemiologists, nutritionists, nurses, optometrists, pharmacists, physical therapists, scientists and speech pathologists. Overall the review judged the site useful, with limitations.
MD Consult
In 2008 AMWA licensed MD Consult, an online subscription-based integrated medical information service, for its members in a one-year pilot, after a survey of members showed a clear preference for it over Ovid. It carried the complete text of more than 50 leading medical reference books, of over 80 journals and of the Clinics of North America; a comprehensive and regularly updated drug database of FDA prescribing information for thousands of drugs; summaries of articles from the top general-interest journals, updated as each new issue was released, and recent top medical stories from the major journals, government agencies and medical conferences, sorted by interest such as cardiology or dermatology and summarised in an easy-to-scan format; extensive patient education material in basic, easy-to-understand language on medical conditions, their treatments and drugs, including generic and brand names, indications, dosing instructions and side effects; and a search of the full text of the online journal collection and millions of MEDLINE abstracts at once.
Because of cost, the association's licence allowed only two members on the service at a time, apart from an initial period of unlimited use when the benefit was introduced, and a customisable time-out logged users out after a period of inactivity. Because access was limited, members were asked to have at hand the facts they needed, and to develop their search strategy, before logging in.
Several members wrote in support of the pilot. One called it probably the second most useful resource for medical writing, after the association's freelance discussion list, and another a truly valuable, educational incentive. A third explained why it helped: one of the most difficult sticking points in medical and scientific research for articles is not having access to the journals.
ClinicalKey
ClinicalKey, Elsevier's clinical search engine for physicians, nurses and other health professionals, offered in 2015 over 1,000 textbooks, 500 journals, thousands of videos and millions of images, with the First Consult point-of-care monographs merged into it. Full-text access to Elsevier content depended on the subscription package, and MEDLINE results from other publishers were not full text unless the subscriber already had access. Its content was Elsevier's medical and surgical books; full text from about 600 Elsevier journals, with MEDLINE articles in full text only where an institutional link resolver reached them; Clinical Pharmacology drug monographs from Elsevier's drug database; about 4,000 practice guidelines from the National Guideline Clearinghouse, journals and professional and government organisations; about 15,000 patient education handouts from MD Consult; images from Elsevier's textbooks and journals; and procedure videos in five specialties, including 312 procedures required by the Accreditation Council for Graduate Medical Education.
A search returned results from many source types, among them full-text Elsevier articles, MEDLINE abstracts, trials from ClinicalTrials.gov, books, drug monographs, practice guidelines, patient education materials, images, videos and First Consult monographs. A search matching one of its 1,400 topic pages displayed an overview from Ferri's Clinical Advisor, Conn's Current Therapy, Goldman's Cecil Medicine or First Consult, with links to recent Elsevier articles, drug information, patient education, practice guidelines and specialty views; a medical librarian's 2015 review read this as an attempt at something of a point-of-care tool. Any other search listed results from all its sources, sorted by relevance or, if the user chose, by date, and they could be narrowed by source type, study type, specialty and date. Books and journals could also be browsed in alphabetical lists, filterable by specialty.
The review had several reservations. After a considerable redesign it was not clear what algorithms worked alongside relevance, and there was no way to jump within the alphabetical lists of books and journals, which had to be scrolled through or searched by title. A book chapter came as HTML under an institutional subscription and as PDF only with a personal login, which an institutional user could create free. It was unclear whether ahead-of-print articles from Elsevier journals were included, and videos mentioned in Elsevier's ExpertConsult book titles could not be retrieved.
Drug information portals
In October 2007 PubMed began an experiment in which randomly selected sessions offered, from the AbstractPlus display, links to patient drug information from MedMaster, a publication of the American Society of Health-System Pharmacists. More than 2 million citations linked to it, and the feature was to be extended to all sessions if it proved popular.
The US National Library of Medicine's drug information portal, reported as new in 2008, takes a drug name and returns a list of consumer and professional sites, with more than 12,000 records available for searching.
Guideline repositories
In 2010 the US National Guideline Clearinghouse, an initiative of the Agency for Healthcare Research and Quality, held more than 2,400 evidence-based clinical practice guideline summaries submitted by organisations and individuals worldwide, with the full-text guidelines available in print or electronic format on request. Each listing was a structured, standardised summary following its published template of guideline attributes. Guidelines could be reached by disease or condition and by chemicals and drugs, both listed by MeSH terms, or alphabetically by developing organisation. Separate lists covered guidelines in progress that had met its inclusion criteria and, in an archive, summaries of guidelines withdrawn from the site. An annotated bibliography of more than 5,750 citations covered guideline development methodology, evaluation, implementation and structure, and expert commentaries, with their own archive, gave editorial views on trends in guideline development and critiques of individual guidelines.
Two of its features compared guidelines. A comparison function set any two or more guidelines side by side against the attributes of a published comparison template, and it held the guidelines selected until the browser was closed, so that guidelines from several searches could be compared together. The guideline syntheses compared selected guidelines on similar topics across areas of agreement and difference, major recommendations with their strength-of-evidence and recommendation rating schemes, methodology, source of funding, and benefits and harms.
The clearinghouse closed on 16 July 2018 because, according to the Agency for Healthcare Research and Quality, which operated it, funds to maintain it were no longer available; its budget for fiscal year 2017 had been $1.2 million, down $0.9 million on the year before. It had been established in 1997 to deliver an accessible mechanism for obtaining objective, detailed information on clinical practice guidelines and to further their dissemination, implementation and use, and had been a free public website since 1999. It was a central repository of carefully vetted guidelines meeting stringent inclusion criteria, with structured summaries of many of them, expert commentaries and a wide variety of advanced searching, features that in one 2018 account made it an invaluable resource for up-to-date, trustworthy and easily digestible evidence-based information. The closure drew criticism and appeals for reconsideration. New guidelines had stopped being accepted on 5 March 2018 so that summaries and the clearinghouse's assessments of adherence to trustworthy standards could be posted by July, and it closed before an alternative could be created.
As of July 2018 the ECRI Institute, the nonprofit that had held the federal contract to help develop and maintain the clearinghouse for 20 years, appeared to be planning a similar resource, and one of its directors said it was doing so without the agency's support or input. According to the same director, several professional organisations had expressed interest in sharing their guidelines, which would be summarised and evaluated against the National Academy of Medicine's standards for trustworthy clinical practice guidelines. The new site would not replicate the clearinghouse but would carry much similar information, an interim version was expected that autumn, and a subscription would be required of anyone not taking part in guideline development, at a cost the institute planned to keep reasonable. The agency had no comment on those plans, and its own future was unclear: the president's budget proposal for fiscal year 2019 called for it to be consolidated into the National Institutes of Health as a new National Institute for Research on Safety and Quality.