What the impact factor and other citation measures can tell you
The journal impact factor averages the citations to a journal's recent articles, so it describes a journal and says little about any one article or researcher in it. Journals can inflate it, sometimes by asking authors to cite their own recent pages. Alternative measures count other things, from citation rates normalised for field to the attention an article receives online.
How the impact factor is calculated
The impact factor rates the influence of a journal on the scientific community and compares it numerically with a large number of other journals. It was first proposed by Eugene Garfield, and the Institute for Scientific Information (ISI) developed it in the 1960s. Another account has it first devised in 1955.
In 2013 Thomson Reuters issued impact factors each June. The numerator is the number of times the journal's articles from the two previous calendar years were cited in the current year, so a journal's 2012 impact factor rests on citations that appeared in 2012 to articles it published in 2010 and 2011. The denominator is the number of substantive articles the journal published in those two years, and the numerator divided by the denominator is the impact factor. Other accounts divide the citations by the total number of articles the journal published over the same two years or by the number of its citable papers. The impact factor is approximately the average number of times the primary research papers a journal published in two consecutive years are cited in the following year, and it reflects the average number of times the journal's articles have been cited. Citable articles can include review articles and editorials as well as reports of clinical trials, which can influence editorial decisions as editors try to achieve the highest impact factor possible. The underlying formula remains constant, but the company that issues the rankings can apply it inconsistently, even within a single journal from one year to the next.
Impact factors in use
In 2013 the impact factor was the most widely used method of assessing the quality of a journal, and it was still heavily used, especially in academic institutions. The higher the number, the greater the presumed impact of the journal. Some journals advertise their impact factors: The Lancet announced by email that its factor had risen to 30.76, and the Irish Journal of Medical Science that its factor had nearly doubled to 0.696.
In 2015 the ten medical journals with the highest impact factors were the New England Journal of Medicine (54.42), The Lancet (39.21), JAMA (30.39), the BMJ (16.38), Annals of Internal Medicine (16.10), PLoS Medicine (14.00), Archives of Internal Medicine (13.24), the Journal of Cachexia, Sarcopenia and Muscle (7.41), BMC Medicine (7.28) and the Cochrane Database of Systematic Reviews (5.94). Impact factors reported as current in 2018 for some leading medical journals were 59.558 for the New England Journal of Medicine, 37.684 for JAMA, 19.967 for the BMJ, 16.593 for Annals of Internal Medicine, 14.430 for Circulation and 2.191 for Archives of Surgery. As reported in 2018, two-thirds of medical journals had an impact factor of less than 1.
Some journals post their current impact factor on their websites and some do not, and a journal's figure can be found by searching the web for "impact factor" followed by the journal's name. The impact factor is seldom of great importance to a beginning medical writer, who will still meet the term and should know what it means.
What it says about a journal
Various indices compare journals by their relative importance or quality. One publication planner holds, while marking the point as questionable, that the impact factor reflects a journal's relative importance in its field. In the same view, journals with a higher impact factor are considered more important than those with lower ones, and journals with very high impact factors, such as the New England Journal of Medicine and Nature, tend to have lower acceptance rates, which usually reflects the quality of the research they publish and the number of articles submitted to them. The impact factor and the other journal indices all have supporters and detractors.
Because the impact factor is an average, it does not on its own reveal whether any specific article in the journal has been cited, although such data are available. An editorial in Nature Materials, reported in 2013, showed that a journal's impact factor is a good predictor of citations to its primary research articles. Across a sample of 100 journals in the physical and chemical sciences, the biological and medical sciences, the earth and environmental sciences and engineering, the 2011 impact factor correlated well with the five-year median of citations to primary research papers published in 2008–2012. The median is the minimum number of citations received by half the papers, so it is robust to outliers and to the shape of the distribution, and citations to reviews, news, editorial material and other non-primary content were excluded from it.
The impact factor has known weaknesses:
- It counts citations to non-primary content such as reviews and news articles as well as to research papers.
- Citations accumulate slowly in many fields.
- The average number of citations per paper can be skewed by a few highly cited manuscripts. The impact factor is an average and is highly skewed, because most publications receive fewer citations than the mean.
- It is unidimensional, because it rests solely on citations, and it does not take into account the other forms of impact that can now be assessed.
- In one group of critics' view, the choice of citations is subjective, the non-pertinence of citations is well known, and the type and size of the journal, its field, the language of publication, self-citations, the coding of articles by their nature and the choice of manuscripts published may intervene.
Two articles of 2007 found journal impact factors often misleading. In one, Rossner and colleagues bought the dataset that Thomson Scientific used to establish impact factors for certain science journals and found numerous misclassifications of front matter as primary research or review articles and an incorrect total number of citations for each journal, discrepancies that affected the impact factors of several journals. In the other, Greenwood used Markov chain Monte Carlo methods to estimate the uncertainty in impact factors and found that only the top and bottom few journals could place any confidence in their rank.
Why it does not judge a paper or a researcher
The impact factor indicates the number of citations and not their quality, and it cannot be used to assess an author's work. Nor can it tell the quality of a particular paper or researcher. The Nature Materials editorial argued that the impact factor does not generally correlate with the performance of individual researchers: when a scientist's papers from five years earlier are ranked by citations alongside the impact factor of the journal each appeared in, the correlation is generally weak at best, with numerous outliers. So a scientist's number of publications weighted by the impact factor of their journals is a poor indicator of their future performance. It recommended article-level metrics for assessing a small subgroup of papers or authors, and no use of impact factors in grant-giving, tenure or appointment committees.
One academic researcher held in 2018 that funding agencies and interview panels often assess research quality, fairly or otherwise, from metrics derived from publications, such as the impact factor, and that feedback on a grant application commonly focuses strongly on the quality of the journals rather than the quality of the work. In his view the impact factor is a good measure of neither the quality nor the impact of research: many papers in high-impact journals are poor science, many excellent papers appear in low-impact journals, and his own most-cited paper is in the lowest-impact journal he has published in. He traces the concept to an artificial scarcity, the limited number of papers an issue can hold, which is a hangover from printed journals. Online journals have essentially unlimited space, yet a journal's impact factor is defined entirely by the editorial selection of its work, a feedback loop that can easily give fields an artificially inflated perceived value.
Selection panels awarding university fellowships are widely acknowledged to be guided by publication metrics, the impact factor the best known of them. The h-index grades a researcher's publication record by the number of articles and the citations to them: an h-index of n means n articles each cited at least n times, so a person with 33 articles each cited at least 33 times has an h-index of 33. In 2012 Acuna and colleagues published in Nature a set of formulas, available online, that estimate a researcher's h-index up to 10 years ahead from the current h-index, the number of articles published, the years since the first article, the number of different journals published in and the number of publications in five named top journals. The formulas were derived from data on neuroscientists, and their authors believed them probably reasonably precise for other life scientists. The formulas predict future h-index more accurately than the current h-index alone. A sceptical columnist pointed out that they were validated on data from 1995 onwards for people who are current researchers, and would have performed less well had people who were researchers in 1995 been followed, since a significant proportion of those will have left research and stopped writing papers. They also rest on the assumption that numbers of publications and citations are good measures of a researcher's quality, which in the columnist's experience is not always the case. In the same columnist's view, over-reliance on improved computer-generated metrics would risk a researcher's personal qualities being completely overshadowed.
The Declaration on Research Assessment
The San Francisco Declaration on Research Assessment (DORA) was developed by a group of journal editors and publishers at the annual meeting of the American Society for Cell Biology in San Francisco in December 2012, as a set of recommendations on the appropriate evaluation of research, including the use of impact factors. Besides specific recommendations for funding bodies, research institutions, researchers and publishers, it makes a general recommendation:
Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist's contributions, or in hiring, promotion, or funding decisions.
Eugene Garfield, the impact factor's creator, strongly advises against using it to assess an individual article or its authors, yet in 2013 it was routinely used to decide who got faculty positions and research funding. DORA was launched with the backing of 82 stakeholder organisations, and as reported in December 2013 it had been signed by 9,305 individuals and 381 organisations. It promotes new digital tools and processes in research assessment and the responsible use of metrics that better align with core academic values. In 2021 governments, funders and research institutions still relied heavily, if not exclusively, on the impact factor as a proxy for the quality, impact and influence of research publications, despite its well-documented flaws.
R Grant Steen called DORA a sprawling document that attempts to serve a variety of needs but may serve none of them well, and criticised it for attacking the impact factor without proposing an alternative. While listing the impact factor's flaws, he argued that it can be used to assess research quality, citing a Wellcome Trust study of 979 papers in which non-blinded expert assessment of importance correlated strongly with the impact factor of the journal of publication. He accepted that it should not be used to assess individual papers or an individual researcher's output, and questioned whether the alternatives are as good.
How journals inflate it
Some journals ask authors to cite their recent pages. One decision letter on a manuscript emphasised the journal's interest in cross-referencing very recent material on the same topic in its own pages, and asked the authors to check its last two years and add all relevant material to the reference list. The two-year window is pertinent, because a journal's impact factor for a year is calculated from citations to articles published in the two preceding years. The journal that sent the letter later signed up to the ICMJE recommendations. The Thammasat International Journal of Science and Technology asked in its instructions for authors for citations to its articles so that its impact factor could be raised; it removed the instruction after scorn on the Retraction Watch blog, and it is not on the ICMJE's list of journals that follow them. Anecdotal evidence suggests that such abuse is not uncommon, and it partly explains an amendment to the references section of the ICMJE recommendations reported in 2015.
A request can also be built into an article type. In 2012 the instructions to authors of Hypertension said that the references of its Recent Advances in Hypertension articles should generally be restricted to those published in Hypertension during the last 2–3 years. Yet of the 25 most frequently cited articles published since 2010 with hypertension in their titles, eight were published in JAMA, The Lancet or the New England Journal of Medicine and only one in Hypertension, on Web of Science data as of 18 July 2012. The international standards for editors agreed at the Second World Conference on Research Integrity, in Singapore in 2010, state that editors should not attempt to influence their journal's ranking inappropriately, and that it is inappropriate to demand that references to the journal's articles be included except for genuine scholarly reasons.
Allen Wilhite and Eric Fong called a request to add unspecified citations from the editor's journal coercive self-citation. In a survey whose findings were reported in February 2012, they invited over 50,000 academics in business, economics, sociology and psychology, and of the 6,672 who responded some 40% were aware of the practice and 20% had personally encountered it. Junior researchers were more likely than senior ones to give in to an editor's demands, and journals with commercial publishers were more likely to coerce than those published by academic societies. The publishing consultant Phil Davis pointed to possible sources of bias, such as responders potentially being more likely than non-responders to be aware of coercion, but accepted that the problem exists. The editors of the two journals that respondents most often named denied involvement.
Referencing a couple of a journal's articles to keep a demanding editor happy may seem trivial to the author who does it, but academic careers depend on impact factors, and citations are their basis. Other abuses have been reported:
- One journal raised its impact factor by 18 ranks by publishing a single article that cited 303 of its own previous papers.
- Authors who submitted to Leukemia received a letter noting that they cited the journal once in 42 references and asking them to add references to articles published in it.
- In protest at the impact factor process, one journal published a single editorial citing a year's worth of its own papers, which raised its impact factor from 0.66 to 1.44.
- The editors of four Brazilian medical journals published articles with hundreds of citations to each other's journals, hoping that not citing their own would escape detection; the ruse, called citation stacking, failed, and penalties followed.
Editorials can boost a journal's impact factor, and excessive self-citation is among their risks. By 2012 Thomson Reuters published impact factors with and without self-citations and temporarily delisted journals that had used self-citation to boost theirs. The director of its Journal Citation Reports felt that Wilhite and Fong's figures were higher than her own data would have led her to expect. Wilhite advocated removing self-citations from impact factors altogether, while acknowledging the need for studies of other disciplines, including the biological sciences. By 2014 Thomson Reuters excluded any journal it found guilty of self-citation abuses from its Journal Citation Reports for two years, which denies the journal an impact factor.
Other citation measures
Alternatives to the impact factor include the h index and the Eigenfactor. Werner Marx points out that the impact factor and the relative citation rate, an article's observed citation rate divided by its mean expected citation rate, both lack normalisation of citation counts for subject and publication year. To overcome this, he describes a percentiles method that gives an impression of the impact an article has achieved compared with similar items in the same publication year and subject category.
In 2016 researchers at the US National Institutes of Health (NIH) described a method of quantifying the influence of a research article from its co-citation network. It yields a Relative Citation Ratio, a field-normalised metric of the citation impact of one or more articles relative to the average NIH-funded paper, offered as an alternative to the journal impact factor for identifying influential papers. The ratio can give supplemental information to decision makers at funding agencies and to others who compare the outcomes of different groups of research investments. The NIH's web tool, iCite, takes uploaded PubMed IDs of articles, optionally grouped for comparison, and displays the number of articles, articles per year, citations per year and the Relative Citation Ratio. It can be filtered by a range of years and by article type, individual articles can be toggled on and off, and the article-level detail can be downloaded as a table.
Reading what citations say
In the view of one developer of a citation index, a citation count is often a poor proxy for a paper's quality, and so are the journal it appeared in and its author list. A count says only that a paper has been cited a lot. Few readers will read all the citations a paper has received, still less do so for every article they meet, and filtering by journal or author instead introduces a bias through the selection criteria. He traces many of these problems, such as numerical reductionism, citation counts and the h-index, to systems built on the traditional citation index designed by Eugene Garfield and productised as the Science Citation Index and Web of Science.
His citation index, built using advances in machine learning and artificial intelligence, shows what each citing article said about the cited paper, taken from the citing article's full text. Citations made in an introduction tend to describe the cited paper's findings, so skimming them quickly shows what a paper was about. Citations made in a discussion show how the findings were discussed in the light of newer research, which helps a reader judge whether they were reproducible before relying on the paper. He holds that researchers had been doing this, perhaps inefficiently and semi-manually, and that the index makes it easier to discover quality research at scale.
Altmetrics and other article-level measures
Citations take months to begin to appear and then, if an article is influential, to build, because a researcher has to notice a paper and then cite it in their own scholarly writing, so an important article can take a year or two to attract a substantial number. An article may be cited only within its own academic niche while influencing specialties beyond it, and methods of tracking that have been inadequate. Neither the impact factor nor citation counts track how an article is discussed, used and cited in blogs, presentation slides, datasets or social media.
Alternative metrics, or altmetrics, emerged to measure influence in online communication. They are tools to measure the real-time reach and influence of an academic article. Altmetrics tools can count how often an article has been bookmarked, cited in a blog post, tweeted or shared online, and can show its influence within hours or days of publication. They can track an article's dissemination across a variety of online communication methods and readership, and can better show the viral nature of research. They measure online views of an article on social media, mainstream media and other sites outside the academic arena, and Altmetric gives its results much more rapidly than the citation scoring systems used before. They can show an article's reach to the general public, which the impact factor was never designed to do. They can potentially be used to assess both the spread of research findings on the internet and researchers' own active contributions to that spread. Altmetric publishes a yearly list of the 100 articles that received the most online attention, and in 2015 the journals most often represented were Nature (14 articles), Science (13) and PLOS One (6).
The tools cover more than articles. Over the decade to 2021 digital tools were developed to track and measure the attention and engagement that published content receives through social media and other web-based channels, and by 2021 they were widely used in medical publications. In 2021 the prominent tools were Altmetric, Plum Analytics and ImpactStory, and the content they surveyed included journal articles, book chapters, software, blogs, datasets, websites and videos. Of particular interest for healthcare is how research publications reach academic and professional digital communities and, beyond them, patients and the general public. In 2013 the Plum Analytics tool measured these categories, for books, figures, presentations and videos as well as articles, and aggregated all the data into author-level metrics:
- usage: downloads, views and document delivery
- captures: bookmarks and saves
- mentions: blog posts, news stories and Wikipedia
- social media: tweets, +1s and likes
- citations: PubMed, Scopus and patents
In 2013 Academia.edu offered a free service for individuals, Altmetric offered plans for individuals, groups and institutions, and ImpactStory was an open-source product for individuals, groups and institutional repositories. In 2022 Altmetric.com tracked 21 metric sources and its main rival, PlumX, more than 40, not all of them relevant to journal articles.
Over the decade to 2021 altmetrics generated great interest as a way of complementing, or potentially replacing, citation metrics and the impact factor. In 2013 there was no indication that altmetrics would soon replace the impact factor. Priem and colleagues found moderate correlations between Mendeley readership and Web of Science citations, and found other altmetric indicators unrelated to citation impact. They also found that articles cluster into impact patterns: one cluster may be heavily read and saved by scholars but not often cited, and another heavily read and cited but not widely shared. They concluded that new metrics are useful as alternatives to citation-based measures for some purposes, are by no means a replacement, and should be deployed alongside them, each complementing the other's strengths. Using both web analytics and traditional citation factors gives a more rounded view of a writer's or researcher's body of work as a whole.
The librarian Jeffrey Beall warned that any system designed to measure impact at the article level will be gamed, rendering the metrics useless and invalid. In 2013 newspapers described click farms of low-wage workers paid to generate social media likes, and socialbots, automated programs designed to interact online and appear to be real people.
Tools and metrics can help authors track the impact of their published work. For one 2016 article, Crossref counted 114 citing articles and books, and the journal's metrics page 201 tweetations. Its Altmetric record showed 2 policy sources, 140 tweeters, 5 Facebook pages, 2 Google+ posts and 216 readers on Mendeley. The profiles of the 140 tweeters placed 27% in the United States, 10% in the United Kingdom, 6% in Australia, 5% in Canada and 33% in unknown countries, and counted 82 members of the public, 32 scientists, 24 practitioners and 2 science communicators.
Using article-level metrics after publication
Many publication teams still judge the success of their publications by the impact factor, a journal-level metric that gives only a rough indication of the impact of an individual publication. One publication consultant holds that looking at the impact of each publication through article-level metrics is far better. These show whether a publication was viewed, shared, engaged with, saved or cited, in news, social media and reference managers, in the peer-reviewed literature, and in grey literature such as blog posts, open peer-review sites, evidence syntheses, policy documents and guidelines. From them a team can learn whether a publication was communicated effectively, whether the right studies were conducted and the right publications developed, which topics attract the most interest, and how all of these compare with competitors' publications. Fundamentally, they show whether the effort put into the publication and the publication plan was invested productively or whether alternative strategies might be more effective.
Each source of these metrics has gaps in its coverage:
- Publishers track article views slightly differently and often do not share the data.
- Social media can be tracked only on platforms that allow machine access, which excludes LinkedIn.
- Some guideline providers, such as the National Comprehensive Cancer Network, do not let automated systems read their reference lists.
- Mendeley is the only reference manager that provides anonymised usage data.
- A provider must choose between broad news monitoring, with more noise, and only the more widely read outlets.
- No service captures discussion of a publication that is not named or referenced.
The value of altmetrics lies in recording different kinds of interaction with a publication, so they can give greater insight than simple citation counts despite the gaps in their coverage. Posting a link on social media differs from citing an article in a peer-reviewed publication and is usually done for quite different reasons, a save to a reference manager may be an intention to cite or only a bookmark, and news outlets tend to pick up newsworthy studies of interest to the wider public rather than those that are scientifically interesting but less immediately relevant to it. Altmetric.com collapses many of its metrics into a single Altmetric Attention Score, which excludes reference-manager saves and peer-reviewed citations and weights the rest, a news article counting about 30 times a tweet. Even so the score is driven almost entirely by news articles and tweets, so it is simply an attention score.
Because the metrics record different actions, the correlations among them can vary. Article-level metrics cannot be reduced to a single dimension. Tomas Rees and Avishek Pal used factor analysis on the metrics of nearly 3,000 publications of phase 3 studies and found that, rather than forming a single score, they fall reliably into three groups:
- Social Impact: Twitter, Facebook, news, blog mentions and Wikipedia citations, with an audience of healthcare professionals, non-specialists, healthcare support staff, patients and other members of the public
- Scholarly Impact: Mendeley saves, citations in peer-reviewed publications and citations in Faculty Opinions, with an audience of specialists, experts, scientists and academics
- Societal Impact: citations in guidelines, policy documents and patents, with an audience of healthcare and policy decision makers and disease management bodies
Weighted so that the typical score in each group is similar, the three make up the EMPIRE Index, which can be compared across articles. Its Total Value score is a weighted average of the three, and a Total Value of 100 equals the average for phase 3 articles published in the New England Journal of Medicine in 2016. An Early Predictor score uses the metrics that accumulate quickly to estimate the future Total Value. The index is fully open for anyone to use.
Metrics show what happened and when, but not how or why, so the platforms' links to the underlying sources, where these are available, are a starting point for deeper insight. Knowing that a publication has been cited is good, and knowing the context of the citation is better. Sentiment analysis, which classifies statements as emotionally positive or negative, is easily confused by medical discussion of death and pain, and it is fundamentally unsuited to understanding a publication's impact. As artificial intelligence advances, automating insight-gathering from unstructured text sources will become easier. More advanced natural language processing can give rapid insight into readers' perspectives, and Scite already assesses whether a citation supports or contrasts with the original publication. This fast-moving area has the potential to transform a second time the understanding of the true impact of publications.