The practice gap and the needs assessment
A needs assessment documents the gap between how clinicians practise and how they should, and it justifies an educational programme and its learning objectives before the programme is written. It rests on a gap analysis and evidence of need, aligns learning objectives with the gaps and is organised in three sections; certain properties improve its chance of funding, and surveys of practitioners describe good and poor practice in writing one.
The practice gap
Accredited CME rests on the tenet that an activity must improve how clinicians practise and so improve patient health, so CME providers must first identify where clinicians' professional ability fails to meet best-practice standards and then design education that raises their competency.
A professional practice gap is the difference between what providers already know and what they should know to be competent or expert in their field. A clinical practice gap is the gap between current and optimal medical practice, such as a gap in patient care or a lack of knowledge, skills or attitudes, and it justifies the need for education.
Educational linkage is the process that guides the development of CME through a series of interdependent products, each informing the next when the process is done correctly, so that direct connections can be drawn between an identified deficiency in physician performance, the educational intervention, a change in how clinicians practise and an improvement in patient health outcomes. The sequence runs from identifying the practice gap and documenting the educational need, through developing the educational objectives, the content and the programme format and design, to writing the activity, and then to evaluations and outcome assessments.
The first step in creating CME is gap analysis, also called discrepancy analysis: the study of the difference between two competency points, in which current practice behaviour is compared with an ideal or accepted standard of performance. Competency is the combination of knowledge, skills and attitudes that embody proficiency. The Accreditation Council for Continuing Medical Education defines a clinical practice gap as the difference between health care processes or outcomes observed in practice and those potentially achievable on the basis of current professional knowledge, and it requires accredited providers to address relevant competency gaps by designing activities around the educational needs (knowledge, competence or performance) that underlie their own learners' professional practice gaps. Gap analysis gives direction for the content and also clarifies what measurable outcomes the programme should achieve, by determining what learners ought to be able to do differently in the clinical setting after the activity.
Identifying a gap means establishing an association between two things, a lack of clinician proficiency in a specific area and the poor health outcome it causes, and how the gap is pinpointed depends on which of the two is known and which has to be discovered. In the first and commonest situation, a known gap in competency is associated with a known poor outcome, and this is the simplest to document. In the second, a poor outcome is known and the search is for the competency gap that causes it. In the third, the gap is known and its effect on patient outcomes has to be verified. For example, failure to follow established guidelines for scheduling follow-up visits after initial diagnosis and prescription can be paired with poor medication adherence among adults with attention-deficit/hyperactivity disorder: in the first situation the association is known, and in the other two it is a question to be answered from reliable sources of information.
The needs assessment
The needs assessment is the component of an educational grant proposal that describes why a specific programme should be developed, and it is one of the most important parts of the proposal. It typically appears as a single section of a larger request for commercial support that a CME provider submits to a funding body such as a pharmaceutical company. It is more than a literature review and includes data-driven gap analysis and root-cause investigation. Data have shown that CME programmes based on well-conducted needs assessments are effective in changing providers' practice behaviours.
The assessment determines what the intended audience needs to learn from the planned activity, and it is the foundation of the whole programme, demonstrating that there is a definite need to improve clinician competency through education. It identifies the gap between current and best practice and is a systematic means of collecting the information that helps to determine the instructional solutions to close it; that information then identifies the programme's goals and learning objectives. Its four key features are the current clinical practice, meaning what learners currently know and do; the best clinical practice, meaning what they should know and do; the clinical practice gap between them; and the learning objectives.
A needs assessment for a CME programme typically analyses clinicians' deficiencies in knowledge, confidence, competence or performance in a therapeutic area. These gaps guide the development of the educational intervention, with improved patient care as the ultimate goal. An assessment can run from one page (less than a page, by another account) to more than ten, depending on the number of gaps, the quantity of supporting evidence and the resources available.
A strong assessment is usually built around a core literature review but may draw on other evidence, and its narrative describes one or more gaps between optimal practice, such as that laid out in practice guidelines, and actual practice in the field. The assessment justifies the programme's agenda: it is the basis for the learning objectives, which in turn justify the agenda, the content and the choice of faculty.
Evidence of need
Identifying the need for a specific activity, from reviews of the literature, from discussions and from analysis of current practice, is a vital first step, and writers can play a crucial part in the preliminaries, the needs assessment and the learning objectives, by helping to identify and delineate the gaps in knowledge. A change in guidelines that may affect a physician's practice is an example of such a need.
Every gap in clinician competency and every need for education must be objectively demonstrated. It is not enough for a programme director to decide that a course on geriatric diabetes would be a good idea; there has to be a way to prove the need exists. A programme that does not bear directly on some aspect of patient care cannot be CME, however interesting or beneficial it is to clinicians. A course in investment finance, for example, is beneficial to the clinician but has no direct relevance to patient care, so it cannot offer credit.
The data for a needs assessment come from three areas. Inferred needs cover new diagnostic or treatment methods, technologies and agents; new indications for current agents; the opinions of key experts about advances in medical knowledge; and regulatory and legislative changes affecting patient care. Verbalised needs cover survey results from potential learners, learner evaluations of previous activities, and the consensus opinion of members of a medical specialty group. Proven needs, based on objective data sources, cover guidelines and recommendations published by professional societies, quality assurance data, reviews of journal articles, and morbidity and mortality data.
A second classification divides the sources of information on educational need into documented, presumed and expressed needs. Documented needs are proven through objective external data sources and are typically evidence-based: epidemiologic data, morbidity and mortality data, quality assurance and audit data, changes in health care legislation or regulation, and the evidence-based literature. They are the commonest category used to substantiate a need; for example, prevalence and incidence data show that adults aged 65 and over are disproportionately burdened by type 2 diabetes and its complications. Presumed needs are inferred from developments that suggest clinicians will need training, such as changes in diagnostic criteria, emerging treatments, innovations in technology and new regulations; for example, a professional society's consensus statement provides a framework for treatment goals in older adults with diabetes. Expressed needs are conveyed directly by clinicians, and evidence of them is collected in a variety of ways, including targeted surveys and interviews with medical experts and clinical administrators. Sources of information on educational need also include new methods of diagnosis or treatment, new medications, new technology, new guidelines, regulations, expert opinion, evaluations of past activities, patient inventories, consensus reports and clinician surveys.
Gaps can be identified from sources including the literature, interviews with subject matter experts and surveys of clinicians. Inferred gaps are often derived from guideline updates, drug approvals and new technologies such as diagnostic tests; verbalised gaps come from interviews and surveys, which makes them a valuable source; and statistical gaps come from epidemiological data and morbidity and mortality reports. The writer needs to keep the end in view, better patient outcomes and how to achieve them, and then examines each gap for the educational needs beneath it: the deficits in knowledge, strategy, skill, performance or systems that could contribute to the problem.
One instructor of a CME writing course requires that one source of information about the need in each student's needs assessment come from a live interview, since much can be learned by talking to clinicians directly about the challenges they face that meta-analyses and review papers do not record.
Evidence-based programmes, such as the prescriber education programme that a Massachusetts law enacted in 2008 funds, require research studies to identify gaps in provider knowledge, absent knowledge, or inaccurate understandings that need to be addressed.
One needs-assessment writer employed by a CME provider searches the medical literature and the internet for recent clinical trial data, practice guidelines, key opinion leader perspectives and other material that justifies education in a therapeutic area and identifies gaps in clinicians' knowledge or practice habits, and also has access to survey data, participant data and outcomes data from past activities. As new grant requests arrive, this writer schedules start-up meetings with the therapeutic team to determine the focus of the assessment, the target audiences, suggested faculty and tactics.
At a medical liability insurer, physicians submit medical-legal questions on issues such as releasing medical records, treating minors and informed consent to the risk management department's telephone and email help lines. In 2008 the staff reported that questions on prescription drug diversion, the use of licit drugs for illicit purposes, were arriving with increasing frequency: some physicians suspected patients of trying to divert drugs, and others had caught staff members doing so. Given these queries, the insurer published a continuing medical education article to help physicians learn how to prevent it.
Learning objectives and the alignment of gaps
A learning objective is what participants can expect to get out of the activity. Unlike the overall goals, learning objectives are precise and measurable aims that define how the programme will improve participants' knowledge and skills and subsequently enhance patient outcomes, and each is, in essence, an action statement that contributes to the goal being achieved. For a programme on colorectal cancer, for example, the goal might be to provide education that will enhance participants' competence in applying what they learn to patient care strategies, while its objectives are more specific and focused. The regulatory bodies that set accreditation standards for continuing medical education require programmes to state objectives based on the identified needs of the target audience.
Learning objectives should be prepared before the content and should be a well-defined number of actionable items. The website of the Accreditation Council for Continuing Medical Education is a resource for preparing CME materials, down to the choice of words for learning objectives.
When writing a needs assessment, writers are often asked to identify and research the practice gaps, meaning the difference between actual and ideal clinical practice, also known as the professional practice gap, and to develop learning objectives and desired outcomes, which are summarised in an alignment table.
An example alignment table on advanced non-small cell lung cancer lists as gaps that many newly diagnosed patients do not receive biomarker testing, or do not receive appropriate therapy despite it, and that oncologists may be unaware of current guideline recommendations for pretreatment evaluation, may need education on the available targeted therapies and may have limited knowledge of trial data on emerging ones. Its objectives include describing the role of biomarker testing, integrating recent trial data on FDA-approved targeted therapies and evaluating investigational agents, and its outcomes include using current evidence-based guidelines to make informed treatment decisions. Writers are sometimes asked to develop objectives from the gaps they have identified, but it has become common for them to be asked instead to gather evidence that justifies objectives the employer or client has already drafted. Objectives are often written to Miller's progression from acquiring knowledge to competence and performance. They should close the identified gaps and needs and target the behaviours learners must change to give better care, and their action verbs should come from the appropriate level of Bloom's taxonomy. To make sure the objectives are appropriate, writers should ask about the educational format before writing the needs assessment.
The three sections of an assessment
The assessment's purpose determines its length and format. Providers write needs assessments to document the need for education, which fulfils a requirement of the accreditation council; to guide the development of the education programme; and to include in grant proposals for commercial funding, as validation of the need for education. An assessment written to fulfil the documentation requirement can be a bulleted list or a table, whereas one written for a grant proposal is formatted as a report that follows the page limits and other requirements in the grantor organisation's request for proposals. Providers have not developed a common template, but certain types of information are universally included, and the result is a succinct report that describes the condition of interest and its epidemiology, compares how it is treated and managed with what guidelines recommend, and shows how education can change clinicians' behaviour so that their practice is better aligned with best-practice standards, with the goal of improving patient health outcomes.
In one experienced writer's account, the format of a needs assessment is not fixed and varies by the company the writer is working for, so flexibility matters. The guidelines governing accredited education are also frequently modified, and needs assessments change to conform with them. The needs assessment may be one component of a proposal a team is assembling, so the writer has to work with others and understand their perspectives.
Whatever the format, the information can be organised in three main sections. The introduction gives the purpose of the assessment and the source of the documentation; identifies the condition, disease or clinical problem being addressed, with its cause and risk factors; defines the affected patient population by demographics such as sex, age, ethnicity and socioeconomic status, by prevalence and incidence, and by morbidity and mortality; and describes the condition's effect on society at large, both economic, such as work days lost and health care costs, and as a trend, predicting the impact if the condition grows unchecked. The second section covers best and current practice: the best-practice standards, meaning the way clinicians should practise; the gap in clinician competency, meaning the way they currently practise; and the barriers that prevent best practice. The third presents education as the solution, explaining the value of continuing medical education and giving examples of its success in changing clinician behaviour. In a grant proposal, these examples can lead into how the activity for which funding is sought has been designed to improve competency.
A sample needs assessment on the appropriate management of hypertension in primary care shows how the three sections are written.
Its introduction labels each element in turn. The goal is to identify potential professional practice gaps in, and barriers to, the appropriate management of hypertension in the primary care setting [purpose]. Hypertension is increasingly diagnosed in both older and younger individuals, and the rate of uncontrolled hypertension is suboptimal [condition identified]. Clinical practice guidelines, physician and patient surveys, government reports and systematic reviews of the current literature document the gaps and barriers [sources]. Hypertension affects 31% of the United States population, and among adults aged 45 and older it is the number one diagnosis at visits to office-based physicians and hospital outpatient departments [prevalence and incidence]. Individuals who are normotensive at 55 have a 90% lifetime risk of the condition, so an ageing population makes it a greater problem still [risk and trends]. The rate among younger individuals aged 24 to 32 had recently been found to be 19%, against a previous estimate of 4% [incidence and prevalence]. Hypertension carries considerable morbidity and mortality, is a major risk factor for cardiovascular disease and stroke, is the cause of death in one of every seven deaths in the United States, and accounts for nearly half of all deaths related to cardiovascular disease [morbidity, mortality and risk]. It places huge economic demands on an overburdened health system, with the American Heart Association estimating direct and indirect costs at more than $93.5 billion a year [effect on society]. The figures date from 2013.
The example's second section alternates best practice with current practice to demonstrate the gap. It opens on a complicating feature: 46% of individuals with the condition have uncontrolled blood pressure, above the goal set by the seventh report of the Joint National Committee. Controlling hypertension according to evidence-based guidelines involves identifying it as a diagnosis, selecting the most effective drug therapy, and routinely monitoring blood pressure and adjusting medication to reach a target. The committee defines hypertension as a blood pressure of 140/90 mm Hg or more and recommends monitoring at approximately monthly intervals until the pressure falls below that limit; it recommends a thiazide-type diuretic as initial therapy, advises that more than one agent is usually required, and recommends intensifying therapy where the goal has not been reached [best practice]. Physicians report 94% familiarity with the guidelines, but appropriate treatment increased shortly after their dissemination and was not sustained beyond a few years [current practice]. Fixed-dose combination medications are more effective than single agents given together and enhance compliance [best practice], yet they remain underused [current practice]. The approximate 50% rate of controlled hypertension indicates that physicians are not adhering to the guidelines, for reasons that are unclear. Clinical inertia has been identified as the primary barrier to controlling hypertension through adjustment of drug therapy: in one study it was the reason 63% of patients with uncontrolled hypertension did not have their medications changed, and predictors of no change included diabetes as a comorbidity, a blood pressure less than 10 mm Hg from target, a patient load of more than 26 a day and non-Hispanic white ethnicity. In other studies, a pressure within 5 to 10 mm Hg of target was less likely to prompt a change. Physicians themselves report waiting until the next visit before intensifying therapy, inadequate time to discuss management with patients, patient-related factors such as non-compliance with drug therapy or follow-up, and staff-related factors such as inaccurate blood pressure measurement. Several studies document patient-related factors as major contributors to uncontrolled hypertension, so improving control relies on strategies that motivate patients to adhere to therapy, lifestyle changes and follow-up.
The example's third section argues that education can close the gap the second section documented. Controlling hypertension requires a comprehensive approach, with improvements in health care delivery systems, physician behaviour and practice, and patient adherence to prescribed treatment, and continuing medical education gives clinicians an excellent opportunity to review guidelines and learn new strategies for treatment and management [value of education]. Several strategies have improved control, and interactive programmes have been shown to change how physicians manage hypertension by implementing them [education is effective]. Activities that focus on evidence-based strategies can improve control and ultimately reduce the morbidity and mortality of the condition [education is effective]. Because patients do not adhere to antihypertensive therapy, activities that highlight documented interventions to enhance compliance and patient education can help physicians engage patients in self-management [education is effective].
Properties of a fundable assessment
Since funding for CME programmes is increasingly difficult to obtain, an assessment must stand out to gain approval, and to do so it must have the following properties.
Be specific. The data must demonstrate a mechanism for measuring and publishing the outcomes of the activity, so the assessment should target specific aspects of a condition rather than give a general overview of it. A literature review may be informative yet do little to document the focused needs of the target audience, and so may fail to gain funding approval. An assessment that discusses data about particular practice gaps, describes how the proposed programme would target those needs and shows how education would overcome the specific issues is more likely to be funded.
Incorporate evidence-based data. Gaps in provider knowledge are typically identified by two routes: published or collected data, such as evidence-based journal articles, healthcare statistics on government websites, quality assurance reports and pre-test and post-test results; and physician self-assessment, including self-testing results and quality assurance reports. The data should be linked to the target audience's specialty where possible, and to a particular geographic region where relevant.
Present fair balance. Even where one pharmaceutical company is the sponsor, it is in the best interest of the target audience and their patients that the programme presents a fair balance among competing therapeutic and diagnostic choices.
Include measurable learning objectives. Objectives are identified by investigating the issues that led to the gap, and they must be measurable through observation or documentation. The ideal objective has three components: the learner, a measurable action verb, and the desired result of learning. The action verb is a key component, and verbs differ. Verbs such as discuss, identify and perform are more measurable as a direct outcome of a programme and are considered more effective; verbs such as understand, appreciate and learn are considered weak because they are less measurable, and should be avoided.
Match practice behaviours, gaps and objectives. The current and best practice behaviours should match the practice gaps, and the objectives should reflect those gaps. In a programme on colorectal cancer, for example, the current behaviour is that clinicians lack knowledge of the available therapeutic agents for metastatic disease and of how to select the most appropriate options for individual patients. The best practice is that clinicians must stay abreast of available and emerging data to make individualised, evidence-based decisions about those agents. The corresponding gap is that clinicians lack the most up-to-date clinical trial data on available and emerging agents, taking into account patient risk factors and drug-associated adverse events and toxicities, and the learning objective is to select appropriate therapy for an individual after considering patient characteristics, clinical factors and the safety and efficacy of the available agents. Another row of the same table works the chain for screening: physicians are unaware of the current guidelines on risk stratification and alternative screening tests, they must follow the updated guidelines and use an appropriate screening regimen, and the objective is to determine the appropriate risk stratification for an individual patient and select the appropriate screening strategy. A further row, on non-adherence, ends in an objective to identify the therapy-related, patient-related and provider-related factors leading to medication non-adherence.
The assessment should be crafted with provider performance improvement in mind, it must focus on important issues relevant to participants' practice, and it must provide evidence-based information on common practice problems. To improve the chance of funding, the writer must ensure that the assessment is specific, gap-based and free of commercial bias, and that it includes measurable learning objectives.
Who writes needs assessments
Medical writers frequently play a central role in producing needs assessments. They are generally involved once the data on educational needs and clinical practice gaps have been collected, at the third and fourth steps of the CME hierarchy.
Needs assessments are widely used by publishing and education companies within the accredited system to identify the deficits that underlie practice gaps, and as commercial support within that system grows, so does the number of assessments needed. Developing the assessment often falls to an in-house or freelance medical writer, whereas assembling, editing and submitting the proposal are usually handled by staff employees.
Freelance writers who develop needs assessments for education grants form a tight-knit community of practice, though they work from home offices across the United States, and its members often call themselves CME writers.
Writing the needs assessments used in grant proposals to fund accredited continuing medical education activities in the United States had become a specialised niche market for medical writers, and best practices changed constantly in response to market conditions, industry needs and technology.
Continuing-education writers need to master the art of writing needs assessments; for many of them, it is the commonest document they are hired to produce.
Writing needs assessments gives a writer a view of the whole grant proposal, with full details of the proposed initiative and a chance to learn from its contents and process. A writer experienced in needs assessments may come to be employed to write other sections of the proposal.
Training programmes, employer-based or university-based, are scarce, and system-level problems such as labour market supply and demand, industry expectations and standard project lead times may lie beyond any individual writer's control. Writers may gain more from working on problems within their control: their professional knowledge, skills and attitudes, and delivering the kinds of assignment providers most demand, and so becoming more competent.
How writers build assessments in practice
Effective continuing education begins with a high-quality assessment. Unwarranted variation in a health care process can be a sign of poor quality. In 2010 or 2011 (accounts differ), a small pilot analysis of assessments by various authors, collected from several sources including clients and an AMWA roundtable, found a great deal of variation. The assessments varied in their sources of evidence, in how the evidence was presented and cited, in whether writers interviewed experts or included charts and graphs, and in whether they used American Medical Association, American Psychological Association or hybrid reference styles.
Annual surveys of needs-assessment writers followed from 2014. The 2014 survey, open from 3 to 19 September 2014, asked how long respondents had been writing assessments, what evidence they used to assess practice gaps and how they presented it; it received 109 responses, 103 after excluding six from people who had never written an assessment. The 2015 survey, open from 13 September to 12 October 2015, received 67 responses, 62 after the same exclusion.
In 2014 most respondents had been writing assessments for five years or more, followed a template supplied by the client or employer, included at least one chart or table, and considered a review of the medical literature the most essential evidence. In 2015, 68% had written their first assessment more than five years earlier, 47% had written assessments as staff employees in the previous six months and 34% as freelances. Asked what data clients or employers consider essential to a first-rate assessment, 69% named a medical literature review, 63% practitioner surveys, 58% practice guidelines, 53% interviews with key opinion leaders, 31% outcomes data and 24% evaluation reports (N=62). Respondents in 2014 used charts or tables to show the alignment of gaps, objectives and outcomes, and in 2015 the commonest column headings were, in descending order, "Learning Objective", "Practice Gap" and "Desired Outcome", with many variations. Asked how often their assessments include the perspective of patients or their care partners, 8% said always, 5% in 67% to 99% of assessments, 16% in 34% to 66%, 56% in 1% to 33%, and 15% never (N=60). In the survey authors' professional experience, patient perspectives were becoming more common.
When writing assessments to justify education for physicians and nurses, writers often ask about barriers to effective practice, and the survey put a similar question to the writers themselves: respondents rated a list of barriers by importance to professional practice on a scale from 1 (low) to 10 (high) (N=60). Clinical practice guidelines out of date or non-existent scored highest (7.1), followed by not enough lead time for adequate research (7.0), few published data on the topic (6.3), a poor match between the identified gaps and the desired outcome levels (6.0), an unclear assignment from the requester (5.3), several ways to interpret clinical studies and limited space in the needs assessment section of the supporter's submission portal (5.1 each), uncertainty about which information to include (4.8) and how to present it (4.2), and fear of appearing biased (3.5).
The two years of survey data yield three cumulative "recipes" for assessments supporting requests for commercial funding, which differ in their ingredients and in the time and money needed to produce them.
- Standard: a medical literature review, a practitioner survey, reference to clinical practice guidelines, one or more key opinion leader interviews, and an alignment chart with columns labelled "Learning Objective", "Practice Gap" and "Desired Outcome".
- Better: adds the perspective of a patient or patient advocacy group on patient-level gaps, text or a chart showing outcomes data, and evaluation reports from participants in previous activities, which some practitioners consider part of a standard assessment.
- Deluxe: adds references to national healthcare quality standards, such as those published by the National Quality Forum, the Agency for Healthcare Research and Quality or the Patient-Centered Outcomes Research Institute, and evidence of change measured against a validated quality benchmark.
The lack of up-to-date practice guidelines underscores the importance of CME and of the writers who develop it. The patient's perspective had gained prominence in CME, though whether it would grow or shrink remained to be seen. To follow best practice, a chart showing the alignment of a programme's key components must use the column headings respondents reported. Respondents named the literature review the single most important type of evidence two years running, so unless another type overtakes it in the next annual survey, training for future assessment writers should teach how to carry out a first-rate review of the medical literature; since needs assessments are also an important part of publication planning, the skill could benefit other medical writers too.
Poor practices reviewers report
A 2018 survey, open from 5 to 19 October 2018, asked for the first time an open question: what unprofessional or poor practices, if any, have you noticed while reviewing needs assessments written by others that might be appropriate for future survey research? Two reviewers working separately sorted the answers into categories defined in advance, and for an answer naming two or more poor practices they chose the category that best described the most salient.
There were 104 responses, roughly balanced between freelances (50%) and staff employees (44%); 60% of respondents had written at least 26 assessments in their careers and 44% more than 50, a figure another account gives as 45%. Sixty-seven respondents described poor practices. Sources and referencing was the largest category for both reviewers, at 19 responses each, followed, for the first and second reviewer respectively, by irrelevance or poor focus (13 and 15), organisation, coherence and readability (11 and 14), plagiarism, fabrication and bias (10 and 9), outdated information (7 and 4), grammar (4 and 2) and other (3 and 4).
The poor practices respondents described concern evidence, focus, reasoning and presentation. On evidence: gap statements not supported by evidence, or gap claims exaggerated without solid information behind them; insufficient, outdated (one respondent said ten years old) or inapplicable references; current guidelines not cited; tertiary sources heavily cited without the primary one, and secondary sources listed; only abstracts read; small-scale studies or studies not relevant to the target population; reliance on what one provider sees in practice; press releases from pharmaceutical companies, news articles and sources that are not peer reviewed; a literature search not thorough enough to find the references that back up the need; unreferenced statements in support of key messages; incomplete identification of the key opinion leaders quoted; and non-standard citation formats. One respondent described news articles as a stretch but allowed that they sometimes help emphasise an initial peer-reviewed point. On focus: an assessment that is 100% literature review, or a data dump that never reaches the gaps in clinical practice or why the need is unmet; too much background on the disease or its whole natural history, material the sponsor already knows, and clinical data better suited to the activity itself; pages of clinical trial results that do not state clearly whether the gap is emerging data, conflicting data, practice contrary to guidelines or patient selection; a statement of why the education is wanted, sometimes for personal goals, in place of a need; what one respondent called the book-report approach, which surveys existing and emerging developments without showing any specific need beyond clinicians having trouble keeping up; and a myopic discussion of drugs at the expense of other forms of therapy such as surgery or radiation, or a failure to discuss the potential grantor's class of agent. One respondent holds that the writer must grasp the current thinking and issues relevant to the gap, and describes an earlier writer on obesity who overlooked new biological developments in the field and missed how clinicians needed to change their behaviour. On reasoning: failing to link gaps to poor patient outcomes, poor coverage of the reasons for gaps, no examination of clinicians' attitudes and beliefs, a poor grasp of the subject matter, and low data literacy. On presentation: dense narrative where summaries and bullets are needed, no figures or charts, poor organisation and unclear structure, excessive length and redundancy, editorialising about the politics of health care, clinical data pasted without context, poor grammar and punctuation, typos and cut-and-paste errors, and failure to proofread. Some assessments do not meet the accreditation council's requirements.
Other complaints concern the chain from gap to objective: too many gaps or learning objectives; objectives written sloppily or not easily measurable; several objectives for one gap or the reverse, where the ratio should mostly be one to one; a narrative that does not support the objectives; pretest and posttest questions that do not reflect mastery or an intent to change practice, and evaluation that assesses the quality of the intervention rather than its impact on patient outcomes; and outdated outcomes data offered as proof of need, or heavy reliance on outcomes data at the expense of current science. Others concern recycling: cookie-cutter assessments moved from one therapeutic area to another, pastiches of earlier assessments, failure to rewrite adequately when given an assessment to update, canned language, references misnumbered when an assessment is refreshed, details left over from deleted versions, and, as sponsors report, freelances submitting essentially the same proposal for several clients. One respondent warned that refreshing an old assessment sounds like a shortcut but is often a detour into a rat's nest of detail that loses the big picture, and that sometimes it is better to tear the house down and rebuild it.
Complaints about integrity include plagiarism, made-up faculty quotes and outcomes data, outcomes data misrepresented or embellished to favour the need for education, an assessment spun to favour the potential grantor's product or made so friendly to it that it strays from a true educational directive, and too much industry influence. Several responses also raise questions about the start of the work: whether there is a chance to talk to prospective faculty, whether the sponsor's request for proposals arrives first, and, for joint providership, whether the accredited provider has given its input before submission. One respondent finds that most providers do not understand what a needs assessment is, but that asking them to brain dump on the topic and taking notes yields everything needed to rewrite their assessment. Survey data are themselves among the weak evidence named: a sample of physicians with a small response rate, no comparison to justify generalising, and no stated limitations.
Because the reference list is the heart of a literature review, which respondents in four of five earlier surveys named the most essential evidence in an assessment, weak sourcing may suggest an inexperienced, rushed or sloppy writer who had trouble finding valid data, identifying sources or compiling a clear and orderly reference list. A reader may notice these problems if an editor does not identify and correct them, but even skilled writers may underperform when given too little lead time, poor templates or vague editorial direction. Either way, a skilled and meticulous researcher who can work quickly with a client to overcome obstacles, identify a bona fide educational need and marshal detailed evidence for it adds great value to the process. A workshop or online exercise on conducting a high-quality literature review may help, including how to appraise clinical study results and extract data to support statements of need, and a separate one for the editors, medical directors and others who hire freelances may also be useful, on two-way communication, reasonable deadlines, editorial direction and incorporating evidence-based best practices into proprietary templates. Given the reports of plagiarism, fabrication and commercial bias, the accreditation council's effort at the time to revisit independence appeared necessary and timely.
Despite resources and hands-on workshops that teach best practice, writers still struggle with referencing, organisation, coherence and readability. This may be an opportunity for the industry to consider new best practices that encourage standardisation and eliminate some of the poor practices. One of the two reviewers proposed that the industry adopt a structured template for writing assessments, similar to those already used in regulatory writing, and held that such standardisation may enable writers to produce an organised, readable narrative that identifies gaps clearly supported by evidence.
The commonest poor practices concern fundamental skills, in the literature review and in writing itself, while the least common concern grammar. The referencing complaints (references to guidelines missing, original sources not cited, references that do not support the gap, older references) indicate a gap in knowledge about the most appropriate data sources, and point to a need for education on the basics of citation and on conducting a literature review that supports a practice gap. A needs assessment should be free of irrelevant information and focused on the key message readers need to make decisions.
Where poor practices come from beginning writers, fundamental education on writing needs assessments can enhance their skills, while a different tactic is needed where they come from experienced practitioners. Self-plagiarism is unlikely to be the offender in needs assessments; plagiarism there may be text lifted wholesale, paraphrase without citation, or source-based, with an incorrect source cited or a secondary source read and only the primary one cited. Knowing which type occurs can help target education on citing sources appropriately and on interpreting the literature to create original statements.