Setting outcome levels and evaluating an activity
A continuing medical education activity is expected to show what it achieved. Its design aims at seven outcome levels, the field has moved from counting attendance and satisfaction toward the higher levels, and an evaluation draws on frameworks and data sources to measure the outcomes.
The seven outcome levels
Seven levels of potential and beneficial outcomes for an activity, presented by Donald Moore and colleagues in 2009, were widely adopted by the CME community as the de facto levels of outcomes for physician education: level 1 is attendance, level 2 satisfaction with the location, accommodation and content, level 3 what the physicians learned, level 4 the new competence acquired, level 5 the change in practice behaviour, level 6 the effect on patient health outcomes, and level 7 the effect on community health.
The move from attendance to measurable outcomes
Before 2006 the impact and effectiveness of physician education were measured by how many physicians attended and how satisfied they were, with little attention paid to the five levels above attendance and satisfaction. The Accreditation Council for Continuing Medical Education's accreditation criteria of 2006 focused the enterprise on a learner-centred, continuous improvement model requiring providers to identify practice and knowledge gaps, create relevant learning objectives, and measure the outcomes of activities. By 2012 the outcomes requirements in place since 2006 were recognised by US accrediting bodies, providers, funders and a growing number of physicians.
By 2010 educating health care providers for the sake of education was no longer good enough. The focus moved to performance improvement, education with a measurable impact on a participant's performance. In a ninety-minute programme on the risk of type 2 diabetes in the Hispanic population, for example, participants are expected to complete pre- and post-activity surveys showing how the education did or did not affect their overall level of care, and some broader initiatives ask physicians to review data from patient charts over several months. The shift let statistical specialists emerge in CME, with activities regularly designed around multilayered surveys and post-activity analyses that give a provider a tangible measure of the return on education, as opposed to return on investment, a term common in the promotional world. It also brought additional opportunities for medical writers and editors able to work with these experts in measuring educational outcomes, pairing carefully worded questionnaires and surveys with specific educational activities. In 2012 one practitioner saw the enterprise settling into one more clearly dedicated not only to physician education but also to the improvement of medical practice and community health.
By 2012 providers were improving their skills at developing measurable outcomes and ploddingly moving up Moore's scale from levels 1 and 2 more reliably to levels 3, 4 and 5 (did physicians learn, has their competence improved, are they changing how they practise), while innovative ways of measuring improvements in patient health at level 6 and community health at level 7 were being discussed at conferences and piloted. One practitioner invited readers to imagine physician education with a notable measurable impact on four major US lifestyle epidemics: heart disease, diabetes, obesity and allergy.
Evaluation frameworks and their data sources
After an activity has taken place, a framework can be used to assess how effectively it changed the learner's knowledge, competence and performance and patient outcomes. The reach, effectiveness, adoption, implementation and maintenance (RE-AIM) framework evaluates the activity's impact on public health, while the Kirkpatrick-Barr framework and Moore's levels assess the learner's educational outcomes. Moore's levels are the most commonly used framework for evaluating outcomes in continuing medical and health professions education, and were developed to combine assessment with planning and teaching throughout the learning process. In the scale as presented in 2024, level 3 is divided into declarative and procedural learning, and each level has potential data sources: attendance records for participation; post-activity questionnaires for satisfaction; pre- and posttests and self-reported knowledge gain for declarative and procedural learning; observation in an educational setting and self-reported competence or intention to change for competence; observation of performance in the patient care setting, patient charts, administrative databases and self-reported performance for performance; health status measures recorded in patient charts or administrative databases and patients' self-reported health status for patient health; and epidemiological data and reports and community self-report for community health. Participation and satisfaction are easy to assess with feedback surveys, but they give no information on how much the participant learned or whether the knowledge is integrated into clinical practice. Improvements in learning and knowledge gaps can be evaluated with pre- and posttest questions, competence can be assessed from reflective statements after the activity, and commitment to change can be gauged by semistructured interviews, while improvements in patient and community health can be estimated later from administrative databases and epidemiological data and reports. Using a framework to evaluate activities continually after they have taken place helps to improve them and shows their value to patients and the health care community.