Evaluating Your Evaluations: A Practical Guide

John A. Owen, Ronald T. Murray, Jann T. Balmer, Colleen A. Conner, Judith E. Marymor, Tammy S. Rogers

Abstract


Background: Measuring the effectiveness of continuing medical education (CME) activities has increasingly necessitated relating it to improvements in health care delivery and ultimately to patient outcomes. Recent increased emphasis on patient outcomes involves moving from a traditional evaluation model to an outcomes-based evaluation model. Making this shift to an outcomes-based model requires that CME providers develop a process for “evaluating their evaluations.”

Methods: We designed and completed a qualitative study of our evaluation instruments. Content analysis was used as the textual-analysis method for identifying coherent and important themes in the data, with the evaluation item being our unit of analysis. Once this analysis was completed, we identified the evaluation level and the updated Accreditation Council for Continuing Medical Education (ACCME) criteria that related to each evaluation item. We then developed strategies to increase the percentage of our outcomes-based evaluation items.

Results: The majority (83%) of our evaluation items pertained to learners’ participation and satisfaction, and the remaining items (17%) pertained to learners’ enhanced knowledge and skills. All of our evaluation items were categorized as traditional evaluation items. The development of strategies to increase the percentage of outcomes-based evaluation items facilitated the shift from a traditional to an outcomes-based evaluation model.

Conclusions: Completing this process of evaluating our evaluations provided us with the percentages of our evaluation items that were associated with the traditional evaluation model. The results of this analysis enabled us to develop and incorporate new strategies designed to shift us toward an outcomes-based evaluation model as a means of linking our CME activities to significant improvements in patient care.


Full Text:

PDF

Refbacks

  • There are currently no refbacks.