Introduction: Evaluation of emergency medicine (EM) learners based on observed performance in the emergency department (ED) is limited by factors such as reproducibility and patient safety. EM educators depend on standardized and reproducible assessments such as the objective structured clinical examination (OSCE). The validity of the OSCE as an evaluation tool in EM education has not been previously studied. The objective was to assess the validity of a novel management-focused OSCE as an evaluation instrument in EM education through demonstration of performance correlation with established assessment methods and case item analysis.
Methods: We conducted a prospective cohort study of fourth-year medical students enrolled in a required EM clerkship. Students enrolled in the clerkship completed a five-station EM OSCE. We used Pearson’s coefficient to correlate OSCE performance with performance in the ED based on completed faculty evaluations. Indices of difficulty and discrimination were computed for each scoring item.
Results: We found a moderate and statistically-significant correlation between OSCE score and ED performance score [r(239) =0.40, p<0.001]. Of the 34 OSCE testing items the mean index of difficulty was 63.0 (SD =23.0) and the mean index of discrimination was 0.52 (SD =0.21).
Conclusion: Student performance on the OSCE correlated with their observed performance in the ED, and indices of difficulty and differentiation demonstrated alignment with published best-practice testing standards. This evidence, along with other attributes of the OSCE, attest to its validity. Our OSCE can be further improved by modifying testing items that performed poorly and by examining and maximizing the inter-rater reliability of our evaluation instrument.
Introduction: Electronic health records (EHR) have become ubiquitous in emergency departments. Medical students rotating on emergency medicine (EM) clerkships at these sites have constant exposure to EHRs as they learn essential skills. The Association of American Medical Colleges (AAMC), the Liaison Committee on Medical Education (LCME), and the Alliance for Clinical Education (ACE) have determined that documentation of the patient encounter in the medical record is an essential skill that all medical students must learn. However, little is known about the current practices or perceived barriers to student documentation in EHRs on EM clerkships. Methods: We performed a cross-sectional study of EM clerkship directors at United States medical schools between March and May 2016. A 13-question IRB-approved electronic survey on student documentation was sent to all EM clerkship directors. Only one response from each institution was permitted. Results: We received survey responses from 100 institutions, yielding a response rate of 86%. Currently, 63% of EM clerkships allow medical students to document a patient encounter in the EHR. The most common reasons cited for not permitting students to document a patient encounter were hospital or medical school rule forbidding student documentation (80%), concern for medical liability (60%), and inability of student notes to support medical billing (53%). Almost 95% of respondents provided feedback on student documentation with supervising faculty being the most common group to deliver feedback (92%), followed by residents (64%). Conclusion: Close to two-thirds of medical students are allowed to document in the EHR on EM clerkships. While this number is robust, many organizations such as the AAMC and ACE have issued statements and guidelines that would look to increase this number even further to ensure that students are prepared for residency as well as their future careers. Almost all EM clerkships provided feedback on student documentation indicating the importance for students to learn this skill.
Introduction: Physicians dedicate substantial time to documentation. Scribes are sometimes used to improve efficiency by performing documentation tasks, although their impacts have not been prospectively evaluated. Our objective was to assess a scribe program's impact on emergency department (ED) throughput, physician time utilization, and job satisfaction in a large academic emergency medicine practice. Methods: We evaluated the intervention using pre-and post-intervention surveys and administrative data. All site physicians were included. Pre-and post-intervention data were collected in fourmonth periods one year apart. Primary outcomes included changes in monthly average ED length of stay (LOS), provider-specific average relative value units (RVUs) per hour (raw and normalized to volume), self-reported estimates of time spent teaching, self-reported estimates of time spent documenting, and job satisfaction. We analyzed data using descriptive statistics and appropriate tests for paired pre-post differences in continuous, categorical, and ranked variables. Results: Pre-and post-survey response rates were 76.1% and 69.0%, respectively. Most responded positively to the intervention, although 9.5% reported negative impressions. There was a 36% reduction (25%-50%; p<0.01) in time spent documenting and a 30% increase (11%-46%, p<0.01) in time spent in direct patient contact. No statistically significant changes were seen in job satisfaction or perception of time spent teaching. ED volume increased by 88 patients per day (32-146, p=0.04) pre-to post-and LOS was unchanged; rates of patients leaving against medical advice dropped, and rates of patients leaving without being seen increased. RVUs per hour increased 5.5% and per patient 5.3%; both were statistically significant. No statistically significant changes were seen in patients seen per hour. There was moderate correlation between changes in ED volume and changes in productivity metrics. Conclusion: Scribes were well received in our practice. Documentation time was substantially reduced and redirected primarily to patient care. Despite an ED volume increase, LOS was maintained, with fewer patients leaving against medical advice but more leaving without being seen. RVUs per hour and per patient both increased.