A Data-Driven Iterative Approach for Semi-automatically Assessing the Correctness of Medication Value Sets: A Proof of Concept Based on Opioids

• Type of Material: Article
• Authors: Linyi Li, Emory University; Adela Grando, Arizona State University; Md Sarker, Emory University
• Language: English
• Date: 2021-12-01
• Publisher: GEORG THIEME VERLAG KG
• Publication Version: Final Published Version
• Copyright Statement: The Author(s).
• License: Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International
• Final Published Version (URL): http://dx.doi.org/10.1055/s-0041-1740358
• Title of Journal or Parent Work: METHODS OF INFORMATION IN MEDICINE
• Volume: 60
• Issue: S 02
• Start Page: E111
• End Page: E119
• Grant/Funding Information: A.G.'s efforts were funded by the National Institute of Mental Health through the “ My Data Choices, evaluation of effective consent strategies for patients with behavioral health conditions ” (R01 MH108992) grant. A.S.'s efforts were funded by the National Institute on Drug Abuse through the “ Mining Social Media Big Data for Toxicovigilance: Automating the Monitoring of Prescription Medication Abuse via Natural Language Processing and Machine Learning Methods ” (R01 DA046619) grant.
• Keywords: Medical Informatics; opioids; Science & Technology; Health Care Sciences & Services; Computer Science, Information Systems; medication value sets; Technology; Life Sciences & Biomedicine; prescription drugs; natural language processing; Computer Science
• Research Categories: Computer Science

Abstract

Background Value sets are lists of terms (e.g., opioid medication names) and their corresponding codes from standard clinical vocabularies (e.g., RxNorm) created with the intent of supporting health information exchange and research. Value sets are manually-created and often exhibit errors. Objectives The aim of the study is to develop a semi-automatic, data-centric natural language processing (NLP) method to assess medication-related value set correctness and evaluate it on a set of opioid medication value sets. Methods We developed an NLP algorithm that utilizes value sets containing mostly true positives and true negatives to learn lexical patterns associated with the true positives, and then employs these patterns to identify potential errors in unseen value sets. We evaluated the algorithm on a set of opioid medication value sets, using the recall, precision and F 1-score metrics. We applied the trained model to assess the correctness of unseen opioid value sets based on recall. To replicate the application of the algorithm in real-world settings, a domain expert manually conducted error analysis to identify potential system and value set errors. Results Thirty-eight value sets were retrieved from the Value Set Authority Center, and six (two opioid, four non-opioid) were used to develop and evaluate the system. Average precision, recall, and F 1-score were 0.932, 0.904, and 0.909, respectively on uncorrected value sets; and 0.958, 0.953, and 0.953, respectively after manual correction of the same value sets. On 20 unseen opioid value sets, the algorithm obtained average recall of 0.89. Error analyses revealed that the main sources of system misclassifications were differences in how opioids were coded in the value sets-while the training value sets had generic names mostly, some of the unseen value sets had new trade names and ingredients. Conclusion The proposed approach is data-centric, reusable, customizable, and not resource intensive. It may help domain experts to easily validate value sets.