Text classification models for the automatic detection of nonmedical prescription medication use from social media

• Type of Material: Article
• Authors: Mohammed Ali Al-Garadi, Emory University; Yuan-Chi Yang, Emory University; Haitao Cai, University of Pennsylvania; Yucheng Ruan, University of Pennsylvania; Karen O'Connor, University of Pennsylvania; Gonzalez-Hernandez Graciela, University of Pennsylvania; Jeanmarie Perrone, University of Pennsylvania; Md. Abeed Sarker, Emory University
• Language: English
• Date: 2021-01-26
• Publisher: BMC
• Publication Version: Final Published Version
• Copyright Statement: © The Author(s) 2021
• License: Creative Commons Attribution 4.0 International
• Final Published Version (URL): http://dx.doi.org/10.1186/s12911-021-01394-0
• Title of Journal or Parent Work: BMC MEDICAL INFORMATICS AND DECISION MAKING
• Volume: 21
• Issue: 1
• Start Page: 27
• End Page: 27
• Grant/Funding Information: Research reported in this publication is supported by the NIDA of the NIH under award number R01DA046619. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
• Keywords: Medical Informatics; Social media; Machine learning; Life Sciences & Biomedicine; Prescription medication misuse; Science & Technology; Natural language processing
• Research Categories: Health Sciences, Public Health; Health Sciences, Pharmacy

Abstract

Background: Prescription medication (PM) misuse/abuse has emerged as a national crisis in the United States, and social media has been suggested as a potential resource for performing active monitoring. However, automating a social media-based monitoring system is challenging—requiring advanced natural language processing (NLP) and machine learning methods. In this paper, we describe the development and evaluation of automatic text classification models for detecting self-reports of PM abuse from Twitter. Methods: We experimented with state-of-the-art bi-directional transformer-based language models, which utilize tweet-level representations that enable transfer learning (e.g., BERT, RoBERTa, XLNet, AlBERT, and DistilBERT), proposed fusion-based approaches, and compared the developed models with several traditional machine learning, including deep learning, approaches. Using a public dataset, we evaluated the performances of the classifiers on their abilities to classify the non-majority “abuse/misuse” class. Results: Our proposed fusion-based model performs significantly better than the best traditional model (F1-score [95% CI]: 0.67 [0.64–0.69] vs. 0.45 [0.42–0.48]). We illustrate, via experimentation using varying training set sizes, that the transformer-based models are more stable and require less annotated data compared to the other models. The significant improvements achieved by our best-performing classification model over past approaches makes it suitable for automated continuous monitoring of nonmedical PM use from Twitter. Conclusions: BERT, BERT-like and fusion-based models outperform traditional machine learning and deep learning models, achieving substantial improvements over many years of past research on the topic of prescription medication misuse/abuse classification from social media, which had been shown to be a complex task due to the unique ways in which information about nonmedical use is presented. Several challenges associated with the lack of context and the nature of social media language need to be overcome to further improve BERT and BERT-like models. These experimental driven challenges are represented as potential future research directions.