PathologyBERT - Pre-trained Vs. A New Transformer Language Model for Pathology Domain.

• Type of Material: Article
• Authors: Thiago Santos, Emory University; Amara Tariq, Mayo Clinic, Phoenix, Arizona, USA.; Susmita Das, Indian Institute of Technology (IIT), Centre of Excellence in Artificial Intelligence, Kharagpur, West Bengal, India.; Kavyasree Vayalpati, Arizona State University, School of Computing and Augmented Intelligence, Tempe, Arizona, USA.; Geoffrey Smith, Emory University; Hari Trivedi, Emory University; Imon Banerjee, Emory University
• Language: English
• Date: 2022
• Publisher: AMIA Annual Symposium Proceesings Archive
• Publication Version: Final Published Version
• Copyright Statement: ©2022 AMIA - All rights reserved.
• Title of Journal or Parent Work: AMIA Annu Symp Proc
• Volume: 2022
• Start Page: 962
• End Page: 971
• Keywords: Natural Language Processing; Data Mining; Breast Neoplasms; Language; Female; Humans; Big Data
• Research Categories: Computer Science

Abstract

Pathology text mining is a challenging task given the reporting variability and constant new findings in cancer sub-type definitions. However, successful text mining of a large pathology database can play a critical role to advance 'big data' cancer research like similarity-based treatment selection, case identification, prognostication, surveillance, clinical trial screening, risk stratification, and many others. While there is a growing interest in developing language models for more specific clinical domains, no pathology-specific language space exist to support the rapid data-mining development in pathology space. In literature, a few approaches fine-tuned general transformer models on specialized corpora while maintaining the original tokenizer, but in fields requiring specialized terminology, these models often fail to perform adequately. We propose PathologyBERT - a pre-trained masked language model which was trained on 347,173 histopathology specimen reports and publicly released in the Huggingface1 repository2. Our comprehensive experiments demonstrate that pre-training of transformer model on pathology corpora yields performance improvements on Natural Language Understanding (NLU) and Breast Cancer Diagnose Classification when compared to nonspecific language models.