Developing Pretrained Language Models for Turkish Biomedical Domain
Conference paper > Full Text
Authors: Hazal TÜRKMEN, Oğuz DİKENELLİ, Cenk ERASLAN, Mehmet Cem Calli, Süha Süreyya ÖZBEK
2022 IEEE 10th International Conference on Healthcare Informatics (ICHI), , , International, Full Text, pp 597-598
DOI: 10.1109/ichi54592.2022.00117
Summary
Pretrained language models elevated with in-domain corpora show impressive results in biomedicine and clinical NLP tasks in English. However, there is minimal work in low-resource languages. This work introduces the BioBERTurk family, three pretrained models in Turkish for biomedicine. To evaluate models, we also introduce a labeled dataset to classify radiology reports of CT exams. Our first model was initialized from BERTurk and pretrained with biomedical corpus. The second model again continues to pretrain the general BERT model with a corpus of Ph.D. theses on radiology to test the effect of the task-related text. The final model combines radiology and biomedicine corpora with the corpus of BERTurk and pretrained a BERT model from scratch. F-scores of our models in the radiology resort classification are 92.99, 92.75, and 89.49 respectively. As far as we know, this is the first model that evaluates the effect of small size in-domain corpus in pretraining from scratch.