Fine-Tuned Transformers and Large Language Models for Entity Recognition in Complex Eligibility Criteria for Clinical Trials
[ 1 ] Wydział Informatyki i Telekomunikacji, Politechnika Poznańska | [ 2 ] Instytut Informatyki, Wydział Informatyki i Telekomunikacji, Politechnika Poznańska | [ SzD ] doktorant ze Szkoły Doktorskiej | [ P ] pracownik
2024
rozdział w monografii naukowej / referat
angielski
- prompt learning
- LLM
- clinical trial eligibility criteria
- named entity recognition
EN This paper evaluates the gpt-4-turbo model’s proficiency in recognizing named entities within the clinical trial eligibility criteria. We employ prompt learning to a dataset comprising 49 903 criteria from 3 314 trials, with 120 906 annotated entities in 15 classes. We compare the performance of gpt-4-turbo to state-of-the-art BERT-based Transformer models. Contrary to expectations, BERT-based models outperform gpt-4-turbo after moderate fine-tuning, in particular in low-resource settings. The CODER model consistently surpasses others in both lowand high-resource environments, likely due to term normalization and extensive pre-training on the UMLS thesaurus. However, it is important to recognize that traditional NER evaluation metrics, such as precision, recall, and the F1 score, can unfairly penalize generative language models, even if they correctly identify entities.
20
140