Enhanced adverse drug event extraction using prefix-based multi-prompt tuning in transformer models
Extracting mentions of adverse drug events and relationships between them is crucial for effective pharmacovigilance and drug safety surveillance. Recently, transformer-based models have significantly improved this task through fine-tuning. However, traditional fine-tuning of transformer models, esp...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Politeknik Negeri Padang
2024
|
| Online Access: | http://psasir.upm.edu.my/id/eprint/117021/ http://psasir.upm.edu.my/id/eprint/117021/1/117021.pdf |
| _version_ | 1848867144596455424 |
|---|---|
| author | Modi, Salisu Kasmiran, Khairul Azhar Mohd Sharef, Nurfadhlina Sharum, Mohd Yunus |
| author_facet | Modi, Salisu Kasmiran, Khairul Azhar Mohd Sharef, Nurfadhlina Sharum, Mohd Yunus |
| author_sort | Modi, Salisu |
| building | UPM Institutional Repository |
| collection | Online Access |
| description | Extracting mentions of adverse drug events and relationships between them is crucial for effective pharmacovigilance and drug safety surveillance. Recently, transformer-based models have significantly improved this task through fine-tuning. However, traditional fine-tuning of transformer models, especially those with many parameters, is resource-intensive, memory-inefficient, and often leaves a gap between pre-training and downstream task-specific objectives. Soft prompting is a lightweight approach that updates a trainable prompt to guide task-specific fine-tuning, showing comparable performance to traditional fine-tuning for large language models on simple tasks. However, its effectiveness on complex tasks like token-based sequence labeling requiring multiple predictions for a single input sequence remains underexplored, particularly in multi-task settings. In addition, using holistic prompts in multi-task learning settings may be biased to other subtasks. Additionally, some prompt tokens hurt the model prediction. This study proposes a prefix-based multi-prompt soft tuning method with attention-driven prompt token selection for tuning transformer models on multi-task dual sequence labelling for concept and relation extraction. We experimented with BERT and SciBERT models using frozen and unfrozen parameter strategies. Our approach achieved state-of-the-art performance on the n2c2 2018 and TAC 2017 datasets for adverse drug event extraction, with multi-prompt tuning in unfrozen models surpassing traditional fine-tuning. Moreover, it outperforms the largest clinical natural language processing model, GatorTron, on the n2c2 2018 dataset. This research highlights the potential of soft prompts in efficiently adapting large language models to complex downstream NLP tasks. |
| first_indexed | 2025-11-15T14:31:50Z |
| format | Article |
| id | upm-117021 |
| institution | Universiti Putra Malaysia |
| institution_category | Local University |
| language | English |
| last_indexed | 2025-11-15T14:31:50Z |
| publishDate | 2024 |
| publisher | Politeknik Negeri Padang |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | upm-1170212025-07-07T08:07:42Z http://psasir.upm.edu.my/id/eprint/117021/ Enhanced adverse drug event extraction using prefix-based multi-prompt tuning in transformer models Modi, Salisu Kasmiran, Khairul Azhar Mohd Sharef, Nurfadhlina Sharum, Mohd Yunus Extracting mentions of adverse drug events and relationships between them is crucial for effective pharmacovigilance and drug safety surveillance. Recently, transformer-based models have significantly improved this task through fine-tuning. However, traditional fine-tuning of transformer models, especially those with many parameters, is resource-intensive, memory-inefficient, and often leaves a gap between pre-training and downstream task-specific objectives. Soft prompting is a lightweight approach that updates a trainable prompt to guide task-specific fine-tuning, showing comparable performance to traditional fine-tuning for large language models on simple tasks. However, its effectiveness on complex tasks like token-based sequence labeling requiring multiple predictions for a single input sequence remains underexplored, particularly in multi-task settings. In addition, using holistic prompts in multi-task learning settings may be biased to other subtasks. Additionally, some prompt tokens hurt the model prediction. This study proposes a prefix-based multi-prompt soft tuning method with attention-driven prompt token selection for tuning transformer models on multi-task dual sequence labelling for concept and relation extraction. We experimented with BERT and SciBERT models using frozen and unfrozen parameter strategies. Our approach achieved state-of-the-art performance on the n2c2 2018 and TAC 2017 datasets for adverse drug event extraction, with multi-prompt tuning in unfrozen models surpassing traditional fine-tuning. Moreover, it outperforms the largest clinical natural language processing model, GatorTron, on the n2c2 2018 dataset. This research highlights the potential of soft prompts in efficiently adapting large language models to complex downstream NLP tasks. Politeknik Negeri Padang 2024 Article PeerReviewed text en http://psasir.upm.edu.my/id/eprint/117021/1/117021.pdf Modi, Salisu and Kasmiran, Khairul Azhar and Mohd Sharef, Nurfadhlina and Sharum, Mohd Yunus (2024) Enhanced adverse drug event extraction using prefix-based multi-prompt tuning in transformer models. International Journal on Informatics Visualization, 8 (3-2). pp. 1713-1719. ISSN 2549-9904; eISSN: 2549-9904 https://joiv.org/index.php/joiv/article/view/3454 10.62527/joiv.8.3-2.3454 |
| spellingShingle | Modi, Salisu Kasmiran, Khairul Azhar Mohd Sharef, Nurfadhlina Sharum, Mohd Yunus Enhanced adverse drug event extraction using prefix-based multi-prompt tuning in transformer models |
| title | Enhanced adverse drug event extraction using prefix-based multi-prompt tuning in transformer models |
| title_full | Enhanced adverse drug event extraction using prefix-based multi-prompt tuning in transformer models |
| title_fullStr | Enhanced adverse drug event extraction using prefix-based multi-prompt tuning in transformer models |
| title_full_unstemmed | Enhanced adverse drug event extraction using prefix-based multi-prompt tuning in transformer models |
| title_short | Enhanced adverse drug event extraction using prefix-based multi-prompt tuning in transformer models |
| title_sort | enhanced adverse drug event extraction using prefix-based multi-prompt tuning in transformer models |
| url | http://psasir.upm.edu.my/id/eprint/117021/ http://psasir.upm.edu.my/id/eprint/117021/ http://psasir.upm.edu.my/id/eprint/117021/ http://psasir.upm.edu.my/id/eprint/117021/1/117021.pdf |