GPT-3 Models are Poor Few-Shot Learners in the Biomedical Domain (2109.02555v2)

Abstract: Deep neural LLMs have set new breakthroughs in many tasks of NLP. Recent work has shown that deep transformer LLMs (pretrained on large amounts of texts) can achieve high levels of task-specific few-shot performance comparable to state-of-the-art models. However, the ability of these LLMs in few-shot transfer learning has not yet been explored in the biomedical domain. We investigated the performance of two powerful transformer LLMs, i.e. GPT-3 and BioBERT, in few-shot settings on various biomedical NLP tasks. The experimental results showed that, to a great extent, both the models underperform a LLM fine-tuned on the full training data. Although GPT-3 had already achieved near state-of-the-art results in few-shot knowledge transfer on open-domain NLP tasks, it could not perform as effectively as BioBERT, which is orders of magnitude smaller than GPT-3. Regarding that BioBERT was already pretrained on large biomedical text corpora, our study suggests that LLMs may largely benefit from in-domain pretraining in task-specific few-shot learning. However, in-domain pretraining seems not to be sufficient; novel pretraining and few-shot learning strategies are required in the biomedical NLP domain.