Does QA-based intermediate training help fine-tuning language models for text classification? (2112.15051v1)

Abstract: Fine-tuning pre-trained LLMs for downstream tasks has become a norm for NLP. Recently it is found that intermediate training based on high-level inference tasks such as Question Answering (QA) can improve the performance of some LLMs for target tasks. However it is not clear if intermediate training generally benefits various LLMs. In this paper, using the SQuAD-2.0 QA task for intermediate training for target text classification tasks, we experimented on eight tasks for single-sequence classification and eight tasks for sequence-pair classification using two base and two compact LLMs. Our experiments show that QA-based intermediate training generates varying transfer performance across different LLMs, except for similar QA tasks.