Abstract
Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors. However, tuning these models has become increasingly resource-intensive, or impossible when model weights are private. We introduce proxy-tuning, a lightweight decoding-time algorithm that operates on top of black-box LMs to achieve the same end as direct tuning, but by accessing only its predictions over the output vocabulary, not its parameters. Our method tunes a smaller LM, then applies the difference between the predictions of the small tuned and untuned LMs to shift the original predictions of the larger untuned model in the direction of tuning, while retaining the benefits of larger-scale pretraining. In experiments, when we apply proxy-tuning to Llama2-70B using proxies of only 7B size, we can close 88% of the gap between Llama2-70B and its truly-tuned chat version, when evaluated across knowledge, reasoning, and safety benchmarks. Interestingly, on TruthfulQA, proxy-tuned models are actually more truthful than directly tuned models, possibly because decoding-time guidance better retains the model's factual knowledge. We then demonstrate the generality of proxy-tuning by applying it to domain adaptation on code, and task-specific finetuning on question-answering and math problems. Finally, we show how to proxy-tune a truly black-box LM, GPT-3.5, for temporal adaptation, increasing its knowledge about recent events. Our work demonstrates the promise of using small tuned LMs to efficiently customize large, potentially proprietary LMs through decoding-time guidance.
Overview
-
Proxy-tuning is a new method for adapting LLMs to specific tasks without altering the model parameters directly.
-
It uses a decoding-time algorithm with smaller 'proxy' models, known as experts and anti-experts, to adjust the prediction logits of the base LLM.
-
Experiments demonstrate that proxy-tuning can nearly match the performance of directly fine-tuned models in different tasks such as instruction-following and coding.
-
Proxy-tuning offers substantial performance improvements over base models, with significant gains in areas like question-answering and math problems.
-
The method allows for efficient customization of proprietary LLMs and may preserve knowledge better than traditional fine-tuning.
Overview of Proxy-Tuning
Proxy-tuning represents a novel methodology for adapting LLMs to specific tasks or domains without direct tweaking of the model parameters. Traditional fine-tuning has become resource-intensive, and in some cases, impractical when dealing with proprietary models with inaccessible weights. Proxy-tuning introduces a solution that adapts the prediction behavior of LLMs through a lightweight decoding-time algorithm that leverages smaller, more manageable models known as "proxies".
Methodology
The technique involves utilizing smaller trained models (experts) and their untuned counterparts (anti-experts) as references to adjust the prediction logits of a larger, base LLM. During the decoding phase, for every output token, the logits of the base model are shifted in the direction suggested by the difference between the expert and anti-expert. Experiments applying this method have shown that proxy-tuning can substantially close the performance gap between a base LLM and a version that has been directly tuned, indicating that larger models gain the benefits of fine-tuning while preserving the vast knowledge acquired during pre-training.
Experimental Findings
Proxy-tuning has shown remarkable results across various areas. In instruction-following benchmarks, it closed up to 91% of the performance gap between base and directly-tuned models. When used for domain adaptation in coding tasks, it yielded up to a 32% absolute improvement over base models. For task-specific tuning in question-answering and math problems, there was an average absolute improvement of 31%. These outcomes validate that proxy-tuning is not only effective but it can also implement fine-tuning for tasks with stringent constraints.
Implications
This method is particularly advantageous for adapting large proprietary LLMs for user-specific needs when only output probabilities are available. It demonstrates the promise for efficient and effective customization without the need for direct model parameter modification. Also, proxy-tuning might preserve learned knowledge better than direct fine-tuning, which can be invasive and risk forgetting previously acquired information. Its resource efficiency and adaptability make it a compelling alternative to conventional tuning methods and open new avenues for leveraging the capabilities of LLMs for various applications.
Create an account to read this summary for free: