DoRA: Weight-Decomposed Low-Rank Adaptation

Abstract: Among the widely used parameter-efficient fine-tuning (PEFT) methods, LoRA and its variants have gained considerable popularity because of avoiding additional inference costs. However, there still often exists an accuracy gap between these methods and full fine-tuning (FT). In this work, we first introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed Low-Rank Adaptation (DoRA). DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. By employing \ours, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead. \ours~consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding. Code is available at https://github.com/NVlabs/DoRA.

• The paper introduces DoRA, a parameter-efficient method that decomposes pre-trained weight matrices into magnitude and directional components to better mimic full fine-tuning behavior.
• DoRA directly fine-tunes magnitude vectors and applies low-rank updates to the directional component, improving accuracy by up to 3.7% over standard LoRA on models like LLaMA-7B.
• The approach supports merging adapted weights for zero inference overhead and extends to quantized models, enabling efficient fine-tuning on resource-constrained hardware.

This paper introduces DoRA (Weight-Decomposed Low-Rank Adaptation), a parameter-efficient fine-tuning (PEFT) method designed to bridge the performance gap between standard LoRA and full fine-tuning (FT) while retaining LoRA's efficiency benefits.

The core idea of DoRA stems from an analysis of how FT and LoRA update model weights. The authors propose decomposing a pre-trained weight matrix $W_0$ into a magnitude component $m$ and a directional component $V$, such that $W_0 = m \frac{V}{||V||_c}$, where $||V||_c$ is the vector-wise norm across columns. By analyzing the changes in magnitude ($\Delta M$) and direction ($\Delta D$) during fine-tuning, they observe that FT exhibits a distinct learning pattern compared to LoRA, characterized by a different correlation between magnitude and directional updates (negative for FT, positive for LoRA in their analysis). This difference is hypothesized to contribute to LoRA's lower learning capacity compared to FT.

DoRA leverages this insight by explicitly decomposing the pre-trained weight $W_0$ into its magnitude $m = ||W_0||_c$ and direction $V = W_0$ at initialization. Instead of directly adding a low-rank update $m$0 to $m$1 as in standard LoRA, DoRA fine-tunes the magnitude component $m$2 directly and applies the low-rank update $m$3 to the directional component $m$4. The fine-tuned weight $m$5 is then constructed as:

$m$6

where $m$7 and the low-rank matrices $m$8 and $m$9 are trainable parameters, and $V$0 (initialized as $V$1) is kept frozen. This allows DoRA to separately manage magnitude and directional updates, aiming to replicate FT's learning behavior more closely.

Practical Implementation:

1. Weight Decomposition: For each weight matrix $V$2 in the pre-trained model where PEFT is applied (typically attention and sometimes feed-forward layers in Transformers), calculate the initial magnitude vector $V$3 where each element $V$4 (L2 norm of the i-th column). The initial directional matrix $V$5 is simply $V$6. $V$7 is initialized as a trainable parameter, while $V$8 remains fixed.
2. LoRA Application: Standard LoRA matrices $V$9 and $W_0 = m \frac{V}{||V||_c}$0 are introduced. They are initialized as in standard LoRA ($W_0 = m \frac{V}{||V||_c}$1 with Kaiming uniform, $W_0 = m \frac{V}{||V||_c}$2 with zeros) so that $W_0 = m \frac{V}{||V||_c}$3 initially, making $W_0 = m \frac{V}{||V||_c}$4 equal to $W_0 = m \frac{V}{||V||_c}$5. These matrices are trainable.
3. Forward Pass: To compute the output of a layer with weight $W_0 = m \frac{V}{||V||_c}$6 and input $W_0 = m \frac{V}{||V||_c}$7:
   • Compute the updated directional component: $W_0 = m \frac{V}{||V||_c}$8.
   • Compute the column-wise norms of $W_0 = m \frac{V}{||V||_c}$9: $||V||_c$0.
   • Compute the fine-tuned weight: $||V||_c$1 (where $||V||_c$2 denotes element-wise multiplication after broadcasting $||V||_c$3).
   • Compute the output: $||V||_c$4.
   • Note that the magnitude $||V||_c$5 is a $||V||_c$6 vector, and $||V||_c$7 is $||V||_c$8. The magnitude is applied element-wise to the columns of the normalized directional matrix.
4. Backward Pass (Gradient Calculation): Standard gradient calculations apply to $||V||_c$9, $\Delta M$0, and $\Delta M$1. To reduce memory overhead during backpropagation, the authors propose treating $\Delta M$2 as a constant (detaching it from the gradient graph). This means $\Delta M$3 is computed based on the current values of $\Delta M$4 and $\Delta M$5, but its gradient is not backpropagated through. The gradient w.r.t $\Delta M$6 is $\Delta M$7, where $\Delta M$8 is treated as a constant. This modification reduces memory usage with minimal impact on accuracy, as shown in ablation studies (e.g., 24.4% memory reduction for LLaMA tuning with 0.2 accuracy drop).
5. Inference: Like LoRA, DoRA allows merging the trained parameters ($\Delta M$9) into the original weight matrix $\Delta D$0 before deployment. The merged weight $\Delta D$1 can be pre-calculated, resulting in a standard $\Delta D$2 matrix. This means DoRA introduces no additional inference latency compared to the original pre-trained model or merged LoRA.

Real-world Applications and Performance:

The paper demonstrates DoRA's effectiveness across various tasks and model architectures:

• Commonsense Reasoning (LLaMA, LLaMA2, LLaMA3): DoRA consistently outperforms LoRA across multiple benchmarks. On LLaMA-7B, DoRA improves average accuracy by 3.7% over LoRA. Even with half the parameters ($\Delta D$3), DoRA surpasses LoRA. This indicates improved learning capacity with comparable or fewer parameters.
• Image/Video-Text Understanding (VL-BART): DoRA shows better performance than LoRA (nearly 1% average improvement on image-text, 2% on video-text), reaching performance levels comparable to full fine-tuning but with significantly fewer trainable parameters.
• Visual Instruction Tuning (LLaVA-1.5-7B): DoRA achieves higher average accuracy than both LoRA and FT.
• Instruction Tuning (LLaMA, LLaMA2): DoRA is compatible with other LoRA variants like VeRA, creating DoVA. DoVA significantly improves over VeRA (which uses very few parameters) and achieves accuracy comparable to or better than standard LoRA with substantially fewer parameters.
• Robustness: DoRA consistently outperforms LoRA across different numbers of training samples and varying LoRA ranks, demonstrating its stability and improved performance, particularly at lower ranks where LoRA's performance degrades significantly.
• Tuning Granularity: By selectively applying directional updates (LoRA part) only to certain modules (e.g., QKV layers in attention) while updating magnitude for more modules, DoRA can achieve better accuracy than LoRA with a smaller total number of trainable parameters.

Broader Impact (QDoRA):

DoRA's concept has been extended to quantized models. QDoRA combines 4-bit quantization (from QLoRA) with DoRA. Early results on fine-tuning LLaMA2-7B on Orca-Math (Mitra et al., 2024) show that QDoRA not only outperforms QLoRA significantly but can even surpass the accuracy of the non-quantized full fine-tuning baseline, all while requiring substantially less GPU memory. This suggests QDoRA is a promising approach for efficiently fine-tuning large models on consumer hardware.

Implementation Considerations:

• Implementing DoRA requires modifying the forward and backward passes for the layers where PEFT is applied to incorporate the magnitude and directional decomposition and updates.
• The gradient modification (treating $\Delta D$4 as constant) is crucial for practical memory efficiency during training.
• The trainable parameters are the magnitude vector $\Delta D$5 (size $\Delta D$6) and the LoRA matrices $\Delta D$7 ($\Delta D$8) and $\Delta D$9 ($W_0$0). The total trainable parameters are $W_0$1. Compared to LoRA ($W_0$2), DoRA adds $W_0$3 parameters for the magnitude vector per modified weight matrix.
• For deployment, the trained $W_0$4 are merged into the original $W_0$5 to form the final weight matrix $W_0$6, which can then be used like any standard weight matrix without special DoRA computation, ensuring zero inference overhead.

In summary, DoRA is a practical and effective PEFT method that improves upon LoRA by explicitly handling magnitude and directional updates based on insights from FT. It achieves better performance with similar or fewer trainable parameters than LoRA, is compatible with LoRA variants, is robust to varying ranks and data sizes, and can be merged for inference, incurring no additional latency. Its extension to quantized fine-tuning (QDoRA) further highlights its practical significance for training large models on resource-constrained hardware.