Researchers report Deletion Tuning matches full fine-tuning by deleting under 0.05% of weights

Top Paper

In the paper's reinforcement learning tests, full fine-tuning, LoRA, and random sparse fine-tuning lost accuracy on prior tasks. Deletion Tuning learns a binary mask over transformer-block gate projections, zeroing fewer than 0.05% of weights without making any other modification. The authors report performance matching full fine-tuning at scale.

Here's what changed:

• Matching results span reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation.
• Prior-task accuracy stays near the base level under reinforcement learning, while full fine-tuning, LoRA, and random sparse fine-tuning lose up to 5.4 points.
• The trained mask is hundreds to thousands of times smaller than the corresponding full-fine-tuning checkpoint.
• The authors' analysis of weight changes places Deletion Tuning closest to full fine-tuning among the compared methods, using stable rank, spectrum drift, and principal-weight overlap.

Sources: arXiv paper