Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbations

Explore P-GAP, a novel zeroth-order optimization framework that accelerates LLM fine-tuning by aligning perturbations with projected gradients to minimize va...

Level: advanced

By Zhendong Mi, Qitao Tan, Grace Li Zhang, Zhaozhuo Xu, Geng Yuan, Shaoyi Huang

Category: research