LoRA: Low-Rank Adaptation of Large Language Models
LoRA This repo contains the implementation of LoRA in GPT-2 and steps to replicate the results in our recent paper LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu*, Yelong Shen*, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Weizhu ChenPaper: https://arxiv.org/abs/2106.09685 LoRA reduces the number of trainable parameters by learning pairs of rank-decompostion matrices and freezing the original weights. This vastly reduces the storage requirement for large language models adapted to specific tasks and enables efficient task-switching during deployment […]
Read more