MTL-LoRA enhances the multi-task learning capabilities of LoRA by introducing task-adaptive parameters that distinguish task-specific information while efficiently capturing shared knowledge across tasks in low-dimensional spaces. MTL-LoRA outperforms LoRA and its variants with comparable or fewer learnable parameters in multi-task learning scenarios.

The standalone implementation of MTL-LoRA can be found here. Currently, we only support for tuning linear layer.

• Setup

  conda create -n mtl-lora python==3.10 -y
  conda activate mtl-lora
  pip install torch
  pip install -r requirements.txt
  

• Datasets

  • Please follow the instruction here to prepare the dataset.
  • For training MTL-LoRA or MoE-LoRA, please use the dataset with task IDs available here

• Fine-tuning

  • The script for fine-tuning different adapters can be found at ./script.

  • For fine-tuning with MTL-LoRA:

    • Specify the DATA_PATH, OUTPUT_PATH, and CACHE_PATH (For caching the tokenized data).

    bash ./script/llama2_7B_mlora_qkvo.sh 8 16
    

• Evaluation

  • The script for evaluation can be found at ./script
  • For evaluation MTL-LoRA:

    bash ./script/llama2_7B_mlora_qkvo_eval.sh $CHECKPOINT_PATH $OUTPUT_PATH
    

Please follow the instruction here.

We gratitude to the following repositories for their wonderful works:

• LoRA
• DoRA

Please contact us or post an issue if you have any questions: [email protected].