开源所有数据集来源、数据清洗、tokenizer训练、模型预训练、SFT指令微调、RLHF优化等流程的全部代码

accelerate==0.25.0 colorlog==6.8.0 datasets==2.15.0 datasketch==1.6.4 fastapi==0.105.0 fastparquet==2023.10.1 fire==0.5.0 jieba==0.42.1 matplotlib==3.8.2 nltk==3.8.1 numpy==1.26.2 opencc_python_reimplemented==0.1.7 pandas==2.1.4 peft==0.6.2 psutil==5.9.6 pyarrow==14.0.1 pydantic==2.5.2 rich==13.7.0 safetensors==0.4.1 sentencepiece==0.1.99 tokenizers==0.15.0 torch==2.1.1 torch_optimizer==0.3.0 tqdm==4.66.1 transformers==4.36.0 trl==0.7.4 ujson==5.8.0 uvicorn==0.24.0.post1