Implementation of "Scaling Large Language Models for Next-Generation Single-Cell Analysis", the latest Cell2Sentence (C2S-Scale) framework for applying Large Language Models (LLMs) to single-cell transcriptomics. The previous paper establishing the C2S methodology can be found here. C2S proposes a rank-ordering transformation of expression vectors into cell sentences—space-separated gene names ordered by descending expression—which allows LLMs to natively model scRNA-seq data using natural language.
The new C2S-Scale models scale to 27B parameters, unify transcriptomic and textual data, and enable advanced single-cell tasks including perturbation prediction, dataset summarization, and cluster captioning, and biological question answering. C2S-Scale models based on the Pythia architecture are already available on Huggingface, and models based on Gemma-2 will soon be made available.
For more information, check out the paper, explore the documentation, or reach out to us at the van Dijk Lab!
🚀 (10/15/2025) The C2S-Scale Gemma-2 2B and 27B pretrained models are now available on HuggingFace: C2S-Scale-Gemma-Models. Check out what's new in our updated preprint and blog post from Google!
📰 (04/17/2025) We published a blog post in collaboration with Google Research and Google DeepMind! Read it here to learn how Cell2Sentence is enabling next-gen single-cell discovery with LLMs.
🧠 (04/17/2025) We released the new C2S-Scale preprint, including larger models and tons of new experiments! Read the full paper on bioRxiv here.
🚀 (04/17/2025) The C2S-Scale-1B pretrained model is now available on HuggingFace: vandijklab/c2s-scale-1b.
🎉 (12/08/2024) Added built-in support for finetuning on your own custom prompt templates as well as multi-cell prompt formatting! See tutorials 7 & 8 for examples.
🎉 (09/03/2024) We release the new C2S code base, with core functionalities for working with C2S models in common single-cell workflows!
🎉 (09/03/2024) We release a new suite of Pythia models for cell type prediction, cell type conditioned generation, and a diverse multi-cell multi-task model! These models are trained on over 57 million human and mouse cells from CellxGene and Human Cell Atlas.
🎉 (05/01/2024) Cell2Sentence was accepted to ICML 2024! Congratulations to all of the authors involved in the project. See you in Vienna!
🎉 (02/15/2024) pythia-160m-c2s trained on full cell sentences is available on the Hugging Face hub here! This new model generates and predicts cell types from entire cells directly in text with the Pythia-160 base model.
🎉 (02/15/2024) Our updated preprint was posted on BioRxiv here. We introduce our latest results, including full cell sentence generation, combinatorial cell label prediction, abstract generation, and training on a large multi-tissue dataset of 36M cells.
To set up a cell2sentence environment, first pull the repository locally:
git clone https://github.com/vandijklab/cell2sentence.gitNavigate a terminal into the root of the repository. Next, create an Anaconda environment using python3 using anaconda with:
conda create -n cell2sentence python=3.8Next, activate the environment:
conda activate cell2sentenceFinally, run the setup:
make installThis will install the latest development environment of cell2sentence, along with other pacakge dependendies. You can also install cell2sentence itself using pip:
pip install cell2sentence==1.1.0The C2S package will allow usage of the core functionalities of C2S, including inference using existing C2S models and finetuning your own C2S models on your own datasets.
If you would like to speed up inference, you can optionally install flash-attention, which speeds up inference times particularly for long sequences (e.g. for generating cells with more than a few hundred genes):
pip install flash-attn --no-build-isolationDetailed instructions for installing flash-attention can be found in the official installation instructions. To enable flash attention 2, see the example in tutorial notebook 5 (cell generation).
The following notebooks provide guides on common workflows with C2S models. For more information about specific functions and modules, check out the official C2S documentation page.
| Notebook | Description |
|---|---|
| c2s_tutorial_0_data_preparation.ipynb | Data loading and preprocessing example on an immune tissue dataset |
| c2s_tutorial_1_rank_transformation_and_reconstruction.ipynb | C2S rank transformation to cell sentences and inverse transformation |
| c2s_tutorial_2_cell_embedding.ipynb | Obtaining cell type embeddings with C2S models |
| c2s_tutorial_3_finetuning_on_new_datasets.ipynb | Finetuning C2S models on new datasets |
| c2s_tutorial_4_cell_type_prediction.ipynb | Cell type prediction using C2S models |
| c2s_tutorial_5_cell_generation.ipynb | Cell generation conditioned on cell type |
| c2s_tutorial_6_cell_annotation_with_foundation_model.ipynb | Cell type annotation with foundation model |
The following is a collection of C2S pretrained models which are released for open use. Each is available through Huggingface, and has different capabilities and pretraining recipies. More details on each model are provided in their respective Huggingface cards, and tutorial notebooks for different C2S workflows each explain which model they use.
| Model name | Task(s) | Training Data |
|---|---|---|
| C2S-Pythia-410M cell type prediction model | Next gene prediction, cell type prediction | CellxGene, HCA, cell type annotations |
| C2S-Pythia-410M cell type conditioned single cell generation model | Next gene prediction, cell type conditioned single-cell generation | CellxGene, HCA, cell type annotations |
| C2S-Pythia-410M diverse single- and multi-cell tasks model | Next gene prediction, diverse single- and multi-cell prediction and generation tasks | CellxGene, HCA, cell type and tissue annotations, paper abstracts |
| C2S-Pythia-160M full cell generation (legacy) | Next gene prediction, cell type prediction | Immune tissue (Domínguez Conde et al.), cell type annotations |
| GPT-2 Large foundation model (legacy) | Next gene prediction, cell type prediction, conditional single-cell generation | CellxGene, HCA, cell type and tissue annotations, paper abstracts |
- Add core functionality for cell sentence conversion, data manipulation
- Add CSModel wrapper, finetuning functionality
- Add prompt formatting, support for multiple tasks
- Add functionality for inverse reconstruction back to expression vectors
- Add tutorial notebooks for main C2S workflows: cell type prediction, cell generation
- Add multi-cell prompt formatting
- Add support for legacy C2S-GPT-2 model prompts
- Add parameter-efficient finetuning methods (LoRA)
If you use Cell2Sentence in your work, please cite our paper:
@article{levine2023cell2sentence,
title={Cell2sentence: Teaching large language models the language of biology},
author={Levine, Daniel and L{\'e}vy, Sacha and Rizvi, Syed Asad and Pallikkavaliyaveetil, Nazreen and Chen, Xingyu and Zhang, David and Ghadermarzi, Sina and Wu, Ruiming and Zheng, Zihe and Vrkic, Ivan and others},
journal={bioRxiv},
pages={2023--09},
year={2023},
publisher={Cold Spring Harbor Laboratory}
}