Murthy Devarakonda, PhD’s Post

🚀 Excited to share our latest preprint: "Single Cell Foundation Models Evaluation (scFME) for In-Silico Perturbation" Now live on bioRxiv: https://lnkd.in/eXjZ8q3u In this study, we introduce scFME, a robust benchmarking framework designed to evaluate single-cell foundation models for in-silico perturbation (ISP), a critical capability for target discovery and disease modeling. 🔬 Background: Many single cell foundation models have been built and many more are coming, which makes it an active area of research with broad interest. While early models like Geneformer and scGPT have been in use and evaluated there are new virtual cell models like #TranscriptFormer and flow matching models like #CellFlow. It is important to understand how they perform using a standardized, model-independent benchmark for a critical downstream application. Often models are evaluated on cell or phenotype classification, but what is also important is how they perform on gene perturbation tasks, otherwise we miss the mark on drug discovery and disease understanding relevance. scFME fills this gap by aligning model evaluation with realistic perturbation scenarios, enabling deeper insights into model performance for gene perturbation across gene categories and biological functions. 📊 Key highlights: The study shows that fine-tuning on disease-relevant data improves ISP accuracy significantly. Scaling laws apply, larger models pretrained on larger datasets perform better. The scFME method introduces new metrics based on familiar cosine shift, silhouette score, and ranking to quantify ISP success and specificity. We also identify the need for larger perturbation datasets for improved analysis. We hope that this work lays the foundation for more rigorous, task-specific evaluation of single-cell foundation models which will help accurate in-silico modeling of molecular biology. Congratulations to the fantastic team at Novartis: James Boylan, Elizaveta Solovyeva, Theophile Bouiller, Xiong Liu, Sebastian Hoersch, Bulent Ataman, and Jeremy Jenkins #SingleCellRNA #InSilicoPerturbation #FoundationModels #scFME #Geneformer #scGPT #TargetIdentification #AIinBiotech #BiomedicalAI #Transcriptomics #PerturbSeq #MachineLearning #DeepLearning #PrecisionMedicine #DrugDiscovery