About Me
Hello! I am Kaiyue Wen. I am a second-year Phd student at Stanford University, where I am grateful to be advised by Tengyu Ma and Percy Liang. I graduated from Tsinghua University, where I was a member of Yao’s pilot class. Here are my CV and Publications. During my undergraduate study, I am fortunate to be advised by Tengyu Ma, Zhiyuan Liu, Andrej Risteski, Jingzhao Zhang, Yuhao Wang and Zhiyuan Li.
My research interest spreads broadly in deep learning. My long-term goal is to understand the physics behind deep learning and I believe a combination of theoretical analysis and empirical study is essential for this goal.
Recently, I’ve become fascinated by two fundamental axes of scaling in deep learning.
Demystifying pretraining: Pretraining has been the driving force behind the evolution of large language models, yet many foundational algorithmic choices remain poorly understood. Key aspects such as optimizers, architectures, and hyperparameter scaling strategies still lack consensus. My goal is to clarify these choices through rigorous benchmarking (e.g., benchmarking modern optimizers) and theoretical analysis (e.g., exploring the representation limitation of RNNs, architectures beyond $\mathrm{TC}^0$, and river-valley loss landscape). Most of my research in this direction is carried out in the open-source project Marin.
New algorithmic paradigms in reasoning: With the recent progress in reasoning reinforcement learning (RL), particularly innovations like long-chain-of-thought RL, there is growing potential to push the limits of model reasoning. While I am new to this field, my aim is to design end-to-end trainable multi-agent RL systems that build upon and extend the capabilities of current long-CoT RL paradigms.
Recent News
- Sep, 2025 New preprint (Fantastic Pretraining Optimizers and Where to Find Them ) on arxiv!
- May, 2025 WSD-S is used in training the best open-source 8B model Marin 8B.
- Jan, 2025 New preprint (Global Load Balancing Helps Expert Specialization) on arxiv!
- Jan, 2025 3 papers (River Valley Landscape, RNNs are not Transformers (Yet), Optimization Analysis on Chain-of-Thought) accepted at ICLR 2025!
- Dec, 2024 Residual Permutation Test is accepted at AoS!
- Oct, 2024 New preprints River Valley Landscape and Optimization Analysis on Chain-of-Thought on arxiv!
- Sep, 2024 Start my Ph.D. study at Stanford University! I am currently rotating with Percy Liang.
- Jul, 2024 Graduated from Tsinghua University with a Bachelor’s degree in Computer Science.
- May, 2024 Receive and accept the offer from Stanford University! I am honored to receive the Stanford Graduate Fellowship.
- Feb, 2024 New preprint RNNs are not Transformers (Yet) on arxiv!
- Oct, 2023 Awarded the National Scholarship (top 0.2%)!
- Sep, 2023 2 papers (Sharpness&Generalization, (Un)interpretability of Transformers) accepted at Neurips 2023! Sharpness&Generalization is received as oral.
- Sep, 2023 Receive the silver medal for Yao Award (Top 4 in Yao’s pilot class)!
- Aug, 2023 Return to China for my senior year in Tsinghua.
- Jul, 2023 Visit Hawaii for ICML 2023! Always great to see old friends.
- Jun, 2023 Residual Permutation Test receive Major Revision from AoS.
- Jun, 2023 Visiting Tengyu Ma at Stanford!
- May, 2023 Visit Rwanda for ICLR 2023!
- Mar, 2023 Reviewing ICML for the first time!
- Mar, 2023 New preprint Solving LPN with Neural Networks on arxiv!
- Feb, 2023 Visiting Andrej Risteski at CMU!
- Jan, 2023 2 papers (Understanding SAM, Not Benign Overfitting) accepted at ICLR 2023!
- Dec, 2022 New preprint Residual Permutation Test on arxiv!
- Dec, 2022 New preprint Understanding SAM on arxiv!
- Oct, 2022 1 paper (Skill Neurons) accepted at EMNLP 2022.
- Jun, 2022 New preprint Not Benign Overfitting on arxiv!
One More Thing
I keep a firm faith in analytical thinking, hard work, and consistent self-improvement. Any advice or feedback is welcome. You can use this Anonymous Form or discuss with me in person.