Siddharth Nayyar

Some recent, exciting news on compressed tokenizers and how individual tokens modify images in encoder-decoder models. I hope this opens up a new avenue of research with smaller models across frameworks (perhaps JEPA?). https://lnkd.in/evFw8WBu

70

2 Comments

Wanted to understand how GPU works, but found yourself lost in jargons like warp and SM? My latest article explains the optimisation goal of GPUs, and how software constructs (Grids, Blocks, Threads) and hardware components (Streaming Multiprocessors, Warps, CUDA Cores) are related to each other. It also explains the difference between CUDA and tensor cores, and how constraining a block to a SM helps to maximise memory throughput. I also explain the difference between intra-node GPU interconnects, CPU interconnects, and inter-nodes GPU interconnects to enable high-speed communication both within a single node and across entire GPU clusters. Credits to Stanford CS 336 for providing a base for me to understand these concepts, and Google Gemini 2.5 Pro for speeding up my research. 

2,324

58 Comments

Are reward models (RMs) still useful in the age of reasoning / verifiable rewards? Here’s a break down… Early research: Initial post-ChatGPT LLMs were almost always post-trained using the three-step alignment procedure proposed by InstructGPT; see image. This procedure has three steps: 1. Supervised finetuning (SFT) trains the model using next-token prediction over examples of good completions. 2. An RM is trained over a human preference dataset. 3. Reinforcement learning (RL) is used to finetune the LLM by using the output of the RM as a training signal. Collectively, steps two and three in this procedure are called reinforcement learning from human feedback (RLHF). Modern LLMs & reasoning models. Post-training is slightly more complicated today; an example of a modern post-training pipeline (from Tulu-3) is provided in the image. Key changes include: - The SFT phase—although still very common— is not always used, especially for recent reasoning models; e.g., some variants of DeepSeek-R1 forgo SFT and apply RL directly to the pretrained model. - RL training is usually performed in several rounds, where fresh data is collected for each round to further improve the LLM’s capabilities. - Several variants of RL (and non-RL-based alternatives like DPO) are used—potentially in tandem—that may or may not require an RM. Different types of RL. There are two broad categories of reinforcement learning (RL) training that are heavily leveraged by LLMs: RLHF and reinforcement learning with verifiable rewards (RLVR); see image. The primary difference between these techniques lies in how we define the reward: - In RLHF, the reward comes from the RM. - RLVR uses deterministic (or verifiable) rewards. Notably, the deterministic—usually rules-based—rewards in RLVR eliminate the need for an RM! Usually, rewards are derived by extracting the LLM’s final answer from its generated output and comparing this answer to a known, ground truth answer. Are RMs still relevant? Although RMs do not play a role in RLVR, most reasoning models (e.g., DeepSeek-v3/R1 is a relevant example) are still post-trained using both RLHF and RLVR. These techniques serve different purposes: - RLHF aligns the LLM to human preferences. - RLVR boosts the LLMs performance on (verifiable) reasoning tasks. Although it’s possible to use non-RL alternatives like DPO that do not require an RM for alignment, using PPO-based RLHF still seems to yield the best results at scale. For this reason, RMs are still necessary for aligning LLMs to human preferences.

254

10 Comments

Excited to announce that our group has had two papers accepted at NeurIPS this year, covering two very different and exciting topics! 1. 𝐏𝐚𝐫𝐞𝐭𝐨𝐐: 𝐒𝐜𝐚𝐥𝐢𝐧𝐠 𝐋𝐚𝐰𝐬 𝐢𝐧 𝐄𝐱𝐭𝐫𝐞𝐦𝐞𝐥𝐲 𝐋𝐨𝐰-𝐛𝐢𝐭 𝐋𝐋𝐌 𝐐𝐮𝐚𝐧𝐭𝐢𝐳𝐚𝐭𝐢𝐨𝐧 ParetoQ is a framework designed to effectively quantize LLMs to sub-4-bit precision. It surpasses all previous methods tailored to specific bit widths. Our ternary 600M-parameter model outperforms the previous SoTA ternary 3B-parameter model proposed in 1-bit era paper, using only one-fifth of the parameters. Paper: https://lnkd.in/ga7rP7Qb Code: https://lnkd.in/eWPYFryC  Model: https://lnkd.in/gBcmsBnu PyTorch blog: https://lnkd.in/g_wy2cr5 Congratulations to Zechun Liu, Changsheng Zhao and all the co-authors! 2. 𝐀𝐮𝐭𝐨𝐏𝐚𝐫𝐭𝐆𝐞𝐧: 𝐀𝐮𝐭𝐨𝐠𝐫𝐞𝐬𝐬𝐢𝐯𝐞 3𝐃 𝐏𝐚𝐫𝐭 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐃𝐢𝐬𝐜𝐨𝐯𝐞𝐫𝐲 AutoPartGen is an autoregressive model that generates complex 3D objects by producing one 3D part at a time and seamlessly assembling them into coherent final reconstructions. Leveraging a latent 3D representation, the model can automatically determine the number and type of parts based on various inputs, including 2D images, 2D part masks, or an existing 3D object. AutoPartGen achieves state-of-the-art performance in 3D part generation. Project: https://lnkd.in/gTUE_Ceb  Paper: https://lnkd.in/gtPHHd9B Congratulations to Minghao Chen, Andrea Vedaldi and all the co-authors!

203

1 Comment

There’s a new RL algorithm in town… and it’s already shaking up the LLM research community. 🤯 Group Sequence Policy Optimization (GSPO) was created by the research team at Qwen and is purpose built for operating at the sequence level rather than the token level. This means it evaluates, clips, and optimizes entire response sequences by their overall likelihood. The important upshot is that it directly aligns the optimization process with how rewards are actually assigned in real world scenarios! Some key reasons to be excited! 1. Better stability. GSPO improves training stability and helps avoids model collapse that can happen with algorithms like GRPO. 2. Scalable RL for mixture of experts (MoE) models. GSPO inherently solves stability challenges teams face when scaling RL for MoE architectures. Large scale RL becomes more practically feasible! 3. Simplifies Infra. Sequence-level optimization is inherently less sensitive to small numerical differences between training and inference systems compared to token-level methods, which should lead to more robust RL pipelines! We can already see signs that this has paid off in big ways for the open-source community. Qwen3 models are doing really well on benchmarks, boasting competitive performance with top models like Gemini-2.5 Pro on GPQA, AIME25, and LiveCodeBench! https://lnkd.in/gCZMQvUj P.S. 👀 If you want to run RL on your own Qwen3 model with GRPO/GSPO, keep an eye on our open-source post-building repo 🧡 : https://lnkd.in/gXZ32qkm #AI #LLM #ReinforcementLearning

82

10 Comments

Recent research by UIUC and Intel Labs has introduced a new jailbreak technique for Large Language Models (LLMs) known as InfoFlood. This method takes advantage of a vulnerability termed "Information Overload," where excessive linguistic complexity can circumvent safety mechanisms without the need for traditional adversarial prefixes or suffixes.   InfoFlood operates through a three-stage process: Linguistic Saturation, Rejection Analysis, and Saturation Refinement. Initially, it reformulates potentially harmful queries into more complex structures. If the first attempt does not succeed, the system analyzes the response to iteratively refine the query until a successful jailbreak is achieved. Empirical validation across four notable LLMs—GPT-4o, GPT-3.5-turbo, Gemini 2.0, and LLaMA 3.1—indicates that InfoFlood significantly surpasses existing methods, achieving success rates up to three times higher on various benchmarks.   The study underscores significant vulnerabilities in current AI safety measures, as widely used defenses, such as OpenAI’s Moderation API, proved ineffective against InfoFlood attacks. This situation raises important concerns regarding the robustness of AI alignment systems and highlights the necessity for more resilient safety interventions. As LLMs become increasingly integrated into diverse applications, addressing these vulnerabilities is crucial for ensuring the responsible deployment of AI technologies and enhancing their safety against emerging adversarial techniques.  Arxiv: https://lnkd.in/eBty6G7z

53

4 Comments

🔎 Recent Observations in Embedding Modeling (Gemini & Qwen3) The frontier in embedding modeling is increasingly shaped by large LLMs like Google Gemini (https://lnkd.in/dp8ygAuy) and Alibaba Qwen3 (https://lnkd.in/dREPHW8m). Both teams are converging on a similar recipe for building state-of-the-art embeddings for retrieval, ranking, and search. 🎯 Common Patterns Emerging: • Leverage LLM Backbones: Both approaches start from large, pretrained LLMs (decoder-only, causal architectures) rather than building encoder models from scratch. • Synthetic Data Generation: Synthetic positive and hard negative pairs are created in bulk—using the LLM itself—to augment or even replace scarce labeled data. • Contrastive Fine-Tuning: Embedding quality is improved by fine-tuning with contrastive objectives, with in-batch / hard negatives and multi-stage training. • Task Conditioning: Simple instruction prompts (like "query:" or "passage:") help the model generalize across tasks without re-architecting. • Pooling over Causal Outputs: Instead of modifying the attention mask to be bidirectional, these models pool the outputs of a decoder-only, left-to-right model for sequence-level representations. 💡 As these practices become standard, a tough question remains for downstream adoption: how can we fine-tune LLM-based embedding models for a specific task without catastrophic forgetting, if we lack the diverse multi-task or general-domain data that major labs have access to? • Mixing in original language modeling or general tasks is a proven way to retain model generality, but it requires substantial extra data and resources. • In practice, many organizations have only narrow downstream data. This creates a real risk of overfitting or catastrophic forgetting when fine-tuning for new embedding tasks. Would love to hear thoughts, practical experiences, or solutions from those tackling this problem!

55

5 Comments