Elena Gagliardoni

The complexity of PPO leads practitioners to avoid online RL in favor of RL-free or offline algorithms (e.g., DPO), but why not just use simpler versions of online RL? TL;DR: REINFORCE and RLOO have been shown to work well for training LLMs. And, they do not require a value model, which drastically reduces cost and complexity. Therefore, instead of completely avoiding online RL due to the overhead associated with PPO, we can just use algorithms that provide the benefits of online RL without the unnecessary complexity. Policy gradient: RL optimizers operate by computing the gradient of the RL objective (maximizing rewards) w.r.t. policy parameters—this is called the policy gradient—and performing gradient ascent. A basic policy gradient is easy to compute (see figure) but has high variance. So, we introduce some extra complexity to get stable gradients / training. Actor-critic: We can reduce variance by incorporating the advantage function (instead of the raw reward for a trajectory) into the policy gradient estimate. When we do this, the value function acts as a normalizer / baseline to the reward estimate for a state. PPO estimates advantage with an additional copy of the LLM—called a value model or critic—that is trained via an MSE loss to predict the final reward for every token in a sequence. The critic is on-policy (i.e., it depends on the current parameters of the policy during training) and is trained alongside the policy itself during RL. This is called an actor-critic framework. REINFORCE: Due to keeping and extra copy of the LLM in memory that is trained, actor-critic frameworks come with additional cost and complexity. As an alternative, we can use the much simpler REINFORCE algorithm, which estimates the value function via an average of rewards (either a moving average of all rewards or an average of rewards in a batch) throughout training. Estimating the value in this way is low cost and removes the need for a critic. REINFORCE does yield higher variance relative to actor-critic methods, but recent work has shown that this does not matter in the context of LLMs. Algorithms like PPO were developed in a prior generation of DeepRL research, where neural networks were primarily trained from scratch. In the LLM domain, we are finetuning pretrained models with a strong prior—unstable gradients are much less of a worry. For this reason, even simple online RL algorithms like REINFORCE work well for training LLMs despite their higher variance. RLOO: To reduce variance relative to REINFORCE, we can also use REINFORCE leave one out (RLOO). This algorithm reduces variance by sampling multiple completions for every prompt and estimating the value function as the average of rewards for other completions—excluding the completion for which the value is being estimated—to the same prompt.

217

6 commenti

If you’re an AI engineer, this paper is a fascinating exploration of what happens when large language models move from reasoning to acting. I was reading “Collaborating Action by Action: A Multi-Agent LLM Framework for Embodied Reasoning”, and it’s one of the most fascinating reads I’ve come across lately. It introduces MINDCraft, a framework that lets LLM agents collaborate, plan, and act inside the open-world environment of Minecraft. What “multi-agent LLM with embodied reasoning” means? Multiple LLM-driven agents act in a world, not just in text. They see partial state, plan over long horizons, talk to each other, and use tools to change the environment. This setup stresses multi-turn dialogue, partial observability, and quantitative evaluation of behavior, not just answers. Unlike standard benchmarks that test LLM reasoning in static or text-based settings, MINDCraft gives models agency, letting them see, decide, and act in a complex sandbox environment. Here’s what makes the framework so insightful: → It introduces a task-driven workflow where players (or configs) define high-level goals, like “build a house”, and the agent autonomously plans, queries the LLM, and executes step-by-step commands in Minecraft. → The Agent Runtime handles communication between the player’s task, the LLM API, and the Minecraft environment via a rich library of commands and history logs. → This setup makes it possible to evaluate not just output accuracy, but behavioral intelligence, things like persistence, resource gathering, planning depth, and tool use. → It’s also modular, allowing researchers to plug in different LLMs, run comparative tests, and measure emergent capabilities in a grounded world. I found this a fascinating read because it moves us closer to understanding how language models perform when they’re not just “thinking,” but acting, reasoning through goals, feedback, and real-world constraints. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

273

51 commenti

GLM-4.5, an extremely powerful Chinese open-weights model, has been available for two weeks. A detailed paper is now out with fascinating technical insights. The pace of progress in open-weights AI development is nothing short of astounding 🤩 Paper: https://lnkd.in/eww-4FSy

88

11 commenti

🤗 Welcome EmbeddingGemma — a multilingual embedding model from Google DeepMind, built for on‑device use cases and now leading the MTEB leaderboard for models under 500M parameters! And with Hugging Face Text Embeddings Inference v1.8.1, it's supported from day-zero across CPU, MPS, and CUDA for fast and efficient inference. TL;DR - 🤏 **308M parameters**: compact yet powerful, striking a balance between efficiency and performance, and specifically designed to make on‑device use cases practical. - ⚡ **Full precision + QAT checkpoints**: run under 200MB of RAM while preserving quality with Q4_0 and Q8_0 quantization. - 🏆 **State‑of‑the‑art below 500M parameters**: leading the MTEB leaderboard in its class. - 🌍 **Over 100 languages supported**: broad multilingual coverage out of the box. - ✍️ **Context length up to 2k tokens**: generous context for embedding long passages. - 🪆 **Matryoshka Representation Learning (MRL)**: 768‑dim by default, or truncate to 512, 256, or 128 to match task and compute budget. - 🤝 **Day‑zero support**: integrated with SentenceTransformers, Transformers.js, Text Embeddings Inference (TEI), and many more open‑source solutions. - 📄 **Gemma community license** Discover more details in our blog post — link in the first comment!

888

11 commenti

Why #SmallLanguageModels are the real future? ⚡ Most agents route every task, no matter how trivial - through GPT-4 or Claude → That’s expensive, slow, and overkill. ⚡ NVIDIA’s new paper argues→ #SLMs models small enough to run on consumer hardware are faster, cheaper, and just as capable for most agentic tasks. ⚡ Prioritize SLMs for cost-effective deployment → Cut latency, energy use, and infra costs, especially for real-time or on-device inference. ⚡ Design modular agentic systems → Use SLMs for routine, narrow tasks; reserve #LLMs only for complex reasoning. ⚡ Leverage SLMs for rapid specialization → Fine-tune small models quickly for evolving use cases, enabling faster iteration and easier adaptation. ⚡Agentic workloads are rarely open-ended conversations. They’re structured, repetitive, and scoped: ✸ Summarize a doc. ✸ Extract a field. ✸ Call an API. ✸ Generate boilerplate code. For these, you don’t need a trillion-parameter giant. A well-tuned 2–7B model often does the job just as well or better. ⚡ Proof in Numbers: ✸ Phi-3 Small (7B) rivals 70B on reasoning + code. ✸ DeepSeek-R1-Distill (7B) beats Claude 3.5 + GPT-4o on reasoning. ✸ SmolLM2 (≤1.7B) matches 70B models from 2 years ago. ✸ Nemotron-H (2–9B) delivers 30B-level tool use at a fraction of compute. ✸ Toolformer (6.7B) outperforms GPT-3 (175B) by learning APIs. Smaller ≠ weaker. ⚡The Efficiency Edge: ✸ 10–30× cheaper to run. ✸ Lower energy use + faster response. ✸ Overnight fine-tuning with LoRA/QLoRA. ✸ More reliable with strict formats (JSON, XML, Python). ✸ Deployable locally for cost savings + data control. ⚡Smarter Architectures: Don’t build monolithic agents. ✸ Use SLMs by default. ✸ Call LLMs only when absolutely necessary. This modular approach is cheaper, more controllable, and easier to debug. ⚡Migration path (LLM → SLM): ✸ Log usage data. ✸ Cluster recurring tasks. ✸ Fine-tune SLMs. ✸ Replace LLM calls. ✸ Iterate + refine. ⚡Real-world Results: ✸ MetaGPT → 60% of LLM calls replaceable. ✸ Cradle → 70%. ✸ Open Operator → 40%. ⚡ Why hasn’t Flipped yet: ✸ Billions sunk into LLM infra. ✸ Benchmarks biased toward generalist models. ✸ Less hype for SLMs But none of these are technical blockers. ⚡ The Takeaway: ✸ The future of agents isn’t bigger models. ✸ It’s smarter architectures built on SLMs. ✸ SLMs give you speed, control, and affordability.

57

6 commenti

I’m thrilled to share that the new STAC machine learning benchmark, covering Gradient-Boosted Tree models, is now live. Machine learning is being pushed closer to the edge as advances in hardware and software stacks reduce latency and enable more complex decision logic at the point of trade. Gradient-boosted tree models are a natural fit in electronic trading. They deliver strong predictive performance on market data while keeping inference times extremely low. This new benchmark focuses on inference latency and stability, two key factors for trading applications. Specifically, the benchmark measures how quickly different systems can score live market data using GBT models of varying size and complexity. A big thank you to the STAC-ML Working Group for their contributions in shaping this benchmark. Like all STAC benchmarks, it was developed with substantial input from quantitative researchers and low-latency engineers, ensuring its rigor and direct applicability to real-world trading environments. Submissions are now open, and we look forward to publishing the first results. STAC - Strategic Technology Analysis Center

35

1 commento