Svetlana Popova

The complexity of PPO leads practitioners to avoid online RL in favor of RL-free or offline algorithms (e.g., DPO), but why not just use simpler versions of online RL? TL;DR: REINFORCE and RLOO have been shown to work well for training LLMs. And, they do not require a value model, which drastically reduces cost and complexity. Therefore, instead of completely avoiding online RL due to the overhead associated with PPO, we can just use algorithms that provide the benefits of online RL without the unnecessary complexity. Policy gradient: RL optimizers operate by computing the gradient of the RL objective (maximizing rewards) w.r.t. policy parameters—this is called the policy gradient—and performing gradient ascent. A basic policy gradient is easy to compute (see figure) but has high variance. So, we introduce some extra complexity to get stable gradients / training. Actor-critic: We can reduce variance by incorporating the advantage function (instead of the raw reward for a trajectory) into the policy gradient estimate. When we do this, the value function acts as a normalizer / baseline to the reward estimate for a state. PPO estimates advantage with an additional copy of the LLM—called a value model or critic—that is trained via an MSE loss to predict the final reward for every token in a sequence. The critic is on-policy (i.e., it depends on the current parameters of the policy during training) and is trained alongside the policy itself during RL. This is called an actor-critic framework. REINFORCE: Due to keeping and extra copy of the LLM in memory that is trained, actor-critic frameworks come with additional cost and complexity. As an alternative, we can use the much simpler REINFORCE algorithm, which estimates the value function via an average of rewards (either a moving average of all rewards or an average of rewards in a batch) throughout training. Estimating the value in this way is low cost and removes the need for a critic. REINFORCE does yield higher variance relative to actor-critic methods, but recent work has shown that this does not matter in the context of LLMs. Algorithms like PPO were developed in a prior generation of DeepRL research, where neural networks were primarily trained from scratch. In the LLM domain, we are finetuning pretrained models with a strong prior—unstable gradients are much less of a worry. For this reason, even simple online RL algorithms like REINFORCE work well for training LLMs despite their higher variance. RLOO: To reduce variance relative to REINFORCE, we can also use REINFORCE leave one out (RLOO). This algorithm reduces variance by sampling multiple completions for every prompt and estimating the value function as the average of rewards for other completions—excluding the completion for which the value is being estimated—to the same prompt.

215

6 comentarios

The Context Engineering Framework is quickly becoming one of the most important tools for anyone building reliable LLM systems. Getting the model to respond is the easy part. The real challenge is: → What should the model know right now? → Where should that info come from? → How should it be structured, stored, retrieved, or compressed? That’s exactly what this framework solves. 🧠 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 Context engineering = designing dynamic systems that deliver the right info, in the right structure, at the right time, so models can reason, retrieve, and respond effectively. This matters most in agents, copilots, retrieval-augmented pipelines, and anything with memory or tools. ⚙️ 𝗜𝗻𝘀𝗶𝗱𝗲 𝘁𝗵𝗲 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 Here’s the 3-layer system I use when designing end-to-end LLM workflows 👇 1️⃣ Context Retrieval & Generation → Prompt Engineering & Context Generation → External Knowledge Retrieval → Dynamic Context Assembly 2️⃣ Context Processing → Long Sequence Processing → Self-Refinement & Adaptation → Structured + Relational Information Integration 3️⃣ Context Management → Fundamental Constraints (tokens, latency, structure) → Memory Hierarchies & Storage Architectures → Context Compression & Trimming 🧱 All of this feeds into the Context Engine, which handles: → User Prompts → Retrieved Info → Available Tools → Long-Term Memory This is what gives your system continuity, task awareness, and reasoning depth across steps. ⚙️ Tools I would recommend: → LangGraph for orchestration + memory → Fireworks AI for fast, open-weight inference → LlamaIndex for modular retrieval → Redis & Vector DBs for scoped memory recall → Claude/Mistral for summarization and compression If your system is hallucinating, drifting, or missing the mark, it’s likely a context failure, not a prompt failure. 📌 Save this framework. 📩 Share it with your team before your next agent or RAG deployment. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for real-world GenAI system breakdowns, and subscribe to my Substack for deep dives and weekly insights: https://lnkd.in/dpBNr6Jg

321

59 comentarios

CAG is RAG’s quiet but powerful cousin -- and it fixes RAG’s biggest weakness. Over the past year, most of the industry has been focusing on RAG. But Cache Augmented Generation (CAG) is showing up as a powerful alternative. Why: - RAG is great when you need fresh data and external retrieval. - CAG shines when you work with data that changes rarely but is accessed very frequently. This avoids repeated retrieval and reduces latency and cost. How CAG works: - You preload “cold” data (rarely changing, high-usage datasets) directly into the model’s KV cache. - The cache is then reused across queries without recomputation. - For example, long compliance rules, internal guidelines, or product manuals can be stored once and applied instantly for every request. - This removes the overhead of retrieval from databases or embeddings for information you always need. It also allows the LLM to respond faster and with higher consistency. Why it’s better than RAG (in some cases): - RAG adds context at runtime -- higher latency, more complexity. - CAG brings the context “pre-baked” into the session -- faster, cheaper, and more reliable for static data. Of course, the real power comes when you combine RAG and CAG -- where you're caching what rarely changes, and retrieving what must stay fresh. RAG retrieves. CAG remembers. And in doing so, it eliminates RAG’s biggest problem: latency :) ♻️ Share it with anyone who is trying to get RAG to work :) I regularly share AI Agents and projects on my newsletter: https://lnkd.in/gaJTcZBR #AI #GenAI #AIAgents

249

8 comentarios

Scared of using LLMs in Python? It’s easier than you think. I procrastinated on using LLM for a long time, Even though I saw the power of LLM models in ✔️ Analyzing unstructured, long-form text ✔️ Turning data into plain-English summaries ✔️ Automating repeated tasks, like weekly reporting But it seemed so complex. It felt like “real developer” work. But then I tried it. And it literally takes <10 lines of code. So I’m sharing it here, so you can try it too! Currently, my favorite models to work with are: - Sonnet 4 - Gemini Flash 2.5 - GPT 5 (sometimes I switch back to 4o) ------ 𝗗𝗮𝘄𝗻'𝘀 𝗛𝗼𝘁 𝗧𝗶𝗽𝘀 𝗧𝗶𝗽 #1 never work backwards from the solution. Don’t “force” an LLM into your analyses. That’s adding unnecessary complexity. Make sure you have a good use case for an LLM, before doing the work. 𝗧𝗶𝗽 #2 Use providers, like OpenRouter to quickly switch between models Not sure which model best fits your use case? Use a provider like OpenRouter, so you can test out different models, by changing just the model field. 𝘗𝘚: 𝘵𝘩𝘪𝘴 𝘪𝘴 𝘸𝘩𝘢𝘵 𝘐 𝘶𝘴𝘦 𝘵𝘰 𝘲𝘶𝘪𝘤𝘬𝘭𝘺 𝘵𝘦𝘴𝘵 𝘥𝘪𝘧𝘧𝘦𝘳𝘦𝘯𝘵 𝘮𝘰𝘥𝘦𝘭𝘴 𝘧𝘰𝘳 Interview Master . 𝗧𝗶𝗽 #3  Always sanity check outputs LLMs can speed you up, but they still make mistakes. ------ ♻️ Was this useful? Repost it so others can see it too. 📩 I’m writing about this in more detail in my next newsletter. Article is going out on Wednesday. Sign up → www.askdatadawn.com

1173

49 comentarios

Out there MLOps idea: What if Evidently AI used Ibis and/or Narwhals? In Evidently AI, there are a bunch of data tests you can set up. Typically either a "data quality test", like what you'd think of from Great Expectations or Soda e.g. missing values. Or a "data drift test" that compares distributions, e.g. wasserstein distance. === It seemed like a good idea, but my team chose not to go with it recently for a few reasons: 1. We wanted to be able to run our tests on dataframes AND against our lakehouse directly. Sometimes we want to run tests on our lakehouse (currently Snowflake--possibly soon Trino+S3+Iceberg+Glue Catalog). As far as we could tell, Evidently only supports pandas or PySpark dataframes. 2. Calculating the test suite using pandas was really slow (5 mins) on a medium+ sized dataset (I forget the exact size we tried). 3. We wanted a declarative format for our tests, like Soda has with SodaCL. === I think it'd be sick if these data testing tools supported both dataframes and varied compute engines. === Deepyaman Datta (or people from Voltron Data) am I thinking about this right? === Issue I opened on the Evidently repo: https://lnkd.in/gw3bAb8y

32

4 comentarios

Memory is becoming the new infrastructure layer of AI. Every week, new frameworks emerge promising to fix the same fundamental problem: LLMs forget too quickly, and retrieval systems can’t keep up with the complexity of real work. Let’s break down a few of the most influential approaches — and where they still fall short: 🔹 REFRAG (Meta + NUS) – Instead of stuffing long documents directly into context, it compresses retrieval chunks into embeddings, reducing latency and context bloat. Smart for efficiency, but still struggles with knowledge integration: how do you connect ideas across multiple documents, not just compress them? 🔹 KG-RAG (Adobe Research) – Builds structured knowledge graphs out of enterprise docs. Great for reducing hallucination and noise, but graph construction is brittle and costly. Updating ontologies fast enough for dynamic teams remains unsolved. 🔹 Human-Inspired Frameworks (SALM, CAIM, PREMem) – Bring in concepts like episodic, semantic, and procedural memory. These are closer to how humans think, shifting some reasoning into memory itself. But they’re mostly research-grade, not production-ready: integration into daily team workflows is minimal. 🔹 System-Level OS Approaches (MemOS, MemoryOS) – Treat memory like an operating system, with lifecycle management, scheduling, and migration across “short-term,” “mid-term,” and “long-term” layers. Strong architectural vision, but too abstract for non-research teams who just want answers today, not a full OS overhaul. 🔹 Social & Group Memory (Social-RAG) – Retrieves not just facts but social context from group interactions. Crucial for collaboration use cases, but limited to niche scenarios right now. Across all of these, two unsolved challenges keep coming up: Integration vs. Efficiency – Compressing memory helps scale, but often destroys the relational context teams actually need. Lifecycle & Evolution – Few frameworks handle the reality that knowledge is constantly being updated, contradicted, and reshaped. 💡 Why Tanka takes a different path Instead of treating memory as just a retrieval hack, Tanka treats it as the native infrastructure of work. Every message, doc, and decision is captured once, then structured into living memory — automatically. Agents don’t just retrieve snippets; they act on evolving context, update it, and carry it across workflows. Memory isn’t abstracted away into embeddings or graphs nobody sees — it’s visible, searchable, and agent-actionable. For teams drowning in information every day — switching between Slack, Notion, Google Docs, and email — this difference is existential. Most frameworks solve part of the puzzle. Tanka solves the team problem: how do we stop losing context every time someone leaves, a tool changes, or a project shifts? The answer: memory-native collaboration. Because the future of work isn’t just about smarter AI — it’s about AI that never forgets.

62

7 comentarios