Javier Fernández Briones

What is Function calling & MCP for LLMs? (explained with visuals and code) Before MCPs became popular, AI workflows relied on traditional Function Calling for tool access. Now, MCP is standardizing it for Agents/LLMs. The visual below explains how Function Calling and MCP work under the hood. Today, let's learn: - Function calling by building custom tools for Agents. - How MCPs help by building a local MCP client with mcp-use and using tools from Browserbase MCP server. In Function Calling: - The LLM receives a prompt. - The LLM decides the tool. - The programmer implements a procedure to accept a tool call request from the LLM and prepare a function call. The tool call request is found in the LLM's response when you prompt it. - A backend service executes the tool. This Function Calling takes place within our stack: - We host the tool. - We implement a logic to determine the tool to invoke and its parameters. - We execute it. So Function Calling requires us to wire everything manually. MCP simplifies this! Instead of hard-wiring tools, MCP: - Standardizes defining, hosting, and exposing tools. - Makes it easy to discover tools, understand schemas, and use them. - Demands approval before invoking them. - Detaches implementation from consumption. For instance, whenever you integrate an MCP server, you never write a line of Python code to integrate the tools. Instead, you just integrate the MCP server and everything beyond this follows a standard protocol handled by the MCP client and the LLM: - They identify the MCP tool. - They prepare the input argument. - They invoke the tool. - They use the tool’s output to generate a response. Everything happens through a standard (but abstracted) protocol. So here’s the key point: MCP and Function Calling are not in conflict. They’re two sides of the same workflow. - Function Calling helps an LLM decide what it wants to do. - MCP ensures that tools are reliably available, discoverable, and executable, without you needing to custom-integrate everything. For example, an agent might say, “I need to search the web,” using function calling. That request can be routed through MCP to select from available web search tools, invoke the correct one, and return the result. Check the workflow in the diagram below. In this setup, to build a local MCP client, I used mcp-use because it lets us connect any LLM to MCP servers & build private MCP clients, unlike Claude/Cursor. - Compatible with Ollama & LangChain - Stream Agent output async - Built-in debugging mode, etc Find the mcp-use GitHub repo in the comments! ____ Find me → Avi Chawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

269

11 comentarios

The Context Engineering Framework is quickly becoming one of the most important tools for anyone building reliable LLM systems. Getting the model to respond is the easy part. The real challenge is: → What should the model know right now? → Where should that info come from? → How should it be structured, stored, retrieved, or compressed? That’s exactly what this framework solves. 🧠 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 Context engineering = designing dynamic systems that deliver the right info, in the right structure, at the right time, so models can reason, retrieve, and respond effectively. This matters most in agents, copilots, retrieval-augmented pipelines, and anything with memory or tools. ⚙️ 𝗜𝗻𝘀𝗶𝗱𝗲 𝘁𝗵𝗲 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 Here’s the 3-layer system I use when designing end-to-end LLM workflows 👇 1️⃣ Context Retrieval & Generation → Prompt Engineering & Context Generation → External Knowledge Retrieval → Dynamic Context Assembly 2️⃣ Context Processing → Long Sequence Processing → Self-Refinement & Adaptation → Structured + Relational Information Integration 3️⃣ Context Management → Fundamental Constraints (tokens, latency, structure) → Memory Hierarchies & Storage Architectures → Context Compression & Trimming 🧱 All of this feeds into the Context Engine, which handles: → User Prompts → Retrieved Info → Available Tools → Long-Term Memory This is what gives your system continuity, task awareness, and reasoning depth across steps. ⚙️ Tools I would recommend: → LangGraph for orchestration + memory → Fireworks AI for fast, open-weight inference → LlamaIndex for modular retrieval → Redis & Vector DBs for scoped memory recall → Claude/Mistral for summarization and compression If your system is hallucinating, drifting, or missing the mark, it’s likely a context failure, not a prompt failure. 📌 Save this framework. 📩 Share it with your team before your next agent or RAG deployment. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for real-world GenAI system breakdowns, and subscribe to my Substack for deep dives and weekly insights: https://lnkd.in/dpBNr6Jg

321

59 comentarios

GPT-OSS introduces an architectural detail that I haven’t seen discussed much: trainable attention sinks. While attention sinks have been used before—particularly in sparse attention mechanisms—they were traditionally implemented by preserving the first few tokens in the sequence during KV-cache pruning. This is based on the observation that removing these tokens causes a measurable drop in model quality. In previous approaches, these “sink” tokens were not trainable parameters—they were simply the original sequence tokens retained for stability. GPT-OSS takes a different approach. The model uses trainable sink parameters that are incorporated only during the attention score computation: - The sink is added to the attention logits. - Softmax is computed over the augmented scores. - The sink is then removed before continuing the attention computation. This is not a common design choice. Sebastian Raschka, PhD, suggests that this might serve to stabilize attention in long-context scenarios (https://lnkd.in/dpHw4dDZ). Whether this still holds in the sliding-window attention used by GPT-OSS remains to be confirmed, but it’s a plausible explanation. You can experiment with GPT-OSS on Cerebras Systems hardware here, achieving up to 3000 tokens/sec throughput: 🔗 https://lnkd.in/dACuEQgs

90

3 comentarios

The complexity of PPO leads practitioners to avoid online RL in favor of RL-free or offline algorithms (e.g., DPO), but why not just use simpler versions of online RL? TL;DR: REINFORCE and RLOO have been shown to work well for training LLMs. And, they do not require a value model, which drastically reduces cost and complexity. Therefore, instead of completely avoiding online RL due to the overhead associated with PPO, we can just use algorithms that provide the benefits of online RL without the unnecessary complexity. Policy gradient: RL optimizers operate by computing the gradient of the RL objective (maximizing rewards) w.r.t. policy parameters—this is called the policy gradient—and performing gradient ascent. A basic policy gradient is easy to compute (see figure) but has high variance. So, we introduce some extra complexity to get stable gradients / training. Actor-critic: We can reduce variance by incorporating the advantage function (instead of the raw reward for a trajectory) into the policy gradient estimate. When we do this, the value function acts as a normalizer / baseline to the reward estimate for a state. PPO estimates advantage with an additional copy of the LLM—called a value model or critic—that is trained via an MSE loss to predict the final reward for every token in a sequence. The critic is on-policy (i.e., it depends on the current parameters of the policy during training) and is trained alongside the policy itself during RL. This is called an actor-critic framework. REINFORCE: Due to keeping and extra copy of the LLM in memory that is trained, actor-critic frameworks come with additional cost and complexity. As an alternative, we can use the much simpler REINFORCE algorithm, which estimates the value function via an average of rewards (either a moving average of all rewards or an average of rewards in a batch) throughout training. Estimating the value in this way is low cost and removes the need for a critic. REINFORCE does yield higher variance relative to actor-critic methods, but recent work has shown that this does not matter in the context of LLMs. Algorithms like PPO were developed in a prior generation of DeepRL research, where neural networks were primarily trained from scratch. In the LLM domain, we are finetuning pretrained models with a strong prior—unstable gradients are much less of a worry. For this reason, even simple online RL algorithms like REINFORCE work well for training LLMs despite their higher variance. RLOO: To reduce variance relative to REINFORCE, we can also use REINFORCE leave one out (RLOO). This algorithm reduces variance by sampling multiple completions for every prompt and estimating the value function as the average of rewards for other completions—excluding the completion for which the value is being estimated—to the same prompt.

215

6 comentarios

"How One AI Model Creates a Physical Intuition of Its Environment" A nice article about our work with the Video-JEPA model showing how it learns a kind of "physical common sense." The model's prediction error shoots up whenever it is shown a video in which something impossible happens. The article has quoted from the lead author Quentin Garrido, collaborators like Randall Balestriero, scientists not connected to our work like neuroscientist Karl Friston and cognitive scientist Micha Heilbron. https://lnkd.in/e5uhG-_P

994

63 comentarios

Agents are getting most of the attention in the Generative AI space lately — and fairly so. But when it comes to enhancing LLM performance, there are other powerful levers we shouldn’t overlook. One of them is Fine-Tuning — and techniques like QLoRA are key for hyper-personalization, especially when working with domain-specific or user-centric data. In this video, I walk through how you can leverage Amazon SageMaker’s Multi-Adapter Inference feature — combined with vLLM’s Async Engine using the new Large Model Inference (LMI) container — to serve multiple adapters efficiently at scale. 👇 Check it out below and feel free to adapt it for your own use-cases. I’ve also attached the code sample if you want to experiment directly. #GenerativeAI #SageMaker #vLLM #QLoRA #MachineLearning #AWS #FineTuning https://lnkd.in/eEFAgFBF

51

2 comentarios

Very excited to announce Llama-Nemotron-Super-V1.5! Llama-Nemotron-Super-V1.5 is now better than Llama-Nemotron-Ultra-V1. This is currently the best model that can be deployed on a single H100. Reasoning On/Off and drop in replacement for V1. Open-weight, code and data on HF. https://lnkd.in/gbARuDJH

196

2 comentarios

Excited to announce that our group has had two papers accepted at NeurIPS this year, covering two very different and exciting topics! 1. 𝐏𝐚𝐫𝐞𝐭𝐨𝐐: 𝐒𝐜𝐚𝐥𝐢𝐧𝐠 𝐋𝐚𝐰𝐬 𝐢𝐧 𝐄𝐱𝐭𝐫𝐞𝐦𝐞𝐥𝐲 𝐋𝐨𝐰-𝐛𝐢𝐭 𝐋𝐋𝐌 𝐐𝐮𝐚𝐧𝐭𝐢𝐳𝐚𝐭𝐢𝐨𝐧 ParetoQ is a framework designed to effectively quantize LLMs to sub-4-bit precision. It surpasses all previous methods tailored to specific bit widths. Our ternary 600M-parameter model outperforms the previous SoTA ternary 3B-parameter model proposed in 1-bit era paper, using only one-fifth of the parameters. Paper: https://lnkd.in/ga7rP7Qb Code: https://lnkd.in/eWPYFryC  Model: https://lnkd.in/gBcmsBnu PyTorch blog: https://lnkd.in/g_wy2cr5 Congratulations to Zechun Liu, Changsheng Zhao and all the co-authors! 2. 𝐀𝐮𝐭𝐨𝐏𝐚𝐫𝐭𝐆𝐞𝐧: 𝐀𝐮𝐭𝐨𝐠𝐫𝐞𝐬𝐬𝐢𝐯𝐞 3𝐃 𝐏𝐚𝐫𝐭 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐃𝐢𝐬𝐜𝐨𝐯𝐞𝐫𝐲 AutoPartGen is an autoregressive model that generates complex 3D objects by producing one 3D part at a time and seamlessly assembling them into coherent final reconstructions. Leveraging a latent 3D representation, the model can automatically determine the number and type of parts based on various inputs, including 2D images, 2D part masks, or an existing 3D object. AutoPartGen achieves state-of-the-art performance in 3D part generation. Project: https://lnkd.in/gTUE_Ceb  Paper: https://lnkd.in/gtPHHd9B Congratulations to Minghao Chen, Andrea Vedaldi and all the co-authors!

207

1 comentario