Raquel Urtasun

What is Function calling & MCP for LLMs? (explained with visuals and code) Before MCPs became popular, AI workflows relied on traditional Function Calling for tool access. Now, MCP is standardizing it for Agents/LLMs. The visual below explains how Function Calling and MCP work under the hood. Today, let's learn: - Function calling by building custom tools for Agents. - How MCPs help by building a local MCP client with mcp-use and using tools from Browserbase MCP server. In Function Calling: - The LLM receives a prompt. - The LLM decides the tool. - The programmer implements a procedure to accept a tool call request from the LLM and prepare a function call. The tool call request is found in the LLM's response when you prompt it. - A backend service executes the tool. This Function Calling takes place within our stack: - We host the tool. - We implement a logic to determine the tool to invoke and its parameters. - We execute it. So Function Calling requires us to wire everything manually. MCP simplifies this! Instead of hard-wiring tools, MCP: - Standardizes defining, hosting, and exposing tools. - Makes it easy to discover tools, understand schemas, and use them. - Demands approval before invoking them. - Detaches implementation from consumption. For instance, whenever you integrate an MCP server, you never write a line of Python code to integrate the tools. Instead, you just integrate the MCP server and everything beyond this follows a standard protocol handled by the MCP client and the LLM: - They identify the MCP tool. - They prepare the input argument. - They invoke the tool. - They use the tool’s output to generate a response. Everything happens through a standard (but abstracted) protocol. So here’s the key point: MCP and Function Calling are not in conflict. They’re two sides of the same workflow. - Function Calling helps an LLM decide what it wants to do. - MCP ensures that tools are reliably available, discoverable, and executable, without you needing to custom-integrate everything. For example, an agent might say, “I need to search the web,” using function calling. That request can be routed through MCP to select from available web search tools, invoke the correct one, and return the result. Check the workflow in the diagram below. In this setup, to build a local MCP client, I used mcp-use because it lets us connect any LLM to MCP servers & build private MCP clients, unlike Claude/Cursor. - Compatible with Ollama & LangChain - Stream Agent output async - Built-in debugging mode, etc Find the mcp-use GitHub repo in the comments! ____ Find me → Avi Chawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

269

11 Comments

🚀 New blog post: How we're using multiple LLMs (Gemini, Claude, ChatGPT & more) to revolutionize software performance optimization at Fastcode ✨ Feeding academic papers + source code into AI models to uncover optimization opportunities that would take months to find manually 📈 Results: ~67% of AI recommendations lead to measurable performance gains 🔗 https://bit.ly/45WGw6u #AI #PerformanceEngineering #LLMs #SoftwareOptimization #Fastcode

65

3 Comments

The resurgence of reinforcement learning is great, but it’s missing the big picture. Chasing traditional RL is the wrong direction. In pursuing better policies and reward shaping, it’s easy to miss a simpler tool we already have: language. There is no reason to work in weights and gradients if you can work in natural language. Today we’re releasing our first set of research on Prompt Learning — not prompt tuning, not chain-of-thought. Something new. To our knowledge, no one has yet rigorously researched, characterized, and measured a full implementation of this approach. Instead of adjusting weights, we use MetaPrompting — where English evals and critiques (rather than just scalar metrics like previously done by the industry) drive targeted prompt updates. It’s a promising avenue of research, and the early results show that it works. Writeup in the comments.

173

10 Comments

More big news coming out of AiHello! We’re collaborating with Masters and PhD researchers at the University of Toronto to tackle one of the toughest challenges in AI today: reducing hallucinations in large language models. At AiHello, we’ve already built and scaled automation systems that handle over $1B in yearly ad revenue for e-commerce & Amazon sellers worldwide. Running AI at this scale has taught us what works, what doesn't. Now, we’re bringing that same expertise into the next frontier: building a hallucination-free LLM. The goal is simple but ambitious.  Creating an AI that can provide factual numbers because in ecommerce business, hallucinating about your business stats can be devastating. This is more than just research. It’s about applying real-world, scaled experience to solve a problem that will define the next generation of trustworthy AI. Our ML department still has a long way to go but we are on the right track. My goal is to get our LLM on the global leader board before 2025 Excited to start work on it!

54

6 Comments

🦾 Great milestone for open-source robotics: pi0 & pi0.5 by Physical Intelligence are now on Hugging Face, fully ported to PyTorch in LeRobot and validated side-by-side with OpenPI for everyone to experiment with, fine-tune & deploy in their robots! π₀.₅ is a Vision-Language-Action model which represents a significant evolution from π₀ to address a big challenge in robotics: open-world generalization. While robots can perform impressive tasks in controlled environments, π₀.₅ is designed to generalize to entirely new environments and situations that were never seen during training. Generalization must occur at multiple levels: - Physical Level: Understanding how to pick up a spoon (by the handle) or plate (by the edge), even with unseen objects in cluttered environments - Semantic Level: Understanding task semantics, where to put clothes and shoes (laundry hamper, not on the bed), and what tools are appropriate for cleaning spills - Environmental Level: Adapting to "messy" real-world environments like homes, grocery stores, offices, and hospitals The breakthrough innovation in π₀.₅ is co-training on heterogeneous data sources. The model learns from: - Multimodal Web Data: Image captioning, visual question answering, object detection - Verbal Instructions: Humans coaching robots through complex tasks step-by-step - Subtask Commands: High-level semantic behavior labels (e.g., "pick up the pillow" for an unmade bed) - Cross-Embodiment Robot Data: Data from various robot platforms with different capabilities - Multi-Environment Data: Static robots deployed across many different homes - Mobile Manipulation Data: ~400 hours of mobile robot demonstrations This diverse training mixture creates a "curriculum" that enables generalization across physical, visual, and semantic levels simultaneously. Huge thanks to the Physical Intelligence team & contributors Model: https://lnkd.in/eAEr7Yk6 LeRobot: https://lnkd.in/ehzQ3Mqy

367

10 Comments