MLflow

How do you track your datasets in machine learning experiments? At our recent MLflow Community Meetup, Benjamin Wilson shared a deep dive into a major improvement — dataset tracking made simple. With a single unified interface, you can create, merge, and manage datasets directly in the MLflow tracking server. No more juggling multiple function calls or losing visibility into lineage. 🎥 Watch the full video: https://lnkd.in/gnAxi2ur #opensource #oss #linuxfoundation #mlflow

⏰ Don’t forget — MLflow Office Hours are this Wednesday, October 22 at 8AM PT on Zoom! Join the MLflow maintainers and contributors for a live, interactive session. Whether you’re debugging production issues or experimenting with LLMs and GenAI, this is your chance to ask questions directly and get expert feedback. Come for: 🔹 Real-time MLflow troubleshooting and guidance 🔹 Best practices for managing LLM & GenAI experiments 🔹 A look ahead at new MLflow features 🔗 Reserve your spot: https://lnkd.in/ezg-R8tc #opensource #oss #mlflow #genai #llm

Missed last week’s #MLflow Community Meetup? Check out this clip with Benjamin Wilson on 𝗮𝗴𝗲𝗻𝘁𝗶𝗰 𝗷𝘂𝗱𝗴𝗲𝘀! 🙌 “The judge no longer works as an LLM as a judge—it actually works as an agent as a judge.” In this mode, the trace metadata (the trace info object in MLflow) is passed in: the input to the call, the output, and basically the root span ID for that trace. With that metadata—and MLflow’s MCP features recently released—the judge can make tool calls to MLflow to do things like searching spans and querying different aspects of the trace. 🎥 Watch the full video to go deeper: https://lnkd.in/eDzZmd_E Have questions? Bring them to MLflow Office Hours next Wed, Oct 22 🔗 https://lnkd.in/ezg-R8tc #opensource #oss #mlflow #agenticjudges #llm #genai

In MLflow 3.4, the make_judge method introduces a declarative way to create MLflow Scorers, the core abstraction for automated evaluation. With simple instructions, you can build judges that understand your domain’s quality requirements and automatically align with feedback from human experts. This post shows how to: 🔹 Create custom scorers with make_judge using simple declarative instructions. 🔹 Build scorers that act as agents with built-in tools for trace introspection, handling complex evaluation without complicated prompts or complex span parsing logic. 🔹 Automatically align scorers with subject-matter expert preferences to improve scorer accuracy over time. 🔗 Dive in: https://lnkd.in/eHkcBvHN #opensource #oss #mlflow #LLM #genai #llmops

📣 MLflow Office Hours — Wednesday, October 22 at 8AM PT Connect directly with MLflow maintainers and contributors for live Q&A! Bring your production challenges or your latest #LLM and #GenAI experiments—this session is dedicated to hands-on technical discussion and feedback. You'll get: 🔹 Real-time support for both production and experimental MLflow setups 🔹 Expert guidance on LLM & GenAI model management 🔹 Early insight into future releases and new features 🔹 Best practices for tracking ML & LLM experiments Save your spot ➡️ https://lnkd.in/ezg-R8tc #opensource #oss #mlflow #genai #llm

MLflow 3.4 introduces an MCP server that enables conversational analysis of MLflow traces with assistants like Claude. Ask Claude—or any MCP‑compatible tool—to find failed traces, compare successful and unsuccessful runs, and identify patterns in retrieval steps. In this post, Daniel Liden walks through: 🔹 Setting up MLflow tracing with example data (including intentional failures) 🔹 Configuring the MCP server to work with Claude Desktop and Claude Code 🔹 Practical examples of using AI to analyze and debug AI application traces Setup takes about five minutes, and once configured, your trace data becomes something you can have a conversation with. Use AI to debug AI! ✅ 🔗 Read the full post: https://lnkd.in/eKhMt2kE #MLflow #MLOps #GenAI #AI #ClaudeCode

MLflow Ambassador Thor Steen Larsen will share insights on "How to use MLflow for Experiment Tracking and Deployment and how we use it in DSB" at Industrial Machine Learning in Action: From Models to Operations! 📅 Monday, November 3, 2025 ⏰ 09:00 – 13:30 📍IDA Conference Center, Copenhagen 🇩🇰 Explore practical ML deployment strategies and experiment tracking in real-world operations. ⬇️ 🔗 Registration link: https://lnkd.in/eJ4x9hDg #opensource #ml #MLflow #oss

⚡ In this lightning talk at MLOps World | GenAI Summit, Danny Chiao tackled a top agent challenge: ensuring high quality output. Rather than labeling and analyzing traces by hand, MLflow makes it easy to log, evaluate, and iterate faster—using techniques leading companies rely on to deploy agents in production. ✅ #mlopsworld #MLflow #opensource #oss #mlops #genai #agents

🚀 Join the next MLflow Community Meetup on Oct 8 at 4PM PT! 🔹 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗦𝗺𝗮𝗿𝘁𝗲𝗿 𝗘𝘃𝗮𝗹𝘀 𝘄𝗶𝘁𝗵 𝗧𝗿𝗮𝗰𝗲-𝗔𝘄𝗮𝗿𝗲, 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸-𝗔𝗹𝗶𝗴𝗻𝗲𝗱 𝗝𝘂𝗱𝗴𝗲𝘀: MLflow’s judges assess not just answers, but also the retrievals and steps behind them—and improve continually with real user feedback. 🔹 𝗞𝗲𝗲𝗽𝗶𝗻𝗴 𝗘𝘃𝗮𝗹 𝗗𝗮𝘁𝗮𝘀𝗲𝘁𝘀 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝘁 𝗮𝘀 𝗬𝗼𝘂𝗿 𝗔𝗽𝗽 𝗖𝗵𝗮𝗻𝗴𝗲𝘀: Update, version, and track your eval datasets in MLflow so every test stays aligned with your evolving app. Bring your questions about dataset management, evaluation workflows, or how to best contribute to MLflow OSS development! ✅ RSVP 👉 https://lnkd.in/e3Mivwje #opensource #oss #mlflow #communitymeetup #llm #genai

Attending MLOps World | GenAI Summit in Austin? 🙌 Don’t miss tomorrow’s lightning talk on the expo floor: 10:35–10:50 AM CT — “Techniques to build high‑quality agents faster with MLflow” with Danny Chiao 🔗 Learn more: https://lnkd.in/eTjQ7UQu Have MLflow questions? Join the LIVE, text‑based AMA in the MLflow Slack #General channel from 1–3 PM CT with Danny Chiao and Daniel Liden! 🤝 Bring questions on production deployments, labeling strategies, how to integrate MLflow for agent quality, & more. 💬 Join Slack: https://lnkd.in/easDYPqv #mlflow #opensource #oss #mlops #genai #agents