First Hands-On Experience with Promptfoo for LLM Testing

No code tools are the new coding of the development world But what does it mean to the competeitors (Make, n8n, zapier)? This is what I think OpenAI’s new AgentKit changes what “automation” means. It brings models, tools, and memory into one place turning AI agents into full-fledged doers. That doesn’t put Make.com or n8n in danger. It opens new ground. They still excel at what AgentKit can’t......yet: ✅ Deep integrations across systems (CRMs, thirdparty apps) ✅ Reliable workflows with complex branching ✅ Flexibility for builders who need the extra umf in control The real opportunity? Collaboration, not competition. Think AgentKit as the brain and Make or n8n as the nervous system connecting it to the world. Automation isn’t getting replaced. It’s getting upgraded. The next wave belongs to those who bridge AI intelligence with real-world execution. What do you think about this new plot twist?

Evaluations for LLMs often get treated as an afterthought. But without them, you’re essentially shipping blind. Honestly, evaluating LLMs is not straightforward. Some of the biggest challenges I see teams run into: → No consistent framework for benchmarking models across tasks → Evals are run ad-hoc, not integrated into CI/CD pipelines → Hard to simulate multi-turn, agentic workflows with tools + memory → Regression detection (“did I break the prompt?”) is manual and error-prone → Test suites don’t evolve as systems retrain and scale Eval Protocol (EP) directly addresses these pain points. It’s an open protocol + SDK built by Fireworks AI, that brings engineering rigor (unit tests, regression checks), CI/CD pipelines to LLM applications. Why it matters: → EP supports both single-turn and multi-turn/agentic evals → You can define evals with thresholds, rollout processors, and even simulate agents with MCP configs → It scales with your workflow, from local model selection to long-term system maintenance If you’re building GenAI applications at scale, you should check this out: evalprotocol.io Because reliable systems don’t just depend on good prompts or models, they depend on disciplined evaluation. How are you currently approaching evals in your workflow? 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

🔥 Last week, we launched AgentCompass and our team is getting questions, feedback, appreciation - left, right and centre. The inbound rush has been crazy (and energizing). But in the middle of that, we kept getting one genuine question: “How exactly do you categorize errors in agent traces? And how does it help in root cause analysis?” Here’s the short version ⬇️ AgentCompass introduces a structured taxonomy of 5 trace error categories: 🧠 Thinking & Response Issues ->  e.g. hallucinations, misinterpretations, poor decision logic 🛡️ Safety & Security Risks -> e.g. leaks, biased or dangerous content 🔧 Tool & System Failures ->  e.g. API errors, misconfigurations, runtime faults 📉 Workflow & Task Gaps ->  e.g. lost context, goal drift, orchestration errors 🔍 Reflection Gaps ->  meta-level failures like no self-correction or blind retries of failed actions% 👉 Why this matters: 1. Instead of “your agent failed,” you now know what failed, where it failed, and why. 2. That means faster root cause discovery, automated clustering of recurring issues, and fewer hours spent digging through raw traces. 3. For builders, this means debugging agents like debugging code: structured, repeatable, measurable. That’s the part we are most excited about. Reliable AI isn’t about more metrics, it’s about the right categories that tell you where to look. Run Compass ONCE. You’ll know where your agent falters most often and why- https://lnkd.in/giceCBmC

Every week, I get questions like — “What exactly is an AI Agent?” “Isn’t it just a bot with an LLM?” Not really. An AI Agent is more like a 𝘀𝘆𝘀𝘁𝗲𝗺 𝘁𝗵𝗮𝘁 𝗰𝗮𝗻 𝘁𝗵𝗶𝗻𝗸, 𝗿𝗲𝗮𝘀𝗼𝗻, 𝗮𝗻𝗱 𝗮𝗰𝘁. The LLM is just one part — it gives the brain power.  • 𝗢𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻 𝗟𝗮𝘆𝗲𝗿: breaks high-level goals into smaller reasoning and execution steps  • 𝗧𝗼𝗼𝗹𝘀 & 𝗘𝘅𝘁𝗲𝗻𝘀𝗶𝗼𝗻𝘀: allow real-world actions — querying databases, calling APIs, automating workflows  • 𝗗𝗮𝘁𝗮 𝗦𝘁𝗼𝗿𝗲𝘀: ground the agent in facts and real-time context  • 𝗠𝗲𝗺𝗼𝗿𝘆 𝗟𝗮𝘆𝗲𝗿𝘀:  --𝗦𝗵𝗼𝗿𝘁-𝘁𝗲𝗿𝗺 𝗺𝗲𝗺𝗼𝗿𝘆 holds the current conversation or task context  --𝗟𝗼𝗻𝗴-𝘁𝗲𝗿𝗺 𝗺𝗲𝗺𝗼𝗿𝘆 helps the agent retain insights across sessions for adaptive behavior  • 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 & 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆: ensure the agent operates transparently, tracks lineage, and respects policies  • 𝗧𝗲𝗹𝗲𝗺𝗲𝘁𝗿𝘆 & 𝗟𝗼𝗴𝗴𝗶𝗻𝗴: provide insight into what the agent is doing, how decisions are made, and when to intervene  • 𝗣𝗿𝗼𝘁𝗼𝗰𝗼𝗹𝘀 (𝗠𝗖𝗣, 𝗔𝟮𝗔): connect agents, tools, and systems for smooth coordination I put everything in one place — 𝘁𝗵𝗶𝘀 𝗔𝗜 𝗔𝗴𝗲𝗻𝘁𝘀 𝗖𝗵𝗲𝗮𝘁 𝗦𝗵𝗲𝗲𝘁 — to help you understand each component and how it all connects. If you’re experimenting with multi-agent systems, or building orchestration layers around LLMs, this will help you see the big picture before you dive into the code.

“Zapier always breaks though doesn't it.. I've heard Make.com is better.” I've heard this twice now this week (and many times before). Both can break, and so do a lot of other platforms i've used. This shouldn't stop you using them - they have different strengths depending on your need. I use both interchangeably all the time. Make.com is brilliant for complex workflows, but Zapier is quicker for simple ones (the AI Copilot is really good for this). Both are fine so long as you design with failure in mind. Because here’s the truth: no automation tool is ‘set and forget’. APIs change, triggers misfire, data fields get edited. It’s not the tool’s fault - it’s just software being software and people being people. What matters isn’t if it breaks but: → Do you know when it happens? → Can you recover without losing data? → Have you built a way to roll back or re-run? → Most importantly: have you trained your team on how it works? Its worth saying i do love both tools, but right now my favourite is Gumloop - powerful, flexible, great for AI-first ops. But give it time and I’m sure i'll hear someone else say the same about it too.

⚙️ Best Practices, Tips & Pro Tricks ✅ Follow These Rules: Use descriptive names (getCurrentDateTime) Write natural-language descriptions Separate Info Tools & Action Tools Use directReturn = true for fast responses Keep output simple & human-readable 🧠 Analogy: Each tool = microservice → clean input/output → single purpose 💡 Pro Tip: Declarative (@Tool) = Simplicity Programmatic (ToolCallback) = Control 🎯 Conclusion: Organized tools make AI code cleaner, faster, and more explainable. #SpringAI, #AIEngineering, #BestPractices, #LLM, #FunctionCalling, #QAAutomation, #JavaDevelopment, #AITools, #CleanCode, #PromptEngineering

Amazing how far we’ve come in not even 3 years. 🔥 I have finally tested the new Perplexity Comet Browser. ☄️ I got the invitation and decided to give it a shot after Emanuel Stadler and Jakub Popluhar strongly recommended it. And this might be the beginning of how we actually start working with AI like a teammate. I recorded a video showing exactly how it works. The Comet Browser doesn’t just answer questions. 👉 It tracks your research. 👉 It understands your context. 👉 And it stays with you while you move across tabs, projects, and tools. In the video, I ask Comet to help me plan a flowchart for Email Automation. Comet create the plan, breaks it down, and builds a flowchart inside the canvas. 🤯 This is the first time I’ve felt like an AI tool is thinking with me instead of just reacting to me. We’re used to AI giving outputs. This feels more like a teammate. You can watch the demo in the video I posted. Curious what this could replace for you? 👀 Happy building 🧑💻

Just shipped selfvcs: A lightweight, local VCS that supercharges your dev workflow with AI smarts! No more bland commits get semantic messages, change insights (feat/fix/refactor?), risk levels, auto-version bumps, and changelog gen in one CLI. Core Implementations 1)Local Repository System: Each project is initialized into a structured repository that stores snapshots of code versions locally using file hashing and timestamp-based version mapping. 2)Commit & History Management: The system supports both manual and AI-assisted commit messages (via a local LLM), automatically logging metadata such as time, user, and file change summary. 3)Change Detection Algorithm: A custom comparison module tracks file additions, deletions, and modifications using hashing for efficiency simulating Git’s diff mechanism in a simplified, educational way. 4)Pull & Restore Functionality: Users can revert to any previous commit safely, restoring project files to their exact state at that point in history. 5)Secure Offline Operation: Every process from commit creation to history management runs completely offline. No data leaves the machine, ensuring privacy and full local autonomy. pip install selfvcs Key perks: 1)Private & local: Everything stays on your machine. 2)AI-powered: LLM detects changes, suggests pro messages, tags impacts. 3)Simple: Drop-in CLI with core commands like push, log, and more. All commands: 1)selfvcs init - Initialize repository 2)selfvcs push <files> -m "<message>" - Push with manual message 3)selfvcs push <files> --auto-detect - Push with AI-generated message 4)selfvcs log -n <num> - Show last N commits 5)selfvcs pull <commit_id> -o <dir> - Restore commit to directory 6)selfvcs delete <commit_id> - Delete a commit 7)selfvcs status - Show repo status 8)selfvcs auth set <groq-key> - Set GROQ API key 9)selfvcs changelog -n <num> - Generate changelog from last N commits 10)selfvcs insight <commit_id> - Show insights for a commit Install & try! Open to feedback! #ai #groq #innovation

This guy literally builds n8n AI Agents with 1 prompt I'm sharing his prompt below but first some thoughts on how he wrote the prompt: 1. Context Dump Everything > Don't hold back. Your use case, data sources, desired outputs... throw it all in there. The AI needs to understand your entire ecosystem, not just the happy path. 2. Demand Production-Ready Code > Explicitly ask for error handling, retry logic, and fallback mechanisms. The difference between a demo and something you can actually ship? Those three things right there. 3. Request the Full Stack > Node configurations. JSON structures. Testing checklists. Environment variables. When you ask for everything upfront, you get a complete blueprint, not just fragments you'll struggle to piece together later. 4. Include Edge Cases > Ask for "common edge cases and solutions" in your prompt. This one line saves you from those 2am debugging sessions when everything breaks in production. 5. Make It Maintainable > Specify "make it maintainable by someone else." This forces cleaner architecture, proper naming conventions, and documentation that your future self (or teammate) will actually thank you for. Check out his full prompt in comments and use it to build AI Agents in n8n. P.S. "This Guy" = X / alex_prompter

Every week, I get questions like — “What exactly is an AI Agent?” “Isn’t it just a bot with an LLM?” Not really. An AI Agent is more like a 𝘀𝘆𝘀𝘁𝗲𝗺 𝘁𝗵𝗮𝘁 𝗰𝗮𝗻 𝘁𝗵𝗶𝗻𝗸, 𝗿𝗲𝗮𝘀𝗼𝗻, 𝗮𝗻𝗱 𝗮𝗰𝘁. The LLM is just one part — it gives the brain power. • 𝗢𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻 𝗟𝗮𝘆𝗲𝗿: breaks high-level goals into smaller reasoning and execution steps • 𝗧𝗼𝗼𝗹𝘀 & 𝗘𝘅𝘁𝗲𝗻𝘀𝗶𝗼𝗻𝘀: allow real-world actions — querying databases, calling APIs, automating workflows • 𝗗𝗮𝘁𝗮 𝗦𝘁𝗼𝗿𝗲𝘀: ground the agent in facts and real-time context • 𝗠𝗲𝗺𝗼𝗿𝘆 𝗟𝗮𝘆𝗲𝗿𝘀: --𝗦𝗵𝗼𝗿𝘁-𝘁𝗲𝗿𝗺 𝗺𝗲𝗺𝗼𝗿𝘆 holds the current conversation or task context --𝗟𝗼𝗻𝗴-𝘁𝗲𝗿𝗺 𝗺𝗲𝗺𝗼𝗿𝘆 helps the agent retain insights across sessions for adaptive behavior • 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 & 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆: ensure the agent operates transparently, tracks lineage, and respects policies • 𝗧𝗲𝗹𝗲𝗺𝗲𝘁𝗿𝘆 & 𝗟𝗼𝗴𝗴𝗶𝗻𝗴: provide insight into what the agent is doing, how decisions are made, and when to intervene • 𝗣𝗿𝗼𝘁𝗼𝗰𝗼𝗹𝘀 (𝗠𝗖𝗣, 𝗔𝟮𝗔): connect agents, tools, and systems for smooth coordination I put everything in one place — 𝘁𝗵𝗶𝘀 𝗔𝗜 𝗔𝗴𝗲𝗻𝘁𝘀 𝗖𝗵𝗲𝗮𝘁 𝗦𝗵𝗲𝗲𝘁 — to help you understand each component and how it all connects. If you’re experimenting with multi-agent systems, or building orchestration layers around LLMs, this will help you see the big picture before you dive into the code.