Dr. Branislava Šandrih Todorović

📊 Agent Evaluation Frameworks 📚 👉 LangSmith - A toolkit for building and debugging LLM apps with step-by-step logging, trajectory analysis and human-in-the-loop validation via interactive UI. https://lnkd.in/eeZuK3dx 👉 Langfuse - Open-source observability for LLMs with support for OpenTelemetry, step-level logging and feedback capture. Ideal for self-hosting and real-time tracing. 📌 https://lnkd.in/eyAA253B 👉 Galileo Agentic Evaluation - Tracks agent behavior with fine-grained metrics beyond success/failure, enabling detailed analysis of actions and decision quality. 📌 https://lnkd.in/ekDFJYTe 👉 AgentEvals - A lightweight, LLM-judge-based framework for evaluating agent behavior by comparing action paths to expected workflows. 📌 https://lnkd.in/ekpSFwh3 👉 Databricks Mosaic ML Eval - Enterprise-grade tool for evaluating agents at scale, with cost optimization, synthetic data and stepwise assessment integrated into Databricks. 📌 https://lnkd.in/e9i96EDY 👉 AWS Labs: Agent Evaluation - A generative AI testing framework where an LLM acts as an evaluator to orchestrate and assess multi-turn interactions with agent. 📌 https://lnkd.in/e8DjFwsz 👉 Microsoft AgentEval (AutoGen) - LLM-based evaluation framework with Critic, Quantifier and Verifier agents for assessing multi-agent behavior. 📌 https://lnkd.in/efyUBNGf 👉 Vertex AI Gen AI Evaluation - Google Cloud’s managed service for LLM evaluation—includes live monitoring, A/B testing and OpenTelemetry support for scalable deployments. 📌 https://lnkd.in/eicWy5Dz 👉 Patronus AI - Focuses on safety, compliance and domain-specific benchmarks. Supports automated scoring and synthetic data for regulated sectors. 📌 https://lnkd.in/eRX2Gfm5 👉 Arize AI Phoenix - End-to-end eval platform for LLMs with features like drift/bias tracking, synthetic data generation and enterprise-ready A/B comparisons. 📌 https://lnkd.in/emBuFNtp #AgentEval #Evaluation #Framework

220

4 Comments

Anthropic wrote about a way to decrease RAG retrieval error rate by 67%. It’s called Contextual Retrieval and it improves RAG by adding meaningful context to each chunk before embedding and indexing with BM25. First, for each chunk of your document, a concise explanation of key details is generated. For example -- In a medical study database, a chunk "Patient A showed symptoms of fatigue" becomes -- "In a 2022 clinical trial, Patient A (Group 1, receiving Drug X) showed symptoms of fatigue." These details situate every single chunk within the whole document. While creating the vectordb, embeddings are calculated on these contextualised chunks. Then, BM25 is also applied to capture exact keyword matches. BM25 is a ranking function that improves text retrieval by measuring word relevance using TF-IDF while also accounting for doct length and saturation effects. At query time, both methods are used to retrieve relevant chunks, which are ranked, deduplicated, reranked and passed to the model. Contextual Retrieval reduces retrieval failures by 49%, and combining it with reranking boosts the reduction to 67%. ♻️ Share it with anyone who might benefit :) I regularly share AI Agents and RAG projects on my newsletter: https://lnkd.in/gaJTcZBR Link to the full article: https://lnkd.in/gubfPfvN #AI #LLMs #RAG

1,374

84 Comments

Just published a video tutorial on how to build and iterate on a Graph RAG pipeline using Kuzu, marimo and DSPy (Community). I've been using all these tools a lot lately, but when used together, I believe they have a lot of synergy with one another, so allow me to elaborate: ✅ When working on an idea like Graph RAG, the starting point is that you have a knowledge graph that you can query (ideally with Cypher in Kuzu 😄). DSPy makes it very simple to compose together modules that help you use LMs to write Cypher. This tutorial covers the key steps. ✅ The traditional way to work with interactive workflows has (thus far), been Jupyter notebooks. However, marimo is paving the way for a new paradigm in Python: the SAME Python file can simultaneously serve as a notebook, a script AND an interactive app! ✅ To retrieve from the graph, a modular Text2Cypher pipeline is built with the following components that are written as a custom DSPy module: 1. Schema pruning (to trim down the graph schema as context for the next step) 2. Text2Cypher (Using an LM to generate Cypher queries) 3. Answer generation (Using an LM to inspect the retrieved context from the graph and answer in natural language). It's a joy to be able to build with a truly modern stack like uv, marimo, Kuzu and DSPy, and to be building composable pipelines that allow a rapid iteration feedback cycle. And the best part: these are ALL open source software! ✅ The DSPy pipeline shown in the tutorial was quite simple (no optimization). It's very clear that even without optimization, DSPy shows a lot of promise to make it easier to work with LMs for any arbitrary tasks. Graph RAG isn't something that needs to be a pre-built, commoditized library - with the flexibility of DSPy's modules, you can build almost anything of any complexity, and interweave your language model calls with deterministic code. I want to go a lot deeper into the abstraction philosophy of DSPy and why I'm excited about it, and am working on a blog post related to the same, so stay tuned! In the meantime, install marimo, fire up a Kuzu database and give this workflow a try! 😄 https://lnkd.in/gX_wN2Dh

89

6 Comments

Who are The Magnificent 7 of Open Source AI? Tracking the explosion of open-source AI repos on Hugging Face over the last year, here is the leading pack: 1. Allen Institute for AI - Ai2 is the non-profit that just keeps shipping OSAI 2. NVIDIA - as noted by Clem Delangue 🤗 recent post, NVIDIA is now a full-stack OSAI player, open-sourcing models and frameworks that fuel the entire ecosystem 3. Meta is still by far the largest contributor by repo count, and not slowing down this year. Congrats Yann LeCun & team for setting the pace for others to follow 4. Hugging Face of course - not just hosting models but building the infrastructure, community, and tools that make open-source AI accessible 5. Qwen (Alibaba Group) - scaling fast, setting new benchmarks, and showing that open-source AI isn’t just a 🇺🇸 game 6. Google - the OG of the LLM revolution - and Google’s open-source AI work is still top-tier 7. Microsoft - enterprise focus & highly committed to on open source AI, with major models like Phi and Orca This list is based on real repository growth and activity on Hugging Face, as visualized by my team at AI World. Full story by Leon Oliver Wolf Francisco Ríos Fierro here: https://lnkd.in/grTEUq7Z Florent Daudens Brigitte Tousignant Thomas Wolf - a lot of material for a 2025 state of open-source AI!

155

8 Comments

Google just open-sourced the LangExtract Python library! It extracts structured data from unstructured documents with precise sources in just a few lines of code. Flexible LLM support (Gemini, OpenAI, Ollama) and 100% Open-source. Link in the comments:

147

7 Comments

🧠 Not all neural networks are the same—each one has its own superpower! Just like different tools in a toolbox, they’re built for specific AI tasks. ➡️ Feedforward Neural Networks (FNNs) – The “OG” of AI 🛠️. Simple, straightforward, and great for basic pattern recognition. 📸 Convolutional Neural Networks (CNNs) – The visionaries 👀. Perfect for image recognition, object detection & facial recognition. 🔄 Recurrent Neural Networks (RNNs) – The storytellers 📖. Ideal for speech, time-series data & sequences—but sometimes forgetful. ⏳ Long Short-Term Memory (LSTMs) – The memory keepers 🗂️. Great for translations, chatbots & remembering long-term patterns. 🎨 Generative Adversarial Networks (GANs) – The artists 🎭. They create hyper-realistic images, art & even deepfakes. ⚡ Transformers – The game-changers 🚀. Powering ChatGPT, translators & AI tools with unmatched speed & accuracy. 💡 Knowing which neural network to use = unlocking AI’s true potential! #NeuralNetworks #DeepLearning #MachineLearning #AI #DataScience #ArtificialIntelligence #TechExplained

263

22 Comments

What is grokking, and why is it important? Grokking is a training phenomenon where a neural network first memorizes the training data and only much later—sometimes after a long plateau—suddenly generalizes well to unseen data. In other words, test accuracy stays low for a long time after the model has already overfit, then jumps abruptly to near-perfect performance. Why does this happen? On small, algorithmic tasks, models often begin by brute-force memorizing examples. With continued training (and usually some regularization such as weight decay), the network gradually builds an internal representation that captures the true structure of the problem. Once this representation becomes strong enough, the memorized patterns fade away, and the model “clicks” into generalization. A well-known example is modular addition (e.g., computing c = (a + b) mod p). Researchers have shown that when a one-layer transformer is trained on this, the model eventually discovers a geometric shortcut: it represents numbers as points on a circle and performs addition as rotation using Fourier features and trigonometric identities. During training, three phases emerge—memorization, circuit formation, and cleanup—culminating in the sudden grokking jump in accuracy. Why it matters: in engineering and scientific domains, many formulas are empirical and miss hidden variables. If we can both train models on rich datasets and open up what they are doing internally, we may uncover elegant, mechanistic solutions that outperform long-standing heuristics. Mechanistic interpretability is the key here: it gives us a way to inspect, validate, and even reuse the algorithms that neural networks invent. #AI #MachineLearning #DeepLearning #Interpretability #Grokking

37

3 Comments