Introducing SCORE: A New Evaluation Framework for Document Parsing

🚀 Excited to share our latest research from our Unstructured team! We're proud to introduce SCORE: A Semantic Evaluation Framework for Generative Document Parsing – addressing blind spots in how we evaluate AI systems. Traditional document parsing metrics have a fundamental flaw – they penalize systems for producing semantically correct outputs that happen to be structurally different. This creates evaluation blind spots as generative AI becomes dominant. Our Solution – SCORE's Four Key Innovations: ✅ Adjusted edit distance that tolerates valid reorganization ✅ Token-level diagnostics separating hallucinations from omissions ✅ Spatially aware table evaluation ✅ Hierarchy-aware consistency checks Key Findings from 1,114 pages analyzed: 📊 2-5% of documents showed alternative but valid interpretations that traditional metrics unfairly penalized by 12-25% 🔍 Surprising Discovery: GPT-5 Mini actually outperforms Gemini 2.5 Flash in semantic accuracy (0.896 vs 0.895 adjusted NED) despite appearing weaker under traditional metrics. 💡 GPT-5 Mini consistently shows the lowest hallucination rates across datasets – completely masked by conventional evaluation methods. Why This Matters: As generative models become dominant for document processing, we need evaluation frameworks that capture semantic correctness. SCORE establishes new principles for semantically grounded benchmarking that the field desperately needs. 📄 Read the full paper: https://lnkd.in/gB_3nifC

Evals of unstructured document processing is the most important problem for many enterprise ingest pipelines, and it is not an easy one. At Unstructured, we remain hyper focused on this, continually improving our eval data and framework which translates into better transformation capabilities in the Unstructured Platform. On a personal note, it's been such an honor to work with the best minds in document understanding and transformation in the industry, and I'm really excited for what comes next. For anyone working in this space, I strongly recommend reading the paper!

𝗦𝘁𝗼𝗽 𝗰𝗮𝗹𝗹𝗶𝗻𝗴 𝗲𝘃𝗲𝗿𝘆𝘁𝗵𝗶𝗻𝗴 "𝗽𝗿𝗼𝗺𝗽𝘁 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴." The real skill that separates functional AI systems from production-ready ones? 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴. I've been watching teams struggle with this exact problem. They obsess over prompt wording while their agents fail because they're drowning in irrelevant information or starving for the right context. Here's what changed my thinking: 🔗 [https://lnkd.in/gJFkN5GX] Andrey Karpathy nailed it recently—people think of prompts as short task descriptions. But in every industrial-strength LLM application, the real work is the delicate art of filling the context window with exactly the right information for the next step. This isn't just semantic wordplay. It's a fundamental shift in how we architect AI systems. 𝗪𝗵𝗮𝘁 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗺𝗮𝗸𝗲𝘀 𝘂𝗽 "𝗰𝗼𝗻𝘁𝗲𝘅𝘁"? Most people think it's just the user's question and maybe some retrieved documents. But your LLM is processing far more: → System instructions and user input (the obvious ones) → Short and long-term memory from conversations → Retrieved information from knowledge bases → Tool definitions and their responses → Structured output schemas → Global state across agent steps Every single one of these elements is competing for limited context window space. 𝗧𝗵𝗲 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗰 𝗶𝗺𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀: When you frame the challenge as "context engineering," you start asking better questions: Are we selecting the right knowledge base or tool before we even retrieve? Should we compress or rank retrieved information by relevance? Do we need all the chat history, or just extracted facts? Could structured outputs give us condensed context instead of walls of text? Are we orchestrating LLM calls in a sequence that optimizes context at each step? 𝗛𝗲𝗿𝗲'𝘀 𝘄𝗵𝗮𝘁 𝗜'𝗺 𝘀𝗲𝗲𝗶𝗻𝗴 𝘄𝗼𝗿𝗸: The teams that get this right aren't throwing everything at the LLM and hoping for the best. They're building explicit workflows that break complex tasks into focused steps—each with its own optimized context window. They're using structured extraction to condense long documents into relevant data points. They're implementing memory blocks that retrieve only what matters. They're ranking and filtering before adding to context. In other words: they're treating the context window as a scarce resource that requires intentional design. 𝚃̲𝚑̲𝚎̲ ̲𝚋̲𝚘̲𝚝̲𝚝̲𝚘̲𝚖̲ ̲𝚕̲𝚒̲𝚗̲𝚎̲:̲ If your AI agent is underperforming, the problem probably isn't your prompt. It's that you haven't engineered your context strategy. Think about it: you're building workflows whether you realize it or not. The question is whether you're building them intentionally. What's your biggest challenge with context management in your AI systems? #AIEngineering #MachineLearning #ArtificialIntelligence #LLM #GenerativeAI #PromptEngineering #AIAgents #MLOps #AIStrategy

#Recent Paper release: Beyond Prompt Engineering, ACE - A Framework for Evolving, Self-Improving LLM Contexts Traditional methods for adapting LLMs through context (prompts, memory) are hitting fundamental limits: brevity bias and context collapse. Techniques like GEPA or monolithic rewriting often compress away crucial domain heuristics or degrade into uninformative summaries, severely limiting agents and knowledge-intensive applications. New paper (published in Cornell) introduces ACE (#Agentic Context Engineering), a framework that treats context not as a static prompt, but as a dynamic, evolving playbook. ACE introduces a structured, agentic workflow to solve these issues: Architectural Core: A Modular Triad ACE decomposes context adaptation into three specialized roles, preventing the overload of a single model: - Generator: Produces reasoning trajectories for new queries. - Reflector: Critiques traces to distill concrete insights and error root causes from execution feedback (e.g., unit test results, API errors). - Curator: Synthesizes insights into compact delta entries and integrates them into the existing context via deterministic, non-LLM logic. Key Technical Innovations: Incremental Delta Updates: Instead of costly full-context rewrites, ACE produces small sets of candidate "bullets" (structured knowledge units with metadata). This enables localized edits, fine-grained retrieval, and efficient merging, drastically reducing latency and cost. Grow-and-Refine Mechanism: New insights are appended as new bullets, while existing ones are updated in-place. A semantic de-duplication step prunes redundancy, ensuring the context remains compact and relevant as it expands. 📈 Empirical Results: Evaluated on AppWorld (agent benchmark) and financial reasoning tasks (FINER, Formula), ACE consistently outperforms strong baselines (GEPA, MIPROv2, Dynamic Cheatsheet): +10.6% average improvement on agent tasks. +8.6% on domain-specific (financial) benchmarks. Achieved 86.9% lower adaptation latency and significantly fewer rollouts than GEPA. Critically, ACE matches the top-ranked production agent (IBM CUGA on GPT-4.1) on the AppWorld leaderboard using a smaller open-source model (DeepSeek-V3.1), demonstrating the power of context over sheer model scale. Implications: ACE provides a scalable and efficient pathway for online and continuous learning, offering a flexible alternative to weight fine-tuning. The human-interpretable nature of the curated playbook also opens the door to more manageable selective unlearning and responsible AI governance. This work underscores that for complex reasoning tasks, comprehensive, evolving contexts are more effective than compressed summaries, and that a structured, multi-agent approach is key to building robust, self-improving LLM systems. Paper: https://lnkd.in/gj_sWNDr #LLM #AIResearch #MachineLearning #DataScience #PromptEngineering #AIAgents #LLMOps #SelfImprovingAI #TechDeepDive

AI Tip: Write, Select, Compress, Isolate — The Four Pillars of Context Engineering 📝🔍 The secret to scalable, high-quality AI isn’t “more context” — it’s better context. Top systems rely on four core operations to manage what goes into the window. 🧰 The Four Pillars 1️⃣ Write Craft or reformulate context pieces (e.g. system instructions, few-shot examples, templates) tailored to each task. These are your building blocks. 2️⃣ Select From memory, documents, tool outputs, or history — pick only the most relevant. Use ranking, similarity, heuristics. 3️⃣ Compress Summarize or condense long texts so they fit — e.g. use embeddings, summaries, chunking, or skeletons. 4️⃣ Isolate Prevent interference by isolating unrelated context (e.g. domain separation, context partitioning, “slots” for tool output vs memory vs user profile). > “Context engineering is the art and science of filling the context window with just the right information at each step of an agent’s trajectory.” — from LangChain’s blog on context strategies 🔄 Why This Matters ▫️Without selection, your context gets noisy and distracts the model. ▫️Without compression, you blow your token budget on verbosity. ▫️Without isolation, different context types can conflict and confuse the model’s reasoning. ▫️Without writing, you lack structure and consistency across tasks. These operations turn context from a “dump everything and hope for the best” approach into a disciplined, architected system. ✅ Try This Today ▫️For one of your agent paths, implement all four operations (write, select, compress, isolate). ▫️Track how performance (accuracy, coherence, latency) shifts as you add or remove each step. ▫️Inspect “wrong answers” and ask: was something selected incorrectly? Or compressed too much? ▫️Iterate on your compression thresholds and separation rules to strike balance. 📚 🔗 https://lnkd.in/gr6qMdv3 🔗 https://lnkd.in/g4NzSFg8 🔗 https://lnkd.in/gxfspsSK 🔗 https://lnkd.in/g72PZCXZ 👇 What’s the trickiest of these four pillars for you — selecting, compressing, or isolating? #AI #ContextEngineering #AITip #PromptVsContext #Memory #Retrieval #Compression #AgenticAI #AIArchitecture #Innovation

𝗪𝗵𝘆 “𝗰𝗼𝗻𝘁𝗲𝘅𝘁 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴” 𝗶𝘀 𝗯𝗲𝗰𝗼𝗺𝗶𝗻𝗴 𝘁𝗵𝗲 𝗻𝗲𝘄 𝗳𝗿𝗼𝗻𝘁𝗶𝗲𝗿 𝗶𝗻 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁 𝗱𝗲𝘀𝗶𝗴𝗻 (And how does it differ from prompt engineering?) Anthropic’s latest post "Effective Context Engineering for AI Agents" is a must-read for anyone building AI products. Here’s what stood out (and what I am reflecting on): + 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗶𝘀 𝗮 𝗳𝗶𝗻𝗶𝘁𝗲, 𝗽𝗿𝗲𝗰𝗶𝗼𝘂𝘀 𝗿𝗲𝘀𝗼𝘂𝗿𝗰𝗲 Large language models (LLMs) have limits. As you add more tokens, their ability to "attend" effectively degrades (a phenomenon dubbed “context rot”). This means every token in your agent’s context should pull weight. + 𝗣𝗿𝗼𝗺𝗽𝘁 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗲𝘃𝗼𝗹𝘃𝗲𝘀 𝗶𝗻𝘁𝗼 𝗰𝗼𝗻𝘁𝗲𝘅𝘁 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 Rather than merely crafting the perfect prompt, the art is in curating the set of tokens (system instructions, tools, external data, history) that you allow the model to see at each step. + 𝗝𝘂𝘀𝘁-𝗶𝗻-𝘁𝗶𝗺𝗲 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝘃𝘀. 𝘂𝗽𝗳𝗿𝗼𝗻𝘁 𝗹𝗼𝗮𝗱𝗶𝗻𝗴 Instead of dumping all data into context from the start, let the agent dynamically fetch what’s needed via tool calls or metadata pointers. This is both more scalable and more focused. 𝗧𝗲𝗰𝗵𝗻𝗶𝗾𝘂𝗲𝘀 𝘁𝗼 𝗲𝘅𝘁𝗲𝗻𝗱 𝗮𝗴𝗲𝗻𝘁 𝗰𝗮𝗽𝗮𝗯𝗶𝗹𝗶𝘁𝗶𝗲𝘀 𝗼𝘃𝗲𝗿 𝗹𝗼𝗻𝗴 𝗵𝗼𝗿𝗶𝘇𝗼𝗻𝘀 + 𝗖𝗼𝗺𝗽𝗮𝗰𝘁𝗶𝗼𝗻 | summarizing history and restarting a new context window without losing critical information.  + 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱 𝗻𝗼𝘁𝗲-𝘁𝗮𝗸𝗶𝗻𝗴 / 𝗺𝗲𝗺𝗼𝗿𝘆 | persist essential information outside the immediate context window and recall it selectively. + 𝗠𝘂𝗹𝘁𝗶-𝗮𝗴𝗲𝗻𝘁 / 𝘀𝘂𝗯-𝗮𝗴𝗲𝗻𝘁 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲𝘀 | delegate specialized tasks to smaller agents and only bring back distilled summaries.  𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 & 𝗶𝗺𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 (𝗳𝗼𝗿 𝘁𝗲𝗮𝗺𝘀 & 𝗽𝗿𝗮𝗰𝘁𝗶𝘁𝗶𝗼𝗻𝗲𝗿𝘀) + When building conversational agents or long-running workflows, aggressively weed out low-value context. Strive for lean, high-signal context design. + Use compaction and memory strategies to support coherence over sessions. + Design tool APIs with token efficiency in mind. The agent’s inputs and responses should be compact but expressive. + Embrace hybrid strategies: preload critical context, but rely on runtime exploration for depth. + When your agent or task becomes too complex for a single context window, partition responsibilities into sub-agents. 𝗪𝗵𝘆 𝗶𝘁 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 The next competitive edge won’t be increasingly clever prompts. It’ll be how we orchestrate information flow so agents stay focused, efficient, and useful over long horizons. Curious how others are approaching this: Are you already treating context like a product design challenge? What’s worked (or failed) for you in keeping agents coherent over long sessions? #AI #LLM #AIagents #ProductThinking #ContextEngineering

Context Engineering for AI Agents: The Discipline of Forgetting Well Instead of throwing more memory at LLMs, let me share how to engineer context: when to trim, summarize, prevent drift, and defend against context poisoning. The real challenge isn’t just remembering more, but remembering smarter. Context engineering is about designing the rules of forgetting so AI agents stay reliable, coherent, and trustworthy. Sharing 10 principles, expanded with nuance from my practice: 1. Memory per task matters: A reasoning agent benefits from trimmed, high-signal windows; a planning agent thrives on long, structured summaries. The context strategy should follow the workload. 2. Trim by intent, not brute force: Always cut entire conversational turns. Mid-message trimming destroys semantic integrity and destabilizes reasoning. 3. Smarter summaries: Summaries should be structured into environment, steps, blockers, decisions. Order matters—contradictions and overwrites should be flagged automatically. 4. Context budgets: Treat tokens as a scarce resource. Define max_turns relative to the task and pin the latest N verbatim. This balances recency with continuity. 5. Async summarization: Summarization shouldn’t block agent flow. Re-check after operations complete to prevent drift between intent and execution. 6. Metadata hygiene: Debug logs, timestamps, or system chatter can pollute agent reasoning. Strip aggressively, keep only semantic payload. 7. Idempotency: Repeated calls must yield stable summaries. Deduplicate aggressively to avoid compounding noise. 8. Progressive summarization: Compress older layers, but mark boundaries clearly. This creates a time-layered memory that resists hallucination. 9. Evaluation harnesses: Don’t guess—replay transcripts through an LLM-as-judge, scoring for coherence, accuracy, and loss of critical detail. 10. Context poisoning defense: Monitor for false info injection. Track token counts pre/post and verify semantic shifts. Context poisoning is subtle, cumulative, and fatal if ignored. Context evolves over time it drifts, contradicts, and decays. A robust context engineering stack must therefore integrate freshness checks, conflict resolution policies, and human-in-the-loop safeguards for high-stakes domains. The real power of context engineering lies in mastering the art of dynamic memory—what to keep, what to compress, and what to erase—so AI agents can think with clarity, act with trust, and scale with resilience. For a deeper dive into these principles, see my blog on Context Engineering: https://lnkd.in/eBzUcefM

1️⃣ An interesting article about context engineering from Anthropic, which uses its own software Claude Code to explain how they conduct context engineering for various scenarios. 2️⃣ Prompt engineering focuses on crafting inputs to guide LLMs, but context engineering is about curating the entire information ecosystem an agent needs—think system instructions, tools, examples, and dynamic memory. This ecosystem lets agents do what prompts alone can’t: run tool-calling loops to iterate on tasks, recover from errors, and tackle multi-step work (like debugging code or analyzing large datasets). 3️⃣ Anthropic highlights 3 tactics for context management: 🎯 Embedding searches for lightweight, just-in-time access to history (no bulky static memory). 🎯 Claude’s code-driven workflow: Querying big databases to pull only relevant data chunks (e.g., head/tail sections) instead of flooding the context window. 🎯 Metadata smarts: Agents use file details (name, size, format) to decide when/which files to use—mimicking how humans prioritize resources. 4️⃣ For long-horizon tasks three strategies shine, and Anthropic spells out when to use each: 🎯 Context compression (summarizing key points into new context) for multi-conversation tasks. 🎯 Structured external memory (task lists, relationship notes) for step-by-step milestones (prevents goal drift). 🎯 Multi-agent systems (segregating context across specialized agents) for parallel deep work (no single agent needs to “remember everything”). 5️⃣ The takeaway? Great AI agents aren’t built on one perfect prompt—they’re built on intentional context design. Every step needs careful choices about which info to feed the LLM: the most informative, yet concise, inputs win. 6️⃣ Article Link: https://lnkd.in/gM4JA8qX

We don’t just talk about prompt engineering anymore. It is now about context engineering. Anthropic’s latest blog on the topic highlights principles I have already been applying in practice. The idea is simple: LLMs don’t just need good prompts, they need the right context. What you feed them, from system prompts to examples, metadata, and history, determines how useful the output will be. This is something I have seen first-hand. After years working on how people manage and curate data, the same rules apply to LLMs. Good data and clear metadata lead to better results. A few points from the article that stood out to me: • LLMs are only as good as the slice of information you give them. • Metadata such as file names, paths, and timestamps provides important signals, just as it does for humans. • Context is a finite resource. If you overload an LLM with too much raw data, it begins to lose track of what matters. • Sub-agents help manage complexity by focusing on narrow tasks while a main agent coordinates. That last point is something I have been experimenting with. I built a Product Management agent that manages the work of other agents, one coding agent and two code review agents. It feels less like chatting with a single model and more like working with a small AI team. My takeaway: do not throw raw data at an LLM. Curate it, structure it, and think about context the same way you would when working with people. https://lnkd.in/eq7Mrv7v

📄 Meet Granite-Docling: Smarter Document Conversion with AI Working with complex documents has always been a challenge. Tables, code blocks, equations, and even different languages often get lost in conversion, slowing down workflows and hurting accuracy. That’s why the release of Granite-Docling-258M is so exciting. It’s an open-source, enterprise-ready vision-language model designed to convert documents into machine-readable formats while keeping everything intact: layout, tables, lists, equations, and more. ✨ Why it matters: - Preserves structure, no more broken tables or messy formatting - Compact & efficient, powerful performance in a small, cost-effective model - Multimodal, understands both text and images in documents - Multilingual, early support for Arabic, Chinese, and Japanese ⚡ How it can be used: Granite-Docling is perfect for turning documents into clean, structured data that’s ready for downstream AI use cases, like: - RAG systems that rely on precise document context - Enterprise search where accuracy is critical - Data preparation for fine-tuning LLMs - Agentic AI workflows where documents need to be read like an expert would 🔗 Explore Granite-Docling: - https://lnkd.in/ebDnaaJk - https://lnkd.in/exvWxQ7T At SIA Innovations Inc., we are building advanced enterprise solutions powered by Agentic AI and models, making document intelligence more accurate, scalable, and accessible. #GraniteDocling #DocumentAI #OpenSourceAI #EnterpriseAI #FutureOfWork