Docling Adds Structured Data Extraction from Documents

Very cool 🚀 — this feels like a must-have tool for any data or AI engineer working with documents in enterprise settings. Extracting structured fields directly from messy PDFs or images without extra conversion is a huge time saver. Excited to see how this evolves! #AI #Docling #DocumentAI #DataEngineering #MLOps #IBMResearch

Good to see the steady progress Docling is making. These are important innovations that would make the use of AI tools run locally far more efficient and with added privacy and confidentiality built-in. #FOSS ftw!

Interesting update from Docling on structured data extraction 🔹 No need to convert to text or JSON first. 🔹 Define free-form schemas and extract exactly the fields you need. 🔹 Looks perfect for messy → structured workflows. I can already see use cases in areas where documents are complex and schema alignment is critical. Definitely worth a look 👇

🔥 Cool update in Docling: Structured Data Extraction. Docking helps with your document AI needs. It’s an opensource project from IBM and has had collaboration from Hugging Face. 👉 Context in JSON out. 🍓 Check out the example here: https://lnkd.in/g32tjatY 🎁 GitHub Repo: https://lnkd.in/gvJTRGEC #LLMs #structuredData #JSON #docling #documentAI #ml

Spent 6 months learning RAG the hard way. Here's the checklist I wish I had from day one: (valid for interviews as well)👇 𝗥𝗲𝗮𝗹𝗶𝘁𝘆 𝗖𝗵𝗲𝗰𝗸: RAG = Information Retrieval + LLM grounding Goal: Fetch closest, useful context to ground your LLM Why: Token limits, fewer hallucinations, private data But here's how it's really built in production (not just OpenAI + vector DB): 1️⃣ 𝗣𝗿𝗼𝗯𝗹𝗲𝗺 𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗶𝗼𝗻 → What's the user problem? → Do we really need RAG? Rules or simple search might work better → What's the feature we're enabling? 2️⃣ 𝗗𝗮𝘁𝗮 𝗔𝘂𝗱𝗶𝘁 & 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀 → Document types, lengths, structure, freshness → Text + images? OCR/vision embeddings needed → Clean? Complete? Extraction required? 3️⃣ 𝗙𝗲𝗮𝘁𝘂𝗿𝗲 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 (𝗖𝗵𝘂𝗻𝗸𝗶𝗻𝗴) → Section-aware chunks (title+content) → Keep fields separate for boosting → Metadata extraction, HTML cleanup, language detection 4️⃣ 𝗕𝗮𝘀𝗲𝗹𝗶𝗻𝗲 & 𝗣𝗿𝗼𝘁𝗼𝘁𝘆𝗽𝗶𝗻𝗴 → Create query ↔ chunk pairs (your ground truth) → Try BM25 (lexical), then vector similarity (semantic) → Score with Recall@k, nDCG, MRR → Start small: get 10-20 examples right 5️⃣ 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗦𝘁𝗿𝗮𝘁𝗲𝗴𝘆 → BM25 → vectors → Hybrid → re-rankers → Tune hyperparameters (k1/b of bm25, top-k, similarity thresholds) → Add business logic, filters, fallback mechanisms 6️⃣ 𝗟𝗟𝗠 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 → Model selection (OpenAI, Claude, Mistral) → Prompt design + guardrails → Handle hallucinations, costs, context limits 7️⃣ 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 (𝗖𝗿𝗶𝘁𝗶𝗰𝗮𝗹!) → Offline: nDCG, Recall@k, latency, cost/query → Online: A/B tests, user feedback, RAGAS scores → Citation correctness, success rates 8️⃣ 𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝗜𝗻𝗳𝗿𝗮 → Vector DB setup (not just local FAISS) → Data pipelines, CI/CD, monitoring → Logs, alerts, dashboards (latency, failure rates) → Schema validation, PII redaction, timeouts 9️⃣ 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸 & 𝗜𝘁𝗲𝗿𝗮𝘁𝗶𝗼𝗻 → Mine real queries + hard negatives → Continuous improvement: embeddings, re-ranking → New data? Repeat preprocessing pipeline → Requirements will change (they always do) 𝗕𝗼𝗻𝘂𝘀 𝗥𝗲𝗮𝗹𝗶𝘁𝗶𝗲𝘀 → Legal/GDPR compliance (always remember this constraint) → Explaining retrieval logic to non-tech teams → Stakeholder analytics requests → Cost optimization under scale 𝗕𝗼𝘁𝘁𝗼𝗺 𝗹𝗶𝗻𝗲: Start lexical, add semantic, fuse, then re-rank. Measure at every step. It's system design, not magic. The process never ends. That's the point. Keep iterating! What's your biggest RAG challenge right now? ♻️ Repost and share to help others on similar journey 😇

Finally, the cursor for data scientists is here. Software development and data science are not the same, most IDEs assume schemas live in your head, not your files. Notebooks break, kernels crash, and messy data derails progress. Zerve AI is an agentic development environment for data scientists. → It generates code, orchestrates compute, and adapts to your workflow. → You stay in control, preview data, edit code, configure compute. It comes with everything you need: • Tracks data and code across every iteration • Scales from one experiment to thousands in parallel • Captures and versions every artifact and result for sharing • Exposes workflows as secure APIs or interfaces • Deploys in the cloud, on-prem, or self-hosted Test → I gave the task: “Summarize the latest developments in LLM training techniques.” Zerve: ▸ Pulled recent research papers and GitHub repo updates ▸ Filtered for relevance automatically ▸ Ran summarization and scheduled it to repeat daily ▸ Used context from past SQL queries, notebooks, and pipelines Go try it yourself, they have a free tier. Link in comments.

LexiFlow Series — Post 1/15 When building LexiFlow, my document intelligence system, I realized something early: The toughest part is not APIs or cloud infra — it’s the data model. In Designing Data-Intensive Applications (DDIA), Martin Kleppmann says: “The most important part of a data system is the data model it exposes.” That hit me hard. In LexiFlow, a PDF is not just a file — it’s a knowledge unit. And that shift in perspective reshaped the whole architecture. Our evolving data model became a layered structure: 🔹 Raw content — bytes, scanned pages, OCR text 🔹 Metadata — title, author, timestamps, compliance tags 🔹 Semantic vectors — embeddings powering retrieval & GenAI chat 🔹 Relationships — cross-document links, version comparisons, traceability Why does this matter? Because every downstream decision flows from it: 1. Ingestion pipelines became smoother (adding new formats is just another data layer, not a redesign). 2. Semantic search scales naturally across thousands of contracts, research reports, or compliance docs. 3. Chat & comparison features feel natural, because they’re grounded in a consistent, extensible data model. Instead of patching features on top, we’re building upwards from the core data model — just as DDIA emphasizes. This reinforced a principle I now carry into every system I design: Get the data model right, and features will follow. Get it wrong, and you’ll fight complexity forever. This is Post #1 in my LexiFlow series — where I connect real engineering choices with timeless system design principles. In the coming days, I’ll share how we applied DDIA’s other foundations (reliability, scalability, maintainability) to LexiFlow’s AWS + FastAPI stack. Next up → Query Models in LexiFlow: balancing flexibility vs performance. Repo: 🔗 https://lnkd.in/dc9ST8YU #GenAI #ArtificialIntelligence #LLMOps #MachineLearning #AIMLProjects #SystemDesign #DataIntensiveApplications #Scalability #Reliability #Maintainability #AWS #Fargate #CloudComputing #DevOps #MLOps #LexiFlow #OpenSource #TechLeadership #Innovation #AIProjects #BhagwatChate

GenAI 90 Days Challenge – Post 14/90 Series: Document Loaders and Text Splitting in LangChain To build retrieval-augmented generation (RAG) pipelines, we need a reliable way to transform raw files, web pages, or APIs into usable formats for LLMs. LangChain provides document loaders and text splitters that standardize content and prepare it for embeddings, retrieval, and prompts. Highlights from Post 14: 1. Document Loaders: Convert diverse formats like PDFs, HTML, JSON, DOCX, emails, and S3-hosted files into standardized Document objects. Each object contains page_content (text) and metadata (file name, URL, page number). 2. Common Loaders: • UnstructuredFileLoader for PDFs, DOCX, PPTX, and emails • BSHTMLLoader for cleaning HTML pages with BeautifulSoup • PyMuPDFLoader for fine-grained PDF page loading • JSONLoader for structured extraction with jq queries • S3FileLoader for enterprise cloud storage ingestion • RecursiveUrlLoader for crawling entire websites 4. Why Text Splitting Matters: LLMs have token limits. Splitting documents into smaller overlapping chunks preserves context, ensures efficiency, and improves retrieval quality. 5. Key Text Splitters: • RecursiveCharacterTextSplitter: Preferred for natural breakpoints • CharacterTextSplitter: Simple and fast for pre-cleaned text • HTMLHeaderTextSplitter: Retains structure from HTML and Markdown headers 6. Chunking Best Practices: Keep 400–1000 tokens per chunk, maintain 10–20% overlap, avoid mid-sentence cuts, and always preserve metadata for traceability. 7. Real-World Pipeline: Load → Split → Embed → Retrieve → Prompt. This sequence is the backbone of production RAG systems. Why it matters? Proper loading and splitting directly impact retrieval accuracy, embedding quality, and final response coherence. Without structured ingestion and thoughtful chunking, even the best LLMs can produce irrelevant or inconsistent outputs. Full details, loader examples, and pipeline code are inside the attached PDF. Next, we continue with Post 15 in the GenAI series. #GenAI #AgenticAI #SystemDesign #LangChain #RAG #90DaysChallenge #BhagwatChate #AI

Big O Notation: do you really get it? 📈 🔎 What is it? Big O is a way to measure the efficiency of an algorithm, showing how it behaves in terms of time and space as the input size grows. ⚡ What is it for? It helps compare algorithms and make better choices: which one is faster, which uses less memory, which one scales better as the data increases. 💡 Why use it? Because in practice, performance matters. A piece of code that works fine with 100 items may crash with 1 million. Big O gives us a mathematical lens to anticipate these scenarios. 👩💻 Who is it for? For software engineers, data scientists, and developers who want to write scalable solutions and make conscious architecture decisions. ⸻ Classic examples of Time Complexity (execution time ⏱️) • O(1) → constant: accessing an item in an array. • O(log n) → logarithmic: binary search. • O(n) → linear: iterating through a list. • O(n log n) → quasi-linear: efficient sorting algorithms like Merge Sort. • O(n²) → quadratic: nested loops, e.g., comparing each element with all others. • O(2ⁿ) → exponential: brute-force recursion, like naive Fibonacci. ⸻ Examples of Space Complexity (memory 💾) • A fixed-size array → O(n) (needs space proportional to the number of elements). • Simple variables → O(1) (constant space). • Recursive algorithms → often include stack usage, which grows with input size. ⸻ ⚡ In short: Big O isn’t just academic theory, it’s a practical tool for building systems that can grow and perform well at scale. 💬 What about you? Can you remember the last time you had to analyze the complexity of an algorithm in your code? #BigO #algorithms #datastructures #softwareengineering #performance #dev