Hamza Hachani

🚀 New in Docling: Structured Data Extraction from Documents! 🚀 We’ve just added a brand-new functionality: extraction of structured data from complex documents using free-form schemas 🤩 What does that mean? 🔹 You can now skip the conversion step — instead of turning documents into text or JSON first, Docling directly extracts the structured fields you care about. 🔹 The requested fields are defined in a free-form schema, so you can instantly align the extraction with the schemas of your own databases. 🔹 This makes it ideal for data-pipelines that don’t need full document conversion, but do need to populate structured databases from messy, unstructured documents. ✨ It’s: 1️⃣ Super simple to use (check out the code snippet 👉 https://lnkd.in/eK6vaMEe ) 2️⃣ 100% open-source and runs fully local – no API calls needed 🙌 3️⃣ Powered by cutting-edge models from NuMind (YC S22) 4️⃣ Perfect for data-pipelines where you need to populate structured databases from documents (think invoices, Curriculum Vitae, contracts, product datasheets, etc) 5️⃣ Currently focused on PDFs and images (PNG) — support for pure text coming soon! The example below shows how easy it is to define a schema and extract structured fields directly from an invoice. 👉 Try it out, break it, send us feedback, and if you like what we’re building, don’t forget to ⭐ the repo (https://lnkd.in/d4UT-6_2)! After a long and fruitful summer, the Docling team has been cooking up many new features — and this is just the beginning! 🌟 #opensource #AI #documentAI #docling #IBM #IBMResearch

1 382

83 commentaires

Kafka vs. Spark vs. Flink: which one should you use? My zero fluff, quick breakdown 👇 When building a data platform, choosing the right tool isn’t just a technical decision but also a strategic one. Because each framework serves a different purpose. ➊ Kafka = Real-time messaging at scale Perfect to reliably stream blazing-fast pipelines. ✅ Very low latency ✅ High throughput ✅ Durable ⚠️ But… limited processing power. You’ll need other tools for transformations. ➋ Spark = Batch + Micro-batch king Ideal for crunching massive datasets or doing batch ETL. ✅ Excellent for batch pipelines ✅ Unified engine for SQL, ML and graphs ✅ Mature ecosystem + tooling ⚠️ Watch out for higher streaming latency—it’s not true real-time. ➌ Flink = Event-by-event precision Built for complex, stateful stream processing. ✅ True real-time (no batching) ✅ Exactly-once process guarantees ✅ Handles late data and out-of-order events natively ⚠️ Steeper learning curve & fewer high-level APIs use. ~~~ So, instead of wondering which tool is better, ask yourself what does your use case demand. Is it latency? Scalability? Flexibility? Governance? Always map your needs to the right engine and your data stack will thank you. PS. Curious to know which one you use the most and why #dataengineering #infographics

295

14 commentaires

📌 For all data folks — engineers, scientists, analysts — this is a great read to take a step back on the limits of current AI coding tools in our field. 👉 All you need is context. But without data & AI context, you’ll go nowhere. Ido Bronstein article does a great job explaining — with concrete examples — why today’s AI-generated SQL and pipelines often look right but work wrong. ⚠️The problem: AI doesn’t know your data. It can generate syntactically correct queries, but without context it creates fragile code, duplicates, broken filters, and endless debugging. ✅The missing context: To really support data teams, AI needs to understand and formalize: 1️⃣ Semantics – what each field, table, or entity means 2️⃣ Lineage – how data flows and where dependencies exist 3️⃣ Usage patterns – how teams consume and interpret the same data 4️⃣ Constraints & rules – what must always hold true That’s the next big challenge: bringing this hidden, tribal knowledge into a structured “data context” that AI can reason with. And this resonates strongly with us at Mirakl: it’s exactly the kind of learnings and hurdles we had to go through to crack text-to-analytics, together with Valentin Binjacar, Samuel Baker, and Marius H-F 🔗Link to the article in comments ⬇️

62

10 commentaires

What I love about #ApacheIceberg is how many different workloads folks are running on the same copy of data. Graph queries are a really cool example. Instead of copying data into specialized graph databases, PuppyGraph lets you run graph traversals directly on your existing Iceberg tables. Pretty cool stuff. But here's the thing - to get truly great performance, you need to ensure your Iceberg tables are well maintained and optimized. That's where Ryft comes in. We make sure your Iceberg tables are automatically maintained and optimized, no matter what engine you are using. It was terrific to work with Zhenni Wu and Weimo Liu, to create this blog post on how you can use Ryft with PuppyGraph to run graph queries on a fully optimized Iceberg lake! https://lnkd.in/giwnwMuc #ApacheIceberg #GraphQueries

99

2 commentaires

Steps here👇the game has completely changed since 2022 - it’s not just leetcode anymore. Here’s what Viktoria Semaan shared that actually worked: ✅ Actually USE the platform (yes, she had to learn Databricks + Spark from scratch) ✅ Nail the coding assignment without ChatGPT crutches ✅ Come with CLEAR impact stories for leadership rounds ✅ Do your homework on every interviewer (LinkedIn stalking is real prep 😅) The technical bar is higher, but the human connection piece is everything. When you can tie your stories to their experience and show genuine interest in their work - that’s when you stand out. 2025 interviews hit different. Preparation is everything 💪 What’s the biggest interview challenge you’re facing right now? Drop it in the comments 👇 #techjobs #interviewtips

187

5 commentaires

🔙 Throwback to one of my favorite articles! Some time back, I wrote a detailed piece on Optimizing Big Data with Apache Iceberg in Databricks – and it’s still one of my most-read Medium articles. In this article, I cover: - How to set up your Databricks environment for Apache Iceberg - Deep dive into the table structure of Iceberg - Practical insights on why Iceberg is a game-changer for Big Data optimization This was part of my Apache Iceberg x Databricks series, which also included: Understanding Apache Iceberg Understanding the Table Structure of Apache Iceberg Iceberg Series 4: Setting up Databricks Environment If you missed it earlier, here’s your chance to check it out 👇 📖 - https://lnkd.in/dpkaAKTm - https://lnkd.in/d6nQBNZn - https://lnkd.in/dErW5rjP - https://lnkd.in/d2BuSxq3 I’d love to hear — how are you leveraging Apache Iceberg in your data platform today? #Databricks #ApacheIceberg #BigData #DataEngineering #DataLakehouse #Spark #DataOptimization #MediumArticle #Throwback

57

4 commentaires

Excited to see OpenAI's new gpt-oss. Enterprises don’t want to send sensitive data to a closed API/black-box model. The real value here is that you can bring open-source models to your data — not the other way around. With Databricks and OSS, you keep control: ✅ Your data stays in your environment ✅ You choose the model ✅ You maintain governance and security It’s a smarter way to build AI. Give it a try on Databricks: https://lnkd.in/gDsrdBhY #GenAI #OSS #EnterpriseAI #Databricks #CIO #CDO #AIstrategy

62

2 commentaires

#OpenSource is just like open science: shared, collaborative, and the foundation of progress. 🚀 At PyData Paris 2025, I had the privilege of moderating the panel “Open-source Business” with Yann Lechelle (CEO, :probabl., former CEO of Scaleway) and Sylvain Corlay (CEO, QuantStack) at the Cité des sciences et de l'industrie. Both leaders are deeply embedded in projects like scikit-learn and Project Jupyter—the invisible infrastructure used daily by millions from scientists to business analysts. Our conversation unpacked what it really takes to grow sustainable businesses on top of open source.: • No one-size-fits-all: Every project needs its own model; hybrids are the norm. • Community = momentum. Every user is an advocate. Marketing spend shrinks when you invest in shared value • Safeguard the team: Visibility comes with risks; support and protection are non-negotiable. • Diversify funding. From VC to contracts to public investment (like France’s support of scikit-learn), sustainability depends on many levers. • Play the long game. The most critical projects are often the least glamorous. Openness as strategy. Beyond code: open science, data, source, weights, standards, hardware. A European playbook for digital sovereignty—with France and Germany catalyzing all 27 EU countries 🇪🇺 🗽. Sustainable open-source businesses balance community values with commercial viability—and “openness” is not ideology; it’s an industrial strategy that lets society understand and control its destiny. Thanks to Yann and Sylvain for a candid, deeply practical discussion. Looking forward to Yann's book "ouvertarisme.fr" that talks about how openness (from open science to open source to open models) can become an industrial policy. In fact, it has become one. We need to seize the occasion without falling into cynicism nor "open washing”.

77

3 commentaires

Delta Lake 4.0 now supports Variant data type designed for working with semi-structured data like JSON with evolving schemas! Key capabilities include: 🔹 𝗦𝗰𝗵𝗲𝗺𝗮-𝗼𝗻-𝗿𝗲𝗮𝗱: No need to define a fixed schema in advance. Structured data can be extracted at query time. 🔹 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝘀𝘁𝗼𝗿𝗮𝗴𝗲: Dynamic payloads are stored using high-performance encoding for better storage efficiency. 🔹 𝗦𝗵𝗿𝗲𝗱𝗱𝗲𝗱 𝗩𝗮𝗿𝗶𝗮𝗻𝘁𝘀 (𝗣𝗿𝗲𝘃𝗶𝗲𝘄): Allows extraction of specific columns from variant data and collection of statistics to speed up queries and file skipping. This feature was developed collaboratively with contributions from Apache Parquet, Apache Arrow, Apache Iceberg, #DeltaLake, Apache Spark, and others, supporting integration across the broader data ecosystem. 🔗 For more details, see the full release notes: https://lnkd.in/edemXCS5 #opensource #oss #linuxfoundation #variantdata

223

6 commentaires

What is DLT in Databricks? If you’ve been working with Databricks, you might have come across the term DLT (Delta Live Tables). But what exactly is it, and why do data engineers love it? 🔹 What is DLT? DLT is a framework in Databricks that simplifies the creation, deployment, and management of data pipelines. It allows you to define transformations using SQL or Python and ensures your pipelines are reliable, scalable, and automatically managed. 🔹 Where is it used? DLT is widely used in scenarios like: ✅ Batch & streaming pipelines – Process real-time data along with historical data. ✅ Data quality enforcement – Apply expectations to ensure clean and trustworthy data. ✅ CDC (Change Data Capture) – Manage incremental changes without complex coding. ✅ ETL/ELT workloads – Build modern, declarative pipelines for analytics and ML. ✅ Governance & monitoring – Track lineage, handle schema evolution, and monitor jobs easily. In short, DLT helps data teams move faster while ensuring pipelines are production-ready from day one. 👉 Like to show your support. 💬 Comment with your thoughts or questions. 👥 Follow Shilpa Das to get such more like this. #databricks #DLT #deltalivetables

67

3 commentaires