Delta Lake

𝗗𝗲𝗹𝘁𝗮 𝗞𝗲𝗿𝗻𝗲𝗹 𝗣𝗼𝘄𝗲𝗿𝘀 𝗖𝗼𝗻𝗳𝗹𝘂𝗲𝗻𝘁 𝗧𝗮𝗯𝗹𝗲𝗳𝗹𝗼𝘄 🚀 Confluent Tableflow’s Delta Lake support is built on Delta Kernel, the open source library developed by Databricks to make Delta Lake universally accessible across any processing engine. Delta Kernel provides Java and Rust APIs that let developers read, write, and commit to Delta tables directly—without handling low-level Delta protocol details. This simplifies connector development and makes it easy for connectors to adopt the latest Delta innovations, such as catalog-managed commits, VARIANT data type, and type widening. By leveraging Delta Kernel, Tableflow achieves high performance, strong compatibility, and open interoperability while simplifying the path from real-time streaming data to fully governed Delta tables in Databricks Unity Catalog. Specifically, Delta Kernel enables Tableflow to do the following: ✅ Maintain full atomicity, consistency, isolation, and durability (ACID) guarantees and Delta-compliant transaction logs for reliable table updates ✅ Write at scale, efficiently materializing Kafka topics into optimized Parquet data and metadata commits ✅ Stay future-proof with transparent adoption of new Delta Lake features and protocol versions 🔗 Learn more: https://lnkd.in/eyrzK7Yu #opensource #deltalake #deltakernel #tableflow #confluent

Missed the recent Open Lakehouse + AI webinar with Jacopo Tagliabue and Ciro Greco from bauplan? 🎥 Watch the full discussion: https://lnkd.in/eR6ragtJ They break down: 🔹 How a simple, uniform layer cuts through today’s fragmented data stacks. 🔹 Why function-based execution, Git‑for‑Data, and programmable abstractions make lakehouses agent‑ready. 🔹 What you get: safer, deterministic, reproducible ops for AI agents. #opensource #oss #openlakehouse #ai #aiagents

Discover how deletion vectors can take Delta Lake performance to the next level. 🚀 In this 5-minute video, Youssef Mrini breaks down how deletion vectors speed up delete, update, and merge operations by marking rows logically—no need to rewrite entire files. He also shares real-world merge benchmarks and shows how this feature enhances Change Data Capture and streaming workloads. 𝗧𝗼𝗽𝗶𝗰𝘀 𝗰𝗼𝘃𝗲𝗿𝗲𝗱: 🔹 What deletion vectors are and how they work 🔹 Merge-on-read performance improvements 🔹 How to enable deletion vectors in Python and SQL 🔹 When to use deletion vectors for large datasets 🎥 Watch: https://lnkd.in/eHnuvxfT #deltalake #opensource #oss #deletionvectors #python #SQL

🚀 Massive data, mission‑critical pipelines, zero excuses. “Scaling Apache Spark at OpenAI” with Chao Sun reveals how to make Apache Spark™ reliable at scale. You’ll see: 🔹 Production patterns for cluster management, job submission, and access control. 🔹 Dynamic scaling strategies to handle spiky, multi-tenant workloads. 🔹 How Apache Spark™ + Delta Lake underpin large-scale pipelines—and the lessons learned keeping them fast and dependable. Two tracks. One epic afternoon. Open Lakehouse + AI Mini Summit. 🙌 📍 Mountain View 🗓️ Nov 13 🔗 Register → https://lu.ma/OLMS-1113 #OpenLakehouse #ApacheSpark #DeltaLake #DataInfra #BigData #DataOps

In this hands-on walkthrough, Youssef Mrini (Databricks) shows you how to handle special characters, rename columns, and safely modify schemas in Delta Lake! 🤝 By leveraging column mapping, you’ll learn how to make Delta tables easier to evolve, support more flexible schema operations, and prepare for advanced capabilities like Liquid Clustering. See how to: 🔹 Understand what column mapping does and when to enable it 🔹 Safely rename and drop columns 🔹 Manage schema evolution for dynamic data sources 🔹 Configure your table with delta.columnMapping.mode = 'name' 🎥 Get started: https://lnkd.in/e8nkH7sX #opensource #oss #deltalake #columnmapping

🚨 THIS WEEK! The data world doesn’t need more lineage graphs — it needs more understanding. 🤝 Join us on 𝗧𝗵𝘂𝗿𝘀𝗱𝗮𝘆, 𝗢𝗰𝘁 𝟯𝟬 𝗮𝘁 𝟵𝗔𝗠 𝗣𝗧 for our next Open Lakehouse + AI webinar, where we’ll explore why “total visibility” often creates noise, not clarity. Discover how oleander is redefining observability by unifying OpenLineage, OpenTelemetry, and #LLM reasoning into one continuous feedback loop — connecting data movement, system behavior, and AI-driven insights. 🔗 Save your spot: https://lnkd.in/eihgBzdV 🎤 Speaker: Willy Lulciuc, hosted by Lisa N. Cao #openlakehouse #ai #oss #opensource #openlineage #opentelemetry

We’re diving deep into Flight SQL + Delta on November 11 at 9AM PT to show how open technologies like Apache Arrow Flight, #DataFusion, and Delta Lake are reshaping high-performance analytics. 🚀 See how: 🔹 Flight SQL provides a super-fast semantic layer for SQL-over-Flight 🔹 Delta operations like VACUUM and OPTIMIZE become seamless extensions of streaming compute Get ready to see data exchange the way it should be — fast, distributed, and open. 🗓️ November 11 🕝 9:00 AM PT 🔗 Register: https://lnkd.in/e_8T5zRp #apachearrow #opensource #oss #flightsql

🚀 From data to AI—without the governance headaches. Don't miss “From Data to AI: Leveraging Unity Catalog to Train at Scale” with Aniruth N. (Databricks) at Open Lakehouse + AI Mini Summit in Mountain View on Nov 13! 🗓️ Discover how a unified governance plane streamlines security, lineage, and scale for multimodal AI—across tabular data, documents, and images. 🎯 What you’ll learn: unify open table formats like #DeltaLake and Apache Iceberg, govern unstructured data with Volumes, and move from governed data to production models using MLflow + Unity Catalog’s model registry. 🧩 Two exciting tracks, one epic afternoon. 😎 🔗 Register → https://luma.com/OLMS-1113 #opensource #oss #unitycatalog #mlflow #apacheiceberg #openlakehouse #ai

Hard deletes break incremental loads. Delta Lake’s Change Data Feed captures inserts, updates, and deletes — including those produced by MERGE — so downstream streams can maintain consistency. In this clip, Bartosz Konieczny shows how CDF simplifies merge/update logic when rows disappear from the base table. 🎥 Watch the full webinar: https://lnkd.in/ePTaYhj6 #DeltaLake #ChangeDataFeed #OpenSource