Tiger Lake Public Beta: Postgres-Lakehouse Bridge for Real-Time Analytics

Unifying database and lakehouse data. We announced our baselake architecture the day before Databricks announced lakebase. Names aside, the underlying idea resonates with many larger customers. Databases and lakehouses serve different use cases, different teams, and different needs. But both benefit from seamless data movement and a shared foundation. Start building with Tiger Cloud today 🚀

Databricks recently published a great breakdown of some of the most common (and expensive) S3 storage mistakes-and how to fix them fast. Key takeaways: • Versioning pitfalls that quietly inflate costs • Lifecycle policies that look right but work against you • Hidden egress fees from NAT gateways • Practical steps to clean up buckets and cut spend without hurting performance If you care about FinOps, data platform efficiency, or just avoiding surprise AWS bills, this one’s worth 5 minutes. 👉 https://lnkd.in/e6fZ7ATG #DataEngineering #DeltaLake #Databricks #FinOps #AWS #S3 #Lakehouse

Running AI systems on premises is becoming practical again, especially for teams dealing with latency, data control, and rising cloud costs. This write-up outlines an on-premises architecture in which embeddings, vector search, and RAG run directly within PostgreSQL. It covers real-world production concerns such as query planning, write-heavy ingestion, replication, indexing behavior, and operational trade-offs versus external vector stores. If you are building AI systems that require predictable latency, clear security boundaries, and SQL-first workflows, this approach is worth considering. Read the full article: https://lnkd.in/dXzpmdkE #PostgreSQL #AIEngineering #VectorSearch #RAG #OnPrem #Databases #MachineLearning

Predictable and reliable doesn’t have to mean boring. We just shipped a major Lakebase update with: ⚡ Autoscaling of compute based on load, with scale to zero when idle 🌿 Instant database branching, enabling git-like workflows with isolated, copy-on-write environments for dev/test/staging 🚀 Fast provisioning and sub-second resume 🛡️ Built-in backups and point-in-time recovery ✨ A refreshed Lakebase UI for everyday workflows Lakebase defines a new category of operational database: OLTP running directly on cloud object storage, with serverless Postgres compute on top. It’s built to simplify how teams run the operational layer for Data Intelligent Applications.

Parquet Isn’t Enough at Scale ❌ — That’s Why Delta Lake Exists ✅ While working on Databricks pipelines in Azure, I realized that simply storing data in Parquet is not enough when pipelines scale and multiple jobs start writing concurrently. That’s when I truly understood why Delta Lake is a core component of Databricks, not just an add-on. At its core, Delta Lake is a transactional storage layer built on top of Parquet and cloud object storage. Every Delta table maintains a _delta_log, which records all file-level changes and metadata. This transaction log enables ACID guarantees and snapshot isolation, ensuring readers never see partial or corrupt writes even with concurrent workloads. Delta Lake also enforces data quality through schema enforcement, while allowing controlled changes via schema evolution. Combined with features like time travel, OPTIMIZE, and Z-ORDER, Delta solves common data lake challenges such as schema drift, small files, and unpredictable query performance. Delta Lake is what truly bridges the gap between data lakes and data warehouses, making the Lakehouse architecture possible in Databricks. Delta Lake — Core Features ✔ ACID transactions enabled through a transaction log (_delta_log) on cloud storage ✔ Native upserts, updates, and deletes (MERGE, UPDATE, DELETE) ✔ Schema enforcement & controlled schema evolution to maintain data quality ✔ Time travel for data versioning, auditing, and rollback ✔ Performance optimizations with OPTIMIZE and Z-ORDER #Databricks #DeltaLake #DataEngineering #Azure #Lakehouse

🚀 New on the AWS Blog: Building an Open Warehouse Architecture We’re excited to share how Supabase is bridging the gap between transactional and analytical workloads using Amazon S3 Tables. This "Open Warehouse" architecture launched at AWS re:Invent, allows developers to combine the speed of Postgres with the scale of Apache Iceberg, all without managing complex pipelines. Read the full deep dive here: https://lnkd.in/ge4kMGnJ A huge thank you to my co-authors for this collaboration: Riccardo Busetti, Fabien Gaud, Ameen Khan S , and Pritesh Patel. #AWS #Supabase #reInvent2025 #OpenWarehouse #ApacheIceberg #S3Tables

Ever wondered if ClickHouse can query your lakehouse catalog directly? Spoiler: it can. The DataLakeCatalog engine connects to AWS Glue Catalog and Databricks Unity Catalog, automatically detecting whether your tables are Iceberg or Delta Lake, and enables you to query them instantly - no data movement is required. The post walks through: • Connecting to catalogs in under a minute • Querying Iceberg and Delta Lake tables like they're native • Running federated queries across multiple catalogs in a single statement If it's in your catalog, you can query it with ClickHouse. https://lnkd.in/gKgvQNSB

In modern data platforms, orchestration is no longer just about scheduling tasks—it’s about control, security, and scale. Apache Airflow now sits at the center of this complexity, quietly coordinating critical systems across cloud and enterprise boundaries. Yet one design choice is often underestimated: how we define and manage connection details. As pipelines grow and environments multiply, flat configurations start to break. This is why advanced teams are moving toward structured, JSON-based connection definitions. Not as a preference—but as a platform decision. Here’s why this approach is shaping production-grade Airflow architectures today. #ApacheAirflow #DataEngineering #PlatformEngineering #CloudArchitecture #ModernDataStack #DataOps #DevOps

Is Lineage HOT again? I heard that statement from multiple folks recently and it has started to settle with me. Here's what we're seeing: - Data flow visibility and lineage tracking are not really solved. It's "ok" if you're running a warehouse, but once you have some legacy ETL, on-prem databases, applications, custom transformation code, Spark pipelines and large-scale BI deployments, it's far from solved - Interestingly, with cloud it doesn't go away, in particular when AI code is introduced. Extracting lineage for Python Notebooks is a hard problem. Even if it's Databricks. And good luck getting lineage for something like MongoDB. But teams now need even more - because now there are numerous AI calls and agents going around, and no one really knows where do they go and what sensitive data they might access. It reminds me of the 2015-2016 days pre cloud security. All of that information is in code and in runtime - to create visibility and controls you HAVE to go to the code. So is lineage HOT again? Very excited for what we have in store for 2026. Happy Holidays!

Lineage "hot" 🔥 again because AI needs it. 🤖 If you're building with AI agents or using Claude/Cursor, you know that hallucinations happen when the AI doesn't understand your data dependencies. Foundational solves this by: 💡 Providing real-time, deep, code-based context (even for Spark/Python). 💡 Act as safeguard on any code changes that may break downstream dashboards Alon Nafta