Kai Waehner

What I love about #ApacheIceberg is how many different workloads folks are running on the same copy of data. Graph queries are a really cool example. Instead of copying data into specialized graph databases, PuppyGraph lets you run graph traversals directly on your existing Iceberg tables. Pretty cool stuff. But here's the thing - to get truly great performance, you need to ensure your Iceberg tables are well maintained and optimized. That's where Ryft comes in. We make sure your Iceberg tables are automatically maintained and optimized, no matter what engine you are using. It was terrific to work with Zhenni Wu and Weimo Liu, to create this blog post on how you can use Ryft with PuppyGraph to run graph queries on a fully optimized Iceberg lake! https://lnkd.in/giwnwMuc #ApacheIceberg #GraphQueries

99

2 Kommentare

Enterprise data pulls in two directions. On one side: governed BI workloads — structured, compliant, and built for business reporting. On the other: unstructured pipelines — streaming feeds, heavy transformations, and machine learning at scale. Most organizations now run Snowflake and Databricks in parallel to handle this dual reality. Yet stitching them together creates new challenges: deciding which workloads belong where, moving data without waste, and keeping governance consistent across environments. Each platform brings distinct strengths to the table — Snowflake excels at governed analytics and BI reporting, while Databricks powers heavy transformations, streaming, and machine learning. When combined thoughtfully, these strengths converge into a unified architecture that supports the full data lifecycle. > Watch the on-demand discussion on how Janus Henderson Investors designed a unified architecture that balances both worlds: https://lnkd.in/eG-8YBqm In a financial services setting where accuracy, performance, and cost discipline matter most, Janus Henderson’s approach connects Snowflake’s strengths in analytics and governance with Databricks’ capabilities in engineering and ML. #unifieddataarchitecture #snowflake #databricks #usecase

135

38 Kommentare

What if your Databricks workloads could continuously self-improve their performance by up to 10x or more while you sleep, with one simple command? Enable zero touch performance with Automatic Liquid Clustering: CLUSTER BY AUTO -- Create a new table with Automatic Liquid Clustering CREATE OR REPLACE TABLE catalog.schema.table_name ( col1 INT, col2 STRING ) CLUSTER BY AUTO; or -- Enable Automatic Liquid Clustering on an existing table ALTER TABLE catalog.schema.table_name CLUSTER BY AUTO; Docs: https://lnkd.in/g8PWTsxz Most data teams are stuck in the endless cycle of manual optimization hell: analyzing query patterns, debating clustering strategies, running expensive ZORDER operations, and watching performance degrade as access patterns evolve. Meanwhile, compute costs keep climbing as data volumes grow. But here's what's actually possible: ✅ Healthrise deployed it across all gold tables and saw queries run up to 10x faster with zero manual work ✅ CFC Underwriting's compute costs went DOWN even as they scaled up data volume ✅ 3000+ customers are already writing 200+ PB monthly to auto-optimized tables 🔥 Don't wait while your competitors gain the advantage. This feature is available NOW in Unity Catalog managed tables. The magic happens in three steps: 1️⃣ Predictive Optimization analyzes your actual query patterns (not guesses) 2️⃣ Intelligently selects optimal clustering keys that maximize data skipping 3️⃣ Only triggers clustering when cost savings exceed overhead—pure ROI-driven decisions 💡 CHALLENGE: Try CLUSTER BY AUTO on just ONE of your busiest tables this week and drop a comment below with your results. I'll personally reply to every success story. 👇 What's your biggest query optimization pain point? Let's solve it together in the comments! Shannon Barrow Kyle Hale Mostafa Mokhtar 🚀 Laurent Leturgez 🚀 Matthieu Lamairesse Abhishek Dey Fabien Contaminard

159

4 Kommentare

Apache Iceberg is attracting a lot of attention in the analytic community, and I encounter an increasing number companies with production apps on it. Iceberg V3 is a big step forward in capabilities, especially around data types. VARIANT and nanosecond timestamps sound obscure to the uninitiated, but they are critical in real-time applications. Here's a nice summary from Google. https://lnkd.in/gVtmsZuq #apache #iceberg #realtime #analytics #opensource

36

PSA: OpenAI released their latest GPT-5 models, and we're proud to be a launch partner. We're making these models available in Snowflake Cortex AI the same day, all from within the secure Snowflake perimeter: ❄️ GPT-5: Designed for logic and multi-step tasks. ❄️ GPT-5 mini: A lightweight version for cost-sensitive applications. ❄️ GPT-5 nano: Optimized for speed and ideal for applications requiring low latency. Learn more about what you can do with GPT-5 in Cortex AI: https://lnkd.in/gGUgQvXQ 

130

We tested Snowflake's Gen2 warehouses on 2 billion query profiles across 14K warehouses here at Capital One. Here is where we saw improvement: 🍎 Approximately 30% of queries fell into categories that could benefit from Gen2—specifically those involving DELETE, UPDATE or MERGE. 🍎 Long-running DML operations: A significant subset of queries running longer than 30 minutes were performing complex DML tasks. These are prime candidates for Gen2 migration. 🍎 Large scans: Numerous queries scanned over 1 TB of data, where Gen 2’s improved scan performance can provide material runtime reduction. 🍎 Combined heavy patterns: A smaller but critical set of queries both scanned over 1 TB and ran for 30 minutes or more, often involving DML operations—these represent the highest potential gain from Gen2. These findings support a targeted migration strategy: focus on warehouses with a significant concentration of eligible workloads rather than migrating all warehouses indiscriminately. Check out the full blow written by our resident expert and all around awesome guy - Ganesh Bharathan https://lnkd.in/g8XufZaV

292

14 Kommentare