Joris Van den Borre

Enterprise data pulls in two directions. On one side: governed BI workloads — structured, compliant, and built for business reporting. On the other: unstructured pipelines — streaming feeds, heavy transformations, and machine learning at scale. Most organizations now run Snowflake and Databricks in parallel to handle this dual reality. Yet stitching them together creates new challenges: deciding which workloads belong where, moving data without waste, and keeping governance consistent across environments. Each platform brings distinct strengths to the table — Snowflake excels at governed analytics and BI reporting, while Databricks powers heavy transformations, streaming, and machine learning. When combined thoughtfully, these strengths converge into a unified architecture that supports the full data lifecycle. > Watch the on-demand discussion on how Janus Henderson Investors designed a unified architecture that balances both worlds: https://lnkd.in/eG-8YBqm In a financial services setting where accuracy, performance, and cost discipline matter most, Janus Henderson’s approach connects Snowflake’s strengths in analytics and governance with Databricks’ capabilities in engineering and ML. #unifieddataarchitecture #snowflake #databricks #usecase

135

38 Comments

We spent 10 years building the Modern Data Stack. It's already obsolete. The old architecture (what 99% of brands still use): ➣ Shopify API -> ETL Script -> Snowflake -> Looker ➣ Facebook Ads API -> Fivetran -> Snowflake -> Looker ➣ Klaviyo API -> … Result: A dozen brittle pipelines feeding a passive, read-only dashboard that tells you what happened yesterday. This entire "look-at-the-past" model collapses during a live, high-stakes event like Black Friday. By the time your dashboard refreshes, the opportunity is gone. The paradigm shift isn't a better dashboard. It's moving from a passive stack to an active, agentic platform. The new architecture has two layers: 1⃣ A Unified Data Layer: All sources (sales, ads, inventory) are synced and modeled in real-time into a single source of truth. 2⃣ An Agentic Layer: Autonomous AI agents that can read, analyze, and execute tasks on top of that live data stream. This isn't a theory. This is the architecture behind 𝐓𝐫𝐢𝐩𝐥𝐞 𝐖𝐡𝐚𝐥𝐞's Moby Agents. It unlocks a level of real-time automation that was previously impossible. For BFCM, this means: ➣ An agent can perform a live join between Facebook ad spend and Shopify margin data, identify a low-profit campaign, and then trigger a webhook to pause it - no human required. ➣ An agent can monitor inventory levels against sales velocity, predict a stock-out in 3 hours, and execute a command to automatically remove the item from active campaigns. The job is no longer about building brittle pipelines to look at yesterday's charts. The results are insane: One brand cut $187,000 in wasted ad spend because an agent paused 3 unprofitable campaigns mid-flight. Another unlocked an extra 22% 𝐦𝐚𝐫𝐠𝐢𝐧 by auto-shifting budget toward their highest-profit SKUs in real time. A team running 40+ SKUs avoided a $92𝐊 𝐬𝐭𝐨𝐜𝐤-𝐨𝐮𝐭 when an agent predicted sell-through 3 hours in advance and pulled ads instantly. The future isn't building dashboards. It's building agents. 📌 You can them out here: https://bit.ly/4m8O8sz ------- PS. Want Triple Whale's complete pre-trained AI Agent Library plus the whole data set? 💎 Comment "MOBY" and I'll send it to you. I really like their vision and am glad to collaborate with Triple Whale.

337

66 Comments

🤯 How the hell did I miss this update to Databricks SQL last month? 💡 Databricks have been serious about their investments in SQL for some time now. In July they introduced stored procedures in their Databricks SQL (version 2025.20 in Preview)! As in other data platforms, stored procedures in Databricks SQL are pre-compiled collections of SQL statements allowing users to manage their SQL logic into a single, reusable unit. As expected, procedures in Databricks are stored in Unity Catalog. And yes, they can be nested and recursive ;-) ✨ Why I think this is important? Because: 1) it's another important step to make Databricks more friendly for SQL developers willing to use their skills (and habits) built on other platforms (often on-premises RDBMSs) to Databricks and 2) support for stored procedures can simplify the migration process when moving from SQL-based platforms to Databricks. ❓ Anyone has already given a serious test drive to stored procs in Databricks SQL? ➡️ Read more here: https://lnkd.in/dYUwzevW #Databricks #SQL #StoredProcedures #DataPlatform #DataEngineering

583

77 Comments

𝐃𝐚𝐭𝐚𝐛𝐫𝐢𝐜𝐤𝐬 𝐯𝐬 𝐅𝐚𝐛𝐫𝐢𝐜 – 𝐖𝐡𝐚𝐭 𝐟𝐢𝐭𝐬 𝐲𝐨𝐮𝐫 𝐛𝐮𝐬𝐢𝐧𝐞𝐬𝐬 𝐛𝐞𝐬𝐭? (𝐓𝐨𝐦𝐨𝐫𝐫𝐨𝐰, 𝐨𝐧𝐥𝐢𝐧𝐞) If you want to understand how Databricks and Microsoft Fabric are evolving, and what that means in practice for data engineering, governance and reporting, then tune in tomorrow morning! Five months ago, Anders Krohn and I hosted a webinar on the same topic, and the interest much higher than expected. So we’re doing it again, this time with an updated perspective. Both platforms are moving fast, and some of the opinions we had back then might already look a bit different today. 𝐖𝐞’𝐥𝐥 𝐬𝐡𝐚𝐫𝐞 𝐨𝐮𝐫 𝐥𝐚𝐭𝐞𝐬𝐭 𝐭𝐚𝐤𝐞 𝐨𝐧 𝐡𝐨𝐰 𝐭𝐡𝐞 𝐩𝐥𝐚𝐭𝐟𝐨𝐫𝐦𝐬 𝐜𝐨𝐦𝐩𝐚𝐫𝐞 𝐢𝐧 terms of: ▪️ Data engineering ▪️ Governance ▪️ Reporting ▪️ How teams are actually using them in real scenarios. This isn’t a feature-by-feature walkthrough. We will focus on understanding how the platforms are developing, where they differ, and where they might overlap. 🗓️ 𝐎𝐧𝐥𝐢𝐧𝐞 𝐭𝐨𝐦𝐨𝐫𝐫𝐨𝐰 🔗 https://lnkd.in/drPYpmrU Evidi Evidi Denmark Torben Nyvang Jakobsen Martin Bundesen Anette Kilebu Brage Rønning Tukkensæter Lars Yngvar Tømmerholt #Databricks #Microsoft #MicrosoftFabric #DatabricksMVP

162

5 Comments

When data agents fail, they often fail silently - giving confident-sounding answers that are wrong, and it can be hard to figure out what caused the failure. "Building and Evaluating Data Agents" is a new short course created with Snowflake and taught by Anupam Datta and Josh Reini. This course teaches you to build data agents with comprehensive evaluation built in. Skills you'll gain: - Build reliable LLM data agents using the Goal-Plan-Action framework and runtime evaluations that catch failures mid-execution - Use OpenTelemetry tracing and evaluation infrastructure to diagnose exactly where agents fail and systematically improve performance - Orchestrate multi-step workflows across web search, SQL, and document retrieval in LangGraph-based agents The result: visibility into every step of your agent's reasoning, so if something breaks, you have a systematic approach to fix it. Sign up to get started: https://lnkd.in/gVwKuCCQ

6,637

224 Comments

What I love about #ApacheIceberg is how many different workloads folks are running on the same copy of data. Graph queries are a really cool example. Instead of copying data into specialized graph databases, PuppyGraph lets you run graph traversals directly on your existing Iceberg tables. Pretty cool stuff. But here's the thing - to get truly great performance, you need to ensure your Iceberg tables are well maintained and optimized. That's where Ryft comes in. We make sure your Iceberg tables are automatically maintained and optimized, no matter what engine you are using. It was terrific to work with Zhenni Wu and Weimo Liu, to create this blog post on how you can use Ryft with PuppyGraph to run graph queries on a fully optimized Iceberg lake! https://lnkd.in/giwnwMuc #ApacheIceberg #GraphQueries

99

2 Comments