Zbigniew Wyszomierski

When data agents fail, they often fail silently - giving confident-sounding answers that are wrong, and it can be hard to figure out what caused the failure. "Building and Evaluating Data Agents" is a new short course created with Snowflake and taught by Anupam Datta and Josh Reini. This course teaches you to build data agents with comprehensive evaluation built in. Skills you'll gain: - Build reliable LLM data agents using the Goal-Plan-Action framework and runtime evaluations that catch failures mid-execution - Use OpenTelemetry tracing and evaluation infrastructure to diagnose exactly where agents fail and systematically improve performance - Orchestrate multi-step workflows across web search, SQL, and document retrieval in LangGraph-based agents The result: visibility into every step of your agent's reasoning, so if something breaks, you have a systematic approach to fix it. Sign up to get started: https://lnkd.in/gVwKuCCQ

6638

224 komentarze

Death by a thousand defaults on #Databricks. Databricks clusters: latest ML runtime + auto-termination set to 3 days. Dashboards: refresh every few minutes. Alerts: happily running queries on your most expensive cluster. Online tables: syncing more often than anyone needs. Not obvious. Sometimes hidden. Always expensive.

136

10 komentarzy

What I love about #ApacheIceberg is how many different workloads folks are running on the same copy of data. Graph queries are a really cool example. Instead of copying data into specialized graph databases, PuppyGraph lets you run graph traversals directly on your existing Iceberg tables. Pretty cool stuff. But here's the thing - to get truly great performance, you need to ensure your Iceberg tables are well maintained and optimized. That's where Ryft comes in. We make sure your Iceberg tables are automatically maintained and optimized, no matter what engine you are using. It was terrific to work with Zhenni Wu and Weimo Liu, to create this blog post on how you can use Ryft with PuppyGraph to run graph queries on a fully optimized Iceberg lake! https://lnkd.in/giwnwMuc #ApacheIceberg #GraphQueries

99

2 komentarze

✨ Big news: the 2nd Edition of Data Modeling with Snowflake is officially out today! It's hard to believe that only two years have passed since the first edition, yet so much has changed. New Snowflake features, such as Dynamic and Iceberg tables, have transformed the way data is shared and processed. Additionally, the introduction of Cortex and the semantic layer allows AI to execute queries using natural language, adding even more capabilities. As AI continues to evolve, it's evident that data modeling is playing an increasingly important role in creating scalable and cost-effective data architectures. The rising costs of inefficiency highlight this relevance. However, despite its importance, data modeling remains far from ubiquitous. One major reason data modeling initiatives often fail is not due to technical issues, but rather personal ones: the engineering team often struggles to effectively communicate the business value to stakeholders. My favorite chapter in the second edition addresses this by breaking down business metrics and the ROI of data modeling. It provides data teams with a playbook for engaging stakeholders, persuading the budget committee, and fostering business alignment. Since publishing the first edition, I've heard directly from readers who have used this book to transform their careers and convince colleagues that data modeling is a worthwhile endeavor. I am confident that the second edition will build on that success and provide readers with even more tools to establish a solid foundation for creating efficient and scalable data solutions. A huge thank you to Joe Reis for writing the introduction to this edition and for promoting data modeling within the data community through his ongoing work. If you aren't already following him, be sure to do so, and keep an eye out for his highly anticipated data modeling publication. Also, stay tuned for a release webinar and AMA session in the coming week. I'll be taking questions and giving away a copy of the book as well. Get your copy on Amazon: https://a.co/d/iVak9qF or direct from the publisher: https://lnkd.in/dEjk-Xf3

170

19 komentarzy

Delta Lake 4.0 now supports Variant data type designed for working with semi-structured data like JSON with evolving schemas! Key capabilities include: 🔹 𝗦𝗰𝗵𝗲𝗺𝗮-𝗼𝗻-𝗿𝗲𝗮𝗱: No need to define a fixed schema in advance. Structured data can be extracted at query time. 🔹 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝘀𝘁𝗼𝗿𝗮𝗴𝗲: Dynamic payloads are stored using high-performance encoding for better storage efficiency. 🔹 𝗦𝗵𝗿𝗲𝗱𝗱𝗲𝗱 𝗩𝗮𝗿𝗶𝗮𝗻𝘁𝘀 (𝗣𝗿𝗲𝘃𝗶𝗲𝘄): Allows extraction of specific columns from variant data and collection of statistics to speed up queries and file skipping. This feature was developed collaboratively with contributions from Apache Parquet, Apache Arrow, Apache Iceberg, #DeltaLake, Apache Spark, and others, supporting integration across the broader data ecosystem. 🔗 For more details, see the full release notes: https://lnkd.in/edemXCS5 #opensource #oss #linuxfoundation #variantdata

223

6 komentarzy

Apache Iceberg is attracting a lot of attention in the analytic community, and I encounter an increasing number companies with production apps on it. Iceberg V3 is a big step forward in capabilities, especially around data types. VARIANT and nanosecond timestamps sound obscure to the uninitiated, but they are critical in real-time applications. Here's a nice summary from Google. https://lnkd.in/gVtmsZuq #apache #iceberg #realtime #analytics #opensource

36

🚀 Run Microsoft Power BI with DuckDB Server! 🦀 We added extensive Postgres compatible interface and put it around DuckDB as a server with Rust. You can use psql, bgbench, DBeaver, Grafana, Superset and so on. The interface supports both text and binary encoding and has extensive type support and pg catalog for schema discovery. Latest we integrated with Power BI, which so far was the most demanding integration from the rest. 🦆 You have one-to-one connections directly into DuckDB memory. ✈️ You can ingest data to the DuckDB server with high throughput by using the DuckDB Airport extension - while at the same time run realtime analytics queries through the postgres interface. 🪣 All the ingested data goes through diskless pipeline directly to S3 as Parquet (DuckLake) and to local DuckDB on-disk databases, directly queryable through the Postgres interface. Download and use for free! https://lnkd.in/eMfSUqQ6 #powerbi #analytics #duckdb #rust #claudecode #ducklake

27

[The Dash Board] ✨ One of the most requested features is now GA! 🙌 Snowflake customers can now store objects up to 128MB in FDN (Snowflake Native Tables) and Iceberg Tables! 🤯 Why it matters: This update unlocks new possibilities for working with VARIANT, VARCHAR, BINARY, structured, and geospatial types — especially when dealing with large documents, rich JSON payloads, or complex nested structures. So whether you're building data apps, ingesting semi-structured logs, or working across open formats with Iceberg, this enhancement removes a major bottleneck. 💥 Real-world use cases: ↳Store large geospatial shapes (like full regions or boundaries) ↳Record events data without sharding ↳Ingest JSON logs that can't be flattened from upstream devices as-is ↳Capture clickstream data ↳Archive HR exports as single, large JSON files 👏 Huge kudos to Oleksii Bielov and the entire team for making this awesome feature generally available for our customers!

348

10 komentarzy