Vibhaas Vinay Kotwal

Since last many months I was curating the most advanced topics on Data Engineering. I had covered the basics in depth beforehand so from today to next 40-50 days I will be posting the NOT so common topics. Expert DE Day #1: Delta vs Iceberg vs Hudi - Which Lakehouse Table Format Should You Choose? I’ve put together a practical, side-by-side guide comparing Delta Lake, Apache Iceberg, and Apache Hudi - focusing on time travel, compaction, schema evolution, and upserts. This resource includes: ✔️ Feature matrix for quick reference ✔️ How-to snippets in SQL & configs ✔️ Operational runbooks for each format ✔️ Decision tips for choosing the right fit If you’re building modern data platforms, this will help you navigate the trade-offs between the three leading formats. #gsglearn

216

13 Comments

When data agents fail, they often fail silently - giving confident-sounding answers that are wrong, and it can be hard to figure out what caused the failure. "Building and Evaluating Data Agents" is a new short course created with Snowflake and taught by Anupam Datta and Josh Reini. This course teaches you to build data agents with comprehensive evaluation built in. Skills you'll gain: - Build reliable LLM data agents using the Goal-Plan-Action framework and runtime evaluations that catch failures mid-execution - Use OpenTelemetry tracing and evaluation infrastructure to diagnose exactly where agents fail and systematically improve performance - Orchestrate multi-step workflows across web search, SQL, and document retrieval in LangGraph-based agents The result: visibility into every step of your agent's reasoning, so if something breaks, you have a systematic approach to fix it. Sign up to get started: https://lnkd.in/gVwKuCCQ

6,261

215 Comments

A butterfly chart is used to compare two data sets side-by-side. Typically this the same data broken down by a dimension. For example, this year sales goes right and last year sales goes left. They work especially well with population data. Here’s how you make a butterfly chart in Tableau. Let me know in the comments how you would use a butterfly chart.

82

7 Comments

Inference shall be a differentiating factor among the Data Scientists and an Important Element to focus in the Enterprise level data science and decision making. Even for any organisation that cares about the evidence that could answer the business question need to emphasise the Inference (simple statistical inference to causal inference), Statistical Inference is something helps in building better predictive models, This is the right time that “Noise in the Data Science Community is settling down”, and giving space for such dialogues and unique opportunities! A new practice of DS needs to be “Evidence and Inference” focused! Even if we want to understand the patterns of how adapted Gen AI systems helps in the business problems , how to perform evals of an Agent or a RAG solution - Inferential Mindset is somethting critical! ps - just thought of recording my thoughts and ideas as a video for better communication and engagement! 🙂‍↔️

130

8 Comments

What Are Data Silos? Data silos refer to isolated pockets of data controlled by specific departments or systems, inaccessible to the broader organization. Why Are Data Silos Problematic? 1.Hinder cross-department collaboration and data sharing 2.Lead to inconsistent, duplicate, and unreliable data across systems 3.Slow down decision-making and reduce organizational agility 4.Increase operational costs due to redundant data management 5.Limit the effectiveness of analytics and AI initiatives How Can Organizations Break Them Down? 1.Advocate a unified data governance framework with clear policies 2.Implement centralized, cloud-based data platforms for integration 3.Foster a culture of openness and data democratization 4.Utilize integration tools and real-time data-sharing technologies Keep exploring the world of Data — Engineering, Analytics and AI. Follow Anil Patel for more content on Data Engineering, Analytics and AI. 𝐁𝐨𝐨𝐤 𝐚 Free 𝐜𝐚𝐥𝐥-https://lnkd.in/gca6jRJD #data #sql #datamesh #warehouse

76

6 Comments