Bhupesh kothari

In one of my YouTube videos, I demonstrated how to architect AUTOMATED DATABRICKS NOTEBOOK CI/CD USING GIT ACTIONS which is completely a proprietary solution. Watch on YouTube if you're interested in CI/CD: https://lnkd.in/gg4HC2Qd Based on popular demand, I am going to show you how to architect the same solution using GitLab and Azure DevOps Git Providers and unique YML syntax. This is going to help you to understand how CI/CD works in the field of data engineering across different providers Stay tuned. Follow Me. Report. Subscribe #Databricks #GitLab #AzureDevOps #CICD #CommunityRocks #DataEngineering

102

5 Comments

Building AI That Works at Scale Starts with the Right Foundation Many organizations are ready to embrace the next generation of AI but face roadblocks like siloed data, legacy infrastructure, and rising costs. The solution? A modern data platform purpose-built for AI. MongoDB Atlas + Microsoft Azure AI Foundry offers a secure, unified, and scalable environment designed to power GenAI and autonomous systems at scale. Why this matters for GenAI success: 🔍 Unified architecture: Combine vector, hybrid, and transactional search in a single data layer ⚙️ Developer first: Streamline GenAI pipelines using live, structured data , no stitching required 🌍 Built to scale: Global reach and high availability to support production workloads Learn how enterprises are moving from GenAI pilots to production with: Scalable vector search on Azure Enterprise grade security and compliance Support for RAG, agents, and real-time reasoning Rapid deployment from prototype to production 🎙️ Featuring expert insights from: Paul Leury (MongoDB) & Azim Uddin (Microsoft) 🎯 Free Webinar | August 6 @ 11AM ET Topic: From Vectors to Value: Real-World AI with MongoDB Atlas on Azure 🔗 Register here : https://bit.ly/46ATb0Y Ready to move beyond AI experiments? Join this free webinar to learn how to build AI systems that scale, adapt, and deliver value fast. #genai #microsoft #mongodb #ai #dataengineering #free #datascientist #data #career #ml

251

71 Comments

𝐀𝐩𝐚𝐜𝐡𝐞 𝐒𝐩𝐚𝐫𝐤 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐚𝐭𝐢𝐨𝐧: 𝐉𝐨𝐛 𝐜𝐨𝐦𝐩𝐥𝐞𝐭𝐞𝐝 𝐢𝐧 90 𝐦𝐢𝐧𝐬 We have a requirement to perform a one-time load of a reporting table for all our enterprise customers which was never done before. So we are sure we will run into optimization issues and in the same way the job is failing with executors get lost, heartbeat from executor lost, RPC timeout issues etc Issues: Memory pressure: Missed heartbeats > 120s timeout = executor death Network_saturation: RPC failures when communicating with executors Rpc_system_overload: Driver overwhelmed managing too many concurrent tasks and RPC queue saturation during executor failures Solution: Reduced executor cores (5→3): Less memory pressure per executor, shorter GC pauses Increased the no. of executors: Better parallelism for delta operations Increased network timeout (420s→800s): Accommodates S3 throttling and network congestion Shorter heartbeat interval (60s→30s): Earlier detection of executor issues, faster recovery Introduced executor and driver memoryOverhead parameters: To handle the GC cleanup and off-heap operations better In our job, we had both CPU intensive operations like de-duplication, scheduling & coordination of tasks as well as memory intensive operations like broadcasting of smaller datasets, shuffle operations and delta log processing so we tried a balanced approach of resource management and spark properties. We have fixed the performance issues and let the job complete in 90mins which is not even going through before. Given its a one-time load, didn’t spend more time to optimize it further. #dataengineer #bigdata #dataengineering #apachespark #sql #datascience #aws #azure #ai #sparkoptimization

105

4 Comments

Building My First End-to-End Data Pipeline — The Real Lessons When I started building my first data pipeline, I thought the toughest part would be writing the transformations. But I was wrong — the real challenge was making everything work together. Here’s what that project really taught me 👇 1. A pipeline isn’t just code — it’s orchestration. From ingestion to storage to transformation and load — every step has dependencies, retries, and SLAs. It’s like designing a relay race where every runner must start exactly when the previous one finishes. 2. Metadata matters more than you think. Tracking schema, timestamps, and data quality metrics saves you hours of debugging later. Good metadata = easier lineage and trust in your data. 3. Incremental processing is an art. Loading everything daily looks simple — until your job scales. Building pipelines that process only what changed is where you start thinking like a real data engineer. 4. Testing and validation are not optional. Unit tests for transformations and sanity checks for outputs prevent bad data from quietly flowing downstream. 5. Monitoring is the unsung hero. A failed job is fine. A failed job that no one notices — that’s the problem. Set up alerts, logs, and metrics early — they’re your best friend at 2 AM. Every pipeline you build teaches you a bit more about systems thinking. It’s no longer about writing code — it’s about designing reliability. If you’re looking to crack Data Engineering interviews at top product and service-based companies, this resource can really help you prepare in a structured way 👇 https://lnkd.in/d5pUkVMf

66

6 Comments

🚀 These Spark optimizations helped me reduce execution time from 6 hours to just 25 minutes — and mastering them can help you land a 25+ LPA job in 2025! 💼🔥 • spark.sql.adaptive.enabled = true Enables Adaptive Query Execution (AQE) to optimize query plans dynamically at runtime. • spark.sql.adaptive.skewJoin.enabled = true Handles skewed joins automatically by splitting large skewed partitions into smaller ones. • spark.sql.broadcastTimeout = 3000 Extends the broadcast timeout for large dimension tables to avoid query failures. • mapreduce.fileoutputcommitter.algorithm.version = 2 Uses a more efficient file output commit algorithm, improving write performance. • spark.sql.adaptive.forceOptimizeSkewedJoin = true Forces Spark to always optimize skew joins, even when not obvious. • spark.sql.autoBroadcastJoinThreshold = 1GB Allows larger tables (up to 1GB) to be broadcasted for faster joins. • spark.sql.shuffle.partitions = 3000 Optimizes the number of shuffle partitions for better parallelism and reduced data skew. • spark.sql.serializer.KryoSerializer.buffer.max.mb = 2048 Boosts serialization efficiency to handle large objects without memory errors. 💡 These small tweaks can be a game-changer in interviews and in real-world big data pipelines. #DataEngineering #ApacheSpark #BigData #PerformanceTuning #SparkOptimization #Scaler #JobPreparation2025 #DataEngineer

138

7 Comments

MERGE is a common query that is executed in Lakehouse. While working with ICEBERG and MERGE (read → merge → write), we can do the following to improve the MERGE performance. → Batch update (CoW), Micro batch (MoR) → Apply MERGE statements on the relevant partitions. → Enable metrics collection (write.metadata.metrics.*) → Add filters in the ON clause of the MERGE statement.  → Enable write fanout (write.spark.fanout.enabled) → Tweak distribution mode (spark.sql.iceberg.distribution-mode) → If the job can, disable sort merge join (spark.sql.join.preferSortMergeJoin) → Check if Storage Partition Join (Include partition col in ON clause of MERGE) → Do regular table maintenance #apacheIceberg #iceberg #datalake #lakehouse #openosurce

45

1 Comment

I didn’t realise just how powerful SQL is when I was first learning it To quote Lukas Eder, the creator of jOOQ: > 𝘚𝘘𝘓 𝘪𝘴 𝘢 𝘥𝘦𝘷𝘪𝘤𝘦 𝘸𝘩𝘰𝘴𝘦 𝘮𝘺𝘴𝘵𝘦𝘳𝘺 𝘪𝘴 𝘰𝘯𝘭𝘺 𝘦𝘹𝘤𝘦𝘦𝘥𝘦𝘥 𝘣𝘺 𝘪𝘵𝘴 𝘱𝘰𝘸𝘦𝘳 Below are three of my favourite videos which aim to prove this, and which significantly altered the way I thought about SQL and relational databases when I was first learning about them 𝟭𝟬 𝗦𝗤𝗟 𝘁𝗿𝗶𝗰𝗸𝘀 𝘁𝗵𝗮𝘁 𝘆𝗼𝘂 𝗱𝗶𝗱𝗻’𝘁 𝘁𝗵𝗶𝗻𝗸 𝘄𝗲𝗿𝗲 𝗽𝗼𝘀𝘀𝗶𝗯𝗹𝗲 (Lukas Eder) This is the first video that really showed me what’s possible in SQL: it includes recursive CTEs, various window function usages, the PIVOT clause, Oracle’s MODEL clause, and more! Also, the thumbnail is velociraptor wielding an RPG and riding a shark: pretty awesome https://lnkd.in/eaG_52mJ 𝗠𝗼𝗱𝗲𝗿𝗻 𝗦𝗤𝗟: 𝗮 𝗹𝗼𝘁 𝗵𝗮𝘀 𝗰𝗵𝗮𝗻𝗴𝗲𝗱 𝘀𝗶𝗻𝗰𝗲 𝗦𝗤𝗟-𝟵𝟮 (Markus Winand) SQL evolves a lot over time. This video covers a bunch of the new syntax that were added to the ANSI standard since the SQL-92 standard, including (recursive) CTEs, advanced GROUP BY options, window functions, object time travel, and more! This complements the previous video because Markus also adds extra details around the syntax and which database engines support the syntax (at the time of the recording), and one particular part of ANSI standard you should avoid (spoiler: the OFFSET clause) https://lnkd.in/eR7rP-Dv 𝗦𝗤𝗟𝗶𝘁𝗲: 𝗧𝗵𝗲 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲 𝗮𝘁 𝘁𝗵𝗲 𝗘𝗱𝗴𝗲 𝗼𝗳 𝘁𝗵𝗲 𝗡𝗲𝘁𝘄𝗼𝗿𝗸 (𝗥𝗶𝗰𝗵𝗮𝗿𝗱 𝗛𝗶𝗽𝗽) SQLite doesn’t follow the usual client-server pattern, making it extremely easy to start using. In this video, the creator (Richard) explains how SQLite’s architecture doesn’t compromise on its power: it just changes the use case This helped me realise that, although they all speak SQL, different relational databases have different use cases and you should aim to pick the one right for your scenario https://lnkd.in/eGwVVmGt What have been some of your favourite SQL videos? #sqlwithbill #sql

50

5 Comments

From 1 TB to 56 GB: How file formats & transformations can make or break your Snowflake bill (and speed) When you’re working in Snowflake with dbt transformations, file size isn’t just a storage number—it directly impacts query cost, load times, and performance. The same dataset might take up 1 TB in CSV format, but shrink dramatically if you use optimized columnar formats like Parquet or ORC before loading into Snowflake. Typical size impact: ╰┈➤ CSV: 1 TB ╰┈➤ Avro: ~240 GB (~76% smaller) ╰┈➤ Parquet: ~104 GB (~89.6% smaller) ╰┈➤ ORC: ~56 GB (~94.4% smaller) That’s up to 95% smaller, before you even start optimizing queries in dbt. ⸻ 📄 CSV in Snowflake • Plain text, no compression. • Slower bulk loads into Snowflake stages. • Higher cloud storage & compute costs when transforming. • In dbt, transformations will have to scan all columns, even unused ones. ⸻ 📦 Avro in Snowflake • Row-based binary format—great for event ingestion. • Works well with Snowpipe or streaming loads into Snowflake. • Supports schema evolution (helpful for dbt incremental models). • But for analytics, columnar formats beat it on query speed. ⸻ 📊 Parquet in Snowflake • Columnar format = Snowflake can prune unnecessary columns during scans. • Highly compressed, faster to load via COPY INTO. • Perfect for dbt’s incremental models—less data scanned = lower costs. • Ideal for analytical workloads on large datasets. ⸻ 📘 ORC in Snowflake • Also columnar, with excellent compression. • Stores statistics (min/max) to speed up predicate pushdown. • Great for batch loads before dbt runs transformations. ⸻ 💡 Snowflake + dbt takeaway: • Store data in stages as Parquet/ORC for maximum compression and query pruning. • Use dbt incremental models to avoid reprocessing unchanged data. • Leverage Snowflake’s micro-partition metadata for lightning-fast queries. ⸻ 💰 Before-and-After Cost Simulation Let’s say your warehouse is Medium (4 credits/hour) and Snowflake charges $2/credit. Scenario 1 – CSV + Full dbt Refresh • Data size scanned: 1 TB • Full scan every run (no incremental) • Query time: ~1 hour • Cost per run: 4 credits x $2 = $8 • Monthly (daily run): $8 x 30 = $240 Scenario 2 – Parquet + dbt Incremental • Data size scanned: 104 GB (89.6% smaller) • Only 10% of data changes each run (incremental model) • Effective scan: ~10 GB • Query time: ~5 minutes • Cost per run: ~0.33 credits x $2 ≈ $0.66 • Monthly (daily run): $0.66 x 30 ≈ $19.80 💥 Savings: From $240 → $19.80/month → ~92% cost reduction. ⸻ Choosing the right file format isn’t just about storage—it’s about controlling your Snowflake bill and speeding up dbt runs. #Snowflake #dbt #DataEngineering #Parquet #BigData #seniormanagers #topITforce Author - Kunwar-Sa

164

9 Comments