Erik Poelhuis

Databricks on Azure is everywhere , stealing the spotlight and making waves in the market! But let’s be honest, it can be tricky to truly understand these concepts until you dive into them yourself. That’s where these free end-to-end projects come in your shortcut to clarity and hands-on mastery! 🚀. Believe me, these projects are cutting-edge and align with the latest technologies. 𝗕𝘆 𝗲𝘅𝗽𝗹𝗼𝗿𝗶𝗻𝗴 𝘁𝗵𝗲𝘀𝗲 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀, 𝘆𝗼𝘂’𝗹𝗹 𝗴𝗮𝗶𝗻 𝗮 𝗰𝗼𝗺𝗽𝗿𝗲𝗵𝗲𝗻𝘀𝗶𝘃𝗲 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝗼𝗳 𝗸𝗲𝘆 𝗔𝘇𝘂𝗿𝗲 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝘀𝗲𝗿𝘃𝗶𝗰𝗲𝘀 𝘄𝗶𝘁𝗵 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗰𝗼𝗻𝗰𝗲𝗽𝘁𝘀 , 𝗶𝗻𝗰𝗹𝘂𝗱𝗶𝗻𝗴: ▪️ How Databricks works and its integration with Azure ▪️ Delta Lake and its benefits ▪️ Full and incremental data loads ▪️ Implementing SCD Type 2 ▪️ CI/CD DevOps integration ▪️ Lakehouse Medallion Architecture with Bronze, Silver, and Gold layers ▪️ Building ETL pipelines using Azure Data Factory ▪️ An overview of PySpark ▪️ Data Governance with unity catalog ▪️ Insights into Azure Synapse Analytics …and so much more! 𝗖𝗵𝗲𝗰𝗸 𝗼𝘂𝘁 𝘁𝗵𝗲𝘀𝗲 𝗳𝗿𝗲𝗲 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀. 👉 End to End Azure Data Engineering Project by Sumit Sir (Sumit Mittal) 🔗 https://lnkd.in/giN6pEEy 👉 Azure End-To-End Data Engineering Project (Job Ready) | Azure Data Engineering Bootcamp (Ansh Lamba) 🔗 https://lnkd.in/gJfPHQsV 👉 Olympic Data Analytics | Azure End-To-End Data Engineering Project (Darshil Parmar ) 🔗 https://lnkd.in/gb_gwT3R 👉 An End to End Azure Data Engineering Real Time Project Demo | Get Hired as an Azure Data Engineer ( Mr. K Talks Tech ) 🔗 https://lnkd.in/gQ7RT5kJ 👉 End to End Big Data Engineering Project with Azure | Big Data Engineering (From Scratch) (Mayank Aggarwal ) 🔗 https://lnkd.in/gTJFAu64 👉 Azure Data Engineer Project | Azure Data Engineer End to End Project | Azure data Engineer Job ( Aditya Chandak) 🔗 https://lnkd.in/gsPWybWw 👉 Azure End-To-End Data Engineering Project for Beginners (FREE Account) | SQL DB Tutorial (Luke J. Byrne) 🔗 https://lnkd.in/g-gPtg2P 👉 Azure End-To-End Data Engineering Project (From Scratch!) (Ansh Lamba) 🔗 https://lnkd.in/gAQaJpZm 👉 Football Data Analytics | Azure End To End Data Engineering Project ( 𝐂𝐨𝐝𝐞𝐖𝐢𝐭𝐡𝐘𝐮 (Yusuf Ganiyu) 🔗 https://lnkd.in/gNHrhDV5 Dive in to elevate your skills and master the world of Azure data engineering! 📕 Sharing 'Azure Databricks End to End Project' for your learning reference. Doc Credits : Linkedin Respective Creator. ♻️ Repost this to help the Job Seekers. P.S : I share valuable job search tips, industry insights, learning resources, and the latest updates in the data domain – all for free. Join thousands of other readers here → https://lnkd.in/g-ZtB4Yf Lets Learn Azure Databricks !! #dataengineer #azure #databricks #projects #data #azuredataengineer #cloud #microsoft #cloudcomputing

464

43 commentaren

A few days ago I published my article “Apache Parquet vs. Newer File Formats (BtrBlocks, FastLanes, Lance, Vortex)” looking at why new formats are emerging and how they compare to Apache Parquet/ORC. Last week, a new research paper introduced F3 (Future-proof File Format) - a next-gen open-source format designed to move beyond Parquet/ORC’s limitations. It is an extremely detailed read but here are some of the things you might want to know. ✅ Problem Framing: Parquet and ORC were built for hardware/workload assumptions that don’t hold anymore. Cloud object storage, wide ML tables, vector embeddings & random access make them inefficient. ✅ Core Principles: F3 is built on interoperability, extensibility, and efficiency - exactly the gaps newer formats have been trying to fill. ✅ Metadata Redesign: FlatBuffers replace Thrift/Protobuf for zero-copy column-level access, avoiding full footer deserialization. ✅ Decoupled Layout: F3 introduces IOUnits and EncUnits, breaking the tight coupling of row groups, hence better memory usage, predictable I/O, cloud-friendly flushes. ✅ Flexible Dictionaries: Dictionaries can be local, global or shared across columns, enabling better compression ratios than Parquet’s fixed row-group scope. ✅ Decoding API: A stable, language-agnostic API that always outputs Apache Arrow arrays, hence consistent interoperability across systems. ✅ Wasm-Embedded Decoders: Every F3 file ships with its decoder in WebAssembly, ensuring files are always readable without waiting for library upgrades. ✅ Performance: Faster metadata parsing, competitive compression/throughput & much better random access vs. Parquet/ORC. What’s different here is that the F3 authors explicitly call out that even newer formats like Lance, Nimble, BtrBlocks are repeating the same mistake as Parquet/ORC - locking into present-day assumptions. Their bet is that by embedding extensibility and interoperability into the core (e.g., via Wasm decoders), F3 won’t need to be replaced a decade later. Irrespective of how this will see adoption, I think this reinforces what I highlighted in my blog: the file format layer is being actively re-imagined! Link of the paper & my blog in comments. #dataengineering #softwareengineering

332

6 commentaren

👷‍♂️ 𝐖𝐡𝐨 𝐛𝐮𝐢𝐥𝐭 𝐭𝐡𝐢𝐬 𝐀𝐳𝐮𝐫𝐞 𝐃𝐚𝐭𝐚 𝐅𝐚𝐜𝐭𝐨𝐫𝐲 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞?! Late-night debugging. No documentation. No alerts. Just deeply nested ForEach loops and broken lookups. If you’ve ever inherited an ADF pipeline, you know the pain. 🚨 Real talk: Azure Data Factory is powerful, but without proper naming conventions, logging, and alerting, it quickly becomes a black box of chaos. 💡 Pro tip: Document your pipelines Set alerts for failure triggers Use parameterized datasets Respect your future self (and teammates) 😅 Because when ADF breaks in production, it doesn't whisper... it screams. #DataEngineering #AzureDataFactory #ADF #DataPipelines #TechHumor #CloudEngineering #DevOps #LinkedInHumor #DataOps

396

11 commentaren

Just in case you want to explain the difference between ETL (extract-transform-load) and ELT (extract-load-transform) approaches in data engineering to your 5 y.o. child, I have something for you. It’s actually easy: either you get something, store it, and make sense out of it later; or you get something, make some sense out of it, and store only the results. What is better? Depends on the use case and a few parameters like size of your storage space and complexity of the processing needed. UPD: for more information about ETL and ELT please check the episode of my video-blog, which is dedicated to this topic: https://lnkd.in/eP52bjFc #data #leadership #etl #elt #apples

698

65 commentaren

We finally have 𝘂𝘀𝗲𝗿-𝗱𝗲𝗳𝗶𝗻𝗲𝗱 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝘀 (𝗨𝗗𝗙) in DAX! Don't make mistakes; this is not a user data function in Fabric. This is a new DAX feature, the most important one in the last five years. First article and video to introduce how UDFs in DAX work.

80

1 commentaar

Part 4. I often see data lineage tools do not perform well short during cross-platform data movement. Especially when data pipelines mix legacy and cloud systems. In these cases, data flows can get messy. Example: - mainframe → COBOL files → Azure Delta Lake → Snowflake - COBOL files → Hadoop → Oracle → SQL Server → Tableau Most lineage tools see only parts, not the full picture: legacy hops, parallel paths, and cross-platform transformations are  not captured. While these tools can track data within a single system, they struggle across multiple technologies. As a result, end-to-end lineage is rarely automatic. We need a mix of tools, expertise, and manual mapping to see the whole journey. This incomplete lineage makes impact analysis, governance, and troubleshooting much harder than expected. Part 1 of the common lineage issues: https://lnkd.in/gwPs5Zfq Part2: https://lnkd.in/gQAxhciw

58

20 commentaren

𝗠𝗮𝘀𝘁𝗲𝗿𝗶𝗻𝗴 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 – 𝗧𝗵𝗲 𝗘𝗮𝘀𝘆 𝗪𝗮𝘆! When I first started learning PySpark, it felt overwhelming — so many new terms, distributed computing, lazy evaluation So, I decided to 𝗹𝗲𝗮𝗿𝗻 𝗶𝘁 𝗳𝗿𝗼𝗺 𝘁𝗵𝗲 𝗴𝗿𝗼𝘂𝗻𝗱 𝘂𝗽 — 𝘀𝘁𝗲𝗽 𝗯𝘆 𝘀𝘁𝗲𝗽, 𝘁𝗵𝗲 𝗳𝗼𝘂𝗻𝗱𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝘄𝗮𝘆. And it worked. I’ve documented this journey and compiled a simple, beginner-friendly 𝗣𝗗𝗙 𝘄𝗶𝘁𝗵 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗯𝗮𝘀𝗶𝗰𝘀 𝗮𝗻𝗱 𝗰𝗼𝗱𝗲 𝘀𝗻𝗶𝗽𝗽𝗲𝘁𝘀. 📘 Whether you’re just getting started or need a quick refresher — this PDF will help you: ✅ Understand SparkSession ✅ Read data from CSV ✅ Perform basic transformations ✅ Write data back ✅ And more... 🧠 This foundational approach made PySpark click for me. I hope it helps you too! Please follow Md Sohail Tanveer for more informative content #PySpark #BigData #DataEngineering #ApacheSpark #LearningByDoing #CareerInData

72

1 commentaar

𝐋𝐢𝐬𝐭 𝐯𝐬 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐨𝐫 𝐢𝐧 𝐏𝐲𝐭𝐡𝐨𝐧, 𝐰𝐡𝐚𝐭’𝐬 𝐭𝐡𝐞 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐜𝐞? 𝐥𝐢𝐬𝐭 holds everything in memory at once. 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐨𝐫 gives you one item at a time, only when needed. When working with large data, this difference matters. Generators are more memory efficient and better for performance. #waeldagash #dataEngineering

32

3 commentaren

#DataWarehouse, #DataLake, and #DataMesh ..... What’s the Difference? If you are working in the data space, you have probably come across these terms. While they often get used interchangeably, each serves a very different purpose. A Data Warehouse is a centralized system that stores clean, structured, and processed data from different sources. It is designed for reporting, business intelligence, and analytics. Think of it as a well-organized library where everything is neatly labeled and easy to search. In contrast, a Data Lake is built to handle large volumes of raw data in any format structured, semi-structured, or unstructured. It is widely used in big data and machine learning environments where flexibility is more important than immediate usability. Imagine it as a giant reservoir where all data flows in, but it needs further processing before it becomes useful. Then comes the Data Mesh, which takes a different approach. Rather than relying on a single, centralized data team, Data Mesh promotes a decentralized architecture. Each business domain like marketing, sales, or HR owns and manages its own data as a product, while following shared standards and governance. This model scales well in large organizations and encourages accountability and faster data delivery. In short, a data warehouse gives you reliable reports, a data lake gives you flexibility with raw data, and a data mesh gives you scalability through distributed ownership. Each has its place in a modern data architecture and often, organizations use a combination of all three to meet different needs. Which one are you using or moving towards in your organization? Avinash S. #DataEngineering #DataArchitecture #DataWarehouse #DataLake #DataMesh #Analytics #CloudDataPlatform #ModernDataStack

67

4 commentaren

⭕ 𝗗𝗔𝗧𝗔𝗕𝗥𝗜𝗖𝗞𝗦 𝗗𝗔𝗧𝗔 𝗘𝗡𝗚𝗜𝗡𝗘𝗘𝗥 𝗜𝗡𝗧𝗘𝗥𝗩𝗜𝗘𝗪 𝗤𝗨𝗘𝗦𝗧𝗜𝗢𝗡𝗦 - 𝗥𝗘𝗖𝗘𝗡𝗧 𝗨𝗣𝗗𝗔𝗧𝗘𝗦 ⭕ Preparing for a Databricks Data Engineer interview? - This 3+ HOUR MASTERCLASS is packed with the latest concepts you need to know in 2025! What you’ll learn: ✅ Declarative Pipelines – simplify complex workflows ✅ Serverless Compute – run workloads without managing clusters ✅ Streaming Tables & EXPECTATIONS – handle real-time data like a pro ✅ Alerts & Monitoring – stay on top of data quality & system health ✅ System Design & Architecture – design scalable, robust solutions ✅ Dynamic Queries with Jobs – make jobs flexible & reusable 📍 Find the video link in the COMMENTS ▶️ Watch, take notes, and boost your chances of landing that dream role #databricks #dataengineer #interview

187

10 commentaren

🚀 𝗦𝗽𝗮𝗿𝗸 𝗜𝗻𝗰𝗿𝗲𝗺𝗲𝗻𝘁𝗮𝗹 𝗟𝗼𝗮𝗱𝘀 𝗝𝘂𝘀𝘁 𝗚𝗼𝘁 𝗤𝘂𝗶𝗰𝗸𝗲𝗿 & 𝗖𝗹𝗲𝗮𝗻𝗲𝗿! 🚀 Tired of reprocessing your entire dataset every time you need to update your analytics? When dealing with large volumes of data, especially from cloud storage, efficient incremental loading is key to performance and cost savings. One of the most elegant and powerful ways to achieve this in Databricks Spark, particularly with Auto Loader, is by leveraging hashtag #𝗳𝗶𝗹𝗲_𝗺𝗼𝗱𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻_𝘁𝗶𝗺𝗲 𝗮𝗻𝗱 𝘁𝗵𝗲 𝗺𝗼𝗱𝗶𝗳𝗶𝗲𝗱𝗔𝗳𝘁𝗲𝗿 option. 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗮𝗽𝗽𝗿𝗼𝗮𝗰𝗵 𝗶𝘀 𝗮 𝗴𝗮𝗺𝗲-𝗰𝗵𝗮𝗻𝗴𝗲𝗿: * Precision Loading: Instead of blindly scanning all historical files, modifiedAfter allows you to tell Auto Loader exactly where to start – only processing files that have been modified (or created) after a specific timestamp. * Optimized Initial Scans: For massive source directories, this drastically reduces the time taken for the initial scan when your stream first starts or restarts. No more sifting through years of old data! * Clean & Efficient Data Pipelines: By focusing only on new or updated data, you streamline your ingestion process, leading to faster job execution and less resource consumption. * Simplicity with Auto Loader: Auto Loader's robust checkpointing combined with modifiedAfter provides a nearly hands-off experience for maintaining exactly-once processing guarantees for your incremental data. How it works (in essence): You simply set the modifiedAfter option in your spark.readStream.format("cloudFiles") call with a precise timestamp. Auto Loader then intelligently filters out anything older than that time during its initial discovery phase. This method is particularly effective for scenarios where new data arrives as new files or existing files are updated (if cloudFiles.allowOverwrites is configured carefully). If you're building data lakes or data warehouses on Databricks, mastering incremental loads with modifiedAfter is a must for building scalable and cost-effective data pipelines. #Databricks #SparkSQL #DataEngineering #DataPipelines #IncrementalLoad #AutoLoader #BigData #DataLake #CloudData #AnalyticsEngineering

143

1 commentaar

Running dbt models on Snowflake? Did you know you can easily leverage the query tag to get cost insights per dbt model. Here's my process and code to: - Identify which dbt models burn through those credits 🔥 - Understand how much your dbt tests are costing you 💸 - Calculate how much you're paying for 'empty' warehouses 📦 https://lnkd.in/eqAdqMEt

41

1 commentaar