Daan Bakboord

Databricks on Azure is everywhere , stealing the spotlight and making waves in the market! But let’s be honest, it can be tricky to truly understand these concepts until you dive into them yourself. That’s where these free end-to-end projects come in your shortcut to clarity and hands-on mastery! 🚀. Believe me, these projects are cutting-edge and align with the latest technologies. 𝗕𝘆 𝗲𝘅𝗽𝗹𝗼𝗿𝗶𝗻𝗴 𝘁𝗵𝗲𝘀𝗲 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀, 𝘆𝗼𝘂’𝗹𝗹 𝗴𝗮𝗶𝗻 𝗮 𝗰𝗼𝗺𝗽𝗿𝗲𝗵𝗲𝗻𝘀𝗶𝘃𝗲 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 𝗼𝗳 𝗸𝗲𝘆 𝗔𝘇𝘂𝗿𝗲 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝘀𝗲𝗿𝘃𝗶𝗰𝗲𝘀 𝘄𝗶𝘁𝗵 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗰𝗼𝗻𝗰𝗲𝗽𝘁𝘀 , 𝗶𝗻𝗰𝗹𝘂𝗱𝗶𝗻𝗴: ▪️ How Databricks works and its integration with Azure ▪️ Delta Lake and its benefits ▪️ Full and incremental data loads ▪️ Implementing SCD Type 2 ▪️ CI/CD DevOps integration ▪️ Lakehouse Medallion Architecture with Bronze, Silver, and Gold layers ▪️ Building ETL pipelines using Azure Data Factory ▪️ An overview of PySpark ▪️ Data Governance with unity catalog ▪️ Insights into Azure Synapse Analytics …and so much more! 𝗖𝗵𝗲𝗰𝗸 𝗼𝘂𝘁 𝘁𝗵𝗲𝘀𝗲 𝗳𝗿𝗲𝗲 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀. 👉 End to End Azure Data Engineering Project by Sumit Sir (Sumit Mittal) 🔗 https://lnkd.in/giN6pEEy 👉 Azure End-To-End Data Engineering Project (Job Ready) | Azure Data Engineering Bootcamp (Ansh Lamba) 🔗 https://lnkd.in/gJfPHQsV 👉 Olympic Data Analytics | Azure End-To-End Data Engineering Project (Darshil Parmar ) 🔗 https://lnkd.in/gb_gwT3R 👉 An End to End Azure Data Engineering Real Time Project Demo | Get Hired as an Azure Data Engineer ( Mr. K Talks Tech ) 🔗 https://lnkd.in/gQ7RT5kJ 👉 End to End Big Data Engineering Project with Azure | Big Data Engineering (From Scratch) (Mayank Aggarwal ) 🔗 https://lnkd.in/gTJFAu64 👉 Azure Data Engineer Project | Azure Data Engineer End to End Project | Azure data Engineer Job ( Aditya Chandak) 🔗 https://lnkd.in/gsPWybWw 👉 Azure End-To-End Data Engineering Project for Beginners (FREE Account) | SQL DB Tutorial (Luke J. Byrne) 🔗 https://lnkd.in/g-gPtg2P 👉 Azure End-To-End Data Engineering Project (From Scratch!) (Ansh Lamba) 🔗 https://lnkd.in/gAQaJpZm 👉 Football Data Analytics | Azure End To End Data Engineering Project ( 𝐂𝐨𝐝𝐞𝐖𝐢𝐭𝐡𝐘𝐮 (Yusuf Ganiyu) 🔗 https://lnkd.in/gNHrhDV5 Dive in to elevate your skills and master the world of Azure data engineering! 📕 Sharing 'Azure Databricks End to End Project' for your learning reference. Doc Credits : Linkedin Respective Creator. ♻️ Repost this to help the Job Seekers. P.S : I share valuable job search tips, industry insights, learning resources, and the latest updates in the data domain – all for free. Join thousands of other readers here → https://lnkd.in/g-ZtB4Yf Lets Learn Azure Databricks !! #dataengineer #azure #databricks #projects #data #azuredataengineer #cloud #microsoft #cloudcomputing

464

43 commentaren

Just Published a New Blog "Apache Parquet Vs Newer File Formats" 🎉 There’s no denying the impact Apache Parquet has had on the analytics ecosystem. - Columnar layout - Strong compression & encoding - The de facto format in data lakes and warehouses for over a decade - Open Table formats like Apache Iceberg, Hudi & Delta uses them But there are also a lot of new things happening with our analytical workloads. ❌ Workloads are no longer just batch analytics - they involve AI pipelines, low-latency use cases, and hardware-accelerated compute ❌ Hardware is not only CPU but also GPU, from x86 to ARM/RISC-V, with wide SIMD instruction sets becoming the norm ❌ And performance bottlenecks are seen in decompression speed, memory pressure and lack of vectorized execution paths As a result, we’re seeing a wave of innovation around next-gen file formats. I have been exploring/understanding some of these newer formats. So, I wrote about them. Link in comments! #dataengineering #softwareengineering

247

11 commentaren

Just in case you want to explain the difference between ETL (extract-transform-load) and ELT (extract-load-transform) approaches in data engineering to your 5 y.o. child, I have something for you. It’s actually easy: either you get something, store it, and make sense out of it later; or you get something, make some sense out of it, and store only the results. What is better? Depends on the use case and a few parameters like size of your storage space and complexity of the processing needed. UPD: for more information about ETL and ELT please check the episode of my video-blog, which is dedicated to this topic: https://lnkd.in/eP52bjFc #data #leadership #etl #elt #apples

698

65 commentaren

Part 4. I often see data lineage tools do not perform well short during cross-platform data movement. Especially when data pipelines mix legacy and cloud systems. In these cases, data flows can get messy. Example: - mainframe → COBOL files → Azure Delta Lake → Snowflake - COBOL files → Hadoop → Oracle → SQL Server → Tableau Most lineage tools see only parts, not the full picture: legacy hops, parallel paths, and cross-platform transformations are  not captured. While these tools can track data within a single system, they struggle across multiple technologies. As a result, end-to-end lineage is rarely automatic. We need a mix of tools, expertise, and manual mapping to see the whole journey. This incomplete lineage makes impact analysis, governance, and troubleshooting much harder than expected. Part 1 of the common lineage issues: https://lnkd.in/gwPs5Zfq Part2: https://lnkd.in/gQAxhciw

58

20 commentaren

We finally have 𝘂𝘀𝗲𝗿-𝗱𝗲𝗳𝗶𝗻𝗲𝗱 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝘀 (𝗨𝗗𝗙) in DAX! Don't make mistakes; this is not a user data function in Fabric. This is a new DAX feature, the most important one in the last five years. First article and video to introduce how UDFs in DAX work.

80

1 commentaar

New data infrastructure lore just dropped: The Pendantic Layer. Joe Reis has a great article introducing this idea [https://lnkd.in/gaXw_TPr], but this quote sums it up perfectly. "The Pedantic Layer slows us down by focusing attention on taxonomy and semantics over outcomes. It makes practitioners feel like they need to pass some purity test of definitions before they can get value. It feeds a culture where thought leaders posture about whether CSVs are structured “enough,” while the practitioners who ignore the debate are quietly building models that drive millions in revenue." Coming from startups where you are obsessed with execution over perfection, the "pedantic layer" has been one of my largest gripes with the wider data industry. In celebration of yet another made-up layer (YAML), leave a comment about your favorite data concept, and I'll tell you why you are wrong for obscure nuances that don't apply to most use cases.

44

11 commentaren

Don't write a 1000-line SQL Query. Build a data transformation pipeline with modular components, version control, and column-level lineage. Cramming every edge case into one SELECT statement is exhausting. Modularity with maintainability is where it's at. Ready to get started? / Break monoliths into small, named models / Reuse logic functions and macros / Add tests for keys, nulls, ranges, and referential integrity / Put everything in version control / Document inputs, outputs, owners, and assumptions / Make sure column-level lineage is visible / Boring predictability > latest tech tricks Want to level up? Follow - → Zach Wilson at DataExpert.io → Tobias (Toby) Mao at Tobiko → Benjamin Rogojan at Seattle Data Guy → Xinran Waibel at Data Engineer Things → Carly Taylor, MSc at Rebel Data Science → Madison Schott at Learn Analytics Engineering Happy Engineering! #dataengineering #datatransformation #sqlmesh Tobiko

84

12 commentaren

Running dbt models on Snowflake? Did you know you can easily leverage the query tag to get cost insights per dbt model. Here's my process and code to: - Identify which dbt models burn through those credits 🔥 - Understand how much your dbt tests are costing you 💸 - Calculate how much you're paying for 'empty' warehouses 📦 https://lnkd.in/eqAdqMEt

41

1 commentaar

NODUPKEY vs NODUP in #SAS PROC SORT Many SAS users confuse these options. Here's what sets them apart: NODUPKEY: - Removes duplicates based only on BY variables - Keeps first observation when BY values match - Ignores non-BY variables during comparison NODUP: - Removes duplicates based on all variables - Keeps first observation when entire record is identical - Requires 100% match across every column Key Takeaway: NODUPKEY ensures uniqueness at the key level. NODUP eliminates only exact duplicate rows. Choosing the wrong option can cause unintended data loss or retain duplicates. Example: data test; input id marks; cards; 1 45 1 56 1 62 2 65 2 65 2 45 ; run; proc print data = test; title "Sample Data"; run; proc sort data = test out= test_1 nodupkey; by id; run; proc print data = test_1; title "Nodupkey"; run; proc sort data = test out= test_2 nodup; by id; run; proc print data = test_2; title "Nodup"; run; Understanding this distinction is essential for data integrity in your analytical workflows. #datascience #dataengineering #analytics #dataanalytics #dataanalyst #sql #data #learning #growthmindset

22

1 commentaar