Daniel Bravo Capote

Don’t get confused while learning PySpark. Understand these core concepts step by step to become truly proficient in PySpark. 𝗕𝗮𝘀𝗶𝗰𝘀 𝗼𝗳 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - PySpark Architecture - SparkContext and SparkSession - RDDs (Resilient Distributed Datasets) - DataFrames - Transformations and Actions - Lazy Evaluation 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗗𝗮𝘁𝗮𝗙𝗿𝗮𝗺𝗲𝘀: - Creating DataFrames - Reading Data from CSV, JSON, Parquet - DataFrame Operations - Filtering, Selecting, and Aggregating Data - Joins and Merging DataFrames - Working with Null Values 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗖𝗼𝗹𝘂𝗺𝗻 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Defining and Using UDFs (User Defined Functions) - Column Operations (Select, Rename, Drop) - Handling Complex Data Types (Array, Map) - Working with Dates and Timestamps 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗦𝗵𝘂𝗳𝗳𝗹𝗲 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Understanding Partitions - Repartitioning and Coalescing - Managing Shuffle Operations - Optimizing Partition Sizes for Performance 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗮𝗻𝗱 𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗶𝗻𝗴 𝗗𝗮𝘁𝗮: - When to Cache or Persist - Memory vs Disk Caching - Checking Storage Levels 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗪𝗶𝘁𝗵 𝗦𝗤𝗟: - Spark SQL Introduction - Creating Temp Views - Running SQL Queries - Optimizing SQL Queries with Catalyst Optimizer - Working with Hive Tables in PySpark 𝗪𝗼𝗿𝗸𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗗𝗮𝘁𝗮 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Data Cleaning and Preparation - Handling Missing Values - Data Normalization and Transformation - Working with Categorical Data 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗧𝗼𝗽𝗶𝗰𝘀 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Broadcasting Variables - Accumulators - PySpark Window Functions - PySpark with Machine Learning (MLlib) - Working with Streaming Data (Spark Streaming) 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗧𝘂𝗻𝗶𝗻𝗴 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Understanding Job, Stage, and Task - Tungsten Execution Engine - Memory Management and Garbage Collection - Tuning Parallelism - Using Spark UI for Performance Monitoring Once you master these PySpark fundamentals, you won’t just write code — you’ll design efficient, scalable, and production-ready data pipelines that harness the full potential of Spark.

268

33 comentarios

You can go from 0 -> PySpark pro with just these four steps. No expensive courses needed. Here’s how you do it 👇 1️⃣ Understand What Spark Is (and WHY you need it) - Spark is the backbone of big data processing - You must know why it matters for real-world data jobs Start here → 🔗 https://lnkd.in/g47yWA5v 2️⃣ Learn What PySpark Is (and how it works inside Spark) - PySpark lets you use Python with Spark - See how your code runs behind the scenes Watch this → 🔗 https://lnkd.in/gSWt2Mbh 3️⃣ Use the BEST PySpark Learning Resources - Manish Kumar | YT Playlist 🔗 https://lnkd.in/gQJqD_mr - Rajas Data Engineering | YT Playlist 🔗 https://lnkd.in/gan3FkTK - Subham Khandelwal | YT Playlist 🔗 https://lnkd.in/gZdvSVb8 - SparkByExamples | Website 🔗 https://lnkd.in/gbxzq5gE 4️⃣ PRACTICE, PRACTICE, PRACTICE - Real hands-on projects are where you grow - Use these to get started: 🔗 https://lnkd.in/gjXBVbmf 🔗 https://lnkd.in/g7Sm2Rcc Trust ME, you do NOT need to spend big money to master PySpark. You need good resources and focused practice (consistency is key!). Let me know if you have any other top resources for learning PySpark or want to add to this list! 🎯 𝗖𝗼𝗻𝗻𝗲𝗰𝘁 𝟭:𝟭 𝗳𝗼𝗿 𝗰𝗮𝗿𝗲𝗲𝗿 𝗴𝘂𝗶𝗱𝗮𝗻𝗰𝗲 → https://lnkd.in/gH4DeYb4 🎯 𝗔𝗧𝗦 𝗢𝗽𝘁𝗶𝗺𝗶𝘀𝗲𝗱 𝗿𝗲𝘀𝘂𝗺𝗲 𝘁𝗲𝗺𝗽𝗹𝗮𝘁𝗲 → https://lnkd.in/g-iw7FaQ ♻️ Liked it? Repost this! ➕ Follow for more daily insights on how to grow your career in the data field

294

11 comentarios

Just in case you want to explain the difference between ETL (extract-transform-load) and ELT (extract-load-transform) approaches in data engineering to your 5 y.o. child, I have something for you. It’s actually easy: either you get something, store it, and make sense out of it later; or you get something, make some sense out of it, and store only the results. What is better? Depends on the use case and a few parameters like size of your storage space and complexity of the processing needed. UPD: for more information about ETL and ELT please check the episode of my video-blog, which is dedicated to this topic: https://lnkd.in/eP52bjFc #data #leadership #etl #elt #apples

699

65 comentarios

Write-Audit-Publish Pattern in Apache Iceberg. Data quality is non-negotiable in any data engineering workflow. It directly impacts downstream workloads like BI reports or ML pipelines. Bad data in production tables can have serious impact. That’s why validation before publishing to production is critical. And this is where the Write-Audit-Publish (WAP) pattern comes in! WAP introduces a safety step between writing and publishing: ✅ Write - New data is ingested into a staging area. ✅ Audit - Data quality checks (schema validation, null checks, deduplication, business rules) are run on this staged version. ✅ Publish - If checks pass, the data is atomically committed into the main table. If not, the staged data is discarded. This ensures production tables remain consistent even if incoming data batches contain errors. Once you understand the value of WAP, the next question becomes: how is this realized in practice? Different systems approach it differently - some rely on shallow clones or pre-commit validators, others build it into orchestration. Apache Iceberg provides native support for WAP at the table format level. Iceberg implements WAP in two ways: 1. WAP.id (legacy approach): - Writers set a WAP.id during commit. - The commit stays invisible to readers until a publish step promotes it. - This is less flexible for multi-branch workflows. 2. Branching: - Writers commit new data to a dedicated branch while main stays unchanged. - Validation queries run directly against the branch. - If successful, the branch can be fast-forwarded to main. If not, simply drop the branch. This aligns with Git semantics and enables parallel ingestion or experimentation. By combining atomic commits with branching semantics, WAP in Iceberg gives data engineers a practical mechanism to stage, validate & promote data safely to production. I wrote about this sometime back. Blog link in comments. #dataengineering #softwareengineering

215

2 comentarios

We finally have 𝘂𝘀𝗲𝗿-𝗱𝗲𝗳𝗶𝗻𝗲𝗱 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝘀 (𝗨𝗗𝗙) in DAX! Don't make mistakes; this is not a user data function in Fabric. This is a new DAX feature, the most important one in the last five years. First article and video to introduce how UDFs in DAX work.

80

1 comentario

🚀 𝗗𝗲𝗺𝘆𝘀𝘁𝗶𝗳𝘆𝗶𝗻𝗴 𝗟𝗮𝗸𝗲𝗵𝗼𝘂𝘀𝗲 𝗙𝗲𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝗶𝗻 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 Curious about how 𝗾𝘂𝗲𝗿𝘆 𝗳𝗲𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻 and 𝗰𝗮𝘁𝗮𝗹𝗼𝗴 𝗳𝗲𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻 work in Databricks? 🤔 I’ve put together a 𝗾𝘂𝗶𝗰𝗸 𝗴𝘂𝗶𝗱𝗲 that explains: 🔹 What Lakehouse Federation is 🔹 Key differences between Query vs Catalog Federation 🔹 Practical use cases for both 🔹 Simple setup steps to get started Whether you’re a 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿, 𝗮𝗻𝗮𝗹𝘆𝘀𝘁, 𝗼𝗿 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁, this PDF will help you understand how to run queries seamlessly across multiple data sources without moving all your data into one place. 📥 Download & Read the PDF: [Attached] 💡 𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 for more insights on Databricks, Azure Data, Data Engineering & AI. #Databricks #Lakehouse #DataEngineering #AzureData #BigData #CloudComputing #AI #MachineLearning #DataArchitecture #TechLearning #MicrosoftFabric #LearningOnLinkedIn

65

1 comentario

𝗠𝗮𝘀𝘁𝗲𝗿𝗶𝗻𝗴 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 – 𝗧𝗵𝗲 𝗘𝗮𝘀𝘆 𝗪𝗮𝘆! When I first started learning PySpark, it felt overwhelming — so many new terms, distributed computing, lazy evaluation So, I decided to 𝗹𝗲𝗮𝗿𝗻 𝗶𝘁 𝗳𝗿𝗼𝗺 𝘁𝗵𝗲 𝗴𝗿𝗼𝘂𝗻𝗱 𝘂𝗽 — 𝘀𝘁𝗲𝗽 𝗯𝘆 𝘀𝘁𝗲𝗽, 𝘁𝗵𝗲 𝗳𝗼𝘂𝗻𝗱𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝘄𝗮𝘆. And it worked. I’ve documented this journey and compiled a simple, beginner-friendly 𝗣𝗗𝗙 𝘄𝗶𝘁𝗵 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗯𝗮𝘀𝗶𝗰𝘀 𝗮𝗻𝗱 𝗰𝗼𝗱𝗲 𝘀𝗻𝗶𝗽𝗽𝗲𝘁𝘀. 📘 Whether you’re just getting started or need a quick refresher — this PDF will help you: ✅ Understand SparkSession ✅ Read data from CSV ✅ Perform basic transformations ✅ Write data back ✅ And more... 🧠 This foundational approach made PySpark click for me. I hope it helps you too! Please follow Md Sohail Tanveer for more informative content #PySpark #BigData #DataEngineering #ApacheSpark #LearningByDoing #CareerInData

72

1 comentario

🧊  I’m excited to announce the official launch of Ryft Adaptive Optimization for Apache Iceberg! “All optimized Iceberg tables are alike; each unoptimized table is unoptimized in its own way” Every table is unique, with different workloads, write patterns, data characteristics and query usage patterns. This is what makes managing Iceberg tables so manual and complex. You find yourself in an endless cycle of monitoring and manual tuning - adjusting compaction schedules, debugging failed maintenance jobs, tweaking compression settings, optimizing file sizes, indexing columns, and reordering data. With Ryft Adaptive Optimization, engineers don’t need to think about when, how and with what parameters to run maintenance and optimize Iceberg tables. It’s all automatic. Ryft's engine learns the unique characteristics of each table and automatically adapts optimization strategies accordingly. This results in up to 5× faster queries, ~7× higher compaction efficiency, up to 10× storage reduction, and mostly - zero manual management. Originally, we set out to build the simplest solution for managing Apache Iceberg Tables. Today, we’re moving one step closer towards making data management simple, and I can’t wait to share with you what else we have in the works over the next few months! One last thing - I hate benchmarks. Never trust a vendor posting a benchmark. That's why we decided to share real world results from what we saw with our customers. For more details, read the full blog post: https://lnkd.in/gTXmiWPu

188

28 comentarios

#DataWarehouse, #DataLake, and #DataMesh ..... What’s the Difference? If you are working in the data space, you have probably come across these terms. While they often get used interchangeably, each serves a very different purpose. A Data Warehouse is a centralized system that stores clean, structured, and processed data from different sources. It is designed for reporting, business intelligence, and analytics. Think of it as a well-organized library where everything is neatly labeled and easy to search. In contrast, a Data Lake is built to handle large volumes of raw data in any format structured, semi-structured, or unstructured. It is widely used in big data and machine learning environments where flexibility is more important than immediate usability. Imagine it as a giant reservoir where all data flows in, but it needs further processing before it becomes useful. Then comes the Data Mesh, which takes a different approach. Rather than relying on a single, centralized data team, Data Mesh promotes a decentralized architecture. Each business domain like marketing, sales, or HR owns and manages its own data as a product, while following shared standards and governance. This model scales well in large organizations and encourages accountability and faster data delivery. In short, a data warehouse gives you reliable reports, a data lake gives you flexibility with raw data, and a data mesh gives you scalability through distributed ownership. Each has its place in a modern data architecture and often, organizations use a combination of all three to meet different needs. Which one are you using or moving towards in your organization? Avinash S. #DataEngineering #DataArchitecture #DataWarehouse #DataLake #DataMesh #Analytics #CloudDataPlatform #ModernDataStack

67

4 comentarios

𝐋𝐢𝐬𝐭 𝐯𝐬 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐨𝐫 𝐢𝐧 𝐏𝐲𝐭𝐡𝐨𝐧, 𝐰𝐡𝐚𝐭’𝐬 𝐭𝐡𝐞 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐜𝐞? 𝐥𝐢𝐬𝐭 holds everything in memory at once. 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐨𝐫 gives you one item at a time, only when needed. When working with large data, this difference matters. Generators are more memory efficient and better for performance. #waeldagash #dataEngineering

32

3 comentarios

#Delivering End-to-End Data Engineering with Azure #2025 ----------------------------------------------------------------------------------- . . . . . Recently, I worked on an Azure Data Engineering project where the goal was to build a scalable ETL pipeline for handling large volumes of e-commerce data. Here’s a quick breakdown of how I approached it as a Senior Data Engineer: ✅ Data Ingestion – Used Azure Data Factory to orchestrate ingestion from multiple sources (APIs, SQL DB, and Blob Storage). ✅ Transformation – Leveraged Azure Databricks (PySpark & Delta Lake) for cleaning, joining, and transforming raw data into analytics-ready datasets. ✅ Data Storage – Designed a Lakehouse architecture: - Raw data → Azure Data Lake Gen2 - Curated & transformed → Delta tables in Databricks ✅ Analytics Layer – Exposed the data for the Power BI team via Azure Synapse, enabling real-time dashboards. ✅ CI/CD – Automated deployments with Azure DevOps pipelines to ensure consistency across environments. 📌 Key Learning: Having a well-defined orchestration strategy + Delta Lake reduced processing costs and improved data reliability. 👉 Question for the community: What’s your preferred tool for data transformation in Azure – Databricks, Synapse, or SQL-based ELT? #Azure #DataEngineering #Databricks #AzureDataFactory #PowerBI #SeniorDataEngineer

25

2 comentarios

⭕ 𝗗𝗔𝗧𝗔𝗕𝗥𝗜𝗖𝗞𝗦 𝗗𝗔𝗧𝗔 𝗘𝗡𝗚𝗜𝗡𝗘𝗘𝗥 𝗜𝗡𝗧𝗘𝗥𝗩𝗜𝗘𝗪 𝗤𝗨𝗘𝗦𝗧𝗜𝗢𝗡𝗦 - 𝗥𝗘𝗖𝗘𝗡𝗧 𝗨𝗣𝗗𝗔𝗧𝗘𝗦 ⭕ Preparing for a Databricks Data Engineer interview? - This 3+ HOUR MASTERCLASS is packed with the latest concepts you need to know in 2025! What you’ll learn: ✅ Declarative Pipelines – simplify complex workflows ✅ Serverless Compute – run workloads without managing clusters ✅ Streaming Tables & EXPECTATIONS – handle real-time data like a pro ✅ Alerts & Monitoring – stay on top of data quality & system health ✅ System Design & Architecture – design scalable, robust solutions ✅ Dynamic Queries with Jobs – make jobs flexible & reusable 📍 Find the video link in the COMMENTS ▶️ Watch, take notes, and boost your chances of landing that dream role #databricks #dataengineer #interview

187

10 comentarios