Negin Fazel

Don’t get confused while learning PySpark. Understand these core concepts step by step to become truly proficient in PySpark. 𝗕𝗮𝘀𝗶𝗰𝘀 𝗼𝗳 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - PySpark Architecture - SparkContext and SparkSession - RDDs (Resilient Distributed Datasets) - DataFrames - Transformations and Actions - Lazy Evaluation 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗗𝗮𝘁𝗮𝗙𝗿𝗮𝗺𝗲𝘀: - Creating DataFrames - Reading Data from CSV, JSON, Parquet - DataFrame Operations - Filtering, Selecting, and Aggregating Data - Joins and Merging DataFrames - Working with Null Values 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗖𝗼𝗹𝘂𝗺𝗻 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Defining and Using UDFs (User Defined Functions) - Column Operations (Select, Rename, Drop) - Handling Complex Data Types (Array, Map) - Working with Dates and Timestamps 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗦𝗵𝘂𝗳𝗳𝗹𝗲 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Understanding Partitions - Repartitioning and Coalescing - Managing Shuffle Operations - Optimizing Partition Sizes for Performance 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗮𝗻𝗱 𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗶𝗻𝗴 𝗗𝗮𝘁𝗮: - When to Cache or Persist - Memory vs Disk Caching - Checking Storage Levels 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗪𝗶𝘁𝗵 𝗦𝗤𝗟: - Spark SQL Introduction - Creating Temp Views - Running SQL Queries - Optimizing SQL Queries with Catalyst Optimizer - Working with Hive Tables in PySpark 𝗪𝗼𝗿𝗸𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗗𝗮𝘁𝗮 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Data Cleaning and Preparation - Handling Missing Values - Data Normalization and Transformation - Working with Categorical Data 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗧𝗼𝗽𝗶𝗰𝘀 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Broadcasting Variables - Accumulators - PySpark Window Functions - PySpark with Machine Learning (MLlib) - Working with Streaming Data (Spark Streaming) 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗧𝘂𝗻𝗶𝗻𝗴 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Understanding Job, Stage, and Task - Tungsten Execution Engine - Memory Management and Garbage Collection - Tuning Parallelism - Using Spark UI for Performance Monitoring Once you master these PySpark fundamentals, you won’t just write code — you’ll design efficient, scalable, and production-ready data pipelines that harness the full potential of Spark.

268

33 Kommentare

Just in case you want to explain the difference between ETL (extract-transform-load) and ELT (extract-load-transform) approaches in data engineering to your 5 y.o. child, I have something for you. It’s actually easy: either you get something, store it, and make sense out of it later; or you get something, make some sense out of it, and store only the results. What is better? Depends on the use case and a few parameters like size of your storage space and complexity of the processing needed. UPD: for more information about ETL and ELT please check the episode of my video-blog, which is dedicated to this topic: https://lnkd.in/eP52bjFc #data #leadership #etl #elt #apples

699

65 Kommentare

Just Published a New Blog "Apache Parquet Vs Newer File Formats" 🎉 There’s no denying the impact Apache Parquet has had on the analytics ecosystem. - Columnar layout - Strong compression & encoding - The de facto format in data lakes and warehouses for over a decade - Open Table formats like Apache Iceberg, Hudi & Delta uses them But there are also a lot of new things happening with our analytical workloads. ❌ Workloads are no longer just batch analytics - they involve AI pipelines, low-latency use cases, and hardware-accelerated compute ❌ Hardware is not only CPU but also GPU, from x86 to ARM/RISC-V, with wide SIMD instruction sets becoming the norm ❌ And performance bottlenecks are seen in decompression speed, memory pressure and lack of vectorized execution paths As a result, we’re seeing a wave of innovation around next-gen file formats. I have been exploring/understanding some of these newer formats. So, I wrote about them. Link in comments! #dataengineering #softwareengineering

253

11 Kommentare

#DataWarehouse, #DataLake, and #DataMesh ..... What’s the Difference? If you are working in the data space, you have probably come across these terms. While they often get used interchangeably, each serves a very different purpose. A Data Warehouse is a centralized system that stores clean, structured, and processed data from different sources. It is designed for reporting, business intelligence, and analytics. Think of it as a well-organized library where everything is neatly labeled and easy to search. In contrast, a Data Lake is built to handle large volumes of raw data in any format structured, semi-structured, or unstructured. It is widely used in big data and machine learning environments where flexibility is more important than immediate usability. Imagine it as a giant reservoir where all data flows in, but it needs further processing before it becomes useful. Then comes the Data Mesh, which takes a different approach. Rather than relying on a single, centralized data team, Data Mesh promotes a decentralized architecture. Each business domain like marketing, sales, or HR owns and manages its own data as a product, while following shared standards and governance. This model scales well in large organizations and encourages accountability and faster data delivery. In short, a data warehouse gives you reliable reports, a data lake gives you flexibility with raw data, and a data mesh gives you scalability through distributed ownership. Each has its place in a modern data architecture and often, organizations use a combination of all three to meet different needs. Which one are you using or moving towards in your organization? Avinash S. #DataEngineering #DataArchitecture #DataWarehouse #DataLake #DataMesh #Analytics #CloudDataPlatform #ModernDataStack

67

4 Kommentare

❄️#How Snowflake transformed one of my most complex ETL projects ----------------------------------------------------------------------------------- . . . . . . Every growing business eventually hits the same wall: data pipelines that can’t keep up with scale. One of my recent clients faced this challenge. Their ETL pipelines were slow, expensive, and rigid. Queries often failed under heavy loads, and adding new data sources required weeks of rework. Leadership wanted faster insights at lower cost—but the legacy stack simply couldn’t deliver. We rebuilt their architecture with Snowflake at the center: 🔹 AWS S3 for raw data ingestion and secure storage. 🔹 Databricks for heavy transformations, leveraging Spark to process billions of records. 🔹 Snowflake as the cloud data warehouse, powering clean, analytics-ready datasets. What made Snowflake a game-changer? Separation of compute & storage → ETL pipelines ran independently of analyst queries. Virtual warehouses → Scaled compute up or down in minutes. Time Travel & Zero-Copy Cloning → Allowed safe testing, debugging, and rollback with no extra cost. Seamless integration → Fit perfectly with AWS + Databricks for a modern data ecosystem. ✅ Pipelines that once took 5+ hours finished in under 40 minutes. ✅ Query performance improved by 10x. ✅ Infrastructure costs dropped significantly thanks to autoscaling. 💡 Takeaway: Snowflake isn’t just a data warehouse—it’s a business enabler. When paired with the right orchestration and transformation tools, it unlocks speed, scalability, and cost-efficiency. 👉 If you’re looking to modernize your ETL pipelines or migrate to a Snowflake-based data platform, let’s connect. Always happy to share insights—or help design a solution tailored to your needs. #DataEngineering #Snowflake #AWS #Databricks #ETL #BigData #CloudComputing #FreelanceDataEngineer #DataPipeline

34

Disaster Recovery for PostgreSQL databases is built on proven practice, refined through real production experience. In this webinar, Semab Tariq and Warda Bibi will share how to design PostgreSQL DR strategies that deliver reliability and resilience at scale. Drawing from critical deployments, they will cover: ✅ Trusted approaches to safeguard data ✅ Methods to align RPO and RTO with business priorities ✅ Lessons learned from production environments where uptime is essential This session is designed for teams that want their PostgreSQL environments to be fully prepared and resilient. Reserve your spot here: https://lnkd.in/d5_VC_Ah

31

Planning to upgrade SAS Enterprise Guide 8.5 to 8.6? The default encoding had changed to UTF-8 in SAS EG 8.6. However there is now a hotfix available to specify the default encoding to use. Have a read through the SAS Support community thread and hotfix. #SAS #SASusers #encoding #SAStip https://lnkd.in/g--UcdS8

25

𝐋𝐢𝐬𝐭 𝐯𝐬 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐨𝐫 𝐢𝐧 𝐏𝐲𝐭𝐡𝐨𝐧, 𝐰𝐡𝐚𝐭’𝐬 𝐭𝐡𝐞 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐜𝐞? 𝐥𝐢𝐬𝐭 holds everything in memory at once. 𝐠𝐞𝐧𝐞𝐫𝐚𝐭𝐨𝐫 gives you one item at a time, only when needed. When working with large data, this difference matters. Generators are more memory efficient and better for performance. #waeldagash #dataEngineering

32

3 Kommentare

You can go from 0 -> PySpark pro with just these four steps. No expensive courses needed. Here’s how you do it 👇 1️⃣ Understand What Spark Is (and WHY you need it) - Spark is the backbone of big data processing - You must know why it matters for real-world data jobs Start here → 🔗 https://lnkd.in/g47yWA5v 2️⃣ Learn What PySpark Is (and how it works inside Spark) - PySpark lets you use Python with Spark - See how your code runs behind the scenes Watch this → 🔗 https://lnkd.in/gSWt2Mbh 3️⃣ Use the BEST PySpark Learning Resources - Manish Kumar | YT Playlist 🔗 https://lnkd.in/gQJqD_mr - Rajas Data Engineering | YT Playlist 🔗 https://lnkd.in/gan3FkTK - Subham Khandelwal | YT Playlist 🔗 https://lnkd.in/gZdvSVb8 - SparkByExamples | Website 🔗 https://lnkd.in/gbxzq5gE 4️⃣ PRACTICE, PRACTICE, PRACTICE - Real hands-on projects are where you grow - Use these to get started: 🔗 https://lnkd.in/gjXBVbmf 🔗 https://lnkd.in/g7Sm2Rcc Trust ME, you do NOT need to spend big money to master PySpark. You need good resources and focused practice (consistency is key!). Let me know if you have any other top resources for learning PySpark or want to add to this list! 🎯 𝗖𝗼𝗻𝗻𝗲𝗰𝘁 𝟭:𝟭 𝗳𝗼𝗿 𝗰𝗮𝗿𝗲𝗲𝗿 𝗴𝘂𝗶𝗱𝗮𝗻𝗰𝗲 → https://lnkd.in/gH4DeYb4 🎯 𝗔𝗧𝗦 𝗢𝗽𝘁𝗶𝗺𝗶𝘀𝗲𝗱 𝗿𝗲𝘀𝘂𝗺𝗲 𝘁𝗲𝗺𝗽𝗹𝗮𝘁𝗲 → https://lnkd.in/g-iw7FaQ ♻️ Liked it? Repost this! ➕ Follow for more daily insights on how to grow your career in the data field

294

11 Kommentare

❄️ Do you know - Snowflake Cortex Functions can now be used to process any type of images, audio-video files? You can just upload it to cloud storage and run Cortex functions to generate metadata, predictions, summarize , or classify them. Snowflake Cortex is a suite of AI features that use large language models (LLMs) to understand unstructured data, answer freeform questions, and provide intelligent assistance. You can read more about using AISQL functions as part of Cortex suite- https://lnkd.in/gxxRpJC4 #snowflake #datasuperheroes #snowflake_advocate #genai #snowflakecortex

13