Daniyal Bashir

Just in case you want to explain the difference between ETL (extract-transform-load) and ELT (extract-load-transform) approaches in data engineering to your 5 y.o. child, I have something for you. It’s actually easy: either you get something, store it, and make sense out of it later; or you get something, make some sense out of it, and store only the results. What is better? Depends on the use case and a few parameters like size of your storage space and complexity of the processing needed. UPD: for more information about ETL and ELT please check the episode of my video-blog, which is dedicated to this topic: https://lnkd.in/eP52bjFc #data #leadership #etl #elt #apples

٧٠٠

٦٥ تعليق

Apache Parquet Vs New File Formats. There’s no denying the impact Apache Parquet has had on the analytics ecosystem. - Columnar layout - Strong compression & encoding - The de facto format in data lakes and warehouses for over a decade But there are also a lot of new things happening with our analytical workloads. ❌ Workloads are no longer just batch analytics - they involve AI pipelines, low-latency use cases, and hardware-accelerated compute ❌ Hardware is not only CPU but also GPU, from x86 to ARM/RISC-V, with wide SIMD instruction sets becoming the norm ❌ And performance bottlenecks are seen in decompression speed, memory pressure and lack of vectorized execution paths As a result, we’re seeing a wave of innovation around next-gen file formats. Efforts like: BtrBlocks – introduces cascaded compression with lightweight codecs Nimble (Meta) – optimized for fast scan & inference LanceDB – tailored for vector search and ML use cases And the past week I read about “Fastlanes” Here are some highlights from their VLDB paper ✅ Expression Encoding: Flexible chains of lightweight codecs (FFOR, DELTA, DICT, FSST) that outperform heavyweight compression like Zstd ✅ Multi-Column Compression: Exploits inter-column correlations (e.g. equality, one-to-one mappings) to go beyond traditional columnar compression ✅ Segmented Layout: Decompresses in small vectors (1024 values) rather than rowgroups, reducing memory pressure and improving cache efficiency ✅ Compressed Execution Support: Returns compressed vectors to engines like DuckDB/Velox for SIMD/GPU-friendly query execution Here are some numbers I picked up on Performance: - 800× faster random access than Parquet+Zstd - 41% better compression than Parquet+Snappy - 2% better compression than Parquet+Zstd (without using heavyweight codecs) - Decoding accelerates with AVX512 (up to 40% faster) I think we are going to see more of fit-for-purpose formats in the future as we tackle specific use cases around AI workloads and to target performance. Paper link in comments. #dataengineering #softwareengineering

٣٧٧

١٣ تعليق

etcd is a distributed key-value store used to coordinate state across nodes in a PostgreSQL cluster managed by Patroni. In a PostgreSQL setup aiming for 99.99% availability, etcd plays a crucial role. Here is why: 👉 Cluster State Source of Truth etcd holds the authoritative state of your PostgreSQL cluster. Patroni nodes read from and write to etcd to coordinate leader election, replica health, and failover decisions. Without it, there is no consensus—just confusion. 👉 Reliable Automatic Failover When the primary fails, Patroni does not guess. It asks etcd. If etcd confirms consensus, failover is triggered instantly and safely. No split-brain. No manual intervention. Just uptime. 👉 Enabler of 99.99% Availability You cannot hit four nines of uptime if failover depends on manual processes. etcd provides the consistency and speed needed for fully automated, fault-tolerant failover—keeping SLAs intact when it matters most. 👉 Lightweight and Secure etcd is designed for fast, distributed, and consistent reads/writes. It brings consensus-based decision making to PostgreSQL HA in a lightweight and secure package. 👉 Built for Scale As your application grows, etcd scales with you. It supports multi-node deployment and integrates tightly with cloud-native infrastructure. For technology leaders who prioritize keeping mission-critical systems online, avoiding costly downtime, and scaling smoothly, this is the architecture that delivers. If you are building for resilience with PostgreSQL, etcd is the cornerstone of availability. Read more here: https://lnkd.in/dwrdP3kn

٨٥

٢ تعليق

PostgreSQL 18 is almost out. Two new game changing features: 🐘 Asynchronous I/O AIO 🐘 UUID7 AIO Introduction of a native asynchronous I/O (AIO) subsystem. This new architecture is designed to significantly increase I/O throughput and reduce latency by allowing the database to perform other work while waiting for data to be read from storage. The AIO feature allows PostgreSQL to issue read requests without blocking, which is especially beneficial for systems with slower, network-attached storage common in cloud environments. The initial implementation specifically targets read-heavy operations, including: ✔️ Sequential scans ✔️ Bitmap heap scans ✔️ Vacuums For databases running these types of workloads, this could translate into performance gains of 2-3 times. However, it's crucial to note that this first release focuses on reads, so write-intensive OLTP applications will not see a direct benefit just yet. UUID7 PostgreSQL 18 now includes native support for UUIDv7, a modern standard for generating globally unique and sortable identifiers. This is a game-changing feature for developers building distributed systems who need efficient, index-friendly primary keys. Traditional random UUIDs (like UUIDv4) are excellent for ensuring uniqueness but are terrible for database performance when used as primary keys. Because they are completely random, new entries are inserted at random locations within a B-tree index. This leads to poor data locality and frequent page splits, a costly operation that fragments the index and slows down writes and reads. UUIDv7 solves this problem by embedding a timestamp at the beginning of the identifier. This makes the UUIDs sequential and naturally ordered by creation time. When new UUIDv7 keys are inserted, they are always added to the end of the index, which is far more efficient. This approach dramatically improves caching and minimizes page splits, resulting in significantly better performance. #PostgreSQL #PostgreSQL18 #Database #DBA #DatabaseAdministrator #DevOps #SystemAdministrator #CloudComputing #DataEngineering #DatabaseManagement #ITInfrastructure #CloudDatabase #DatabasePerformance #postgres #aio #uuid7

٣٦٢

٢ تعليق

Building an Airflow DAG is complicated. Building a beautiful one really reduces tech debt. Here's some tips: - Don't write directly to production. The data quality risk is too high. Use the stage-check-exchange pattern instead. - Use clear task names. Don't call your task "load_data." Call it "load_user_data" or something more clear - Declare all your partitions sensors near the top of the DAG. Declaring them in the middle or as you go is very confusing. - Don't write SQL directly in the DAG. Move your SQL into a separate file so the DAG is clean and concise. - Don't hardcode table names. Abstracting them to a constants file where they can be reused for easy renaming is better.

٦٦٠

٢٩ تعليق

3 Suggestions for Aspiring & Growing Data Engineers 🚀 Having worked with data engineering teams over the years, I’ve noticed a few things that can make a big difference in your career trajectory and impact. Here are my top 3 suggestions: 1️⃣ Contribute to Platforms—Not Just Pipelines or Models Data engineers with deeper platform-oriented thinking make their teams more productive, efficient, and capable. Whether you’re designing pipelines or building models, focus on how your work strengthens the broader platform—ensuring it’s scalable, reliable, and valuable for the entire organization. 2️⃣ Lean Towards Self-Serve The most impactful data teams empower others. Invest time in building tools, abstractions, and documentation that enable analysts, scientists, and business teams to answer their own questions without depending on you for every dataset. 3️⃣ Think Mid to Long Term Avoid optimizing just for today’s problem. Choices around architecture, tooling, and data modeling should be made with the next 6–24 months in mind. Short-term hacks often cost more in the long run. 💡 Great data engineering isn’t just about moving data—it’s about building enabling systems for the future. #BigData #DataEngineering #DataPlatform #PlatformEngineering #DataScience #DataAnalytics #ETL #DataEngineer #DataAnalyst

١٩

Don’t get confused while learning PySpark. Understand these core concepts step by step to become truly proficient in PySpark. 𝗕𝗮𝘀𝗶𝗰𝘀 𝗼𝗳 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - PySpark Architecture - SparkContext and SparkSession - RDDs (Resilient Distributed Datasets) - DataFrames - Transformations and Actions - Lazy Evaluation 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗗𝗮𝘁𝗮𝗙𝗿𝗮𝗺𝗲𝘀: - Creating DataFrames - Reading Data from CSV, JSON, Parquet - DataFrame Operations - Filtering, Selecting, and Aggregating Data - Joins and Merging DataFrames - Working with Null Values 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗖𝗼𝗹𝘂𝗺𝗻 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Defining and Using UDFs (User Defined Functions) - Column Operations (Select, Rename, Drop) - Handling Complex Data Types (Array, Map) - Working with Dates and Timestamps 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗦𝗵𝘂𝗳𝗳𝗹𝗲 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Understanding Partitions - Repartitioning and Coalescing - Managing Shuffle Operations - Optimizing Partition Sizes for Performance 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗮𝗻𝗱 𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗶𝗻𝗴 𝗗𝗮𝘁𝗮: - When to Cache or Persist - Memory vs Disk Caching - Checking Storage Levels 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗪𝗶𝘁𝗵 𝗦𝗤𝗟: - Spark SQL Introduction - Creating Temp Views - Running SQL Queries - Optimizing SQL Queries with Catalyst Optimizer - Working with Hive Tables in PySpark 𝗪𝗼𝗿𝗸𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗗𝗮𝘁𝗮 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Data Cleaning and Preparation - Handling Missing Values - Data Normalization and Transformation - Working with Categorical Data 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗧𝗼𝗽𝗶𝗰𝘀 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Broadcasting Variables - Accumulators - PySpark Window Functions - PySpark with Machine Learning (MLlib) - Working with Streaming Data (Spark Streaming) 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗧𝘂𝗻𝗶𝗻𝗴 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Understanding Job, Stage, and Task - Tungsten Execution Engine - Memory Management and Garbage Collection - Tuning Parallelism - Using Spark UI for Performance Monitoring Once you master these PySpark fundamentals, you won’t just write code — you’ll design efficient, scalable, and production-ready data pipelines that harness the full potential of Spark.

٢٧٠

٣٣ تعليق

Hi. Sample design of data bars (SVG involved) Integrated variance bar & Progress bar 🟢 🔴 🟦This is based on a sample/anonymized data set. The origin requirement was derived from a different use-case. 🟦 It's more of a design example rather than recommended path to take. In most cases we would "need" to go with native bars. 🟦In most cases - we will be using native (standard) data bars. If we need to omit the limitations - we can strategically use SVG (taking into accounts its pros & cons) 🟦Here - I intentionally use SVG only to show data bars (visual representation) without using text components. Here - the values are available through standard measures. 🟦With such row-height - the misalignment between top-aligned text and bars is not to noticeable. 🟦The logic behind the left-side progress bars can be adjusted/improved (for now it's higlighting red/green based on whether it's below/above the 100% threshold. 🟦If we would like to push it further, we could potentially use some techniques proposed by Nick Desbarats (higlighting items which needs attention while keeping the "safe" range less prominent). For the technical simplicty - here 0/1 color cosind is used. Have you ever needed SVG-out of the box solutions in your PBI reports? All best 🖐 #TheDataVisForge #analytics #microsoft #data #powerbi #datavisualization #dashboard #visualization #UX #UI

٢٢٦

١٤ تعليق