Andrés Navidad León

Apache Parquet Vs New File Formats. There’s no denying the impact Apache Parquet has had on the analytics ecosystem. - Columnar layout - Strong compression & encoding - The de facto format in data lakes and warehouses for over a decade But there are also a lot of new things happening with our analytical workloads. ❌ Workloads are no longer just batch analytics - they involve AI pipelines, low-latency use cases, and hardware-accelerated compute ❌ Hardware is not only CPU but also GPU, from x86 to ARM/RISC-V, with wide SIMD instruction sets becoming the norm ❌ And performance bottlenecks are seen in decompression speed, memory pressure and lack of vectorized execution paths As a result, we’re seeing a wave of innovation around next-gen file formats. Efforts like: BtrBlocks – introduces cascaded compression with lightweight codecs Nimble (Meta) – optimized for fast scan & inference LanceDB – tailored for vector search and ML use cases And the past week I read about “Fastlanes” Here are some highlights from their VLDB paper ✅ Expression Encoding: Flexible chains of lightweight codecs (FFOR, DELTA, DICT, FSST) that outperform heavyweight compression like Zstd ✅ Multi-Column Compression: Exploits inter-column correlations (e.g. equality, one-to-one mappings) to go beyond traditional columnar compression ✅ Segmented Layout: Decompresses in small vectors (1024 values) rather than rowgroups, reducing memory pressure and improving cache efficiency ✅ Compressed Execution Support: Returns compressed vectors to engines like DuckDB/Velox for SIMD/GPU-friendly query execution Here are some numbers I picked up on Performance: - 800× faster random access than Parquet+Zstd - 41% better compression than Parquet+Snappy - 2% better compression than Parquet+Zstd (without using heavyweight codecs) - Decoding accelerates with AVX512 (up to 40% faster) I think we are going to see more of fit-for-purpose formats in the future as we tackle specific use cases around AI workloads and to target performance. Paper link in comments. #dataengineering #softwareengineering

378

13 comentarios

PostgreSQL 18 is almost out. Two new game changing features: 🐘 Asynchronous I/O AIO 🐘 UUID7 AIO Introduction of a native asynchronous I/O (AIO) subsystem. This new architecture is designed to significantly increase I/O throughput and reduce latency by allowing the database to perform other work while waiting for data to be read from storage. The AIO feature allows PostgreSQL to issue read requests without blocking, which is especially beneficial for systems with slower, network-attached storage common in cloud environments. The initial implementation specifically targets read-heavy operations, including: ✔️ Sequential scans ✔️ Bitmap heap scans ✔️ Vacuums For databases running these types of workloads, this could translate into performance gains of 2-3 times. However, it's crucial to note that this first release focuses on reads, so write-intensive OLTP applications will not see a direct benefit just yet. UUID7 PostgreSQL 18 now includes native support for UUIDv7, a modern standard for generating globally unique and sortable identifiers. This is a game-changing feature for developers building distributed systems who need efficient, index-friendly primary keys. Traditional random UUIDs (like UUIDv4) are excellent for ensuring uniqueness but are terrible for database performance when used as primary keys. Because they are completely random, new entries are inserted at random locations within a B-tree index. This leads to poor data locality and frequent page splits, a costly operation that fragments the index and slows down writes and reads. UUIDv7 solves this problem by embedding a timestamp at the beginning of the identifier. This makes the UUIDs sequential and naturally ordered by creation time. When new UUIDv7 keys are inserted, they are always added to the end of the index, which is far more efficient. This approach dramatically improves caching and minimizes page splits, resulting in significantly better performance. #PostgreSQL #PostgreSQL18 #Database #DBA #DatabaseAdministrator #DevOps #SystemAdministrator #CloudComputing #DataEngineering #DatabaseManagement #ITInfrastructure #CloudDatabase #DatabasePerformance #postgres #aio #uuid7

362

2 comentarios

Just in case you want to explain the difference between ETL (extract-transform-load) and ELT (extract-load-transform) approaches in data engineering to your 5 y.o. child, I have something for you. It’s actually easy: either you get something, store it, and make sense out of it later; or you get something, make some sense out of it, and store only the results. What is better? Depends on the use case and a few parameters like size of your storage space and complexity of the processing needed. UPD: for more information about ETL and ELT please check the episode of my video-blog, which is dedicated to this topic: https://lnkd.in/eP52bjFc #data #leadership #etl #elt #apples

698

65 comentarios

PostgreSQL feels slower than usual. Now what? Before reaching for a bigger instance, take a breath and check the basics. Most Postgres slowdowns follow familiar patterns. These are the first places worth looking: ✅ Query plans — Is the planner doing what you expect? ✅ Indexes — Are they aligned with current access patterns? ✅ Autovacuum — Is it keeping up with churn-heavy tables? ✅ Connections — Are you pooling efficiently under load? ✅ Memory — Are key queries spilling to disk? ✅ Locks — Is something blocking the queue? ✅ Logs — Are you logging slow queries and temp files? ✅ App patterns — Are your ORMs over-fetching or repeating work? Asking the right questions early can save hours of firefighting. Try this, before 'scaling by credit card'!

48

6 comentarios

Don’t get confused while learning PySpark. Understand these core concepts step by step to become truly proficient in PySpark. 𝗕𝗮𝘀𝗶𝗰𝘀 𝗼𝗳 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - PySpark Architecture - SparkContext and SparkSession - RDDs (Resilient Distributed Datasets) - DataFrames - Transformations and Actions - Lazy Evaluation 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗗𝗮𝘁𝗮𝗙𝗿𝗮𝗺𝗲𝘀: - Creating DataFrames - Reading Data from CSV, JSON, Parquet - DataFrame Operations - Filtering, Selecting, and Aggregating Data - Joins and Merging DataFrames - Working with Null Values 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗖𝗼𝗹𝘂𝗺𝗻 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Defining and Using UDFs (User Defined Functions) - Column Operations (Select, Rename, Drop) - Handling Complex Data Types (Array, Map) - Working with Dates and Timestamps 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗦𝗵𝘂𝗳𝗳𝗹𝗲 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Understanding Partitions - Repartitioning and Coalescing - Managing Shuffle Operations - Optimizing Partition Sizes for Performance 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗮𝗻𝗱 𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗶𝗻𝗴 𝗗𝗮𝘁𝗮: - When to Cache or Persist - Memory vs Disk Caching - Checking Storage Levels 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗪𝗶𝘁𝗵 𝗦𝗤𝗟: - Spark SQL Introduction - Creating Temp Views - Running SQL Queries - Optimizing SQL Queries with Catalyst Optimizer - Working with Hive Tables in PySpark 𝗪𝗼𝗿𝗸𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗗𝗮𝘁𝗮 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Data Cleaning and Preparation - Handling Missing Values - Data Normalization and Transformation - Working with Categorical Data 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗧𝗼𝗽𝗶𝗰𝘀 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Broadcasting Variables - Accumulators - PySpark Window Functions - PySpark with Machine Learning (MLlib) - Working with Streaming Data (Spark Streaming) 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗧𝘂𝗻𝗶𝗻𝗴 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Understanding Job, Stage, and Task - Tungsten Execution Engine - Memory Management and Garbage Collection - Tuning Parallelism - Using Spark UI for Performance Monitoring Once you master these PySpark fundamentals, you won’t just write code — you’ll design efficient, scalable, and production-ready data pipelines that harness the full potential of Spark.

267

33 comentarios

🚀 Delta Table vs Delta Live Table – The Complete Difference In Azure Databricks, both Delta Tables and Delta Live Tables (DLT) are essential for building a robust Lakehouse — but they’re not the same thing. 🔹 Delta Table What it is: Storage format built on Parquet with ACID transactions. Key features: ✅ Create, read, update, delete data ✅ Schema enforcement & evolution ✅ Versioning & Time Travel (VERSION AS OF) ✅ Batch or streaming ingestion (manual setup) When to use: Reliable storage for Bronze, Silver, and Gold layers in the Medallion Architecture. 🔹 Delta Live Table (DLT) What it is: Managed ETL pipeline framework that uses Delta Tables under the hood. Key features: ✅ Declarative pipelines (SQL or Python) ✅ Automated ingestion, transformation, dependency management ✅ Built-in data quality checks (Expectations) ✅ Native batch + streaming support without extra configuration ✅ Rollback Options:   🔹 Option A (Pipeline level): Revert to an earlier pipeline definition → DLT automatically recomputes tables with the correct logic   🔹 Option B (Data level): Use Delta Time Travel → RESTORE TABLE … TO VERSION AS OF N to roll back bad data When to use: Production-ready, automated ingestion + transformation pipelines. 💡 Simple analogy: Delta Table → Your warehouse shelves (organized, reliable storage) Delta Live Table → The conveyor belt + quality inspector keeping those shelves stocked and clean automatically. 💡 Pro Tip: Use Option A when pipeline logic is wrong (ensures consistency) Use Option B when data itself is wrong (quick recovery) If you found this helpful, like 👍, share 🔄, and comment 💬 so more friends can understand the difference between Delta Tables and Delta Live Tables. #DataEngineering #Databricks #DeltaLake #ETL #BigData #AzureDatabricks #MedallionArchitecture #StreamingData #DataPipelines

174

20 comentarios

🚀 𝗗𝗲𝗺𝘆𝘀𝘁𝗶𝗳𝘆𝗶𝗻𝗴 𝗟𝗮𝗸𝗲𝗵𝗼𝘂𝘀𝗲 𝗙𝗲𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝗶𝗻 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 Curious about how 𝗾𝘂𝗲𝗿𝘆 𝗳𝗲𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻 and 𝗰𝗮𝘁𝗮𝗹𝗼𝗴 𝗳𝗲𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻 work in Databricks? 🤔 I’ve put together a 𝗾𝘂𝗶𝗰𝗸 𝗴𝘂𝗶𝗱𝗲 that explains: 🔹 What Lakehouse Federation is 🔹 Key differences between Query vs Catalog Federation 🔹 Practical use cases for both 🔹 Simple setup steps to get started Whether you’re a 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿, 𝗮𝗻𝗮𝗹𝘆𝘀𝘁, 𝗼𝗿 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁, this PDF will help you understand how to run queries seamlessly across multiple data sources without moving all your data into one place. 📥 Download & Read the PDF: [Attached] 💡 𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 for more insights on Databricks, Azure Data, Data Engineering & AI. #Databricks #Lakehouse #DataEngineering #AzureData #BigData #CloudComputing #AI #MachineLearning #DataArchitecture #TechLearning #MicrosoftFabric #LearningOnLinkedIn

65

1 comentario