Jorge Salazar

We finally have 𝘂𝘀𝗲𝗿-𝗱𝗲𝗳𝗶𝗻𝗲𝗱 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝘀 (𝗨𝗗𝗙) in DAX! Don't make mistakes; this is not a user data function in Fabric. This is a new DAX feature, the most important one in the last five years. First article and video to introduce how UDFs in DAX work.

80

1 comentario

#DataWarehouse, #DataLake, and #DataMesh ..... What’s the Difference? If you are working in the data space, you have probably come across these terms. While they often get used interchangeably, each serves a very different purpose. A Data Warehouse is a centralized system that stores clean, structured, and processed data from different sources. It is designed for reporting, business intelligence, and analytics. Think of it as a well-organized library where everything is neatly labeled and easy to search. In contrast, a Data Lake is built to handle large volumes of raw data in any format structured, semi-structured, or unstructured. It is widely used in big data and machine learning environments where flexibility is more important than immediate usability. Imagine it as a giant reservoir where all data flows in, but it needs further processing before it becomes useful. Then comes the Data Mesh, which takes a different approach. Rather than relying on a single, centralized data team, Data Mesh promotes a decentralized architecture. Each business domain like marketing, sales, or HR owns and manages its own data as a product, while following shared standards and governance. This model scales well in large organizations and encourages accountability and faster data delivery. In short, a data warehouse gives you reliable reports, a data lake gives you flexibility with raw data, and a data mesh gives you scalability through distributed ownership. Each has its place in a modern data architecture and often, organizations use a combination of all three to meet different needs. Which one are you using or moving towards in your organization? Avinash S. #DataEngineering #DataArchitecture #DataWarehouse #DataLake #DataMesh #Analytics #CloudDataPlatform #ModernDataStack

67

4 comentarios

Don't write a 1000-line SQL Query. Build a data transformation pipeline with modular components, version control, and column-level lineage. Cramming every edge case into one SELECT statement is exhausting. Modularity with maintainability is where it's at. Ready to get started? / Break monoliths into small, named models / Reuse logic functions and macros / Add tests for keys, nulls, ranges, and referential integrity / Put everything in version control / Document inputs, outputs, owners, and assumptions / Make sure column-level lineage is visible / Boring predictability > latest tech tricks Want to level up? Follow - → Zach Wilson at DataExpert.io → Tobias (Toby) Mao at Tobiko → Benjamin Rogojan at Seattle Data Guy → Xinran Waibel at Data Engineer Things → Carly Taylor, MSc at Rebel Data Science → Madison Schott at Learn Analytics Engineering Happy Engineering! #dataengineering #datatransformation #sqlmesh Tobiko

84

12 comentarios

🚀 𝗗𝗲𝗺𝘆𝘀𝘁𝗶𝗳𝘆𝗶𝗻𝗴 𝗟𝗮𝗸𝗲𝗵𝗼𝘂𝘀𝗲 𝗙𝗲𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝗶𝗻 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 Curious about how 𝗾𝘂𝗲𝗿𝘆 𝗳𝗲𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻 and 𝗰𝗮𝘁𝗮𝗹𝗼𝗴 𝗳𝗲𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻 work in Databricks? 🤔 I’ve put together a 𝗾𝘂𝗶𝗰𝗸 𝗴𝘂𝗶𝗱𝗲 that explains: 🔹 What Lakehouse Federation is 🔹 Key differences between Query vs Catalog Federation 🔹 Practical use cases for both 🔹 Simple setup steps to get started Whether you’re a 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿, 𝗮𝗻𝗮𝗹𝘆𝘀𝘁, 𝗼𝗿 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁, this PDF will help you understand how to run queries seamlessly across multiple data sources without moving all your data into one place. 📥 Download & Read the PDF: [Attached] 💡 𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 for more insights on Databricks, Azure Data, Data Engineering & AI. #Databricks #Lakehouse #DataEngineering #AzureData #BigData #CloudComputing #AI #MachineLearning #DataArchitecture #TechLearning #MicrosoftFabric #LearningOnLinkedIn

65

1 comentario

PostgreSQL 18 is almost out. Two new game changing features: 🐘 Asynchronous I/O AIO 🐘 UUID7 AIO Introduction of a native asynchronous I/O (AIO) subsystem. This new architecture is designed to significantly increase I/O throughput and reduce latency by allowing the database to perform other work while waiting for data to be read from storage. The AIO feature allows PostgreSQL to issue read requests without blocking, which is especially beneficial for systems with slower, network-attached storage common in cloud environments. The initial implementation specifically targets read-heavy operations, including: ✔️ Sequential scans ✔️ Bitmap heap scans ✔️ Vacuums For databases running these types of workloads, this could translate into performance gains of 2-3 times. However, it's crucial to note that this first release focuses on reads, so write-intensive OLTP applications will not see a direct benefit just yet. UUID7 PostgreSQL 18 now includes native support for UUIDv7, a modern standard for generating globally unique and sortable identifiers. This is a game-changing feature for developers building distributed systems who need efficient, index-friendly primary keys. Traditional random UUIDs (like UUIDv4) are excellent for ensuring uniqueness but are terrible for database performance when used as primary keys. Because they are completely random, new entries are inserted at random locations within a B-tree index. This leads to poor data locality and frequent page splits, a costly operation that fragments the index and slows down writes and reads. UUIDv7 solves this problem by embedding a timestamp at the beginning of the identifier. This makes the UUIDs sequential and naturally ordered by creation time. When new UUIDv7 keys are inserted, they are always added to the end of the index, which is far more efficient. This approach dramatically improves caching and minimizes page splits, resulting in significantly better performance. #PostgreSQL #PostgreSQL18 #Database #DBA #DatabaseAdministrator #DevOps #SystemAdministrator #CloudComputing #DataEngineering #DatabaseManagement #ITInfrastructure #CloudDatabase #DatabasePerformance #postgres #aio #uuid7

362

2 comentarios

𝗦𝗰𝗮𝗹𝗮𝗯𝗹𝗲 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗣𝗿𝗼𝗷𝗲𝗰𝘁𝘀: 𝗠𝗼𝗻𝗼𝗿𝗲𝗽𝗼 𝗼𝗿 𝗠𝗶𝗰𝗿𝗼-𝗕𝘂𝗻𝗱𝗹𝗲𝘀? 𝗔𝗻 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗮𝗹 𝗖𝗼𝗺𝗽𝗮𝗿𝗶𝘀𝗼𝗻 Hej! 👋 Your first project with Databricks Asset Bundles (DABs) is a success, and the CI/CD pipeline is running. But now the team is growing, new projects are being added, and the big question arises: How do we structure all of this to keep it maintainable, scalable, and efficient? At the core, there are two opposing architectural patterns: the monorepo and the micro-bundle approach. Choosing the right pattern is one of the most important strategic decisions for your Databricks development. Pattern 1: The Monorepo (One central bundle for everything) In this approach, all the code for multiple projects or teams resides in a single large Git repository, often managed by a central databricks.yml file with many targets. Advantages: - Easy Code-Sharing: Common libraries and utilities can be imported directly. A change is immediately available to everyone. - Atomic Commits: Changes to a library and the jobs that use it can be made in a single, traceable commit. - Easy Discoverability: All code is in one place, which simplifies finding things. Disadvantages: - Slow CI/CD: A small change can trigger a massive pipeline that tests and validates everything. - Tight Coupling: A failing test in an insignificant project can block deployment for all others. - Complex Access Management: It's difficult to restrict access to individual project folders within the repo. Ideal for: Smaller teams, tightly coupled projects, or when you want to get started quickly. Pattern 2: Micro-Bundles (A separate bundle per team/project) Here, each team or each major, independent data product has its own Git repository with its own databricks.yml. Advantages: - Independent Deployments: Teams can deploy their changes whenever they want without impacting others. This significantly increases velocity. - Clear Ownership: Responsibility for code, quality, and deployments is clearly assigned to the respective team. - Focused CI/CD Pipelines: The pipelines are fast, lean, and only test the relevant code. Disadvantages: - Shared Code is More Complex: Common utilities must be versioned as standalone Python packages and published in a package repository. - Potential Code Duplication: Without clean package management, there's a risk of teams reinventing the wheel. Ideal for: Larger organizations, multiple independent teams, and architectures in the style of Data Mesh. There is no universal answer, but a good rule of thumb is: "Start with a monolith, refactor to microservices." What strategy do you follow and why? Share your lessons learned in the comments! 👇 #Databricks #AssetBundles #DevOps #DataEngineering #CICD #DataInsightConsulting

61

3 comentarios