Chandramani Tiwary

A lot has changed in Databricks. Over the years Databricks has evolved as a End to End Data Platform. When we talk about Data Engineering, Databricks now offers - Lakeflow Connect (for data Ingestion) - Lakeflow Declarative Pipelines (for ETL, earlier referred as DLT) - Lakeflow Jobs (for Orchestration) - Unity Catalog (for Metadata handling and Governance) In the latest versions of databricks, the workspace is unity catalog enabled. In unity catalog, at the top level is the Metastore Under metastore we have one or more catalogs Under catalog you have schemas/Databases Under schemas you have tables. So its a 3 level namespace catalog -> schema -> table Now whenever we create a databricks workspace we will get - an automatically provisioned unity catalog metastore.. - and a catalog under that named after your workspace. The best practise is to just have 1 metastore in one region and then attach this metastore to all the workspaces in that region. BTW whats the need for more than one workspace? you can have it for different environments (dev, stg, prod) for different BU's (sales, finance, IT) can one workspace have multiple metastores? No, it will have just one metastore attached to it. can multiple workspaces have one metastore only? Yes, if these workspaces are in the same region. you might know the days when hive metastore is the default one. Now its not default, but we still get one hive metastore along with each workspace. Hive Metastore follows a 2 level namespace Schema -> table Now lets understand, what were the issues with hive metastore? - Hive metastore offers basic metadata management but lacks, fine grained access controls. - It lacks build in data governance features like data lineage, auditing and access controls. - Moreover hive metastore can deal only with structured data. so basically, tables in the hive metastore do not benefit from the full set of security and governance features provided by the unity catalog. such as built in auditing, lineage and access control. Its a legacy feature now. When developing any solution in Databricks make sure we should always use Unity catalog. I am sure you would have learned something new from this post. In my next post I will talk about tables in Databricks in good depth. Also, I recently launched my Databricks Masters Program. My students always say that my pyspark program is the best in the world. I have taken it as a challenge that my students should say that your Databricks program is even better than your pyspark program. It will be a 15 weeks of extensive program (8 weeks of Databricks + 7 weeks of Databricks performance tuning) DM to know more!

633

57 Comments

🚀 New in Docling: Structured Data Extraction from Documents! 🚀 We’ve just added a brand-new functionality: extraction of structured data from complex documents using free-form schemas 🤩 What does that mean? 🔹 You can now skip the conversion step — instead of turning documents into text or JSON first, Docling directly extracts the structured fields you care about. 🔹 The requested fields are defined in a free-form schema, so you can instantly align the extraction with the schemas of your own databases. 🔹 This makes it ideal for data-pipelines that don’t need full document conversion, but do need to populate structured databases from messy, unstructured documents. ✨ It’s: 1️⃣ Super simple to use (check out the code snippet 👉 https://lnkd.in/eK6vaMEe ) 2️⃣ 100% open-source and runs fully local – no API calls needed 🙌 3️⃣ Powered by cutting-edge models from NuMind (YC S22) 4️⃣ Perfect for data-pipelines where you need to populate structured databases from documents (think invoices, Curriculum Vitae, contracts, product datasheets, etc) 5️⃣ Currently focused on PDFs and images (PNG) — support for pure text coming soon! The example below shows how easy it is to define a schema and extract structured fields directly from an invoice. 👉 Try it out, break it, send us feedback, and if you like what we’re building, don’t forget to ⭐ the repo (https://lnkd.in/d4UT-6_2)! After a long and fruitful summer, the Docling team has been cooking up many new features — and this is just the beginning! 🌟 #opensource #AI #documentAI #docling #IBM #IBMResearch

1,382

83 Comments

Advanced RAG Techniques 1. Basic Index Retrieval + Metadata Uses traditional vector search enhanced with metadata filters (e.g., tags, source). 2. Query Routing Routes queries to specialized indexes or retrievers based on query intent or topic. 3. Sentence Window Retrieval Expands retrieval scope by pulling nearby sentences to add richer context. 4. Query Transformation Modifies or expands the original query to improve retrieval accuracy (e.g., query rewriting, paraphrasing). 5. HyDE Retrieval (Hypothetical Document Embeddings) Generates hypothetical answers from the LLM and embeds them to retrieve supporting documents. 6. Hierarchical Index Retrieval Organizes documents in multi-level indexes (e.g., chapter → section → paragraph) for structured search. 7. Parent-Child Chunks Retrieval Links document chunks to parents (e.g., section headers) to preserve context during retrieval. 8. Hybrid Search (Fusion Rank) Combines keyword search (BM25) with vector search (dense embeddings) using ranking algorithms. 9. Agents + Query Routing Employs intelligent agents that interpret the query and route to the right tools or indexes dynamically. 🌟 Follow the AIKaDoctor (Free AI & Data Science Resources)  channel on WhatsApp: https://lnkd.in/dCTCEKKc 📌Follow Habib Shaikh For more such content.

42

4 Comments