Deepak Mittal

I created a video covering 5 Scenario based Interview questions related to Deletion Vectors in Databricks (a really hot topic for interviews) Question 1) You are managing a large Delta table with billions of rows.  Deleting even a few thousand rows used to be very expensive because it required rewriting files. How do deletion vectors, help solve this problem? Question 2) Imagine two cases: Case A: Deleting 100 rows Case B: Deleting 100 million rows In which case do deletion vectors bring the most benefit, and when would you consider a file rewrite instead? Question 3) Your company must comply with the GDPR (right to be forgotton). Since deletion vectors only mark rows as deleted, the data still exists in storage. How would you design a process to ensure deleted records are physically removed for compliance? Question 4) Over time, analysts complain that queries on the Delta table are becoming slower after frequent deletes. How would you confirm whether deletion vectors are the cause, and what steps can you take to restore performance? Question 5) After months of deletes and updates, you notice storage costs have gone up and queries are slower, even though deletion vectors were meant to save space and time. Why does this happen, and what Databricks operations would you apply to balance storage, performance, and compliance? Link to the video in Comments! If you know the answers, then do mention here. I am sure this video will help you a lot. This is a really hot topic for Interviews these days! Do support by liking, commenting & sharing if you truly find it valuable :) #databricks

٢٤١

٩ تعليق

Enterprise data pulls in two directions. On one side: governed BI workloads — structured, compliant, and built for business reporting. On the other: unstructured pipelines — streaming feeds, heavy transformations, and machine learning at scale. Most organizations now run Snowflake and Databricks in parallel to handle this dual reality. Yet stitching them together creates new challenges: deciding which workloads belong where, moving data without waste, and keeping governance consistent across environments. Each platform brings distinct strengths to the table — Snowflake excels at governed analytics and BI reporting, while Databricks powers heavy transformations, streaming, and machine learning. When combined thoughtfully, these strengths converge into a unified architecture that supports the full data lifecycle. > Watch the on-demand discussion on how Janus Henderson Investors designed a unified architecture that balances both worlds: https://lnkd.in/eG-8YBqm In a financial services setting where accuracy, performance, and cost discipline matter most, Janus Henderson’s approach connects Snowflake’s strengths in analytics and governance with Databricks’ capabilities in engineering and ML. #unifieddataarchitecture #snowflake #databricks #usecase

١٣٥

٣٨ تعليق

Death by a thousand defaults on #Databricks. Databricks clusters: latest ML runtime + auto-termination set to 3 days. Dashboards: refresh every few minutes. Alerts: happily running queries on your most expensive cluster. Online tables: syncing more often than anyone needs. Not obvious. Sometimes hidden. Always expensive.

١٣٦

١٠ تعليق

What I love about #ApacheIceberg is how many different workloads folks are running on the same copy of data. Graph queries are a really cool example. Instead of copying data into specialized graph databases, PuppyGraph lets you run graph traversals directly on your existing Iceberg tables. Pretty cool stuff. But here's the thing - to get truly great performance, you need to ensure your Iceberg tables are well maintained and optimized. That's where Ryft comes in. We make sure your Iceberg tables are automatically maintained and optimized, no matter what engine you are using. It was terrific to work with Zhenni Wu and Weimo Liu, to create this blog post on how you can use Ryft with PuppyGraph to run graph queries on a fully optimized Iceberg lake! https://lnkd.in/giwnwMuc #ApacheIceberg #GraphQueries

٩٩

٢ تعليق