The XY Problem and How to Avoid It

This edition of the newsletter contains

• a write-up to help you grow in your career
• a video I posted
• a paper I read

By the way,

Being hands-on is the best way for you to learn. Practice interesting programming challenges like building your own BitTorrent client, Redis, DNS server, and even SQLite from scratch on CodeCrafters.

Sign up, and become a better engineer.

The XY Problem and How to Avoid It

We often focus on the wrong problem when asking for help. Instead of describing what we actually want to achieve, we end up asking about a side issue, i.e., instead of asking about X, we end up asking about Y. Let me explain...

Say you’re implementing image upload in your service, and while doing so, you get stuck converting a base64 string into an image. So, you ask for help decoding base64.

But the real problem isn’t decoding base64 - it’s handling image uploads. If you had asked about photo uploads, someone might have shown you a much simpler approach.

That’s the classic XY problem: you ask about your assumed solution (Y) instead of explaining the actual problem (X) you want to solve.

• say what you are really trying to do
• don’t assume your solution is the only way
• explain the full problem, not just the code snippet
• describe what you have already tried

You can and should always start with context. A bit of background helps others understand the bigger picture. Then, state your end goal clearly. What does success actually look like?

Be open to alternative solutions, even if they go beyond your original plan. Sometimes the best fix isn’t what you imagined at first. A fresh perspective can save you hours of unnecessary work.

Show what you’ve already researched or tested. It signals effort and prevents people from repeating what you already know. Plus, it helps others pick up right where you left off.

Here’s an example: Instead of “How do I decode base64 in Python?” Try “I’m building an image upload feature and need to store uploaded photos efficiently. What’s the best way to handle this?”

Before you ask, check yourself: are you describing Y, or explaining X?

Focus on the real problem, not your first guess at the solution, and that’s how you get better answers and solve the right issue.

Here’s a video from me

I published a video - Database per Service Pattern in Microservices

Keeping microservices loosely coupled is our ultimate goal, and a high-level pattern that enables us to achieve it is - database-per-service pattern. I published a video where I cover

• the database-per-service architecture pattern,
• nuances around how it helps in modeling massive systems,
• understand the advantages of adopting it, and
• conclude by going through some drawbacks of it

Here’s a paper I recently read

I spent some time reading Tenzing: A SQL Implementation On The MapReduce Framework

In 2011, Google built a SQL Engine on top of MapReduce, which ran over 10,000 queries across 1.5 PB of data and powered Google Ads.

Some time back, I read a 2011 paper from Google titled Tenzing, which is a SQL engine built on top of MapReduce. This paper laid the foundation for all the big data technologies we see today, such as Spark SQL, Presto, and BigQuery, and included key features like joins, aggregations, and OLAP functions.

Tenzing leveraged persistent worker pools, shared in-memory state between nodes, and data streaming to bring end-to-end query latency under 10 seconds for most queries. They also implemented hash aggregation and joins by doing block-level shuffles and not sorting.

I would highly recommend you read this paper, as it will give you a glimpse into what it takes for a database to support SQL on a distributed database. You will be able to connect the dots when you use or dig deeper into the modern big data ecosystem.

You can download this and other papers I recommend from my papershelf.

Three interesting articles I read

I read a few engineering blogs almost every day, and here are the three articles I read and would recommend you read.

• An MVCC-like columnar table on S3 with constant-time deletes
• CPU Cache-Friendly Data Structures in Go
• Love C, Hate C: Web Framework Memory Problems

Thank you so much for reading this edition of the newsletter 🔮 If you found it interesting, you will also love my courses

1. System Design Course for Beginners
2. System Design Course for SDE-2, SDE-3, and above
3. Redis Internals Course

I keep sharing no-fluff stuff across my socials, so if you resonate, do give me a follow on Twitter, LinkedIn, YouTube, and GitHub.