Before designing the solution, design the problem

by Harsh Choudhary

The human mind is not very well with statistics or calculations. Give it some calculations and your pupils will dilate. Don’t believe me - try 9*13 or google the following keywords

Add-3 experiment, Attention and Effort, Thinking Fast and Slow, Daniel Kahneman.

What it is actually good at - visualizing things, remembering relations and meanings, understanding stories. Just try to cram a random number and you will forget it after some time. I had remembered this random number 9462 some months back. My technique to remember it was

  • It is a 4-digit number
  • The first two digits are 94, a year after my birth (93)
  • The third digit is a flip of 9 -> 6
  • The fourth digit is half of the second digit 4/2 -> 2

Maybe, it was overkill but I remember this number even now. I am actually very bad at remembering pins otherwise. Though, this is not a pin I use anywhere (e.g., my phone or credit card) - so don’t try.

I think it’s enough background. My primary job is to provide solutions to problems of my customers and my colleagues. For this, there is one approach that I really follow. Though this applies to any kind of problem, here I will talk specifically from the perspective of Data Architect.

Whenever I come across a problem statement, I first try to create a visual picture/diagram of it. This way I can

  • get a high-level view of the problem statement at hand
  • get my understanding validated with the problem stater(s)
  • decide whether the original problem is actually the right problem or needs reframing
  • find the missing links and point the missing information
  • note the assumptions, which are not present otherwise
  • discuss it with various kind of people - engineers, product/project managers, salespeople, or non-tech stakeholders
  • get the framework of the solution ready

Before even coming up with the solution, it is very important to get the problem right. Jumping directly to solve the problem can lead to a solution that solves something else and not the original problem in hand. The idea is to cover the breadth and then the depth of the problem.

The mechanism

Let’s understand the process with an example

A company wants to create an AI product recommendation engine for its retail customers. They want to use their existing Campaign Manager to send personalized product recommendations to their customers via channels like Email, SMS, Push notifications of Mobile application.

Step 1: Put down all the components

My intent at this step is to understand the high-level use case and come up with follow-up questions to understand different aspects of it. So, I come up with the following questions

  • Does any kind of recommendation exist today? If yes, what is the setup?
  • Recommendation Engine involves a Machine Learning model. To build that what data is available and where is it?
  • Where will the infrastructure be hosted? On some cloud or on-premise?
  • How does the existing Campaign Manager work?

The customer (the client) tells me that the data is stored in their transactional databases. They do their current analytics directly on them and don’t currently have any sort of analytics database. The databases are hosted on-premise in their data center. They want to use AWS for this use case. The current campaign manager works as software hosted on AWS only where they upload customers' offers for the campaigns.

Step 2: Fill up more details

Subsequent steps:

Keep going deep to understand all the components, constraints, requirements, and details required to come up with the right solution and architecture.

See, this is the way I approach a problem no matter how hard it is. Our brain can better comprehend this way and the analytical part of our brain can really complement it with the details required to design the right solution.

So, always keep a pen and diary with you. Who knows a problem comes and you need to doodle.