Getting started with hypothesis-driven experimentation

January 4, 2021

Today marked the first day of work in 2021. Aside from Slack being down for a few hours, I’d say everything went well, and I finally applied something I had been learning about. 


The big task I had to get through was wrapping up the product requirement document (PRD) for an initiative I will be tackling. 


This initiative is not new but is rather a redesign. I have high-level goals for what needs to happen for this to be a success. 


But with that, I had both lots of ideas and lots of questions, making it hard to start. 


I did some research on how to approach this in the most experimental manner, given the limited resources and time I have to hit milestones that determine further investments. The answer: hypothesis-driven design. 


Step 1: Capture questions and assumptions

This is critical. Prior to sitting down and dumping all the assumptions and questions I had, I was already coming up with solutions. 


Ideation is easy, what isn’t is solving the right user problem. 


Step 2: Organize and prioritize 

All the questions and assumptions are captured, now you have to organize them and bucket them into groups that will make it really easy to review. 


Next, prioritize. My criteria think of the impact. What would be most valuable? 


Take into consideration I did this alone given the pod hasn’t come together yet. In the ideal scenario, you are organizing and prioritizing with your team. 

Step 3: Hypothesize 

This to me was the biggest novelty. How to write out the hypotheses for the questions that were prioritized. 


In hypothesis-driven experimentation, the hypothesis is the focal point and it’s what determines both the capability and the measurements required to either prove or disprove this. Remember that in growth both are successful outcomes. 


The hypothesis template looks like: 

  1. ‘We believe that…’: this is where you want to put in the informed guess of the user’s behavior. 
  2. ‘so if we …’:   is the action, it’s what we want or think the user is going to do. 
  3. ‘leading to …’ and finally, the expected result or measure of success. 


What you are doing here is basically combining the solution with a question, and the output is the hypothesis. 


What I found most helpful about this method is that you can clearly define the question and solution, while eliminating assumptions. 

Time to experiment

With that, I now move on to building out the experiments. I’ll follow-up with how I went about experiment formulation once I get to it. 

Takeaway

For me, the main benefit of approaching hypothesizing like this on a new surface area is that the team and cross-functional partner (or stakeholders) will understand the intent and we’ll put up some guardrails to keep us on track and not get lost in an implementation loop. 


Reframing the work something more discovery based gave more space to think instead of getting locked into solutions based on assumptions.