Dealing with Uncertainty by Microsoft Product Leader, Vaibhavi Phadnis

Minimizing uncertainty and determining the right next step for your product is what product managers spend most of their time in their careers doing.

PM`s Most Important task

Reduce uncertainty to ensure product success

The most important job of a product manager is to create a successful product that satisfies our customers and makes a profit for our business. The success of this product depends on three things: product desirability, feasibility, and viability.
It becomes really important for product managers to minimize the uncertainty and risks associated with product desirability, feasibility and viability and to find that common ground between these three factors that really increases your chances of creating a successful product.

Gather Evidence to Reduce Uncertainty

"When you know little, your only goal should be to gather evidence that points you in the right direction."

Sources of Evidence

• Customers

  • Major source for gathering evidence related to desirability, viability

• Competitors

  • Moderate source for gathering evidence related to desirability, viability

• Internal teams

  • Major source for gathering evidence related to feasibility

  • Moderate source for gathering evidence related to viability

Rules of thumb while gathering evidence from different sources

• Customers>Competitors

• Focus on volume; not a single data point or anecdote

• Focus on gathering evidence for desirability first

The Framework

• Modes of enquiry for weak evidence

• Remove biases

• Hypothesize

• Prioritize

• Modes of enquiry for strong evidence

• Remove biases

• Repeat/Iterate

Gather weak evidence

• Modes of enquiry

  • Trend/competitor analysis

  • Customer interviews

  • Customer surveys

  • Internal team communications

  • Learnings from past experiments/experience

• Don't spend too much time on gathering weak evidence

• Weak evidence

  • Not statistically significant

  • Opinions/beliefs

• Filter out ideas that will NOT work

Remove Biases and Derive Key Insights

• Data and insights are different

• We all have conscious and unconscious biases

  • Work to disprove your beliefs

  • Have multiple people attend user interviews

  • Share raw data and not summaries

  • Have multiple people look at the evidence and share their insights

  • Encourage and carefully consider critical views

Key Hypothesis

• Convert your assumptions into the key hypothesis

  • Desirability hypothesis – pain-points, value proposition, engagement etc.

  • Feasibility hypothesis – resources, partnerships key activities etc.

  • Viability hypothesis – costs, pricing, profit etc.

• A good hypothesis

  • Is testable – ability to prove and disprove

    • X - Re-Designing the website will increase user engagement

    • V - Re-Designing the website with “design A” will increase total time spent on the website per user

• Has a clear goal and a success criteria

  • X - Re-Designing the website with “design A” will increase average time spent on the website per user

  • V - Re-Designing the website with “design A” will increase average time spent on the website per user, per week by 10% for the users between ages 20-45

• Is atomic – testing one variable at a time

  • X – We can buy and sell good for profit

  • V – H1: We can buy item X from the market at the rate of 3$ per item, if we buy more than 1000 pieces.

    • H2: We can sell each item for $6.

Prioritize

• Which hypothesis needs to be true for my idea to succeed?

• For which hypothesis do I lack concrete evidence?

Gather Strong Evidence

• Modes of enquiry

  • Hypothesis driven experiments – focus on volume

• Remember that while experiments provide strong evidence, they are costly to build and execute.

  • Run scrappy experiments if you can – Wizard of Oz, Fake landing pages etc.

• Strong evidence

  • Facts

  • What people do

  • Real world settings

  • Large investments

  • Statistically significant

Remove Biases and Derive Key Insights

You've collected more evidence. Convincing evidence that you can use again, you must make sure that you have removed your prejudices. It is very important to remove your biases when you analyze your evidence. In cases of experiments, it is really important to have a hypothesis that has a clear goal and success criteria. If you have a well-defined measure of success, the analysis of the results of the experiment becomes very clear. You either reached your goal or you didn't. If the results of your experiment are inconclusive, you might be tempted to put it into production, because it won't hurt anything if it doesn't improve anything. But it is here that the elimination of bias is what we must remember.
Just make sure you have a few people looking at the input as well, getting their perspective on it, removing bias, and doing the right thing for the product.

Repeat/Iterate

• Running multiple experiments for a single hypothesis increases strength of the validity of the hypothesis.

• Now you know more, you have reduced uncertainty. Repeat the same process to know even more and iteratively build your product.

• Remember, Product development is not a linear process, and you'll never know it all. But you can minimize uncertainty and increase chances of success.

Summary

• PMs need to minimize uncertainty associated with product desirability, feasibility and viability in order to build a successful product.

• When you are uncertain, your only goal should be to gather evidence that points you in the right direction.

• Major sources of evidence – customers, competitors and internal teams.

• Different modes of enquiry provide different degrees of evidence. Start with weak evidence and move quickly towards the modes of enquiry that gather strong evidence.

• Work to disprove hypothesis than to prove it and consciously remove biases when driving insights from the data.

• Remember that product development is not a linear process.

Vaibhavi Phadnis, Microsoft Product Leader