Machine Learning 101 for PMs by Waymo Product Leader, Malavica Sridhar

Types of Machine Learning Product Managers

1. Data Product Manager

  • Focused on data infrastructure, pipeline health, and data readiness

  • Problem Areas: Data Acquisition, Data Lake/Warehouse Strategy & Success, Access Controls, Ease of Data Retrieval

  • Key Metrics:

    • Data Quality

    • Availability

    • Query Traffic

    • Latency and Reliability

    • Error Rates

    • Response Time

2. ML Product Manager

  • Focused on successful applications of machine learning in an organization

  • Problem Areas:

    • 1) Vertically towards a product or problem

    • 2) Horizontally as a capability is "pricing".

  • Key Metrics

    • 1) Business outcomes themselves

    • 2) Experimentation results

    • 3)Model performance metrics

Types of Machine Learning

Largely, you can...

  1. Predict stuff.

  2. Classify stuff.

  3. Group stuff.

  4. Teach stuff.

Machines can do all of these things as long as it doesn`t look too different from what it`s "training on."

Therefore, it is always important for a product manager to understand the bias and what exactly you are learning.

You use these characteristics to predict variable outcomes that are home value.
There are multiple functions, inputs or variables, and on the right side, data scientists would predict the outcome or response variable.
For the next step in machine learning, if you want to predict any value, which are base values ​​themselves, you can actually span the invisible ones, which span multiple records, or what people would call that invisible data, and practice on a few. Take the data, and then say, knowing the weight of these variables, how much should we weigh. We can actually come up with this weighted equation. For example, we can predict the value of a house using data on the number of bedrooms up to the city. The data you train on is representative of this unseen data.

Can you find the "weights" of each of these variables, learn the equation to home value, and predict the home values that are unseen?

Yes but...

Only if the data I can see now is representative of the weights that would apply to the data I can`t see.

Oh and...

And I`d probably do a better job if I practice with what the data I can see now by pretending to hide some of the records myself and seeing how I do...

The ML Life Cycle: A Little Less Simple

Things Going Wrong Part I: Organizational Readiness

The Problem Assessment

  • What business problem are you solving?

  • How do you know the use case is amenable to ML?

  • Ask yourself...

  • Does this problem have fast feedback loops or ways to assess if we`re making better decisions because of this model?

  • Do we have a lot of data to train for this problem? Is the training data representative?

    How easy is it to get more training data?

    • What would success look like?

The Data & Infrastructure

  • What does the training data actually look like?

  • Is there enough?

  • Is it representative?

  • How hard is it to get more?

  • How hard is it to get more of the long-tail cases?

  • What do the features look like?

  • How do you assess data quality?

  • How easy is it to maintain?

The Metrics

  • Apply a heuristic to advance the problem. Can you even measure if this is better than the status quo? If you can't, don't apply ML.

  • If you can, great. At least you know what business metrics you're looking to improve.

  • What are your model's performance metrics and how will they relate to this business metric improvement? This is important for your team's morale and when you've decided you want to reallocate resources to another problem.

Experimentation

  • How will you know a model is ready to be put in production? How does it compare to status quo?

  • What metrics are you assessing?

  • And under what time horizon?

  • What is your experimentation framework?

Optimization: Time for ML!

  • Now that you've understood all of the above, it might be time to start applying ML to optimize or accelerate business outcomes.

  • But be careful. Most organizations over-invest in complex solutions without understanding how much cost is associated in acquiring the data and infrastructure, resources to build it out, and maintain the model in a production environment.

  • Sometimes (most of the time...) simpler is better.

Things Going Wrong Part II: ML Complexity

Data

  • Define a strong annotation schema

  • How does labeling work in your organization

  • Improving your model generally involves things like hard sample mining (adding new training data similar to other samples the model failed on), rebalancing your dataset based on biases your model has learned, and updating your annotations and schema to add new labels and refine existing ones.

Model

  • Modify the output structure of the model to match the classes and structure of your labels.

  • Experiment Tracking – This entire cycle will likely require multiple iterations. You will end up training a lot of different models so being meticulous in your tracking of different versions of a model and the hyperparameters and data it was trained on will help a great deal to keep things organized.

Evaluation

  • Visualize Outputs

  • Choose the Right Metrics (& not too many metrics)

  • Look at Failure Cases

  • Dig into Failure Cases

Production

  • Monitor Model

  • Evaluate New Data

  • Continue Understanding Model ( interpretability will matter here); what edge cases and biases do you need to watch out for?

The Role of an ML PM

Why does she keep talking about performance metrics?

Classification Metrics

  • Confusion Matrix

  • Accuracy

  • Precision/Recall

  • F1 Score

  • Sensitivity/Specificity

  • ROC/AUC

Regression Metrics

  • MSE (Mean Squared Error), RMSE

  • MAE (Mean Absolute Error), MAPE

  • R^2, Adjusted R^2

  • MAD (Mean Absolute Deviation)

  • Bayesian Information Criteria (BIC)

  • Correlation Coefficient

Lots More...! You should know these.

Where has ML typically gone right vs. wrong?

Good Use Cases

  • ML is the product

  • B2C, large-scale products

  • Small changes, big impact

  • Fast feedback loops

  • Clean classifications, easy training data

  • ML big bets

Bad Use Cases

  • B2B account acquisition or conversion

  • Pricing with X months of data

  • Pricing with X training data set

  • Lack of experimentation to test value

  • Super nuanced human decision making is in these cases how do you get consistent training data?

Takeaways

  1. Understand the problem → readiness → model build → metrics → evaluation cycle. What's not working? What are the bottle necks?

  2. Understand the outcomes/know your performance metrics so you can help make decisions.

  3. Understand the data assumptions so you have a handle on biases.

  4. Practice the ML PM toolkit in traditional ways.

  5. Keep up to date on ML foundational skills & new trends.

Malavica Sridhar, Waymo Product Leader