Focused on data infrastructure, pipeline health, and data readiness
Problem Areas: Data Acquisition, Data Lake/Warehouse Strategy & Success, Access Controls, Ease of Data Retrieval
Key Metrics:
Data Quality
Availability
Query Traffic
Latency and Reliability
Error Rates
Response Time
Focused on successful applications of machine learning in an organization
Problem Areas:
1) Vertically towards a product or problem
2) Horizontally as a capability is "pricing".
Key Metrics
1) Business outcomes themselves
2) Experimentation results
3)Model performance metrics
Predict stuff.
Classify stuff.
Group stuff.
Teach stuff.
Machines can do all of these things as long as it doesn`t look too different from what it`s "training on."
Therefore, it is always important for a product manager to understand the bias and what exactly you are learning.
You use these characteristics to predict variable outcomes that are home value.
There are multiple functions, inputs or variables, and on the right side, data scientists would predict the outcome or response variable.
For the next step in machine learning, if you want to predict any value, which are base values themselves, you can actually span the invisible ones, which span multiple records, or what people would call that invisible data, and practice on a few. Take the data, and then say, knowing the weight of these variables, how much should we weigh. We can actually come up with this weighted equation. For example, we can predict the value of a house using data on the number of bedrooms up to the city. The data you train on is representative of this unseen data.
Can you find the "weights" of each of these variables, learn the equation to home value, and predict the home values that are unseen?
Yes but...
Only if the data I can see now is representative of the weights that would apply to the data I can`t see.
Oh and...
And I`d probably do a better job if I practice with what the data I can see now by pretending to hide some of the records myself and seeing how I do...
What business problem are you solving?
How do you know the use case is amenable to ML?
Ask yourself...
Does this problem have fast feedback loops or ways to assess if we`re making better decisions because of this model?
Do we have a lot of data to train for this problem? Is the training data representative?
How easy is it to get more training data?
What would success look like?
What does the training data actually look like?
Is there enough?
Is it representative?
How hard is it to get more?
How hard is it to get more of the long-tail cases?
What do the features look like?
How do you assess data quality?
How easy is it to maintain?
Apply a heuristic to advance the problem. Can you even measure if this is better than the status quo? If you can't, don't apply ML.
If you can, great. At least you know what business metrics you're looking to improve.
What are your model's performance metrics and how will they relate to this business metric improvement? This is important for your team's morale and when you've decided you want to reallocate resources to another problem.
How will you know a model is ready to be put in production? How does it compare to status quo?
What metrics are you assessing?
And under what time horizon?
What is your experimentation framework?
Now that you've understood all of the above, it might be time to start applying ML to optimize or accelerate business outcomes.
But be careful. Most organizations over-invest in complex solutions without understanding how much cost is associated in acquiring the data and infrastructure, resources to build it out, and maintain the model in a production environment.
Sometimes (most of the time...) simpler is better.
Define a strong annotation schema
How does labeling work in your organization
Improving your model generally involves things like hard sample mining (adding new training data similar to other samples the model failed on), rebalancing your dataset based on biases your model has learned, and updating your annotations and schema to add new labels and refine existing ones.
Modify the output structure of the model to match the classes and structure of your labels.
Experiment Tracking – This entire cycle will likely require multiple iterations. You will end up training a lot of different models so being meticulous in your tracking of different versions of a model and the hyperparameters and data it was trained on will help a great deal to keep things organized.
Visualize Outputs
Choose the Right Metrics (& not too many metrics)
Look at Failure Cases
Dig into Failure Cases
Monitor Model
Evaluate New Data
Continue Understanding Model ( interpretability will matter here); what edge cases and biases do you need to watch out for?
Confusion Matrix
Accuracy
Precision/Recall
F1 Score
Sensitivity/Specificity
ROC/AUC
MSE (Mean Squared Error), RMSE
MAE (Mean Absolute Error), MAPE
R^2, Adjusted R^2
MAD (Mean Absolute Deviation)
Bayesian Information Criteria (BIC)
Correlation Coefficient
Lots More...! You should know these.
ML is the product
B2C, large-scale products
Small changes, big impact
Fast feedback loops
Clean classifications, easy training data
ML big bets
B2B account acquisition or conversion
Pricing with X months of data
Pricing with X training data set
Lack of experimentation to test value
Super nuanced human decision making is in these cases how do you get consistent training data?
Understand the problem → readiness → model build → metrics → evaluation cycle. What's not working? What are the bottle necks?
Understand the outcomes/know your performance metrics so you can help make decisions.
Understand the data assumptions so you have a handle on biases.
Practice the ML PM toolkit in traditional ways.
Keep up to date on ML foundational skills & new trends.
Malavica Sridhar, Waymo Product Leader