TL;DR “a classification decision depends on why the system exists and what we’re trying to achieve with it.” — Cassie Kozyrkov
The above video offered a simple yet insightful observation about the role of end-to-end system understanding into modeling. Here are key take-aways for any product managers:
- Every ML model that the product depends on need to align to the end goal of the system. Don’t take a generic model for granted as someone might have created/tuned the recall/precision of that model for a totally different purpose.
- Be empathized with ML people when we push for ML feedback in design mock-up, MLP, private beta, PM manual review, etc. We know that it might go against the lean principle. However, you need to understand the downstream impacts to get ML right and it’s hard to do so by looking at a proposal or design doc.
- Offline performance treats all errors the same. However, some errors are more important than others (in a pet recommender system, wrongly classifying a tiger as a “cat” is more harmful than mistakenly recognizing a hamster). So, PM needs to intimately follow the modeling development and guide the team.
