Scenario: You are a Product Manager working on the ML-focused part of the product or a Machine Learning Engineer/researcher working on several project ideas.
Question:
- How do you prioritize between two or three ML ideas?
- What are the common red flags of machine learning projects?
Disclaimer: this process is for big companies only. If you’re a start-up, it will not always work.
How are (ML) Modeling projects different?
ML modeling projects are different from software development projects in three main things:
- Modeling problem vs Product problems:
- Uncertainty of Modeling Headroom:
- The Needs for Labeled data:
Steps for prioritization:
Step 1: Light-filter your idea list to two or three ideas.
You need to narrow down your ideas to two to three ideas at max. Why? Because if not, you can spend a quarter to run various analyses in Step 2.
If you’re part of a big company, ask your (product or engineering) managers for the company’s and department’s priorities. Then, map out how each of your ideas can benefit the company’s prioritized workstreams.
You want to be in the most important workstreams because that is where your company needs help and where you can have more resources/support if you hit a wall.
Special scenarios:
- If such a list doesn’t exist anywhere and you’re in a big company, you’re in big trouble. It’s time to talk to your managers or skip for clarity.
- If most of your ideas cluster in one workstream of the company, maybe the right question is not prioritization but sequencing. If so, the rule of thumb for all modeling projects is to start simple, build a baseline, and iterate your way up (more features, complex architectures, etc.).
Step 2: Modified the RICE Framework for ML projects
- Reach: Dissect your target “customers” and understand your reach. If you develop a model, your “customers” are your prediction target. Some of the ways to think about reach are: the scope of the prediction (e.g. if you build a Twitter recommendation model, does the model act on original tweets, retweets, replies or only replies?), countries, and surface areas (is this model be applied to multiple places in the product?).
- Headroom (Impacts & Confidence): For modeling projects, I usually have problems to estimate impacts and confidence independently. To me, they are related. So, I want to tackle them together.
First and foremost, ask yourself if your customers’ problem is a product problem or modeling one? Product problems have to do with the hypotheses of the product (“if we provide a nudge to customers’ replies, we expect less abusive replies because customers will think twice”) whereas the modeling problem deals with the relationship between model performance and the overall end-to-end value chain (“if we increase the model precision by x, we expect a reduction in bounce rate”). If there is a flaw in your product hypotheses (not all abusive replies are “naive” or happens in a heated moment; most of them are intentional and no matter how we nudge customers, they will ignore), your modeling impacts on the end problem will be limited.
Second, examine the baseline and compare it with your ideas. For modeling projects, diagnose the model before blindly increasing training data, making the model more complex, or adding more features. Your first modeling improvement task should always be a diagnostic task.
Third, hopefully, at this point, you can say with confidence the potential improvement headrooms. Because of the expense of your diagnostic tasks, you can’t do this for 10 ideas. You must narrow down to a few ideas/hypotheses first.
At this point, you can multiply Reach (e.g. number of customers) and Headroom (e.g. % conversion improvements) to arrive at the final impacts. If you can’t get a quantitative assessment, a high-level qualitative (High, Medium, Low) is still better than nothing. If you reach a dire High Reach — Medium Impacts versus Low to Medium Reach — High Impacts, let’s talk about Opportunity Cost.
- Opportunity Cost
If you don’t jump on this idea now, what would happen?
- If you block a feature General Availability, ask for the launch criteria and go back to Step 2. For most of the modeling projects, always seek simple baseline or even rule-based for the MVP launch. You need data to train a more complex model anyway, and you need to learn/validate the product hypotheses before you fall in love with a modeling technique.
- If this is part of an evergreen workstream, maybe you have time to join later… but maybe not if there is another team with similar ideas. In this case, talk to that team and see if join-force makes sense.
- If your managers want to fund a bigger bet next year but they want to see some early results first, jump right in! Chances are you will own such big pieces if you lead the change now. “The best way to predict the future is to create it.”
It’s hard to see the opportunity cost. It’s ok, explicitly state your assumptions so that your partner teams and managers understand your thought process.
Step 3:
Usually, the below priorities should have fewer impacts on your prioritization decisions. Assign them a lighter weight or rate them with a smaller scale (0/1 vs 0/4).
- Align with Your Team’s Direction (On Theme):
Your team should have a charter or a conviction around the areas that you want to focus on. Review with your teammates and managers if this project adds value to your team’s portfolio and paves the way for you to expand your portfolio. If not, move on.
Your team should also possess a differentiator from other teams in the same company. Review why/how this project must be done by you and not any other teams. If there is a team that is better suited to execute this, but somehow you’re asked to do it, understand why before committing.
- External Visibility (Publications):
Don’t assume that only researchers can publish papers. We have Medium, Blog Posts, PodCast, and the trend of ML is towards open source. Check if such projects have potentials for external visibility. Such visibility can help you build a community in your industry, which is good for team morale and recruitment. Your manager will appreciate it.
- Personal Growth:
Last, you should always learn something new in every project. If you have done the same project three times, something is not right. Check if there is a more junior team member whom you can coach or if it’s not a prioritization but a career planning session with your managers.
Ok, if you make it here and decide on a project, let’s understand the red flags.
Red Flags to look out for:
- Ownership & Resources
- Data Availability
- Launch Blocker
