Bugs
Bugs, like OKRs, always require some tweaking and refining. Rather than tell one specific iteration we made towards optimizing bug process and prioritization, I’ll share where I’ve ended up after 9 years of trying to make this work, and have been quite pleased with where we stand today.
Reasoning.
I’ve seen where bugs have been handled by product, project, quality, and support teams. Here are some of the areas where I’ve seen the biggest breakdowns:
All bugs being too noisy and being a distraction to teams
Critical bugs not being addressed with enough urgency
Misalignment of the priority of bugs
Unclear expectations as to when a bug will be addressed
Framework.
Priority Level
-
Indicators
• Severity Score Range: 18+
• Directly impacting conversion and revenue
• Outages
• All users are unable to use the product
Action
• Once identified as a potential P0, Product Director(s) and Group Leader(s) are to be looped in immediately to evaluate with key stakeholders.
• Engineering to halt all existing work and immediate work to resolve the issue.
• Post the issue in #incident for:
• Communication of status
• Collaborate to get key people looped in
• Report regularly on progress
-
Indicators
• Severity Score Range: 8-18
• High Traffic: +50% of the potential population affected.
• Most users are unable to use the product
• Indirectly impacting conversion and revenue
Action
• Once identified as a potential P1, Product Director(s) and Group Leader(s) are to be looped in immediately to evaluate with key stakeholders.
• Once the priority is confirmed, a card is to be placed into the existing sprint of the appropriate squad.
• Engineering is to finish existing work and then begin working to resolve the issue next.
-
Indicators
• Severity Score Range: 6-8
• Medium Traffic: <50% of the potential population affected.
• Some users are unable to use the product
• No impact on conversion and revenue
Action
• Once identified as a potential P1, Product Director(s) and Group Leader(s) are to be looped in immediately to evaluate with key stakeholders.
• Once the priority is confirmed, a card is to be placed into the existing sprint of the appropriate squad.
• Engineering is to finish the existing sprint and then begin working to resolve the issue next.
-
Indicators
• Severity Score Range: 3-6
• Low Traffic: <30% of the potential population affected.
• A few users are unable to use the product
• No impact on conversion and revenue
Action
• Once the priority is confirmed, a card is to be placed into the backlog of the appropriate squad.
• The Product Trio is to evaluate against existing priorities with a commitment to get it started by within the next 2 sprints.
-
Indicators
• Severity Score Range: -3
• Cosmetic issues, but not impacting a users ability to use the product
Action
• Once the priority is confirmed, a card is to be placed into the existing sprint of the appropriate squad.
• The Product Trio is to evaluate against existing priorities and fit into the backlog accordingly.
Prioritization Scoring
In order to remove the biases of a bug, use the following framework to get a priority score: (Reach * User Impact * Severity * Workaround) + Effort + Long Term Impact
-
Score:
≥35% of users affected: 5
<35% of users affected: 4
<20% of users affected: 3
<10% of users affected: 2
<5% of users affected: 1
What percent of users going through this experience have this issue?
-
Score:
Critical Usability Issue: 2
Usability issues either prevent users from completing their task or cause significant confusion as to the operation of the UI.Major Usability Issue: 1
Usability issues result in significant inefficiency in completing common tasks.Minor Usability Issue: 0.5
Usability issues do not prevent task completion but do cause inefficiency or annoyance with prolonged use.
-
Score:
Critical impact to business: 5
Significant impact to business: 2
Moderate impact to business: 1
Low impact to business: 0.5
Minimal impact to business: 0.25
No impact to business: 0
Is this impact of this issue impacting the business from a revenue perspective?
-
Score:
Yes: 0.5
No: 1
Is there a workaround for the user to complete the task?
-
Score:
Unknown: 1
<1 day to fix: 4
≤1 week to fix: 3
≤1 sprint to fix: 2
What is the scope to fix the bug? Based upon the context as to how much effort it takes to fix an issue, it makes the priority score higher.
-
Score:
Significant Benefit: 4
Moderate Benefit: 3
Minimal Benefit: 2
Unsure of Benefit: 1
No Benefit: 0
What are the long-term repercussions to this bug continuing to exist?
Process.
1. Bug Intake
All reported issues from users and internal stakeholders are brought into a single board, and preliminary efforts to diagnose where the issue is either confirmed to be a bug or feature request.
2. Bug Vetting
All reported issues confirmed to be a bug are then vetted by the quality team to:
A technical summary of the bug
Provide the steps to reproduce the bug
The expected amount of users to be impacted
Define the location and browsers showing the bug
3. Bug Prioritization
With the context from bug vetting, group-level leadership will score the bug and assign it to the appropriate product teams with clear expectations as to when the bug must be addressed.
4. Bug Reporting
The support team sends bug updates to key stakeholders discussing the amount of prioritized bugs and how many outstanding issues exist. On a monthly basis, critical bugs are reported with other delivery items.
Outcomes.
Utilizing the following bug process and framework, it increases transparency of how bugs have achieved a priority level, sets clear expectations and creates accountability for all parties as to when a bug will be addressed and how it impacts existing priorities.
At homie, we were able to greatly reduce the friction as to bugs being reported, the chaos for the squads in scrambling to meet company expectations to fix issues, as well as ensure that critical issues are being addressed with that appropriate urgency.