To know whether a shipped change worked, write down the expected effect before release — which metric, which accounts, by how much, over what window — then compare against it afterwards. For fixes, the clearest signals are the repeat-ticket rate for that pattern and usage of the affected feature by the affected accounts. For features, it's adoption and whether adopters stick around. The point isn't the report; it's deciding the next action: done, iterate, or roll back.
Most teams skip this step. They ship, watch Slack for complaints, and if nothing blows up they call it a win. That's not measurement, it's the absence of noise. It also means every release is a one-way door: you can't tell your next prioritisation round what actually happened, so it starts from the same guesswork as the last one.
What should you write down before you ship?
A pre-release hypothesis takes five minutes and forces the only honest question: what do you expect to happen, specifically? Use this template for every fix and every feature above trivial size:
- Change: what shipped, one sentence.
- Metric: the single number you expect to move (not three — one, with others as secondary).
- Population: which accounts or users should be affected. Everyone, or the 40 accounts who filed the ticket?
- Expected direction and size: "repeat-ticket rate for this pattern drops to near zero" or "adoption among invited users passes 30%." Vague targets like "improves" don't let you fail, which means you'll never learn.
- Measurement window: when you'll check, and when you'll check again.
- Decision rule: what result triggers what action, decided now, not after you see the number.
Writing this down is the whole discipline. If you can't fill it in, you probably haven't decided why you're shipping the change — see Outcome-Based Roadmaps for how to tie work to a goal before it reaches a sprint.
Which metrics prove a bug fix actually worked?
Bug fixes are easier to verify than features because you have a before-state: the pattern of tickets that prompted the work. Two signals matter:
- Repeat-ticket rate for that specific pattern. Not overall ticket volume — the exact cluster of complaints tied to the bug. If it doesn't drop to near zero within one or two support cycles, the fix didn't land or didn't cover the full case.
- Usage of the affected feature, by the affected accounts. A bug that was blocking a workflow should show usage recover for the accounts who were stuck, not just hold steady in aggregate.
Checking tickets and usage per account, not in aggregate, is the difference between "looks fine" and "fixed for the people who actually hit it." See Bug Triage: Repeat Tickets Into Fixes for how to trace a ticket pattern back to the code in the first place.
Which metrics prove a feature shipped well?
Features need a different lens because there's no "before" complaint to resolve — you're trying to prove people want and keep using something new.
| Change type | Primary metric | Secondary signal |
|---|---|---|
| Bug fix | Repeat-ticket rate for the pattern | Usage recovery in affected accounts |
| New feature | Adoption rate (% of target users who try it) | Retention of adopters at week 2/4 |
| Performance work | The latency/error metric itself (p95, error rate) | Downstream usage or completion rate on the affected flow |
Adoption without retention is a vanity number — people clicked it once out of curiosity. Amplitude and Pendo both write at length about adoption-curve shapes; worth a read if you want the deeper mechanics (Amplitude's product metrics blog, Pendo on feature adoption). This article won't repeat A/B testing methodology — that's its own discipline, worth a dedicated read elsewhere — but the hypothesis template above works whether you split-test the change or ship it to everyone at once.
How long should you wait before measuring?
Long enough to cover a full usage cycle for that workflow, short enough that the result still informs the next planning cycle. As a rule of thumb:
- Bug fixes: one full support cycle (usually 1–2 weeks) to confirm the ticket pattern stopped recurring.
- Features used daily: 1–2 weeks for initial adoption, 3–4 weeks for retention of adopters.
- Features used weekly or monthly (reporting, billing, admin tools): at least one full cycle of that cadence — checking a monthly report feature after four days tells you nothing.
- Performance fixes: measure within hours for the metric itself, but watch downstream usage for a week to see if the fix changed behaviour, not just the dashboard.
Pick the window before you ship, not after you see an inconvenient number — that's the whole point of writing it down.
What can throw off your attribution?
A metric moving isn't the same as your change causing it. Before you credit (or blame) a release, check for:
- Seasonality. Usage dips every August or every Monday regardless of what you shipped. Compare against the same period last cycle, not just the week before.
- Concurrent releases. If three things shipped the same week, you can't cleanly attribute a metric move to any one of them without holding other variables constant — which is what proper A/B testing is for.
- Small samples. A fix for 12 accounts moving from "3 tickets a week" to "0" is meaningful. A feature adopted by 8 out of 400 eligible users is not yet a trend either way — wait for more data before you decide.
- Measuring the wrong population. If the fix targeted enterprise accounts on a specific integration, checking overall product usage dilutes the signal to nothing.
When you genuinely can't measure — the change is too small, the population too thin, or the metric too noisy — say so explicitly rather than inventing a result. "Not enough volume to call it yet, recheck in 30 days" is a legitimate, honest outcome. It's also a decision: hold, don't iterate blind.
What's the decision rule once you have a result?
Every measurement should resolve to one of three actions, decided in advance as part of the hypothesis:
- Done. Metric moved as expected, within the window, for the right population. Close it out, tell the customers who asked — see Closing the Feedback Loop — and move on.
- Iterate. Directionally right but short of target, or adoption without retention. Scope a follow-up, don't re-litigate the whole feature.
- Roll back or rethink. No movement, or the metric moved the wrong way. This is the result teams most avoid writing down — which is exactly why it needs to be a rule, not a judgment call made under pressure to defend the work.
How do you make this continuous instead of a one-off report?
A single post-mortem is better than nothing, but the real value is making the check automatic and feeding it straight back into prioritisation — so the next ICE or RICE score (see ICE Scoring Model) is informed by what actually happened last time, not just what the team hopes will happen this time.
This is the step most tooling in this category stops short of. Feedback and roadmapping tools help you decide what to build; almost none of them close the loop on whether it worked. Insight Tools vs Fixing Tools covers that gap in more depth.
FAQ
How do you measure the success of a feature?
Define the target population and expected adoption rate before release, measure adoption within one to two weeks, then check retention of those adopters three to four weeks out. Adoption without retention usually means curiosity, not value.
What metrics show a bug fix worked?
The repeat-ticket rate for that specific pattern should drop to near zero within one support cycle, and usage of the affected feature should recover among the accounts who were previously blocked — check per-account, not just in aggregate.
How long should you wait before measuring a release?
Long enough to cover one full usage cycle of the affected workflow: 1–2 weeks for daily-use features or bug fixes, a full month for monthly-cadence features like reporting or billing, and hours (plus a week of downstream watching) for performance fixes.
What is feature adoption rate?
Feature adoption rate is the percentage of your target user population who use a feature within a defined window after it's available to them — not the percentage of your whole user base, which understates adoption for features aimed at a specific segment.
What should you do if a feature didn't work?
Follow the decision rule you set before shipping: if the metric moved partway, scope a focused iteration; if it didn't move at all or moved the wrong way, roll back or rework the approach rather than leaving it live and hoping usage catches up later.
See how VocxAI builds this for you
VocxAI is the AI agent that keeps improving your product: it finds the patterns in customer feedback from any tool you use, works out what's wrong in your code, has AI agents build the fix for your approval, then checks whether it worked and acts on what it learns.
Sign up for free