How to test a feature removal before you ship it.
A quarter-close CSV export shows how to look beyond monthly usage, test the handoff the feature supports, and decide whether it can go.
Keep a feature until you know what happens to the work it supports. A CSV export can look unused on a monthly dashboard and still be the only way a finance team gets invoice records into its accounting system at quarter close. Before retiring it, review usage across the full close cycle and watch customers complete the same job with the proposed replacement.
Count accounts through the deadline
Consider an invoicing tool whose product team wants to replace a manual CSV export with a dashboard download. The counts and test results in this example are illustrative. The team follows 120 accounts that had export access throughout the last completed quarter and its five-business-day close window.
Six of those 120 accounts exported in the middle month of the quarter, a usage rate of 5%. Across the quarter and close window, 27 of 120 accounts exported, or 22.5%. Eighteen of the 27 exported only during the five days after the quarter ended. A team using the monthly figure to recruit test participants would have missed every one of those 18 accounts.
Cover the period leading to the deadline and the days customers spend closing it. Count distinct eligible accounts that completed the export and check whether scheduled jobs or support-assisted exports show up in a separate record. Then speak with the people who receive the output. The person who presses Download may be handing the file to a controller who never logs in.
Rehearse the handoff with the new workflow
Give the people who do the work a realistic invoice batch and the same completion criteria for each route. Ask them to find the right reporting period, download the file, import it into the system they use, and reconcile the totals. Use equivalent data for both attempts and vary which route participants try first so the second attempt has less of a practice advantage. Include the permissions they normally have at close.
In the example, six accounts rehearse the task, including accounts that use the CSV only at close. All six complete it with the original export. Four complete it with the dashboard download. The other two can download the file, but their importer requires an invoice-level tax code that the new file omits. Counting successful downloads would miss both failures.
Record where each attempt stops, how long it takes, and what help the customer needs. Agree before testing which failures should hold the release. For this team, a file that cannot be imported or reconciled by the reporting deadline is enough to keep the old route available.
Make the keep, replace, or phase-out call
Copy the decision sheet into the release brief. Beside the chosen row, record the affected accounts, the work cycle reviewed, the task results, and the owner of the next check.
Decision | Evidence and next step |
|---|---|
Keep | A required job still depends on the feature, or the replacement breaks a critical handoff. Preserve access and assign the specific gap to an owner. |
Replace | The job still matters and affected customers can complete it with the new workflow. Migrate in stages and verify the first live cycle. |
Phase out | The full-cycle review finds no remaining dependency on the original route. Give notice, name the cutoff date, and define when to restore access. |
For the CSV export, keep the existing download through the next close. Add the missing field and ask both blocked accounts to rerun the import. If they can reconcile the output, run the two routes in parallel for a close before setting the retirement date. Give users time to update downstream processes and tell them where to go for help. GOV.UK’s guide to retiring a service
If accounts leave during the rollout, trace the timeline from their last successful close through any failed task, support request, and cancellation. Compare those episodes with accounts that stayed to see which explanation fits the evidence. Our churn research guide
