Resilience Solutions
Articles

Reducing testing time without reducing coverage

When a regression pass takes two days, the instinct is to automate more of it. The faster answer is usually to move it to a different layer.

Written by
Quality Engineering practice
Reviewed by
Engineering review
Published
Reading time
6 min

"Reduce testing time" is one of the most common briefs we receive, and it is almost always framed as an automation problem. The team has a two-day manual regression pass, so the proposal is to automate the two-day regression pass.

That works, briefly. An automated end-to-end suite that reproduces a manual script tends to arrive somewhere around four hours, then grows back toward a day as the product grows, and brings a flake rate that erodes trust in it. The underlying issue was never that the tests were manual. It was that they were at the wrong layer.

What the layer actually costs you

An end-to-end test exercises the browser, the application, every service it calls and the database. That is why it is slow, and why it fails for reasons unrelated to the behavior it was written to check. When it fails, someone has to work out which of those layers broke.

The same assertion made at the service or unit layer runs in milliseconds, fails for exactly one reason, and tells you where. The behavior verified is identical. The cost of verifying it is not.

A workable redistribution

The aim is not to delete end-to-end tests. It is to leave them doing the job only they can do, which is proving that the pieces are wired together.

  • Unit tests for business logic and edge cases, which is where most of the real risk lives.
  • Contract tests at service boundaries, so services can deploy independently without integrating to find out.
  • API-level tests for workflow behavior, replacing most of what UI tests are currently asked to do.
  • A small end-to-end suite covering the critical journeys, and only those.
  • Exploratory testing by people who understand the domain, which finds a different class of defect than any automation.

Measure the right things

Coverage percentage is a poor signal on its own. More useful: how long the pipeline takes to give a verdict, what proportion of failures are genuine, how many defects escape to production, and how long they take to detect.

A suite with lower line coverage, a one-percent flake rate and a ten-minute runtime is worth considerably more than one with higher coverage that nobody trusts and everybody reruns.

Where this does not apply

Configuration-heavy platforms such as ERP and CRM change through configuration as well as code, and much of their behavior genuinely only exists at the integrated layer. There the emphasis shifts toward business-process automation and regression against each vendor release, rather than toward unit tests.

The principle still holds, though: push each assertion to the lowest layer that can honestly make it.


Discuss your testing strategy

If this describes something you are working through, we can talk it through against your specific environment.

  • Banking & Financial Services

    Modernize banking experiences, strengthen digital operations and activate trusted data while keeping security and resilience central.

  • Retail & E-commerce

    Create relevant customer experiences while improving merchandising, fulfilment and margin.

  • Telecommunications

    Modernize platforms and operations to improve service quality, field productivity and network insight.