Release Testing is high cost, low value risk management theatre

Described by Elisabeth Hendrickson as originating with the misguided belief that “testers test, programmers code, and the separation of the two disciplines is important“, the traditional segregation of development and testing into separate phases has disastrous consequences for product quality and validates Jez Humble’s adage that “bad behavior arises when you abstract people away from the consequences of their actions“. When a development team has authority for changes and a testing team has responsibility for quality, there will be an inevitable increase in defects and feedback loops that will inflate lead times and increase organisational vulnerability to opportunity costs.

Release Testing - Develop and Test

Agile software development aims to solve this problem by establishing cross-functional product teams, in which testing is explicitly recognised as a continuous activity and there is a shared commitment to product quality. Developers and testers collaborate upon a testing strategy described by Lisa Crispin as the Testing Pyramid, in which Test Driven Development drives the codebase design and Acceptance Test Driven Development documents the product design. The Testing Pyramid values unit and acceptance tests over manual and end-to-end tests due to the execution times and well-publicised limitations of the latter, such as Martin Fowler stating that “end-to-end tests are more prone to non-determinism“.

Release Testing - Product Team

Given Continuous Delivery is predicated upon the optimisation of product integrity, lead times, and organisational structure in order to deliver business value faster, the creation of cross-functional product teams is a textbook example of how to optimise an organisation for Continuous Delivery. However, many organisations are prevented from fully realising the benefits of product teams due to Release Testing – a risk reduction strategy that aims to reduce defect probability via manual and/or automated end-to-end regression testing independent of the product team.

Release Testing - Release Testing

While Release Testing is traditionally seen as a guarantee of product quality, it is in reality a fundamentally flawed strategy of disproportionately costly testing due to the following characteristics:

  1. Extensive end-to-end testing – as end-to-end tests are slow and less deterministic they require long execution times and incur substantial maintenance costs. This ensures end-to-end testing cannot conceivably cover all scenarios and results in an implicit reduction of test coverage
  2. Independent testing phase – a regression testing phase brazenly re-segregates development and testing, creating a product team with authority for changes and a release testing team with responsibility for quality. This results in quality issues, longer feedback delays, and substantial wait times
  3. Critical path constraints – post-development testing must occur on the critical path, leaving release testers under constant pressure to complete their testing to a deadline. This will usually result in an explicit reduction of test coverage in order to meet expectations

As Release Testing is divorced from the development of value-add by the product team, the regression tests tend to either duplicate existing test scenarios or invent new test scenarios shorn of any business context. Furthermore, the implicit and explicit constraints of end-to-end testing on the critical path invariably prevent Release Testing from achieving any meaningful amount of test coverage or significant reduction in defect probability.

This means Release Testing has a considerable transaction cost and limited value, and attempts to reduce the costs or increase the value of Release Testing are a zero-sum game. Reducing transaction costs requires fewer end-to-end tests, which will decrease execution time but also decrease the potential for defect discovery. Increasing value requires more end-to-end tests, which will marginally increase the potential for defect discovery but will also increase execution time. We can therefore conclude that Release Testing is an example of what Jez Humble refers to as Risk Management Theatre - a process providing an artificial sense of value at a disproportionate cost:

Release Testing is high cost, low value Risk Management Theatre

To undo the detrimental impact of Release Testing upon product quality and lead times, we must heed the advice of W. Edwards Deming that “we cannot rely on mass inspection to improve quality“. Rather than try to inspect quality into each product increment, we must instead build quality in by replacing Release Testing with feedback-driven product development activities in which release testers become valuable members of the product team. By moving release testers into the product team everyone is able to collaborate in tight feedback loops, and the existing end-to-end tests can be assessed for removal, replacement, or retention. This will reduce both the wait waste and overprocessing waste in the value stream, empowering the team to focus upon valuable post-development activities such as automated smoke testing of environment configuration and the manual exploratory testing of product features.

Release Testing - Final Product Team

A far more effective risk reduction strategy than Release Testing is batch size reduction, which can attain a notable reduction in defect probability with a minimal transaction cost. Championed by Eric Ries asserting that “small batches reduce risk“, releasing smaller change sets into production more frequently decreases the complexity of each change set, therefore reducing both the probability and cost of defect occurrence. In addition, batch size reduction also improves overheads and product increment flow, which will produce a further improvement in lead times.

Release Testing is not the fault of any developer, or any tester. It is a systemic fault that causes blameless teams of individuals to be bedevilled by a sub-optimal organisational structure, that actively harms lead times and product quality in the name of risk management theatre. Ultimately, we need to embrace the inherent lessons of Agile software development and Continuous Delivery – product quality is the responsibility of everyone, and testing is an activity not a phase.

Upcoming talks on Continuous Delivery

I will be presenting a new design-oriented Continuous Delivery talk called “Refactoring for Continuous Delivery: Breaking the Mould” at the following events:

Hope to see you at one of the above!

Consumer Driven Contracts enable independent releases of interdependent applications

When applying Continuous Delivery to an application estate, our ability to rapidly release individual applications is constrained by inter-application dependencies. We want to release applications independently in order to minimise batch size and cycle time, but the traditional risk reduction methods associated with inter-application dependencies compromise lead times and are fundamentally incompatible with Continuous Delivery. How can we enable the independent evolution of interdependent applications with minimal risk?

For example, consider an estate initially comprised of two independently versioned applications – a Provider application that offers name/address/phone details, and a Consumer A application that uses name/address details.

Provider with Consumer and Provider Contract

In this scenario, the interdependent relationship between Provider and Consumer A means an independent release of either application will increase the probability of a runtime communications failure between the applications. As Consumer A has little knowledge of the Provider a new Consumer A binary might unintentionally alter its usage of the Provider API, and conversely as the Provider has zero knowledge of Consumer A a new Provider binary might break API compatibility for an unchanged version of Consumer A.

The fear of change and uncertainty associated with this inter-application dependency is exacerbated when another consumer with different requirements is added to the application estate, such as a Consumer B application requesting name/address/email details from the Provider.

Provider with Consumers and Provider Contract

The introduction of Consumer B adds another inter-application dependency to our estate and increases the complexity of our independent release strategy. We must be able to release a new Provider binary capable of servicing Consumer B without endangering the integrity of existing API conversations with Consumer A, but this is difficult when the Provider is unaware of Consumer A and Consumer B requirements. To solve this problem a number of risk reduction methods may be proposed, each of which comes at a significant cost:

  • Coupled releases. Releasing new versions of the Provider, Consumer A, and Consumer B simultaneously is a direct impediment to Continuous Delivery, with increased holding costs and transactions costs resulting in artificially inflated batch sizes and lengthy lead times
  • End-to-end testing. An end-to-end test suite covering all possible API interactions would be costly to maintain and a drain upon lead times, hence Jez Humble arguing it “delays feedback on the effect of the change on the system as a whole” and JB Rainsberger stating “integrated tests are a scam
  • Multiple Producer API operations. This shares the cost of change between the Provider and Consumer B without impacting Consumer A, but there is an increase in Provider application complexity and there is no incentive for Consumer A to migrate to the same service as Consumer B

Given that all of the above methods will increase the cost of change and cause the evolutionary design of our application estate to stagnate, we need a more adaptive solution that will favour Build Quality In over Risk Management Theatre. We can reduce the consumer-side risk of each inter-application dependency by implementing the Tolerant Reader and API Examples patterns within Consumer A and Consumer B, but this solution is incomplete as it cannot address the provider-side risk – a new Provider binary could still harm unchanged versions of Consumer A and/or Consumer B running in production.

Consumer Driven Contracts is a pattern originally described by Ian Robinson for Service-Oriented Architectures that has since proven applicable to any scenario in which evolving consumer-provider relationships need to be documented and continually verified. Characterised by Ian as an attempt “to create a vocabulary for describing the roles of the participants in a loosely coupled relationship“, the Consumer Driven Contracts pattern defines three related contract types:

  1. A Provider Contract is a description of a service offered by a provider
  2. A Consumer Contract is a description of the extent to which a consumer utilises a Provider Contract
  3. A Consumer Driven Contract is a description of how a provider satisfies an aggregate of Consumer Contracts

With Consumer Driven Contracts, the subset of information within the Provider Contract that a consumer is actually dependent upon forms a Consumer Contract that is communicated back to the Provider at build time. It is then the responsibility of the Provider to run a suite of unit tests in its commit build against each of its Consumer Contracts to ensure that no provider change could unexpectedly cause a communications failure at run time.

Consumer Driven Contracts

Consumer Driven Contracts are relatively cheap to implement, yet enormously powerful. As the Consumer Contracts supplied to the provider are executable specifications they provide an automated documentation mechanism that increases the visibility of existing inter-application relationships, and enable a testing strategy that favours the fast feedback of unit tests over the complexity and unreliability of end-to-end tests. Given that Consumer Driven Contracts enable us to visualise inter-application relationships and detect potentially harmful changes within seconds, Ian was not exaggerating when he stated that Consumer Driven Contracts “give us the fine-grained insight and rapid feedback we require to plan changes and assess their impact“.

If we apply the Consumer Driven Contracts vocabulary to our earlier example, it is apparent that our Provider application is offering a name/address/phone/email Provider Contract, with Consumer A encapsulating a private name/address Consumer Contract and Consumer B encapsulating a private name/address/email Consumer Contract.

Provider and Consumers with Consumer Contracts

These Consumer Contracts should be elevated to first class concepts and persisted within the Provider application as a Consumer Driven Contract, so that the Provider is made aware of consumer expectations and can be independently released without fear of harming any inter-application dependencies.

Provider and Consumers with Consumer Driven Contracts

Regardless of original intent, the Consumer Driven Contracts pattern is of immense value to Continuous Delivery and is an essential tool when we wish to independently release interdependent applications.

Divide and conquer vertically to reduce cost of change

Application architecture is an oft-overlooked aspect of Continuous Delivery, and an application that encapsulates orthogonal business capabilities is a direct impediment to our stated aim of rapidly releasing small changesets to improve our cycle time. How can we better align technical capabilities with business capabilities, and consequently improve our release cadence?

For example, consider a Fruit Basket application that contains unrelated Apples and Bananas business capabilities, both of which rely upon a messaging service to integrate with a third-party endpoint.

Divide Conquer Monolith

This is clearly an ineffective architecture as it encapsulates unrelated business capabilities with different rates of change, violating Bob Martin’s Single Responsibility Principle and Kevlin Henney’s assertion that “an effective architecture is one that generally reduces the significance of design decisions“. A Fruit Basket release that contains new Apples functionality must also include the unchanged Bananas functionality, creating an inflated changeset that increases transaction costs and the probability of failure.

We can reduce the cost of change and enable Continuous Delivery by using Divide and Conquer to split Fruit Basket into independent applications, but it is important to assess the merits of different strategies. Horizontal Divide and Conquer favours a division of technical capabilities, and would result in separate Fruit Basket and Messaging applications.

Horizontal Divide Conquer

While Horizontal Divide and Conquer allows for independent releases of Fruit Basket and Messaging, it ignores the fact that the variation between individual business capabilities will be greater than the variation between business and technical capabilities. Over time there will be far more orthogonal Apples/Bananas requirements than orthogonal Fruit Basket/Messaging requirements, with each Fruit Basket release still a bloated changeset – and when an Apples or Bananas requirement needs a Messaging change, the changeset will grow even larger.

In order to have a significant impact upon cycle time, we must value the decoupling of business capabilities over the deduplication of technical capabilities and adopt a Vertical Divide and Conquer strategy. Vertical Divide and Conquer favours a division of business capabilities, and would result in separate Apples and Bananas applications.

Vertical Divide Conquer

By creating independent Apples and Bananas applications we align our technical capabilities with our business capabilities, and respect the different rates of change in the Apples and Bananas business domains. This will ensure each Apples and Bananas release consists of a minimal changeset, unlocking batch size reduction benefits such as improved transaction costs, improved risk, and improved cycle time.

If we identify Messaging duplication as an issue after Apple and Bananas have been decoupled, we can further improve our architecture by extracting Messaging as an independently versioned library. This will further shrink Apples and Bananas changesets, and the introduction of a Messaging Published Interface will make it easier to reason about responsibilities and collaborators.

Optimise Vertical Divide Conquer

The corollary to Vertical Divide and Conquer is Conway’s Law, which tells us for our vertically aligned business and technical capabilities to be truly successful we must also re-structure our organisation so that our development teams are vertically aligned with singular responsibility for specific business capabilities.

API Examples enable consumer unit testing of producer APIs

When an application consumes data from a remote service, we wish to verify the correctness of consumer-producer interactions via a testing strategy that encompasses the following characteristics:

  • Fast feedback
  • 100% scenario coverage
  • Representative test data
  • Auto-detect API changes

The simplest method of verifying parser behaviour would be to use Test Driven Development and create a suite of unit tests reliant upon self-generated test data. These tests could provide feedback in milliseconds, and would be able to cover all happy/sad path scenarios. However, consumer ownership of test data increases the probability of errors as highlighted by Brandon Byars warning that “hard-coding assumptions about data available at the time of the test can be a fragile approach“, and it leaves the consumer entirely unaware of API changes when a new producer version is released.

Consumer Producer Unit Testing

To address these concerns, we could write some integration tests to trigger interactions between running instances of the producer and consumer applications to implicitly test parser behaviour. This could encourage the use of representative test data and warn the consumer of producer API changes, but the increase in run time from milliseconds to minutes would result in significant feedback delays and a corresponding reduction in scenario coverage. Given JB Rainsberger’s oft-quoted assertion that “integrated tests are a scam… you write integrated tests because you can’t write perfect unit tests“, it seems prudent to explore how we might equip our unit testing strategy with representative test data and an awareness of API changes.

Consumer Producer Integration Testing

API Examples is an application pattern originally devised by Daniel Worthingon-Bodart, in which a new version of a producer application is accompanied by a sibling artifact that solely contains example API requests and example API responses. These example files should be raw textual data recorded from the acceptance tests of the producer application, meaning that all happy/sad path scenarios known to the producer become freely available for unit testing within the consumer commit build without any binary dependencies or feedback delays. This approach satisfies Brandon’s recommendation that “each service publish a cohesive set of golden test data that it guarantees to be stable“, and when combined with a regular update policy ensures new versions of the consumer application will have  early warning of API changes.

Consumer Producer API Examples

As API Examples are exercised within the consumer commit build, they can warn a new consumer version of an API change but cannot warn an existing consumer version already in production. The solution to this problem is for the consumer to derive its parser behaviour from the API Examples and publish it as a Consumer Driven Contract - a testable specification embedded within the producer commit build to document how the consumer uses the API and to immediately warn the producer if an API change will harm a consumer.

Consumer Producer Examples and Contracts

« Older entries