(Cross-posted from https://blog.freetrade.io/serverless-integration-testing-at-freetrade-5359fb0b0e57)
Our tech stack at Freetrade is 99% serverless. This means that for the vast
majority of our applications, the physical infrastructure, deployment and
other operational concerns are largely abstracted away by the cloud provider
(Google Cloud in our case).
We focus on building what our users need, and our cloud provider focuses on
what we need to keep it running smoothly at scale.
Serverless computing has huge benefits in terms of allowing more resources to
be devoted to solving user problems, but it also presents some different
challenges. One of those is how you go about applying automated testing to
your software systems when you’re in a serverless environment.
Automated testing is essential to produce and maintain high-quality software.
With traditional infrastructure where you’re responsible for the actual server
instances and processes that run your applications, it can be easier to
implement automated testing because you have direct control over more of the
stack. That extra responsibility is exactly what you’re trying to abstract
away with serverless computing, though, so you have to adapt your testing
Unit testing serverless applications
Unit testing is the bedrock of an automated testing strategy. Unit tests are the
most numerous and lowest-level tests, which isolate tiny parts of application
logic and test them on their own.
With a well-architected application, the infrastructure that will ultimately run
the application shouldn’t impact unit testing at all. Each individual unit is
oblivious to what server environment it might be running in. Serverless
computing facilitates this, as units of code cannot rely on particular
environment details being available, such as disk i/o or some particular
application state being persisted across requests. Good abstractions have to be
used for these instead.
In other words, serverless encourages you to write code that depends on
abstractions (see dependency inversion). It also encourages stateless
Both of these make it easier to write comprehensive unit tests, as you can
isolate those parts out.
As an example, requests from Freetrade’s apps to the platform are handled by
HTTP handlers in Google Cloud Functions. There is little to no logic in these
handlers; they merely pass the request on to services that don’t need to know
about how the request came in or even about HTTP at all.
Those services and their components are then easy to unit test because they’re
not tied to particular details of the environment.
Integration testing serverless applications in Google Cloud Functions
Integration testing could be described as the next level up from unit testing.
Unit testing mocks out interactions between components to isolate logic, but
what if you’ve got that interaction wrong?
Integration tests fill this gap in a testing strategy by confirming that
components also play nicely with each other.
While serverless does not impact unit testing, it can present a couple of
challenges for integration testing. You need to test the interactions between
HTTP requests, databases, queues and other systems, so you need those to be
available for the integration tests. With a traditional server infrastructure,
you might run instances of all those things locally on a single machine and run
the integration tests that way.
There are options to try and emulate serverless components locally for this
purpose, but that kind of defeats the point of the integration testing — what if
the emulation is wrong? You’re reliant on the emulation being accurate and up to
date with the serverless platform, and that’s the kind of work you want to avoid
It is possible to spin up a fresh environment in Google Cloud for each
integration test run, and that’s certainly an option. However, it takes a bit of
time for each setup, and you want the integration tests to be applied frequently
and rapidly, so this approach might not be ideal.
So far we’ve taken a middle-ground approach by having a dedicated CI (continuous
integration) environment in Google Cloud. It isn’t spun up and torn down between
test runs. Instead, our integration tests aim to achieve that robustness by
creating the specific state to be tested within each test, and avoiding
dependency on any other state (as that would be a bad practice).
As an imaginary example:
// Given I have a user account;
(code to create a fresh user account)
// When I make a valid sign-in request;
(code to send a real sign-in request over the Internet to the test environment)
// Then I should get a valid auth token;
(code to check received token works in the test environment)
// And a sign-in event should have been persisted.
(code to check a sign-in event was persisted)
We have various helper code to set up and assert on the expected state within a
test, and a test client to mimic user app requests against the system.
Individual tests or the entire test suite can be run as needed without needing
to refresh the whole environment.
This is a pragmatic approach that is working well for us so far. Tests can be
run continuously, and each test run completes in reasonable time with the
individual tests executed in parallel.
The serverless cloud functions are spun up and down as in production use, and
interact realistically with other components.
Because serverless functions are spun up to meet demand, they can have cold
starts. As the name suggests, this is when a function that had no “hot”
instances running is spun up.
Inevitably this will happen during integration testing, which can add several
seconds to the time taken by the test. However, we actually want this to occur,
as it mimics behaviour in production. We want our integration tests to help us
identify potential issues due to cold function starts just as we would with any
other integration issue.
End-to-end testing serverless applications
The top tier of a testing strategy is end-to-end (E2E) testing. This applies
user-level interactions via the user interface, and also checks that the
application behaviour is correct via the user interface. These tests are the
most time-consuming to run, but also provide the strongest confirmation that the
entire stack works as expected for our users.
As with unit tests, the fact that serverless infrastructure is being used should
not impact end-to-end testing. They only care about what they can interact with
in the user interface, with the platform behind that being a black box.
That said, serverless computing does offer some benefits to end-to-end testing.
You don’t need to maintain dedicated infrastructure to handle E2E tests: the
infrastructure spins up automatically to meet the demand. This can reduce costs
as you don’t pay for idle infrastructure that might sit waiting for a test run.
Getting more specific, Google Cloud also makes it easier to manage E2E testing
by grouping components within specific environments, with consistent URLs based
Managing test suites and their environments is then straightforward.
To sum up
We’ve had a great experience applying different tiers of automated testing to
our serverless infrastructure.
Having strong abstractions provided by the cloud platform lets us devote more of
our time to building things for our users, and that carries through to our
approach to testing.
This allows a lot of our test code to be highly readable and meaningful from a
user perspective, keeping us on track to bring free investing to more and more
(Cross-post) Serverless integration testing at Freetrade