Why You Should Canary Test In Production

Posted on 2019-06-15

Testing, test strategies and tooling change often. Eight years ago we were all drinking the BDD coolaid, dreaming about our Product Managers writing our specs in Cucumber while we just happily write the code that makes them pass.

It took the testing world by a storm. But then it stopped.

The pendulum has swung, and today we’re seeing more of the simplistic style of testing taking the stage. Go’s testing package is one notable example of mainstream, simple, testing approaches. Some people even wanted a simpler test output.

Jest for example, is a mix. It takes a simplistic BDD-style testing approach; it takes BDD-the-language but not the bells and whistles of the frameworks, which leaves the extreme idealistic approaches out. It even has a minimalistic set of matchers, as opposed to the massive amount of matcher libraries that we used to see come out every now and then.

So, testing, test strategies, and frameworks all change as years go by.

Even their backing technologies change.

Contracts are King

Contracts don’t want to change often. SLAs ( Service Level Agreements) and nonfunctional requirements might never change, because you’ve got a lawyer to sign an actual courtroom-backed contract specifying your uptime, your support policy, and your service quality.

The result is that we have to test our contracts as if we’re our own clients — that means black box testing.

We can expect SLA definitions such as:

  1. %99.9 uptime (or any other specification in percent).
  2. All requests come below one (1) second for main flows, eight (8) seconds for secondary flows.
  3. Clean security postures on all APIs (i.e. don’t expose server headers).
  4. Early notification of breakage and maintenance, up to 72 hours ahead.
  5. And so on… (you get the point).

We can translate these requirements to tests:

  1. Periodical test for a main endpoints (see if up or down, such as the /login endpoint).
  2. Benchmark public endpoint requests after deploys and periodically.
  3. Verify security standards on public endpoints.
  4. Test things that usually break — response contracts, error codes, SSL/TLS certificates.

What tools do we have? well, we can use BDD with Cucumber to test APIs, like in this article:

Scenario Outline: Check if user is able to submit GET API request
   Given I want to execute service <serviceName>
    When I submit the GET request as per the data in Excel Worksheet <EWorksheet>
    And I validate status code
    And I validate mandatory tag in response from Excel Worksheet <EWorksheet>
    And I validate response content
    And I validate tag values in response from Excel Worksheet <EWorksheet>
    And I validate header parameter in response

This looks fancy. It’s declarative and it’s english. But if you read the rest of the article, before you know it, you’ll find out you have to fill in and implement an API testing framework yourself.

But even if you did, there’s arguments to be had about how maintainable this look like for an API testing solution. And, you have to carry cucumber around every time you want to do this, possibly in projects and teams that don’t care for, or don’t have cucumber support.

Fine, Let’s Program Everything

You might say, if we’re building everything from scratch, why not get rid of cucumber, and just code our own framework?

Here’s the idea using Jest:

it("makes a request", ()=>{

This would have been a reasonable approach. But there’s a ton of boilerplate in it — if I’m expecting the response body to be in a certain shape, isn’t that all I’m supposed to specify?

Contract Driven Testing

Maybe what we’re really looking for is contract driven testing? A bunch of testing frameworks and contract specification formats:

All of these have tools to validate their own contract specifications, to varying levels of success. Tools like Dredd, and Swagger CLI.

But what about actual content? nope. There’s partial support. Also because for example, in Swagger (and the OpenAPI spec) examples are kind of secondary; they’re an extension. For good reason — Open API is a contract specification format; a contract says nothing about actual API response content values, it specifies types and constraints.

Trust No One

What about testing your dependencies? Say testing Twitter, if your product depends on that? How about some of your enterprise service partners?

You might not be able to get the specs for their contract, and they might not even have one. But you certainly know how what response and SLAs you’re expecting from them.

How do we test that with contract driven testing tools? (spoiler: we don’t).

Running Fast And Breaking Things

Some times you run fast and break things. But some things cannot break.

We usually take a bottom up approach — unit tests, integration tests, functional tests, and sanity tests.

But when running fast, we deploy a lot. The match between:

  1. Deploy workflow, orchestration, and tools (relatively static).
  2. Your software (changes a lot).

Is being exercised many times a day. When either change, and especially the deploy workflow changes, we don’t really have a mainstream way to test, or actually, meta-test all that. Does this even have a name?

That’s a top-to-bottom approach — black box testing. And for matching your workflow and software — that’s called Canary tests; and we need a tool to write and execute these quickly and easily.

Introducing: Canary

Canary is the Swiss army-knife of canary testing. It lets you test:

  • Contracts
  • Content, body, headers, status
  • Certificates (SSL/TLS)
  • Benchmarks and performance
  • Uptime and SLAs

All in a generic, technology-independent way. Canary is written in Rust a modern systems programming language that is: fast, secure, and has an extremely low footprint.

Here’s your first Canary test:

- request:
    uri: https://google.com
    status_code: "200"

And here’s what happens when we execute it:

$ canary run
✔ verify: request - (https://google.com)
Ran 1 interactions in 246ms. Success: 1 Failure: 0 Skipped: 0

You can try this right now with nothing but Docker on your machine:

$ docker run -v "$(pwd)":/canary jondot/canary init
$ docker run -v "$(pwd)":/canary jondot/canary run

Quick. Simple. Fast.

A Bag of Opportunities

Canary is built to accommodate many use cases at once:

Your own project and repository flow.

Simply check in your .canary.yaml file, and include Canary in your CI. Here it is on Travis CI:

# ...
  - yarn test
  - yarn deploy:staging
  - curl http://get.canaryops.io/install | sh
  - canary run && yarn deploy:production

Canary will fail the build if needed.

As a centralized SLA validation tool.

If you have an engineering effectiveness or SRE team in your organization, they will see Canary as a great tool for validating healthy production standards.

A central repo with benchmarks, validations, and cert checks is something Canary can handle well.

Here’s how to generate various types of interactions:

$ canary init [with-vars|certs|benchmark|default]

As a homebrew global testing service.

Wrap Canary in a Lambda and have it execute from any point on the globe. Canary is built with Rust, and so startup time is extremely low, and its footprint is as compact as it gets.

You can also use Canary’s various reporters to ship back JSON results to your aggregated logs service, for analytics, aggregation, and forensics. You’re in control.

Take Canary for a Spin

Canary is optimized for a quick and lightweight experience. There’s nothing to learn but using your engineering instincts.

Check out the documentation and the features, as well as sign up to get a full license for free

For everything else, check out Canary’s official website: https://canaryops.io