Snapshot Testing in TestBox

Snapshot testing is the latest craze. Popularized by Facebook’s Jest testing framework, snapshot testing is a simple way to add regression testing into your application.

What is regression testing, you ask? Regression testing is testing that the output did not change from what it previously returned.

Take the following example:

In this handler we are retrieving a list of users, optionally filtered by the request context. We can use ColdBox’s integration testing capabilities to test that this code is coming back correctly.

This is a great test! It will make sure that this users endpoint returns the values we expect.

Fast-forward a month or two (or maybe just a few weeks) and there is a new property on the User bean. With our current approach, our users endpoint test will fail, even though the logic hasn’t changed. We’ll need to go update our tests to match.

Not with snapshot testing. Let’s take a look at the same test using snapshot testing:

When running this test for the first time, the tests will fail mentioning that no snapshots exist.

Creating and updating snapshots is a deliberate action with snapshot testing. You have to pass a updateSnapshots url flag to create or update the snapshots for the tests ran.

Matchers that have their snapshots updated always return true. In addition, a debug message is logged with the name of the snapshot created.

You can see the snapshots created in your tests/resources folder:

Now, running the tests without the updateSnapshots url flag returns green.

The real power of snapshot tests come when you make changes. Let’s add an email field to the User bean and return it. When we run our tests now, we see the following failure:

This failure gives us the opportunity to check what changed in our system and if we were did break some existing functionality. If we did, our tests helped us catch it! If not, we can now update these snapshots by running the tests with the updateSnapshots url flag.

A really compelling use case for this is testing HTML. In fact, the snapshot intelligently normalizes HTML output so that simple indentation changes should cause the snapshot to fail.

To get started with snapshot testing in TestBox, install the testbox-snapshots module from ForgeBox. Then add the matcher in your beforeAll method and you’re good to go!

For more precise changes, make sure to include the optional java library in your load paths. It produces a diff of the files instead of just outputting the contents in their entirety. You can enable this by including the following in your tests/Application.cfc:

this.javaSettings = {
    loadPaths = [ "testbox-snapshots/lib" ],
    reloadOnChange = false
};

And that’s it! Snapshot testing is another tool in your testing tool belt. You can get started with box install testbox-snapshots --saveDev in your project right now.

One great use case that I’ve seen is using snapshot testing to start to get a hold on a legacy app. Add a snapshot assertion, create the snapshot, then start refactoring with confidence!

Snapshot Testing in TestBox

When does `BeforeAll` run in TestBox?

Just a quick gotcha I found tonight.

I was trying to get Selenium running in a project using Ortus Solution’s fork of CFSelenium. Here’s the test for the login page:

I thought it would be neat and clean to write the setup for the tests after the given and when sections. Only one problem.

Cannot find Selenium in variables

This was confusing to me. Dumping the variables scope in the beforeAll method showed Selenium, but not in my when block.

Turns out this is due to how TestBox packages specs and when it runs its lifecycle methods like beforeAll. The new spec keywords — feature, story, scenario, given, and when — are just aliases to describe. describe blocks are executed before the test lifecycle in order to find all the actual specs — the it and then blocks. That’s why Selenium was not available there.

Moving all the login to the then block fixed this up nice and quick:

An opportunity might exist here for a Pull Request to instead run the beforeAll before any describe blocks, but I’m not sure if it matters all that much. For now, I’m content to just be aware of the fact and start testing with Selenium.

When does `BeforeAll` run in TestBox?