Not All Failures Are Equal

The noise problem of failed tests

When a test fails, you’re usually looking for what broke and where. For tests to be helpful we don’t want that mix of assertion to bury that information we need. We also don’t want to stop too early if one of many fields of the struct fails.

In Go, the testify library splits assertions into two packages: assert and require. They check the same conditions, but differ in what happens on failure. assert records the failure and continues. require records the failure and stops the test immediately by calling t.FailNow().

The distinction mirrors Google’s C++ testing framework where EXPECT_* (nonfatal) and ASSERT_* (fatal) serve the same roles. The design intent is identical: not all failures deserve the same response.

Why we can’t use only one

Let’s consider following example. We will test an imaginary database and a function that creates a user in it. We want to check that the user is created with the correct name, but also that the database was migrated successfully.

func TestUserCreation(t *testing.T) {
    // if service fails, nothing to check below
    service, err := startServer(t)
    require.NoError(t, err) 

    // if migration fails, we can't expect service to work
    err = db.Migrate()
    require.NoError(t, err) 

    // the creation can still write to the database, 
    // so we want to see all the failures here
    err = service.CreateUser("alice")
    assert.NoError(t, err) 

    user, err := service.GetUser("alice")
    assert.NoError(t, err)
    assert.Equal(t, "alice", user.Name)
    assert.NotZero(t, user.CreatedAt)
}

Now picture the two extremes.

We use all require: A failure on CreateUser stops the test immediately. You never learn whether GetUser also has a bug. You fix one thing, rerun, discover the next failure, fix, rerun, repeat. You get a very slow and frustrating feedback loop.

We use all assert: The migration fails, and then you see three or four subsequent failures, but they are fully irrelevant, you can’t expect anything to run if one of the required dependencies failed at setup time. So you want to see only that required condition failed to run.

How to use them together

The rule is simple: use require for preconditions and assert for behavioral checks. What counts as a precondition? A precondition is any step where failure makes the rest of the test meaningless. And behavioral checks are any assertions that validate the expected behavior of the code under test, but don’t necessarily invalidate the entire test if they fail.

This gives you the best of both approaches: noisy failures from broken preconditions are eliminated, and genuine behavioral failures are all surfaced in one run.

Note: require calls t.FailNow(), which internally calls runtime.Goexit(). This means require must be called from the goroutine running the test function, not from goroutines spawned during the test. If you need fatal assertions inside a spawned goroutine, you will need a different approach.

How they behave in subtests

The same rule can be applied to subtests. Each t.Run creates its own scope for the subtest, so require will exit only that case, where it was called and not the entire test. This allows you to have multiple subtests with their own preconditions and assertions, and failures in one subtest won’t affect the execution of others.

func TestStatusCodes(t *testing.T) {
    // precondition for all subtests
    service, err := startServer(t)
    require.NoError(t, err) 

    cases := []struct {
        path string
        want int
    }{
        {"/health", 200},
        {"/missing", 404},
    }

    for _, tc := range cases {
        t.Run(tc.path, func(t *testing.T) {
            resp, err := http.Get(service.URL + tc.path)
            require.NoError(t, err)  // can't check status without a response

            assert.Equal(t, tc.want, resp.StatusCode)
            assert.NotEmpty(t, resp.Header.Get("Content-Type"))
            // more assertions...
        })
    }
}

Conclusion

This combination of different assertion types controls what information you get when tests fail. The goal is always the same: see the real problem as soon as possible.

Sources

• testify library
• Google’s C++ testing framework