What to Test

Alexander Avery

Thu | Nov 30, 2023

computer-science

What to test (with examples in Go)

When first exposed to automated testing, many people wonder what code they should test. The following article focuses on what to test and is not a guide to using the Go testing package. The code comes from real solutions to a kata on codewars.com.

The Kata

Let’s imagine we are writing code for our startup that provides software for making computations with odd numbers. Our first task is to write a function that finds which integer within a slice appears an odd number of times. Here is our solution:

package odd

func FindOdd(seq []int) int {
	// for each value in the slice...
	for _, v := range seq {
		// ...count how many times it appears
		if countInt(v, seq)%2 == 1 {
			return v
		}
	}
	panic("no candidate integers found!")
}

// countInt returns the number of times a value appears in a slice
func countInt(val int, values []int) int {
	var count int
	for _, v := range values {
		if val == v {
			count++
		}
	}
	return count
}

Our first test

We have heard that testing is something you should do, so we’ll write some tests. Without a strategy for evaluating what we should test, we start writing tests for our package in order. The first function that appears in odd.go is FindOdd.

package odd

import "testing"

type testCase struct {
	arr  []int
	want int
}

func TestFindOdd(t *testing.T) {
	tests := []testCase{
		{arr: []int{20, 1, -1, 2, -2, 3, 3, 5, 5, 1, 2, 4, 20, 4, -1, -2, 5}, want: 5},
		{arr: []int{1, 1, 2, -2, 5, 2, 4, 4, -1, -2, 5}, want: -1},
		{arr: []int{20, 1, 1, 2, 2, 3, 3, 5, 5, 4, 20, 4, 5}, want: 5},
		{arr: []int{10}, want: 10},
		{arr: []int{1, 1, 1, 1, 1, 1, 10, 1, 1, 1, 1}, want: 10},
		{arr: []int{5, 4, 3, 2, 1, 5, 4, 3, 2, 10, 10}, want: 1},
	}

	for _, tt := range tests {
		if ans := FindOdd(tt.arr); ans != tt.want {
			t.Fatalf("FindOdd(%v) = %d; wanted %d", tt.arr, ans, tt.want)
		}
	}
}

alexander@calcifer:~/testing$ go test .
ok      odd     0.003s

What have we gained?

To better understand the value of our test, let’s enumerate some ways it benefits us:

1. The test allows users of the odd package to verify the expected behavior of FindOdd.
2. The test allows users of the odd package to see an example of how to invoke FindOdd.
3. The test allows developers of the odd package to change the implementation of FindOdd without modifying its behavior.

There are possibly other benefits we have achieved, but consider which of the above three we would gain if we added a test for countInt. Because countInt is not exported, testing it won’t provide us with benefits one or two. In fact, it even loses us benefit three entirely because the tests now depend on the existence of countInt.

We can improve our standing by simply exporting countInt as CountInt, but we should consider the purpose of the odd package. Dave Cheney wrote an excellent post encouraging gophers to use the name of a package to describe what the package does. The name odd describes a package one would expect to export functions related to odd numbers, therefore, CountInt wouldn’t apply. It is merely an implementation detail we use to provide the functionality of FindOdd.

Implementation details and APIs

Anything unexported from a package is typically referred to as an “implementation detail”. These implementation details are important only insofar as they assist you in supporting your package’s API. Refraining from testing implementation details lets you refactor the way your package operates with minimal testing friction.

By labeling countInt as an implementation detail, we are free to refactor the package without breaking our tests. Our tests written earlier will continue to pass, and the expectations of our users will remain met.

package odd

func FindOdd(seq []int) int {
    res := 0
    for _, x := range seq {
        res ^= x
    }
    return res
}

Whether you are developing code for clients, coworkers, or yourself, deciding what API you are providing is a foundational programming skill. If you test all your code, you’re likely writing fragile tests that need continuous updates as your codebase evolves. If you test only the surface of your API, you will only need to update your tests when the surface of your API evolves.

Tests are beneficial because they keep you constrained. When utilized properly, these constraints allow you to provide a stable API for your users and avoid reintroducing old bugs. The trick is to not let these constraints prevent you from shipping your product or making improvements. By writing tests for implementation details, you constrain yourself more than is necessary to provide a working API. You can easily see how such a practice is not the best use of your time.

Clear direction

Keeping the above example in mind, the following are questions I ask myself about code I am considering testing. I will repeatedly use the term function, but this can apply to any function, method, or type. An answer “yes” to any of the following questions is a reason to more strongly consider testing:

• Is the function exported from the package?
• Is the function a commonly used feature of the package?
• Is the function difficult to implement correctly?
• Is the function’s implementation regularly changing?
• Is the function’s proper usage difficult to understand without an example?

Next: Do Not Fear Systems Programming
Previous: Work Updates
>> Home