testing.md

Testing

So programs are tested with the so test command and the so/testing package. The model mirrors Go's testing, but stays deliberately small: no subtests, no parallelism, no formatted message helpers.

Layout

Tests for a package live in a test subdirectory of that package:

so/bytes/
    bytes.go
    ...
    test/
        bytes.go     # your tests
        main.go      # generated by `so test`

Test files are plain .go files (no _test.go suffix) in package main. Two consequences follow from this convention:

• go test ignores them (they carry no _test.go suffix), so you can still keep ordinary Go-level tests elsewhere and run them with go test.
• Because the directory is package main, it needs a func main. That function is provided by the generated main.go (see How it works), which is committed alongside the tests and keeps the directory buildable by the Go toolchain.

Since test is a separate package, tests see only the exported API of the package under test (black-box testing).

Writing tests

A test is a function named TestXxx taking a single *testing.T:

package main

import (
	"solod.dev/so/bytes"
	"solod.dev/so/testing"
)

func TestEqual(t *testing.T) {
	if !bytes.Equal([]byte("abc"), []byte("abc")) {
		t.Error("Equal(abc, abc) = false, want true")
	}
}

The T type records failure and skip state for one test:

Method	Description
Name() string	Name of the running test.
Fail()	Mark the test failed, keep running.
Failed() bool	Whether the test has failed.
Log(msg string)	Record a log line.
Error(msg string)	Log + Fail.
Fatal(msg string)	Log + Fail. The test must return afterwards (see below).
Skip(msg string)	Mark the test skipped. The test must return afterwards.

Running

so test ./so/sync

=== RUN   TestCond
--- PASS: TestCond
=== RUN   TestMutex_LockUnlock
--- PASS: TestMutex_LockUnlock
=== RUN   TestMutex_TryLock
--- PASS: TestMutex_TryLock
=== RUN   TestOnce
--- PASS: TestOnce
ok	so/sync	4 tests

so test exits non-zero if any test fails. The === RUN line is printed before each test, so if a test hard-crashes (a panic or a segfault, which cannot be recovered), the output still identifies the culprit.

How it works

so test:

1. Scans the test directory for TestXxx(t *testing.T) functions.
2. Generates test/main.go, a runner that dispatches them via testing.RunTests. The file is deterministic (tests sorted by name, no timestamps) and carries a // Code generated by "so test". DO NOT EDIT. header, so re-running produces no spurious diffs. It is meant to be committed.
3. Compiles and runs the test package with the equivalent of so run.

Benchmarks

Benchmarks work just like tests, but live in a bench subdirectory and are run with so bench:

so/bytes/
    bytes.go
    ...
    bench/
        bytes.go     # your benchmarks
        main.go      # generated by `so bench`

A benchmark is a function named BenchmarkXxx taking a single *testing.B. The measured code goes in a for b.Loop() loop; setup before the loop and cleanup after it are not timed:

package main

import (
	"solod.dev/so/bytes"
	"solod.dev/so/testing"
)

func BenchmarkEqual(b *testing.B) {
	x := []byte("hello world")
	for b.Loop() {
		bytes.Equal(x, x)
	}
}

Run them with:

so bench ./so/bytes

BenchmarkEqual  482547852           2.215 ns/op         0 B/op         0 allocs/op

To measure allocations, allocate through b.Allocator(); the benchmark tracks memory routed through it and reports B/op and allocs/op.

so bench mirrors so test: it scans the bench directory for BenchmarkXxx(b *testing.B) functions, generates a deterministic, committed bench/main.go runner (// Code generated by "so bench". DO NOT EDIT.), then compiles and runs it.

The generated runner always uses the system allocator (mem.System). If a benchmark needs a different allocator, maintain bench/main.go yourself (it calls testing.RunBenchmarks) and run it with so run instead of so bench.

Comparing against Go

so bench (and so test) ignore _test.go files. This lets you drop native Go benchmarks of the same code into the bench directory and compare the two side by side: so bench ./so/bufio runs the So versions, while go test -bench=. ./so/bufio/bench runs the Go ones. Give the Go functions distinct names (e.g. a _Go suffix) so both sets can share the package main directory without colliding.

Caveats

No Errorf / formatted helpers

So cannot forward a ...any argument to fmt, so there is no Errorf, Fatalf, or Logf. Format the message yourself with fmt.Sprintf:

func TestIndex(t *testing.T) {
	got := bytes.Index([]byte("hello world"), []byte("world"))
	if got != 6 {
		buf := fmt.NewBuffer(64)
		t.Error(fmt.Sprintf(buf, "Index = %d, want 6", got))
	}
}

Fatal and Skip need an explicit return

So has no recover, so a test cannot be unwound from the outside. Fatal and Skip only set state and print the message; they do not stop the function. Always return right after:

if err != nil {
	t.Fatal("open failed")
	return
}

Use Fatal when continuing makes no sense (a precondition failed), and Error when you want to report several problems from one test.

A hard crash aborts the whole run

Tests run in a single process, and So has no recover. So a hard crash — a panic or a segfault — in one test aborts the entire run, not just that test: the tests after it do not execute. Reported failures (Error, Fatal) are unaffected; only an actual crash stops the run. The === RUN line printed before each test tells you which one crashed.