1# Datawire build-aux Test Harness
2
3`common.mk` includes a built-in test harness that allows you to plug
4in your own tests, and have them aggregated and summarized, like:
5
6 $ make check
7 ...
8 PASS: go-test 5 - github.com/datawire/apro/cmd/amb-sidecar.TestAppNoToken
9 PASS: go-test 6 - github.com/datawire/apro/cmd/amb-sidecar.TestAppBadToken
10 PASS: go-test 7 - github.com/datawire/apro/cmd/amb-sidecar.TestAppBadCookie
11 PASS: go-test 8 - github.com/datawire/apro/cmd/amb-sidecar.TestAppCallback
12 PASS: go-test 9 - github.com/datawire/apro/cmd/amb-sidecar.TestAppCallbackNoCode
13 PASS: tests/local/apictl.tap.gen 1 - check_version
14 ============================================================================
15 test-suite summary
16 ============================================================================
17 # TOTAL: 10
18 # SKIP: 0
19 # PASS: 10
20 # XFAIL: 0
21 # FAIL: 0
22 # XPASS: 0
23 # ERROR: 0
24 ============================================================================
25 make[1]: Leaving directory '/home/lukeshu/src/apro'
26
27(If you were viewing that in a terminal, it would also be pretty and
28colorized.)
29
30Each test-case can have one of 6 results:
31
32 - `pass`, `fail`, `skip`: You can guess what these mean.
33 - `xfail` ("expected fail"): The test failed, but it was expected to
34 fail. Perhaps you wrote a test for a feature for before
35 implementing the feature. Perhaps you wrote a test that captures a
36 bug report, but haven't written the fix yet. Makes
37 Test-Driven-Development possible.
38 - `xpass` ("unexpected pass"): The test passed, but it was expected
39 to xfail. This either means you did a poor job implementing the
40 test, and it doesn't really check what you think it checks, or it
41 means you've implemented the fix, but forgot to replace `xfail`
42 with `fail` in the test.
43 - `error`: It isn't that the test decided that there's bug in the
44 code-under-test, it's that the test itself encountered an error.
45
46You can use any test framework or language you like, as long as it can
47emit [TAP, the Test Anything Protocol][TAP] (or something that can be
48translated to TAP). Both TAP version 12 and version 13 are supported.
49`not ok # TODO` is "xfail", while , `ok # TODO` is "xpass"
50
51 > Side-Note: pytest-tap emits `ok # TODO` for both xfail and xpass,
52 > which is wrong, and I consider to be a bug in pytest-tap.
53
54## Built-in test runners
55
56By default, `common.mk` knows how to run test cases of 2 types (but
57you can easily add more):
58
59 - `*.test` GNU Automake-compatible standalone test cases. One file
60 is one test case. `FOO.test` must be an executable file. It is
61 run with no arguments, stdout and stderr are ignored (but logged to
62 `FOO.log`); it is the exit code that determines the test result:
63
64 * 0 => pass
65 * 77 => skip
66 * anything else => fail
67
68 It is not possible to xfail or xpass with this type of test.
69
70 - `*.tap.gen` TAP-emitting test cases. You may have multiple test
71 cases per file. `FOO.tap.gen` must be an executable file. It is
72 run with no arguments; stdout and stderr are merged, and are taken
73 to be a TAP stream (both v12 and v13 are supported). The exit code
74 is ignored.
75
76`common.mk` does *not* scan your source directory for tests (but
77`go-*.mk` will scan for `go test` tests). In your `Makefile` You must
78explicitly tell it about any `.test` or `.tap.gen` files that you
79would like it to include in `make check`. For example, if you would
80like it to include any `.test` or `.tap.gen` files in the `./tests/`
81directory, you could write:
82
83 test-suite.tap: $(patsubst %.test,%.tap,$(wildcard tests/*.test))
84 test-suite.tap: $(patsubst %.tap.gen,%.tap,$(wildcard tests/*.tap.gen))
85
86## Adding your own test runners
87
88To add a new test runner, you just need a command that emits TAP:
89`tee` it to a `.tap` file, and pipe that to `$(tools/tap-driver) stream -n
90TEST_GROUP_NAME` to pretty-print the results as they happen:
91
92 # Tell Make how to run the test command, and stream the results to
93 # `$(tools/tap-driver) stream` to pretty-print the results as they happen.
94 my-test.tap: my-test.input $(tools/tap-driver) FORCE
95 @SOME_COMMAND_THAT_EMITS_TAP 2>&1 | tee $@ | $(tools/tap-driver) stream -n my-test
96
97 # Tell Make to include 'my-test' in `make check`
98 test-suite.tap: my-test.tap
99
100For example, to use [BATS (Bash Automated Testing System)][BATS], you
101would write:
102
103 %.tap: %.bats $(tools/tap-driver) FORCE
104 @bats --tap $< | tee $@ | $(tools/tap-driver) stream -n $<
105
106 # Automatically include `./tests/*.bats`
107 test-suite.tap: $(patsubst %.bats,%.tap,$(wildcard tests/*.bats))
108
109If your test framework of choice doesn't support TAP output, you can
110pipe it to a helper program that can translate it. For example, `go
111test` doesn't support TAP output, but `go test -json` output is
112parsable, so we pipe that to [gotest2tap][gotest2tap], which
113translates it to TAP.
114
115If you set `SHELL = sh -o pipefail` in your `Makefile` (the pros and
116cons of which I won't comment on here), you should be sure that if
117your test-runner indicates success or failure with an exit code, that
118you ignore that exit code:
119
120 %.tap: %.bats $(tools/tap-driver) FORCE
121 @{ bats --tap $< || true; } | tee $@ | $(tools/tap-driver) stream -n $<
122
123## Adding dependencies of tests
124
125It is assumed that *all* tests depend on `make build`. To add a
126dependency shared by all tests, to declare a dependency that all tests
127should depend on, declare it as a dependency of `check` itself. For
128example, `common.mk` says:
129
130 check: lint build
131
132As another example, the `Makefile` for Ambassador Pro says:
133
134 check: $(if $(HAVE_DOCKER),deploy proxy)
135
136To declare a dependency for an individual test is a little trickier,
137because you must keep in mind what type of test it is. For `.tap.gen`
138tests, you must declare it both on the `.tap` file and (if the
139dependency is not a `.tap`) on `check` itself:
140
141 test-suite.tap: tests/cluster/oauth-e2e.tap
142 check tests/cluster/oauth-e2e.tap: tests/cluster/oauth-e2e/node_modules
143
144If that were a `.test` test, you would need to declare it on the
145`.log` file instead of `.tap`:
146
147 test-suite.tap: tests/cluster/oauth-e2e.tap
148 check tests/cluster/oauth-e2e.log: tests/cluster/oauth-e2e/node_modules
149
150If you need one test to depend on another test, write the dependency
151using the `.tap` suffix (not `.log`). You do not need to write the
152depenency for `check`, since it will already depend on the `.tap`
153through `test-suite.tap`:
154
155 test-suite.tap: foo.tap bar.tap
156 foo.log: bar.tap
157
158[TAP]: https://testanything.org
159[BATS]: https://github.com/sstephenson/bats
160[gotest2tap]: ../bin-go/gotest2tap/
View as plain text