1 // Copyright 2018 The Go Authors. All rights reserved. 2 // Use of this source code is governed by a BSD-style 3 // license that can be found in the LICENSE file. 4 5 /* 6 Package testscript provides support for defining filesystem-based tests by 7 creating scripts in a directory. 8 9 To invoke the tests, call testscript.Run. For example: 10 11 func TestFoo(t *testing.T) { 12 testscript.Run(t, testscript.Params{ 13 Dir: "testdata", 14 }) 15 } 16 17 A testscript directory holds test scripts with extension txtar or txt run during 'go test'. 18 Each script defines a subtest; the exact set of allowable commands in a 19 script are defined by the parameters passed to the Run function. 20 To run a specific script foo.txtar or foo.txt, run 21 22 go test cmd/go -run=TestName/^foo$ 23 24 where TestName is the name of the test that Run is called from. 25 26 To define an executable command (or several) that can be run as part of the script, 27 call RunMain with the functions that implement the command's functionality. 28 The command functions will be called in a separate process, so are 29 free to mutate global variables without polluting the top level test binary. 30 31 func TestMain(m *testing.M) { 32 os.Exit(testscript.RunMain(m, map[string] func() int{ 33 "testscript": testscriptMain, 34 })) 35 } 36 37 In general script files should have short names: a few words, not whole sentences. 38 The first word should be the general category of behavior being tested, 39 often the name of a subcommand to be tested or a concept (vendor, pattern). 40 41 Each script is a text archive (go doc golang.org/x/tools/txtar). 42 The script begins with an actual command script to run 43 followed by the content of zero or more supporting files to 44 create in the script's temporary file system before it starts executing. 45 46 As an example: 47 48 # hello world 49 exec cat hello.text 50 stdout 'hello world\n' 51 ! stderr . 52 53 -- hello.text -- 54 hello world 55 56 Each script runs in a fresh temporary work directory tree, available to scripts as $WORK. 57 Scripts also have access to these other environment variables: 58 59 PATH=<actual PATH> 60 HOME=/no-home 61 TMPDIR=$WORK/.tmp 62 devnull=<value of os.DevNull> 63 /=<value of os.PathSeparator> 64 :=<value of os.PathListSeparator> 65 $=$ 66 67 The environment variable $exe (lowercase) is an empty string on most 68 systems, ".exe" on Windows. 69 70 The script's supporting files are unpacked relative to $WORK 71 and then the script begins execution in that 72 directory as well. Thus the example above runs in $WORK 73 with $WORK/hello.txtar containing the listed contents. 74 75 The lines at the top of the script are a sequence of commands to be 76 executed by a small script engine in the testscript package (not the system 77 shell). The script stops and the overall test fails if any particular 78 command fails. 79 80 Each line is parsed into a sequence of space-separated command words, 81 with environment variable expansion and # marking an end-of-line comment. 82 Adding single quotes around text keeps spaces in that text from being 83 treated as word separators and also disables environment variable 84 expansion. Inside a single-quoted block of text, a repeated single 85 quote indicates a literal single quote, as in: 86 87 'Don''t communicate by sharing memory.' 88 89 A line beginning with # is a comment and conventionally explains what is 90 being done or tested at the start of a new phase in the script. 91 92 A special form of environment variable syntax can be used to quote 93 regexp metacharacters inside environment variables. The "@R" suffix 94 is special, and indicates that the variable should be quoted. 95 96 ${VAR@R} 97 98 The command prefix ! indicates that the command on the rest of the line 99 (typically go or a matching predicate) must fail, not succeed. Only certain 100 commands support this prefix. They are indicated below by [!] in the synopsis. 101 102 The command prefix [cond] indicates that the command on the rest of the line 103 should only run when the condition is satisfied. The predefined conditions are: 104 105 - [short] for testing.Short() 106 - [net] for whether the external network can be used 107 - [link] for whether the OS has hard link support 108 - [symlink] for whether the OS has symbolic link support 109 - [exec:prog] for whether prog is available for execution (found by exec.LookPath) 110 - [gc] for whether Go was built with gc 111 - [gccgo] for whether Go was built with gccgo 112 - [go1.x] for whether the Go version is 1.x or later 113 - [unix] for whether the OS is Unix-like (that is, would match the 'unix' build 114 constraint) 115 116 Any known values of GOOS and GOARCH can also be used as conditions. They will be 117 satisfied if the target OS or architecture match the specified value. For example, 118 the condition [darwin] is true if GOOS=darwin, and [amd64] is true if GOARCH=amd64. 119 120 A condition can be negated: [!short] means to run the rest of the line 121 when testing.Short() is false. 122 123 Additional conditions can be added by passing a function to Params.Condition. 124 125 The predefined commands are: 126 127 - cd dir 128 Change to the given directory for future commands. 129 130 - chmod perm path... 131 Change the permissions of the files or directories named by the path arguments 132 to the given octal mode (000 to 777). 133 134 - [!] cmp file1 file2 135 Check that the named files have (or do not have) the same content. 136 By convention, file1 is the actual data and file2 the expected data. 137 File1 can be "stdout" or "stderr" to use the standard output or standard error 138 from the most recent exec or wait command. 139 (If the files have differing content and the command is not negated, 140 the failure prints a diff.) 141 142 - [!] cmpenv file1 file2 143 Like cmp, but environment variables in file2 are substituted before the 144 comparison. For example, $GOOS is replaced by the target GOOS. 145 146 - cp src... dst 147 Copy the listed files to the target file or existing directory. 148 src can include "stdout" or "stderr" to use the standard output or standard error 149 from the most recent exec or go command. 150 151 - env [key=value...] 152 With no arguments, print the environment (useful for debugging). 153 Otherwise add the listed key=value pairs to the environment. 154 155 - [!] exec program [args...] [&] 156 Run the given executable program with the arguments. 157 It must (or must not) succeed. 158 Note that 'exec' does not terminate the script (unlike in Unix shells). 159 160 If the last token is '&', the program executes in the background. The standard 161 output and standard error of the previous command is cleared, but the output 162 of the background process is buffered — and checking of its exit status is 163 delayed — until the next call to 'wait', 'skip', or 'stop' or the end of the 164 test. At the end of the test, any remaining background processes are 165 terminated using os.Interrupt (if supported) or os.Kill. 166 167 If the last token is '&word&` (where "word" is alphanumeric), the 168 command runs in the background but has a name, and can be waited 169 for specifically by passing the word to 'wait'. 170 171 Standard input can be provided using the stdin command; this will be 172 cleared after exec has been called. 173 174 - [!] exists [-readonly] file... 175 Each of the listed files or directories must (or must not) exist. 176 If -readonly is given, the files or directories must be unwritable. 177 178 - [!] grep [-count=N] pattern file 179 The file's content must (or must not) match the regular expression pattern. 180 For positive matches, -count=N specifies an exact number of matches to require. 181 182 - mkdir path... 183 Create the listed directories, if they do not already exists. 184 185 - mv path1 path2 186 Rename path1 to path2. OS-specific restrictions may apply when path1 and path2 187 are in different directories. 188 189 - rm file... 190 Remove the listed files or directories. 191 192 - skip [message] 193 Mark the test skipped, including the message if given. 194 195 - [!] stderr [-count=N] pattern 196 Apply the grep command (see above) to the standard error 197 from the most recent exec or wait command. 198 199 - stdin file 200 Set the standard input for the next exec command to the contents of the given file. 201 File can be "stdout" or "stderr" to use the standard output or standard error 202 from the most recent exec or wait command. 203 204 - [!] stdout [-count=N] pattern 205 Apply the grep command (see above) to the standard output 206 from the most recent exec or wait command. 207 208 - ttyin [-stdin] file 209 Attach the next exec command to a controlling pseudo-terminal, and use the 210 contents of the given file as the raw terminal input. If -stdin is specified, 211 also attach the terminal to standard input. 212 Note that this does not attach the terminal to standard output/error. 213 214 - [!] ttyout [-count=N] pattern 215 Apply the grep command (see above) to the raw controlling terminal output 216 from the most recent exec command. 217 218 - stop [message] 219 Stop the test early (marking it as passing), including the message if given. 220 221 - symlink file -> target 222 Create file as a symlink to target. The -> (like in ls -l output) is required. 223 224 - unquote file... 225 Rewrite each file by replacing any leading ">" characters from 226 each line. This enables a file to contain substrings that look like 227 txtar file markers. 228 See also https://godoc.org/github.com/rogpeppe/go-internal/txtar#Unquote 229 230 - wait [command] 231 Wait for all 'exec' and 'go' commands started in the background (with the '&' 232 token) to exit, and display success or failure status for them. 233 After a call to wait, the 'stderr' and 'stdout' commands will apply to the 234 concatenation of the corresponding streams of the background commands, 235 in the order in which those commands were started. 236 237 If an argument is specified, it waits for just that command. 238 239 When TestScript runs a script and the script fails, by default TestScript shows 240 the execution of the most recent phase of the script (since the last # comment) 241 and only shows the # comments for earlier phases. For example, here is a 242 multi-phase script with a bug in it (TODO: make this example less go-command 243 specific): 244 245 # GOPATH with p1 in d2, p2 in d2 246 env GOPATH=$WORK/d1${:}$WORK/d2 247 248 # build & install p1 249 env 250 go install -i p1 251 ! stale p1 252 ! stale p2 253 254 # modify p2 - p1 should appear stale 255 cp $WORK/p2x.go $WORK/d2/src/p2/p2.go 256 stale p1 p2 257 258 # build & install p1 again 259 go install -i p11 260 ! stale p1 261 ! stale p2 262 263 -- $WORK/d1/src/p1/p1.go -- 264 package p1 265 import "p2" 266 func F() { p2.F() } 267 -- $WORK/d2/src/p2/p2.go -- 268 package p2 269 func F() {} 270 -- $WORK/p2x.go -- 271 package p2 272 func F() {} 273 func G() {} 274 275 The bug is that the final phase installs p11 instead of p1. The test failure looks like: 276 277 $ go test -run=Script 278 --- FAIL: TestScript (3.75s) 279 --- FAIL: TestScript/install_rebuild_gopath (0.16s) 280 script_test.go:223: 281 # GOPATH with p1 in d2, p2 in d2 (0.000s) 282 # build & install p1 (0.087s) 283 # modify p2 - p1 should appear stale (0.029s) 284 # build & install p1 again (0.022s) 285 > go install -i p11 286 [stderr] 287 can't load package: package p11: cannot find package "p11" in any of: 288 /Users/rsc/go/src/p11 (from $GOROOT) 289 $WORK/d1/src/p11 (from $GOPATH) 290 $WORK/d2/src/p11 291 [exit status 1] 292 FAIL: unexpected go command failure 293 294 script_test.go:73: failed at testdata/script/install_rebuild_gopath.txt:15 in $WORK/gopath/src 295 296 FAIL 297 exit status 1 298 FAIL cmd/go 4.875s 299 $ 300 301 Note that the commands in earlier phases have been hidden, so that the relevant 302 commands are more easily found, and the elapsed time for a completed phase 303 is shown next to the phase heading. To see the entire execution, use "go test -v", 304 which also adds an initial environment dump to the beginning of the log. 305 306 Note also that in reported output, the actual name of the per-script temporary directory 307 has been consistently replaced with the literal string $WORK. 308 309 If Params.TestWork is true, it causes each test to log the name of its $WORK directory and other 310 environment variable settings and also to leave that directory behind when it exits, 311 for manual debugging of failing tests: 312 313 $ go test -run=Script -testwork 314 --- FAIL: TestScript (3.75s) 315 --- FAIL: TestScript/install_rebuild_gopath (0.16s) 316 script_test.go:223: 317 WORK=/tmp/cmd-go-test-745953508/script-install_rebuild_gopath 318 GOARCH= 319 GOCACHE=/Users/rsc/Library/Caches/go-build 320 GOOS= 321 GOPATH=$WORK/gopath 322 GOROOT=/Users/rsc/go 323 HOME=/no-home 324 TMPDIR=$WORK/tmp 325 exe= 326 327 # GOPATH with p1 in d2, p2 in d2 (0.000s) 328 # build & install p1 (0.085s) 329 # modify p2 - p1 should appear stale (0.030s) 330 # build & install p1 again (0.019s) 331 > go install -i p11 332 [stderr] 333 can't load package: package p11: cannot find package "p11" in any of: 334 /Users/rsc/go/src/p11 (from $GOROOT) 335 $WORK/d1/src/p11 (from $GOPATH) 336 $WORK/d2/src/p11 337 [exit status 1] 338 FAIL: unexpected go command failure 339 340 script_test.go:73: failed at testdata/script/install_rebuild_gopath.txt:15 in $WORK/gopath/src 341 342 FAIL 343 exit status 1 344 FAIL cmd/go 4.875s 345 $ 346 347 $ WORK=/tmp/cmd-go-test-745953508/script-install_rebuild_gopath 348 $ cd $WORK/d1/src/p1 349 $ cat p1.go 350 package p1 351 import "p2" 352 func F() { p2.F() } 353 $ 354 */ 355 package testscript 356