1# Azure Pipelines Matrix Generator
2
3* [Usage in a pipeline](#usage-in-a-pipeline)
4* [Matrix config file syntax](#matrix-config-file-syntax)
5 * [Fields](#fields)
6 * [matrix](#matrix)
7 * [include](#include)
8 * [exclude](#exclude)
9 * [displayNames](#displaynames)
10 * [$IMPORT](#import)
11* [Matrix Generation behavior](#matrix-generation-behavior)
12 * [all](#all)
13 * [sparse](#sparse)
14 * [include/exclude](#includeexclude)
15 * [displayNames](#displaynames-1)
16 * [Filters](#filters)
17 * [Replace/Modify/Append](#replacemodifyappend-values)
18 * [NonSparseParameters](#nonsparseparameters)
19 * [Under the hood](#under-the-hood)
20* [Testing](#testing)
21
22
23This directory contains scripts supporting dynamic, cross-product matrix generation for azure pipeline jobs.
24It aims to replicate the [cross-product matrix functionality in github actions](https://docs.github.com/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#example-running-with-more-than-one-version-of-nodejs),
25but also adds some additional features like sparse matrix generation, cross-product includes and excludes, and programmable matrix filters.
26
27This functionality is made possible by the ability for the azure pipelines yaml to take a [dynamic variable as an input
28for a job matrix definition](https://docs.microsoft.com/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#multi-job-configuration) (see the code sample at the bottom of the linked section).
29
30## Usage in a pipeline
31
32In order to use these scripts in a pipeline, you must provide a config file and call the matrix creation script within a powershell job.
33
34For a single matrix, you can include the `/eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml` template in a pipeline (see /eng/common/scripts/job-matrix/samples/matrix-test.yml for a full working example):
35
36```
37jobs:
38 - template: /eng/common/pipelines/templates/jobs/archetype-sdk-tests-generate.yml
39 parameters:
40 MatrixConfigs:
41 - Name: base_product_matrix
42 Path: eng/scripts/job-matrix/samples/matrix.json
43 Selection: all
44 NonSparseParameters:
45 - framework
46 GenerateVMJobs: true
47 - Name: sparse_product_matrix
48 Path: eng/scripts/job-matrix/samples/matrix.json
49 Selection: sparse
50 GenerateVMJobs: true
51 JobTemplatePath: /eng/common/scripts/job-matrix/samples/matrix-job-sample.yml
52 AdditionalParameters: []
53 CloudConfig:
54 SubscriptionConfiguration: $(sub-config-azure-cloud-test-resources)
55 Location: eastus2
56 Cloud: Public
57 MatrixFilters: []
58 MatrixReplace: []
59 PreGenerationSteps: []
60```
61
62### A note regarding PreGenerationSteps
63
64The generation template laid out above runs as its own job. A limitation of this method is that it disallows any runtime matrix customization due to the fact that an individual job clones the targeted build SHA. The stepList `PreGenerationSteps` allows users to update matrix json however they like prior to actually invoking the matrix generation. Injected steps are run after the repository checkout, but before any matrix generation is invoked.
65
66## Matrix config file syntax
67
68Matrix parameters can either be a list of strings, or a set of grouped strings (represented as a hash). The latter parameter
69type is useful for when 2 or more parameters need to be grouped together, but without generating more than one matrix permutation.
70
71```
72"matrix": {
73 "<parameter1 name>": [ <values...> ],
74 "<parameter2 name>": [ <values...> ],
75 "<parameter set>": {
76 "<parameter set 1 name>": {
77 "<parameter set 1 value 1": "value",
78 "<parameter set 1 value 2": "<value>",
79 },
80 "<parameter set 2 name>": {
81 "<parameter set 2 value 1": "value",
82 "<parameter set 2 value 2": "<value>",
83 }
84 }
85}
86"include": [ <matrix>, <matrix>, ... ],
87"exclude": [ <matrix>, <matrix>, ... ],
88"displayNames": { <parameter value>: <human readable override> }
89```
90
91See `samples/matrix.json` for a full sample.
92
93### Fields
94
95#### matrix
96
97The `matrix` field defines the base cross-product matrix. The generated matrix can be full or sparse.
98
99Example:
100```
101"matrix": {
102 "operatingSystem": [
103 "windows-2019",
104 "ubuntu-18.04",
105 "macOS-10.15"
106 ],
107 "framework": [
108 "net461",
109 "netcoreapp2.1",
110 "net50"
111 ],
112 "additionalTestArguments": [
113 "",
114 "/p:UseProjectReferenceToAzureClients=true",
115 ]
116}
117```
118
119#### include
120
121The `include` field defines any number of matrices to be appended to the base matrix after processing exclusions.
122
123#### exclude
124
125The `include` field defines any number of matrices to be removed from the base matrix. Exclude parameters can be a partial
126set, meaning as long as all exclude parameters match against a matrix entry (even if the matrix entry has additional parameters),
127then it will be excluded from the matrix. For example, the below entry will match the exclusion and be removed:
128
129```
130matrix entry:
131{
132 "a": 1,
133 "b": 2,
134 "c": 3,
135}
136
137"exclude": [
138 {
139 "a": 1,
140 "b": 2
141 }
142]
143```
144
145#### displayNames
146
147Specify any overrides for the azure pipelines definition and UI that determines the matrix job name. If some parameter
148values are too long or unreadable for this purpose (e.g. a command line argument), then you can replace them with a more
149readable value here. For example:
150
151```
152"displayNames": {
153 "/p:UseProjectReferenceToAzureClients=true": "UseProjectRef"
154},
155"matrix": {
156 "additionalTestArguments": [
157 "/p:UseProjectReferenceToAzureClients=true"
158 ]
159}
160```
161
162#### $IMPORT
163
164Matrix configs can also import another matrix config. The effect of this is the imported matrix will be generated,
165and then the importing config will be combined with that matrix (as if each entry of the imported matrix was a parameter).
166To import a matrix, add a parameter with the key `$IMPORT`:
167
168```
169"matrix": {
170 "$IMPORT": "path/to/matrix.json",
171 "JavaVersion": [ "1.8", "1.11" ]
172}
173```
174
175Importing can be useful, for example, in cases where there is a shared base matrix, but there is a need to run it
176once for each instance of a language version. Importing does not support overriding duplicate parameters. To achieve
177this, use the [Replace](#replacemodifyappend-values) argument instead.
178
179The `Selection` and `NonSparseParameters` parameters are respected when generating an imported matrix.
180
181The processing order is as follows:
182
183Given a matrix and import matrix like below:
184```
185{
186 "matrix": {
187 "$IMPORT": "example-matrix.json",
188 "endpointType": [ "storage", "cosmos" ],
189 "JavaVersion": [ "1.8", "1.11" ]
190 },
191 "include": [
192 {
193 "operatingSystem": "windows",
194 "mode": "TestFromSource",
195 "JavaVersion": "1.8"
196 }
197 ]
198}
199
200### example-matrix.json to import
201{
202 "matrix": {
203 "operatingSystem": [ "windows", "linux" ],
204 "client": [ "netty", "okhttp" ]
205 },
206 "include": [
207 {
208 "operatingSystem": "mac",
209 "client": "netty"
210 }
211 ]
212}
213```
214
2151. The base matrix is generated (sparse in this example):
216 ```
217 {
218 "storage_18": {
219 "endpointType": "storage",
220 "JavaVersion": "1.8"
221 },
222 "cosmos_111": {
223 "endpointType": "cosmos",
224 "JavaVersion": "1.11"
225 }
226 }
227 ```
2281. The imported base matrix is generated (sparse in this example):
229 ```
230 {
231 "windows_netty": {
232 "operatingSystem": "windows",
233 "client": "netty"
234 },
235 "linux_okhttp": {
236 "operatingSystem": "linux",
237 "client": "okhttp"
238 }
239 }
240 ```
2411. Includes/excludes from the imported matrix get applied to the imported matrix
242 ```
243 {
244 "windows_netty": {
245 "operatingSystem": "windows",
246 "client": "netty"
247 },
248 "linux_okhttp": {
249 "operatingSystem": "linux",
250 "client": "okhttp"
251 },
252 "mac_netty": {
253 "operatingSystem": "mac",
254 "client": "netty"
255 }
256 }
257 ```
2581. The base matrix is multipled by the imported matrix (in this case, the base matrix has 2 elements, and the imported
259 matrix has 3 elements, so the product is a matrix with 6 elements:
260 ```
261 "storage_18_windows_netty": {
262 "endpointType": "storage",
263 "JavaVersion": "1.8",
264 "operatingSystem": "windows",
265 "client": "netty"
266 },
267 "storage_18_linux_okhttp": {
268 "endpointType": "storage",
269 "JavaVersion": "1.8",
270 "operatingSystem": "linux",
271 "client": "okhttp"
272 },
273 "storage_18_mac_netty": {
274 "endpointType": "storage",
275 "JavaVersion": "1.8",
276 "operatingSystem": "mac",
277 "client": "netty"
278 },
279 "cosmos_111_windows_netty": {
280 "endpointType": "cosmos",
281 "JavaVersion": "1.11",
282 "operatingSystem": "windows",
283 "client": "netty"
284 },
285 "cosmos_111_linux_okhttp": {
286 "endpointType": "cosmos",
287 "JavaVersion": "1.11",
288 "operatingSystem": "linux",
289 "client": "okhttp"
290 },
291 "cosmos_111_mac_netty": {
292 "endpointType": "cosmos",
293 "JavaVersion": "1.11",
294 "operatingSystem": "mac",
295 "client": "netty"
296 }
297 }
298 ```
2991. Includes/excludes from the top-level matrix get applied to the multiplied matrix, so the below element will be added
300 to the above matrix, for an output matrix with 7 elements:
301 ```
302 "windows_TestFromSource_18": {
303 "operatingSystem": "windows",
304 "mode": "TestFromSource",
305 "JavaVersion": "1.8"
306 }
307 ```
308
309## Matrix Generation behavior
310
311#### all
312
313`all` will output the full matrix, i.e. every possible permutation of all parameters given (p1.Length * p2.Length * ...).
314
315#### sparse
316
317`sparse` outputs the minimum number of parameter combinations while ensuring that all parameter values are present in at least one matrix job.
318Effectively this means the total length of a sparse matrix will be equal to the largest matrix dimension, i.e. `max(p1.Length, p2.Length, ...)`.
319
320To build a sparse matrix, a full matrix is generated, and then walked diagonally N times where N is the largest matrix dimension.
321This pattern works for any N-dimensional matrix, via an incrementing index (n, n, n, ...), (n+1, n+1, n+1, ...), etc.
322Index lookups against matrix dimensions are calculated modulus the dimension size, so a two-dimensional matrix of 4x2 might be walked like this:
323
324```
325index: 0, 0:
326o . . .
327. . . .
328
329index: 1, 1:
330. . . .
331. o . .
332
333index: 2, 2 (modded to 2, 0):
334. . o .
335. . . .
336
337index: 3, 3 (modded to 3, 1):
338. . . .
339. . . o
340```
341
342#### include/exclude
343
344Include and exclude support additions and subtractions off the base matrix. Both include and exclude take an array of matrix values.
345Typically these values will be a single entry, but they also support the cross-product matrix definition syntax of the base matrix.
346
347Include and exclude are parsed fully. So if a sparse matrix is called for, a sparse version of the base matrix will be generated, but
348the full matrix of both include and exclude will be processed.
349
350Excludes are processed first, so includes can be used to add back any specific jobs to the matrix.
351
352#### displayNames
353
354In the matrix job output that azure pipelines consumes, the format is a dictionary of dictionaries. For example:
355
356```
357{
358 "net461_macOS1015": {
359 "framework": "net461",
360 "operatingSystem": "macOS-10.15"
361 },
362 "net50_ubuntu1804": {
363 "framework": "net50",
364 "operatingSystem": "ubuntu-18.04"
365 },
366 "netcoreapp21_windows2019": {
367 "framework": "netcoreapp2.1",
368 "operatingSystem": "windows-2019"
369 },
370 "UseProjectRef_net461_windows2019": {
371 "additionalTestArguments": "/p:UseProjectReferenceToAzureClients=true",
372 "framework": "net461",
373 "operatingSystem": "windows-2019"
374 }
375}
376```
377
378The top level keys are used as job names, meaning they get displayed in the azure pipelines UI when running the pipeline.
379
380The logic for generating display names works like this:
381
382- Join parameter values by "_"
383 a. If the parameter value exists as a key in `displayNames` in the matrix config, replace it with that value.
384 b. For each name value, strip all non-alphanumeric characters (excluding "_").
385 c. If the name is greater than 100 characters, truncate it.
386
387#### Filters
388
389Filters can be passed to the matrix as an array of strings, each matching the format of `<key>=<regex>`. When a matrix entry
390does not contain the specified key, it will default to a value of empty string for regex parsing. This can be used to specify
391filters for keys that don't exist or keys that optionally exist and match a regex, as seen in the below example.
392
393Display name filters can also be passed as a single regex string that runs against the [generated display name](#displaynames) of the matrix job.
394The intent of display name filters is to be defined primarily as a top level variable at template queue time in the azure pipelines UI.
395
396For example, the below command will filter for matrix entries with "windows" in the job display name, no matrix variable
397named "ExcludedKey", a framework variable containing either "461" or "5.0", and an optional key "SupportedClouds" that, if exists, must contain "Public":
398
399```
400./Create-JobMatrix.ps1 `
401 -ConfigPath samples/matrix.json `
402 -Selection all `
403 -DisplayNameFilter ".*windows.*" `
404 -Filters @("ExcludedKey=^$", "framework=(461|5\.0)", "SupportedClouds=^$|.*Public.*")
405```
406
407#### Replace/Modify/Append Values
408
409Replacements for values can be passed to the matrix as an array of strings, each matching the format of `<keyRegex>=<valueRegex>/<replacementValue>`.
410The replace argument will find any permutations where the key fully matches the key regex and the value fully matches the value regex, and replace the value with
411the replacement specified.
412
413NOTE:
414- The replacement value supports regex capture groups, enabling substring transformations, e.g. `Foo=(.*)-replaceMe/$1-replaced`. See the below examples for usage.
415- For each key/value, the first replacement provided that matches will be the only one applied.
416- If `=` or `/` characters need to be part of the regex or replacement, escape them with `\`.
417
418For example, given a matrix config like below:
419
420```
421{
422 "matrix": {
423 "Agent": {
424 "ubuntu-1804": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" }
425 },
426 "JavaTestVersion": [ "1.8", "1.11" ]
427 }
428}
429
430```
431
432The normal matrix output (without replacements), looks like:
433
434```
435$ ./Create-JobMatrix.ps1 -ConfigPath <test> -Selection all
436{
437 "ubuntu1804_18": {
438 "OSVmImage": "MMSUbuntu18.04",
439 "Pool": "azsdk-pool-mms-ubuntu-1804-general",
440 "JavaTestVersion": "1.8"
441 },
442 "ubuntu1804_111": {
443 "OSVmImage": "MMSUbuntu18.04",
444 "Pool": "azsdk-pool-mms-ubuntu-1804-general",
445 "JavaTestVersion": "1.11"
446 }
447}
448```
449
450Passing in multiple replacements, the output will look like below. Note that replacing key/values that appear nested within a grouping
451will not affect that segment of the job name, since the job takes the grouping name (in this case "ubuntu1804").
452
453The below example includes samples of regex grouping references, and wildcard key/value regexes:
454
455```
456$ $replacements = @('.*Version=1.11/2.0', 'Pool=(.*ubuntu.*)-general/$1-custom')
457$ ../Create-JobMatrix.ps1 -ConfigPath ./test.Json -Selection all -Replace $replacements
458{
459 "ubuntu1804_18": {
460 "OSVmImage": "MMSUbuntu18.04",
461 "Pool": "azsdk-pool-mms-ubuntu-1804-custom",
462 "JavaTestVersion": "1.8"
463 },
464 "ubuntu1804_20": {
465 "OSVmImage": "MMSUbuntu18.04",
466 "Pool": "azsdk-pool-mms-ubuntu-1804-custom",
467 "JavaTestVersion": "2.0"
468 }
469}
470```
471
472#### NonSparseParameters
473
474Sometimes it may be necessary to generate a sparse matrix, but keep the full combination of a few parameters. The
475NonSparseParameters argument allows for more fine-grained control of matrix generation. For example:
476
477```
478./Create-JobMatrix.ps1 `
479 -ConfigPath /path/to/matrix.json `
480 -Selection sparse `
481 -NonSparseParameters @("JavaTestVersion")
482```
483
484Given a matrix like below with `JavaTestVersion` marked as a non-sparse parameter:
485
486```
487{
488 "matrix": {
489 "Agent": {
490 "windows-2019": { "OSVmImage": "MMS2019", "Pool": "azsdk-pool-mms-win-2019-general" },
491 "ubuntu-1804": { "OSVmImage": "MMSUbuntu18.04", "Pool": "azsdk-pool-mms-ubuntu-1804-general" },
492 "macOS-10.15": { "OSVmImage": "macOS-10.15", "Pool": "Azure Pipelines" }
493 },
494 "JavaTestVersion": [ "1.8", "1.11" ],
495 "AZURE_TEST_HTTP_CLIENTS": "netty",
496 "ArmTemplateParameters": [ "@{endpointType='storage'}", "@{endpointType='cosmos'}" ]
497 }
498}
499```
500
501A matrix with 6 entries will be generated: A sparse matrix of Agent, AZURE_TEST_HTTP_CLIENTS and ArmTemplateParameters
502(3 total entries) will be multipled by the two `JavaTestVersion` parameters `1.8` and `1.11`.
503
504NOTE: NonSparseParameters are also applied when generating an imported matrix.
505
506#### Under the hood
507
508The script generates an N-dimensional matrix with dimensions equal to the parameter array lengths. For example,
509the below config would generate a 2x2x1x1x1 matrix (five-dimensional):
510
511```
512"matrix": {
513 "framework": [ "net461", "netcoreapp2.1" ],
514 "additionalTestArguments": [ "", "/p:SuperTest=true" ]
515 "pool": [ "ubuntu-18.04" ],
516 "container": [ "ubuntu-18.04" ],
517 "testMode": [ "Record" ]
518}
519```
520
521The matrix is stored as a one-dimensional array, with a row-major indexing scheme (e.g. `(2, 1, 0, 1, 0)`).
522
523## Testing
524
525The matrix functions can be tested using [pester](https://pester.dev/). The test command must be run from within the tests directory.
526
527```
528$ cd tests
529$ Invoke-Pester
530
531Starting discovery in 3 files.
532Discovery finished in 75ms.
533[+] /home/ben/sdk/azure-sdk-tools/eng/common/scripts/job-matrix/tests/job-matrix-functions.filter.tests.ps1 750ms (309ms|428ms)
534[+] /home/ben/sdk/azure-sdk-tools/eng/common/scripts/job-matrix/tests/job-matrix-functions.modification.tests.ps1 867ms (250ms|608ms)
535[+] /home/ben/sdk/azure-sdk-tools/eng/common/scripts/job-matrix/tests/job-matrix-functions.tests.ps1 2.71s (725ms|1.93s)
536Tests completed in 4.33s
537Tests Passed: 141, Failed: 0, Skipped: 4 NotRun: 0
538```
View as plain text