1# GEP-1709: Conformance Profiles
2
3* Issue: [#1709](https://github.com/kubernetes-sigs/gateway-api/issues/1709)
4* Status: Experimental
5* Probationary Period: Re-evaluate in February 2024
6
7## TLDR
8
9Add high level profiles for conformance tests which implementations can select
10when running the conformance test suite. Also add opt-in automated conformance
11reporting to the conformance test suite to report conformance results back to
12the Gateway API project and receive recognition (e.g. badges).
13
14## Goals
15
16- Add high level profiles which downstream implementations can subscribe to in
17 order to run tests for the associated supported feature sets.
18- Add a reporting mechanism where conformance results can be reported back to
19 the Gateway API project and provide "badges" to visibly decorate the
20 implementations as conformant according to their profiles.
21- Expand conformance testing documentation significantly so that it becomes the
22 "landing zone" for new prospective implementations and provides a clear and
23 iterative process for how to get started implementing Gateway API support.
24
25## Non-Goals
26
27- We want to avoid adding any infrastructure for the reporting mechanism if
28 feasible.
29- For this iteration we don't want to add configuration files for conformance
30 tests, instead leaving that to future iterations and working on the raw
31 machinery here (see [alternatives considered](#alternatives-considered)).
32- For this iteration we don't want to add container images for conformance test
33 runs, instead leaving that to future iterations (see
34 [alternatives considered](#alternatives-considered).
35
36## Introduction
37
38Since our conformance test suite was conceived of it's been our desire to
39provide simple high level profiles that downstream implementations can
40subscribe to.
41
42Today we have `SupportedFeatures` which get us some of what we want in terms of
43easily configuring the conformance test suite, but in this GEP we will describe
44taking that a step further (and a level higher) to create named profiles which
45indicate a "level of conformance" which implementations can prove they satisfy
46and receive certification for.
47
48An API will be provided as the format for conformance test reports. We will
49provide tooling to assist with the reporting and certification process of
50submitting those reports and displaying the results.
51
52## API
53
54The API for conformance profiles will include an API resource called
55`ConformanceReport` which will be at the center of a workflow that
56implementations can opt into to generate and submit those resources.
57
58The workflow implementers will follow will include the following high-level
59steps:
60
611. select a [profile](#profiles)
622. [integrate](#integration) tests in the downstream project
633. [report results and get certified](#certification)
64
65The goal is to make selecting a conformance profile as simple and automatic of a
66process as feasible and support both the existing command line integration
67approach (e.g. `go test`) as well as a [Golang][go] approach using the
68conformance suite as a library.
69
70[go]:https://go.dev
71
72### Profiles
73
74"Profiles" are effectively categories which represent the high level grouping of
75tests related to some feature (or feature set) of Gateway API. When conformance
76is reported using one of these profiles extra features can be covered according
77to support levels:
78
79- `core`
80- `extended`
81
82> **NOTE**: `implementation-specific` doesn't really have much in the way of
83> tests today, but it is something users want to be able to display. We leave
84> door open for it later and mention it in the [alternatives
85> considered](#alternatives-considered) section below.
86
87We will start with the following named profiles:
88
89- `HTTP`
90- `TLSPassthrough`
91
92These profiles correspond with `*Route` type APIs that we currently have tests
93for. As the tests roll in, we'll also eventually have:
94
95- `UDP`
96- `TCP`
97- `GRPC`
98
99> **NOTE**: In time we may have higher level groupings, like `Layer4` (which
100> would include at least `TCP` and `UDP`) but feedback from the community has
101> been strong for a preference on the `*Route` level (see the
102> [alternatives considered](#alternatives-considered) for some more notes on
103> this) for the moment.
104
105> **NOTE**: APIs that are referenced to or by `*Route` APIs will be tested as
106> a part of a profile. For instance, running the `HTTP` profile will also run
107> tests for `GatewayClass` and `Gateway` implicitly as these are required
108> components of supporting `HTTP`.
109
110The technical implementation of these profiles is very simple: effectively a
111"profile" is a static compilation of existing [SupportedFeatures][feat] which
112represent the named category. Features that aren't covered under a "core" level
113of support are opt-in.
114
115[feat]:https://github.com/kubernetes-sigs/gateway-api/blob/c61097edaa3b1fad29721e787fee4b02c35e3103/conformance/utils/suite/suite.go#L33
116
117### Integration
118
119Integrating the test suite into your implementation can be done using one of
120the following methods:
121
122- The [go test][go-test] command line interface which enables projects of any
123 language to run the test suite on any platform [Golang][go] supports.
124- Using the conformance test suite as a [Golang library][lib] within an already
125 existing test suite.
126
127> **NOTE**: Usage as a library is already an established colloquialism in the
128> community, this effort simply intends to make that more official.
129
130Conformance profiles are passed as arguments when running the test suite. For
131instance when running via command line:
132
133```console
134$ go test ./conformance/... -args -gateway-class=acme -conformance-profile=Layer7
135```
136
137Or the equivalent configuration using the Golang library:
138
139```go
140cSuite, err := suite.New(suite.Options{
141 GatewayClassName: "acme",
142 Profiles: sets.New(Layer7),
143 // other options
144})
145require.NoError(t, err, "misconfigured conformance test suite")
146cSuite.Setup(t)
147
148for i := 0; i < len(tests.ConformanceTests); i++ {
149 test := tests.ConformanceTests[i]
150 test.Run(t, cSuite)
151}
152```
153
154> **NOTE**: In the `suite.Options` above it's still possible to add `SkipTests`
155> but when used in conjunction with `Profile` this will result in a report that
156> the profile is not valid for reporting. Implementations in this state may be
157> able to report themselves as "in progress", see the
158> [certification section](#certification) for details.
159
160Alternatively for an `Extended` conformance profile where not all of the
161features are implemented (as described in the [profiles](#profiles) section
162above):
163
164```console
165$ go test ./conformance/... -args \
166 -gateway-class=acme \
167 -conformance-profiles=HTTP,TCP \
168 -unsupported-features=HTTPResponseHeaderModification,HTTPRouteMethodMatching,HTTPRouteQueryParamMatching,
169```
170
171Or the equivalent configuration using the Golang library:
172
173```go
174cSuite, err := suite.New(suite.Options{
175 GatewayClassName: "acme",
176 Profiles: sets.New(
177 HTTP,
178 TCP,
179 ),
180 UnsupportedFeatures: sets.New(
181 suite.SupportHTTPResponseHeaderModification,
182 suite.SupportHTTPRouteMethodMatching,
183 suite.SupportHTTPRouteQueryParamMatching,
184 ),
185 // other options
186})
187require.NoError(t, err, "misconfigured conformance test suite")
188cSuite.Setup(t)
189
190for i := 0; i < len(tests.ConformanceTests); i++ {
191 test := tests.ConformanceTests[i]
192 test.Run(t, cSuite)
193}
194```
195
196> **NOTE**: You can't disable features that are `Core` conformance as `Core` is
197> a minimum requirement for the profile to be considered fulfilled.
198
199Some implementations may support more or less extended features than others,
200so in some cases it could be cumbersome to have to list ALL features that you
201_don't_ support so we optionally and inversely allow `SupportedFeatures` so
202you can pick which option makes sense to you, and under the hood the
203expressions will compile to the same overall list:
204
205```go
206cSuite, err := suite.New(suite.Options{
207 GatewayClassName: "acme",
208 Profiles: sets.New(
209 HTTP,
210 TCP,
211 ),
212 SupportedFeatures: sets.New(
213 suite.SupportHTTPRouteMethodMatching,
214 ),
215 // other options
216})
217```
218
219> **NOTE**: The `UnsupportedFeatures` and `SupportedFeatures` fields are
220> mutually exclusive.
221
222So to have your YAML report include details about extended features you support
223you must either opt-in using `SupportedFeatures` to the exact features you
224support, or opt-out of the features you _don't_ support using
225`UnsupportedFeatures`.
226
227Once an implementation has integrated with the conformance test suite, they can
228move on to [certification](#certification) to report the results.
229
230[go-test]:https://go.dev/doc/tutorial/add-a-test
231[go]:https://go.dev
232[lib]:https://pkg.go.dev/sigs.k8s.io/gateway-api@v0.6.2/conformance/utils/suite
233
234### Gateway API version and channel
235
236The certification is related to a specific API version and a specific channel,
237therefore such information must be included in the final report. At test suite
238setup time, the conformance profile machinery gets all the CRDs with the field
239`.spec.group` equal to `gateway.networking.k8s.io`, and for each of them checks
240the annotations `gateway.networking.k8s.io/bundle-version` and
241`gateway.networking.k8s.io/channel`. If there are `CRD`s with different
242versions, the certification fails specifying that it's not possible to run the
243tests as there are different Gateway API versions installed in the cluster. If
244there are CRDs with different channels, the certification fails specifying that
245it's not possible to run the tests as there are different Gateway API channels
246installed in the cluster. If all the Gateway API `CRD`s have the same version
247and the same channel, the tests can be run and the detected version and channel
248will be set in the `GatewayAPIVersion` and `gatewayAPIChannel` fields of the
249final report. Furthermore, the suite must run all the experimental tests when
250the channel is `experimental`, and the related features are enabled.
251
252In addition to the `CRD`s version, the suite needs to check its version in
253relation to the `CRD`s one. To do so, a new `.go` file containing the current
254Gateway API version is introduced in the project and compiled with the
255conformance profile suite:
256
257```go
258const GatewayAPIVersion = "0.7.0"
259```
260
261At test suite setup time the conformance profile suite checks the `CRD`s version
262and the suite version; if the two versions differ, the certification fails. A
263new generator will be introduced in the project to generate the aforementioned
264`.go` file starting from a VERSION file contained in the root folder. Such a
265VERSION file contains the semver of the latest release and is manually bumped at
266release time. The script hack/verify-all.sh will be updated to ensure the
267generated `.go` file is up to date with the VERSION file.
268
269### Certification
270
271Implementations will be able to report their conformance testing results using
272our [reporting process](#reporting-process). Implementations will be able to
273visibly demonstrate their conformance results on their downstream projects and
274repositories using our [certification process](#certification-process).
275
276#### Reporting Process
277
278When conformance tests are executed an argument can be provided to the test
279suite to emit `ConformanceReport` resource with the test results. This resource
280can be configured to emit to the test output itself, or to a specific file.
281
282The following is an example report:
283
284```yaml
285apiVersion: v1alpha1
286kind: ConformanceReport
287implementation:
288 organization: acme
289 project: operator
290 url: https://acme.com
291 version: v1.0.0
292 contact:
293 - @acme/maintainers
294date: "2023-02-28 20:29:41+00:00"
295gatewayAPIVersion: v0.7.0
296gatewayAPIChannel: standard
297profiles:
298 - name: tcp
299 core:
300 result: success
301 summary: "all core functionality passed"
302 statistics:
303 passed: 4
304 skipped: 0
305 failed: 0
306 extended:
307 result: skipped
308 summary: "no extended features supported"
309 statistics:
310 passed: 0
311 skipped: 6
312 failed: 0
313 unsupportedFeatures:
314 - ExtendedFeature1
315 - ExtendedFeature2
316 - ExtendedFeature3
317 - name: http
318 core:
319 result: success
320 summary: "all core functionality passed"
321 statistics:
322 passed: 20
323 skipped: 0
324 failed: 0
325 extended:
326 result: success
327 summary: "all extended features supported"
328 statistics:
329 passed: 8
330 skipped: 0
331 failed: 0
332 supportedFeatures:
333 - ExtendedFeature1
334 - ExtendedFeature2
335 - ExtendedFeature3
336 - ExtendedFeature4
337 - ExtendedFeature5
338```
339
340> **WARNING**: It is an important clarification that this is NOT a full
341> Kubernetes API. It uses `TypeMeta` for some fields that made sense to re-use
342> and were familiar, but otherwise has it's own structure. It is not a [Custom
343> Resource Definition (CRD)][crd] nor will it be made available along with our
344> CRDs. It will be used only by conformance test tooling.
345
346> **NOTE**: The `implementation` field in the above example includes an
347> `organization` and `project` field. Organizations can be an open source
348> organization, an individual, a company, e.t.c.. Organizations can
349> theoretically have multiple projects and should submit separate reports for
350> each of them.
351
352> **NOTE**: The `contact` field indicates the Github usernames or team
353> names of those who are responsible for maintaining this file, so they can be
354> easily contacted when needed (e.g. for relevant release announcements
355> regarding conformance, e.t.c.). Optionally, it can be an email address or
356> a support URL (e.g. Github new issue page).
357
358The above report describes an implementation that just released `v1` and has
359`Core` support for `TCP` functionality and fully supports both `Core` and
360`Extended` `HTTP` functionality.
361
362`ConformanceReports` can be stored as a list of reports in chronological order.
363The following shows previous releases of the `acme`/`operator` implementation and
364its feature progression:
365
366```yaml
367apiVersion: v1alpha1
368kind: ConformanceReport
369implementation:
370 organization: acme
371 project: operator
372 url: https://acme.com
373 version: v0.91.0
374 contact:
375 - @acme/maintainers
376date: "2022-09-28 20:29:41+00:00"
377gatewayAPIVersion: v0.6.2
378gatewayAPIChannel: standard
379profiles:
380 - name: tcp
381 core:
382 result: partial
383 summary: "some tests were manually skipped"
384 statistics:
385 passed: 2
386 skipped: 2
387 failed: 0
388 skippedTests:
389 - TCPRouteBasics
390 - UDPRouteBasics
391 extended:
392 result: skipped
393 summary: "no extended features supported"
394 statistics:
395 passed: 0
396 skipped: 4
397 failed: 0
398 unsupportedFeatures:
399 - ExtendedFeature1
400 - ExtendedFeature2
401 - ExtendedFeature3
402 - name: http
403 core:
404 result: success
405 summary: "all core functionality passed"
406 statistics:
407 passed: 20
408 skipped: 0
409 failed: 0
410 extended:
411 result: success
412 summary: "all extended features supported"
413 statistics:
414 passed: 5
415 skipped: 3
416 failed: 0
417 supportedFeatures:
418 - ExtendedFeature1
419 - ExtendedFeature2
420 - ExtendedFeature3
421 unsupportedFeatures:
422 - ExtendedFeature4
423 - ExtendedFeature5
424---
425apiVersion: v1alpha1
426kind: ConformanceReport
427implementation:
428 organization: acme
429 project: operator
430 url: https://acmeorg.com
431 version: v0.90.0
432 contact:
433 - @acme/maintainers
434date: "2022-08-28 20:29:41+00:00"
435gatewayAPIVersion: v0.6.1
436gatewayAPIChannel: standard
437profiles:
438 - name: tcp
439 core:
440 result: failed
441 summary: "all tests are failing"
442 statistics:
443 passed: 0
444 skipped: 0
445 failed: 4
446 failedTests:
447 - TCPRouteExampleTest1
448 - TCPRouteExampleTest2
449 - TCPRouteExampleTest3
450 - TCPRouteExampleTest4
451 - name: http
452 core:
453 result: success
454 summary: "all core functionality passed"
455 statistics:
456 passed: 20
457 skipped: 0
458 failed: 0
459 extended:
460 result: skipped
461 summary: "no extended features supported"
462 statistics:
463 passed: 2
464 skipped: 6
465 failed: 0
466 supportedFeatures:
467 - ExtendedFeature1
468 unsupportedFeatures:
469 - ExtendedFeature2
470 - ExtendedFeature3
471 - ExtendedFeature4
472 - ExtendedFeature5
473---
474apiVersion: v1alpha1
475kind: ConformanceReport
476implementation:
477 organization: acme
478 project: operator
479 url: https://acmeorg.com
480 version: v0.89.0
481 contact:
482 - @acme/maintainers
483date: "2022-07-28 20:29:41+00:00"
484gatewayAPIVersion: v0.6.0
485gatewayAPIChannel: standard
486profiles:
487 - name: http
488 core:
489 result: partial
490 summary: "some tests were skipped"
491 statistics:
492 passed: 16
493 skipped: 2
494 failed: 0
495 skippedTests:
496 - HTTPRouteTestExample1
497 - HTTPRouteTestExample2
498 extended:
499 result: skipped
500 summary: "no extended features supported"
501 statistics:
502 passed: 0
503 skipped: 8
504 failed: 0
505 unsupportedFeatures:
506 - ExtendedFeature1
507 - ExtendedFeature2
508 - ExtendedFeature3
509 - ExtendedFeature4
510 - ExtendedFeature5
511```
512
513> **NOTE**: In the above you can see the `acme` implementation's progression. In
514> their release `v0.89.0` they had started adding `HTTP` support and added the
515> conformance tests to CI, but they were still skipping some core tests. In
516> their next release `v0.90.0` they completed adding `HTTP` `Core`
517> functionality (and even added one extended feature), and also starting adding
518> `TCP` functionality during `v0.90.0` (but it was failing at that time). In
519> `v0.91.0` they had completed core `HTTP` supported and added two more
520> `Extended` features, and also started to get their `TCP` functionality to
521> partially pass.
522
523Implementers can submit their reports upstream by creating a pull request to
524the Gateway API repository and adding new reports to a file specific to their
525implementation's organization and project name:
526
527```console
528conformance/reports/<version>/<organization>-<project>.yaml
529```
530
531The `<version>` directory in the above refers to the version of Gateway API. The
532`latest` release will include a symlink that points to the latest version
533directory.
534
535For instance:
536
537```console
538conformance/reports/v0.7.1/acme-operator.yaml
539```
540
541> **NOTE**: Implementations **MUST** report for a specific release version
542> (e.g. `v0.7.1`) and not use branches or Git SHAs. Some exceptions will be
543> made for initial reports to help make it easier for implementations to get
544> started, but as we move to standard everyone should be reporting on specific
545> releases.
546
547Creating a pull request to add the `ConformanceReport` will start the
548[certification process](#certification-process).
549
550> **NOTE**: No verification process (to prevent intentionally incorrect
551> conformance results) will be implemented at this time. We expect that this wont
552> be an issue in our community and even if someone were to try and "cheat" on
553> the reporting the reputation loss for being caught would make them look very
554> bad and would not be worth it.
555
556[crd]:https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/
557
558#### Certification Process
559
560For this initial iteration the raw report data of the `ConformanceReports` will
561live in its own directory and _is predominantly meant for machine consumption_.
562Report data will be compiled into human-friendly displays during the an
563automated certification process.
564
565Certification starts with the pull request described during the [reporting
566process](#reporting-process). Once the `ConformanceReport` is created or
567updated a display layer in the implementations page will need to be updated to
568point to the new data.
569
570> **NOTE**: The `ConformanceReport` API will be defined in Golang like our
571> other `apis/` so that we can utilize build tags from kubebuilder for defaults
572> and validation, and so that there exists a common Golang type for it in the
573> conformance test suite. When PRs are created the Gateway API repositories'
574> CI will run linting and validation against the reports.
575
576Maintainers will provide [badges][bdg] to implementations at the end of the process
577which link to the implementations page for that specific implementation and can
578be easily added via markdown to Git repositories.
579
580[impl]:https://gateway-api.sigs.k8s.io/implementations/
581[bdg]:https://shields.io
582
583## Alternatives Considered
584
585### Conformance Test Configuration File
586
587Conformance testing is currently done mainly through command line with
588`go test` or via use of the conformance test suite as a library in Golang
589projects. We considered whether adding the alternative to provide a
590configuration file to the test suite would be nice, but we haven't heard
591any specific demand for that from implementors yet so we're leaving the
592idea here for later iterations to consider.
593
594### Conformance Test Container Image
595
596Providing a container image which could be fed deployment instructions for an
597implementation was considered while working on this GET but it seemed like a
598scope all unto itself so we're avoiding it here and perhaps a later iteration
599could take a look at that if there are asks in the community.
600
601### Implementation-Specific Reporting
602
603Users have mentioned the desire to report on `implementation-specific` features
604they support as a part of conformance. At the time of writing, there's not much
605in the way of structure or testing for us to do this with but we remain open to
606the idea. The door is left open in the `ConformanceReport` API for a future
607iteration to add this if desired, but it probably warrants its own GEP as we
608need to make sure we have buy-in from multiple stakeholders with different
609implementations that are implementing those features.
610
611### High Level Profiles
612
613We originally started with two high level profiles:
614
615- `Layer4`
616- `Layer7`
617
618However the overwhelming feedback from the community was to go a step down and
619define profiles at the level of each individual API (e.g. `HTTPRoute`,
620`TCPRoute`, `GRPCRoute`, e.t.c.). One of the main reasons for this was that we
621already have multiple known implementations of Gateway API which only support
622a single route type (`UDPRoute`, in particular as it turns out).
623
624We may consider in the future doing some of these higher level profiles if
625there's a technical reason or strong desire from implementers.
626
627## Graduation Criteria
628
629The following are items that **MUST** be resolved to move this GEP to
630`Standard` status (and before the end of the probationary period):
631
632- [x] some kind of basic level of display for the report data needs to exist.
633 It's OK for a more robust display layer to be part of a follow-up effort.
634- [ ] initially we were OK with storing reports in the Git repository as files.
635 While this is probably sufficient during the `Experimental` phase, we need to
636 re-evaluate this before `Standard` and see if this remains sufficient or if
637 we want to store the data elsewhere.
638- [ ] We have been actively [gathering feedback from SIG
639 Arch][sig-arch-feedback]. Some time during the `experimental` phase needs to
640 be allowed to continue to engage with SIG Arch and incorporate their feedback
641 into the test suite.
642
643[sig-arch-feedback]:https://groups.google.com/g/kubernetes-sig-architecture/c/YjrVZ4NJQiA/m/7Qg7ScddBwAJ
644
645## References
646
647- https://github.com/kubernetes-sigs/gateway-api/issues/1709
648- https://github.com/kubernetes-sigs/gateway-api/issues/1329
649
View as plain text