1# GEP-1911: Backend Protocol Selection
2
3* Issue: [#1911](https://github.com/kubernetes-sigs/gateway-api/issues/1911)
4* Status: Experimental
5
6(See status definitions [here](overview.md#status).)
7
8## TLDR
9
10Not all implementations support automatic protocol selection. Even in some cases protocols are disabled without an explicit opt-in (eg. websockets with Contour & NGINX). Thus application developers need the ability to specify the protocol(s) that their application supports.
11
12## Goals
13
14- Support protocols that can have a Gateway `*Route` resource as a frontend
15- Standardize Gateway API implementations on the protocols & constants defined by the Kubernetes [Standard Application Protocols (KEP-3726)][kep-3726]
16- Support backends with multiple protocols on the same port (ie. tcp/udp)
17
18## Non-Goals
19
20- Backend TLS (covered in [GEP-1897](https://github.com/kubernetes-sigs/gateway-api/issues/1897))
21- Additional protocol specific configuration
22- Disabling Protocols
23
24## Introduction
25
26Since Kubernetes 1.20 the [`core/v1.Service`][k8s-service] and [`core/v1.EndpointSlice`][k8s-endpointslices] resource has a stable `appProtocol` field. It's purpose is to allow end-users to specify an application protocol (L7) for each service port.
27
28Originally the use of this field in the Gateway API was rejected in [GEP-1282](/geps/gep-1282#non-goals):
29> v1.Service’s appProtocol field is not fit for purpose, because it is defined as accepting values either from the IANA Service Name registry, or domain-prefixed values and we need more flexibility than that.
30
31Since then a Kubernetes enhancement proposal was created [KEP-3726][kep-3726] to repurpose `appProtocol` to include a convention for protocols that are not IANA service names. This would involve prefixing protocol names with `kubernetes.io/*`.
32
33Note: Kubernetes will automatically create `EndpointSlices` for `Services` that have a selector. [Custom `EndpointSlices`](https://kubernetes.io/docs/concepts/services-networking/service/#custom-endpointslices) can manually be created.
34
35## API Semantics
36
37A Gateway implementation MUST recognize the Kubernetes Standard Application Protocols ([KEP-3726][kep-3726]) for specifying the protocol for a backend reference in a Gateway API `*Route` resource
38
39Thus when a `*Route` points to a Kubernetes Service, implementations SHOULD honor the appProtocol field if it
40is set for the target Service Port.
41
42
43At the moment there exists three defined constants:
44
45- `kubernetes.io/h2c` - HTTP/2 over cleartext as described in [RFC7540](https://www.rfc-editor.org/rfc/rfc7540)
46- `kubernetes.io/ws` - WebSocket over cleartext as described in [RFC6445](https://www.rfc-editor.org/rfc/rfc6455)
47- `kubernetes.io/wss` - WebSocket over TLS as described in [RFC6455](https://www.rfc-editor.org/rfc/rfc6455)
48
49### New Protocols & Reserved Prefix
50
51To add support for a new protocol it should first become a Kubernetes Standard Application Protocol by updating the [KEP-3726][kep-3726]. [KEP-3726][kep-3726] also states the `appProtocol` field accepts a domain-prefixed implementation specific value. Thus, if the suggested protocol is not suited to have a `kubernetes.io/*` prefix, then the Gateway API MAY support the new protocol using it's own prefix `gateway.networking.k8s.io/*`. Please make a PR to this GEP.
52
53For example we may want to add a sentinel `appProtocol` value that prevents Gateway implementations from discovering the protocol of the application. Instead they should just refer to the Service's `protocol` field. Such a constant was rejected upstream (https://github.com/kubernetes/enhancements/pull/4106) but as an example it could be defined in a future addition to this GEP as `gateway.networking.k8s.io/no-sniff`.
54
55### Default Protocols
56
57If a Service `appProtocol` isn't specified an implementation MAY infer the backend protocol through its own means. Implementations MAY infer the protocol from the `Route` type referring to the backend Service.
58
59Absence of the `appProtocol` field does not imply the implementation should disable any features (eg. websocket upgrades).
60
61### Multiple Protocols on the Same Port
62
63Only the Kubernetes `Service` `protocol` field supports multiple protocols on the same port. See the details in [KEP-1435][kep-1435].
64
65Implementations MAY support Kubernetes Service BackendRefs that are multiplexing TCP and UDP on the same port. Otherwise implementations MUST set *Route ResolvedRefs condition to False with the "UnsupportedProtocol" Reason with a clear message that multiplexing is not supported.
66
67Currently Kubernetes `Service` API only allows different `appProtocol` values for the same port when `protocol` fields differs. At this time there seems to be interest in changing `appProtocol` to be a list in order to facilitate this use-case.
68
69### Supporting Protocols
70
71If a Route is not able to send traffic to the backend using the specified protocol then the backend is considered invalid. Implementations MUST set `ResolvedRefs` condition to `False` with the Reason `UnsupportedProtocol`.
72
73Implementations MAY support the following combinations below:
74
75ServicePort Protocol | ServicePort AppProtocol | Route Type | Supported
76-|-|-|-
77`TCP` | `kubernetes.io/h2c` | `GRPCRoute` | Yes [1]
78`TCP` | `kubernetes.io/h2c` | `HTTPRoute` | Yes
79`TCP` | `kubernetes.io/ws` | `HTTPRoute` | Yes
80`TCP` | `kubernetes.io/wss` | `TLSRoute` | Yes
81
821. GRPC works over h2c - so a GRPCRoute should be able to connect to an h2c backend
83
84Implementations MAY support the following combinations below:
85
86ServicePort Protocol | ServicePort AppProtocol | Route Type | Supported
87-|-|-|-
88ServicePort Protocol | ServicePort AppProtocol | Route Type | Supported
89`TCP` | `kubernetes.io/wss` | `HTTPRoute` | Conditional [1]
90
911. Only if there is a corresponding `BackendTLSPolicy` - see [GEP-1897](/geps/gep-1897)
92
93## Open Questions
94
951. TLSRoute & UDP protocol
96
97TLS over UDP seems to be a thing via QUIC/HTTP3 [ref](https://www.smashingmagazine.com/2021/08/http3-core-concepts-part1/).
98Likewise there's also [DTLS](https://en.wikipedia.org/wiki/Datagram_Transport_Layer_Security). But it's unclear if Gateway's TLSRoute
99applies to an underlying UDP protocol.
100
1012. Websockets & HTTP/2/3
102
103Should we upstream new constants for websocket over [HTTP/2](https://www.rfc-editor.org/rfc/rfc8441.html) & [HTTP/3](https://www.rfc-editor.org/rfc/rfc9220.html) ? HTTP/3 makes things more complicated since its supports UDP as the underlying protocol.
104
105## Alternatives
106
107### Single Meta-resource
108
109The first pass of this GEP proposed a new meta-resource [GEP-713](/geps/gep-713) called `BackendProtocol`.
110
111This allows end-users to specify a list of ports and a list of corresponding protocols that that single
112port supports.
113
114This was dropped in favour of supporting Kubernetes Standard Application Protocols.
115
116### Multiple Protocol Meta-resources
117
118Rather than bundle protocol details into a single resource an alternative would be to create distinct meta resources.
119ie. `HTTP2Backend`, `GPRCBackend`, `WebsocketBackend`.
120
121The advantages of this approach are:
122
123- Easy to introduce new protocols
124- Definitions/types would be simpler
125
126The disadvantages of this approach are:
127
128- N resources for N protocols need to be created to describe a single backend
129- No easy mechanic to specify priority of protocols
130
131### Adding Properties on Gateway Route Objects
132
133From [GEP-1282](/geps/gep-1282#tldr):
134> some types of configuration requested by users are more about defining functionality that describes capabilities of the backend more than the route you take to get to the backend.
135
136Backend protocol is specifying capabilities. This configuration is less about routing.
137
138### Kubernetes Service - Expanding Protocol field
139
140The `protocol` field on a Kubernetes service is used to specify a L4 protocol over IP. This field isn't appropriate to describe protocols
141that operate at a higher 'application' level (eg. HTTP/GRPC etc.)
142
143### Extending Kubernetes Service
144
145This is considered untenable due to the 'the turnaround time for those changes can be years.' ([ref-1282](/geps/gep-1282#non-goals))
146
147### Unstructured Data/Special Values
148
149Unstructured data refers to using labels and annotations.
150
151From [GEP-1282](/geps/gep-1282#non-goals):
152> these are very sticky and hard to get rid of once you start using them.
153
154Special values refers to using special strings in existing Kubernetes Resources.
155For example Istio allows for protocol to be specified by prefixing the Kubernetes
156Service's port name with the protocol (ie. `http-`, `grpc-`). This approach is
157limiting as it doesn't allow for multiple protocols on the same port and future
158configuration per protocol. One protocol per port may be relaxed in the future see
159[KEP 1435][kep-1435]
160
161Additionally, annotations are not self-documenting unlike CRD fields which can display
162documentation via `kubectl explain`
163
164## References
165
166- [GitHub Discussion](https://github.com/kubernetes-sigs/gateway-api/discussions/1244)
167- GEP-1282 - Describing Backend Properties
168 - [GEP](/geps/gep-1282)
169 - [Issue](https://github.com/kubernetes-sigs/gateway-api/issues/1911)
170- [GEP-713 - Metaresources](/geps/gep-713)
171- [Linkerd Protocol Detection](https://linkerd.io/2.12/features/protocol-detection/)
172- [Istio Protocol Selection](https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/)
173- Contour Protocol Selection
174 - [Websockets](https://projectcontour.io/docs/1.24/config/websockets/)
175 - [GRPC](https://projectcontour.io/docs/1.24/guides/grpc/#httpproxy-configuration)
176- [AWS Gateway Protocol Selection](https://github.com/aws/aws-application-networking-k8s/blob/a277fb39449383f53cd7d1e5576b4fa190a1a853/config/crds/bases/application-networking.k8s.aws_targetgrouppolicies.yaml#L109)
177- [Google GKE AppProtocol Selection](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress-xlb#https_tls_between_load_balancer_and_your_application)
178
179[k8s-service]: https://kubernetes.io/docs/concepts/services-networking/service/
180[k8s-endpointslices]: https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/
181[kep-3726]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/3726-standard-application-protocols
182[kep-1435]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1435-mixed-protocol-lb
View as plain text