1# GEP-1762: In Cluster Gateway Deployments
2
3* Status: Experimental
4
5## Overview
6
7Gateway API provides a common abstraction over different implementations, whether they are implemented by cloud load balancers, in-cluster deployments, or other mechanisms. However, many in-cluster implementations have solved some of the same problems in different ways.
8
9Related discussions:
10
11* [Support cluster-local Gateways](https://github.com/kubernetes-sigs/gateway-api/discussions/1247)
12* [Scaling Gateway Resources](https://github.com/kubernetes-sigs/gateway-api/discussions/1355)
13* [Manual deployments](https://github.com/kubernetes-sigs/gateway-api/issues/1687)
14* [Merging Gateways](https://github.com/kubernetes-sigs/gateway-api/pull/1863/)
15* [Per-Gateway Infrastructure](https://github.com/kubernetes-sigs/gateway-api/pull/1757)
16
17## Goals
18
19* Provide prescriptive guidance for how in-cluster implementations should behave.
20* Provide requirements for how in-cluster implementations should behave.
21
22Note that some changes will be suggestions, while others will be requirements.
23
24## Non-Goals
25
26* Provide guidance to how out-of-cluster implementations should behave. Rather, this document aims to bring consistency between these types.
27
28## Terminology
29
30This document uses a few terms throughout. To ensure consistency, they are defined below:
31
32* In-cluster deployment: refers to an implementation that actuates a `Gateway` by running a data plane in the cluster.
33 This is *often*, but not necessarily, by deploying a `Deployment`/`DaemonSet` and `Service`.
34* Automated deployment: refers to an implementation that automatically deploys the data plane based on a `Gateway`.
35 That is, the user simply creates a `Gateway` resource and the rest is handled behind the scenes by the implementation.
36
37## Design
38
39This GEP both introduces new API fields, and standardizes how implementations should behave when implementing the existing API.
40
41### Automated Deployments
42
43A simple `Gateway`, as is configured below is assumed to be an automated deployment:
44
45```yaml
46apiVersion: gateway.networking.k8s.io/v1beta1
47kind: Gateway
48metadata:
49 name: my-gateway
50spec:
51 gatewayClassName: example
52 listeners:
53 - name: default
54 port: 80
55 protocol: HTTP
56```
57
58With this configuration, an implementation:
59
60* MUST mark the Gateway as `Programmed` and provide an address in `Status.Addresses` where the Gateway can be reached on each configured port.
61* MUST label all generated resources (Service, Deployment, etc) with `gateway.networking.k8s.io/gateway-name: my-gateway` (where `my-gateway` is the name of the Gateway resource).
62* MUST provision generated resources in the same namespace as the Gateway if they are namespace scoped resources.
63 * Cluster scoped resources are not recommended.
64* SHOULD name all generated resources `my-gateway-example` (`<NAME>-<GATEWAY CLASS>`).
65 This is not simply `NAME` to reduce the chance of conflicts with existing resources.
66 Where required, this can also serve as the prefix for the object.
67
68### Customizations
69
70With any in-cluster deployment, customization requirements will arise.
71
72Some common requirements would be:
73
74* `Service.spec.type`, to control whether a service is a `ClusterIP` or `LoadBalancer`.
75* IP in the Service to assign to it.
76* Arbitrary labels and annotations on generated resources.
77* Any other arbitrary fields; the list is unbounded. Some examples would be:
78 * CPU and memory requests
79 * Service `externalTrafficPolicy`
80 * Affinity rules.
81
82This GEP currently only aims to solve a subset of these concerns. Additional concerns may be addressed in future revisions or other GEPs.
83
84#### Gateway Type
85
86This is handled by [GEP-1651](https://github.com/kubernetes-sigs/gateway-api/pull/1653), so won't be described here.
87
88#### Gateway IP
89
90This section just clarifies and existing part of the spec, how to handle `.spec.addresses` for in-cluster implementations.
91Like all other Gateway types, this should impact the address the `Gateway` is reachable at.
92
93For implementations using a `Service`, this means the `clusterIP` or `loadBalancerIP` (depending on the `Service` type).
94
95For example:
96
97```yaml
98apiVersion: gateway.networking.k8s.io/v1beta1
99kind: Gateway
100metadata:
101 name: my-gateway
102spec:
103 addresses:
104 - type: IPAddress
105 value: 1.1.1.1
106 gatewayClassName: example
107 listeners:
108 - name: default
109 port: 80
110 protocol: HTTP
111```
112
113This would generate a `Service` with `clusterIP` or `loadBalancerIP`, depending on the Service type.
114
115
116#### Labels and Annotations
117
118Labels and annotations for generated resources are specified in `infrastructure`:
119
120```yaml
121apiVersion: gateway.networking.k8s.io/v1beta1
122kind: Gateway
123metadata:
124 name: my-gateway
125spec:
126 infrastructure:
127 labels:
128 foo: bar
129 annotations:
130 name: my-annotation
131```
132
133These are both `map[string]string` types, just like in `ObjectMeta`.
134
135Any labels or annotations here are added to all generated resources.
136Note this may mean an annotation intended for a `Service` may end up on a `Deployment` (for example).
137This is typically not a concern; however, if an implementation is aware of specific meanings of certain labels or annotations, they MAY
138exclude these from irrelevant resources.
139
140This is intended to clearly identify resources associated with a specific application, environment, or Gateway.
141Additionally, it can be used support integration with the kitchen-sync of Kubernetes extensions which rely on labels and annotations.
142
143Validation will be added to prevent any usage with `gateway.networking.k8s.io/` prefix, to avoid conflicts with `gateway.networking.k8s.io/gateway-name` or other future additions.
144
145#### Arbitrary Customization
146
147GEP-1867 introduces a new `infrastructure` field, which allows customization of some common configurations (version, size, etc)
148and allows a per-Gateway generic `parametersRef`.
149This can be utilized for the remainder of customizations.
150
151### Resource Attachment
152
153Resources Generated in response to the `Gateway` will have two attributes:
154
155* A `gateway.networking.k8s.io/gateway-name: <NAME>` label.
156* A name `<NAME>-<GATEWAY CLASS>`. This format is not strictly required for implementations, but strongly recommended for consistency in attachment.
157
158The generated resources MUST be in the same namespaces as the `Gateway`.
159
160Implementations MAY set `ownerReferences` to the `Gateway` in most cases, as well, but this is not required
161as some implementations may have different cleanup mechanisms.
162
163The `gateway.networking.k8s.io/gateway-name` label and standardize resource naming format can be relied on to attach resources to.
164While "Policy attachment" in Gateway API would use attachment to the actual `Gateway` resource itself,
165many existing resources attach only to resources like `Deployment` or `Service`.
166
167An example using these:
168```yaml
169apiVersion: gateway.networking.k8s.io/v1beta1
170kind: Gateway
171metadata:
172 name: gateway
173spec:
174 gatewayClassName: example
175 listeners:
176 - name: default
177 hostname: "example.com"
178 port: 80
179 protocol: HTTP
180---
181apiVersion: autoscaling/v2
182kind: HorizontalPodAutoscaler
183metadata:
184 name: gateway
185spec:
186 # Match the generated Deployment by reference
187 # Note: Do not use `kind: Gateway`.
188 scaleTargetRef:
189 apiVersion: apps/v1
190 kind: Deployment
191 name: gateway-example
192 minReplicas: 2
193 maxReplicas: 5
194 metrics:
195 - type: Resource
196 resource:
197 name: cpu
198 target:
199 type: Utilization
200 averageUtilization: 50
201---
202apiVersion: policy/v1
203kind: PodDisruptionBudget
204metadata:
205 name: gateway
206spec:
207 minAvailable: 1
208 selector:
209 # Match the generated Deployment by label
210 matchLabels:
211 gateway.networking.k8s.io/metadata.name: gateway
212```
213
214Note: there is [discussion](https://github.com/kubernetes-sigs/gateway-api/discussions/1355) around a way to attach a HPA to a Gateway directly.
215
216## API
217
218This GEP extends the `infrastructure` API introduced in [GEP-1867](https://gateway-api.sigs.k8s.io/geps/gep-1867).
219
220```go
221type GatewayInfrastructure struct {
222 // Labels that should be applied to any resources created in response to this Gateway.
223 //
224 // For implementations creating other Kubernetes objects, this should be the `metadata.labels` field on resources.
225 // For other implementations, this refers to any relevant (implementation specific) "labels" concepts.
226 //
227 // An implementation may chose to add additional implementation-specific labels as they see fit.
228 //
229 // Support: Extended
230 // +kubebuilder:validation:MaxItems=8
231 Labels map[string]string `json:"labels,omitempty"`
232 // Annotations that should be applied to any resources created in response to this Gateway.
233 //
234 // For implementations creating other Kubernetes objects, this should be the `metadata.annotations` field on resources.
235 // For other implementations, this refers to any relevant (implementation specific) "annotations" concepts.
236 //
237 // An implementation may chose to add additional implementation-specific annotations as they see fit.
238 //
239 // Support: Extended
240 // +kubebuilder:validation:MaxItems=8
241 Annotations map[string]string `json:"annotations,omitempty"`
242 ...
243}
244```
245
246## Future Work
247
248* Allow various policies, [such as HPA](https://github.com/kubernetes-sigs/gateway-api/discussions/1355), to attach directly to `Gateway` rather than just `Deployment`.
View as plain text