# GEP-1762: In Cluster Gateway Deployments * Status: Experimental ## Overview Gateway API provides a common abstraction over different implementations, whether they are implemented by cloud load balancers, in-cluster deployments, or other mechanisms. However, many in-cluster implementations have solved some of the same problems in different ways. Related discussions: * [Support cluster-local Gateways](https://github.com/kubernetes-sigs/gateway-api/discussions/1247) * [Scaling Gateway Resources](https://github.com/kubernetes-sigs/gateway-api/discussions/1355) * [Manual deployments](https://github.com/kubernetes-sigs/gateway-api/issues/1687) * [Merging Gateways](https://github.com/kubernetes-sigs/gateway-api/pull/1863/) * [Per-Gateway Infrastructure](https://github.com/kubernetes-sigs/gateway-api/pull/1757) ## Goals * Provide prescriptive guidance for how in-cluster implementations should behave. * Provide requirements for how in-cluster implementations should behave. Note that some changes will be suggestions, while others will be requirements. ## Non-Goals * Provide guidance to how out-of-cluster implementations should behave. Rather, this document aims to bring consistency between these types. ## Terminology This document uses a few terms throughout. To ensure consistency, they are defined below: * In-cluster deployment: refers to an implementation that actuates a `Gateway` by running a data plane in the cluster. This is *often*, but not necessarily, by deploying a `Deployment`/`DaemonSet` and `Service`. * Automated deployment: refers to an implementation that automatically deploys the data plane based on a `Gateway`. That is, the user simply creates a `Gateway` resource and the rest is handled behind the scenes by the implementation. ## Design This GEP both introduces new API fields, and standardizes how implementations should behave when implementing the existing API. ### Automated Deployments A simple `Gateway`, as is configured below is assumed to be an automated deployment: ```yaml apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: my-gateway spec: gatewayClassName: example listeners: - name: default port: 80 protocol: HTTP ``` With this configuration, an implementation: * MUST mark the Gateway as `Programmed` and provide an address in `Status.Addresses` where the Gateway can be reached on each configured port. * MUST label all generated resources (Service, Deployment, etc) with `gateway.networking.k8s.io/gateway-name: my-gateway` (where `my-gateway` is the name of the Gateway resource). * MUST provision generated resources in the same namespace as the Gateway if they are namespace scoped resources. * Cluster scoped resources are not recommended. * SHOULD name all generated resources `my-gateway-example` (`-`). This is not simply `NAME` to reduce the chance of conflicts with existing resources. Where required, this can also serve as the prefix for the object. ### Customizations With any in-cluster deployment, customization requirements will arise. Some common requirements would be: * `Service.spec.type`, to control whether a service is a `ClusterIP` or `LoadBalancer`. * IP in the Service to assign to it. * Arbitrary labels and annotations on generated resources. * Any other arbitrary fields; the list is unbounded. Some examples would be: * CPU and memory requests * Service `externalTrafficPolicy` * Affinity rules. This GEP currently only aims to solve a subset of these concerns. Additional concerns may be addressed in future revisions or other GEPs. #### Gateway Type This is handled by [GEP-1651](https://github.com/kubernetes-sigs/gateway-api/pull/1653), so won't be described here. #### Gateway IP This section just clarifies and existing part of the spec, how to handle `.spec.addresses` for in-cluster implementations. Like all other Gateway types, this should impact the address the `Gateway` is reachable at. For implementations using a `Service`, this means the `clusterIP` or `loadBalancerIP` (depending on the `Service` type). For example: ```yaml apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: my-gateway spec: addresses: - type: IPAddress value: 1.1.1.1 gatewayClassName: example listeners: - name: default port: 80 protocol: HTTP ``` This would generate a `Service` with `clusterIP` or `loadBalancerIP`, depending on the Service type. #### Labels and Annotations Labels and annotations for generated resources are specified in `infrastructure`: ```yaml apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: my-gateway spec: infrastructure: labels: foo: bar annotations: name: my-annotation ``` These are both `map[string]string` types, just like in `ObjectMeta`. Any labels or annotations here are added to all generated resources. Note this may mean an annotation intended for a `Service` may end up on a `Deployment` (for example). This is typically not a concern; however, if an implementation is aware of specific meanings of certain labels or annotations, they MAY exclude these from irrelevant resources. This is intended to clearly identify resources associated with a specific application, environment, or Gateway. Additionally, it can be used support integration with the kitchen-sync of Kubernetes extensions which rely on labels and annotations. Validation will be added to prevent any usage with `gateway.networking.k8s.io/` prefix, to avoid conflicts with `gateway.networking.k8s.io/gateway-name` or other future additions. #### Arbitrary Customization GEP-1867 introduces a new `infrastructure` field, which allows customization of some common configurations (version, size, etc) and allows a per-Gateway generic `parametersRef`. This can be utilized for the remainder of customizations. ### Resource Attachment Resources Generated in response to the `Gateway` will have two attributes: * A `gateway.networking.k8s.io/gateway-name: ` label. * A name `-`. This format is not strictly required for implementations, but strongly recommended for consistency in attachment. The generated resources MUST be in the same namespaces as the `Gateway`. Implementations MAY set `ownerReferences` to the `Gateway` in most cases, as well, but this is not required as some implementations may have different cleanup mechanisms. The `gateway.networking.k8s.io/gateway-name` label and standardize resource naming format can be relied on to attach resources to. While "Policy attachment" in Gateway API would use attachment to the actual `Gateway` resource itself, many existing resources attach only to resources like `Deployment` or `Service`. An example using these: ```yaml apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: gateway spec: gatewayClassName: example listeners: - name: default hostname: "example.com" port: 80 protocol: HTTP --- apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: gateway spec: # Match the generated Deployment by reference # Note: Do not use `kind: Gateway`. scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: gateway-example minReplicas: 2 maxReplicas: 5 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50 --- apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: gateway spec: minAvailable: 1 selector: # Match the generated Deployment by label matchLabels: gateway.networking.k8s.io/metadata.name: gateway ``` Note: there is [discussion](https://github.com/kubernetes-sigs/gateway-api/discussions/1355) around a way to attach a HPA to a Gateway directly. ## API This GEP extends the `infrastructure` API introduced in [GEP-1867](https://gateway-api.sigs.k8s.io/geps/gep-1867). ```go type GatewayInfrastructure struct { // Labels that should be applied to any resources created in response to this Gateway. // // For implementations creating other Kubernetes objects, this should be the `metadata.labels` field on resources. // For other implementations, this refers to any relevant (implementation specific) "labels" concepts. // // An implementation may chose to add additional implementation-specific labels as they see fit. // // Support: Extended // +kubebuilder:validation:MaxItems=8 Labels map[string]string `json:"labels,omitempty"` // Annotations that should be applied to any resources created in response to this Gateway. // // For implementations creating other Kubernetes objects, this should be the `metadata.annotations` field on resources. // For other implementations, this refers to any relevant (implementation specific) "annotations" concepts. // // An implementation may chose to add additional implementation-specific annotations as they see fit. // // Support: Extended // +kubebuilder:validation:MaxItems=8 Annotations map[string]string `json:"annotations,omitempty"` ... } ``` ## Future Work * Allow various policies, [such as HPA](https://github.com/kubernetes-sigs/gateway-api/discussions/1355), to attach directly to `Gateway` rather than just `Deployment`.