1# GEP-1364: Status and Conditions Update
2
3* Issue: [#1364](https://github.com/kubernetes-sigs/gateway-api/issues/1364)
4* Status: Standard
5
6## TLDR
7
8The status, particularly the Conditions, across the whole Gateway API have very much
9grown organically, and so have many inconsistencies and odd behaviors.
10This GEP covers doing a review and consolidation to make Condition behavior consistent
11across the whole API.
12
13## Goals
14
15* Update Conditions design to be consistent across Gateway API resources
16* Provide a model and guidelines for Conditions for future new resources
17* Specify changes to conformance required for Condition updates
18
19## Non-Goals
20
21* Define the full set of Conditions that will ever be used with Gateway API
22
23## Introduction
24
25Gateway API currently has a lot of issues related to status, especially that
26status is inconsistent ([#1111][1111]), that names are hard to understand ([#1110][1110]),
27and that Reasons aren't explained properly ([#1362][1362]).
28
29As the API has grown, the way we talk about resources has changed a lot, and some of the
30status design hasn't been updated since resources were created.
31
32So, for example, we have GatewayClass with `Accepted`, Gateway with `Scheduled`,
33the Gateway Listeners with `Detached` (which you want to be `false`, unlike the previous
34two), and then Gateways and Gateway Listeners have `Ready`, but Route doesn't (and which
35also you want to be `true`).
36
37This document lays out large-scale changes to the way that we talk about resources,
38and the Conditions to match them. This means that there will be an unavoidable break
39in what constitutes a healthy or unhealthy resource, and there will be changes
40required for all implementations to be conformant with the release that includes
41these changes.
42
43The constants that mark the deprecated types will be also marked as deprecated,
44and will no longer be tested as part of conformance. They'll still be present,
45and will work, but they won't be part of the spec any more. This should give
46implementations and users a release to transition to the new design (in UX terms).
47This grace period should be one release (so, the constants will be removed in
48v0.7.0.)
49
50This level of change is not optimal, and the intent is to make this a one-off change
51that can be built upon for future resources - since there are definitely more resources
52on the way.
53
54## Background: Kubernetes API conventions and prior art on Conditions
55
56Because this GEP is mainly concerned with updating the Conditions we are setting in
57Gateway API resources' `status`, it's worth reviewing some important points about
58Conditions. (This information is mainly taken from the [Typical status properties][typstatus]
59section of the API conventions document.)
60
611. Conditions are a standard type used to represent arbitrary higher-level status from
62a controller.
632. They are a listMapType, a list that is enforced by the apiserver to have only
64one entry of each item, using the `type` field as a key. (So, this is effectively
65a map that looks like a list in YAML form).
663. Each has a number of fields, the most important of which for this discussion
67are `type`, `status`, `reason`, and `observedGeneration`.
68
69 * `type` is a string value indicating the Condition type. `Accepted`, `Scheduled`,
70 and `Ready` are current examples.
71 * `status` indicates the state of the condition, and can be one of three values,
72 `true`, `false`, or `unknown`. Unknown in particular is important, because it
73 means that the controller is unable to determine the status for some reason.
74 (Also notable is that "" is also valid, and must be treated as `Unknown`.
75 Controllers must not set the value to "", but consumers should accept it
76 as meaning the same thing as `Unknown`.)
77 * `reason` is a CamelCase string that is a brief description of the reason why
78 the `status` is set the way it is.
79 * `observedGeneration` is an optional field that sets what the `metadata.generation`
80 field was when the controller last saw a resource. Note that this is optional
81 _in the struct_, but is required for Gateway API conditions. This will be
82 enforced in the conformance tests in the future.
83
844. Conditions should describe the _current state_ of the resource at observation
85time, which means that they should be an adjective (like `Ready`), or a past-tense
86verb (like `Accepted`). This one in particular is documented pretty closely in the
87[Typical status properties][typstatus] section of the guidelines.
885. Conditions should be applied to a resource the first time the controller sees
89the resource. This seems to imply that _all conditions should be present on every
90resource owned by a controller_, but the rest of the conventions don't make this
91clear, and it is often not complied with.
926. It's helpful to have a top-level condition which summarizes more detailed conditions.
93The guidelines suggest using either `Ready` for long-running processes, or `Succeeded`
94for bounded execution.
95
96From these guidelines, we can see that Conditions can be either _positive polarity_
97(healthy resources have them as `status: true`) or _negative polarity_ (healthy
98resources have them as `status: false`). `Ready` is an example of a positive polarity
99condition, and conditions like `Conflicted` from Listener or `NetworkUnavailable`,
100`MemoryPressure`, or `DiskPressure` from the Node resource are examples of
101negative-polarity conditions.
102
103There is also some extra context that's not in the API conventions doc:
104
105SIG-API Machinery has been reluctant to add fields that would aid in machine-parsing
106of Conditions, especially fields that would indicate the polarity, because they
107are intended more for human consumption than machine consumption. Probably the best
108example of this was in the PR [#4521](https://github.com/kubernetes/community/pull/4521#issuecomment-64894206).
109
110This means that there's no guidance from upstream about condition polarity. We'll
111discuss this more when we talk about new conditions.
112
113The guidance about Conditions being added as soon as a controller sees a resource
114is a bit unclear - as written in the conventions, it seems to imply that _all_
115relevant conditions should always be added, even if their status has to be set to
116`unknown`.
117Gateway API resources do not currently require this, and the practice seems to be
118uncommon.
119
120## Proposed changes
121
122### Proposed changes summary
123
124* All the current Conditions that indicate that the resource is okay and ready
125for processing will be replaced with `Accepted`.
126* In general, resources should be considered `Accepted` if their config is valid
127enough to generate some config in the underlying data plane. Examples are provided
128below.
129* There will be a limited set of positive polarity summary conditions, and a number
130of other specific negative-polarity error conditions.
131* All relevant positive-polarity summary Conditions for a resource must be added
132when it's observed.
133For example, HTTPRoutes must always have `Accepted` and `ResolvedRefs`, regardless
134of their state.
135* Negative polarity error conditions must only be added when the error is True.
136* The `Ready` condition will be moved to Extended conformance, and we'll re-evaluate
137if it's used by any implementations after some time has passed. If not, it may be
138removed.
139* To capture the behavior that `Ready` currently captures, `Programmed` will be
140introduced. This means that the implementation has seen the config, has everything
141it needs, parsed it, and sent configuration off to the data plane. The configuration
142should be available "soon". We'll leave "soon" undefined for now.
143* Resolving a comment that came up, documentation will be added to clarify that
144it's okay to add your own Conditions, and that implementations should namespace
145their custom Conditions with a domain prefix (so `implementation.io/CustomType`
146rather than just `CustomType`), or run the risk of using a word that's reserved later.
147* It's recommended that implementations publish both new and old conditions to
148provide a smoother transition, but conformance tests will only require the new
149conditions.
150
151The exact list of changes is detailed below. The next few sections detail
152the reasons for these large-scale changes.
153
154### Conceptual and language changes
155
156Gateway API resources are, conceptually, all about breaking up the configuration for a
157data plane into separate resources that are _expressive_ and _extensible_, while being
158split up along _role-oriented_ boundaries.
159
160So, when we talk about Gateway API, it's _always_ about a _system of related resources_.
161
162We already acknowledge this when we talk about Routes "attaching" to Gateways, or Gateways
163referencing Services, or Gateways requiring a GatewayClass in their spec.
164
165However, this GEP is proposing that we move all our discussion into using
166"accepted" to indicate that a resource has attached correctly enough to be
167_accepted_ for processing.
168
169So resources are `Accepted` for processing when their attachment succeeds enough
170to generate some configuration. This allows us to make calls about when partially
171valid objects should be accepted and when they shouldn't.
172
173Of course, because we're using all of this configuration to describe some sort of data
174path from "outside"/lacking cluster context to "inside"/enriched with cluster context,
175we also need a way to describe when that data path is configured and working.
176
177We already have a word in the Kubernetes API, but it comes with some expectations
178that implementations are not currently able to meet. That word is `Ready`, but it
179implies that the data path is Ready _when you read the status_, rather than that
180it _will be ready soon_ (which is what most implementations can guarantee currently.)
181
182So we have an unresolved question as to what to do with the `Ready` condition.
183This is addressed further below.
184
185### Condition polarity
186
187In terms of the polarity of conditions, we have three options, of which only two are
188really viable:
189* All conditions must be negative polarity
190* All conditions must be positive polarity
191* Some conditions can be positive polarity, but most should be negative.
192
193The fact that the user experience of `Ready` or conditions like `Accepted` being `true`
194in the healthy case is much better rules out the first option, so we are left to
195decide between enforcing that all conditions are positive, or that we have a mix.
196
197Having an arbitrary mix will make doing machine-based extraction of information
198much harder, so here I'm going to talk about the distinction between having all
199conditions positive or some, summary conditions positive, and the rest negative.
200
201#### All Conditions Positive
202
203In this case, all Condition types are written in such a way that they're positive
204polarity, and are `true` in the healthy case.
205
206As already discussed, `Ready`, and `Accepted` are current examples, but another
207one that's a little more important here is `ResolvedRefs` which is set to `true`
208when all references to other resources have been successfully resolved. This is
209not a _blocking_ Condition that affects the `Ready` condition, since having _some_
210references valid is enough to produce some configuration in the underlying data
211plane.
212
213So, All Conditions Positive pros:
214* We're close already. Most conditions in the API are currently positive polarity.
215* Easier to understand - there are no double negatives. "Good: true" is less
216cognitive overhead than "NotGood: false".
217
218Cons:
219* Reduces flexibility - it can surprisingly difficult to avoid double negatives for
220conditions that describe error states, as in general programmers are more used
221to reporting "something went wrong" than they are "everything's okay".
222
223Not sure if pro or con:
224* Leans the design towards favoring conditions always being present, since you
225can't be sure if everything is good unless you see `AllGood: true`. The absence
226of a positive-polarity condition implies that the condition could be false. This
227puts this option more in line with the API guidelines on this point.
228
229#### Some Conditions Positive
230
231In this case, only a limited set of summary conditions are positive, and the rest
232are negative.
233
234Pros:
235* Error states can be described with `Error: true` instead of `NoError: false`.
236* Negative polarity error conditions are more friendly to not being present (since
237absence of `Error: true` implies everything's okay).
238
239Cons:
240* Any code handling conditions will need a list of the positive ones, and will
241need to assume that any others are negative.
242
243#### Decision
244
245Gateway API conditions will be positive for conditions that describe the happy
246state of the object, which is currently `Accepted` and `ResolvedRefs`, and will
247also include the new `Programmed` condition, and the newly-Extended condition
248`Ready`. A separate set of negative-polarity Error conditions will be set on an
249object when they are true.
250
251
252### Should conditions always be added?
253
254Not all of them.
255
256Positive polarity Conditions that describe the desirable state of the object must
257always be set. These are currently `Accepted`, `ResolvedRefs`, and `Programmed`.
258Implementations that use `Ready` must also add it before programming the Route.
259
260### Partial validity and Conditions
261
262One of the trickiest parts of Gateway API objects is that it's very possible to
263end up with an object that has some parts with valid configuration and some that
264don't. We refer to this as _partial validity_, and communicating this via status
265conditions is difficult.
266
267The intent with the `Accepted` condition is that it serves as an indicator that
268_something_ is working, that _some traffic_ from what the config specifies will
269be routed as configured.
270
271At this time, we haven't added a "no errors at all present" Condition, choosing
272to have a "some config is working" condition, with specific errors to aid in
273finding the exact problem with the objects. We could conceivably add this later
274if users find `Accepted` insufficient, but we're erring on the side of having
275less positive Conditions for now.
276
277### New and Updated Conditions
278
279#### `Accepted`
280
281This GEP proposes replacing all conditions that indicate syntactic and semantic
282validity with one, `Accepted` condition type.
283
284That is, the proposal is to replace:
285
286* `Scheduled` on Gateway
287* `Detached` on Listener
288
289with `Accepted` in all these locations.
290
291GatewayClass and Route will maintain the `Accepted` condition.
292
293All of these conditions share the following meanings:
294
295* The resource has been accepted for processing by the controller
296* The resource is syntactically and semantically valid, and internally consistent
297* The resource fits into a larger system of Gateway API resources, and there is
298is no missing information, including but not limited to:
299 * Any mandatory references resolve to existing resources (examples here are the
300 Gateway's gatewayClass field, or the `parentRefs` field in Route resources)
301 * Any specified TLS secrets exist
302* The resource is supported by the controller by ensuring things like:
303 * Any Kinds being referred to by the resource are supported
304 * Features being used by the resource are supported
305
306All of these rules can be summarized into:
307
308* The resource is valid enough to produce some configuration in the underlying
309data plane.
310
311For Gateway, `Accepted` also subsumes the functions of `Scheduled`: `Accepted`
312set to `true` means that sufficient capacity exists on underlying infrastructure
313for the Gateway to be provisioned. If that capacity does not exist, then the
314Gateway cannot be reconciled successfully, and so fails to attach to the
315owning GatewayClass, and cannot be accepted.
316
317Note that some classes of inter-resource reference failure do _not_ cause a resource
318to become unattached and stop being accepted (that is, to have the `Accepted`
319condition set to `status: false`).
320
321* Non-existent Service backends - if the backend does not exist on a HTTPRoute that
322is otherwise okay, then the data plane must generate 500s for traffic that matches
323that HTTPRoute. In this case, the `Accepted` Condition must be true, and the
324`ResolvedRefs` Condition must be false, with reasons and messages indicating that
325the backend services do not exist.
326* HTTPRoutes with *all* backends in other namespaces, but not permitted by ReferenceGrants.
327In this case, the "non-existent service backends" rules apply, and 500s must be
328generated. In this case, again, the `Accepted` condition is true, and the
329`ResolvedRefs` Condition is false, with reasons and messages indicating that the
330backend services are not reachable.
331
332For ReferenceGrant or not-designed-yet Policy resources, `Accepted` means that:
333
334* the resource has a correctly-defined set of resources that it applies to
335* the resource has a syntactically and semantically valid `spec`
336
337Note that having a correctly-defined set of resources that is empty does not make
338these resources unattached, as long as it's possible to create some config in the
339underlying data plane. By "empty" here we mean that there are no backends,
340not that the config is incomplete or missing references. So you can have a
341GatewayClass, Gateway, HTTPRoute and Service all present and referred to correctly
342when there are no endpoints in the Service, and the resource will not stop being
343accepted, because HTTPRoute contains rules about what to program in the data plane
344if there are no endpoints (that is, it should return 500 for any matching request).
345
346Note that for other Route types that don't have a clear mechanism like HTTP does
347for indicating a server failure (like the HTTP code 500 does), not having existing
348backends may not produce any configuration in the data plane, and so may cause
349the resource to fail to attach. (An example here could be a TCP Route with
350no backends, we need to decide if that means that a port should be opened that
351actively closes connections, or if no port should be opened.)
352
353Examples of Conditions:
354
355* HTTPRoute with one match with one backend that is valid. `Accepted` is true,
356`ResolvedRefs` is true.
357* HTTPRoute with one match with one backend that is a non-existent Service backend.
358The `Accepted` Condition is true, the `ResolvedRefs` condition is false, with
359a reason of `BackendNotFound`. `Accepted` is true in this case because the data
360path must respond to requests that would be sent to that backend with a 500 response.
361* HTTPRoute with one match with two backends, one of which is a non-existent Service
362backend. The `Accepted` Condition is true, the `ResolvedRefs` condition is false.
363`Accepted` is true in this case because the data path must respond to a percentage
364of the requests matching the rule corresponding to the weighting of the non-existent
365backend (which would be fifty percent unless weights are applied).
366* HTTPRoute with one match with one backend that is in a different namespace, and
367does _not_ have a ReferenceGrant permitting that access. The `Accepted` condition
368is true, and the `ResolvedRefs` Condition is false, with a reason of `RefNotPermitted`.
369As before, `Accepted` is true because in this case, the data path must be
370programmed with 500s for the match.
371* TCPRoute with one match with a backend that is a non-existent Service. `Accepted`
372is false, and `ResolvedRefs` is false. `Accepted` is false in this case because
373there is not enough information to program any rules to handle the traffic in the
374underlying data plane - TCP doesn't have a way to say "this is a valid destination
375that has something wrong with it".
376* HTTPRoute with one Custom supported filter added that is not supported by the
377implementation. Our spec is currently unclear on what happens in this case, but
378custom HTTP Filters require the use of the `ExtensionRef` filter type, and the
379setting of the ExtensionRef field to the name, group, version, and kind of a
380custom resource that describes the filter. If that custom resource is not supported,
381it seems reasonable to say that this should be a reference failure, and be treated
382like other reference failures (`Accepted` will be set to true, `ResolvedRefs` to
383false with a `InvalidKind` Reason, and traffic that would have matched the filter
384should receive a 500 error.)
385* A HTTPRoute with one rule that specifies a HTTPRequestRedirect filter _and_ a
386HTTPURLRewrite filter. `Accepted` must be false, because there's only one rule,
387and this configuration for the rule is invalid (see [reference][httpreqredirect])
388The error condition in this case is undefined currently - we should define it,
389thanks @sunjayBhatia.
390* A HTTPRoute with two rules, one valid and one which specifies a HTTPRequestRedirect
391filter _and a HTTPURLRewrite filter. `Accepted` is true, because the valid rule
392can produce some config in the data plane. We'll need to raise the more specific
393error condition for an incompatible filter combination as well to make the partial
394validity clear.
395
396
397#### Ready
398
399Currently, the `Ready` condition text for Gateway says:
400```go
401 // This condition is true when the Gateway is expected to be able
402 // to serve traffic. Note that this does not indicate that the
403 // Gateway configuration is current or even complete (e.g. the
404 // controller may still not have reconciled the latest version,
405 // or some parts of the configuration could be missing).
406```
407
408This is pretty unclear - how can the Gateway serve traffic if config is missing?
409In the past, we've been asked to have a Condition that only flips to `true` when
410*all* required configuration is present.
411
412For many implementations (certainly for Envoy-based ones), getting this information
413correctly and avoiding races on applying it is surprisingly difficult.
414
415For this reason, this GEP proposes that we exclude the `Ready` condition from Core
416conformance, and make it a feature that implementations may opt in to - making it
417an Extended condition.
418
419It will have the following behavior:
420
421* `Ready` is an optional Condition that has Extended support, with conformance
422tests to verify the behavior.
423* When it's set, the condition indicates that traffic is ready to flow through
424the data plane _immediately_, not at some eventual point in the future.
425
426We'll need to add conformance testing for this.
427
428#### Programmed
429
430The `Programmed` condition is being added to replicate the functionality that the
431`Ready` condition currently indicates, namely that all the resources in the set
432are valid enough to produce some data plane configuration, and that configuration
433has been sent to the data plane, and should be ready soon.
434
435It is a positive-polarity summary condition, and so should always be present on
436the resource. It should be set to `Unknown` if the implementation performs updates
437to the status before it has all the information it needs to be able to determine
438if the condition is true.
439
440
441## Alternatives
442
443(Most alternatives have been discussed inline. Please comment here if this section
444needs updating.)
445
446## References
447[kep-status]: https://github.com/kubernetes/enhancements/blob/master/keps/NNNN-kep-template/kep.yaml#L9
448
449[1111]: https://github.com/kubernetes-sigs/gateway-api/issues/1111
450[1110]: https://github.com/kubernetes-sigs/gateway-api/issues/1110
451[1362]: https://github.com/kubernetes-sigs/gateway-api/issues/1362
452
453[typstatus]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties
454[httpreqredirect]: https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1beta1.HTTPRequestRedirectFilterView as plain text