1## Purpose
2
3This repository contains the definitions for the Container Runtime Interface (CRI).
4CRI is a plugin interface which enables kubelet to use a wide variety of container runtimes,
5without the need to recompile. CRI consists of a protocol buffers and gRPC API.
6Read more about CRI API at [kubernetes docs](https://kubernetes.io/docs/concepts/architecture/cri/).
7
8The repository [kubernetes/cri-api](https://github.com/kubernetes/cri-api) is a mirror of https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/cri-api.
9Please do **not** file issues or submit PRs against the [kubernetes/cri-api](https://github.com/kubernetes/cri-api)
10repository as it is readonly, all development is done in [kubernetes/kubernetes](https://github.com/kubernetes/kubernetes).
11
12The CRI API is defined in [kubernetes/kubernetes](https://github.com/kubernetes/kubernetes)
13repository and is **only** intended to be used for kubelet to container runtime
14interactions, or for node-level troubleshooting using a tool such as `crictl`.
15It is **not** a common purpose container runtime API for general use, and is intended
16to be Kubernetes-centric. We try to avoid it, but there may be logic within a container
17runtime that optimizes for the order or specific parameters of call(s) that the kubelet
18makes.
19
20## Version skew policy
21
22On a single Node there may be installed multiple components implementing
23different versions of CRI API.
24
25For example, on a single node there might be:
26
27- _Kubelet_ may call into _Container Runtime_ (e.g. [containerd](https://containerd.io))
28 and _Image Service Proxy_ (e.g. [stargz-snapshotter](https://github.com/containerd/stargz-snapshotter)).
29 _Container Runtime_ may be versioned with the OS Image, _Kubelet_ is installed
30 by system administrator and _Image Service proxy_ is versioned by the third party vendor.
31- _Image Service Proxy_ calls into _Container Runtime_.
32- _CRI tools_ (e.g. [crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/))
33 may be installed by end user to troubleshoot, same as a third party daemonsets.
34 All of them are used to call into the _Container Runtime_ to collect container information.
35
36So on a single node it may happen that _Container Runtime_ is serving a newer
37version'd kubelet and older versioned crictl. This is a supported scenario within
38the version skew policy.
39
40### Version Skew Policy for CRI API
41
42CRI API has two versions:
43- Major semantic version (known versions are `v1alpha2` ([removed in 1.26](https://kubernetes.io/blog/2022/12/09/kubernetes-v1-26-release/#cri-v1alpha2-removed)), `v1`).
44- Kubernetes version (for example: `@1.23`). Note, the `cri-api` Golang library is versioned as `0.23` as it doesn't guarantee Go types backward compatibility.
45
46Major semantic version (e.g. `v1`) is used to introduce breaking changes
47and major new features that are incompatible with the current API.
48
49Kubernetes version is used to indicate a specific feature set implemented
50on top of the major semantic version. All changes made without the change
51of a major semantic version API must be backward and forward compatible.
52
53- _Kubelet_ must work with the older _Container Runtime_ if it implements
54 the same semantic version of CRI API (e.g. `v1`) of up to three Kubernetes minor
55 versions back. New features implemented in CRI API must be gracefully degraded.
56 For example, _Kubelet_ of version 1.26 must work with _Container Runtime_
57 implementing `k8s.io/cri-api@v0.23.0`+.
58- _Kubelet_ must work with _Container Runtime_ if it implements
59 the same semantic version of CRI API (e.g. `v1`) of up to
60 three minor versions up. New features implemented in CRI API must not change
61 behavior of old method calls and response values. For example, _Kubelet_ of
62 version 1.22 must work with _Container Runtime_ implementing `k8s.io/cri-api@v0.25.5`.
63
64
65## Versioning
66
67This library contains go classes generated from the CRI API protocol buffers and gRPC API.
68
69The library versioned as `0.XX` as Kubernetes doesn't provide any guarantees
70on backward compatibility of Go wrappers between versions. However CRI API itself
71(protocol buffers and gRPC API) is marked as stable `v1` version and it is
72backward compatible between versions.
73
74Versions like `v0.<minor>.<patch>` (e.g. `v0.25.5`) are considered stable.
75It is discouraged to introduce CRI API changes in patch releases and recommended
76to use versions like `v0.<minor>.0`.
77
78All alpha and beta versions (e.g. `k8s.io/cri-api@v0.26.0-beta.0`) should be
79backward and forward compatible.
80
81## Feature development
82
83Some features development requires changes in CRI API and corresponding changes
84in _Container Runtime_. Coordinating between Kubernetes branches and release
85versions and _Container Runtime_ versions is not always trivial.
86
87The recommended feature development flow is following:
88
89- Review proposed CRI API changes during the KEP review stage.
90 Some field names and types may not be spelled out exactly at this stage.
91- Locally implement a prototype that implement changes in both - Kubernetes and Container Runtime.
92- Submit a Pull Request for Kubernetes implementing CRI API changes alongside the feature code.
93 Feature must be developed to degrade gracefully when used with older Container Runtime
94 according to the Version Skew policy.
95- Once PR is merged, wait for the next Kubernetes release tag being produced.
96 Find the corresponding CRI API tag (e.g. `k8s.io/cri-api@v0.26.0-beta.0`).
97- This tag can be used to implement the feature in Container Runtime. It is recommended
98 to switch to the stable tag like (`k8s.io/cri-api@v0.26.0`) once available.
99
100## Change history
101
102Here is the change history of the Container Runtime Interface protocol:
103
104### v1.20
105
106`git diff v1.19.0 v1.20.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto`
107
108- CRI [v1 introduced](https://github.com/kubernetes/kubernetes/pull/96387)
109
110### v1.21
111
112`git diff v1.20.0 v1.21.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto`
113
114No changes
115
116### v1.22
117
118`git diff v1.21.0 v1.22.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto`
119
120- [Windows host process support](https://github.com/kubernetes/kubernetes/pull/99576)
121 - `PodSandboxConfig` has `windows` field of type `WindowsPodSandboxConfig`
122 - New type `WindowsPodSandboxConfig` introduced
123 - New type `WindowsSandboxSecurityContext` introduced
124 - The type `WindowsContainerSecurityContext` has a new `host_process` boolean field
125
126- [Feature: add unified on CRI to support cgroup v2](https://github.com/kubernetes/kubernetes/pull/102578)
127 - The type `LinuxContainerResources` has a new field `unified` which is a map of strings
128
129- [Alpha node swap support](https://github.com/kubernetes/kubernetes/pull/102823)
130 - The type `LinuxContainerResources` has a new `memory_swap_limit_in_bytes` int64 field
131
132### v1.23
133
134`git diff v1.22.0 v1.23.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto`
135
136- [CRI: add fields for pod level stats to satisfy the /stats/summary API](https://github.com/kubernetes/kubernetes/pull/102789)
137 - New functions `PodSandboxStats`, `ListPodSandboxStats` with the corresponding types of request and response objects are introduced
138
139- [pass sandbox resource requirements over CRI](https://github.com/kubernetes/kubernetes/pull/104886)
140 - New fields on `LinuxPodSandboxConfig`: `overhead` and `resources` of type `LinuxContainerResources`.
141
142- [prevents garbage collection from removing pinned images](https://github.com/kubernetes/kubernetes/pull/103299)
143 - The type `Image` has a new boolean field `pinned`
144
145### v1.24
146
147`git diff v1.23.0 v1.24.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto`
148
149- [Update CRI-API Capabilities to include a field that allows us to set ambient capabilities](https://github.com/kubernetes/kubernetes/pull/104620)
150 - The type `Capability` has a new string field `add_ambient_capabilities`
151
152- [CRI-API - Add rootfs size to WindowsContainerResources](https://github.com/kubernetes/kubernetes/pull/108894)
153 - The type `WindowsContainerResources` has a new int64 field `rootfs_size_in_bytes`
154
155### v1.25
156
157`git diff v1.24.0 v1.25.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto`
158
159
160- [kubelet: add CRI definitions for user namespaces](https://github.com/kubernetes/kubernetes/pull/110535)
161 - The new type `UserNamespace` introduced to represent user namespaces id mapping
162 - The type `NamespaceOption` has a new field `userns_options` of type `UserNamespace`
163
164- [Minimal checkpointing support](https://github.com/kubernetes/kubernetes/pull/104907)
165 - The new method `CheckpointContainer` introduced with the corresponding request and response types
166
167- [Update CRI API to support Evented PLEG](https://github.com/kubernetes/kubernetes/pull/111642)
168 - The new streaming method `GetContainerEvents` is introduced with the corresponding request and response types
169
170- [CRI changes to support in-place pod resize](https://github.com/kubernetes/kubernetes/pull/111645)
171 - The new type `ContainerResources` is introduced
172 - The type `ContainerStatus` has a new field `resources` of type `ContainerResources`
173 - The semantic of `UpdateContainerResources` updated. The method must be implemented as synchronous and return error on failure
174
175### v1.26
176
177`git diff v1.25.0 v1.26.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto`
178
179- [CRI: Add Windows Podsandbox Stats](https://github.com/kubernetes/kubernetes/pull/110754)
180 - Added fields to the type `WindowsPodSandboxStats` expressing stats required to be collected from windows pods.
181
182- [Windows hostnetwork alpha](https://github.com/kubernetes/kubernetes/pull/112961)
183 - New type `WindowsNamespaceOption` introduced
184 - The type `WindowsSandboxSecurityContext` has a new field `namespace_options` of type `WindowsNamespaceOption`
185
186- [Improve the API description of `PodSecurityContext.SupplementalGroups` to clarify its unfamiliar behavior](https://github.com/kubernetes/kubernetes/pull/113047)
187 - Clarified the expected behavior of `SupplementalGroups` field of `PodSecurityContext`
188
189- [Add Support for Evented PLEG](https://github.com/kubernetes/kubernetes/pull/111384)
190 - The type `ContainerEventResponse` updated: the field `pod_sandbox_metadata` removed and fields `pod_sandbox_status` and `containers_statuses` added.
191 - The type `PodSandboxStatusResponse` has a new fields `containers_statuses` and `timestamp`
192
193### v1.27
194
195`git diff v1.26.0 v1.27.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto`
196
197- [CRI: Add CDI device info for containers](https://github.com/kubernetes/kubernetes/pull/115891/)
198 - New type `CDIDevice` was introduced and added to container config
199
200- [Add mappings for volumes](https://github.com/kubernetes/kubernetes/pull/116377)
201 - Added new fields to the type `Mount` expressing runtime UID/GID mappings for the mount.
202
203### v1.28
204
205`git diff v1.27.0 v1.28.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto`
206
207- [cri-api: fix comment lines about PROPAGATION_PRIVATE](https://github.com/kubernetes/kubernetes/pull/115704)
208 - Fixed comment lines about PROPAGATION_PRIVATE
209
210- [Add user specified image to CRI ContainerConfig](https://github.com/kubernetes/kubernetes/pull/118652)
211 - Added the `user_specified_image` field to type `ImageSpec`
212
213- [kubelet: get cgroup driver config from CRI ](https://github.com/kubernetes/kubernetes/pull/118770)
214 - Added rpc for querying runtime configuration
215 - Added cavieats about cgroup driver field
216
217- [Add swap to stats to Summary API and Prometheus endpoints (/stats/summary and /metrics/resource)](https://github.com/kubernetes/kubernetes/pull/118865)
218 - Added `SwapUsage` type
219 - Added `SwapUsage` field to `ContainerStats` type
220
221- [Expose commit memory used in WindowsMemoryUsage struct](https://github.com/kubernetes/kubernetes/pull/119238)
222 - Added the `commit_memory_bytes` field to type `WindowsMemoryUsage`
223
224### v1.29
225
226`git diff v1.28.0 v1.29.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto`
227
228- [Add runtime handler field to ImageSpec struct](https://github.com/kubernetes/kubernetes/pull/121121)
229 - Added `runtime_handler` field to type `ImageSpec`
230
231## Community, discussion, contribution, and support
232
233Learn how to engage with the Kubernetes community on the [community
234page](http://kubernetes.io/community/).
235
236You can reach the maintainers of this repository at:
237
238- Slack: #sig-node (on https://kubernetes.slack.com -- get an
239 invite at [slack.kubernetes.io](https://slack.kubernetes.io))
240- Mailing List:
241 https://groups.google.com/forum/#!forum/kubernetes-sig-node
242
243### Code of Conduct
244
245Participation in the Kubernetes community is governed by the [Kubernetes
246Code of Conduct](code-of-conduct.md).
247
248### Contribution Guidelines
249
250See [CONTRIBUTING.md](CONTRIBUTING.md) for more information. Please note that [kubernetes/cri-api](https://github.com/kubernetes/cri-api)
251is a readonly mirror repository, all development is done at [kubernetes/kubernetes](https://github.com/kubernetes/kubernetes).
View as plain text