1# dra-test-driver
2
3This driver implements the controller and a resource kubelet plugin for dynamic
4resource allocation. This is done in a single binary to minimize the amount of
5boilerplate code. "Real" drivers could also implement both in different
6binaries.
7
8## Usage
9
10The driver could get deployed as a Deployment for the controller, with leader
11election. A DaemonSet could get used for the kubelet plugin. The controller can
12also run as a Kubernetes client outside of a cluster. The same works for the
13kubelet plugin when using port forwarding. This is how it is used during
14testing.
15
16Valid parameters are key/value string pairs stored in a ConfigMap.
17Those get copied into the ResourceClaimStatus with "user_" and "admin_" as
18prefix, depending on whether they came from the ResourceClaim or ResourceClass.
19They get stored in the `ResourceHandle` field as JSON map by the controller.
20The kubelet plugin then sets these attributes as environment variables in each
21container that uses the resource.
22
23Resource availability is configurable and can simulate different scenarios:
24
25- Network-attached resources, available on all nodes where the node driver runs, or
26 host-local resources, available only on the node whether they were allocated.
27- Shared or unshared allocations.
28- Unlimited or limited resources. The limit is a simple number of allocations
29 per cluster or node.
30
31While the functionality itself is very limited, the code strives to showcase
32best practices and supports metrics, leader election, and the same logging
33options as Kubernetes.
34
35## Design
36
37The binary itself is a Cobra command with two operations, `controller` and
38`kubelet-plugin`. Logging is done with [contextual
39logging](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/3077-contextual-logging).
40
41The `k8s.io/dynamic-resource-allocation/controller` package implements the
42interaction with ResourceClaims. It is generic and relies on an interface to
43implement the actual driver logic. Long-term that part could be split out into
44a reusable utility package.
45
46The `k8s.io/dynamic-resource-allocation/kubelet-plugin` package implements the
47interaction with kubelet, again relying only on the interface defined for the
48kubelet<->dynamic resource allocation plugin interaction.
49
50`app` is the driver itself with a very simple implementation of the interfaces.
51
52## Deployment
53
54### `local-up-cluster.sh`
55
56To try out the feature, build Kubernetes, then in one console run:
57```console
58RUNTIME_CONFIG="resource.k8s.io/v1alpha2" FEATURE_GATES=DynamicResourceAllocation=true ALLOW_PRIVILEGED=1 ./hack/local-up-cluster.sh -O
59```
60
61In another:
62```console
63go run ./test/e2e/dra/test-driver --feature-gates ContextualLogging=true -v=5 controller
64```
65
66In yet another:
67```console
68sudo mkdir -p /var/run/cdi && sudo chmod a+rwx /var/run/cdi /var/lib/kubelet/plugins_registry
69go run ./test/e2e/dra/test-driver --feature-gates ContextualLogging=true -v=5 kubelet-plugin
70```
71
72And finally:
73```console
74$ kubectl create -f test/e2e/dra/test-driver/deploy/example/resourceclass.yaml
75resourceclass/example created
76$ kubectl create -f test/e2e/dra/test-driver/deploy/example/pod-inline.yaml
77configmap/pause-claim-parameters created
78pod/pause created
79
80$ kubectl get resourceclaims
81NAME CLASSNAME ALLOCATIONMODE STATE AGE
82pause-resource example WaitForFirstConsumer allocated,reserved 19s
83
84$ kubectl get pods
85NAME READY STATUS RESTARTS AGE
86pause 1/1 Running 0 23s
87```
88
89There are also examples for other scenarios (multiple pods, multiple claims).
90
91### multi-node cluster
92
93At this point there are no container images that contain the test driver and
94therefore it cannot be deployed on "normal" clusters.
95
96## Prior art
97
98Some of this code was derived from the
99[external-resizer](https://github.com/kubernetes-csi/external-resizer/). `controller`
100corresponds to the [controller
101logic](https://github.com/kubernetes-csi/external-resizer/blob/master/pkg/controller/controller.go),
102which in turn is similar to the
103[sig-storage-lib-external-provisioner](https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner).
View as plain text