1# Starting a Windows Kubernetes cluster on GCE using kube-up
2
3## IMPORTANT PLEASE NOTE
4
5Any time the file structure in the `windows` directory changes, `windows/BUILD`
6and `k8s.io/release/lib/releaselib.sh` must be manually updated with the
7changes. We HIGHLY recommend not changing the file structure, because consumers
8of Kubernetes releases depend on the release structure remaining stable.
9
10## Bring up the cluster
11
12Prerequisites: a Google Cloud Platform project.
13
14### 0. Prepare your environment
15
16Clone this repository under your `$GOPATH/src` directory on a Linux machine.
17Then, optionally clean/prepare your environment using these commands:
18
19```bash
20# Remove files that interfere with get-kube / kube-up:
21rm -rf ./kubernetes/; rm -f kubernetes.tar.gz; rm -f ~/.kube/config
22
23# To run e2e test locally, make sure "Application Default Credentials" is set in any of the places:
24# References: https://cloud.google.com/sdk/docs/authorizing#authorizing_with_a_service_account
25# https://cloud.google.com/sdk/gcloud/reference/auth/application-default/
26# 1. $HOME/.config/gcloud/application_default_credentials.json, if doesn't exist, run this command:
27gcloud auth application-default login
28# Or 2. Create a json format credential file as per http://cloud/docs/authentication/production,
29# then export to environment variable
30export GOOGLE_APPLICATION_CREDENTIAL=[path_to_the_json_file]
31```
32
33### 1. Build Kubernetes
34
35NOTE: this step is only needed if you want to test local changes you made to the
36codebase.
37
38The most straightforward approach to build those binaries is to run `make
39release`. However, that builds binaries for all supported platforms, and can be
40slow. You can speed up the process by following the instructions below to only
41build the necessary binaries.
42
43```bash
44# Build binaries for both Linux and Windows:
45KUBE_BUILD_PLATFORMS="linux/amd64 windows/amd64" make quick-release
46```
47
48### 2. Create a Kubernetes cluster
49
50You can create a regular Kubernetes cluster or an end-to-end test cluster.
51
52Only end-to-end test clusters support running the Kubernetes e2e tests (as both
53[e2e cluster creation](https://github.com/kubernetes/kubernetes/blob/b632eaddbaad9dc1430d214d506b72750bbb9f69/hack/e2e-internal/e2e-up.sh#L24)
54and
55[e2e test scripts](https://github.com/kubernetes/kubernetes/blob/b632eaddbaad9dc1430d214d506b72750bbb9f69/hack/ginkgo-e2e.sh#L42)
56are setup based on `cluster/gce/config-test.sh`), also enables some debugging
57features such as SSH access on the Windows nodes.
58
59Please make sure you set the environment variables properly following the
60instructions in the previous section.
61
62First, set the following environment variables which are required for
63controlling the number of Linux and Windows nodes in the cluster and for
64enabling IP aliases (which are required for Windows pod routing). At least one
65Linux worker node is required and two are recommended because many default
66cluster-addons (e.g., `kube-dns`) need to run on Linux nodes. The master control
67plane only runs on Linux.
68
69```bash
70export NUM_NODES=2 # number of Linux nodes
71export NUM_WINDOWS_NODES=2
72export KUBE_GCE_ENABLE_IP_ALIASES=true
73export KUBERNETES_NODE_PLATFORM=windows
74export LOGGING_STACKDRIVER_RESOURCE_TYPES=new
75```
76
77Now bring up a cluster using one of the following two methods:
78
79#### 2a. Create a regular Kubernetes cluster
80
81Ensure your GCP authentication is current:
82
83```bash
84gcloud auth application-default login
85gcloud auth login
86```
87
88Invoke kube-up.sh with these environment variables:
89
90```bash
91# WINDOWS_NODE_OS_DISTRIBUTION: the Windows version you want your nodes to
92# run, e.g. win2019 or win1909.
93# KUBE_UP_AUTOMATIC_CLEANUP (optional): cleans up existing cluster without
94# prompting.
95WINDOWS_NODE_OS_DISTRIBUTION=win2019 KUBE_UP_AUTOMATIC_CLEANUP=true ./cluster/kube-up.sh
96```
97
98If your GCP project is configured with two-factor authentication, you may need
99to tap your security key shortly after running `kube-up`.
100
101To teardown the cluster run:
102
103```bash
104./cluster/kube-down.sh
105```
106
107If you want to run more than one cluster simultaneously, you can use two
108separate GCP projects and:
109
1101. Use a separate shell for each project / cluster.
1111. Set the `CLOUDSDK_CORE_PROJECT` environment variable to the GCP project you
112 want to use in each shell. This variable will override your current gcloud
113 config.
1141. Prefix your `kube-up.sh` and `kube-down.sh` commands with
115 `PROJECT=${CLOUDSDK_CORE_PROJECT}`
116
117#### 2b. Create a Kubernetes end-to-end (E2E) test cluster
118
119If you have built your own release binaries following step 1, run the following
120command to bring up a cluster for running the K8s e2e tests. See the
121[windows-gce](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-windows/windows-gce.yaml)
122e2e test configuration for the latest environment variables.
123
124```bash
125KUBE_GCE_ENABLE_IP_ALIASES=true KUBERNETES_NODE_PLATFORM=windows \
126 LOGGING_STACKDRIVER_RESOURCE_TYPES=new NUM_NODES=2 \
127 NUM_WINDOWS_NODES=3 WINDOWS_NODE_OS_DISTRIBUTION=win2019 \
128 ./hack/e2e-internal/e2e-up.sh
129```
130
131If any e2e cluster exists already, this command will prompt you to tear down and
132create a new one. To teardown existing e2e cluster only, run the command:
133
134```bash
135./hack/e2e-internal/e2e-down.sh
136```
137
138No matter what type of cluster you chose to create, the result should be a
139Kubernetes cluster with one Linux master node, `NUM_NODES` Linux worker nodes
140and `NUM_WINDOWS_NODES` Windows worker nodes.
141
142## Validating the cluster
143
144Invoke this script to run a smoke test that verifies that the cluster has been
145brought up correctly:
146
147```bash
148cluster/gce/windows/smoke-test.sh
149```
150
151Sometimes the first run of the smoke test will fail because it took too long to
152pull the Windows test containers. The smoke test will usually pass on the next
153attempt.
154
155## Running e2e tests against the cluster
156
157If you brought up an end-to-end test cluster using the steps above then you can
158use the steps below to run K8s e2e tests. These steps are based on
159[kubernetes-sigs/windows-testing](https://github.com/kubernetes-sigs/windows-testing).
160
161* Build the necessary test binaries. This must be done after every change to
162 test code.
163
164 ```bash
165 make WHAT=test/e2e/e2e.test
166 ```
167
168* Set necessary environment variables and fetch the `run-e2e.sh` script:
169
170 ```bash
171 export KUBECONFIG=~/.kube/config
172 export WORKSPACE=$(pwd)
173 export ARTIFACTS=${WORKSPACE}/e2e-artifacts
174
175 curl \
176 https://raw.githubusercontent.com/kubernetes-sigs/windows-testing/master/gce/run-e2e.sh \
177 -o ${WORKSPACE}/run-e2e.sh
178 chmod u+x run-e2e.sh
179
180 # Fetch a prepull manifest for the k8s version you're using.
181 curl \
182 https://raw.githubusercontent.com/kubernetes-sigs/windows-testing/master/gce/prepull-1.21.yaml \
183 -o ${WORKSPACE}/prepull-head.yaml
184 ```
185
186 The e2e test scripts make some annoying assumptions about the path to the
187 k8s repository. If your `~/go/src/k8s.io/kubernetes` directory is actually
188 a symlink to `~/go/src/github.com/<username>/kubernetes`, create this
189 additional symlink:
190
191 ```bash
192 cd ~/go/src/github.com; ln -s . github.com
193 ```
194
195 Without this additional symlink you may receive this error when invoking
196 the `run-e2e.sh` script:
197
198 ```bash
199 chdir ../../github.com/<username>/kubernetes/_output/bin: no such file or directory
200 ```
201
202* The canonical arguments for running all Windows e2e tests against a cluster
203 on GCE can be seen by searching for `--test-cmd-args` in the
204 [test configuration](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes/sig-windows/windows-gce.yaml#L78)
205 for the `ci-kubernetes-e2e-windows-gce` continuous test job. These arguments
206 should be passed to the `run-e2e` script; escape the ginkgo arguments by
207 adding quotes around them. For example:
208
209 ```bash
210 ./run-e2e.sh --node-os-distro=windows --minStartupPods=8 \
211 --ginkgo.focus="\[Conformance\]|\[NodeConformance\]|\[sig-windows\]" \
212 --ginkgo.skip="\[LinuxOnly\]|\[Serial\]|\[Feature:.+\]" \
213 --ginkgo.parallel.total=8 # TODO: does this flag actually help?
214 ```
215
216 If you get auth errors, you may need to re-authenticate:
217
218 ```bash
219 gcloud auth application-default login
220 gcloud auth login
221 ```
222
223* Run a single test by setting the ginkgo focus to match your test name; for
224 example, the "DNS should provide DNS for the cluster" test can be run using:
225
226 ```bash
227 ./run-e2e.sh --node-os-distro=windows \
228 --ginkgo.focus="provide\sDNS\sfor\sthe\scluster"
229 ```
230
231 Make sure to always include `--node-os-distro=windows` for testing against
232 Windows nodes.
233
234After the test run completes, log files can be found under the `${ARTIFACTS}`
235directory.
236
237## E2E Testing
238
239Once you've created a pull request you can comment, `/test
240pull-kubernetes-e2e-windows-gce` to run the integration tests that cover the
241changes in this directory.
View as plain text