1# Stackdriver Logging Agent
2==============
3
4Stackdriver Logging Agent is a DaemonSet which spawns a pod on each node
5that reads logs, generated by kubelet, container runtime and containers
6and sends them to the Stackdriver. When logs are exported to the Stackdriver,
7they can be searched, viewed, and analyzed.
8
9Learn more at: https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver
10
11## Troubleshooting
12
13In Kubernetes clusters in version 1.10.0 or later, fluentd-gcp DaemonSet can be
14manually scaled. This is useful e.g. when applications running in the cluster
15are sending a large volume of logs (i.e. over 100kB/s), causing fluentd-gcp to
16fail with OutOfMemory errors. Conversely, if the applications aren't generating
17a lot of logs, it may be useful to reduce the amount of resources consumed by
18fluentd-gcp, making these resources available to other applications. To learn
19more about Kubernetes resource requests and limits, see the official
20documentation ([CPU][cpu], [memory][memory]). The amount of resources requested
21by fluentd-gcp on every node in the cluster can be fetched by running following
22command:
23
24```
25$ kubectl get ds -n kube-system -l k8s-app=fluentd-gcp \
26-o custom-columns=NAME:.metadata.name,\
27CPU_REQUEST:.spec.template.spec.containers[].resources.requests.cpu,\
28MEMORY_REQUEST:.spec.template.spec.containers[].resources.requests.memory,\
29MEMORY_LIMIT:.spec.template.spec.containers[].resources.limits.memory
30```
31
32This will display an output similar to the following:
33
34```
35NAME CPU_REQUEST MEMORY_REQUEST MEMORY_LIMIT
36fluentd-gcp-v2.0.15 100m 200Mi 300Mi
37```
38
39In order to change those values, a [ScalingPolicy][scalingPolicy] needs to be
40defined. Currently, only base values are supported (no automatic scaling). The
41ScalingPolicy can be created using kubectl. E.g. to set cpu request to 101m,
42memory request to 150Mi and memory limit to 400Mi:
43
44```
45$ cat <<EOF | kubectl apply -f -
46apiVersion: scalingpolicy.kope.io/v1alpha1
47kind: ScalingPolicy
48metadata:
49 name: fluentd-gcp-scaling-policy
50 namespace: kube-system
51spec:
52 containers:
53 - name: fluentd-gcp
54 resources:
55 requests:
56 - resource: cpu
57 base: 101m
58 - resource: memory
59 base: 150Mi
60 limits:
61 - resource: memory
62 base: 400Mi
63EOF
64```
65
66To remove the override and go back to GKE-provided defaults, it is enough to
67just remove the ScalingPolicy:
68
69```
70$ kubectl delete -n kube-system scalingpolicies.scalingpolicy.kope.io/fluentd-gcp-scaling-policy
71```
72
73[cpu]: https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/
74[memory]: https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/
75[scalingPolicy]: https://github.com/justinsb/scalerView as plain text