1# Go gRPC Interceptors for Prometheus monitoring
2
3[](https://travis-ci.org/grpc-ecosystem/go-grpc-prometheus)
4[](http://goreportcard.com/report/grpc-ecosystem/go-grpc-prometheus)
5[](https://godoc.org/github.com/grpc-ecosystem/go-grpc-prometheus)
6[](https://sourcegraph.com/github.com/grpc-ecosystem/go-grpc-prometheus/?badge)
7[](https://codecov.io/gh/grpc-ecosystem/go-grpc-prometheus)
8[](LICENSE)
9
10[Prometheus](https://prometheus.io/) monitoring for your [gRPC Go](https://github.com/grpc/grpc-go) servers and clients.
11
12A sister implementation for [gRPC Java](https://github.com/grpc/grpc-java) (same metrics, same semantics) is in [grpc-ecosystem/java-grpc-prometheus](https://github.com/grpc-ecosystem/java-grpc-prometheus).
13
14## Interceptors
15
16[gRPC Go](https://github.com/grpc/grpc-go) recently acquired support for Interceptors, i.e. middleware that is executed
17by a gRPC Server before the request is passed onto the user's application logic. It is a perfect way to implement
18common patterns: auth, logging and... monitoring.
19
20To use Interceptors in chains, please see [`go-grpc-middleware`](https://github.com/mwitkow/go-grpc-middleware).
21
22## Usage
23
24There are two types of interceptors: client-side and server-side. This package provides monitoring Interceptors for both.
25
26### Server-side
27
28```go
29import "github.com/grpc-ecosystem/go-grpc-prometheus"
30...
31 // Initialize your gRPC server's interceptor.
32 myServer := grpc.NewServer(
33 grpc.StreamInterceptor(grpc_prometheus.StreamServerInterceptor),
34 grpc.UnaryInterceptor(grpc_prometheus.UnaryServerInterceptor),
35 )
36 // Register your gRPC service implementations.
37 myservice.RegisterMyServiceServer(s.server, &myServiceImpl{})
38 // After all your registrations, make sure all of the Prometheus metrics are initialized.
39 grpc_prometheus.Register(myServer)
40 // Register Prometheus metrics handler.
41 http.Handle("/metrics", promhttp.Handler())
42...
43```
44
45### Client-side
46
47```go
48import "github.com/grpc-ecosystem/go-grpc-prometheus"
49...
50 clientConn, err = grpc.Dial(
51 address,
52 grpc.WithUnaryInterceptor(grpc_prometheus.UnaryClientInterceptor),
53 grpc.WithStreamInterceptor(grpc_prometheus.StreamClientInterceptor)
54 )
55 client = pb_testproto.NewTestServiceClient(clientConn)
56 resp, err := client.PingEmpty(s.ctx, &myservice.Request{Msg: "hello"})
57...
58```
59
60# Metrics
61
62## Labels
63
64All server-side metrics start with `grpc_server` as Prometheus subsystem name. All client-side metrics start with `grpc_client`. Both of them have mirror-concepts. Similarly all methods
65contain the same rich labels:
66
67 * `grpc_service` - the [gRPC service](http://www.grpc.io/docs/#defining-a-service) name, which is the combination of protobuf `package` and
68 the `grpc_service` section name. E.g. for `package = mwitkow.testproto` and
69 `service TestService` the label will be `grpc_service="mwitkow.testproto.TestService"`
70 * `grpc_method` - the name of the method called on the gRPC service. E.g.
71 `grpc_method="Ping"`
72 * `grpc_type` - the gRPC [type of request](http://www.grpc.io/docs/guides/concepts.html#rpc-life-cycle).
73 Differentiating between the two is important especially for latency measurements.
74
75 - `unary` is single request, single response RPC
76 - `client_stream` is a multi-request, single response RPC
77 - `server_stream` is a single request, multi-response RPC
78 - `bidi_stream` is a multi-request, multi-response RPC
79
80
81Additionally for completed RPCs, the following labels are used:
82
83 * `grpc_code` - the human-readable [gRPC status code](https://github.com/grpc/grpc-go/blob/master/codes/codes.go).
84 The list of all statuses is to long, but here are some common ones:
85
86 - `OK` - means the RPC was successful
87 - `IllegalArgument` - RPC contained bad values
88 - `Internal` - server-side error not disclosed to the clients
89
90## Counters
91
92The counters and their up to date documentation is in [server_reporter.go](server_reporter.go) and [client_reporter.go](client_reporter.go)
93the respective Prometheus handler (usually `/metrics`).
94
95For the purpose of this documentation we will only discuss `grpc_server` metrics. The `grpc_client` ones contain mirror concepts.
96
97For simplicity, let's assume we're tracking a single server-side RPC call of [`mwitkow.testproto.TestService`](examples/testproto/test.proto),
98calling the method `PingList`. The call succeeds and returns 20 messages in the stream.
99
100First, immediately after the server receives the call it will increment the
101`grpc_server_started_total` and start the handling time clock (if histograms are enabled).
102
103```jsoniq
104grpc_server_started_total{grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 1
105```
106
107Then the user logic gets invoked. It receives one message from the client containing the request
108(it's a `server_stream`):
109
110```jsoniq
111grpc_server_msg_received_total{grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 1
112```
113
114The user logic may return an error, or send multiple messages back to the client. In this case, on
115each of the 20 messages sent back, a counter will be incremented:
116
117```jsoniq
118grpc_server_msg_sent_total{grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 20
119```
120
121After the call completes, its status (`OK` or other [gRPC status code](https://github.com/grpc/grpc-go/blob/master/codes/codes.go))
122and the relevant call labels increment the `grpc_server_handled_total` counter.
123
124```jsoniq
125grpc_server_handled_total{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 1
126```
127
128## Histograms
129
130[Prometheus histograms](https://prometheus.io/docs/concepts/metric_types/#histogram) are a great way
131to measure latency distributions of your RPCs. However, since it is bad practice to have metrics
132of [high cardinality](https://prometheus.io/docs/practices/instrumentation/#do-not-overuse-labels)
133the latency monitoring metrics are disabled by default. To enable them please call the following
134in your server initialization code:
135
136```jsoniq
137grpc_prometheus.EnableHandlingTimeHistogram()
138```
139
140After the call completes, its handling time will be recorded in a [Prometheus histogram](https://prometheus.io/docs/concepts/metric_types/#histogram)
141variable `grpc_server_handling_seconds`. The histogram variable contains three sub-metrics:
142
143 * `grpc_server_handling_seconds_count` - the count of all completed RPCs by status and method
144 * `grpc_server_handling_seconds_sum` - cumulative time of RPCs by status and method, useful for
145 calculating average handling times
146 * `grpc_server_handling_seconds_bucket` - contains the counts of RPCs by status and method in respective
147 handling-time buckets. These buckets can be used by Prometheus to estimate SLAs (see [here](https://prometheus.io/docs/practices/histograms/))
148
149The counter values will look as follows:
150
151```jsoniq
152grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.005"} 1
153grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.01"} 1
154grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.025"} 1
155grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.05"} 1
156grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.1"} 1
157grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.25"} 1
158grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="0.5"} 1
159grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="1"} 1
160grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="2.5"} 1
161grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="5"} 1
162grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="10"} 1
163grpc_server_handling_seconds_bucket{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream",le="+Inf"} 1
164grpc_server_handling_seconds_sum{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 0.0003866430000000001
165grpc_server_handling_seconds_count{grpc_code="OK",grpc_method="PingList",grpc_service="mwitkow.testproto.TestService",grpc_type="server_stream"} 1
166```
167
168
169## Useful query examples
170
171Prometheus philosophy is to provide raw metrics to the monitoring system, and
172let the aggregations be handled there. The verbosity of above metrics make it possible to have that
173flexibility. Here's a couple of useful monitoring queries:
174
175
176### request inbound rate
177```jsoniq
178sum(rate(grpc_server_started_total{job="foo"}[1m])) by (grpc_service)
179```
180For `job="foo"` (common label to differentiate between Prometheus monitoring targets), calculate the
181rate of requests per second (1 minute window) for each gRPC `grpc_service` that the job has. Please note
182how the `grpc_method` is being omitted here: all methods of a given gRPC service will be summed together.
183
184### unary request error rate
185```jsoniq
186sum(rate(grpc_server_handled_total{job="foo",grpc_type="unary",grpc_code!="OK"}[1m])) by (grpc_service)
187```
188For `job="foo"`, calculate the per-`grpc_service` rate of `unary` (1:1) RPCs that failed, i.e. the
189ones that didn't finish with `OK` code.
190
191### unary request error percentage
192```jsoniq
193sum(rate(grpc_server_handled_total{job="foo",grpc_type="unary",grpc_code!="OK"}[1m])) by (grpc_service)
194 /
195sum(rate(grpc_server_started_total{job="foo",grpc_type="unary"}[1m])) by (grpc_service)
196 * 100.0
197```
198For `job="foo"`, calculate the percentage of failed requests by service. It's easy to notice that
199this is a combination of the two above examples. This is an example of a query you would like to
200[alert on](https://prometheus.io/docs/alerting/rules/) in your system for SLA violations, e.g.
201"no more than 1% requests should fail".
202
203### average response stream size
204```jsoniq
205sum(rate(grpc_server_msg_sent_total{job="foo",grpc_type="server_stream"}[10m])) by (grpc_service)
206 /
207sum(rate(grpc_server_started_total{job="foo",grpc_type="server_stream"}[10m])) by (grpc_service)
208```
209For `job="foo"` what is the `grpc_service`-wide `10m` average of messages returned for all `
210server_stream` RPCs. This allows you to track the stream sizes returned by your system, e.g. allows
211you to track when clients started to send "wide" queries that ret
212Note the divisor is the number of started RPCs, in order to account for in-flight requests.
213
214### 99%-tile latency of unary requests
215```jsoniq
216histogram_quantile(0.99,
217 sum(rate(grpc_server_handling_seconds_bucket{job="foo",grpc_type="unary"}[5m])) by (grpc_service,le)
218)
219```
220For `job="foo"`, returns an 99%-tile [quantile estimation](https://prometheus.io/docs/practices/histograms/#quantiles)
221of the handling time of RPCs per service. Please note the `5m` rate, this means that the quantile
222estimation will take samples in a rolling `5m` window. When combined with other quantiles
223(e.g. 50%, 90%), this query gives you tremendous insight into the responsiveness of your system
224(e.g. impact of caching).
225
226### percentage of slow unary queries (>250ms)
227```jsoniq
228100.0 - (
229sum(rate(grpc_server_handling_seconds_bucket{job="foo",grpc_type="unary",le="0.25"}[5m])) by (grpc_service)
230 /
231sum(rate(grpc_server_handling_seconds_count{job="foo",grpc_type="unary"}[5m])) by (grpc_service)
232) * 100.0
233```
234For `job="foo"` calculate the by-`grpc_service` fraction of slow requests that took longer than `0.25`
235seconds. This query is relatively complex, since the Prometheus aggregations use `le` (less or equal)
236buckets, meaning that counting "fast" requests fractions is easier. However, simple maths helps.
237This is an example of a query you would like to alert on in your system for SLA violations,
238e.g. "less than 1% of requests are slower than 250ms".
239
240
241## Status
242
243This code has been used since August 2015 as the basis for monitoring of *production* gRPC micro services at [Improbable](https://improbable.io).
244
245## License
246
247`go-grpc-prometheus` is released under the Apache 2.0 license. See the [LICENSE](LICENSE) file for details.
View as plain text