1Metrics and logging
2===
3
4The `kgo` package supports both metrics and logging through options. By
5default, both are disabled.
6
7## Logging
8
9The [`WithLogger`][1] option can be used to enable internal logging. The
10default [`Logger`][2] interface is small but easy to implement, and there is a
11simple [`BasicLogger`][3] provided that you can use while developing or even in
12production. In production, it is recommended that you use a more "real" logger
13such as [zap][4], and to aid this, the franz-go repo provides a drop-in
14[`kzap`][5]. There also exists a drop-in [`zerolog`][6] [`kzerolog`][7]
15package. If you have another relatively standard logger that would be good to
16provide a drop-in package for, please open an issue and we can add it. It is
17recommended to use an info logging level: if you find that too noisy, please
18open an issue and we can figure out if some logs need to be changed.
19
20[1]: https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo#WithLogger
21[2]: https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo#Logger
22[3]: https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo#BasicLogger
23[4]: https://github.com/uber-go/zap
24[5]: https://pkg.go.dev/github.com/twmb/franz-go/plugin/kzap
25[6]: https://pkg.go.dev/github.com/rs/zerolog
26[7]: https://pkg.go.dev/github.com/twmb/franz-go/plugin/kzerolog
27
28## Metrics
29
30`kgo` takes an unopinionated stance on metrics, instead supporting ["hooks"][8]
31that you can provide functions for to implement your own metrics. You can
32provide an interface that hooks into any behavior you wish to monitor and
33provide yourself extremely coarse monitoring or extremely detailed monitoring.
34If there are angles you would like to monitor do not have a hook, please open
35an issue and we can figure out what hook to add where.
36
37Similar to logging, franz-go provides drop-in packages that provide some
38opinion of which metrics may be useful to monitoring: [`kprom`][9] for
39prometheus, and [`kgmetrics`][10] for gmetrics.
40
41[8]: https://pkg.go.dev/github.com/twmb/franz-go/pkg/kgo#Hook
42[9]: https://pkg.go.dev/github.com/twmb/franz-go/plugin/kprom
43[10]: https://pkg.go.dev/github.com/twmb/franz-go/plugin/kgmetrics
44
45## Latency: brokers, requests, records
46
47The hooks mentioned just above can be used to glean insight into client <=>
48broker latency, request latency, and per-record consume & produce latency.
49Latencies are not provided by default in the plugin packages within franz-go
50because latencies can easily result in cardinality explosion, and it is
51difficult to know ahead of time which latency bucketing a user is interested
52in.
53
54The `OnBroker` hooks (write, read, e2e) can be used for broker latency. The
55hook also has a function parameter indicating the request key that was written
56or read; this key can be used to track per-request request kind latency. For
57example, if you are interested in produce request latency, you can hook into
58request key 0 and monitor it. Similar thought for fetch, with request key 1.
59
60Per-record latency is more difficult to track. When a record is produced, its
61`Timestamp` field is set. You can use `time.Since(r.Timestamp)` when the
62record's promise is called to track the e2e latency for an _individual_ record.
63Internally, records are produced in batches, so a chunk of records will be
64promised in a row all at once. The per-record latency for all of these is
65likely to be roughly the same; each `time.Since` to track latency is a
66_discrete_ observation: per-record latencies should not be added together in an
67attempt to learn something. For more thoughts on record & batch latency when
68producing, skim [#130][130]. Per-record fetch latency can be measured similarly
69to producing: it is more beneficial to measure fetch request latency, but
70per-record timestamp delta's can be used to glean producer <=> consumer e2e
71latency.
72
73[130]: https://github.com/twmb/franz-go/issues/130
74
View as plain text