1Group consuming, using a goroutine per partition
2===
3
4This directory contains three examples that demonstrate different ways to
5have per-partition processing as a group consumer. Because each file is
6invoked the same way, this one readme serves all three examples.
7
8These examples consume from a group and start a goroutine to process each
9partition concurrently. This type of code may be useful if processing each
10record per partition is slow, such that processing records in a single
11`PollFetches` loop is not as fast as you want it to be.
12
13A simpler solution would be to have a pool of goroutines selecting from a
14channel and then sending all records from your `PollFetches` loop down this
15channel. However, the simple solution does not preserve per-partition ordering.
16
17## Auto committing
18
19The autocommitting example is the simplest, but is the most prone to duplicate
20consuming due to rebalances. This solution consumes and processes each
21partition individually, but does nothing about a behind-the-scenes rebalance.
22If a rebalance happens after records are sent to the partition goroutines,
23those partition goroutines will process records for partitions that may have
24been lost.
25
26## Auto committing marks
27
28This example adds a few things to the simpler auto-committing example. First,
29we switch to `BlockRebalanceOnPoll` and uses some locking to avoid rebalances
30while the partition goroutines are processing, and we switch to
31`AutoCommitMarks` to have more control over what will actually be committed.
32This example uses `CommitUncommittedOffsets` at the end of being revoked to
33ensure that marked records are committed before revoking is allowed to
34continue. Lastly, we use `EachPartition` rather than `EachTopic` to avoid the
35internal allocations that `EachTopic` may do.
36
37Blocking rebalance while polling allows for a lot of simplifications in
38comparison to plain autocommitting. Compare the differences: we worry less
39about whether partition consumers have gone away, and we are more sure of what
40is actually happening. These simplifications are commented within the file.
41
42The main downside with `BlockRebalanceOnPoll` is that your application is more
43at risk of blocking the rebalance so long that the member is booted from the
44group. You must ensure that your goroutine workers are fast enough to not block
45rebalancing for all of `RebalanceTimeout`.
46
47## Manually commit
48
49This example is a small extension of the autocommit marks example: rather than
50marking records for commit and forcing a commit when revoked, we issue a
51synchronous commit in each partition consumer whenever a partition batch is
52processed.
53
54This example will have more blocking commits, but has even tighter guarantees
55around what is committed when. Because this also uses `BlockRebalanceOnPoll`,
56like above, you must ensure that your partition processing is fast enough to
57not block a rebalance too long.
58
59## Flags
60
61The flags in each example are the same:
62
63`-b` can be specified to override the default localhost:9092 broker to any
64comma delimited set of brokers.
65
66`-t` specifies the topic to consume (required)
67
68`-g` specifies the group to consume in (required)
69
View as plain text