1. Home
  2. Docs
  3. EDP
  5. 6.1.5 Checking consumer position

6.1.5 Checking consumer position

Sometimes it’s useful to see the position of your consumers. We have a tool that will show the position of all consumers in a consumer group as well as how far behind the end of the log they are. To run this tool on a consumer group named my-group consuming a topic named my-topicwould look like this:

  > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group

  TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
  my-topic                       0          2               4               2          consumer-1-029af89c-873c-4751-a720-cefd41a669d6   /                     consumer-1
  my-topic                       1          2               3               1          consumer-1-029af89c-873c-4751-a720-cefd41a669d6   /                     consumer-1
  my-topic                       2          2               3               1          consumer-2-42c1abd4-e3b2-425d-a8bb-e1ea49b29bb2   /                     consumer-2

Managing Consumer Groups

With the ConsumerGroupCommand tool, we can list, describe, or delete the consumer groups. The consumer group can be deleted manually, or automatically when the last committed offset for that group expires. Manual deletion works only if the group does not have any active members. For example, to list all consumer groups across all topics:

  > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list


To view offsets, as mentioned earlier, we “describe” the consumer group like this:

  > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group

  TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                    HOST            CLIENT-ID
  topic3          0          241019          395308          154289          consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /      consumer2
  topic2          1          520678          803288          282610          consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /      consumer2
  topic3          1          241018          398817          157799          consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 /      consumer2
  topic1          0          854144          855809          1665            consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /      consumer1
  topic2          0          460537          803290          342753          consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 /      consumer1
  topic3          2          243655          398812          155157          consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 /      consumer4

There are a number of additional “describe” options that can be used to provide more detailed information about a consumer group:

  • –members: This option provides the list of all active members in the consumer group. > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members CONSUMER-ID HOST CLIENT-ID #PARTITIONS consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 / consumer1 2 consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 / consumer4 1 consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 / consumer2 3 consumer3-ecea43e4-1f01-479f-8349-f9130b75d8ee / consumer3 0
  • –members –verbose: On top of the information reported by the “–members” options above, this option also provides the partitions assigned to each member. > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members --verbose CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT consumer1-3fc8d6f1-581a-4472-bdf3-3515b4aee8c1 / consumer1 2 topic1(0), topic2(0) consumer4-117fe4d3-c6c1-4178-8ee9-eb4a3954bee0 / consumer4 1 topic3(2) consumer2-e76ea8c3-5d30-4299-9005-47eb41f3d3c4 / consumer2 3 topic2(1), topic3(0,1) consumer3-ecea43e4-1f01-479f-8349-f9130b75d8ee / consumer3 0 -
  • –offsets: This is the default describe option and provides the same output as the “–describe” option.
  • –state: This option provides useful group-level information. > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS localhost:9092 (0) range Stable 4

To manually delete one or multiple consumer groups, the “–delete” option can be used:

  > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-group --group my-other-group

  Deletion of requested consumer groups ('my-group', 'my-other-group') was successful.

To reset offsets of a consumer group, “–reset-offsets” option can be used. This option supports one consumer group at the time. It requires defining following scopes: –all-topics or –topic. One scope must be selected, unless you use ‘–from-file’ scenario. Also, first make sure that the consumer instances are inactive. See  KIP-122 for more details.

It has 3 execution options:

  • (default) to display which offsets to reset.
  • –execute : to execute –reset-offsets process.
  • –export : to export the results to a CSV format.

–reset-offsets also has following scenarios to choose from (at least one scenario must be selected):

  • –to-datetime <String: datetime> : Reset offsets to offsets from datetime. Format: ‘YYYY-MM-DDTHH:mm:SS.sss’
  • –to-earliest : Reset offsets to earliest offset.
  • –to-latest : Reset offsets to latest offset.
  • –shift-by <Long: number-of-offsets> : Reset offsets shifting current offset by ‘n’, where ‘n’ can be positive or negative.
  • –from-file : Reset offsets to values defined in CSV file.
  • –to-current : Resets offsets to current offset.
  • –by-duration <String: duration> : Reset offsets to offset by duration from current timestamp. Format: ‘PnDTnHnMnS’
  • –to-offset : Reset offsets to a specific offset.

Please note, that out of range offsets will be adjusted to available offset end. For example, if offset end is at 10 and offset shift request is of 15, then, offset at 10 will actually be selected.

For example, to reset offsets of a consumer group to the latest offset:

  > bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group consumergroup1 --topic topic1 --to-latest

  TOPIC                          PARTITION  NEW-OFFSET
  topic1                         0          0

If you are using the old high-level consumer and storing the group metadata in ZooKeeper (i.e. offsets.storage=zookeeper), pass --zookeeper instead of --bootstrap-server:

  > bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --list