This project exposes Kafka Topic metrics via JMX where they are picked up and stored in InfluxDB and exposed to user via Grafana.
Kakfa Brokers expose topic statistics two ways
- JMX Beans
- As messages in confluent-metrics topic
This project builds a Docker image on top of the confluentinc/cp-server docker iamage.
The Image exposes the Jolokia HTTP endpoint on port 8778.
The Image will receive two tags, the cp-server version and the cp-serverversion appeneded with the build timestamp.
This lets start scripts use the same version number for this cp-server as it does for the rest of the confluent components.
You must run the freemansoft/cp-server with these parameters to enable the Jolokia agent installed in this image.
KAFKA_OPTS: '-javaagent:/opt/jolokia/jolokia-jvm-agent.jar=host=0.0.0.0'
You configure the Telegraf Jolokia2 plugin to use this endpoint as input where it can then be sent to any Telegraf output sink. See the sample telegraf.conf file in this GitHub repo associated with this Docker Image
- Build a docker container that includes the Jolokia agent that exposes JMX over HTTP on port.
cd cp-server-jolokia
build.sh
docker-compose -f docker-compose-kafka.yml up
You may find a pre-built docker image on Dockerhub.
Jolokia exposes about 3000 metrics. The sample telegraf.conf file imports a significant subset of these. telegraf.conf is configured baed on some 3-node work I did even though this repo only has a single broker:
- Poll the kafka broker nodes every 20 seconds instead of 10 seconds. This cuts the amount of data collected in half. The default seems excessive for metrics.
- Increase tinput buffer sizes 4x from the default. The defaults are overrun by 7000 metrics when connected to three servers at the default buffer size. This implies 17000 metrics. The buffer sizes are now 4000/40000 which shold be twice what we need for a 3 node. I just guessed at all that
- Push the data to a Containerized InfluxDB database
This project builds on a Docker Image built on top of the standard confluentinc/cp-server docker image
- Build a Kafka Broker Docker image with the Jolokia Agent
bash build.sh
- Start Kafka
docker-compose -f docker-compose-kafka.yml up
- Verify Kakfa is up by looking at the Confluent Dashboard
- Verify the *Jolokia Agent is coughing up JMX bean data. Browse to the Jolokia REST Endpoint
- Jolokia port 8778 is exposed to the docker host in the docker_compose sample. This will show all the beans available exposed by the Kafka Broker All MBean data is exposed to the Jolokia REST endpoint. Topic statistics should exist for each topic.
- http://localhost:8778/jolokia/list
- Start the monitoring ecosystem - Telegraf, InfluxDB and Grafana
docker-compose -f docker-compose-monitoring.yml up
- Verify the Grafana UI is up. The credentials are admin/admin Browse to the Grafana UI You can ignore the pasword change request for this demo.
Note: These steps are automated in the quickstart subfolder
- Connect to the Grafana Dashboard
- http://localhost:3000/login
- admin/admin
- skip the offered password change
- Navigate to Configuration --> Data Sources
- Click on add data source
- Click on influxDB to start the connection creation
- Create a data source that points at InfluxDB
- Url: http://influxdb:8086
- Database: telegraf
- Leave everything else blank
- Click Save and Test
You can view the datasource configuration via the Grafana REST API http://localhost:3000/api/datasources/
- Connect to the Grafana Dashboard with your browser as described above
- In the Grafana dashboard
- Click on Explore in the left menu
- Make sure the DB droplist at the top is set to InfluxDB
- On the FROM line
- Change the from to the desired metric like cpu or kafka_topics
- On the Select line
- Change the field value to the metric you want to see
- Click on Run Query
Example: FROM default mem WHERE SELECT field(active) mean() GROUP BY time($_interval...) FORMAT AS Time series
We can generate Kafka traffic using the demonstration Data Generator in the Confluence dashboard. See this page https://docs.confluent.io/current/quickstart/ce-quickstart.html The following is a poor cheatsheet
- Open Browser to the Confluent Dashboard
- Create a topic named pageviews
Cluster1 --> Topics --> _+ Add a Topic_ --> name it _pageviews_ --> Press _Create with Defaults_
- Create a topic named users
Cluster1 --> Topics --> _+ Add a Topic_ --> name it _users_ --> Press _Create with Defaults_
- Generate test data using the console and the datagen source
- pageviews
- Connect to Web Console
- Connect to Datagen Connector
- Connect
Clusters > Connect> connect-default > - You will be on the connectors page
- Press Add Connector on the right
- Find then button DatagenConnector press Connect
- Connect
- Set the name to datagent-pageviews
- On the next page Add Connector
- Key converter class field, -> org.apache.kafka.connect.storage.StringConverter.
- kafka.topic field -> pageviews.
- max.interval field -> 100.
- quickstart field -> pageviews.
- Click Continue
- Review the connector configuration and click launch
- users
- Connect to Web Console
- Connect ```Clusters > Connect> connect-default >``
- You will be on the connectors page
- Press Add Connector on the right
- Find the button DatagenConnector press Connect
- Set the name to datagen-users
- On the next page Add Connector
- Key converter class field, -> org.apache.kafka.connect.storage.StringConverter.
- kafka.topic field -> users
- max.interval field -> 100
- quickstart field -> users
- Click Continue
- Review the connector configuration and click launch
- Connect to Web Console
- pageviews
- Verify 2 the two datagen connectors running
Cluster 1 --> connect --> connect-default
- Bring down monitoring
docker-compose -f docker-compose-monitory.yml down
- Bring down kafka
docker-compose -f docker-compose-kafka.yml down
- If you only control-c the open windows then you may need
docker container prune
- You can remove all the locally cached custom images with
docker rmi -f $(docker images | grep freemansoft/cp-server| awk '{print $3}' )
- See sample-jolokia-beans.json for all the mbean data coughed up by Jolokia for a unadulterated Kafka broker
- Jolokia / Telegraf sample config https://telegraf-kafka.readthedocs.io/en/stable/kafka_monitoring.html
- Jolokia / Telegraf sample config https://docs.wavefront.com/kafka.html
- Datagen data generator demo https://docs.confluent.io/current/quickstart/ce-quickstart.html
The same script sits in two places to support DockerHub and human builds.
- /build.sh - for humans
- hooks/build - for DockerHub
freemansoft dockerhub builds are triggered whenever a semver tag is created with the format x.y.z
| Source Type | Source | Docker Tag | Dockerfile location | Build Context |
|---|---|---|---|---|
| tag | /^[0-9.]+$/ | {sourceref} | Dockerfile | / |
Builds are automtically run when tags are created or moved. Tags are created/moved and pushed with the following commands
git tag -f -a <tagname>
git push -f --tags
Version numbers should match the confluent version numbers to make everyone's life simple
- Intro https://youtu.be/w_FQD_pTybQ
- Creating a cp-server Docker image that includes a Jolokia Java Agent https://youtu.be/SHx2PyUl5xE
- Configuring Telegraf with the telegraf.conf file https://youtu.be/2eX8ZAEM3sA
- Displaying captured metrics in Grafana https://youtu.be/WpjCHCjGMhM
You should be abel to Create a grafana database confguration by sending a POST to http://localhost:3000/api/datasources
{
"name": "InfluxDB",
"type": "influxdb",
"url": "http://influxdb:8086",
"database": "telegraf",
}
@see https://grafana.com/docs/grafana/latest/http_api/data_source/