Using Dash to visualize events. Events are expected to be stored in Elasticsearch.
Enable to see all send events at a glance, provide filtering and visualizations for the events. Enhance data by adding information from other services, like a Pact broker.
A consumer gets all Kafka events on a regular basis and writes them into elasticsearch (Note: consuming and writing of events is not part of this application!) On consuming the callbacks for updating the diagrams are called, for example "lastUpdate". The diagrams are build using timeseries between lastCleared and lastUpdate, as well es the consumer group id.
In order to run the application locally, you need have an Elasticsearch instance running. For development and to get started, it is recommended to use the single node dockerized instance, as described here
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.5.2
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --name elastic docker.elastic.co/elasticsearch/elasticsearch:7.5.2
Plantuml is used to create sequence diagrams based on the events. Since the receiver of the event is not contained in the event, the information is added as target to the schema file. In order to work properly please add plantuml.jar along app.py.
It is recommended (but not necessary) to use a virtual environment virtual environment. Checkout the repository and install all required module using command
pip install -r requirements.txt
Afterwards start the app using
python app.py
The dashboard is then available here following the rule
http://<host>:<port>/<url_base_pathname>
# see line: app = dash.Dash(server=server, url_base_pathname=url_base_path, external_stylesheets=[dbc.themes.BOOTSTRAP])
# with url_base_path = config.config['url_base_path']
In order to add some test data you can use the test in test_add_new_event.py. Run
pytest -k test_add_new_event.py
which will add a new event to elasticsearch.
Afterwards you should see the event in the dashboard.
Run command
docker build -t event_monitor .
to build a docker image.
# if running against local elasticsearch
docker run -p 18850:80 -it event_monitor
# if running against dockerized elasticsearch
docker run -p 18850:80 -it --name dashboard --link=elastic:elastic event_monitor
The link in the second command is used to connect the dashboard container to the elasticsearch container. For more information please have a look here.
You can see all events which are available at elasticsearch.
You can also filter the events by timestamp using the input fields marked with the red 1 and 2. At the moment it is required to use the same date format, since no conversion or verification is implemented. To use the configured timestamps for filter please press button "Apply Filter" marked with the red 3. If you want to reset the filter you can click on button "Reset Filter" marked with the red 4. This removes the filter criteria and will return all events and also update the input fields (red 1 and 2) with the oldest and the youngest timestamp.
In addition to timestamp (backend) filtering you can use the (frontend) filter as shown in the next picture.
The filtered table data are used to display various diagrams, for example number of events. The diagrams/charts get visible when clicking on the according buttons, like it can be seen here:
Shows events based on their timestamp. If the correlation id for multiple events match, a connection between those events is displayed, as you can see on right hand side of the diagram.
Based on the timestamp in the events the time between those events is calculated and shown in the diagram.
This diagram shows a sort of sequence diagram. It contains almost the same information as the timeseries diagram, but instead of the timestamp the index of the event is used. Relations between events are based on the correlation id.
Based on the target property in the event schema a plantuml seqnuence diagram is generated and displayed. It is also possible to download the diagram as png.
To validate the events jsonschema is used. An example schema file can be found in folder event_schemas. An updated schema definition can either be placed in the same folder or in can be posted using Rest. See folder
tests/e2e/test_post_schema.py
as reference. Using that api results in creating a new schema file with a timestamp. The app reads that file and stores its content in-memory, to speed up event validation.