Synthetic Heart is a synthetic testing and monitoring framework for Kubernetes clusters. It allows operators to run synthetic tests, then check and monitor the results through an API or through metrics (via Prometheus).
Synthetic Heart is comprised of four components:
- Agent: Runs the actual tests, watches redis/kv-store for test configs
- Rest api: To query test results, logs and agent details
- Controller: K8s controller (watches for SyntheticTest CRD), and stores them in redis for agents to pick up/query
- Redis (storage): To store test results and also synthetic tests
The controller is a Kubernetes controller that watches the cluster for SyntheticTest CRDs. It mirrors the config in the CRDs to storage (i.e. Redis). It runs a reconcile loop whenever the CRDs change or if there are any redis events such as agent (un)registration events (in order to reschedule tests if an agent exits). Finally, It performs a periodic cleanup of the storage to remove any stale agents or tests.
apiVersion: synheart.infra.webex.com/v1
kind: SyntheticTest
metadata:
name: aws-health-check # name of the test - must be unique
spec:
plugin: httpPing # which plugin to run
node: "$" # which nodes run the test '$'=any node (e.g. 'k8snode*', '*')
displayName: HTTP Ping AWS # name to be displayed in the UI
description: HTTP request to the Amazon AWS
timeouts:
run: 1m # time after which the test is considered failed, and the plugin is restarted
repeat: 2m # how often to repeat the test
config: | # config for the plugin
address: https://console.aws.amazon.com
expectedCodeRegex: ^(200|302){1}
retries: 3
The Agent is responsible for watching the storage (Redis) for any new or updated test configs, and running them.
There are three main components in the agent:
- Plugin Manager - responsible for managing lifecycle of plugins
- Prometheus Exporter - Exports metrics to Prometheus (can use push gateway)
- Plugins - Test logic written as separate binaries/scripts.
The plugin manager is responsible for handling the lifecycle of these plugins. It will start/stop, check health and restart them on crash/error. The plugin manager also exports the test results to redis, as well as metrics to Prometheus.
Plugins run the actual tests. They are separate binaries/scripts that are run by the agent.
This is what makes Synthetic Heart extensible. The plugins can be written in any language, as long as they can communicate with the plugin manager via gRPC. Currently only Golang and Python plugins are tested. The plugins use hashicorp go-plugins.
Synthetic Heart also comes with a Rest API which lets external services query test results etc. It is essentially a shim for the Redis storage.
A UI is currently being worked on. It will allow users to view test results, logs, and agent details. You can check it out the UI repo here.
Easiest way to run synthetic-heart is to install the helm chart in the charts/ directory.
Prerequisites:
- Kubernetes cluster (Tested with v1.25+, older versions may work)
- Helm v3
Steps to follow:
- Install the synthetic-heart Helm chart
- Run:
helm upgrade -i synthetic-heart-system -n synthetic-heart-system .incharts/directory. - This installs the RestAPI, Agents (as DaemonSet), Controller, and Redis.
- Run:
- Install the synthetic-tests Helm chart
- Run
helm upgrade -i synthetic-tests -n synthetic-heart-tests .incharts/directory. - This installs two example synthetic tests.
- Run
- You may then port forward the rest api to query results.
- Run
kubectl port-forward svc/synheart-api-svc 8080:8080 -n synthetic-heart - and then query the tests
curl http://localhost:8080/api/v1/test-configs
- Run
It might be a bit more complex to install the components manually. Please explore the Helm chart to check the example configurations for different Kubernetes resources. The dependencies, and major components are described below.
A few Kubernetes resources need to be installed:
- CustomResourceDefinition - The
SyntheticTestCRD needs to be installed. Link to CRD. - Redis - Redis v7 needs to be installed so the test configs and results can be stored.
- A
Serviceis also needed, so the redis endpoint can be accessed by agents.
- A
- Controller - Needs to be deployed as a
Deployment.- The controller needs
ServiceAccount,ClusteRole,ClusterRoleBinding, allowing it to query and modifySyntheticTests.
- The controller needs
- Agents - Can be deployed as a
DaemonSet.- The configuration of the agents is passed in as a file. The file should be mounted from a
ConfigMap
- The configuration of the agents is passed in as a file. The file should be mounted from a
- RestApi - Needs to be deployed as a
Deployment- The configuration of the agents is passed in as a file. The file should be mounted from a
ConfigMap - A
Servicefor the Restapi is needed. - Optionally an
Ingresscan be added to the Restapi to make the API accessible from outside the cluster.
- The configuration of the agents is passed in as a file. The file should be mounted from a
Please check the Deployment document for different deployment strategies.
Thank you to the following folks for contributing to synthetic-heart project:
- Allen Guo: [email protected]
- Ben Zhao: [email protected]
- Cristian Maugeri: [email protected]
- David Hazra: [email protected]
- Jeff Wang: [email protected]
- Jerry Zhang: [email protected]
- Luke Ren: [email protected]
- Lun Zhou: [email protected]
- Meibao Wang: [email protected]
- Mercion Wilathgamuwage: [email protected]
- Shaz Balakumar: [email protected]
- Kieran Shave [email protected]
There's a lot of features that are still being worked on. Get involved!
- UI (currently being worked on)
Re-design agent deployment model (allow agents per node, per namespace etc.)- Allow agents to be deployed in different namespace, mount secrets etc.Python plugins- Agent CRD (to allow for easier agent deployment)
- Plugin versions
- Dynamically add or remove plugins at build time (to reduce binary size)
- More plugins
Please refer to CONTRIBUTING.md
Please refer to LICENSE