Skip to content

Remote Data Integration (Time Series Databases and External APIs)

Jump to bottom
yacoubii edited this page Dec 20, 2024 · 1 revision

InfluxDB and Prometheus time-series databases (TSDBs) are integrated as data providers in Palindrome.js. Users can create custom remote data use cases based on three types of data providers:

  • InfluxDB
  • Prometheus
  • External API (apiEndpoint)

Metrics from these different data providers are displayed in 3D in Palindrome.js. The remote data integration architecture is illustrated in the following figure.

image.png

This architecture demonstrates that TSDB querying can be performed either through the frontend or the backend (via Node.js). It comprises three main modules:

  1. Palindrome.js: The main frontend code.
  2. TSDB Service: Located under services/tsdbService in the backend.
  3. Time Series Databases (TSDBs): Currently supported databases are Prometheus and InfluxDB v2 Flux.

Quick Setup

Palindrome.js includes use cases that require metrics data coming from InfluxDB, Prometheus, Node Exporter, and Telegraf. To execute the predefined remote data use cases, run the following command to launch the TSDBs with the default configuration:

docker compose -f ./services/tsdbService/tsdbSetup/docker-compose.yml up -d

Additionally, you can configure custom remote data use cases by defining your own data structure, as described here.

Predefined Remote Data Use Cases in Palindrome.js

Palindrome.js includes the following default remote data use cases:

Use Case Name Data Structure Location
Remote Data data-examples/remote_data.js
Heavy Remote Data data-examples/heavy_remote_data.js
api.open-meteo.com data-examples/oth_api_open_meteo_com.js
Local Live Monitoring data-examples/oth_localLiveMonitoring.js

All these use cases include the dataProviders field in their data structure. By default, Prometheus' URL is http://localhost:9090, and InfluxDB v2's URL is http://localhost:8086. To configure them with your own data providers, update the dataProviders settings accordingly.

Data Structure Definition

For a remote data use case, a specific data structure is required. The data structure is a JSON object containing four main parts:

  • Metrics: An array of JSON objects defining all metrics.
  • Layers: An array of JSON objects defining layers and specifying the metrics used in each layer.
  • Data Providers: An array of JSON objects defining your time-series databases or external APIs.
  • Options: A JSON object containing additional options, such as fetch pace, whether to use the backend for querying data, etc.
{
    "metrics": [],
    "layers": [],
    "dataProviders": [],
    "options": {}
}

Data providers

A data provider definition can contain up to six fields, as explained in the table below:

Field Used for Type Description
id All data providers String The ID of the data provider
type All data providers influxdb | prometheus | apiEndpoint The type of the data provider
url All data providers String The URL of the data provider
token InfluxDB String InfluxDB token
org InfluxDB String InfluxDB organization
remoteDataFetchPace All data providers Integer Fetch pace (in ms) for all metrics using this data provider

Example of data providers defintions:

dataProviders: [
    {
        id: 'influxdb-1',
        type: 'influxdb',
        url: 'http://localhost:8086',
        token: 'token',
        org: 'smile_rnd',
        remoteDataFetchPace: 1_000
    },
    {
        id: 'prometheus-1',
        type: 'prometheus',
        url: 'http://localhost:9090',
        remoteDataFetchPace: 5_000
    },
    {
        id: 'api-1',
        type: 'apiEndpoint',
        url: 'http://example-api.com',
        remoteDataFetchPace: 20_000
    }
]

Metrics

A metric definition should contain the following fields:

Field Type Description
id String A unique identifier for the metric.
label String | JSON

The metric label to display.

String label can be used, but if you're using an external API, and you want to get the label from the API. You can use a JSON object with jmesPath field that contains the JMESpath of the label in the API response. For example:

label: {
   jmesPath: 'metrics[2].name' 
}
value JSON

For external API metrics only, specifies a jmesPath to extract the metric value. For example:

value: {
    jmesPath: 'metrics[2].value'
}
query String

For TSDB metrics only, specifies the Flux or PromQL query. If this metric is gathered from an external API please use value instead.

Example:

query: "promhttp_metric_handler_requests_total{code='200'}"
unit String | JSON

The unit of the metric. It should be a string for the TSDB metrics. It can be a string or a JSON with a jmesPath field for the external API metrics. For example:

unit: {
     jmesPath: 'metrics[1].unit'
}
ranges

Array <int> | Array of <string>

The min, med, max values of the metrics. It should be an array of integers for TSDB metrics. It can be an array of JMESpath strings for external APIs metrics. For example:

ranges: ['metrics.system.cpu.usage.perCore[1].value', 80, 100]
remoteDataFetchPace Integer Fetch pace (in ms) for all metrics using this data provider
dataProviderId String The ID of the data provider used.

Example of a two metrics defintion:

metrics: [
    {
        label: 'HTTP Requests (success)',
        unit: ' ',
        id: 'http200',
        query: "promhttp_metric_handler_requests_total{code='200'}",
        ranges: [0, 1_000_000, 400_000_000],
        dataProviderId: "prometheus-1",
        remoteDataFetchPace: 8000
    },
    {
        id: 'cpu-per-core-id',
        label: 'Core 0',
        value: {
            jmesPath: 'metrics.system.cpu.usage.perCore[0].value'
        },
        unit: {
            jmesPath: 'metrics.system.cpu.usage.perCore[0].units'
        },
        ranges: ['metrics.system.cpu.usage.perCore[1].value', 80, 100],
        dataProviderId: 'api-1',
        remoteDataFetchPace: 3000
    }
]

Layers

A layer definition shoud contain 3 fields as explained in the table below:

Field Type Description
name String The name of the layer
metrics Array<String | JSON> IDs of defined metrics or new metric definitions to include in the layer.
options JSON Palindrome.js layer options. Refer to the layers' behavior documentation.

Example of one layer defintion:

layers : [
    {
        name: "Jmes-path-layer",
        metrics: [
            'request-count-id', 
            'memory-usage-id', 
            {
                id: 'cpu-per-core-id',
                label: 'Core 0',
                value: {
                    jmesPath: 'metrics.system.cpu.usage.perCore[0].value'
                },
                unit: {
                    jmesPath: 'metrics.system.cpu.usage.perCore[0].units'
                },
                ranges: ['metrics.system.cpu.usage.perCore[1].value', 80, 100],
                dataProviderId: 'api-1',
                remoteDataFetchPace: 3000
            }
        ],
        options: {
            "mainColorStatic": "#319b31",
            "layerColorLow": "#ffffff",
            "layerColorMed": "#f3c60a",
            "layerColorHigh": "#0096FF",
            "sphereColorLow": "#ffffff",
            "sphereColorMed": "#f3c60a",
            "sphereColorHigh": "#0096FF",
        }
    }
]

Options

Options is JSON object that can contain 6 fields detailed in the table below:

Field Type Description
liveData Boolean Fetch and update data every remoteDataFetchPace ms if set to true.
useBackend Boolean Fetch TSDB data via the backend if set to true.
remoteDataFetchPace Integer The global data fetch pace in milliseconds.
benchmarkDataUpdate Boolean Enables benchmarking of frontend vs backend query times. Results logged in the console.
benchmarkDuration Integer Duration of the benchmarkDataUpdate test (in ms).
benchmarkInitial Boolean Measures initial render time of remote data use cases. Results logged in the console.

Fetch paces and priority

Fetch pace can be defined at multiple levels:

  • Metric level: Overrides all other fetch pace definitions.
  • Data provider level: Used if fetch pace is not defined at the metric level.
  • Global options level: Default fetch pace if not defined at the metric or data provider levels.

Full remote data use case example

export function remoteSchema() {
    return {
        metrics: [
            {
                id: 'request-count-id',
                label: {
                    jmesPath: 'metrics[2].name' 
                },
                value: {
                    jmesPath: 'metrics[2].value'
                },
                unit: {
                    jmesPath: 'metrics[2].unit'
                },
                ranges: [0, 50, 150],
                dataProviderId: 'api-2',
            },
            {
                id: 'memory-usage-id',
                label: {
                    jmesPath: 'metrics[1].name' 
                },
                value: {
                    jmesPath: 'metrics[1].value'
                },
                unit: {
                    jmesPath: 'metrics[1].unit'
                },
                ranges: [0, 500, 1600],
                dataProviderId: 'api-2',
                remoteDataFetchPace: 3000
            },
            {
                label: 'Kernal',
                unit: ' ',
                id: 'kernal-id',
                query: 'from(bucket: "Palindrome.js")  |> range(start:-1m)  |> filter(fn: (r) => r["_measurement"] == "kernel")  |> filter(fn: (r) => r["_field"] == "processes_forked")  |> map(fn: (r) => ({r with _value: r._value / 1000000}))',
                ranges: [0, 10, 30],
                dataProviderId: "influxdb-1",
                remoteDataFetchPace: 6000
            },
            {
                label: 'Disk Weighted IO Time',
                unit: 'MS',
                id: 'diskWIOT-id',
                query: 'from(bucket: "Palindrome.js") |> range(start:-1m) |> filter(fn: (r) => r["_measurement"] == "diskio") |> filter(fn: (r) => r["_field"] == "weighted_io_time") |> filter(fn: (r) => r["name"] == "sda1")',
                ranges: [0, 500, 1000],
                dataProviderId: "influxdb-1",
                remoteDataFetchPace: 6000
            },
            {
                label: 'Disk Write Time',
                unit: 'MS',
                id: 'diskWT-id',
                query: 'from(bucket: "Palindrome.js")  |> range(start:-1m) |> filter(fn: (r) => r["_measurement"] == "diskio")  |> filter(fn: (r) => r["_field"] == "write_time")  |> filter(fn: (r) => r["name"] == "sda1")',
                ranges: [0, 500, 1000],
                dataProviderId: "influxdb-1"
            },
            {
                label: 'HTTP Requests (success)',
                unit: ' ',
                id: 'http200',
                query: "promhttp_metric_handler_requests_total{code='200'}",
                ranges: [0, 1_000_000, 400_000_000],
                dataProviderId: "prometheus-1",
                remoteDataFetchPace: 8000
            },
            {
                label: 'node_dmi_info',
                unit: ' ',
                id: 'node_dmi_info',
                query: "node_dmi_info",
                ranges: [0, 3, 5],
                dataProviderId: "prometheus-1",
                remoteDataFetchPace: 8000
            },
            {
                label: 'node_filesystem_files',
                unit: ' ',
                id: 'node_filesystem_files',
                query: 'node_filesystem_files{device="tmpfs", fstype="tmpfs", instance="node_exporter:9100", job="palindrome_remote_data_source", mountpoint="/run/snapd/ns"}',
                ranges: [900_000, 1_000_000, 2_000_000],
                dataProviderId: "prometheus-1"
            },
            {   label: 'node_cpu_seconds_total x 1000',
                unit: ' ',
                id: 'node_cpu_seconds_total',
                query: 'rate(node_cpu_seconds_total{mode="user", cpu="2"}[5m])*1000',
                ranges: [0, 5, 12],
                dataProviderId: "prometheus-1"
            },
            {   label: 'node_intr_total',
                unit: ' ',
                id: 'node_intr_total',
                query: 'rate(node_intr_total[5m])',
                ranges: [200, 3000, 6000],
                dataProviderId: "prometheus-1"
            },
            {   label: 'node_context_switches_total',
                unit: ' ',
                id: 'node_context_switches_total',
                query: 'rate(node_context_switches_total[5m])',
                ranges: [200, 500, 800],
                dataProviderId: "prometheus-1"
            },
        ],
        layers : [
            {
                name: "Jmes-path-layer",
                metrics: [
                    'request-count-id', 
                    'memory-usage-id', 
                    {
                        id: 'cpu-per-core-id',
                        label: 'Core 0',
                        value: {
                            jmesPath: 'metrics.system.cpu.usage.perCore[0].value'
                        },
                        unit: {
                            jmesPath: 'metrics.system.cpu.usage.perCore[0].units'
                        },
                        ranges: ['metrics.system.cpu.usage.perCore[1].value', 80, 100],
                        dataProviderId: 'api-1',
                        remoteDataFetchPace: 3000
                    }
                ],
                options: {
                    "mainColorStatic": "#319b31",
                    "layerColorLow": "#ffffff",
                    "layerColorMed": "#f3c60a",
                    "layerColorHigh": "#0096FF",
                    "sphereColorLow": "#ffffff",
                    "sphereColorMed": "#f3c60a",
                    "sphereColorHigh": "#0096FF",
                }
            },
            {
                name: "Server-layer",
                metrics: [
                    {
                        label: 'CPU usage user',
                        unit: ' ',
                        id: 'usage_user',
                        query: 'from(bucket: "Palindrome.js")  |> range(start:-1m)  |> filter(fn: (r) => r["_measurement"] == "cpu") |> filter(fn: (r) => r["_field"] == "usage_user") |> filter(fn: (r) => r["cpu"] == "cpu-total")',
                        ranges: [0, 50, 150],
                        dataProviderId: "influxdb-1",
                        remoteDataFetchPace: 8500
                    },
                    {
                        label: 'RAM',
                        unit: 'GB',
                        id: 'ram-id',
                        query: 'from(bucket: "Palindrome.js")  |> range(start:-1m)  |> filter(fn: (r) => r["_measurement"] == "mem")  |> filter(fn: (r) => r["_field"] == "used") |> map(fn: (r) => ({r with _value: r._value / 107374182}))',
                        ranges: [0, 16, 32],
                        dataProviderId: "influxdb-1"
                    }, 
                    'kernal-id', 
                    'node_filesystem_files', 
                    'node_intr_total', 
                    'node_cpu_seconds_total'
                ]
            },
            {
                name: "Container-layer",
                metrics: [
                    'diskWIOT-id', 
                    'diskWT-id', 
                    'http200', 
                    'node_dmi_info', 
                    {
                        label: 'node_memory_Percpu_bytes',
                        unit: ' ',
                        id: 'node_memory_Percpu_bytes',
                        query: 'node_memory_Percpu_bytes',
                        ranges: [10_000_000, 50_000_000, 100_000_000],
                        dataProviderId: "prometheus-1"
                    }, 
                    'node_context_switches_total'
                ],
                options: {
                    "mainColorStatic": "#319b31",
                }
            }
        ],
        dataProviders: [
            {
                id: 'influxdb-1',
                type: 'influxdb',
                url: 'http://localhost:8086',
                token: 'token',
                org: 'smile_rnd',
                remoteDataFetchPace: 1_000
            },
            {
                id: 'prometheus-1',
                type: 'prometheus',
                url: 'http://localhost:9090',
                remoteDataFetchPace: 5_000
            },
            {
                id: 'api-1',
                type: 'apiEndpoint',
                url: '/apiV1.json',
                remoteDataFetchPace: 20_000
            },
            {
                id: 'api-2',
                type: 'apiEndpoint',
                url: '/apiV2.json',
                remoteDataFetchPace: 20_000
            },

        ],
        options: {
            liveData: false,
            useBackend: false,
            remoteDataFetchPace: 1_000 * 60,
            benchmarkDataUpdate: false,
            benchmarkDuration: 1_000 * 60 * 2,
            benchmarkInitialData: false
        }
    }
}

Benchmark

We conducted a series of benchmarks to evaluate querying times for a single live time-series database (TSDB) use case along with other live TSDB use cases, arranged in a grid as show in the screenshot below.

image (5).png

The benchmarks aim to compare the performance between the frontend and the backend. Each benchmark ran for a total of 4 minutes, during which we measured the time taken by the frontend to query metrics from the TSDB and compared it with the backend's querying time.

In each iteration of the benchmark, we incrementally increased the number of use cases (thus metrics being fetched), and consequently, the number of queries being executed. At each step, we computed the time spent by the backend, the time spent by the frontend, and the performance gain observed as shown in the table below.

Frontend (ms) Backend (ms) Gain (%) Number Of Metrics
4602.93 3496.29 24.04207755 118
6581.56 4935.47 25.01063578 236
8430.23 6703.73 20.47986828 354
10180.88 7893.17 22.47065087 472
13179.45 9624.82 26.9710041 590
17188.63 11134.39 35.22235338 708
29064.99 14806.37 49.05771514 826
33430.03 15896.79 52.44757483 944
35951.01 15839.43 55.94162723 1062
37902.94 16386.39 56.76749614 1180
38819.29 15324.63 60.52315743 1298
40902.43 18462.48 54.86214389 1416
45849.35 15015.27 67.25085525 1534
46605.03 13233.63 71.60471734 1652
51513.05 15213.56 70.4665905 1770

The results consistently demonstrated that the backend outperformed the frontend. This superior performance can be attributed to the backend's optimized query handling, reduced overhead in processing and transferring data, and better utilization of system resources compared to the browser-based frontend.

The figure below presents the execution time as well as the gain over benchmark iterations.

image (6).png

Storybook API Reference

Here, you will find information about the available options for configuring your remote data use case in Storybook. Below is a table outlining the parameters:

Name Description Type Default
liveData Enable or disable live data fetching Boolean False
remoteDataFetchPace Specifies a periodic fetching pace in milliseconds Number 1000
webWorkersHTTP Enable or disable fetching data with web workers Boolean False

Use Case Declaration

In order to register your use case. To ensure the use case is defined in the dev environment, declare it inside the dev/utils/controls.js file, within the palindromes object.

For example:

export let palindromes = {
    // ... other use cases
    yourGroupName: [
        { name: "Your Use Case Name", isRemoteData: true, data: yourFunctionName()},
    ]
};

To ensure the use case is available in the Storybook environment, create a file in the stories folder, for example, stories/yourStoryName.js, with the following content:

import { defaultControls, defaultValues } from './controls/defaultControls';
import { yourFunctionName } from '../data-examples/yourUseCaseName';

export default {
    title: 'Use Cases/Palindrome/Your Storybook Group',
    argTypes: defaultControls(),
    args: defaultValues(),
};
export const remoteData = createPalindrome.bind({});
remoteData.args = {
    isRemoteData: true,
    data: yourFunctionName()
};
Clone this wiki locally