Replies: 6 comments 15 replies
-
Unnecessary multiple requests with the same esagg query (FIXED in #182919)When a search is sent to ES and a response is received, the search service is looking if the request needs a post-flight request. kibana/src/plugins/data/common/search/search_source/search_source.ts Lines 539 to 548 in 74fdd1b If needed, it transforms the response to a partial response and update the body with the postflight request.This works correctly if the postflight is actually necessary, but due to the current implementation the postflight request is always "applied" even if not needed, causing a subsequent request to be sent to ES. This results to an increase of:
Analysis kibana/src/plugins/data/common/search/search_source/search_source.ts Lines 474 to 483 in 74fdd1b This function is there even if is not required. For example in a All the other cases this is defaulted to an identity function, so the hasPostFlightRequests function will always return true. |
Beta Was this translation helpful? Give feedback.
-
|
lets fix this |
Beta Was this translation helpful? Give feedback.
-
|
I created this issue to track #183199 |
Beta Was this translation helpful? Give feedback.
-
|
Peter already created an issue and a PR for that #182919 |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
from @dej611
|
Beta Was this translation helpful? Give feedback.
-
|
i think we need to be careful with increasing this values as they might lead to substantial increase of open requests to elasticsearch. Lets discuss with es team before going forward with this. |
Beta Was this translation helpful? Give feedback.
-
|
++ @ppisljar for recommendation to double check with ES-team. I do hope we can "push down" this concern. Either we initiate unnecessary polls or we keep the default open longer. The former is the current situation, and there is quite some additional overhead in a full round-trip in multiple layers of the stack. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
+1 for using a simple for-loop, functional programming-style by nature introduces overhead. Maybe we can remove the |
Beta Was this translation helpful? Give feedback.
-
Test 1 utopic baselineI've implemented the most simple HTML page that covers: fetch directly to ES, a bit of data processing, and rendering on canvas. This time can't be achieved in Kibana but at least can be considered as the super-optimal baseline. Simple demo page<!DOCTYPE html>
<html>
<head>
<title>My Empty Page</title>
</head>
<body>
<button onclick="run()" style="position:absolute; right: 0;">RUN</button>
<canvas id="chart"></canvas>
<script>
async function run() {
const maxSeries = 1;
const canvasWidth = 1000;
const chartHeight = 200;
const canvasHeight = (chartHeight) * maxSeries;
const canvas = document.getElementById('chart');
const ctx = canvas.getContext('2d');
ctx.scale(window.devicePixelRatio, window.devicePixelRatio);
canvas.width = canvasWidth * window.devicePixelRatio;
canvas.height = canvasHeight * window.devicePixelRatio;
canvas.style.width = `${canvasWidth}px`;
canvas.style.height = `${canvasHeight}px`;
function linearScale(value, inputDomain, outputRange) {
const [inputMin, inputMax] = inputDomain;
const [outputMin, outputMax] = outputRange;
const inputRange = inputMax - inputMin;
const outputDiff = outputMax - outputMin;
const scaledValue = ((value - inputMin) / inputRange) * outputDiff + outputMin;
return scaledValue;
}
const overallTimes = [];
async function getData(i, t) {
const startDate = `2024-04-28T22:10:27.454Z`;
const endDate = `2024-05-01T21:00:48.656Z`;
const startTime = performance.now();
console.time(`Overall [${i}]`);
console.time(`Fetch [${i}]`);
const response = await fetch("http://localhost:9200/_search?max_concurrent_shard_requests=10", {
"headers": {
'Authorization': `Basic ${btoa("elastic:changeme")}`,
"content-type": "application/json",
},
"body": JSON.stringify({
"aggs": {
"byTerms": {
"terms": {
"field": "geo.dest",
"order": {
"_key": "asc"
},
"size": 30
},
"aggs": {
"byDate": {
"date_histogram": {
"field": "@timestamp",
"fixed_interval": "5m",
"time_zone": "Europe/Rome",
"extended_bounds": {
"min": new Date(startDate).valueOf(),
"max": new Date(endDate).valueOf()
}
},
"aggs": {
"avgMetric": {
"avg": {
"field": "bytes"
}
},
"maxMetric": {
"max": {
"field": "machine.ram"
}
}
}
}
}
}
},
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"format": "strict_date_optional_time",
"gte": startDate,
"lte": endDate
}
}
}
]
}
}
}
),
"method": "POST",
});
console.timeLog(`Fetch [${i}]`, 'response');
const data = await response.json();
console.timeEnd(`Fetch [${i}]`);
console.log(`ES response took: ${data.took}`);
console.time(`Tabify [${i}]`);
const series = data.aggregations.byTerms.buckets.reduce((acc, bucket) => {
const { key, byDate: { buckets } } = bucket;
const seriesData = buckets.reduce((bucketAcc, d) => {
bucketAcc.dataArray.push(
{
key,
x: d.key,
maxMetric: d.maxMetric.value,
avgMetric: d.avgMetric.value,
});
bucketAcc.xDomain[0] = Math.min(bucketAcc.xDomain[0], d.key);
bucketAcc.xDomain[1] = Math.max(bucketAcc.xDomain[1], d.key);
bucketAcc.yDomain[0] = Math.min(bucketAcc.yDomain[0], d.maxMetric.value, d.avgMetric.value);
bucketAcc.yDomain[1] = Math.max(bucketAcc.yDomain[1], d.maxMetric.value, d.avgMetric.value);
return bucketAcc;
}, { dataArray: [], xDomain: [Infinity, -Infinity], yDomain: [Infinity, -Infinity] })
acc.series.set(key, seriesData.dataArray);
acc.xDomain[0] = Math.min(acc.xDomain[0], seriesData.xDomain[0]);
acc.xDomain[1] = Math.max(acc.xDomain[1], seriesData.xDomain[1]);
acc.yDomain[0] = Math.min(acc.yDomain[0], seriesData.yDomain[0]);
acc.yDomain[1] = Math.max(acc.yDomain[1], seriesData.yDomain[1]);
return acc;
}, { series: new Map(), xDomain: [Infinity, -Infinity], yDomain: [Infinity, -Infinity] });
console.timeEnd(`Tabify [${i}]`);
console.time(`Rendering [${i}]`);
// ctx.fillStyle = 'white';
// ctx.fillRect(0, 0, canvasWidth, canvasHeight);
ctx.save();
ctx.translate(0, (chartHeight) * i);
const colors = [
'#54B399',
'#6092C0',
'#9170B8',
'#CA8EAE',
'#D36086',
'#E7664C',
'#AA6556',
'#DA8B45',
'#B9A888',
'#D6BF57',
];
[...series.series.values()].forEach((s, i) => {
let firstMoveA = false;
let firstMoveB = false;
ctx.strokeStyle = colors[i % colors.length];
ctx.beginPath();
s.forEach((d, i) => {
if (d.maxMetric !== null && !firstMoveA) {
ctx.moveTo(linearScale(d.x, series.xDomain, [0, canvasWidth]), linearScale(d.maxMetric, series.yDomain, [chartHeight, 0]));
firstMoveA = true;
} else if (d.maxMetric !== null && firstMoveA) {
ctx.lineTo(linearScale(d.x, series.xDomain, [0, canvasWidth]), linearScale(d.maxMetric, series.yDomain, [chartHeight, 0]));
}
});
ctx.stroke();
ctx.beginPath();
s.forEach((d, i) => {
if (d.avgMetric !== null && !firstMoveB) {
ctx.moveTo(linearScale(d.x, series.xDomain, [0, canvasWidth]), linearScale(d.avgMetric, series.yDomain, [chartHeight, 0]));
firstMoveA = true;
} else if (d.avgMetric !== null && firstMoveB) {
ctx.lineTo(linearScale(d.x, series.xDomain, [0, canvasWidth]), linearScale(d.avgMetric, series.yDomain, [chartHeight, 0]));
}
});
ctx.stroke();
});
ctx.restore();
console.timeEnd(`Rendering [${i}]`);
console.timeEnd(`Overall [${i}]`);
console.log(`-------`);
overallTimes.push(performance.now() - startTime);
}
console.time('Total time parallel');
const requests = Array.from({ length: maxSeries }, (d, i) => getData(i));
await Promise.all(requests);
console.timeEnd('Total time parallel');
// console.time('Total time sequential');
// for(let i = 0; i < maxSeries; i++) {
// await getData(i, 0);
// }
// console.timeEnd('Total time sequential');
console.log(`Overall avg time: ${overallTimes.reduce((s, c) => { s += c; return s }, 0) / overallTimes.length}ms`);
}
</script>
</body>
</html> |
Beta Was this translation helpful? Give feedback.
-
|
This is pretty great illustration :) . It also shows there is quite some room for improvement. |
Beta Was this translation helpful? Give feedback.
-
Kibana Server doesn't reply with 304 to cached requestsDuring the testing, I've noticed that our server doesn't reply with 304 HTTP statuses even if the request is actually the same. When we have cases where nothing changed in a dashboard we should probably respond with 304 instead of querying ES again and waiting for its cached response. 
|
Beta Was this translation helpful? Give feedback.
-
|
this sounds like it would need a cache on kibana server, which can be a memory problem. (as es doesnt respond with 304 kibana server would need to keep previous response in memory to figure out that the new one looks the same, or rely on request params to decide the response will not change). i would prefer to avoid this. |
Beta Was this translation helpful? Give feedback.
-
|
Maybe we could evaluate case by case where to apply a 304. |
Beta Was this translation helpful? Give feedback.
-
|
@ppisljar agreed we'd likely want to avoid introducing more caches on Kibana server, especially for data responses. @dej611 wrt asking the list of indices (so not wrt bsearch) How do we load this now? Overall, rather than necessarily returning 304s, it may also be something we could side-step and rely on cache headers and have the browser figure it out. e.g. using http-cache headers is how Maps ends up not re-requesting identical data. e.g. see how pbf -reqeusts are loaded from disc
It does require loading data with GET for this to work "automagically" I think. @mattkime do you any thoughts here? Some of the issue here feels similar to what ended up for field-cap caching, although that did introduce more caching server-side I believe (but was also needed for server-side alert-usage so ⚖️ ) |
Beta Was this translation helpful? Give feedback.
-
Polling searchAs described earlier, we have a polling mechanism that is used to check for search completion when using async searches. kibana/src/plugins/data/common/search/poll_search.ts Lines 20 to 35 in b8d8c73 Could be interesting knowing how many HTTP open connection we can keep between the ES and Kibana server and between the Kibana server and the client so we can use server-side push or just keep a long living tunnel to receive the ES responses without any polling mechanism in place. |
Beta Was this translation helpful? Give feedback.
-
|
6 open connections between kibana client and server, we work around this by using bfetch |
Beta Was this translation helpful? Give feedback.
-
|
I think @markov00 was also asking for server to server connections limit (which I suspect is way more than 6, correct?) |
Beta Was this translation helpful? Give feedback.
-
|
@ppisljar wrt open connections, we are looking to remove this altogether by moving to http2 eventually #7104 (comment). @dej611 agreed. That is also the assumption we make for doing a trial run and turning of bsearch in Serverless, where the proxy has http2 support. cc @lukasolson |
Beta Was this translation helpful? Give feedback.
-
On the client-side: we could use just the streaming functionality of the |
Beta Was this translation helpful? Give feedback.
-
I will add here some of the findings related to performance bottlenecks the current Lens/SearchStrategy architecture and possible improvements to these solutions.
Please add comments on each of these findings, or add new ones if you found something that is worth considering as performance improvement
Current Test environment and parameters:
Test 1: single chart with date_histogram and terms aggregation
ES running on a single node locally on M1 Max Mac.
10M documentsingested inside ESbreakdown by
30 series(noother)date_histogram over
3 daysat with buckets every5 minutes:~864 date buckets per seriesa total of
864 * 30 data points(counting also empty buckets):~26k data pointsPanel JSON
[ { "type": "lens", "gridData": { "x": 0, "y": 0, "w": 48, "h": 15, "i": "9f528643-9481-418b-b2df-ccb1ec708e0c" }, "panelIndex": "9f528643-9481-418b-b2df-ccb1ec708e0c", "embeddableConfig": { "attributes": { "title": "check", "description": "", "visualizationType": "lnsXY", "type": "lens", "references": [ { "type": "index-pattern", "id": "cffc1414-a0b2-4ea8-b1d0-690289326c93", "name": "indexpattern-datasource-layer-c8abd0c4-a2b4-43b8-97c8-116e65fefc99" } ], "state": { "visualization": { "legend": { "isVisible": true, "position": "right" }, "valueLabels": "hide", "fittingFunction": "Linear", "axisTitlesVisibilitySettings": { "x": true, "yLeft": true, "yRight": true }, "tickLabelsVisibilitySettings": { "x": true, "yLeft": true, "yRight": true }, "labelsOrientation": { "x": 0, "yLeft": 0, "yRight": 0 }, "gridlinesVisibilitySettings": { "x": true, "yLeft": true, "yRight": true }, "preferredSeriesType": "line", "layers": [ { "layerId": "c8abd0c4-a2b4-43b8-97c8-116e65fefc99", "accessors": [ "e0dee44e-457f-48a4-afc2-5221722e3ea7", "0ad75b08-3083-4e1e-bd49-feedc82ed9b6" ], "position": "top", "seriesType": "line", "showGridlines": false, "layerType": "data", "colorMapping": { "assignments": [], "specialAssignments": [ { "rule": { "type": "other" }, "color": { "type": "loop" }, "touched": false } ], "paletteId": "eui_amsterdam_color_blind", "colorMode": { "type": "categorical" } }, "xAccessor": "b1b9ee06-1bef-4b77-b56a-5fae4e0e40db", "splitAccessor": "783bd726-3eae-41a9-bdd1-d2aebbb0446d" } ], "emphasizeFitting": true }, "query": { "query": "", "language": "kuery" }, "filters": [], "datasourceStates": { "formBased": { "layers": { "c8abd0c4-a2b4-43b8-97c8-116e65fefc99": { "columns": { "b1b9ee06-1bef-4b77-b56a-5fae4e0e40db": { "label": "@timestamp", "dataType": "date", "operationType": "date_histogram", "sourceField": "@timestamp", "isBucketed": true, "scale": "interval", "params": { "interval": "auto", "includeEmptyRows": true, "dropPartials": false } }, "e0dee44e-457f-48a4-afc2-5221722e3ea7": { "label": "Average of bytes", "dataType": "number", "operationType": "average", "sourceField": "bytes", "isBucketed": false, "scale": "ratio", "params": { "emptyAsNull": true } }, "783bd726-3eae-41a9-bdd1-d2aebbb0446d": { "label": "Top 30 values of geo.dest", "dataType": "string", "operationType": "terms", "scale": "ordinal", "sourceField": "geo.dest", "isBucketed": true, "params": { "size": 30, "orderBy": { "type": "alphabetical", "fallback": false }, "orderDirection": "asc", "otherBucket": false, "missingBucket": false, "parentFormat": { "id": "terms" }, "include": [], "exclude": [], "includeIsRegex": false, "excludeIsRegex": false } }, "0ad75b08-3083-4e1e-bd49-feedc82ed9b6": { "label": "Maximum of machine.ram", "dataType": "number", "operationType": "max", "sourceField": "machine.ram", "isBucketed": false, "scale": "ratio", "params": { "emptyAsNull": true } } }, "columnOrder": [ "783bd726-3eae-41a9-bdd1-d2aebbb0446d", "b1b9ee06-1bef-4b77-b56a-5fae4e0e40db", "e0dee44e-457f-48a4-afc2-5221722e3ea7", "0ad75b08-3083-4e1e-bd49-feedc82ed9b6" ], "incompleteColumns": {}, "sampling": 1 } } }, "indexpattern": { "layers": {} }, "textBased": { "layers": {} } }, "internalReferences": [], "adHocDataViews": {} } }, "enhancements": {} } } ]ES query
Profile file
Profiling results with hot ES cache and client cache from a click of the refresh button to render-complete event
The overall process takes ~ 1.2seconds to complete.
~60ms between the click event and the call to the fetch function. Looks like most time is spent on:
dashboard: forceRefresh, onembeddable: setInitializationFinishedand onlens: renderUserMessages.~550ms from fetch request to tabified data. As described in #182551 (comment) and #182551 (comment) we have a fetch request due to mentioned reasons.
~650ms A long time is spent on Elastic-charts to compute the geometries to render. We can probably skip this passage, avoid generating geometries objects, and render the values directly, but require an alternative method to detect geometries under the mouse cursor.
Beta Was this translation helpful? Give feedback.
All reactions