Tests failing with constant keywords in transforms with LogsDB

[jsoriano]

Some packages are failing when logsdb is enabled with:

test case failed: errors found in documents of preview for transform logs-ti_cybersixgill.latest_ioc-default-0.3.0 for data stream logs-ti_cybersixgill.threat-34090: [0] parsing field value failed: field "labels.is_ioc_transform_source"'s value "true" does not match the declared constant_keyword value "false"

For testing we use the preview transform API, where the value actually doesn't match with the definition, but this doesn't seem to be an issue in a real deployment. Confirm if this is actually not an issue, and adjust the tests so they don't fail on this case.

[jsoriano]

@kpollich this would be kind of urgent because there are many tests failing in the daily builds because of this.

[kpollich]

Thanks @jsoriano I added this to our current sprint.

[jsoriano]

@chrisberkhout I was working on this issue, using the ti_anomali package, and I have found that the transform is actually failing:

java.lang.IllegalArgumentException: [constant_keyword] field [labels.is_ioc_transform_source] only accepts values that are equal to the value defined in the mappings [false], but got [true]

This can be reproduced with:

elastic-package test system --setup -v --config-file ./data_stream/threatstream/_dev/test/system/test-integrator-http-config.yml

So there seems to be an actual issue.

We would need to work on two things:

• Ensuring that issues in transforms are detected by elastic-package test.
• Fixing these packages. For this maybe we need to add ingest pipelines to remove or modify these labels.is_ioc_transform_source fields in transform destination indexes.

[jsoriano]

Fixing these packages. For this maybe we need to add ingest pipelines to remove or modify these labels.is_ioc_transform_source fields in transform destination indexes.

Tried to add a pipeline but I receive the same failure, even if logsdb is not enabled 🤔 elastic/integrations#11750

[jsoriano]

Fixing these packages. For this maybe we need to add ingest pipelines to remove or modify these labels.is_ioc_transform_source fields in transform destination indexes.

Tried to add a pipeline but I receive the same failure, even if logsdb is not enabled 🤔 elastic/integrations#11750

Ah, I think the problem was on the type, I was setting it to boolean false, instead of to string "false".

[jsoriano]

There is already an open issue about testing transforms: #859
I have opened another one about supporting ingest pipelines in transforms: elastic/package-spec#833

[andrewkroh]

test case failed: errors found in documents of preview for transform logs-ti_cybersixgill.latest_ioc-default-0.3.0 for data stream logs-ti_cybersixgill.threat-34090: [0] parsing field value failed: field "labels.is_ioc_transform_source"'s value "true" does not match the declared constant_keyword value "false"

@jsoriano Where/how was this originally detected? I don't see any automated issues being opened in elastic/integrations for this failure? Or was this detected via some under-development EP version? Trying to see what we are missing that would have highlighted this sooner (obviously #859 is part of that).

It is a real failure for logsdb where synthetic _source adds the constant_keyword value into the _source while the value would not normally be present outside of logsdb since it is only defined in the constant_keyword mapping. This did result in failures for TI integration users in Serverless (where logsdb is on by default). As of elastic/elasticsearch#117182, Elasticsearch has now been changed to NOT insert constant_keywords into synthesized _source.

[andrewkroh]

Where/how was this originally detected? I don't see any automated issues being opened in elastic/integrations for this failure?

I found the answer. It was failing on CI since Sept 26, but because there were more than 20 failures it wasn't reported through GH issues.

[jsoriano]

It is a real failure for logsdb where synthetic _source adds the constant_keyword value into the _source while the value would not normally be present outside of logsdb since it is only defined in the constant_keyword mapping. This did result in failures for TI integration users in Serverless (where logsdb is on by default). As of elastic/elasticsearch#117182, Elasticsearch has now been changed to NOT insert constant_keywords into synthesized _source.

Oh ok, so this will get fixed when this change reaches the snapshot versions used in CI.

[jsoriano]

Issue is indeed fixed now with latest snapshots https://buildkite.com/elastic/integrations-schedule-daily/builds/401

Closing this.