cylance,ti_opencti: fix up package validation issues

[efd6]

Proposed commit message

cylance: fix kibana.version syntax in manifest
ti_opencti: use extended list of expected_values for indicator type

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Screenshots

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[andrewkroh]

LGTM. But we should wait for @chrisberkhout to take look.

[efd6]

Agreed. That's why I did not include this in the previous batch. I'm not wildly satisfied with the available labels.

[chrisberkhout]

The expected values issue seems to keep coming up.

Regarding the validation of expected values

When I created the OpenCTI integration I included pipeline test examples for each type of indicator that I had data for, which included some that weren't in the expected_values list for threat.indicator.type, so elastic-package failed the build.

When the expected_values config option was added it wasn't intended to be a list of the only possible values. The available documentation still says:

expected_values (optional): An array of expected values for the field. Schema consumers can validate integrations and mapped data against the listed values. These values are the recommended convention, but users may also use other values.

There's also an allowed_values config option that is different.

I raised the issue in elastic/elastic-package#1472 and it was decided that when we have expected values, we'll continue to enforce those as the only valid values, but we'll allow integrations to override those settings with a different list of expected values (implemented in elastic/package-spec#616).

I had different variations of the config that got the build to pass at different times. The last version overrode the definition of threat.indicator.type, but without providing a list of expected values. I guess that stopped working? The best update might be to add a list of expected_values to that definition.

Regarding the expected values themselves

The ECS documentation of threat.indicator.type describes it as:

Type of indicator as represented by Cyber Observable in STIX 2.0.

However, the list of expected values misses some types from STIX and adds some others. OpenCTI uses STIX 2.1 type names but adds several extras. In colums D-G of the "OpenCTI observable types" tab of the "OpenCTI Mappings" spreadsheet, I made a summary of all the type names and which lists they are in.

I'm not sure if alert rules or Security UI features depend on having specific threat.indicator.type values, but if so an unknown extra value might still be better than changing to the closest type in the expected list.

The STIX standard has a fixed list of types and no defined way to mark an extra type or unknown type.

I think the ECS documentation should be updated to say threat.indicator.type uses names from STIX 2.1 where possible, but allows other values. The list of expected values could be updated and remain, or it could be changed to a list of example values. If necessary the OpenCTI integration can override the definition with an expanded list. I do think we should keep the expanded list for OpenCTI. However, if we did remove values from threat.indicator.type, we should keep the original value in a new field opencti.indicator.type and update a README section.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: bd7b497

History

• 💚 Build #12582 succeeded c8d840e
• 💚 Build #12580 succeeded 355092a27da4d514ee6699c4718eb4680a6cfc86
• 💚 Build #12502 succeeded f79920f
• 💔 Build #12501 failed e6a37ce03d084d5afc7aa749dbedb184a28a4f97
• 💚 Build #12489 succeeded f79920f
• 💚 Build #12477 succeeded 64978f0ab2c7efb128ec0e1c8b2c6f4344d81172

cc @efd6

[chrisberkhout]

Added a couple of commits.

[efd6]

Thanks

[elasticmachine]

Package cylance - 0.19.3 containing this change is available at https://epr.elastic.co/search?package=cylance

[elasticmachine]

Package ti_opencti - 2.2.0 containing this change is available at https://epr.elastic.co/search?package=ti_opencti