Tests fail due to ECS expected values

[chrisberkhout]

Expected values defined for an ECS field, such as for threat.indicator.type, are not the only valid values (def, emphasis added):

expected_values (optional): An array of expected values for the field. Schema consumers can validate integrations and mapped data against the listed values. These values are the recommended convention, but users may also use other values.

The expected_values field was intended to indicate the "SHOULD" requirement level (as defined for RFCs). For this level it may be appropriate to warn about the use of other values, but not to fail a build or test run.

Currently, pipeline and system tests will fail if they find values other than the expected ones.

A previous workaround when using ECS dynamic mappings was to override with explicit definitions. That makes tests pass now if those definitions include the full set of values or an empty array (example), but now elastic-package will not successfully build a package if it defines its own expected_values, so this workaround no longer works.

Some related links:

• Slack discussion from when the workaround still worked
• Code that does the validation
• PR introducing that code
• Some changes to validate.go earlier this year, regarding import of the ECS schema
• A recent ECS validation PR
• New integration PR blocked by the need to check, build and test with non-expected values for and threat.indicator.type and threat.indicator.name (although threat.indicator.name is a misuse of expected values and will be converted to examples)

[chrisberkhout]

As discussed today with @andrewkroh and @P1llus, and earlier with @kgeller and @ebeahan.

[jsoriano]

Thanks for the detailed description.

The expected_values field was intended to indicate the "SHOULD" requirement level (as defined for RFCs). For this level it may be appropriate to warn about the use of other values, but not to fail a build or test run.

By now I would stay on the restrictive side of things. I don't see other packages using expected_values in integrations.

In any case, it sounds good to allow to define expected_values per field. I am opening elastic/package-spec#616 by now to support that.

Also, once elastic/package-spec#558 is done, we can also add support to selectively disable this validation.

[chrisberkhout]

IMHO breaking a build due to a valid though not expected value is something I'd rather opt into than have as default.

It's not critical for me now since I found a variation on my earlier workaround that still works.