Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide an expected failure "code" or "message" in testcases.json #820

Open
oxisto opened this issue Oct 31, 2024 · 1 comment
Open

Provide an expected failure "code" or "message" in testcases.json #820

oxisto opened this issue Oct 31, 2024 · 1 comment
Labels
csaf 2.0 csaf 2.0 work csaf 2.1 csaf 2.1 work test related to tests in this repo

Comments

@oxisto
Copy link

oxisto commented Oct 31, 2024

This is a follow-up to the discussion in csaf-sbom/kotlin-csaf#80 (comment). We make use of the TC test cases JSONs in testing the validation tests in kotlin-csaf. While exploring options to keep tests in sync with the TC, we came up with the idea whether it is possible to include an expected failure "message" in the testcases.json. This way we could align the error messages in our library with the official ones and we could query the testcases.json to automatically assert good and bad test cases even with the appropriate error.

Further discussion points

  • Does it make sense to introduce some kind of official error codes for the validation?
  • Should it be mandatory for CSAF libraries to use these error codes or messages

This would make developing compliant CSAF libraries much more easier, sort of like a Technology Compatibility Kit.

@oxisto oxisto changed the title Provide a failure "state" or "message" in testcases.json Provide an expected failure "code" or "message" in testcases.json Oct 31, 2024
@tschmidtb51 tschmidtb51 added the test related to tests in this repo label Oct 31, 2024
@tschmidtb51
Copy link
Contributor

We need to take into consideration, what is already existing, e.g. the codes used by the csaf-validator-lib.

The instancePath is crucial as it provides the opportunity for viewers and editors to highlight it message in the right place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
csaf 2.0 csaf 2.0 work csaf 2.1 csaf 2.1 work test related to tests in this repo
Projects
None yet
Development

No branches or pull requests

2 participants