Extend test matrix to cover more possible build combinations of features

[Jakuje]

With the amount of features the kryoptic supports, keeping track that they are self contained and all of the dependencies is quite tedious work.

This extends the test matrix and updates the tests to be able to run with different storages. For now, jsondb, sqlitedb and nssdb work, but the memorydb does not. Additionally, I encountered the issue that the memory db is not possible to provision, unless encryption is turned on, because it tries to find PIN counters, which are not in non-encrypted storage.

Fixes: #118

[Jakuje]

This is mostly ready for review. The only thing to figure out is the issue with the memorydb and why the jsondb fails in CI, but works locally.
If needed, changes can be merged and CI changes could land later after we will figure out last details.

[simo5]

Looks largely ok, only minor nits.
Thanks this is exactly where I wanted to see things headed!

[tomato42]

The idea is to be comprehensive in the coverage, or just have at least some test coverage?

because I has some experience with all-pairs testing and extension of it to triplets, quadruplets, etc....

[Jakuje]

The idea is to be comprehensive in the coverage, or just have at least some test coverage?

because I has some experience with all-pairs testing and extension of it to triplets, quadruplets, etc....

Right now, we have around 25 features:

https://github.com/latchset/kryoptic/blob/main/Cargo.toml#L49

They have some internal dependencies and then there is the "default" feature which we want or dont want to build. I was worried that doing all pairs (or triplets) would explode or would be even more work to exclude meaningless ones.

[tomato42]

The idea is to be comprehensive in the coverage, or just have at least some test coverage?
because I has some experience with all-pairs testing and extension of it to triplets, quadruplets, etc....

Right now, we have around 25 features:

https://github.com/latchset/kryoptic/blob/main/Cargo.toml#L49

They have some internal dependencies and then there is the "default" feature which we want or dont want to build. I was worried that doing all pairs (or triplets) would explode or would be even more work to exclude meaningless ones.

all pairs (n-tuples) testing is especially effective if you have a lot of features, as then you avoid the combinatorial explosion

and no, having meaningless combinations is not a problem, when you generate the combinations you can exclude combinations that are either impossible to support (like testing ECDH on build without ECC) or combinations we explicitly don't want to support (for whatever reason)

[simo5]

The idea is to be comprehensive in the coverage, or just have at least some test coverage?
because I has some experience with all-pairs testing and extension of it to triplets, quadruplets, etc....

Right now, we have around 25 features:
https://github.com/latchset/kryoptic/blob/main/Cargo.toml#L49
They have some internal dependencies and then there is the "default" feature which we want or dont want to build. I was worried that doing all pairs (or triplets) would explode or would be even more work to exclude meaningless ones.

all pairs (n-tuples) testing is especially effective if you have a lot of features, as then you avoid the combinatorial explosion

and no, having meaningless combinations is not a problem, when you generate the combinations you can exclude combinations that are either impossible to support (like testing ECDH on build without ECC) or combinations we explicitly don't want to support (for whatever reason)

In most cases each algorithm is fully independent from each other algorithm, so that testing them in pairs makes little sense.

We have a core that is always involved and therefore is always tested.

Then we have the algorithms which just operate on specific data, regardless of the core or anything else.

The only major variability is the databases, because what they contain may influence the result of specific tests.

So perhaps testing each "true leaf" algorithm on their own with each database may make some sense. Mostly to ensure there is no unexpected dependency between algorithms, but that is rather simple to verify so I would not want to spend a lot of CPU on that, and definitely not make CI much slower due to this.

Perhaps we can have matrix testing done once a week on a scheduled CI run just for completeness.

[Jakuje]

warning: method dbsuffix is never used

With recent changes the dbsuffix is not used anywhere. I kept it for now because it might be useful in the future, even though it works only for SQL and JSON. If you do not agree, I can remove it altogether.

[simo5]

warning: method dbsuffix is never used

With recent changes the dbsuffix is not used anywhere. I kept it for now because it might be useful in the future, even though it works only for SQL and JSON. If you do not agree, I can remove it altogether.

We can always resurrect from git, I am ok dropping anything that is not used anymore.

[simo5]

Only a couple of nits, but great job with this, thanks!!

[simo5]

Thanks for this work, really appreciated!