Tests discovery strategy

[eddycharly]

Chainsaw supports several test syntaxes and this makes tests discovery complicated and error prone.

It would be nice to make this process simpler and easier to reason about.

To me it looks like:

• Tests written with raw manifests:

  • are good for playing with the testing tool
  • are not so good when it comes to overriding some config elements (timeouts for example)
  • don't support all operations (commands, delete, etc...)
  • are not easy to maintain in the long term

• Tests written with Test resource:

  • require a bit more work/typing but we can make this easier with a CLI command
  • support all operations
  • can contain descriptions to build docs and tests catalog, makes tests easier to maintain in the long term
  • well suited for overriding config elements like timeouts

While the Test based syntax requires a bit more work it has strong advantages over raw manifests syntax.

Moreover, once we introduce test templates it will be harder to support raw manifests syntax.

Now the question, do you think supporting raw manifests is worth it ?
We would like to hear from you to make sure we move in the right direction.

[shubham-cmyk]

I believe having raw manifest is easier to get in and write compared to the Test resource in which you have to spend time.
For a basic operator that only requires assert, error, and apply, the raw manifest is a perfect fit.

Why can't we have the template for the Test and continue supporting the raw manifest as well?

[eddycharly]

The difficult part IMHO is combining both, having a Test resource AND raw manifests.

I find the user experience deceptive, like you start with raw manifests, then you need to change the timeout at some point, and then you must convert to the Test syntax... it would have been easier to start with the Test approach from the beginning.

If we add the TestStep in the mix it gets even more complicated.

Let's say you join a team and inherit a couple hundred tests, all written with a different syntax, it sounds difficult to find your way through all the tests vs a single Test based syntax.

To me the complexity of writing a Test (come on, it's not that hard in practice) can be overcome with a bit of tooling, we can add cli commands, generate schemas to help with autocomplete/validation, etc...

And we also have a command to convert raw manifests tests to Test based tests.

WDYT ?

[shubham-cmyk]

I never used both of them combined. ( test and raw manifest )

I have mostly used TestStep + Raw Manifest setup for me switching to Test resource is not a big-deal.
Also if we could have autocomplete it would be good. Test resource is most verbose here so naturally it should be able to perform everything.

But I see only reason for the raw manifest is for someone who has just started on chainsaw is playing and exploring and one with very less requirement

[eddycharly]

I think the same, to me raw manifests makes sense only when starting with the tool.
As soon as you start using it more seriously there's a good chance you will need more features and start with Test or TestStep.

[fab7]

FYI, I am trying to assess the use of Chainsaw to validate the configuration and status of deployed clusters. For this use case, I started with the Test approach, and I am only using this syntax which I like most because of i) its explicit sequential behavior, ii) its full support of all operations and, iii) its flexible interaction with the system via Script and Command. These allow me to write much longer test scenarios and to remain in the "Chainsaw domain space" once a test is started.

Regarding the raw manifest syntax, I find it of interest for people who come from a previous KUTTL experience (which is not my case).

[shubham-cmyk]

You are right people you are coming from the KUTTL could find a convenience in using the similar syntax

[eddycharly]

I'm on the same page here.

But supporting raw manifests only because it's the way KUTTL works is not a valid argument IMO.

If KUTTL didn't exist at all, would Chainsaw support that ? Do we see value in supporting it if we take KUTTL apart ?

[eddycharly]

And I think KUTTL users are already well covered with chainsaw kuttl migrate tests and chainsaw kuttl migrate config commands.

[eddycharly]

How about removing TestStep CRD ? It could a first step in the right direction...

If we find a Test we use that, else we fall back to raw manifests ?

cc @fab7 @shubham-cmyk

[shubham-cmyk]

@eddycharly If you remove this won't is affect the Kuttl migration that we offer in the chainsaw

[shubham-cmyk]

I think this is a good idea Keeping 2 scenarios would be a good idea since most of us would use the Test.
+1 For this

[eddycharly]

i changed the kuttl migration command to output a Test, it doesn't use TestStep anymore.

[shubham-cmyk]

Right. I agree on this we can remove this. I would like to hear it from more users about this

[eddycharly]

Closing, everything was done.