[feature request] Test execution and reporting controller

[ivelichkovich]

It would be nice to have a supported way to package ginkgo tests and run them with various configurations using a "canary controller" (simple pod wrapper of ginkgo executions) that reruns tests on some configurable interval and then exports results in various configurable ways initially maybe as just prometheus metrics. I'm not sure if this belongs in ginkgo but it seems like a pattern that could be useful for others so looking into open source options for building this rather than just doing it internally.

[onsi]

thanks @ivelichkovich

the simplest way to do this would be to orchestrate (i.e. shell out to) the ginkgo cli to have it run suites for you on demand. this gives you clean isolation, will support parallelization, and allows you to write ginkgo suites as you're used to. of course the downside is the need to distribute your binary with additional test binaries (you could include precompiled test binaries, it wouldn't have to be source code, but I get that this is ugly).

If you want a way to invoke ginkgo specs in-process as code this is possible but we'd need to add some additional tooling to make it a decent experience. The only thing that would be very challenging to do would be to run the specs in parallel. This is because Ginkgo's shared memory model (the different closures in a container hierarchy communicate via shared variables) doesn't translate directly to an in-process parallelization model. But if you're ok running specs in serial then read on.

Today, when call Describe or BeforeEach or It, Ginkgo creates a node and attaches it to the current testing tree in a single global Suite object. This is all done under the hood for you so you don't have to bother passing a Suite instance around. Since Ginkgo's primary mode of operation is as a test runner of go test suites this limitation works just fine: each test package is assumed to exist in one singular suite so globbing stuff onto the share global variable as the code is initialized makes sense.

This gets tricky, however, if you want to use Ginkgo as a library and (potentially) have multiple suites - or be able to run a single suite multiple times.

Ginkgo's own internal_integration suite solves this by (swapping out the global suite)[https://github.com/onsi/ginkgo/blob/master/internal/internal_integration/internal_integration_suite_test.go#L75-L82] object on-demand (look here for an example).

Along these lines I could imagine a new package under ginkgo/extensions that would let you do stuff like:

import "github.com/onsi/ginkgo/extensions/suite"
...

// the entry point for our code - this will construct
// a suite, configure it to use the passed-in label filter
// and return a report
func RunSpecs(labelFilter string) (spec.Report, error) {
    s := suite.New()
    gomega.RegisterFailHandler(ginkgo.Fail) // register Gomega's global fail handler

    //build the test tree - you can imagine pulling the callback into a separate package
    // you could call AppendSpecs multiple times
    s.AppendSpecs(func() {
        Describe("specs go here", func() {
            It("...", func() {
                  Expect("foo").To(Equal("bar"))
            })
        })
    })

    // grab and reconfigure the suite configuration    
    conf :=  s.SuiteConfiguration()
    conf.LabelFilter = labelFilter
    
     // run the suite
     passed, report, err := s.Run(conf)
     return report, err
}

There are some important caveats here - since AppendSpecs must swap out Ginkgo's global suite object you must not call AppendSpecs in multiple goroutines at the same time. Also since Gomega's assertions play a similar global trick you must not call s.Run in multiple goroutines at the same time.

We could implement workarounds for both of these limitations but they would require you to pass the suite object and a particular gomega instance around. I think this could get ugly quickly (but it's the sort of pass-things-around boilerplate that Go users are used to):

func MySpecs(s suite.Suite, g gomega.Gomega) {
    s.Describe("specs go here", func() {
        s.It("...", func() {
             g.Expect("foo").To(Equal("bar"))
        })
    })
}

As you can see the caveats start to pile up and I worry that a user new to all this will struggle to make sense of it.

---

So i wonder if we should explore some alternatives. Perhaps a better approach would be tooling that can handle calling ginkgo for you and returning the report. This would allow you to run things in parallel and to reuse suites in various contexts. It doesn't solve the packaging and distribution problem - but perhaps there are some creative options there? How crazy would it be to precompile the test binaries and the ginkgo cli and embed them into your executable (answer: probably very crazy).