The Testrunner saga

[vipulgupta2048]

We have been thinking and rethinking what exactly is Leviathan, its purpose, the product, and why we need it.
While the thinking happened to answer questions, we came up with even more questions.

Several documents and one hack a week later. I think we have finalized what we intend to do.

Architecture change TLDR

Components consist of:

1. A very thin client that is used to send/receive test artifacts and trigger tests
2. A testrunner to accept the artifacts, setup, execute the test suite, teardown at the end of it and provide results to client somehow.
3. AutoKit, QemuKIT that does the interactions with the DUT
4. The DUT

Story

Thin client running a node script to trigger tests with a CLI interface.

The worker is testbot with only balenaOS on it. The interface gets triggered here and the upload is started using rsync or wget a image from the public web. The suites and config are brought using rsync. A container is pulled using the balena engine. The image contains test runner and everything it needs to run the test suite. The suites import the Worker node package and that is how the workers get access to the artifacts. The test runs and uses the Worker API to control the DUT.

Logs and test results are passed back to the thin client at the end of the test.

The how

Thin down the client #729
Build testrunner on the testbot
Publish NPM package for AutoKit (contains, testbot, worker, and qemu classes)
Enable users to write their tests with the autokit package

stretch goals/discussions

core/testrunner - delete helpers (except the HTTP bindings for the worker)
client - parellelisation
client - rework the upload to use rsync
core/testrunner - dockerfile optimisation