Improving unit test developer experience

[alexveden]

Hi @lerno, thank you for a such great language, and I really appreciate the clarity and simplicity of c3 code. I'd like to discuss direction of unit testing part of the language and how to improve developer productivity using c3 test driven development approach.

I'd like to improve the unit testing experience of the c3, and I wanted to know your vision of the problem I'd like to solve, or if you consider them as problems at all.

List of improvements I think would be useful in unit test experience

(The ideas below are the subject of discussion, let me know what you think)

Setup / teardown test functions

It would be great to have setup/teardown functions for test modules (i.e. marked as module test_something @test;). Because in production code, sometimes we need to setup environment (DB, socket, mocks) for each test case in a suite and make some cleanup afterward.
Example:

Special test asserts

Using built-in assert maybe is not optimal for productive test driven development. For example, failure of this assert assert(6 == bar::mul(2,3)); produces an error:

 Error: Assert violation
  - in test_sub test_foo.c3:19.

However, it's not clear what value was returned by bar::mul(2,3)), to figure it out we need to add formatting to assert, like this: assert(6 == bar::mul(2,3), "failed: %s", bar::mul(2,3));. In my experience, maintaining of this approach quickly becomes a hell. So my proposal is to introduce special test asserts which include expected/actual value printing.

Conceptually, it could be a special macro for this, i.e. tassert_eq(a, b); or maybe test suite struct. Concept code which compiles and works below:

Having this value printouts on failures, significantly improving development and debugging velocity. And the developer gets ideas of what's going wrong much faster, with this error message:

 Error: a[3] != b[2]
  - in test_sub test_foo.c3:12.

Code asserts should produce backtrace

Debugging assert failure inside test function might be trivial, but when assert (or @require/@ensure) was triggered somewhere deep in the guts of the project, it's not trivial what caused the failure.
Simple example:

test_add fails with this obscure error:

Testing test_foo::test_add .. [error]

 Error: Assert violation
  - in add foo.c3:5.

It's not clear when it was triggered, especially if we would have multiple non-obvious calls of an asserted function:

Backtrace printing is the solution for this problem (I made some proof of the concept hacking of c3 test runner):

Testing test_foo::test_add .. [error]

 Error: Assert violation
  - in add foo.c3:5.


ERROR: 'Test case failed'
  in std.core.builtin.print_backtrace (/home/ubertrader/code/c3c/lib/std/core/builtin.c3:69) [./build/testrun]
  in std.core.runtime.test_panic (/home/ubertrader/code/c3c/lib/std/core/runtime.c3:192) [./build/testrun]
  in foo.add (/home/ubertrader/code/c3_advent/src/foo.c3:5) [./build/testrun]   
  in test_foo.test_add (/home/ubertrader/code/c3_advent/test/test_foo.c3:7) [./build/testrun] <<<<<< THIS
  in std.core.runtime.run_tests (/home/ubertrader/code/c3c/lib/std/core/runtime.c3:229) [./build/testrun]
  in std.core.runtime.default_test_runner (/home/ubertrader/code/c3c/lib/std/core/runtime.c3:248) [./build/testrun]
  in main (source unavailable) [./build/testrun]
  in __libc_start_call_main (./csu/../sysdeps/nptl/libc_start_call_main.h:58) [/lib/x86_64-linux-gnu/libc.so.6]
  in __libc_start_main_impl (./csu/../csu/libc-start.c:360) [/lib/x86_64-linux-gnu/libc.so.6]
  in _start (source unavailable) [./build/testrun]
Testing test_foo::test_sub .. [error]

The side effect of this, however, that all failed asserts will start producing back traces. So that's why we possibly need to introduce special test asserts.

Code asserts should be debuggable (in unit tests)

It would be great we could run debugger and automatically stop at assert trigger in the code. I did it a lot in my C projects, but c3 test runner is designed a bit different, which hardens debugability of unit tests. But I think it's a solvable issue.

Single test file runner

It's usually crucial in TDD cycle to work with dedicated test file, so it would be great if we could do something like this c3c test test/test_foo.c3. I'm aware of c3c option compile-test however, the downside of it that it requires to pass all other sources in order to compile the test.

Implementing c3 mocks

Sometimes mocks are crucial in building production applications, and sometimes we need to decouple code (for example mocking hardware related functions in embedded device). It's a very broad and complicated topic, however there are some solutions for C mocking (fff.h or C-mock). Or at least we might try linker wrap options sample gist.
At least for now, the mocking stuff might have a mandatory requirement of separating all tests in stand-alone executables. This is long discussion, we may touch it later down the road.

It is a long list, but I think that everything is doable. I can take burden of implementation. @lerno before I start I'd like to know your view on this problem. Maybe my proposals may interfere with minimalistic nature of c3, that's ok, I'll make my own project. But frankly
I think this should be a feature of the language.

[lerno]

Let's break this down, first regarding setup / teardown.

This is harder than you think. While you can add setup and teardown for ALL tests by using @init / @finalizer, there is an issue with adding setup / teardown "to a module".

Because a test does not need to have a module marked with @test. In fact it's just the module section that has @test as the default property of its functions. It doesn't mean that the module is a test module.

This means that there is no strict way to link a test and a setup. So unfortunately that has to be explicit calls. teardown is simple enough to add using defer teardown_something();.

[lerno]

In addition, I must add a personal note that I've not found setup/teardown that useful when writing tests in various languages.

[lerno]

Regarding asserts for testing. Those are trivial to extend from existing macros, and people are encouraged to add those. There are several missing asserts, not the least an assert that ensures that something does indeed fail as expected!

There was someone working on this, but it never got submitted to the stdlib. Pull Requests with improvements are encouraged!

[lerno]

Regarding better backtraces and better stops: The testrunner is intended to be pluggable. An IDE can integrate with the tests by passing an alternative test runner function for example. The current test runner is really a minimum viable product, lots of things could be made to improve the default test runner.

[lerno]

For Single Test Runner I am unsure of what you want to achieve. What do you want to single out?

[lerno]

Regarding mocks there is some work towards @weak functions that allows overriding in the same code base.

The idea is essentially that if you have

fn void foo() @weak { ... }

And then add another file with

fn void foo() { ... }

Then this is not considered a collision, but instead the latter completely replaces the former. Similarly we could imagine:

fn void foo2() { ... }

And then add another file with

fn void foo2() @override { ... }

Then the behaviour would be the same. That the latter is replacing the former. Could not this offer a good way to mock things?

[alexveden]

Let's break this down, first regarding setup / teardown.

This is harder than you think. While you can add setup and teardown for ALL tests by using @init / @finalizer, there is an issue with adding setup / teardown "to a module".

Because a test does not need to have a module marked with @test. In fact it's just the module section that has @test as the default property of its functions. It doesn't mean that the module is a test module.

This means that there is no strict way to link a test and a setup. So unfortunately that has to be explicit calls. teardown is simple enough to add using defer teardown_something();.

Ok, I understood. Maybe direct call + defer is the way to go, at least it fits Python's principle explicit is better than implicit.

While experimenting along those lines I've found that test's defer was not called after assert hit.

Consider this example:

MyTestSuite t @local = {
   .setup_fn = fn void() {
       io::printn("\nsetup func");   
   },
   .teardown_fn = fn void() {
       io::printn("\nteardown func");
   },
};

fn void test_sub() {
    t.setup(); 
    defer t.teardown(); // teardown func was not called when assert failes

    // NOTE: this fails
    assert( 1 == 0 );
}

[alexveden]

For Single Test Runner I am unsure of what you want to achieve. What do you want to single out?

At minimum, it would be great to have test output only for a single test module (maybe as a test case filters).

However, I think it worth discussing single program per test file build. This will help to isolate some big chunks of larger projects, and apply mocking with @weak/@override attributes. For example, we have 3 modules foo, bar, db , foo uses bar and db, bar uses db. db requires side resources and initialization. Sometimes it's needed to test some aspect of foo without initializing a whole chain of modules (bar / db). Sometimes it's needed to mock some function, etc. So the issue raises when we start building all tests into single executable, so we might have linking conflicts between mocks and real functions (some tests requires real functions / some require mocks).

So having 1 executable per test may solve these types of conflicts, this will make testing more flexible and hackable.

[alexveden]

I'm trying to add more colors into boring life of test suites, but it seems I faced escaping problem of some kind:

This works in C, and in terminal also:

in c3

io::printfn("[\033[0;31mFAIL] ^^^ %s ( %s:%s ) %s")

it produces raw escaped sequence:

I tried raw strings in c3 also, but with the same effect.

EDIT: follow-up question, is that possible to pass arguments from c3c CLI to test suite somehow? For example, adding an argument that makes assertion failure to become a breakpoint would be useful. Or silencing individual cases and run only on module level and aggregating total results.

[lerno]

No octal escapes are supported in the formatter. In this case \e will do the same job as \033

[lerno]

Remember to check that the terminal accepts ANSI codes before sending them.

[alexveden]

@lerno Can we discuss test macros naming question? I rejected the idea of having TestSuite struct entity, because it feels as mental overhead to me and unfit to single @test functions. So I came up to conclusion that tests should be namespace based, and I have 2 options:

To me, option 1 has more c3 flavor, and it's less to type also :)

[alexveden]

@lerno could you suggest a proper way of adding CLI parameters to the test runners? I was experimenting with default runner, and was able to add some useful features:

1. test case filter - run only tests with some substring match in case name
2. test may throw a breakpoint() when any assert failed (including deep in the code asserts) - for the purpose of increasing debug cycle

There are other ideas floating around, for example quiet test mode (suppressing stdout verbosity of tests cases and possibly user code).

Conceptually, test filter may work like this (filter is real, but CLI parameter is fake):

[alexveden]

I altered test report layout to solve the following issues:

1. Assert in deep code doesn't reflect meaningful info in the test failure message
2. User code io::printf() breaks test output
3. Made all error failures at one line. I think it's easier to parse them by regex, and easier to read
4. Colors improved readability
5. Moved FAIL/PASS label to the left, because it's easier to read, most important information comes first.

@lerno let me know if this looks good to you. thx

[alexveden]

I think I've figured out the setup/teardown business too (without defer), but works when asserts failed too.

fn void test_sub() {
    test::setup(
        // mandatory setup function it's called immediately
        setup_fn: fn void!() {
            io::printfn("setup fn");
        },
        // optional teardown function it's called after test finish (or failed)
        teardown_fn: fn void!() {
            io::printfn("teardown_fn");
        }
    );
    test::eq(3, foo::sub(3, 1));

Produces the following:

[lerno]

This is looking great, sorry for not getting back to you in time. I'll look at this tomorrow.

[alexveden]

I added sample c3test project: https://github.com/alexveden/c3test

This is an example, of how a sample unit test will look like:
https://github.com/alexveden/c3test/blob/main/src/sample_test.c3