Checking whether testoutput is correct WITHOUT the help of external resources?

[Aliazzzz]

Hi,

The testsresults of the verifier are now checked with the help of an external resource in order to verify it's outcome on accuracy. It should in theory be possible to do this without an external resource. My suggestion is that we put some effort in trying to investigate this idea.

Aliazzz

[sagatowski]

Sounds good. Investigate this when you've got the time!

[Aliazzzz]

I think the following solution suits our needs very well and should be not very difficult to implement.

The idea;
It should be easy to verify the end-result via an internal pre-calculated hashcode e.g. provided as a project constant (hash-algorithm yet to be decided);
The hash algorithm just takes all success and fail bool results and derives a hash from it (not the count, but the bool array!)
The newly calculated hash should then in theory correspond with the hash we have already provided pre-calculated. This way people can be sure that the framework works correct as long as they did not expand the tests (changing the end results and thus the hash input). We just have to warn that changing the verifier sourcecode is at their own risk.

A minor downside to this method is every time the project gets expanded or changed by us (tcUnit project), the pre-calculated hashcode has to be re-calculated. This however is trivial and beats an external tool by far.

Also, the hash algorithm does not to be some fancy 1024bits algorithm with as few collisions as possible. A simple CRC32 or so would do since we have to calculate and check a single value. The chance that changing tests (=results) in the the code yields a similair hash is unlikely.

[sagatowski]

I don’t understand the bool array?... which ones is that?

If you do a change of the output, how do you know the expected hash-value for a certain set of tests? What happens if you change some functionality, and the hash-value changes, how do you know where the error is?

This is some of the information the external tool provides today. It can pinpoint exactly what does not work… how will this be accomplished with hash-values? Will the hash-values somehow provide information specifically what does not work?

Sorry, I might just be tired but I don't understand how this is going to work :-)

[Aliazzzz]

If the test has been run, the output can be taken to calculate a unique fingerprint.
If that fingerprint matches the one we stored pre-calculated in the project, the test outcome is correct.

If someone is 'tinkering' with the tests (especially the outcome of a test) the fingerprint is hopefully not equal anymore to the pre-calculated one.

The bigger the hash and the more inputs it has to evaluate, the better the detection algorithm will work.

The beauty of this mechanism is that it assures that the outcome is either 100% correct or not correct at all.

What I just added to the original idea I had;
Plus if the hashcode proces is reversible, the original expected output can be recalculated and checked against the current output results. This will determine which test is faulty, if any.

[Aliazzzz]

In order to let this work, a way is to take all testsuite output into the hash algorthm. Every test has a struct in which that test information is stored.
The functionality which we should develop takes all these results and calculates a hash and verifies this against a locally pre-stored hash.

[sagatowski]

Does this also take into account the tests that are NOT supposed to output anything?

I think this would be interesting to see. Appreciated if you create a fork and do an implementation (at least for one test suite?), with a combination of both data that is outputted and data that is not supposed to be outputted!

[Aliazzzz]

I presume that as long as the testresults of such tests are always consistent, we should be in the clear; My asumption is that even tests that are not supposed to output anything, still output something. If that something (result) is consistent, the fingerprint should be consistent. The goal is to have each test contribute a small piece of information (which can be nothing) to the fingerprint.

[Aliazzzz]

CRC background info;
By convention the output value for a CRC is called a “checksum”, and the output value for a hash function is called a “digest”. CRCs are a type of error-detecting code used to implement checksums.
CRCs are specifically designed to satisfy the property that they can detect transmission errors in data. The idea is that you’re transmitting a signal over a potentially lossy medium (e.g. Ethernet or radio waves) and you want to detect any data loss or data corruption. The CRC is computed over the data payload and sent as part of the data transmission. The receiver can recompute the CRC and compare it to the received value to detect errors. No property of the CRC matters other than whether it differs when data corruption occurs.

For our purpose, CRC16 or CRC32 will suffice. I presume a CRC function is provided for TwinCAT/CODESYS via open source OSCAT library.

[Aliazzzz]

Plan:
Add a final assert test per testsuite:
This final test asserts the determined total endresult (actual) of the testsuite and correlates it against the pre configured total endresult, set by us (expected).
This should yield no extra messages in the log if everything is okay.
If the user has rewritten test/code it will result into a new log message (assert failed).

Algorithm

1. The final test per testsuite determines a CRC16 or RC32 checksum out of the former testsuite results.
   In order to ensure a solid fingerprint of the test result we take into account as much info as possible of the testsuite (results per test and total results of the testsuite)
2. We should also provide a predetermined hardcoded expected end result into the code as a local variable.
3. Assert if the checksum from 1 (actual) and 2 (expected) match, the original input data is identical and thus we show no log message.

We could try optimize the algorithm for more integrity in various ways, for example by combining the n-1th testsuite result with the n-th testsuite result. This way the testsuite results become interdependant on another. A penalty is added complexity of calculation for current and predetermined CRC fingerprint.

Any other idea's or tricks are welcome!

[KubaO]

What is this "external resource"? Does this refer to the C# project?

I don't quite understand why is this C# project needed at all, other than as a runner for continuous integration. It probably should not do anything other than letting TwinCat run the test project and then forward a result code out of twincat, via ADS.

Here's how I see it (and I'm totally willing to implement it - it's really a small change):

1. The verifier C# project launches the TwinCAT project just as it currently does.
2. The verifier PLC project uses ADS to introspect its own log (i.e. this doesn't require any changes to TcUnit).
3. It computes the hash of the log.
4. It compares the hash to a precomputed value.
5. It outputs a log message indicating the match/mismatch.
6. The C# verifier notices the final log message, outputs it verbatim, and returns an exit code of 0 on match or 42 on mismatch. That can be used in a CI environment for automated tests.

[Aliazzzz]

Preferably it's a standalone PLC solution i.e. it should not depend on any external C# program. The PLC program itself should contain the pre calculated hash allready.. That way the code is truly independant

[KubaO]

Why though? The tests are supposed to be automated. You add the new functionality, update the “golden” expected output, then let the CI do the rest. Remember that this solution should eventually scale to testing any PLC project, not just TcUnit, and any number of PLC projects, since automation matters when you got dozens or hundreds of them.

I don’t want to test 30 PLC builds manually by clicking something in the dreadfully slow TwinCAT. The CI system can spin up VMs quickly and do all that in a minute or two, even for hundreds of projects… this should be hands-off, and running anything on the PLC is not hands-off without tooling that automates TwinCAT actions.

Ideally though, there’s a way of running the compiler and build without starting the Visual Studio hidden instance (the IDE just uses .net libraries that 3S Systems and Beckhoff developed), that is long-term I see this tool using just the libraries from TwinCAT and ADS to upload the files to the PLC and run it. The uploading and PLC logging and state changes are semi-documented and easy to do. It’s only getting a stand-alone build going that needs figuring out a bit.

[Aliazzzz]

I think we are misunderstanding each other. I will read your question more thoroughly and come back with an anwser.

[Aliazzzz]

I agree with you that portability is of the highest order, I only want to steer away from any dependancies that only Beckhoff can leverage like a tight integration with C#. Sure Beckhoff offers this for free, however 3S CODESYS does not(!). At least not to normal customers (OEM only) and certainly not not for free.
In my view the should be as flexible and as manufacterer independant as possible. I am hoping to converse further with you on a scalable but manufacturer independant solutions for the framework.

Mind you that CfUnit is a Codesys based fork of TcUnit and is allready brand independant.
https://forge.codesys.com/svn/prj,cfunit,code/landingpage/index.html?raw#Download

It just needs a good door to external CI/CD systems ;-)

[KubaO]

An aside: native integration with CI/CD from someone like 3S is frankly said maybe coming in a decade. Personally I see all this work as a stop-gap. Some company will come around and make 3S obsolete in a couple of years, sort of like SpaceX made a lot of legacy launch providers obsolete.

[Aliazzzz]

I am hoping to converse further about scalable but manufacturer independant solutions for the framework. Would you mind to explain your suggestion as simple as possible?

[KubaO]

I don't think that this framework is usable at scale without an automation interface of some sort. Whether Beckhoff provides it or 3S (CoDeSys) doesn't matter, but nobody should be expected to be running this manually on a PLC. So the PLC code is just the "backend", whereas the front end is whatever code initializes the PLC and gets the unit tests to run there. In other words, the "manufacturer agnostic" solution only makes sense for small projects, like a single machine with one or two PLCs, or maybe a library that is tested by the developer. And the developer should be running the tests on a CI system anyway (if it's a serious developer), using git or svn hooks.

The ADS-output-independence needs to be implemented in TcUnit (#3), and then the verifier just needs to provide a plugin that will consume the log output directly from TcUnit, as the tests run. The verifier can then use a variety of techniques to check whether the output is correct - or it may even do nothing much, just compute a hash and pass it to the front-end. The hash generator would need to be one of the "plugins" provided by TcUnit, not the verifier, since the verifier PLC code is just a test harness - for a specific project, but that's very similar to how any user would test their own code, except they'd be testing something else than TcUnit. There should also be an option (for convenience) to call ADSLOGDINT to pop up a message box with success/failure when the tests are run interactively, if the user so wishes.

The C# project still has to be used to run the verifier and check some sort of an output code. For example, imagine this:

1. Run verifier on the solution. Verifier outputs a hash code of the test run's output, e.g. 012bf86c3068b563d128207d53663213, and returns with an result code 10 (this code can be used by CI scripting).
2. Run verifier with a -h 012bf86c3068b563d128207d53663213 command line option. If the hash of the output matches the hash computed by the verifier PLC project, result code is 0, otherwise it is 1 (again, to use by whatever script calls the verifier).

Once that's done, the verifier front-end has a general use, and is not limited to running the TcUnit tests. We'd probably need to rename it as TcUnitRunner or something like that. It'd need a couple more command line arguments, to e.g. choose the ADSnetid of the target, optionally add a temporary route to the target, etc.

[Aliazzzz]

"I don't think that this framework is usable at scale without an automation interface of some sort."
I Fully Agree!

I have written (with help of a few people) a CODESYS fork based on TcUnit named CfUnit.
CfUnit/TcUnit is nice and good enough for simpler projects as you have allready noticed. Scalability is key in the evolution.
My only hope is that such an evolution is not based on dependancies like microsoft C# (beckhoff only) or ADS (beckhoff only) as they vendor-lock the solution, they are not truly manufacturer agnostic. People running WAGO, Schneider, ABB or any other CODESYS based platform can then forget leveraging this great piece of software, which actually contradicts the openness and spirit of the basic idea.
The framework (TcUnit/CfUnit) is better off if we think of a enhancement which opens up the framework to any CI/CD system by using some simple but manufacturer independent trick or mechanism.

My view on it is as follows;

Jenkins
The External CI/CD framework (whichever you run) should be able to launch the test in the Framework (PLC) through the IDE (Visual Studio or CODESYS IDE). This can achieved in various ways via a simple script or bat file. The test output should also be made available in a xUnit.xml compatible file. which is picked up by the CI/CD system. The CI/CD framework can then consume the output file and continue on whatever is scripted.

Drone
A simple yaml file is all that is needed.

Resume
If you wish to extend TcUnit with such a mechanism my only wish is that I can port it to CfUnit so that other people who do not use Beckhoff can profit also.

Idea!
Maybe Jenkins/Drone can be extended with an OP-UA client, that makes the IT/OT translation so much easier, as with such a solution we do not need to extend this framework anymore.

[sagatowski]

What is this "external resource"? Does this refer to the C# project?

Yes, it's the C# project.

I don't quite understand why is this C# project needed at all

The purpose of the project is described here:
https://github.com/tcunit/TcUnit-Verifier/blob/master/README.md
The requirements for the project (so far) is that it needs to:

1. Verify that the TcUnit works as expected, with the aid of the TwinCAT project TcUnit-Verifier. It does this by making sure that the verifier outputs certain (pre-defined) outputs.
2. Verify that TcUnit does NOT output certain messages
3. Verify that TcUnit works as expected in the different builds (as there are some discrepancies in how TwinCAT works between the different builds)
4. And do so in such a way that it's easy for a developer to see exactly where things have gone wrong (by a message saying exactly what has gone wrong)

If there is a way to do all of the above without the C# program, I'm still not sure whether I'd want to remove it as the C# gives me an extra level of flexibility for any future requirements.

Some company will come around and make 3S obsolete in a couple of years

I highly doubt this. There are hundreds of brands using the 3S products within the field of industrial automation, that in them-self many have been around since the '70s (obviously though not with 3S then). The field of automation is moving considerably slower than what you might be used to. Things like CI/CD is largely unknown in this field. Heck, I would guess most companies aren't even doing proper version control! That some company will include CI/CD in the toolchain and processes is not going to make the thousands of 3S users switch to another brand. The introduction of CI/CD (amongst many other "modern" software processes/tools) is a process that will grow gradually over long time, and I'm sure that 3S (and Beckhoff) will catch up too once there is a critical mass big enough.

but nobody should be expected to be running this manually on a PLC.

Running it manually on your computer is the first step in this development, just as you can run any other unit testing framework locally. Throwing a framework at the industry that has not (from a software engineering perspective) not been maturing much for a long time is quite a big step in its own. Making the industry understand the benefits of test driven development (TDD) is a first. Remember, a lot about this project is not about the technology in itself (there are a lot of things that can be wished and done for), but rather making developers understand the benefits of TDD altogether.

And the developer should be running the tests on a CI system anyway (if it's a serious developer)

The introduction of TDD into the field of automation is a gradual process. First the concepts need to be properly understood. Next it's necessary to show how to do tests and run them on your machine. Naturally the next step is to introduce the tools/guides for a complete CI/CD toolchain in order for it to scale well. Personally, I've been running all tests locally even for a very large project (the extremely large telescope), and even here it's been running fine even though everything will get massively better once CI/CD processes/tools are in place.

We'd probably need to rename it as TcUnitRunner

For some time I've been working on a tool with exactly this name (well, more exactly TcUnit-Runner), but there is still a lot of work to be done. This software will enable executing TcUnit and support result collecting with the xUnit results, see the two (already existing) issues (that were created almost a year ago, time flies fast!):
#7
#11

Also, I'm really happy with all the talk of ADS-independence, but the ambition of TcUnit has always been to be a unit testing framework for TwinCAT. The first priority is to have a complete CI/CD toolchain (with all the accompanying documentation/eventual instruction videos/example code) for TwinCAT. With this I'm not saying I see it stopping there, and that's the whole reason I'd for example want to use an interface for the logging (so it might be possible to exchange the ADS for something else easily). To add support for all the other PLC brands (+ making the crazily time consuming documentation/instruction videos/example code) is just currently out of scope until it's working for TwinCAT. I want to make sure to package this next step nicely so that the industry can look at this and say "hey, I get the point! I want to try this!".
What I guess I'm trying to say is what I'm trying to achieve with TcUnit is not just to show of the technology, it's much more important to show the benefits of working this way and to do this things have to be packaged in a way that automation engineers can look at it. The biggest challenge is the hurdle to make automation engineers understand the benefits of TDD, which has been one of the ambitions with TcUnit.

With this being said, I'm not against trying to remove the Beckhoff dependencies. I'm all for you to create working software/PRs where the current solution can do other things than ADS (and I'm more than happy then to include them! the reason I for example created issue tcunit/TcUnit-Verifier#3 over in TcUnit), but with the harsh reality of time constraints the focus now is to have a fully working solution for TwinCAT.

[Aliazzzz]

1. There is this CryptoLib to calculate hash value's
2. Maybe OSCAT can give us some functions.
3. The new FB_TestResults can be used for calculation of the hash.

[sagatowski]

Where is the CryptoLib available? I could only find something for codesys (though not sure). Is it available for TwinCAT as well?

[Aliazzzz]

CmpCrypto v3.5.15.0. for CODESYS. I highly doubt this is "rebadged" into TwinCAT by Beckhoff yet as they are at SP13. We should search the TwinCAT libraries for other usefull stuff. Plus we should search for an open source alternative to calculate a HASH code.

[Aliazzzz]

"MD5_STR" from OSCAT NETWORK seems to fit;

Input 
   RUN: BOOL (Positive edge starts the calculation)
Output 
   DONE: BOOL (TRUE if calculations are complete)
   MD5: ARRAY[0..15] OF BYTE (current MD5 hash)
I / O 
   STR: STRING(string_length) (Text for HASH creation)

Example:
the MD5 hash of 'OSCAT' is 30f33ddb9f17df7219e1acdea3386743

We should only agree upon a few things:
What information are we going to feed it?
My suggestion would be to feed in every testresult only (succes, fail or skipped).
Plus how is this information string going to be composed (nomenclature of the string)?
Suggestions?

[sagatowski]

I don't have any opinions on the technical solution as long as it fulfills the requirements I have mentioned (the ones fulfilled by the .NET Verifier).