Add initial version of firmware-test-builder
Introduce the following features to firmware-test-builder:
* Testing on the host machine (PC)
* Provides CMake functions for building/running/evaluating tests
* Checking prerequisites (git, c-picker, libclang, etc.)
* Fetching and building CppUTest
* Defining and building unit test suites (i.e. separate binaries)
* Handling c-picker based extraction of code snippets
* Registering test suites to CTest (CMake's test system) which runs
all the test binaries
* Generating coverage report
* Documentation of the system
Signed-off-by: Imre Kis <imre.kis@arm.com>
Change-Id: Ic0a1af55bef07c6e76071193caa94a9a48f9041f
diff --git a/docs/components/cpputest.rst b/docs/components/cpputest.rst
new file mode 100644
index 0000000..e1c34a9
--- /dev/null
+++ b/docs/components/cpputest.rst
@@ -0,0 +1,281 @@
+CppUTest
+========
+
+This document is based on CppUTest v3.8. CppUtest is a unit testing framework for testing C and C++ code. This document
+introduces the basic features of the framework. For further information check the `official manual of CppUTest`_.
+
+
+Why CppUTest?
+-------------
+
+First of all it was not our goal to develop a new unit testing framework while plenty of open source solutions are already
+available. There were no special requirements agains the unit testing framework that would rule out all existing frameworks so
+we only had to choose a suitable one for our current and possible future needs.
+
+We ended up selecting CppUTest because of its small footprint and easy portability. It also goes along with the standard xUnit
+frameworks' principles and provides a standard interface to the outside world. Some details are listed below.
+
+- C/C++ support
+- Small footprint (compared to Google Test)
+- Easy to use on embedded systems
+- Built-in mocking system (CppUMock)
+- Implements four-phase testing pattern (setup, exercise, verify, teardown)
+- Selective run of test cases
+- Standard output format (JUnit, TeamCity)
+
+
+Assertions
+----------
+
+Generally is a good practice to use more specific assertions because it can output more informative error messages in case of a
+failure. The generic form or assertions is ``assert(expected, actual)``. Each assert type has a _TEXT variant for user defined
+error messages as last parameter.
+
+- Boolean
+
+ - ``CHECK(condition)`` - Same as ``CHECK_TRUE``
+ - ``CHECK_TRUE(condition)``
+ - ``CHECK_FALSE(condition)``
+
+- C string
+
+ - ``STRCMP_EQUAL(expected, actual)``
+ - ``STRNCMP_EQUAL(expected, actual, length)``
+ - ``STRCMP_NOCASE_EQUAL(expected, actual)``
+ - ``STRCMP_CONTAINS(expected, actual)``
+ - ``STRCMP_NOCASE_CONTAINS(expected, actual)``
+
+- Integer
+
+ - ``LONGS_EQUAL(expected, actual)``
+ - ``UNSIGNED_LONGS_EQUAL(expected, actual)``
+ - ``LONGLONGS_EQUAL(expected, actual)``
+ - ``UNSIGNED_LONGLONGS_EQUAL(expected, actual)``
+ - ``BYTES_EQUAL(expected, actual)``
+ - ``SIGNED_BYTES_EQUAL(expected, actual)``
+ - ``POINTERS_EQUAL(expected, actual)``
+ - ``FUNCTIONPOINTERS_EQUAL(expected, actual)``
+
+- Enums
+
+ - ``ENUMS_EQUAL_INT(expected, actual)``
+ - ``ENUMS_EQUAL_TYPE(underlying_type, expected, actual)``
+
+- Other assertions
+
+ - ``CHECK_EQUAL(expected, actual)`` - Requires ``operator=`` and ``StringFrom(type)`` to be implemented
+ - ``CHECK_COMPARE(first, relop, second)`` - Same as ``CHECK_EQUAL`` but with any type of compare
+ - ``DOUBLES_EQUAL(expected, actual, threshold)``
+ - ``MEMCMP_EQUAL(expected, actual, size)``
+ - ``BITS_EQUAL(expected, actual, mask)``
+ - ``FAIL()`` or ``FAIL_TEST()`` - Test case fails if called
+ - ``CHECK_THROWS(expected, expression)`` - Catching C++ exceptions
+
+- Miscellaneous macros
+
+ - ``IGNORE_TEST`` - Same as ``TEST`` but it’s not called
+ - ``TEST_EXIT`` - Exists test
+ - ``UT_CRASH()`` - Crashes the test which is easy to catch with debugger
+ - ``UT_PRINT(text)`` - Generic print function
+
+
+Test runner
+-----------
+
+Test cases are collected automatically. Under the hood the ``TEST`` macros are creating global instances of classes and their
+constructor registers the test cases into the test registry. This happens before entering the ``main`` function. In the ``main``
+function only the ``RUN_ALL_TESTS`` macro needs to be placed with the command line arguments passed to it. On executing the
+binary the command line arguments will control the behaviour of the test process.
+
+.. code-block:: C++
+
+ #include "CppUTest/CommandLineTestRunner.h"
+
+ int main(int argc, char* argv[]) {
+ return RUN_ALL_TESTS(argc, argv);
+ }
+
+The default ``main`` implementation is added to all unit test suites by the
+build system.
+
+
+Command line options
+--------------------
+
+Command line options are available mainly for configuring the output format of
+the test binary and for filtering test groups or cases.
+
+- Output
+
+ - ``-v`` - Prints each test name before running them
+ - ``-c`` - Colorized output
+ - ``-o{normal, junit, teamcity}`` - Output format, junit can be processed by
+ most CIs
+ - ``-k packageName`` - Package name for junit output
+ - ``-lg`` - List test groups
+ - ``-ln`` - List test cases
+
+- Other
+
+ - ``-p`` - Runs each test case in separate processes
+ - ``-ri`` - Runs ignored test cases
+ - ``-r#`` - Repeats testing ``#`` times
+ - ``-s seed`` - Shuffles tests
+
+- Filtering test cases
+
+ - ``"TEST(groupName, testName)"`` - Running single test
+ - ``"IGNORE_TEST(groupName, testName)"`` -- Running single ignored test
+ - ``-g text`` - Runing groups containing text
+ - ``-n text`` - Runing tests containing text
+ - ``-sg text`` - Runing groups matching text
+ - ``-sn text`` - Runing tests matching text
+ - ``-xg text`` - Excluding groups containing text
+ - ``-xn text`` - Excluding tests containing text
+ - ``-xsg text`` - Excluding groups matching text
+ - ``-xsn text`` - Excluding tests matching text
+
+
+Troubleshooting
+---------------
+
+Output messages
+^^^^^^^^^^^^^^^
+
+When one of tests fails the first step is to run it separately and check its
+output message. Usually it shows the exact line of the file where the error
+happened.
+
+::
+
+ test_memcmp.cpp:17: error: Failure in TEST(memcmp, empty)
+ expected <1 0x1>
+ but was <0 0x0>
+
+The executed tests can be followed by adding ``-v`` command line option.
+
+::
+
+ ./memcmp -v
+ TEST(memcmp, different) - 0 ms
+ TEST(memcmp, same) - 0 ms
+ TEST(memcmp, empty) - 0 ms
+
+ OK (3 tests, 3 ran, 1 checks, 0 ignored, 0 filtered out, 0 ms)
+
+
+Catching failure with debugger
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If a failure happens in a helper function or in a loop where the assertion
+is called multiple times it is harder to get the exact environment of a failure.
+In this case it's a good practice to put a ``UT_CRASH()`` call into a
+conditional block which hits when the failure happens. This way the debugger can
+stop on failure because the code emits a signal.
+
+.. code-block:: C++
+
+ TEST(magic, iterate) {
+ int result;
+
+ for(int i = 0; i < 1000; i++) {
+ result = magic_function(i);
+
+ // Debug code
+ if (result) {
+ UT_CRASH();
+ }
+
+ LONGS_EQUAL(0, result);
+ }
+ }
+
+
+Using ``FAIL`` macro
+^^^^^^^^^^^^^^^^^^^^
+
+It's recommended to use ``FAIL`` macro in conditions that should never occur in
+tests. For example if a test case loads test data from an external file but the
+file could not be opened the ``FAIL`` macro should be used with an informative
+message.
+
+.. code-block:: C++
+
+ fd = open("test.bin", O_RDONLY);
+ if (fd < 0) {
+ FAIL("test.bin open failed");
+ }
+
+
+Interference between test cases
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Test cases can interfere if there's a global resource which was not restored to
+its original state after leaving a test case. This can be hard to find but at
+least the it's easy to make sure that this is root case of an error. Let's
+assume there's a global variable which is set during the test case but it
+original value is not restore at the end. CppUTest has an command line option
+for running each test case in separate process. This makes the global variable
+to have its original value at the beginning of the test cases. Basically if the
+test works by passing argument ``-p`` when running but fails without it, there's
+a good chance for having an interference between test cases.
+
+.. code-block:: C++
+
+ int x = 0;
+
+ TEST_GROUP(crosstalk) {
+ };
+
+ TEST(crosstalk, a) {
+ LONGS_EQUAL(0, x);
+ x = 1;
+ }
+
+ TEST(crosstalk, b) {
+ LONGS_EQUAL(0, x);
+ x = 1;
+ }
+
+ TEST(crosstalk, c) {
+ LONGS_EQUAL(0, x);
+ x = 1;
+ }
+
+By running the test executable with different command line arguments it produces
+a different result.
+
+.. code-block::
+
+ ./crosstalk -v
+
+ TEST(crosstalk, c) - 0 ms
+ TEST(crosstalk, b)
+ test_crosstalk.cpp:37: error:
+ Failure in TEST(crosstalk, b)
+ expected <0 0x0>
+ but was <1 0x1>
+
+ - 0 ms
+ TEST(crosstalk, a)
+ test_crosstalk.cpp:32: error: Failure in TEST(crosstalk, a)
+ expected <0 0x0>
+ but was <1 0x1>
+
+ - 0 ms
+
+ Errors (2 failures, 3 tests, 3 ran, 3 checks, 0 ignored, 0 filtered out, 0 ms)
+
+ ./crosstalk -v -p
+ TEST(crosstalk, c) - 1 ms
+ TEST(crosstalk, b) - 0 ms
+ TEST(crosstalk, a) - 0 ms
+
+ OK (3 tests, 0 ran, 0 checks, 0 ignored, 0 filtered out, 2 ms)
+
+
+--------------
+
+*Copyright (c) 2019-2021, Arm Limited. All rights reserved.*
+
+.. _`official manual of CppUTest`: https://cpputest.github.io/