Add initial version of firmware-test-builder
Introduce the following features to firmware-test-builder:
* Testing on the host machine (PC)
* Provides CMake functions for building/running/evaluating tests
* Checking prerequisites (git, c-picker, libclang, etc.)
* Fetching and building CppUTest
* Defining and building unit test suites (i.e. separate binaries)
* Handling c-picker based extraction of code snippets
* Registering test suites to CTest (CMake's test system) which runs
all the test binaries
* Generating coverage report
* Documentation of the system
Signed-off-by: Imre Kis <imre.kis@arm.com>
Change-Id: Ic0a1af55bef07c6e76071193caa94a9a48f9041f
diff --git a/docs/components/build_system.rst b/docs/components/build_system.rst
new file mode 100644
index 0000000..4c26b32
--- /dev/null
+++ b/docs/components/build_system.rst
@@ -0,0 +1,22 @@
+Build system
+============
+
+.. cmake-module:: ../../cmake/UnitTest.cmake
+
+.. cmake-module:: ../../cmake/FindLibClang.cmake
+
+Generating documentation
+------------------------
+
+``sphinx-builder`` is used for building the documentation which is required to be installed on the machine. The documentation
+can be built by executing the following commands.
+
+::
+
+ cd docs
+ make html
+
+
+--------------
+
+*Copyright (c) 2019-2021, Arm Limited. All rights reserved.*
diff --git a/docs/components/c_picker.rst b/docs/components/c_picker.rst
new file mode 100644
index 0000000..da89520
--- /dev/null
+++ b/docs/components/c_picker.rst
@@ -0,0 +1,61 @@
+c-picker
+========
+
+c-picker uses ``libclang``'s Python interface for parsing source files.
+
+Command line options
+--------------------
+
+- ``-h, --help`` - Showing help message
+- ``--root ROOT`` - Root source directory
+- ``--config CONFIG`` - Configuration file (``.json|.yml``)
+- ``--output OUTPUT`` - Output file
+- ``--print-dependencies`` - Prints the dependencies
+- ``--version`` - Shows the program's version number and exit
+- ``--args ...`` - clang arguments
+
+
+Configuration file format
+-------------------------
+
+c-picker configuration can be described in ``JSON`` or ``YAML`` format using the same object structure.
+
+- ``elements`` - List of elements to pick from the original code
+
+ - ``name`` - Name of the element
+ - ``type`` - Type of the element: ``include``, ``function``, ``variable``
+ - ``args`` - Arguments for clang used during picking the element
+ - ``options`` - Currenly the only options is ``remove_static`` which removes
+ the ``static`` qualifier from the element's definition.
+
+- ``options`` - Global options for all elements
+- ``args`` - Global clang arguments for all elements
+
+YAML example
+------------
+
+YAML format is preferred because it can contain metadata like comments and licence information.
+
+.. code-block:: YAML
+
+ elements:
+ - file: bl1/bl1_fwu.c
+ type: variable
+ name: bl1_fwu_loaded_ids
+ options: [remove_static]
+ - file: bl1/bl1_fwu.c
+ type: function
+ name: bl1_fwu_add_loaded_id
+ options: [remove_static]
+ - file: bl1/bl1_fwu.c
+ type: function
+ name: bl1_fwu_remove_loaded_id
+ options: [remove_static]
+ args:
+ - -DFWU_MAX_SIMULTANEOUS_IMAGES=10
+ - -DINVALID_IMAGE_ID=0xffffffff
+
+
+--------------
+
+*Copyright (c) 2019-2021, Arm Limited. All rights reserved.*
diff --git a/docs/components/code_coverage.rst b/docs/components/code_coverage.rst
new file mode 100644
index 0000000..3936e4f
--- /dev/null
+++ b/docs/components/code_coverage.rst
@@ -0,0 +1,27 @@
+Code coverage
+=============
+
+Coverage processing flow
+------------------------
+
+1. Prerequisites
+
+ 1. Building all or selected test binaries with coverage enabled
+
+ 2. Running all or selected test binaries
+
+2. Collecting coverage data from ``.gcda`` and ``.gcno`` file into ``lcov`` coverage info file
+
+3. Mapping c-picker generated files' coverage to the original source lines
+
+4. Filtering coverage data for separating the coverage of the code under tests and the coverage of the test code
+
+5. Generating HTML coverage report from the filtered lcov info files
+
+
+.. cmake-module:: ../../cmake/Coverage.cmake
+
+
+--------------
+
+*Copyright (c) 2019-2021, Arm Limited. All rights reserved.*
diff --git a/docs/components/cppumock.rst b/docs/components/cppumock.rst
new file mode 100644
index 0000000..fd82d95
--- /dev/null
+++ b/docs/components/cppumock.rst
@@ -0,0 +1,197 @@
+CppUMock
+========
+
+CppUMock is the built-in mocking system of CppUTest. This chapter describes the basic features of the system. For details please
+refer the `official CppUMock manual`_.
+
+System functions
+----------------
+
+Checking expectations
+^^^^^^^^^^^^^^^^^^^^^
+
+After defining expected calls an invoking actual call the test should check if all the expected calls have happened. This can be
+done by calling ``mock().checkExpectations()``. The common place to put this function call is the ``TEST_TEARDOWN`` function of
+the test group.
+
+
+Resetting mocking system
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+After the last interaction with the mocking system the developer should reset the state of the system by calling
+``mock().clear()``. The common place to put this function call is the ``TEST_TEARDOWN`` function of the test group after calling
+``mock().checkExpectations()``.
+
+
+Namespaces
+^^^^^^^^^^
+
+All interactions with the mocking system start by calling ``mock()``. This function has an option ``name`` string parameter
+which can be used for limiting the scope of the mocking operation.
+
+.. code-block:: C++
+
+ mock("bl1").expectOneCall("bl1_main");
+
+
+Enable/disable
+^^^^^^^^^^^^^^
+
+The mocking system can be enabled/disabled runtime using the functions below. This causes call expected and actual call to be
+ignored. Default settings are restored by ``mock().clear()``.
+
+- ``enable()``
+- ``disable()``
+
+.. code-block:: C++
+
+ mock().disable();
+ // All CppUMock calls are ignored after this point
+ mock().enable();
+ // CppUMock calls are working again
+
+ mock().disable();
+ // All CppUMock calls are ignored after this point
+ [...]
+
+ mock().clear();
+ // Default settings are restored
+
+
+String order
+^^^^^^^^^^^^
+
+After defining multiple expected function call the mocking system always The mocking system always uses the next matching
+function when an actual call happens but by default it doesn't check if different function calls happen in the order of
+definition. This behaviour can be turned on using the ``mock().strictOrder()`` function. This option is also set to default by
+the ``mock().clear()`` function.
+
+.. code-block:: C++
+
+ mock().expectOneCall("A");
+ mock().expectOneCall("B");
+
+ mock().actualCall("B");
+ mock().actualCall("A");
+ // No error
+
+ mock().strictOrder();
+ mock().expectOneCall("A");
+ mock().expectOneCall("B");
+
+ mock().actualCall("B"); // Error generated here
+ mock().actualCall("A");
+
+
+Ignoring unspecified calls
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If there are addition actual calls happening in the test case which are unrelated to the test case (i.e. log or other messages)
+the unspecified calls can be ignored by adding ``mock().ignoreOtherCalls()`` to the test case. This function should be used
+carefully because it can hide unexpected call which are indicating errors in the code. The affect of this call ends by calling
+``mock().clear()``.
+
+
+Specifying object
+-----------------
+
+In object oriented environment member function should validate if the function call happened on the suitable object. This is
+done by adding the following function to the mock specification.
+
+- ``onObject(objectPtr)``
+
+
+Validating parameters
+---------------------
+
+Each supported parameter type has a corresponding function. These are the same
+in the expected and actual calls.
+
+- ``withBoolParameter(name, bool value)``
+- ``withIntParameter(name, int value)``
+- ``withUnsignedIntParameter(name, unsigned int value)``
+- ``withLongIntParameter(name, long int value)``
+- ``withUnsignedLongIntParameter(name, unsigned long int value)``
+- ``withDoubleParameter(name, double value)``
+- ``withStringParameter(name, const char* value)``
+- ``withPointerParameter(name, void* value)``
+- ``withFunctionPointerParameter(name, void (*value)())``
+- ``withConstPointerParameter(name, const void* value)``
+- ``withMemoryBufferParameter(name, const unsigned char* value, size_t size)``
+
+If custum types are defined and copier/comparator objects were installed the following function can handle these parameters.
+
+- ``withParameterOfType(typeName, name, value)``
+
+There's an option to copying data from the test environment into the mock function. When setting expectations the following
+function can be used to set the pointer and the address of the data. **The mocking system will not create a copy of this data**
+so the original data should be kept intact until the actual call happens.
+
+- ``withOutputParameterReturning(name, value, size)``
+- ``withOutputParameterOfTypeReturning(typeName, name, value)``
+
+In the actual call the pair of these function are shown below.
+
+- ``withOutputParameter(name, output)``
+- ``withOutputParameterOfType(typeName, name, output)``
+
+
+Ignoring parameters
+^^^^^^^^^^^^^^^^^^^
+
+There are cases when the developer doesn't want to specify all parameters. The following function can set this behaviour in the
+expected call.
+
+- ``ignoreOtherParameters()``
+
+
+Specifying return values
+------------------------
+
+Using function name overloading the return values are specified by calling ``andReturnValue`` and the parameter type will
+determine the exact function.
+
+- ``andReturnValue(bool value)``
+- ``andReturnValue(int value)``
+- ``andReturnValue(unsigned int value)``
+- ``andReturnValue(long int value)``
+- ``andReturnValue(unsigned long int value)``
+- ``andReturnValue(double value)``
+- ``andReturnValue(const char* value)``
+- ``andReturnValue(void* value)``
+- ``andReturnValue(const void* value)``
+- ``andReturnValue(void (*value)())``
+
+
+Returning value in actual calls
+-------------------------------
+
+All of these function have version with ``OrDefault(type default_value)`` suffix. These version return a default value if the
+return value was not specified in the expected call.
+
+- ``bool returnBoolValue()``
+- ``int returnIntValue()``
+- ``unsigned int returnUnsignedIntValue()``
+- ``long int returnLongIntValue()``
+- ``unsigned long int returnUnsignedLongIntValue()``
+- ``double returnDoubleValue()``
+- ``const char * returnStringValue()``
+- ``void * returnPointerValue()``
+- ``const void * returnConstPointerValue()``
+- ``void (*returnFunctionPointerValue())()``
+
+
+Debugging CppUMock errors
+-------------------------
+
+Debugging CppUMock errors can be hard unlike assertion errors because a mocking failure can happen multiple layers of function
+calls under the test case. The mocking system has a very similar feature to CppUTest's ``UT_CRASH()`` which is
+``mock().crashOnFailure()``. By enabling this feature the code will crash on mocking errors and the developer could easily catch
+it with the debugger.
+
+
+--------------
+
+*Copyright (c) 2019-2021, Arm Limited. All rights reserved.*
+
+.. _`official CppUMock manual`: https://cpputest.github.io/mocking_manual.html
diff --git a/docs/components/cpputest.rst b/docs/components/cpputest.rst
new file mode 100644
index 0000000..e1c34a9
--- /dev/null
+++ b/docs/components/cpputest.rst
@@ -0,0 +1,281 @@
+CppUTest
+========
+
+This document is based on CppUTest v3.8. CppUtest is a unit testing framework for testing C and C++ code. This document
+introduces the basic features of the framework. For further information check the `official manual of CppUTest`_.
+
+
+Why CppUTest?
+-------------
+
+First of all it was not our goal to develop a new unit testing framework while plenty of open source solutions are already
+available. There were no special requirements agains the unit testing framework that would rule out all existing frameworks so
+we only had to choose a suitable one for our current and possible future needs.
+
+We ended up selecting CppUTest because of its small footprint and easy portability. It also goes along with the standard xUnit
+frameworks' principles and provides a standard interface to the outside world. Some details are listed below.
+
+- C/C++ support
+- Small footprint (compared to Google Test)
+- Easy to use on embedded systems
+- Built-in mocking system (CppUMock)
+- Implements four-phase testing pattern (setup, exercise, verify, teardown)
+- Selective run of test cases
+- Standard output format (JUnit, TeamCity)
+
+
+Assertions
+----------
+
+Generally is a good practice to use more specific assertions because it can output more informative error messages in case of a
+failure. The generic form or assertions is ``assert(expected, actual)``. Each assert type has a _TEXT variant for user defined
+error messages as last parameter.
+
+- Boolean
+
+ - ``CHECK(condition)`` - Same as ``CHECK_TRUE``
+ - ``CHECK_TRUE(condition)``
+ - ``CHECK_FALSE(condition)``
+
+- C string
+
+ - ``STRCMP_EQUAL(expected, actual)``
+ - ``STRNCMP_EQUAL(expected, actual, length)``
+ - ``STRCMP_NOCASE_EQUAL(expected, actual)``
+ - ``STRCMP_CONTAINS(expected, actual)``
+ - ``STRCMP_NOCASE_CONTAINS(expected, actual)``
+
+- Integer
+
+ - ``LONGS_EQUAL(expected, actual)``
+ - ``UNSIGNED_LONGS_EQUAL(expected, actual)``
+ - ``LONGLONGS_EQUAL(expected, actual)``
+ - ``UNSIGNED_LONGLONGS_EQUAL(expected, actual)``
+ - ``BYTES_EQUAL(expected, actual)``
+ - ``SIGNED_BYTES_EQUAL(expected, actual)``
+ - ``POINTERS_EQUAL(expected, actual)``
+ - ``FUNCTIONPOINTERS_EQUAL(expected, actual)``
+
+- Enums
+
+ - ``ENUMS_EQUAL_INT(expected, actual)``
+ - ``ENUMS_EQUAL_TYPE(underlying_type, expected, actual)``
+
+- Other assertions
+
+ - ``CHECK_EQUAL(expected, actual)`` - Requires ``operator=`` and ``StringFrom(type)`` to be implemented
+ - ``CHECK_COMPARE(first, relop, second)`` - Same as ``CHECK_EQUAL`` but with any type of compare
+ - ``DOUBLES_EQUAL(expected, actual, threshold)``
+ - ``MEMCMP_EQUAL(expected, actual, size)``
+ - ``BITS_EQUAL(expected, actual, mask)``
+ - ``FAIL()`` or ``FAIL_TEST()`` - Test case fails if called
+ - ``CHECK_THROWS(expected, expression)`` - Catching C++ exceptions
+
+- Miscellaneous macros
+
+ - ``IGNORE_TEST`` - Same as ``TEST`` but it’s not called
+ - ``TEST_EXIT`` - Exists test
+ - ``UT_CRASH()`` - Crashes the test which is easy to catch with debugger
+ - ``UT_PRINT(text)`` - Generic print function
+
+
+Test runner
+-----------
+
+Test cases are collected automatically. Under the hood the ``TEST`` macros are creating global instances of classes and their
+constructor registers the test cases into the test registry. This happens before entering the ``main`` function. In the ``main``
+function only the ``RUN_ALL_TESTS`` macro needs to be placed with the command line arguments passed to it. On executing the
+binary the command line arguments will control the behaviour of the test process.
+
+.. code-block:: C++
+
+ #include "CppUTest/CommandLineTestRunner.h"
+
+ int main(int argc, char* argv[]) {
+ return RUN_ALL_TESTS(argc, argv);
+ }
+
+The default ``main`` implementation is added to all unit test suites by the
+build system.
+
+
+Command line options
+--------------------
+
+Command line options are available mainly for configuring the output format of
+the test binary and for filtering test groups or cases.
+
+- Output
+
+ - ``-v`` - Prints each test name before running them
+ - ``-c`` - Colorized output
+ - ``-o{normal, junit, teamcity}`` - Output format, junit can be processed by
+ most CIs
+ - ``-k packageName`` - Package name for junit output
+ - ``-lg`` - List test groups
+ - ``-ln`` - List test cases
+
+- Other
+
+ - ``-p`` - Runs each test case in separate processes
+ - ``-ri`` - Runs ignored test cases
+ - ``-r#`` - Repeats testing ``#`` times
+ - ``-s seed`` - Shuffles tests
+
+- Filtering test cases
+
+ - ``"TEST(groupName, testName)"`` - Running single test
+ - ``"IGNORE_TEST(groupName, testName)"`` -- Running single ignored test
+ - ``-g text`` - Runing groups containing text
+ - ``-n text`` - Runing tests containing text
+ - ``-sg text`` - Runing groups matching text
+ - ``-sn text`` - Runing tests matching text
+ - ``-xg text`` - Excluding groups containing text
+ - ``-xn text`` - Excluding tests containing text
+ - ``-xsg text`` - Excluding groups matching text
+ - ``-xsn text`` - Excluding tests matching text
+
+
+Troubleshooting
+---------------
+
+Output messages
+^^^^^^^^^^^^^^^
+
+When one of tests fails the first step is to run it separately and check its
+output message. Usually it shows the exact line of the file where the error
+happened.
+
+::
+
+ test_memcmp.cpp:17: error: Failure in TEST(memcmp, empty)
+ expected <1 0x1>
+ but was <0 0x0>
+
+The executed tests can be followed by adding ``-v`` command line option.
+
+::
+
+ ./memcmp -v
+ TEST(memcmp, different) - 0 ms
+ TEST(memcmp, same) - 0 ms
+ TEST(memcmp, empty) - 0 ms
+
+ OK (3 tests, 3 ran, 1 checks, 0 ignored, 0 filtered out, 0 ms)
+
+
+Catching failure with debugger
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If a failure happens in a helper function or in a loop where the assertion
+is called multiple times it is harder to get the exact environment of a failure.
+In this case it's a good practice to put a ``UT_CRASH()`` call into a
+conditional block which hits when the failure happens. This way the debugger can
+stop on failure because the code emits a signal.
+
+.. code-block:: C++
+
+ TEST(magic, iterate) {
+ int result;
+
+ for(int i = 0; i < 1000; i++) {
+ result = magic_function(i);
+
+ // Debug code
+ if (result) {
+ UT_CRASH();
+ }
+
+ LONGS_EQUAL(0, result);
+ }
+ }
+
+
+Using ``FAIL`` macro
+^^^^^^^^^^^^^^^^^^^^
+
+It's recommended to use ``FAIL`` macro in conditions that should never occur in
+tests. For example if a test case loads test data from an external file but the
+file could not be opened the ``FAIL`` macro should be used with an informative
+message.
+
+.. code-block:: C++
+
+ fd = open("test.bin", O_RDONLY);
+ if (fd < 0) {
+ FAIL("test.bin open failed");
+ }
+
+
+Interference between test cases
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Test cases can interfere if there's a global resource which was not restored to
+its original state after leaving a test case. This can be hard to find but at
+least the it's easy to make sure that this is root case of an error. Let's
+assume there's a global variable which is set during the test case but it
+original value is not restore at the end. CppUTest has an command line option
+for running each test case in separate process. This makes the global variable
+to have its original value at the beginning of the test cases. Basically if the
+test works by passing argument ``-p`` when running but fails without it, there's
+a good chance for having an interference between test cases.
+
+.. code-block:: C++
+
+ int x = 0;
+
+ TEST_GROUP(crosstalk) {
+ };
+
+ TEST(crosstalk, a) {
+ LONGS_EQUAL(0, x);
+ x = 1;
+ }
+
+ TEST(crosstalk, b) {
+ LONGS_EQUAL(0, x);
+ x = 1;
+ }
+
+ TEST(crosstalk, c) {
+ LONGS_EQUAL(0, x);
+ x = 1;
+ }
+
+By running the test executable with different command line arguments it produces
+a different result.
+
+.. code-block::
+
+ ./crosstalk -v
+
+ TEST(crosstalk, c) - 0 ms
+ TEST(crosstalk, b)
+ test_crosstalk.cpp:37: error:
+ Failure in TEST(crosstalk, b)
+ expected <0 0x0>
+ but was <1 0x1>
+
+ - 0 ms
+ TEST(crosstalk, a)
+ test_crosstalk.cpp:32: error: Failure in TEST(crosstalk, a)
+ expected <0 0x0>
+ but was <1 0x1>
+
+ - 0 ms
+
+ Errors (2 failures, 3 tests, 3 ran, 3 checks, 0 ignored, 0 filtered out, 0 ms)
+
+ ./crosstalk -v -p
+ TEST(crosstalk, c) - 1 ms
+ TEST(crosstalk, b) - 0 ms
+ TEST(crosstalk, a) - 0 ms
+
+ OK (3 tests, 0 ran, 0 checks, 0 ignored, 0 filtered out, 2 ms)
+
+
+--------------
+
+*Copyright (c) 2019-2021, Arm Limited. All rights reserved.*
+
+.. _`official manual of CppUTest`: https://cpputest.github.io/
diff --git a/docs/components/index.rst b/docs/components/index.rst
new file mode 100644
index 0000000..301cf9f
--- /dev/null
+++ b/docs/components/index.rst
@@ -0,0 +1,14 @@
+Component user manuals
+======================
+
+.. toctree::
+ cpputest
+ cppumock
+ c_picker
+ build_system
+ code_coverage
+
+
+--------------
+
+*Copyright (c) 2019-2021, Arm Limited. All rights reserved.*