Add initial version of firmware-test-builder
Introduce the following features to firmware-test-builder:
* Testing on the host machine (PC)
* Provides CMake functions for building/running/evaluating tests
* Checking prerequisites (git, c-picker, libclang, etc.)
* Fetching and building CppUTest
* Defining and building unit test suites (i.e. separate binaries)
* Handling c-picker based extraction of code snippets
* Registering test suites to CTest (CMake's test system) which runs
all the test binaries
* Generating coverage report
* Documentation of the system
Signed-off-by: Imre Kis <imre.kis@arm.com>
Change-Id: Ic0a1af55bef07c6e76071193caa94a9a48f9041f
diff --git a/docs/implementing_tests.rst b/docs/implementing_tests.rst
new file mode 100644
index 0000000..fe3cb4a
--- /dev/null
+++ b/docs/implementing_tests.rst
@@ -0,0 +1,331 @@
+Implementing tests
+==================
+
+Concept of unit testing
+-----------------------
+
+First of all unit tests exercise the C code on a function level. The tests should call functions directly from the code under
+tests and verify if their return values are matching the expected ones and the functions are behaving according to the
+specification.
+
+Because of the function level testing the dependencies of the tested functions should be detached. This is done by mocking the
+underlying layer. This provides an additional advantage of controlling and verifying all the call to the lower layer.
+
+
+Adding new unit test suite
+--------------------------
+
+The first step is to define a new unit test suite. If a completely new module is being test the test suite definition should be
+created in a separate ``.cmake`` file which is placed in the test files' directory. Otherwise the test definition can be added
+to an existing ``.cmake`` file. These files should be included in the root ``CMakeLists.txt``.
+
+The ``UnitTest`` CMake module defines the ``unit_test_add_suite`` function so before using this function the module must be
+included in the ``.cmake`` file. The function first requires a unique test name which will be test binary's name. The test
+sources, include directories and macro definition are passed to the function in the matching arguments. CMake variables can be
+used to reference files relative to common directories:
+
+- ``CMAKE_CURRENT_LIST_DIR`` - Relative to the ``.cmake`` file
+- :cmake:variable:`UNIT_TEST_PROJECT_PATH` - Relative to the project's root directory
+
+.. code-block:: cmake
+
+ # tests/new_module/new_test_suite.cmake
+ include(UnitTest)
+
+ unit_test_add_suite(
+ NAME [unique test name]
+ SOURCES
+ [source files]
+ INCLUDE_DIRECTORIES
+ [include directories]
+ COMPILE_DEFINITIONS
+ [defines]
+ )
+
+.. code-block:: cmake
+
+ # Root CMakeLists.txt
+ include(tests/new_module/new_test_suite.cmake)
+
+Example test definition
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: cmake
+
+ unit_test_add_suite(
+ NAME memcmp
+ SOURCES
+ ${CMAKE_CURRENT_LIST_DIR}/test_memcmp.cpp
+ ${CMAKE_CURRENT_LIST_DIR}/memcmp.yml
+ INCLUDE_DIRECTORIES
+ ${UNIT_TEST_PROJECT_PATH}/include
+ ${UNIT_TEST_PROJECT_PATH}/include/lib/libc/aarch64/
+ )
+
+
+Using c-picker
+--------------
+
+c-picker is a simple tool used for detaching dependencies of the code under test. It can copy elements (i.e. functions,
+variables, etc.) from the original source code into generated files. This way the developer can pick functions from compilation
+units and surround them with a mocked environment.
+
+If a ``.yml`` file listed among source files the build system invokes c-picker and the generated ``.c`` file is implicitly added
+to the source file list.
+
+Example .yml file
+^^^^^^^^^^^^^^^^^
+
+In this simple example c-picker is instructed to copy the include directives and the ``memcmp`` function from the
+``lib/libc/memcmp.c`` file. The root directory of the source files referenced by c-picker is the project's root directory.
+
+.. code-block:: yaml
+
+ elements:
+ - file: lib/libc/memcmp.c
+ type: include
+ - file: lib/libc/memcmp.c
+ type: function
+ name: memcmp
+
+
+Writing unit tests
+------------------
+
+Unit test code should be placed in ``.cpp`` files.
+
+Four-phase test pattern
+^^^^^^^^^^^^^^^^^^^^^^^
+
+All tests cases should follow the four-phase test pattern. This consists of four simple steps that altogether ensure the
+isolation between test cases. These steps follows below.
+
+- Setup
+- Exercise
+- Verify
+- Teardown
+
+After the teardown step all global states should be the same as they were at the beginning of the setup step.
+
+CppUTest
+^^^^^^^^
+
+CppUTest is an open source unit testing framework for C/C++. It is written in C++ so all the useful features of the language is
+available while testing. It automatically collects and runs the defined ``TEST_GROUPS`` and provides an interface for
+implementing the four-phase test pattern. Furthermore the framework has assertion macros for many variable types and test
+scenarios.
+
+Include
+'''''''
+
+The unit test source files should include the CppUTest header after all other headers to avoid conflicts.
+
+.. code-block:: C++
+
+ // Other headers
+ // [...]
+
+ #include "CppUTest/TestHarness.h"
+
+Test group
+''''''''''
+
+The next step is to define a test group. When multiple tests cases are written around testing the same function or couple
+related functions these tests cases should be part of the same test group. Basically test cases in a test group share have same
+setup/teardown sequence. In CppUTest the ``TEST_GROUP`` macro defines a new class which can contain member variables and
+functions. Special setup/teardown function are defined using ``TEST_SETUP`` and ``TEST_TEARDOWN`` macros. These functions are
+called before/after running each test case of the group so all the common initilization and cleanup code should go into these
+functions.
+
+.. code-block:: C++
+
+ TEST_GROUP(List) {
+ TEST_SETUP() {
+ list = list_alloc();
+ }
+
+ TEST_TEARDOWN() {
+ list_cleanup(list);
+ }
+
+ bool has_element(int value) {
+ for (int i = 0; i < list_count (list); i++) {
+ if (list_get(i) == value) { return true; }
+ }
+ return false;
+ }
+
+ List* list;
+ };
+
+
+Test case
+'''''''''
+
+Test cases are defined by the ``TEST`` macro. This macro defines a child class of the test group's class so it can access the
+member functions and variables of the test group. The test case's block itself is the body of the function of the child class.
+
+.. code-block:: C++
+
+ TEST(List, add_one) {
+ // Exercise
+ const int test_value = 5;
+ list_add(list, test_value);
+
+ // Verify using CHECK_TRUE assertion and TEST_GROUP member function
+ CHECK_TRUE(has_element(test_value));
+ }
+
+ TEST(List, add_two) {
+ // Exercise
+ const int test_value1 = 5;
+ const int test_value2 = 6;
+ list_add(list, test_value1);
+ list_add(list, test_value2);
+
+ // Verify
+ CHECK_TRUE(has_element(test_value1));
+ CHECK_TRUE(has_element(test_value2));
+ }
+
+CppUMock
+^^^^^^^^
+
+During unit testing the dependencies of the tested functions should be replaced by stubs or mocks. When using mocks the
+developer would usually like to check if the function was called with corrent parameters and would like to return controlled
+values from the function. When a mocked function is called multiple times from the tested function maybe it should check or
+return different values on each call. This is where CppUMock comes handy.
+
+All CppUMock interactions start with calling the ``mock()`` function. This returs a reference to the mocking system. At this
+point the developer either wants to define expected or actual calls. This is achiveable by calling
+``expectOneCall(functionName)`` or ``expectNCalls(amount, functionName)`` or ``actualCall(functionName)`` member functions of
+``mock()`` call's return value. Registering expected calls are done in the test case before exercising the code and actual calls
+happen from the mocked functions.
+
+After this point the following functions can be chained:
+
+- ``onObject(object)`` - In C++ it is usually the ``this`` pointer but it can be
+ useful in C too.
+- ``with[type]Parameter(name, value)`` - Specifying and checking call parameters
+- ``andReturnValue(result)`` - Specifying return value when defining expected
+ call
+- ``return[type]Value()`` - Returning value from function
+
+The mocking system has two important functions. ``mock().checkExpectation()`` checks if all the expected calls have been
+fulfilled and and the ``mock().clear()`` removes all the expected calls from CppUMock's registry. These two functions are
+usually called from the ``TEST_TEARDOWN`` function because there should not be any crosstalk between test cases through the
+mocking system.
+
+CppUMock's typical use-case is shown below by a simple example of the ``print_to_eeprom`` function.
+
+.. code-block:: C++
+
+ int eeprom_write(const char* str); /* From eeprom.h */
+
+ int printf_to_eeprom(const char* format, ...) {
+ char buffer[256];
+ int length, written_bytes = 0, eeprom_write_result;
+ va_list args;
+
+ va_start(args, format);
+ length = vsnprintf(buffer, sizeof(buffer), format, args);
+ va_end(args);
+
+ if (length < 0) {
+ return length;
+ }
+
+ while(written_bytes < length) {
+ eeprom_write_result = eeprom_write(&buffer[written_bytes]);
+ if (eeprom_write_result < 0) {
+ return eeprom_write_result;
+ }
+ written_bytes += eeprom_write_result;
+ }
+
+ return written_bytes;
+ }
+
+Having the code snipped above a real life usage of the function would look like something shown in the following sequence
+diagram.
+
+.. uml:: resources/sequence_print_to_eeprom.puml
+
+It would be really hard to test unit this whole system so all the lower layers should be separated and mock on the first
+possible level. In the following example the ``print_to_eeprom`` function is being tested and the ``eeprom_write`` function is
+mocked. In test cases where ``eeprom_write`` function is expected to be called the test case should first call the
+``expect_write`` function. This registers an expected call to CppUMocks internal database and when the call actually happens it
+matches the call parameters with the entry in the database. It also returns the previously specified value.
+
+.. code-block:: C++
+
+ TEST_GROUP(printf_to_eeprom) {
+ TEST_TEARDOWN() {
+ mock().checkExpectations();
+ mock().clear();
+ }
+
+ void expect_write(const char* str, int result) {
+ mock().expectOneCall("eeprom_write").withStringParameter("str", str).
+ andReturnValue(result);
+ }
+ };
+
+ /* Mocked function */
+ int eeprom_write(const char* str) {
+ return mock().actualCall("eeprom_write").withStringParameter("str", str).
+ returnIntValue();
+ }
+
+ TEST(printf_to_eeprom, empty) {
+ LONGS_EQUAL(0, printf_to_eeprom(""))
+ }
+
+ TEST(printf_to_eeprom, two_writes) {
+ expect_write("hello1hello2", 6);
+ expect_write("hello2", 6);
+ LONGS_EQUAL(12, printf_to_eeprom("hello%dhello%d", 1, 2))
+ }
+
+ TEST(printf_to_eeprom, error) {
+ expect_write("hello", -1);
+ LONGS_EQUAL(-1, printf_to_eeprom("hello"))
+ }
+
+This how the ``printf_to_eeprom/two_writes`` test case's sequence diagram looks like after mocking ``eeprom_write``. The test
+case became able to check the parameters of multiple calls and it could return controlled values.
+
+.. uml:: resources/sequence_print_to_eeprom_mock.puml
+
+
+Analyzing code coverage
+-----------------------
+
+The code coverage reports can be easily used for finding untested parts of the code. The two main parts of the coverage report
+are the line coverage and the branch coverage. Line coverage shows that how many times the tests ran the given line of the
+source code. It is beneficial to increase the line coverage however 100% line coverage is still not enough to consider the code
+fully tested.
+
+Let's have a look on the following example.
+
+.. code-block:: C++
+
+ void set_pointer_value(unsigned int id, unsigned int value) {
+ unsigned int *pointer;
+
+ if (id < MAX_ID) {
+ pointer = get_pointer(id);
+ }
+
+ *pointer = value;
+ }
+
+The 100% line coverage is achievable by testing the function with an ``id`` value smaller than ``MAX_ID``. However if an ``id``
+larger than or equal to ``MAX_ID`` is used as a parameter of this function it will try to write to a memory address pointed by
+an uninitialized variable. To catch untested conditions like this the branch coverage comes handy. It will show that only one
+branch of the ``if`` statement has been tested as the condition was always true in the tests.
+
+
+--------------
+
+*Copyright (c) 2019-2021, Arm Limited. All rights reserved.*