Add initial version of firmware-test-builder
Introduce the following features to firmware-test-builder:
* Testing on the host machine (PC)
* Provides CMake functions for building/running/evaluating tests
* Checking prerequisites (git, c-picker, libclang, etc.)
* Fetching and building CppUTest
* Defining and building unit test suites (i.e. separate binaries)
* Handling c-picker based extraction of code snippets
* Registering test suites to CTest (CMake's test system) which runs
all the test binaries
* Generating coverage report
* Documentation of the system
Signed-off-by: Imre Kis <imre.kis@arm.com>
Change-Id: Ic0a1af55bef07c6e76071193caa94a9a48f9041f
diff --git a/docs/user_guide.rst b/docs/user_guide.rst
new file mode 100644
index 0000000..4c214f4
--- /dev/null
+++ b/docs/user_guide.rst
@@ -0,0 +1,201 @@
+User guide
+==========
+
+This page describes how to get started with compiling and running unit tests on the host machine.
+
+Host machine requirements
+-------------------------
+
+The system has been successfully tested on the following platforms:
+
+- Ubuntu 19.04
+- Ubuntu 18.04
+- Arch Linux
+- MSYS2 MinGW64
+
+Tools
+-----
+
+The following applications are expected to be installed in the build machine:
+
+- CMake >=3.11
+
+- Python >=3.4
+
+- c-picker
+
+ - pyyaml
+
+ - clang (pip package if not included in libclang)
+
+ - libclang
+
+- git
+
+- Toolchain
+
+- Native build system
+
+Ubuntu 19.04
+^^^^^^^^^^^^
+
+On Ubuntu 19.04 use the following commands to install the required tools:
+
+::
+
+ sudo apt-get install cmake git python3 python3-pip libclang-dev build-essential
+ sudo pip3 install git+[TBD]
+
+Ubuntu 18.04
+^^^^^^^^^^^^
+
+The official Ubuntu 18.04 package repository only contains CMake 3.10 which not satisfies the requirements for building unit
+tests. Fortunately there's an official CMake APT repository provided by Kitware: https://apt.kitware.com/ By adding this server
+to the repository list an up-to-date version of CMake can be installed on Ubuntu 18.04.
+
+::
+
+ wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc \
+ 2>/dev/null | sudo apt-key add -
+ sudo apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main'
+
+ sudo apt-get install cmake git python3 python3-pip libclang-dev build-essential
+ sudo pip3 install git+[TBD]
+
+Arch Linux
+^^^^^^^^^^
+
+On Arch Linux use the following commands to install the required tools:
+
+::
+
+ sudo pacman -Sy cmake git python python-pip python-yaml make gcc clang
+ sudo pip install git+[TBD]
+
+MSYS2 MinGW64
+^^^^^^^^^^^^^
+
+::
+
+ pacman -Sy mingw-w64-x86_64-cmake git mingw-w64-x86_64-python3 \
+ mingw-w64-x86_64-python3-pip make mingw-w64-x86_64-gcc mingw-w64-x86_64-clang
+ pip install git+[TBD]]
+
+
+Integrating firmware test builder into a project
+------------------------------------------------
+
+The firmware test builder can be simply integrated into a CMake based project by adding the lines below to its CMakeLists.txt.
+This example shows how to fetch the firmware test builder files on demand using CMake's FetchContent module.
+
+::
+
+ FetchContent_Declare(
+ firmware_test_builder
+ GIT_REPOSITORY ${FIRMWARE_TEST_BUILDER_URL}
+ GIT_TAG ${FIRMWARE_TEST_BUILDER_REFSPEC}
+ GIT_SHALLOW TRUE
+ )
+
+ FetchContent_GetProperties(firmware_test_builder)
+ if(NOT firmware_test_builder_POPULATED)
+ message(STATUS "Fetching Firmware Test Builder")
+ FetchContent_Populate(firmware_test_builder)
+ endif()
+
+ # Appending Firmware Test Builder's cmake directory to CMake module path
+ list(APPEND CMAKE_MODULE_PATH ${firmware_test_builder_SOURCE_DIR}/cmake)
+
+For more details on adding tests see :ref:`Implementing tests` chapter.
+
+
+Building unit tests
+-------------------
+
+Building unit tests start with running CMake which will check all the prerequisites and generate the native build system's input
+files. This example uses Unix Makefiles. Unit tests exercise the code directly by compiling it into the same binary as the test
+code.
+
+::
+
+ cd project
+ mkdir build
+ cd build
+ cmake -G"Unix Makefiles" ..
+
+After running the previous steps the makefiles are generated into the build directory so make can build unit tests. During unit
+test development if only the source files have changed it's not necessary to re-run cmake it's only needed to run make as shown
+below.
+
+::
+
+ make -j
+
+For building single unit test suites the test's name can be used as a makefile target. Let's assume there's a test suite called
+best_unit_tests.
+
+::
+
+ make best_unit_tests
+
+
+Running unit tests
+------------------
+
+CMake provides a built-in tool called ctest for running all the tests using a single command. It is also able to filter tests or
+run them in parallel for speeding up the tests process. Run all the tests using the following command:
+
+::
+
+ ctest
+
+Each unit test suite has its own executable. The easiest way of running single test suite is running it as a simple executable.
+
+::
+
+ ./best_unit_tests
+
+
+Debugging unit tests
+--------------------
+
+As it was mentioned in the previous section test suites are basically separate executables so they can be debugged as any other
+native applications on the host machine. In a Linux environment gdb or IDE's built-in debugger can be utilized for debugging.
+
+::
+
+ gdb ./best_unit_tests
+
+
+Measuring code coverage
+-----------------------
+
+Inspecting code coverage is a useful method for detecting parts of the code which is not exercised by tests. The build system
+includes an option for generating code coverage report of the unit tests. The coverage is processed by ``lcov`` which needs to
+be installed for this feature. Also the coverage measurement in only available when GCC is used as a compiler.
+
+The methods for enabling coverage measurement is project dependent.
+
+Before collecting coverage info and generating reports the tests must be run as the coverage is a runtime measurement. See
+section `Running unit tests`_ for more information about running unit tests.
+
+In case of enabled coverage report the system adds two new build targets called ``coverage`` and ``coverage_report``. They can
+be used simply by running the following commands if ``make`` is used as a build system.
+
+::
+
+ make coverage
+ make coverage_report
+
+The ``coverage`` target generates lcov info files for further processing. If there are coverage files available from different
+sources (i.e. coverages of other tests) they can be merged with the unit test coverage file and evaluated together.
+
+The ``coverage_report`` target generates a HTML report from the coverage info files. The coverage reports can be found in the
+build directory. The report shows the directory structure of the code and each file can be inspected individually. Line,
+function and branch coverage is included.
+
+
+--------------
+
+*Copyright (c) 2019-2021, Arm Limited. All rights reserved.*
+SPDX-License-Identifier: BSD-3-Clause