docs: document coverage measurement
Add basic information about coverage measurement. Document build types,
shared library names and coverage measurement strategy. Measurement
instructions are to be added later.
Change-Id: Ie657d322941952a7152e65a2b6d651f3c7a986ad
Signed-off-by: Gyorgy Szing <gyorgy.szing@arm.com>
diff --git a/docs/deployments/libraries.rst b/docs/deployments/libraries.rst
index eb6d107..8a0e276 100644
--- a/docs/deployments/libraries.rst
+++ b/docs/deployments/libraries.rst
@@ -4,10 +4,19 @@
other deployments or external applications. The following library deployments
are currently supported:
+The libraries will have a build type-specific suffix added to the base name of
+their binaries, allowing multiple binaries to be installed in the same tree.
+(See :ref:`Selecting the build type`) This differentiation applies to the
+``Debug`` and ``DebugCoverage`` build types. The system integrator must choose
+which single ``Release<XXX>`` build type is to be supported in the system.
+Package managers and mutually exclusive release packages can be used to add
+further flexibility.
+
.. _libs-libts:
libts
-----
+
Userspace applications that depend on trusted services may use *libts* for handling
service discovery and RPC messaging. A major benefit to application developers is
that *libts* entirely decouples client applications from details of where a service
@@ -29,12 +38,14 @@
- * | *linux-pc* - service providers integrated into library
* | *arm-linux* - communicates with service providers in secure processing environment
* - Used by
- - * Userspace applications
+ - * Userspace applications.
+
.. _libs-libpsats:
libpsats
--------
+
Trusted Services implements the `PSA Certified APIs`_. Libpsats encapsulates the service client implementations
which implement this API. Linux uuser-space applications can use libpsats to easily access the PSA services
implemented by the project. Libpsats depends on libts for RPC and service discovery services.
@@ -109,6 +120,7 @@
libsp
-----
+
*libsp* provides a functional interface for using FF-A messaging and memory
management facilities. *libsp* is used in SP deployments. For more information, see:
:ref:`libsp`.
diff --git a/docs/developer/build-instructions.rst b/docs/developer/build-instructions.rst
index f52fe55..5e8e333 100644
--- a/docs/developer/build-instructions.rst
+++ b/docs/developer/build-instructions.rst
@@ -25,10 +25,12 @@
.. uml:: uml/BuildFlow.puml
+.. _`Selecting the build type`:
+
Selecting the build type
-------------------------
The build type selects code optimization and debug information related compiler settings. The build system follows the
-standard CMake methodology and uses the `CMAKE_BUILD_TYPE`_ variable.
+standard CMake methodology and can be controlled with the `CMAKE_BUILD_TYPE`_ variable.
The build system uses the following build types:
@@ -37,27 +39,43 @@
* - Build type
- Purpose
+ - Shared library suffix
- Description
* - Debug
- For debugging purposes.
+ - `d`
- Optimization is off, debugging information generation is enabled.
+ * - DebugCoverage
+ - For coverage measurement purposes.
+ - `c`
+ - Optimization is off, debugging information generation is enabled, code is compiled to enable gcov based coverage
+ measurement.
* - MinSizeRel
- Size optimized release build.
+ - None.
- Optimization is configured to prefer small code size, debugging information is not generated.
* - MinSizWithDebInfo
- For debugging size optimized release build.
+ - None.
- Optimization is configured to prefer small code size, debugging information generation is enabled.
* - Release
- Speed optimized release build.
+ - None.
- Optimization is configured to prefer execution speed, debugging information is not generated.
* - RelWithDebugInfo
- For debugging speed optimized release build.
+ - None.
- Optimization is configured to prefer execution speed, debugging information generation is enabled.
-Build type of external components can be configured with command line parameters. Parameter names follow this pattern:
+Build type of external components can be configured with command line arguments. Argument names follow this pattern:
``-D<upper case component name>_BUILD_TYPE=<value>`` e.g. ``-DNANOPB_BUILD_TYPE=Release``. Supported values are
component specific, please refer to the appropriate cmake file under ``<TS_ROOT>/external/<name>``.
+Binaries of the ``Debug`` and ``DebugCoverage`` build types will have a suffix appended to their base name. This allows
+multiple binaries to coexist on the same system at the same time. However the system integrator can only select
+a single "release" build type.
+
+
Building and Installing
-----------------------
When building from a clean environment where no generated build files exist, it is necessary to run
diff --git a/docs/project/quality-assurance/verification-methodology.rst b/docs/project/quality-assurance/verification-methodology.rst
index ce421e8..3dedf00 100644
--- a/docs/project/quality-assurance/verification-methodology.rst
+++ b/docs/project/quality-assurance/verification-methodology.rst
@@ -62,11 +62,31 @@
The project hosts deployment helping compliance testing. For more information please refer to
:ref:`Platform Certification`.
+Measuring Quality of testing
+''''''''''''''''''''''''''''
+
+The primary metric to measure runtime testing quality is code coverage. The project uses two methods to measure code
+coverage:
+
+ - "Host side" coverage measurement, where code coverage is measured using Unit Tests and Component Test executed on
+ the build host.
+
+ The measurement relies on GCCs `-ftest-coverage` compilation switch and the lcov tool. In order to include
+ coverage data of PSA client code and of service providers, `libpsats` and `libts` have to be built using
+ the `DebugCoverage` build type. If the build type override is not used, the build system will automatically build
+ and use the appropriate binaries.
+
+ - Coverage measurement on the target. This feature is only supported by the :ref:`AEM FVP` platform and uses
+ `qa-tools`_.
+
+Coverage measurement is automated in the CI, which is not publicly accessible yet.
+
------------------
.. _`Checkpatch`: https://docs.kernel.org/dev-tools/checkpatch.html
.. _`CppCheck`: https://cppcheck.sourceforge.io/
.. _`codespell dictionary v2.3.0`: https://raw.githubusercontent.com/codespell-project/codespell/v2.3.0/codespell_lib/data/dictionary.txt
+.. _`qa-tools`: https://gitlab.arm.com/tooling/qa-tools
*Copyright (c) 2022, Arm Limited and Contributors. All rights reserved.*