docs/project/quality-assurance/verification-strategy.rst

Verification Strategy
=====================

This page describes verification from a high level concept perspective.

In this context ``source code`` has a wider scope and may mean any text content produced by humans and processed by
other humans or tools. Examples: C/C++ source code, reST documents, build scripts, etc...

Clean Code
----------

Clean code aims to counterfeit issues discussed in the following sub-chapters.

Code Readability
''''''''''''''''

Expressing ideas in a machine readable formats is complicated, and each developer may have a different taste or
different preferences on how source code should be formatted. Some people may find a specific kind of formatting easier
to understand than others. If the source code does not follow a consistent look and feel, human processing of the text
may become error prone. This undermines effectiveness of co-operation and code-review, and may lead to incorrect code.
The project defines coding style rules to counterfeit these problems. For detail please refer to :ref:`Coding Style &
Guidelines`

Undefined and Implementation Defined Behavior
'''''''''''''''''''''''''''''''''''''''''''''

The "standard" defining how to process a specific source code type, may leave some processing behavior to the tool
to be defined, or allow the tool to behave in an undefined way. Coding constructs relying on such behavior are to be
avoided, or used in a well defined way. This adds robustness and helps avoiding errors due to using different version
of the same tool, or different implementations.

The project defines coding guidelines to counterfeit these problems. For detail please refer to
:ref:`Coding Style & Guidelines`

Security
''''''''

Security is a complex topic affecting all steps of the development process. Incorrect code may lead to security issues
and thus "Clean Code" has a vital role in implementing secure software.

Runtime Testing
---------------

Runtime testing focuses of verifying the behavior of one or multiple build products built from source code. This can be
done at multiple levels and in multiple execution environment.


Unit Test
'''''''''

Unit tests aim to verify if the internal operation of a module matches the developers expectation. It helps covering all
code execution paths, and to give confidence on correct operation when code needs to be refactored. Unit tests serve as
a kind of documentation capturing the expected usage of the code.

Unit-testing allays happen on the "host PC"

Component Test
''''''''''''''

Component tests aim to verify the API (and ABI) of a component is matching expectations. Components are tested in
isolation, where the exported APIs are exercised by these code, and APIs the component depends on are implemented by
test doubles.

System Test
'''''''''''

System test verifies correct operation of a set of modules configured to fulfill the requirements of a use-case. For TS
this usually means testing and end-to-end setup on a specific target platform.

Balancing Costs vs Quality
''''''''''''''''''''''''''

Executing build products on target platforms may have high costs in terms of time, complexity and availability and in
turn it gives the hights confidence in verification results, or the best quality. In the development phase it may be
desired to some level of this confidence for lower costs. For this purpose the project defines two main test
set-up types based on the balance choice between cost and quality.

Testing on a Target Platform
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In this environment tests are executed on a target platform. Emulators (e.g. QEMU, FVP) from this aspect are treated
like targets implemented in silicon.

Testing on the Host Machine
~~~~~~~~~~~~~~~~~~~~~~~~~~~

In this environment test executables are compiled to execute as a "standard" user-space application running on the
machine "hosting" the development activity. In most cases these machines are based on a different architecture that the
ones the project is targeting (e.g. x86-64 vs aarch64). This means this environment relies on the assumption that code
is portable and behaves architecture and compiler independent. This puts limitations on the features which can be tested
and lower the confidence level of test output. In turn executing tests int his environment is simple and gives very good
scalability options.

--------------

*Copyright (c) 2022, Arm Limited and Contributors. All rights reserved.*

SPDX-License-Identifier: BSD-3-Clause