docs/integration_guide/tfm_test_suites_addition.rst

#################################
Adding TF-M Regression Test Suite
#################################

.. contents:: Table of Contents

*************************************
Introduction of TF-M regression tests
*************************************

TF-M regression tests test whether changes to TF-M code work as expected.
This documentation focuses on three parts in TF-M regression tests:

1. ``test suite``: A series of tests of a certain function.
2. ``test case``: A specific test instance in test suites.
3. ``test service``: An extra module works like secure partitions to support
   related test suites.

Source structure
================

TF-M tests source code are located in
`tf-m-tests <https://git.trustedfirmware.org/TF-M/tf-m-tests.git/>`__.

+----------------+-------------------------------------------------------------+
| Folder name    | Description                                                 |
+================+=============================================================+
| test/config    | The CMAKE test configurations files.                        |
+----------------+-------------------------------------------------------------+
| test/framework | Source code for test framework code, managing test suites.  |
+----------------+-------------------------------------------------------------+
| test/suites    | Test suites divided into subdirectories.                    |
+----------------+-------------------------------------------------------------+
| test/services  | Test services divided into subdirectories.                  |
+----------------+-------------------------------------------------------------+

Test configuration
==================

A test configuration controls whether one or multiple test suites are enabled.
The doc :doc:`TF-M Build Instructions </docs/technical_references/instructions/tfm_build_instruction>`
shows some test configurations which are already supported in current TF-M.
An example usage of test configuration shows below.

.. code-block:: bash

    cmake -S . -B cmake_build -DTFM_PLATFORM=arm/mps2/an521               \
                              -DTFM_TOOLCHAIN_FILE=toolchain_GNUARM.cmake \
                              -DCMAKE_BUILD_TYPE=Release                  \
                              -DTEST_NS_ATTESTATION=ON

With this command, only non-secure initial attestation test suite will be built.

***********************
Adding a new test suite
***********************

This section introduces how to add a new test suite and control its compilation
with a test configuration in ``tf-m-tests`` repository.

Source code
===========

The test suite example subdirectory named ``<test_name>`` is located under the path
``tf-m-tests/test/suites``. If the new test suite includes both non-secure and
secure parts, the source code shall be divided shared code and specific code.
An example test suite folder can be organized as the figure below.

.. code-block:: bash

    .
    ├── CMakeLists.txt
    ├── <test_name>_tests_common.c
    ├── non_secure
    │   ├── <test_name>_ns_interface_testsuite.c
    │   └── <test_name>_ns_tests.h
    └── secure
        ├── <test_name>_s_interface_testsuite.c
        └── <test_name>_s_tests.h

Creating test configurations
============================

Developers shall assign corresponding test configurations to control the test
suites.

Naming test configurations
--------------------------

The test configurations of example test suites are ``TEST_NS_<TEST_NAME>``
in non-secure and ``TEST_S_<TEST_NAME>`` in secure.

.. Note::
    The test configurations must be named with the prefixes ``TEST_S_`` and
    ``TEST_NS_``, for secure regression tests and non-secure regression tests
    respectively. Otherwise, tf-m-tests build system may not recognize it.

Setting test configurations
---------------------------

When the test configurations have dependences, the default value need to be set.
The setting is performed in following four steps.

#. Command line input: The configuration can be enabled or disabled by the
   command ``-DTEST_NS_<TEST_NAME>=ON/OFF -DTEST_S_<TEST_NAME>=ON/OFF``, and
   the value cannot be changed throughout the whole compilation of TF-M tests.

#. ``tf-m-tests/config/set_config.cmake``: The test configurations shall be
   OFF if its dependences are not supported. The dependences are probably
   from:

    #. TF-M partitions configurations like ``TFM_PARTITION_CRYPTO``,
       ``TFM_PARTITION_INITIAL_ATTESTATION``, etc.
    #. TF-M build mode configuration like ``TFM_LIB_MODEL``.
    #. TF-M other configurations like ``TFM_PARTITION_FIRMWARE_UPDATE``,
       ``FORWARD_PROT_MSG``, etc.

#. ``tf-m-tests/config/default_ns_test_config.cmake`` or
   ``tf-m-tests/config/default_s_test_config.cmake``: It is required to give
   the default value of the new test configuration in these two files when
   ``TEST_NS`` or ``TEST_S`` is ON. The recommended value is ON unless the
   single test's code or data size is very large.

#. ``tf-m-tests/config/default_test_config.cmake``: It is required to give the
   default value of the new test configuration in the file when both
   ``TEST_NS`` and ``TEST_S`` are OFF. The default value must be OFF.

.. Note::
   The test configurations must be set as CACHE value in CMAKE files. The CACHE
   set cannot replace the value from command line, see
   `Set Cache Entry <https://cmake.org/cmake/help/latest/command/set.html#set-cache-entry>`__.

Checking test configurations
----------------------------

The new test configurations must be checked by function ``tfm_invalid_config()``
if they have any dependence. The value comes from command line may be wrong when
the dependences are conflicting. In addition to the dependences quoted in
``tf-m-tests/config/set_config.cmake``, some other test configurations may be
necessary.

Applicating test configurations
===============================

The mission of test configurations is to control the build. They are applied
in ``test/suites/<test_name>/CMakeLists.txt`` like the example below.

.. code-block:: cmake

    cmake_policy(SET CMP0079 NEW)

    if (NOT TEST_NS_<TEST_NAME> AND NOT TEST_S_<TEST_NAME>)
        return()
    endif()

    ####################### Non Secure #########################################

    if (TEST_NS_<TEST_NAME>)
        add_library(tfm_test_suite_<test_name>_ns STATIC EXCLUDE_FROM_ALL)
        # target_sources()
        # target_include_directories()
        target_compile_definitions(tfm_test_suite_<test_name>_ns
            INTERFACE
                TEST_NS_<TEST_NAME>
        )
        # target_link_libraries()
    endif()

    ####################### Secure #############################################

    if (TEST_S_<TEST_NAME>)
        add_library(tfm_test_suite_<test_name>_s STATIC EXCLUDE_FROM_ALL)
        # target_sources()
        # target_include_directories()
        target_compile_definitions(tfm_test_suite_<test_name>_s
            INTERFACE
                TEST_S_<TEST_NAME>
        )
        # target_link_libraries()
    endif()

The function ``target_compile_definitions`` will export the macros
``TEST_NS_<TEST_NAME>`` or ``TEST_S_<TEST_NAME>`` into source code. and in the
file ``tf-m-tests/framework/non_secure_suites.c`` or
``tests/framework/secure_suites.c``, the definitions of these macros will be
checked, and then the head file will be included and test cases will be
registered if the macro is defined.

.. code-block:: c

    #ifdef TEST_NS_<TEST_NAME>
    #include "<test_name>_ns_tests.h"
    #endif

    static struct test_suite_t test_suites[] = {
    /* Non-secure example test cases */
        // ......
    #ifdef TEST_NS_<TEST_NAME>
        {&register_testsuite_ns_<test_name>_interface, 0, 0, 0},
    #endif
    };

.. code-block:: c

    #ifdef TEST_S_<TEST_NAME>
    #include "<test_name>_s_tests.h"
    #endif

    static struct test_suite_t test_suites[] = {
    /* Secure example test cases */
        // ......
    #ifdef TEST_S_<TEST_NAME>
        {&register_testsuite_s_<test_name>_interface, 0, 0, 0},
    #endif
    };

.. Note::
    On most platforms non-secure tests and secure tests run on the same CPU
    core, but dual-core platform is an exception. So secure test library and
    secure sevices shall be linked together in the file
    ``tf-m-tests/test/test_services/CMakeLists.txt``. Thus they can be built on
    secure CPU core and non-secure tests library and RTOS are built on
    non-secure CPU core.

.. code-block:: cmake

    if (TEST_FRAMEWORK_S)
        # ...
        if (TEST_S_<TEST_NAME>)
            add_library(tfm_test_suite_<test_name>_s STATIC EXCLUDE_FROM_ALL)
        endif()
    endif()

************************************
Adding a new test case in test suite
************************************

The test cases usually express as a function in source code. They will be added
into an array with structure type called ``test_t`` defined in
``tf-m-tests/test/framework/test_framework.h``.

.. code-block:: c

    struct test_t {
        TEST_FUN * const test;         /*!< Test function to call */
        const char *name;              /*!< Test name */
        const char *desc;              /*!< Test description */
        struct test_result_t ret;      /*!< Test result */
    };

For example, a new test case called ``TFM_NS_<TEST_NAME>_TEST_1001`` is created
and the function ``tfm_<test_name>_test_1001`` needs to be defined in file
``<test_name>_ns_interface_testsuite.c``. Then the function shall be appended
into the array which will be quoted in function
``register_testsuite_ns_<test_name>_interface``. See the reference code below.

.. code-block:: c

    /* List of test cases */
    static void tfm_<test_name>_test_1001(struct test_result_t *ret);

    /* Append test cases */
    static struct test_t <test_name>_tests[] = {
        {&tfm_<test_name>_test_1001, "TFM_NS_<TEST_NAME>_TEST_1001",
        "Example test case", {TEST_PASSED}},
    };

    /* Register test case into test suites */
    void register_testsuite_ns_<test_name>_interface(struct test_suite_t *p_test_suite)
    {
        uint32_t list_size;

        list_size = (sizeof(<test_name>_tests) / sizeof(<test_name>_tests[0]));

        set_testsuite("<TEST_NAME> non-secure interface test (TFM_NS_<TEST_NAME>_TEST_1XXX)",
                        <test_name>_tests, list_size, p_test_suite);
    }

    static void tfm_<test_name>_test_1001(struct test_result_t *ret)
    {
       /* test case code */
    }

********************
Adding test services
********************

Some test group may need specific test services. These test services may support
one or more groups thus developers shall determine the proper test scope.

Steps
=====

Adding a test service is same as adding a secure partition, generally the
process can be referenced from the document
:doc:`Adding Secure Partition </docs/integration_guide/services/tfm_secure_partition_addition>`

.. Note::
    Each test service must have resource requirements declared in a manifest
    file, the contents of test services are the same as secure partitions,but
    their locations are different. Test service manifests shall be set in
    ``tf-m-tests/test/test_services/tfm_test_manifest_list.yaml``.

Configuration
=============

If the new test service names ``tfm_<test_name>_test_service`` only supports for the
example test, the configuration in
``tf-m-tests/test/test_services/CMakeLists.txt`` forms like below.

.. code-block:: cmake

    if (TEST_S_<TEST_NAME> OR TEST_NS_<TEST_NAME>)
        add_subdirectory(tfm_<test_name>_test_service)
    endif()

**********************************
Out-of-tree regression test suites
**********************************

TF-M supports out-of-tree regression test suites build, whose source code
folders are maintained outside tf-m-tests repo. There are two configurations
for developers to include the source code.

- ``EXTRA_NS_TEST_SUITES_PATHS``

  A list of the absolute directories of the out-of-tree non-secure test suites
  source code folder(s). TF-M build system searches ``CMakeLists.txt`` of
  non-secure test suites in the source code folder(s).
  Use semicolons ``;`` to separate multiple out-of-tree non-secure test suites
  directorires.

- ``EXTRA_S_TEST_SUITES_PATHS``

  A list of the absolute directories of the out-of-tree secure test suites
  source code folder(s).

Example usage
=============

Take non-secure test as an example in
`tf-m-extras <https://git.trustedfirmware.org/TF-M/tf-m-extras.git/>`__.
A single out-of-tree test suite folder can be organized as the figure below:

.. code-block:: bash

    extra_ns
    ├── CMakeLists.txt
    ├── ns_test.c
    ├── ns_test_config.cmake
    └── ns_test.h

In the example above, ``EXTRA_NS_TEST_SUITES_PATHS`` in the build command can be
specified as listed below.

.. code-block:: bash

  -DEXTRA_NS_TEST_SUITES_PATHS=<Absolute-path-extra-test-folder>

Coding instructions
===================

This is a demo of source code so the structure has been simplified. Files like
``ns_test.c`` and ``ns_test.h`` can be expanded to ``src`` and ``include``
folders. The ``CMakeLists.txt`` is required in the root path and
``ns_test_config.cmake`` is optional.

Header Files
------------

The header file ``extra_ns_tests.h`` must be included by out-of-tree source
code. This file contains the definition of ``struct extra_tests_t``,
``int32_t register_extra_tests()`` and declaration of
``int32_t extra_ns_tests_init()``.

Source code
-----------

To connect the out-of-tree source code and tf-m-tests framework, the high-level
test function must be defined first. An example format is:

.. code-block:: c

    int32_t ns_test(void)
    {
        /* Add platform specific non-secure test suites code here. */

        return EXTRA_TEST_SUCCESS;
    }

This function is the main entry to test framework. All the out-of-tree's test
cases can be added into it. The returned error code of this function is
specified as ``int32_t``. This function shall return an expected value which is
same as the part ``expected_ret`` set in ``plat_ns_t`` below, take the macro
EXTRA_TEST_SUCCESS as an example.

After ``ns_test()`` is defined, a structure variable need to be created like:

.. code-block:: c

    const struct extra_tests_t plat_ns_t = {
        .test_entry = ns_test,
        .expected_ret = EXTRA_TEST_SUCCESS
    };

It will be used by function ``extra_ns_tests_init()`` to register the test by
function ``register_extra_tests()``:

.. code-block:: c

    int32_t extra_ns_tests_init(struct extra_tests_t *internal_test_t)
    {
        /* Add platform init code here. */

        return register_extra_tests(internal_test_t, &plat_ns_t);
    }

The platform initialization code can be added in this function because it runs
before ``ns_test()``.

.. Note::
    Function ``extra_ns_tests_init()`` is declared in tf-m-tests repository
    without definition. It is supplied to out-of-tree source code and need to be
    defined with no change of its format, like returns error code and parameter
    name.


CMakeLists.txt
--------------

In addition to the implementation of CMAKE target like ``example_test_ns`` of
out-of-tree source code, the configuration below needs to be appended:

.. code-block:: cmake

    # Example test must link tfm_test_suite_extra_common to use related interface
    target_link_libraries(example_test_ns
        PRIVATE
            tfm_test_suite_extra_common
    )

    # The example_test_ns library must be linked by tfm_test_suite_extra_common
    target_link_libraries(tfm_test_suite_extra_ns
        PRIVATE
            example_test_ns
    )

To use the interfaces come from tf-m-tests repository, library
``tfm_test_suite_extra_common`` must be linked by ``example_test_ns``.
To add out-of-tree test into TF-M, library ``example_test_ns`` must be linked
by ``tfm_test_suite_extra_ns``.

ns_test_config.cmake
--------------------

The CMAKE configuration file is optional. If out-of-tree source already exists
another configuration file, a new one can be ignored.

--------------

*Copyright (c) 2021, Arm Limited. All rights reserved.*