Docs: Documentation about adding test suites in TF-M
With the test refinements and decoupling work between TF-M repo
and test repo, it is necessary to create a new guide about adding
test suites and test configurations.
Signed-off-by: Jianliang Shen <jianliang.shen@arm.com>
Change-Id: Ie4bff72f98ec52c702f59c2e8aeb09bfc9eb2df1
diff --git a/docs/integration_guide/tfm_test_suites_addition.rst b/docs/integration_guide/tfm_test_suites_addition.rst
new file mode 100644
index 0000000..a5a89a5
--- /dev/null
+++ b/docs/integration_guide/tfm_test_suites_addition.rst
@@ -0,0 +1,320 @@
+#################################
+Adding TF-M Regression Test Suite
+#################################
+
+.. contents:: Table of Contents
+
+*************************************
+Introduction of TF-M regression tests
+*************************************
+
+TF-M regression tests test whether changes to TF-M features work as expected.
+This documentation focus on three parts in TF-M regression tests:
+
+1. ``test suite``: A series of tests of a certain function.
+2. ``test case``: A specific test instance in test suites.
+3. ``test service``: An extra module works like secure partitions to support
+ related test suites.
+
+Source structure
+================
+
+TF-M tests source code are located in
+`tf-m-tests <https://git.trustedfirmware.org/TF-M/tf-m-tests.git/>`__.
+
++----------------+-------------------------------------------------------------+
+| Folder name | Description |
++================+=============================================================+
+| test/config | The CMAKE test configurations files. |
++----------------+-------------------------------------------------------------+
+| test/framework | Source code for test framework code, managing test suites. |
++----------------+-------------------------------------------------------------+
+| test/suites | Test suites divided into subdirectories. |
++----------------+-------------------------------------------------------------+
+| test/services | Test services divided into subdirectories. |
++----------------+-------------------------------------------------------------+
+
+Test configuration
+==================
+
+A test configuration controls whether one or multiple test suites are enabled.
+The doc :doc:`TF-M Build Instructions </docs/technical_references/instructions/tfm_build_instruction>`
+shows some test configurations which are already supported in current TF-M.
+An example usage of test configuration shows below.
+
+.. code-block:: bash
+
+ cmake -S . -B cmake_build -DTFM_PLATFORM=arm/mps2/an521 \
+ -DTFM_TOOLCHAIN_FILE=toolchain_GNUARM.cmake \
+ -DCMAKE_BUILD_TYPE=Release \
+ -DTFM_PSA_API=ON \
+ -DTEST_NS_ATTESTATION=ON
+
+With this command, only non-secure initial attestation test suite will be built.
+
+***********************
+Adding a new test suite
+***********************
+
+This section introduces how to add a new test suite and control its compilation
+with a test configuration in ``tf-m-tests`` repository.
+
+Source code
+===========
+
+The test suite example subdirectory named ``<test_name>`` is located under the path
+``tf-m-tests/test/suites``. If the new test suite includes both non-secure and
+secure parts, the source code shall be divided shared code and specific code.
+An example test suite folder can be organized as the figure below.
+
+.. code-block:: bash
+
+ .
+ ├── CMakeLists.txt
+ ├── <test_name>_tests_common.c
+ ├── non_secure
+ │ ├── <test_name>_ns_interface_testsuite.c
+ │ └── <test_name>_ns_tests.h
+ └── secure
+ ├── <test_name>_s_interface_testsuite.c
+ └── <test_name>_s_tests.h
+
+Creating test configurations
+============================
+
+Developers shall assign corresponding test configurations to control the test
+suites.
+
+Naming test configurations
+--------------------------
+
+The test configurations of example test suites are ``TEST_NS_<TEST_NAME>``
+in non-secure and ``TEST_S_<TEST_NAME>`` in secure.
+
+.. Note::
+ The test configurations must be named with the prefixes ``TEST_S_`` and
+ ``TEST_NS_``, for secure regression tests and non-secure regression tests
+ respectively. Otherwise, tf-m-tests build system may not recognize it.
+
+Setting test configurations
+---------------------------
+
+When the test configurations have dependences, the default value need to be set.
+The setting is performed in following four steps.
+
+#. Command line input: The configuration can be enabled or disabled by the
+ command ``-DTEST_NS_<TEST_NAME>=ON/OFF -DTEST_S_<TEST_NAME>=ON/OFF``, and
+ the value cannot be changed throughout the whole compilation of TF-M tests.
+
+#. ``tf-m-tests/config/set_config.cmake``: The test configurations shall be
+ OFF if its dependences are not supported. The dependences are probably
+ from:
+
+ #. TF-M partitions configurations like ``TFM_PARTITION_CRYPTO``,
+ ``TFM_PARTITION_INITIAL_ATTESTATION``, etc.
+ #. TF-M build mode configuration like ``TFM_PSA_API``.
+ #. TF-M other configurations like ``TFM_PARTITION_FIRMWARE_UPDATE``,
+ ``FORWARD_PROT_MSG``, etc.
+
+#. ``tf-m-tests/config/default_ns_test_config.cmake`` or
+ ``tf-m-tests/config/default_s_test_config.cmake``: It is required to give
+ the default value of the new test configuration in these two files when
+ ``TEST_NS`` or ``TEST_S`` is ON. The recommended value is ON unless the
+ single test's code or data size is very large.
+
+#. ``tf-m-tests/config/default_test_config.cmake``: It is required to give the
+ default value of the new test configuration in the file when both
+ ``TEST_NS`` and ``TEST_S`` are OFF. The default value must be OFF.
+
+.. Note::
+ The test configurations must be set as CACHE value in CMAKE files. The CACHE
+ set cannot replace the value from command line, see
+ `Set Cache Entry <https://cmake.org/cmake/help/latest/command/set.html#set-cache-entry>`__.
+
+Checking test configurations
+----------------------------
+
+The new test configurations must be checked by function ``tfm_invalid_config()``
+if they have any dependence. The value comes from command line may be wrong when
+the dependences are conflicting. In addition to the dependences quoted in
+``tf-m-tests/config/set_config.cmake``, some other test configurations may be
+necessary.
+
+Applicating test configurations
+===============================
+
+The mission of test configurations is to control the build. They are applied
+in ``test/suites/<test_name>/CMakeLists.txt`` like the example below.
+
+.. code-block:: cmake
+
+ cmake_policy(SET CMP0079 NEW)
+
+ if (NOT TEST_NS_<TEST_NAME> AND NOT TEST_S_<TEST_NAME>)
+ return()
+ endif()
+
+ ####################### Non Secure #########################################
+
+ if (TEST_NS_<TEST_NAME>)
+ add_library(tfm_test_suite_<test_name>_ns STATIC EXCLUDE_FROM_ALL)
+ # target_sources()
+ # target_include_directories()
+ target_compile_definitions(tfm_test_suite_<test_name>_ns
+ INTERFACE
+ TEST_NS_<TEST_NAME>
+ )
+ # target_link_libraries()
+ endif()
+
+ ####################### Secure #############################################
+
+ if (TEST_S_<TEST_NAME>)
+ add_library(tfm_test_suite_<test_name>_s STATIC EXCLUDE_FROM_ALL)
+ # target_sources()
+ # target_include_directories()
+ target_compile_definitions(tfm_test_suite_<test_name>_s
+ INTERFACE
+ TEST_S_<TEST_NAME>
+ )
+ # target_link_libraries()
+ endif()
+
+The function ``target_compile_definitions`` will export the macros
+``TEST_NS_<TEST_NAME>`` or ``TEST_S_<TEST_NAME>`` into source code. and in the
+file ``tf-m-tests/framework/non_secure_suites.c`` or
+``tests/framework/secure_suites.c``, the definitions of these macros will be
+checked, and then the head file will be included and test cases will be
+registered if the macro is defined.
+
+.. code-block:: c
+
+ #ifdef TEST_NS_<TEST_NAME>
+ #include "<test_name>_ns_tests.h"
+ #endif
+
+ static struct test_suite_t test_suites[] = {
+ /* Non-secure example test cases */
+ // ......
+ #ifdef TEST_NS_<TEST_NAME>
+ {®ister_testsuite_ns_<test_name>_interface, 0, 0, 0},
+ #endif
+ };
+
+.. code-block:: c
+
+ #ifdef TEST_S_<TEST_NAME>
+ #include "<test_name>_s_tests.h"
+ #endif
+
+ static struct test_suite_t test_suites[] = {
+ /* Secure example test cases */
+ // ......
+ #ifdef TEST_S_<TEST_NAME>
+ {®ister_testsuite_s_<test_name>_interface, 0, 0, 0},
+ #endif
+ };
+
+.. Note::
+ On most platforms non-secure tests and secure tests run on the same CPU
+ core, but dual-core platform is an exception. So secure test library and
+ secure sevices shall be linked together in the file
+ ``tf-m-tests/test/test_services/CMakeLists.txt``. Thus they can be built on
+ secure CPU core and non-secure tests library and RTOS are built on
+ non-secure CPU core.
+
+.. code-block:: cmake
+
+ if (TEST_FRAMEWORK_S)
+ # ...
+ if (TEST_S_<TEST_NAME>)
+ add_library(tfm_test_suite_<test_name>_s STATIC EXCLUDE_FROM_ALL)
+ endif()
+ endif()
+
+************************************
+Adding a new test case in test suite
+************************************
+
+The test cases usually express as a function in source code. They will be added
+into an array with structure type called ``test_t`` defined in
+``tf-m-tests/test/framework/test_framework.h``.
+
+.. code-block:: c
+
+ struct test_t {
+ TEST_FUN * const test; /*!< Test function to call */
+ const char *name; /*!< Test name */
+ const char *desc; /*!< Test description */
+ struct test_result_t ret; /*!< Test result */
+ };
+
+For example, a new test case called ``TFM_NS_<TEST_NAME>_TEST_1001`` is created
+and the function ``tfm_<test_name>_test_1001`` needs to be defined in file
+``<test_name>_ns_interface_testsuite.c``. Then the function shall be appended
+into the array which will be quoted in function
+``register_testsuite_ns_<test_name>_interface``. See the reference code below.
+
+.. code-block:: c
+
+ /* List of test cases */
+ static void tfm_<test_name>_test_1001(struct test_result_t *ret);
+
+ /* Append test cases */
+ static struct test_t <test_name>_tests[] = {
+ {&tfm_<test_name>_test_1001, "TFM_NS_<TEST_NAME>_TEST_1001",
+ "Example test case", {TEST_PASSED}},
+ };
+
+ /* Register test case into test suites */
+ void register_testsuite_ns_<test_name>_interface(struct test_suite_t *p_test_suite)
+ {
+ uint32_t list_size;
+
+ list_size = (sizeof(<test_name>_tests) / sizeof(<test_name>_tests[0]));
+
+ set_testsuite("<TEST_NAME> non-secure interface test (TFM_NS_<TEST_NAME>_TEST_1XXX)",
+ <test_name>_tests, list_size, p_test_suite);
+ }
+
+ static void tfm_<test_name>_test_1001(struct test_result_t *ret)
+ {
+ /* test case code */
+ }
+
+********************
+Adding test services
+********************
+
+Some test group may need specific test services. These test services may support
+one or more groups thus developers shall determine the proper test scope.
+
+Steps
+=====
+
+Adding a test service is same as adding a secure partition, generally the
+process can be referenced from the document
+:doc:`Adding Secure Partition </docs/integration_guide/services/tfm_secure_partition_addition>`
+
+.. Note::
+ Each test service must have resource requirements declared in a manifest
+ file, the contents of test services are the same as secure partitions,but
+ their locations are different. Test service manifests shall be set in
+ ``tf-m-tests/test/test_services/tfm_test_manifest_list.yaml``.
+
+Configuration
+=============
+
+If the new test service names ``tfm_<test_name>_test_service`` only supports for the
+example test, the configuration in
+``tf-m-tests/test/test_services/CMakeLists.txt`` forms like below.
+
+.. code-block:: cmake
+
+ if (TEST_S_<TEST_NAME> OR TEST_NS_<TEST_NAME>)
+ add_subdirectory(tfm_<test_name>_test_service)
+ endif()
+
+--------------
+
+*Copyright (c) 2021, Arm Limited. All rights reserved.*