1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
|
#################################
Adding TF-M Regression Test Suite
#################################
.. contents:: Table of Contents
*************************************
Introduction of TF-M regression tests
*************************************
TF-M regression tests test whether changes to TF-M code work as expected.
A new regression test can consist of following 3 components:
1. ``test suite``: A series of tests of a certain function.
2. ``test case``: A specific test instance in test suites.
3. ``test service or partition``: Optional secure services or partitions to
support related test suites.
Source structure
================
+---------------------------------------+---------------------------------------------------------------+
| Folder name | Description |
+=======================================+===============================================================+
| test/bl1 | TF-M bl1 test suites source code. |
+---------------------------------------+---------------------------------------------------------------+
| test/bl2 | TF-M bL2 test suites source code. |
+---------------------------------------+---------------------------------------------------------------+
| test/config | The CMAKE test configurations files. |
+---------------------------------------+---------------------------------------------------------------+
| test/framework | Source code for test framework code, managing test suites. |
+---------------------------------------+---------------------------------------------------------------+
| test/secure_fw/suites | Test suites divided into subdirectories. |
+---------------------------------------+---------------------------------------------------------------+
| test/secure_fw/suites/<suite>/service | Test service divided into corresponding suite subdirectories. |
+---------------------------------------+---------------------------------------------------------------+
| test/secure_fw/common_test_services | Common test services. |
+---------------------------------------+---------------------------------------------------------------+
***********************
Adding a new test suite
***********************
This section introduces how to add a new test suite and control its compilation
with a test configuration in ``tests_reg/test`` repository.
Source code
===========
The test suite example subdirectory named ``<test_name>`` is located under the
path ``tests_reg/test/secure_fw/suites``. If the new test suite includes both
non-secure and secure parts, the source code shall be divided shared code and
specific code. An example test suite folder can be organized as the figure
below.
.. code-block:: bash
.
├── <test_name>_tests_common.c
├── non_secure
| ├── CMakeLists.txt
│ ├── <test_name>_ns_interface_testsuite.c
│ └── <test_name>_ns_tests.h
└── secure
├── CMakeLists.txt
├── <test_name>_s_interface_testsuite.c
└── <test_name>_s_tests.h
Creating test configurations
============================
A test configuration controls whether one or multiple test suites are enabled.
The doc :doc:`TF-M Build Instructions <TF-M:building/tfm_build_instruction>`.
shows some test configurations which are already supported in current TF-M.
Developers shall assign corresponding test configurations to control the test
suites.
Naming test configurations
--------------------------
The test configurations of example test suites are ``TEST_NS_<TEST_NAME>``
in non-secure and ``TEST_S_<TEST_NAME>`` in secure.
.. Note::
The test configurations must be named with the prefixes ``TEST_S_`` and
``TEST_NS_``, for secure regression tests and non-secure regression tests
respectively. Otherwise, tests_reg/test build system may not recognize it.
Setting test configurations
---------------------------
When the test configurations have dependencies, the default value need to be set.
The setting is performed in following four steps.
#. Command line input: The configuration can be enabled or disabled by the
command ``-DTEST_NS_<TEST_NAME>=ON/OFF -DTEST_S_<TEST_NAME>=ON/OFF``, and
the value cannot be changed throughout the whole compilation of TF-M tests.
#. ``tests_reg/test/config/config.cmake``: The test configurations shall be
``OFF`` if its dependencies are not supported. The dependencies are probably from:
#. TF-M partitions configurations like ``TFM_PARTITION_CRYPTO``,
``TFM_PARTITION_INITIAL_ATTESTATION``, etc.
#. TF-M build mode configuration like ``CONFIG_TFM_SPM_BACKEND``.
#. TF-M other configurations like ``TFM_PARTITION_FIRMWARE_UPDATE``, etc.
#. ``tests_reg/test/config/default_ns_test_config.cmake`` or
``tests_reg/test/config/default_s_test_config.cmake``: It is required to give
the default value of the new test configuration in these two files when
``TEST_NS`` or ``TEST_S`` is ON. The recommended value is ON unless the
single test's code or data size is very large.
#. ``tests_reg/test/config/default_test_config.cmake``: It is required to give the
default value of the new test configuration in the file when both
``TEST_NS`` and ``TEST_S`` are OFF. The default value must be OFF.
.. Note::
The test configurations must be set as CACHE value in CMAKE files. The CACHE
set cannot replace the value from command line, see
`Set Cache Entry <https://cmake.org/cmake/help/latest/command/set.html#set-cache-entry>`_.
Checking test configurations
----------------------------
The new test configurations must be checked by function ``tfm_invalid_config()``
if they have any dependence. The value comes from command line may be wrong when
the dependencies are conflicting.
Implement necessary checks in ``tests_reg/test/config/check_config.cmake``.
Export TF-M configurations
==========================
If the new test depends on some TF-M configurations, export their value via
``tests_reg/test/config/config_ns_tests.cmake.in``.
TF-M secure build will install ``config_ns_tests.cmake`` and export configuration values.
tf-m-tests non-secure build will include ``config_ns_tests.cmake`` and receive TF-M configuration
values.
Applying test configurations
============================
The mission of test configurations is to control the build. They are applied
in ``test/secure_fw/suites/<test_name>/CMakeLists.txt`` like the example below.
.. code-block:: cmake
if (NOT TEST_NS_<TEST_NAME> AND NOT TEST_S_<TEST_NAME>)
return()
endif()
####################### Non Secure #########################################
if (TEST_NS_<TEST_NAME>)
add_library(tfm_test_suite_<test_name>_ns STATIC EXCLUDE_FROM_ALL)
# target_sources()
# target_include_directories()
target_compile_definitions(tfm_test_suite_<test_name>_ns
INTERFACE
TEST_NS_<TEST_NAME>
)
# target_link_libraries()
endif()
####################### Secure #############################################
if (TEST_S_<TEST_NAME>)
add_library(tfm_test_suite_<test_name>_s STATIC EXCLUDE_FROM_ALL)
# target_sources()
# target_include_directories()
target_compile_definitions(tfm_test_suite_<test_name>_s
INTERFACE
TEST_S_<TEST_NAME>
)
# target_link_libraries()
endif()
The function ``target_compile_definitions`` will export the macros
``TEST_NS_<TEST_NAME>`` or ``TEST_S_<TEST_NAME>`` into source code. and in the
file ``tests_reg/test/framework/non_secure_suites.c`` or
``tests/framework/secure_suites.c``, the definitions of these macros will be
checked, and then the head file will be included and test cases will be
registered if the macro is defined.
.. code-block:: c
#ifdef TEST_NS_<TEST_NAME>
#include "<test_name>_ns_tests.h"
#endif
static struct test_suite_t test_suites[] = {
/* Non-secure example test cases */
// ......
#ifdef TEST_NS_<TEST_NAME>
{®ister_testsuite_ns_<test_name>_interface, 0, 0, 0},
#endif
};
.. code-block:: c
#ifdef TEST_S_<TEST_NAME>
#include "<test_name>_s_tests.h"
#endif
static struct test_suite_t test_suites[] = {
/* Secure example test cases */
// ......
#ifdef TEST_S_<TEST_NAME>
{®ister_testsuite_s_<test_name>_interface, 0, 0, 0},
#endif
};
.. Note::
On most platforms non-secure tests and secure tests run on the same CPU
core, but dual-core platform is an exception. So secure test library and
secure services shall be linked together in the file
``tests_reg/test/test/secure_fw/secure_tests.cmake``. Thus they can be built on
secure CPU core and non-secure tests library and RTOS are built on
non-secure CPU core.
.. code-block:: cmake
# ...
if (TEST_S_<TEST_NAME>)
add_library(tfm_test_suite_<test_name>_s STATIC EXCLUDE_FROM_ALL)
endif()
************************************
Adding a new test case in test suite
************************************
The test cases usually express as a function in source code. They will be added
into an array with structure type called ``test_t`` defined in
``tests_reg/test/test/framework/test_framework.h``.
.. code-block:: c
struct test_t {
TEST_FUN * const test; /*!< Test function to call */
const char *name; /*!< Test name */
const char *desc; /*!< Test description */
};
For example, a new test case called ``TFM_NS_<TEST_NAME>_TEST_1001`` is created
and the function ``tfm_<test_name>_test_1001`` needs to be defined in file
``<test_name>_ns_interface_testsuite.c``. Then the function shall be appended
into the array which will be quoted in function
``register_testsuite_ns_<test_name>_interface``. See the reference code below.
.. code-block:: c
/* List of test cases */
static void tfm_<test_name>_test_1001(struct test_result_t *ret);
/* Append test cases */
static struct test_t <test_name>_tests[] = {
{&tfm_<test_name>_test_1001, "TFM_NS_<TEST_NAME>_TEST_1001",
"Example test case"},
};
/* Register test case into test suites */
void register_testsuite_ns_<test_name>_interface(struct test_suite_t *p_test_suite)
{
uint32_t list_size;
list_size = (sizeof(<test_name>_tests) / sizeof(<test_name>_tests[0]));
set_testsuite("<TEST_NAME> non-secure interface test (TFM_NS_<TEST_NAME>_TEST_1XXX)",
<test_name>_tests, list_size, p_test_suite);
}
static void tfm_<test_name>_test_1001(struct test_result_t *ret)
{
/* test case code */
}
********************
Adding test services
********************
Some test group may need specific test services. These test services may support
one or more groups thus developers shall determine the proper test scope. Refer
to
:doc:`Adding partitions for regression tests <tfm_test_partitions_addition>`
to get more information.
**********************************
Out-of-tree regression test suites
**********************************
TF-M supports out-of-tree regression test suites build, whose source code
folder is outside tf-m-tests repo. There are two configurations for developers
to include the source code.
- ``EXTRA_NS_TEST_SUITE_PATH``
An absolute directory of the out-of-tree non-secure test suite
source code folder. TF-M build system searches ``CMakeLists.txt`` of
non-secure test suite in the source code folder.
- ``EXTRA_S_TEST_SUITE_PATH``
An absolute directory of the out-of-tree secure test suite
source code folder.
Example usage
=============
Take non-secure test as an example in
`tf-m-extras <https://git.trustedfirmware.org/TF-M/tf-m-extras.git/>`_.
A single out-of-tree test suite folder can be organized as the figure below:
.. code-block:: bash
extra_ns
├── CMakeLists.txt
├── ns_test.c
└── ns_test_config.cmake
In the example above, ``EXTRA_NS_TEST_SUITE_PATH`` in the build command can be
specified as listed below.
.. code-block:: bash
-DEXTRA_NS_TEST_SUITE_PATH=<Absolute-path-extra-test-folder>
Coding instructions
===================
This is a demo of source code so the structure has been simplified. Files like
``.c`` and ``.h`` can be expanded to ``src`` and ``include`` folders
respectively. The ``CMakeLists.txt`` is required in the root path and
``ns_test_config.cmake`` is optional.
Header Files
------------
The header file ``extra_ns_tests.h`` must be included by out-of-tree source
code. This file contains the declaration of
``void register_testsuite_extra_ns_interface(struct test_suite_t *p_test_suite)``.
Source code
-----------
To connect the out-of-tree source code and regression test framework, the test case
function/functions must be defined first. An example format is:
.. code-block:: c
void ns_test(struct test_result_t *ret)
{
/* Add platform specific non-secure test suites code here. */
ret->val = TEST_PASSED;
}
This function follows the standard TF-M test case function prototype.
.. note::
Extra tests can have one or more test cases. This is simplified example so
only one test case is added.
After ``ns_test()`` is defined, a structure variable need to be created like:
.. code-block:: c
static struct test_t plat_ns_t[] = {
{&ns_test, "TFM_EXTRA_TEST_1001",
"Extra Non-Secure test"},
};
It will be used by function ``register_testsuite_extra_ns_interface()`` to
register the test by calling the ``set_testsuite()`` function:
.. code-block:: c
void register_testsuite_extra_ns_interface(struct test_suite_t *p_test_suite)
{
uint32_t list_size;
list_size = (sizeof(plat_ns_t) /
sizeof(plat_ns_t[0]));
set_testsuite("Extra Non-Secure interface tests"
"(TFM_NS_EXTRA_TEST_1XXX)",
plat_ns_t, list_size, p_test_suite);
}
The platform initialization code can be added in this function because it runs
before ``ns_test()``.
.. Note::
Function ``register_testsuite_extra_ns_interface()`` is declared in
tf-m-tests repository without definition. It is supplied to out-of-tree
source code and need to be defined with no change of its format, like
returns error code and parameter name.
CMakeLists.txt
--------------
After extra test suite file were created they must be linked to
``tfm_test_suite_extra_ns`` CMAKE target:
.. code-block:: cmake
target_sources(tfm_test_suite_extra_ns
PRIVATE
${CMAKE_CURRENT_SOURCE_DIR}/ns_test.c
)
ns_test_config.cmake
--------------------
The CMAKE configuration file is optional. If out-of-tree source already exists
another configuration file, a new one can be ignored.
--------------
*Copyright (c) 2021-2022, Arm Limited. All rights reserved.*
*Copyright (c) 2022 Cypress Semiconductor Corporation (an Infineon company)
or an affiliate of Cypress Semiconductor Corporation. All rights reserved.*
|