blob: 5fc5e18e6d15a8d83ccedfcb9063e8ed445986b0 [file] [log] [blame] [view]
Fredrik Hessecc207bc2021-09-28 21:06:08 +02001# Mbed TLS driver interface test strategy
Gilles Peskineb26c8d82019-09-04 19:26:17 +02002
Fredrik Hessecc207bc2021-09-28 21:06:08 +02003This document describes the test strategy for the driver interfaces in Mbed TLS. Mbed TLS has interfaces for secure element drivers, accelerator drivers and entropy drivers. This document is about testing Mbed TLS itself; testing drivers is out of scope.
Gilles Peskineb26c8d82019-09-04 19:26:17 +02004
5The driver interfaces are standardized through PSA Cryptography functional specifications.
6
Gilles Peskinef0e28532020-11-30 17:51:14 +01007## Secure element driver interface testing
Gilles Peskineb26c8d82019-09-04 19:26:17 +02008
Gilles Peskinef0e28532020-11-30 17:51:14 +01009### Secure element driver interfaces
10
11#### Opaque driver interface
12
13The [unified driver interface](../../proposed/psa-driver-interface.md) supports both transparent drivers (for accelerators) and opaque drivers (for secure elements).
14
15Drivers exposing this interface need to be registered at compile time by declaring their JSON description file.
16
17#### Dynamic secure element driver interface
18
Fredrik Hessecc207bc2021-09-28 21:06:08 +020019The dynamic secure element driver interface (SE interface for short) is defined by [`psa/crypto_se_driver.h`](../../../include/psa/crypto_se_driver.h). This is an interface between Mbed TLS and one or more third-party drivers.
Gilles Peskineb26c8d82019-09-04 19:26:17 +020020
Fredrik Hessecc207bc2021-09-28 21:06:08 +020021The SE interface consists of one function provided by Mbed TLS (`psa_register_se_driver`) and many functions that drivers must implement. To make a driver usable by Mbed TLS, the initialization code must call `psa_register_se_driver` with a structure that describes the driver. The structure mostly contains function pointers, pointing to the driver's methods. All calls to a driver function are triggered by a call to a PSA crypto API function.
Gilles Peskineb26c8d82019-09-04 19:26:17 +020022
Gilles Peskine92bcfdb2019-09-04 19:26:50 +020023### SE driver interface unit tests
24
25This section describes unit tests that must be implemented to validate the secure element driver interface. Note that a test case may cover multiple requirements; for example a “good case” test can validate that the proper function is called, that it receives the expected inputs and that it produces the expected outputs.
26
27Many SE driver interface unit tests could be covered by running the existing API tests with a key in a secure element.
28
29#### SE driver registration
30
Gilles Peskinef0e28532020-11-30 17:51:14 +010031This applies to dynamic drivers only.
32
Gilles Peskine92bcfdb2019-09-04 19:26:50 +020033* Test `psa_register_se_driver` with valid and with invalid arguments.
34* Make at least one failing call to `psa_register_se_driver` followed by a successful call.
35* Make at least one test that successfully registers the maximum number of drivers and fails to register one more.
36
37#### Dispatch to SE driver
38
39For each API function that can lead to a driver call (more precisely, for each driver method call site, but this is practically equivalent):
40
41* Make at least one test with a key in a secure element that checks that the driver method is called. A few API functions involve multiple driver methods; these should validate that all the expected driver methods are called.
42* Make at least one test with a key that is not in a secure element that checks that the driver method is not called.
43* Make at least one test with a key in a secure element with a driver that does not have the requisite method (i.e. the method pointer is `NULL`) but has the substructure containing that method, and check that the return value is `PSA_ERROR_NOT_SUPPORTED`.
44* Make at least one test with a key in a secure element with a driver that does not have the substructure containing that method (i.e. the pointer to the substructure is `NULL`), and check that the return value is `PSA_ERROR_NOT_SUPPORTED`.
45* At least one test should register multiple drivers with a key in each driver and check that the expected driver is called. This does not need to be done for all operations (use a white-box approach to determine if operations may use different code paths to choose the driver).
46* At least one test should register the same driver structure with multiple lifetime values and check that the driver receives the expected lifetime value.
47
48Some methods only make sense as a group (for example a driver that provides the MAC methods must provide all or none). In those cases, test with all of them null and none of them null.
49
50#### SE driver inputs
51
52For each API function that can lead to a driver call (more precisely, for each driver method call site, but this is practically equivalent):
53
54* Wherever the specification guarantees parameters that satisfy certain preconditions, check these preconditions whenever practical.
55* If the API function can take parameters that are invalid and must not reach the driver, call the API function with such parameters and verify that the driver method is not called.
Gilles Peskine8b193c12019-09-05 17:58:13 +020056* Check that the expected inputs reach the driver. This may be implicit in a test that checks the outputs if the only realistic way to obtain the correct outputs is to start from the expected inputs (as is often the case for cryptographic material, but not for metadata).
Gilles Peskine92bcfdb2019-09-04 19:26:50 +020057
58#### SE driver outputs
59
Fredrik Hessecc207bc2021-09-28 21:06:08 +020060For each API function that leads to a driver call, call it with parameters that cause a driver to be invoked and check how Mbed TLS handles the outputs.
Gilles Peskine92bcfdb2019-09-04 19:26:50 +020061
62* Correct outputs.
63* Incorrect outputs such as an invalid output length.
64* Expected errors (e.g. `PSA_ERROR_INVALID_SIGNATURE` from a signature verification method).
65* Unexpected errors. At least test that if the driver returns `PSA_ERROR_GENERIC_ERROR`, this is propagated correctly.
66
67Key creation functions invoke multiple methods and need more complex error handling:
68
69* Check the consequence of errors detected at each stage (slot number allocation or validation, key creation method, storage accesses).
70* Check that the storage ends up in the expected state. At least make sure that no intermediate file remains after a failure.
71
72#### Persistence of SE keys
73
74The following tests must be performed at least one for each key creation method (import, generate, ...).
75
76* Test that keys in a secure element survive `psa_close_key(); psa_open_key()`.
77* Test that keys in a secure element survive `mbedtls_psa_crypto_free(); psa_crypto_init()`.
78* Test that the driver's persistent data survives `mbedtls_psa_crypto_free(); psa_crypto_init()`.
79* Test that `psa_destroy_key()` does not leave any trace of the key.
80
81#### Resilience for SE drivers
82
83Creating or removing a key in a secure element involves multiple storage modifications (M<sub>1</sub>, ..., M<sub>n</sub>). If the operation is interrupted by a reset at any point, it must be either rolled back or completed.
84
85* For each potential interruption point (before M<sub>1</sub>, between M<sub>1</sub> and M<sub>2</sub>, ..., after M<sub>n</sub>), call `mbedtls_psa_crypto_free(); psa_crypto_init()` at that point and check that this either rolls back or completes the operation that was started.
86* This must be done for each key creation method and for key destruction.
87* This must be done for each possible flow, including error cases (e.g. a key creation that fails midway due to `OUT_OF_MEMORY`).
88* The recovery during `psa_crypto_init` can itself be interrupted. Test those interruptions too.
89* Two things need to be tested: the key that is being created or destroyed, and the driver's persistent storage.
bootstrap-prime6dbbf442022-05-17 19:30:44 -040090* Check both that the storage has the expected content (this can be done by e.g. using a key that is supposed to be present) and does not have any unexpected content (for keys, this can be done by checking that `psa_open_key` fails with `PSA_ERROR_DOES_NOT_EXIST`).
Gilles Peskine92bcfdb2019-09-04 19:26:50 +020091
92This requires instrumenting the storage implementation, either to force it to fail at each point or to record successive storage states and replay each of them. Each `psa_its_xxx` function call is assumed to be atomic.
93
94### SE driver system tests
95
96#### Real-world use case
97
98We must have at least one driver that is close to real-world conditions:
99
100* With its own source tree.
101* Running on actual hardware.
102* Run the full driver validation test suite (which does not yet exist).
103* Run at least one test application (e.g. the Mbed OS TLS example).
104
Gilles Peskine545c28b2019-09-04 19:41:16 +0200105This requirement shall be fulfilled by the [Microchip ATECC508A driver](https://github.com/ARMmbed/mbed-os-atecc608a/).
Gilles Peskine92bcfdb2019-09-04 19:26:50 +0200106
107#### Complete driver
108
109We should have at least one driver that covers the whole interface:
110
111* With its own source tree.
112* Implementing all the methods.
113* Run the full driver validation test suite (which does not yet exist).
114
115A PKCS#11 driver would be a good candidate. It would be useful as part of our product offering.
Gilles Peskine24cebf62020-11-30 17:51:53 +0100116
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100117## Unified driver interface testing
Gilles Peskine24cebf62020-11-30 17:51:53 +0100118
119The [unified driver interface](../../proposed/psa-driver-interface.md) defines interfaces for accelerators.
120
121### Test requirements
122
123#### Requirements for transparent driver testing
124
125Every cryptographic mechanism for which a transparent driver interface exists (key creation, cryptographic operations, …) must be exercised in at least one build. The test must verify that the driver code is called.
126
127#### Requirements for fallback
128
129The driver interface includes a fallback mechanism so that a driver can reject a request at runtime and let another driver handle the request. For each entry point, there must be at least three test runs with two or more drivers available with driver A configured to fall back to driver B, with one run where A returns `PSA_SUCCESS`, one where A returns `PSA_ERROR_NOT_SUPPORTED` and B is invoked, and one where A returns a different error and B is not invoked.
130
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100131### Test drivers
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100132
133We have test drivers that are enabled by `PSA_CRYPTO_DRIVER_TEST` (not present
134in the usual config files, must be defined on the command line or in a custom
135config file). Those test drivers are implemented in `tests/src/drivers/*.c`
136and their API is declared in `tests/include/test/drivers/*.h`.
137
138We have two test driver registered: `mbedtls_test_opaque_driver` and
139`mbedtls_test_transparent_driver`. These are described in
140`scripts/data_files/driver_jsons/mbedtls_test_xxx_driver.json` (as much as our
141JSON support currently allows). Each of the drivers can potentially implement
142support for several mechanism; conversely, each of the file mentioned in the
143previous paragraph can potentially contribute to both the opaque and the
144transparent test driver.
145
146Each entry point is instrumented to record the number of hits for each part of
147the driver (same division as the files) and the status of the last call. It is
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100148also possible to force the next call to return a specified status, and
149sometimes more things can be forced: see the various
Manuel Pégourié-Gonnard4575d232024-04-15 10:54:49 +0200150`mbedtls_test_driver_XXX_hooks_t` structures declared by each driver (and
151subsections below).
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100152
153The drivers can use one of two back-ends:
154- internal: this requires the built-in implementation to be present.
155- libtestdriver1: this allows the built-in implementation to be omitted from
156 the build.
157
158Historical note: internal was initially the only back-end; then support for
Manuel Pégourié-Gonnard0ca2fd02024-04-12 10:14:17 +0200159libtestdriver1 was added gradually. Support for libtestdriver1 is now complete
160(see following sub-sections), so we could remove internal now. Note it's
161useful to have builds with both a driver and the built-in, in order to test
162fallback to built-in, which is currently done only with internal, but this can
163be achieved with libtestdriver1 just as well.
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100164
Manuel Pégourié-Gonnard4575d232024-04-15 10:54:49 +0200165Note on instrumentation: originally, when only the internal backend was
166available, hits were how we knew that the driver was called, as opposed to
167directly calling the built-in code. With libtestdriver1, we can check that by
168ensuring that the built-in code is not present, so if the operation gives the
169correct result, only a driver call can have calculated that result. So,
170nowadays there is low value in checking the hit count. There is still some
171value for hit counts, e.g. checking that we don't call a multipart entry point
172when we intended to call the one-shot entry point, but it's limited.
173
Manuel Pégourié-Gonnardb18bc802023-11-24 11:59:25 +0100174Note: our test drivers tend to provide all possible entry points (with a few
175exceptions that may not be intentional, see the next sections). However, in
176some cases, when an entry point is not available, the core is supposed to
177implement it using other entry points, for example:
178- `mac_verify` may use `mac_compute` if the driver does no provide verify;
179- for things that have both one-shot and multi-part API, the driver can
180 provide only the multi-part entry points, and the core is supposed to
181implement one-shot on top of it (but still call the one-shot entry points when
182they're available);
183- `sign/verify_message` can be implemented on top of `sign/verify_hash` for
184 some algorithms;
185- (not sure if the list is exhaustive).
186
187Ideally, we'd want build options for the test drivers so that we can test with
188different combinations of entry points present, and make sure the core behaves
189appropriately when some entry points are absent but other entry points allow
Manuel Pégourié-Gonnardae22f042024-04-12 10:18:27 +0200190implementing the operation. This will remain hard to test until we have proper
191support for JSON-defined drivers with auto-generation of dispatch code.
192(The `MBEDTLS_PSA_ACCEL_xxx` macros we currently use are not expressive enough
Manuel Pégourié-Gonnard4575d232024-04-15 10:54:49 +0200193to specify which entry points are supported for a given mechanism.)
Manuel Pégourié-Gonnardb18bc802023-11-24 11:59:25 +0100194
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100195Our implementation of PSA Crypto is structured in a way that the built-in
196implementation of each operation follows the driver API, see
197[`../architecture/psa-crypto-implementation-structure.md`](../architecture/psa-crypto-implementation-structure.html).
198This makes implementing the test drivers very easy: each entry point has a
199corresponding `mbedtls_psa_xxx()` function that it can call as its
200implementation - with the `libtestdriver1` back-end the function is called
201`libtestdriver1_mbedtls_psa_xxx()` instead.
202
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100203A nice consequence of that strategy is that when an entry point has
204test-driver support, most of the time, it automatically works for all
205algorithms and key types supported by the library. (The exception being when
206the driver needs to call a different function for different key types, as is
Manuel Pégourié-Gonnard432e3b42024-04-12 10:25:25 +0200207the case with some asymmetric key management operations.) (Note: it's still
208useful to test drivers in configurations with partial algorithm support, and
209that can still be done by configuring libtestdriver1 and the main library as
210desired.)
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100211
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100212The renaming process for `libtestdriver1` is implemented as a few Perl regexes
213applied to a copy of the library code, see the `libtestdriver1.a` target in
214`tests/Makefile`. Another modification that's done to this copy is appending
215`tests/include/test/drivers/crypto_config_test_driver_extension.h` to
216`psa/crypto_config.h`. This file reverses the `ACCEL`/`BUILTIN` macros so that
217`libtestdriver1` includes as built-in what the main `libmbedcrypto.a` will
218have accelerated; see that file's initial comment for details. See also
219`helper_libtestdriver1_` functions and the preceding comment in `all.sh` for
220how libtestdriver is used in practice.
221
222This general framework needs specific code for each family of operations. At a
223given point in time, not all operations have the same level of support. The
224following sub-sections describe the status of the test driver support, mostly
225following the structure and order of sections 9.6 and 10.2 to 10.10 of the
226[PSA Crypto standard](https://arm-software.github.io/psa-api/crypto/1.1/) as
227that is also a natural division for implementing test drivers (that's how the
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100228code is divided into files).
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100229
230#### Key management
231
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100232The following entry points are declared in `test/drivers/key_management.h`:
233
234- `"init"` (transparent and opaque)
235- `"generate_key"` (transparent and opaque)
236- `"export_public_key"` (transparent and opaque)
237- `"import_key"` (transparent and opaque)
238- `"export_key"` (opaque only)
239- `"get_builtin_key"` (opaque only)
240- `"copy_key"` (opaque only)
241
242The transparent driver fully implements the declared entry points, and can use
243any backend: internal or libtestdriver1.
244
245The opaque's driver implementation status is as follows:
246- `"generate_key"`: not implemented, always returns `NOT_SUPPORTED`.
247- `"export_public_key"`: implemented only for ECC and RSA keys, both backends.
248- `"import_key"`: implemented except for DH keys, both backends.
249- `"export_key"`: implemented for built-in keys (ECC and AES), and for
250 non-builtin keys except DH keys. (Backend not relevant.)
251- `"get_builtin_key"`: implemented - provisioned keys: AES-128 and ECC
252 secp2456r1. (Backend not relevant.)
253- `"copy_key"`: implemented - emulates a SE without storage. (Backend not
254 relevant.)
255
256Note: the `"init"` entry point is not part of the "key management" family, but
257listed here as it's declared and implemented in the same file. With the
258transparent driver and the libtestdriver1 backend, it calls
259`libtestdriver1_psa_crypto_init()`, which partially but not fully ensures
260that this entry point is called before other entry points in the test drivers.
261With the opaque driver, this entry point just does nothing an returns success.
262
263The following entry points are defined by the driver interface but missing
264from our test drivers:
265- `"allocate_key"`, `"destroy_key"`: this is for opaque drivers that store the
266 key material internally.
267
268Note: the instrumentation also allows forcing the output and its length.
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100269
270#### Message digests (Hashes)
271
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100272The following entry points are declared (transparent only):
273- `"hash_compute"`
274- `"hash_setup"`
275- `"hash_clone"`
276- `"hash_update"`
277- `"hash_finish"`
278- `"hash_abort"`
279
280The transparent driver fully implements the declared entry points, and can use
281any backend: internal or libtestdriver1.
282
283This familly is not part of the opaque driver as it doesn't use keys.
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100284
285#### Message authentication codes (MAC)
286
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100287The following entry points are declared (transparent and opaque):
288- `"mac_compute"`
289- `"mac_sign_setup"`
290- `"mac_verify_setup"`
291- `"mac_update"`
292- `"mac_sign_finish"`
293- `"mac_verify_finish"`
294- `"mac_abort"`
295
296The transparent driver fully implements the declared entry points, and can use
297any backend: internal or libtestdriver1.
298
299The opaque driver only implements the instrumentation but not the actual
300operations: entry points will always return `NOT_SUPPORTED`, unless another
301status is forced.
302
303The following entry points are not implemented:
Manuel Pégourié-Gonnardb18bc802023-11-24 11:59:25 +0100304- `mac_verify`: this mostly makes sense for opaque drivers; the core will fall
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100305 back to using `"mac_compute"` if this is not implemented. So, perhaps
306ideally we should test both with `"mac_verify"` implemented and with it not
307implemented? Anyway, we have a test gap here.
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100308
309#### Unauthenticated ciphers
310
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100311The following entry points are declared (transparent and opaque):
312- `"cipher_encrypt"`
313- `"cipher_decrypt"`
314- `"cipher_encrypt_setup"`
315- `"cipher_decrypt_setup"`
316- `"cipher_set_iv"`
317- `"cipher_update"`
318- `"cipher_finish"`
319- `"cipher_abort"`
320
321The transparent driver fully implements the declared entry points, and can use
322any backend: internal or libtestdriver1.
323
324The opaque driver is not implemented at all, neither instumentation nor the
325operation: entry points always return `NOT_SUPPORTED`.
326
327Note: the instrumentation also allows forcing a specific output and output
328length.
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100329
330#### Authenticated encryption with associated data (AEAD)
331
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100332The following entry points are declared (transparent only):
333- `"aead_encrypt"`
334- `"aead_decrypt"`
335- `"aead_encrypt_setup"`
336- `"aead_decrypt_setup"`
337- `"aead_set_nonce"`
338- `"aead_set_lengths"`
339- `"aead_update_ad"`
340- `"aead_update"`
341- `"aead_finish"`
342- `"aead_verify"`
343- `"aead_abort"`
344
345The transparent driver fully implements the declared entry points, and can use
346any backend: internal or libtestdriver1.
347
348The opaque driver does not implement or even declare entry points for this
349family.
350
351Note: the instrumentation records the number of hits per entry point, not just
352the total number of hits for this family.
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100353
354#### Key derivation
355
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100356Not covered at all by the test drivers.
357
Manuel Pégourié-Gonnarda47a3c4e2024-04-12 10:21:42 +0200358That's a test gap which reflects a feature gap: the driver interface does
359define a key derivation family of entry points, but we don't currently
360implement that part of the driver interface, see #5488 and related issues.
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100361
362#### Asymmetric signature
363
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100364The following entry points are declared (transparent and opaque):
365
366- `"sign_message"`
367- `"verify_message"`
368- `"sign_hash"`
369- `"verify_hash"`
370
371The transparent driver fully implements the declared entry points, and can use
372any backend: internal or libtestdriver1.
373
374The opaque driver is not implemented at all, neither instumentation nor the
375operation: entry points always return `NOT_SUPPORTED`.
376
377Note: the instrumentation also allows forcing a specific output and output
378length, and has two instance of the hooks structure: one for sign, the other
379for verify.
380
381Note: when a driver implements only the `"xxx_hash"` entry points, the core is
382supposed to implement the `psa_xxx_message()` functions by computing the hash
383itself before calling the `"xxx_hash"` entry point. Since the test driver does
384implement the `"xxx_message"` entry point, it's not exercising that part of
385the core's expected behaviour.
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100386
387#### Asymmetric encryption
388
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100389The following entry points are declared (transparent and opaque):
390
391- `"asymmetric_encrypt"`
392- `"asymmetric_decrypt"`
393
394The transparent driver fully implements the declared entry points, and can use
395any backend: internal or libtestdriver1.
396
Manuel Pégourié-Gonnarddde1abd2024-04-09 12:12:48 +0200397The opaque driver implements the declared entry points, and can use any
398backend: internal or libtestdriver1. However it does not implement the
399instrumentation (hits, forced output/status), as this [was not an immediate
400priority](https://github.com/Mbed-TLS/mbedtls/pull/8700#issuecomment-1892466159).
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100401
402Note: the instrumentation also allows forcing a specific output and output
403length.
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100404
405#### Key agreement
406
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100407The following entry points are declared (transparent and opaque):
408
409- `"key_agreement"`
410
411The transparent driver fully implements the declared entry points, and can use
412any backend: internal or libtestdriver1.
413
414The opaque driver is not implemented at all, neither instumentation nor the
415operation: entry points always return `NOT_SUPPORTED`.
416
417Note: the instrumentation also allows forcing a specific output and output
418length.
Manuel Pégourié-Gonnard1a827a32023-11-13 10:01:21 +0100419
420#### Other cryptographic services (Random number generation)
Gilles Peskine24cebf62020-11-30 17:51:53 +0100421
Manuel Pégourié-Gonnardb66f9db2023-11-13 11:32:37 +0100422Not covered at all by the test drivers.
423
424The driver interface defines a `"get_entropy"` entry point, as well as a
425"Random generation" family of entry points. None of those are currently
426implemented in the library. Part of it will be planned for 4.0, see #8150.
427
428#### PAKE extension
429
430The following entry points are declared (transparent only):
431- `"pake_setup"`
432- `"pake_output"`
433- `"pake_input"`
434- `"pake_get_implicit_key"`
435- `"pake_abort"`
436
437Note: the instrumentation records hits per entry point and allows forcing the
438output and its length, as well as forcing the status of setup independently
439from the others.
440
441The transparent driver fully implements the declared entry points, and can use
442any backend: internal or libtestdriver1.
443
444The opaque driver does not implement or even declare entry points for this
445family.
Manuel Pégourié-Gonnard6a96f422023-11-16 13:01:22 +0100446
447### Driver wrapper test suite
448
449We have a test suite dedicated to driver dispatch, which takes advantage of the
450instrumentation in the test drivers described in the previous section, in
451order to check that drivers are called when they're supposed to, and that the
452core behaves as expected when they return errors (in particular, that we fall
Manuel Pégourié-Gonnard432e3b42024-04-12 10:25:25 +0200453back to the built-in implementation when the driver returns `NOT_SUPPORTED`).
Manuel Pégourié-Gonnard6a96f422023-11-16 13:01:22 +0100454
455This is `test_suite_psa_crypto_driver_wrappers`, which is maintained manually
456(that is, the test cases in the `.data` files are not auto-generated). The
457entire test suite depends on the test drivers being enabled
458(`PSA_CRYPTO_DRIVER_TEST`), which is not the case in the default or full
459config.
460
Manuel Pégourié-Gonnard432e3b42024-04-12 10:25:25 +0200461The test suite is focused on driver usage (mostly by checking the expected
462number of hits) but also does some validation of the results: for
463deterministic algorithms, known-answers tests are used, and for the rest, some
464consistency checks are done (more or less detailled depending on the algorithm
465and build configuration).
466
Manuel Pégourié-Gonnard6a96f422023-11-16 13:01:22 +0100467#### Configurations coverage
468
469The driver wrappers test suite has cases that expect both the driver and the
470built-in to be present, and also cases that expect the driver to be present
471but not the built-in. As such, it's impossible for a single configuration to
472run all test cases, and we need at least two: driver+built-in, and
473driver-only.
474
475- The driver+built-in case is covered by `test_psa_crypto_drivers` in `all.sh`.
476This covers all areas (key types and algs) at once.
477- The driver-only case is split into multiple `all.sh` components whose names
478 start with `test_psa_crypto_config_accel`; we have one or more component per
479area, see below.
480
481Here's a summary of driver-only coverage, grouped by families of key types.
482
483Hash (key types: none)
484- `test_psa_crypto_config_accel_hash`: all algs, default config, no parity
485 testing.
486- `test_psa_crypto_config_accel_hash_use_psa`: all algs, full config, with
487 parity testing.
488
489HMAC (key type: HMAC)
Manuel Pégourié-Gonnard6c453612024-03-18 10:12:49 +0100490- `test_psa_crypto_config_accel_hmac`: all algs, full config except a few
491 exclusions (PKCS5, PKCS7, HMAC-DRBG, legacy HKDF, deterministic ECDSA), with
492parity testing.
Manuel Pégourié-Gonnard6a96f422023-11-16 13:01:22 +0100493
494Cipher, AEAD and CMAC (key types: DES, AES, ARIA, CHACHA20, CAMELLIA):
Manuel Pégourié-Gonnard98f8da12024-01-10 12:53:58 +0100495- `test_psa_crypto_config_accel_cipher_aead_cmac`: all key types and algs, full
496 config with a few exclusions (NIST-KW), with parity testing.
497- `test_psa_crypto_config_accel_des`: only DES (with all algs), full
Manuel Pégourié-Gonnard6a96f422023-11-16 13:01:22 +0100498 config, no parity testing.
499- `test_psa_crypto_config_accel_aead`: only AEAD algs (with all relevant key
500 types), full config, no parity testing.
501
502Key derivation (key types: `DERIVE`, `RAW_DATA`, `PASSWORD`, `PEPPER`,
503`PASSWORD_HASH`):
Manuel Pégourié-Gonnarda47a3c4e2024-04-12 10:21:42 +0200504- No testing as we don't have driver support yet (see previous section).
Manuel Pégourié-Gonnard6a96f422023-11-16 13:01:22 +0100505
506RSA (key types: `RSA_KEY_PAIR_xxx`, `RSA_PUBLIC_KEY`):
Manuel Pégourié-Gonnardf2089da2023-12-18 11:36:26 +0100507- `test_psa_crypto_config_accel_rsa_crypto`: all 4 algs (encryption &
508 signature, v1.5 & v2.1), config `crypto_full`, with parity testing excluding
509PK.
Manuel Pégourié-Gonnard6a96f422023-11-16 13:01:22 +0100510
511DH (key types: `DH_KEY_PAIR_xxx`, `DH_PUBLIC_KEY`):
512- `test_psa_crypto_config_accel_ffdh`: all key types and algs, full config,
513 with parity testing.
514- `test_psa_crypto_config_accel_ecc_ffdh_no_bignum`: with also bignum removed.
515
516ECC (key types: `ECC_KEY_PAIR_xxx`, `ECC_PUBLIC_KEY`):
517- Single algorithm accelerated (both key types, all curves):
518 - `test_psa_crypto_config_accel_ecdh`: default config, no parity testing.
519 - `test_psa_crypto_config_accel_ecdsa`: default config, no parity testing.
520 - `test_psa_crypto_config_accel_pake`: full config, no parity testing.
521- All key types, algs and curves accelerated (full config with exceptions,
522 with parity testing):
523 - `test_psa_crypto_config_accel_ecc_ecp_light_only`: `ECP_C` mostly disabled
524 - `test_psa_crypto_config_accel_ecc_no_ecp_at_all`: `ECP_C` fully disabled
525 - `test_psa_crypto_config_accel_ecc_no_bignum`: `BIGNUM_C` disabled (DH disabled)
526 - `test_psa_crypto_config_accel_ecc_ffdh_no_bignum`: `BIGNUM_C` disabled (DH accelerated)
527- Other - all algs accelerated but only some algs/curves (full config with
528 exceptions, no parity testing):
529 - `test_psa_crypto_config_accel_ecc_some_key_types`
530 - `test_psa_crypto_config_accel_ecc_non_weierstrass_curves`
531 - `test_psa_crypto_config_accel_ecc_weierstrass_curves`
532
533Note: `analyze_outcomes.py` provides a list of test cases that are not
Manuel Pégourié-Gonnardf2089da2023-12-18 11:36:26 +0100534executed in any configuration tested on the CI. We're missing driver-only HMAC
535testing, but no test is flagged as never executed there; this reveals we don't
536have "fallback not available" cases for MAC, see #8565.
Manuel Pégourié-Gonnard6a96f422023-11-16 13:01:22 +0100537
538#### Test case coverage
539
540Since `test_suite_psa_crypto_driver_wrappers.data` is maintained manually,
Manuel Pégourié-Gonnard432e3b42024-04-12 10:25:25 +0200541we need to make sure it exercises all the cases that need to be tested. In the
542future, this file should be generated in order to ensure exhaustiveness.
Manuel Pégourié-Gonnard6a96f422023-11-16 13:01:22 +0100543
Manuel Pégourié-Gonnard432e3b42024-04-12 10:25:25 +0200544In the meantime, one way to observe (lack of) completeness is to look at line
545coverage in test driver implementaitons - this doesn't reveal all gaps, but it
546does reveal cases where we thought about something when writing the test
547driver, but not when writing test functions/data.
Manuel Pégourié-Gonnard6a96f422023-11-16 13:01:22 +0100548
549Key management:
Manuel Pégourié-Gonnard6a96f422023-11-16 13:01:22 +0100550- `mbedtls_test_transparent_generate_key()` is not tested with RSA keys.
551- `mbedtls_test_transparent_import_key()` is not tested with DH keys.
552- `mbedtls_test_opaque_import_key()` is not tested with unstructured keys nor
553 with RSA keys (nor DH keys since that's not implemented).
554- `mbedtls_test_opaque_export_key()` is not tested with non-built-in keys.
555- `mbedtls_test_transparent_export_public_key()` is not tested with RSA or DH keys.
556- `mbedtls_test_opaque_export_public_key()` is not tested with non-built-in keys.
557- `mbedtls_test_opaque_copy_key()` is not tested at all.
558
559Hash:
560- `mbedtls_test_transparent_hash_finish()` is not tested with a forced status.
561
562MAC:
563- The following are not tested with a forced status:
564 - `mbedtls_test_transparent_mac_sign_setup()`
565 - `mbedtls_test_transparent_mac_verify_setup()`
566 - `mbedtls_test_transparent_mac_update()`
567 - `mbedtls_test_transparent_mac_verify_finish()`
568 - `mbedtls_test_transparent_mac_abort()`
569- No opaque entry point is tested (they're not implemented either).
570
571Cipher:
572- The following are not tested with a forced status nor with a forced output:
573 - `mbedtls_test_transparent_cipher_encrypt()`
574 - `mbedtls_test_transparent_cipher_finish()`
575- No opaque entry point is tested (they're not implemented either).
576
577AEAD:
578- The following are not tested with a forced status:
579 - `mbedtls_test_transparent_aead_set_nonce()`
580 - `mbedtls_test_transparent_aead_set_lengths()`
581 - `mbedtls_test_transparent_aead_update_ad()`
582 - `mbedtls_test_transparent_aead_update()`
583 - `mbedtls_test_transparent_aead_finish()`
584 - `mbedtls_test_transparent_aead_verify()`
585- `mbedtls_test_transparent_aead_verify()` is not tested with an invalid tag
586 (though it might be in another test suite).
587
588Signature:
589- `sign_hash()` is not tested with RSA-PSS
590- No opaque entry point is tested (they're not implemented either).
591
Manuel Pégourié-Gonnard6a96f422023-11-16 13:01:22 +0100592Key agreement:
593- `mbedtls_test_transparent_key_agreement()` is not tested with FFDH.
594- No opaque entry point is tested (they're not implemented either).
595
596PAKE:
597- All lines are covered.