Mbed TLS driver interface test strategy

This document describes the test strategy for the driver interfaces in Mbed TLS. Mbed TLS has interfaces for secure element drivers, accelerator drivers and entropy drivers. This document is about testing Mbed TLS itself; testing drivers is out of scope.

The driver interfaces are standardized through PSA Cryptography functional specifications.

Secure element driver interface testing

Secure element driver interfaces

Opaque driver interface

The unified driver interface supports both transparent drivers (for accelerators) and opaque drivers (for secure elements).

Drivers exposing this interface need to be registered at compile time by declaring their JSON description file.

Dynamic secure element driver interface

The dynamic secure element driver interface (SE interface for short) is defined by psa/crypto_se_driver.h. This is an interface between Mbed TLS and one or more third-party drivers.

The SE interface consists of one function provided by Mbed TLS (psa_register_se_driver) and many functions that drivers must implement. To make a driver usable by Mbed TLS, the initialization code must call psa_register_se_driver with a structure that describes the driver. The structure mostly contains function pointers, pointing to the driver's methods. All calls to a driver function are triggered by a call to a PSA crypto API function.

SE driver interface unit tests

This section describes unit tests that must be implemented to validate the secure element driver interface. Note that a test case may cover multiple requirements; for example a “good case” test can validate that the proper function is called, that it receives the expected inputs and that it produces the expected outputs.

Many SE driver interface unit tests could be covered by running the existing API tests with a key in a secure element.

SE driver registration

This applies to dynamic drivers only.

  • Test psa_register_se_driver with valid and with invalid arguments.
  • Make at least one failing call to psa_register_se_driver followed by a successful call.
  • Make at least one test that successfully registers the maximum number of drivers and fails to register one more.

Dispatch to SE driver

For each API function that can lead to a driver call (more precisely, for each driver method call site, but this is practically equivalent):

  • Make at least one test with a key in a secure element that checks that the driver method is called. A few API functions involve multiple driver methods; these should validate that all the expected driver methods are called.
  • Make at least one test with a key that is not in a secure element that checks that the driver method is not called.
  • Make at least one test with a key in a secure element with a driver that does not have the requisite method (i.e. the method pointer is NULL) but has the substructure containing that method, and check that the return value is PSA_ERROR_NOT_SUPPORTED.
  • Make at least one test with a key in a secure element with a driver that does not have the substructure containing that method (i.e. the pointer to the substructure is NULL), and check that the return value is PSA_ERROR_NOT_SUPPORTED.
  • At least one test should register multiple drivers with a key in each driver and check that the expected driver is called. This does not need to be done for all operations (use a white-box approach to determine if operations may use different code paths to choose the driver).
  • At least one test should register the same driver structure with multiple lifetime values and check that the driver receives the expected lifetime value.

Some methods only make sense as a group (for example a driver that provides the MAC methods must provide all or none). In those cases, test with all of them null and none of them null.

SE driver inputs

For each API function that can lead to a driver call (more precisely, for each driver method call site, but this is practically equivalent):

  • Wherever the specification guarantees parameters that satisfy certain preconditions, check these preconditions whenever practical.
  • If the API function can take parameters that are invalid and must not reach the driver, call the API function with such parameters and verify that the driver method is not called.
  • Check that the expected inputs reach the driver. This may be implicit in a test that checks the outputs if the only realistic way to obtain the correct outputs is to start from the expected inputs (as is often the case for cryptographic material, but not for metadata).

SE driver outputs

For each API function that leads to a driver call, call it with parameters that cause a driver to be invoked and check how Mbed TLS handles the outputs.

  • Correct outputs.
  • Incorrect outputs such as an invalid output length.
  • Expected errors (e.g. PSA_ERROR_INVALID_SIGNATURE from a signature verification method).
  • Unexpected errors. At least test that if the driver returns PSA_ERROR_GENERIC_ERROR, this is propagated correctly.

Key creation functions invoke multiple methods and need more complex error handling:

  • Check the consequence of errors detected at each stage (slot number allocation or validation, key creation method, storage accesses).
  • Check that the storage ends up in the expected state. At least make sure that no intermediate file remains after a failure.

Persistence of SE keys

The following tests must be performed at least one for each key creation method (import, generate, ...).

  • Test that keys in a secure element survive psa_close_key(); psa_open_key().
  • Test that keys in a secure element survive mbedtls_psa_crypto_free(); psa_crypto_init().
  • Test that the driver's persistent data survives mbedtls_psa_crypto_free(); psa_crypto_init().
  • Test that psa_destroy_key() does not leave any trace of the key.

Resilience for SE drivers

Creating or removing a key in a secure element involves multiple storage modifications (M1, ..., Mn). If the operation is interrupted by a reset at any point, it must be either rolled back or completed.

  • For each potential interruption point (before M1, between M1 and M2, ..., after Mn), call mbedtls_psa_crypto_free(); psa_crypto_init() at that point and check that this either rolls back or completes the operation that was started.
  • This must be done for each key creation method and for key destruction.
  • This must be done for each possible flow, including error cases (e.g. a key creation that fails midway due to OUT_OF_MEMORY).
  • The recovery during psa_crypto_init can itself be interrupted. Test those interruptions too.
  • Two things need to be tested: the key that is being created or destroyed, and the driver's persistent storage.
  • Check both that the storage has the expected content (this can be done by e.g. using a key that is supposed to be present) and does not have any unexpected content (for keys, this can be done by checking that psa_open_key fails with PSA_ERROR_DOES_NOT_EXIST).

This requires instrumenting the storage implementation, either to force it to fail at each point or to record successive storage states and replay each of them. Each psa_its_xxx function call is assumed to be atomic.

SE driver system tests

Real-world use case

We must have at least one driver that is close to real-world conditions:

  • With its own source tree.
  • Running on actual hardware.
  • Run the full driver validation test suite (which does not yet exist).
  • Run at least one test application (e.g. the Mbed OS TLS example).

This requirement shall be fulfilled by the Microchip ATECC508A driver.

Complete driver

We should have at least one driver that covers the whole interface:

  • With its own source tree.
  • Implementing all the methods.
  • Run the full driver validation test suite (which does not yet exist).

A PKCS#11 driver would be a good candidate. It would be useful as part of our product offering.

Unified driver interface testing

The unified driver interface defines interfaces for accelerators.

Test requirements

Requirements for transparent driver testing

Every cryptographic mechanism for which a transparent driver interface exists (key creation, cryptographic operations, …) must be exercised in at least one build. The test must verify that the driver code is called.

Requirements for fallback

The driver interface includes a fallback mechanism so that a driver can reject a request at runtime and let another driver handle the request. For each entry point, there must be at least three test runs with two or more drivers available with driver A configured to fall back to driver B, with one run where A returns PSA_SUCCESS, one where A returns PSA_ERROR_NOT_SUPPORTED and B is invoked, and one where A returns a different error and B is not invoked.

Test drivers

We have test drivers that are enabled by PSA_CRYPTO_DRIVER_TEST (not present in the usual config files, must be defined on the command line or in a custom config file). Those test drivers are implemented in tests/src/drivers/*.c and their API is declared in tests/include/test/drivers/*.h.

We have two test driver registered: mbedtls_test_opaque_driver and mbedtls_test_transparent_driver. These are described in scripts/data_files/driver_jsons/mbedtls_test_xxx_driver.json (as much as our JSON support currently allows). Each of the drivers can potentially implement support for several mechanism; conversely, each of the file mentioned in the previous paragraph can potentially contribute to both the opaque and the transparent test driver.

Each entry point is instrumented to record the number of hits for each part of the driver (same division as the files) and the status of the last call. It is also possible to force the next call to return a specified status, and sometimes more things can be forced: see the various mbedtls_test_driver_XXX_hooks_t structures declared by each driver.

The drivers can use one of two back-ends:

  • internal: this requires the built-in implementation to be present.
  • libtestdriver1: this allows the built-in implementation to be omitted from the build.

Historical note: internal was initially the only back-end; then support for libtestdriver1 was added gradually.

Question: if/when we have complete libtestdriver1 support, do we still need internal? Thoughts:

  • It's useful to have builds with both a driver and the built-in, in order to test fallback to built-in, but this could be achieved with libtestdriver1 too.
    • Performance might be better with internal though?
  • The instrumentation works the same with both back-ends.

Our implementation of PSA Crypto is structured in a way that the built-in implementation of each operation follows the driver API, see ../architecture/psa-crypto-implementation-structure.md. This makes implementing the test drivers very easy: each entry point has a corresponding mbedtls_psa_xxx() function that it can call as its implementation - with the libtestdriver1 back-end the function is called libtestdriver1_mbedtls_psa_xxx() instead.

A nice consequence of that strategy is that when an entry point has test-driver support, most of the time, it automatically works for all algorithms and key types supported by the library. (The exception being when the driver needs to call a different function for different key types, as is the case with some asymmetric key management operations.)

The renaming process for libtestdriver1 is implemented as a few Perl regexes applied to a copy of the library code, see the libtestdriver1.a target in tests/Makefile. Another modification that's done to this copy is appending tests/include/test/drivers/crypto_config_test_driver_extension.h to psa/crypto_config.h. This file reverses the ACCEL/BUILTIN macros so that libtestdriver1 includes as built-in what the main libmbedcrypto.a will have accelerated; see that file's initial comment for details. See also helper_libtestdriver1_ functions and the preceding comment in all.sh for how libtestdriver is used in practice.

This general framework needs specific code for each family of operations. At a given point in time, not all operations have the same level of support. The following sub-sections describe the status of the test driver support, mostly following the structure and order of sections 9.6 and 10.2 to 10.10 of the PSA Crypto standard as that is also a natural division for implementing test drivers (that's how the code is divided into files).

Key management

The following entry points are declared in test/drivers/key_management.h:

  • "init" (transparent and opaque)
  • "generate_key" (transparent and opaque)
  • "export_public_key" (transparent and opaque)
  • "import_key" (transparent and opaque)
  • "export_key" (opaque only)
  • "get_builtin_key" (opaque only)
  • "copy_key" (opaque only)

The transparent driver fully implements the declared entry points, and can use any backend: internal or libtestdriver1.

The opaque's driver implementation status is as follows:

  • "generate_key": not implemented, always returns NOT_SUPPORTED.
  • "export_public_key": implemented only for ECC and RSA keys, both backends.
  • "import_key": implemented except for DH keys, both backends.
  • "export_key": implemented for built-in keys (ECC and AES), and for non-builtin keys except DH keys. (Backend not relevant.)
  • "get_builtin_key": implemented - provisioned keys: AES-128 and ECC secp2456r1. (Backend not relevant.)
  • "copy_key": implemented - emulates a SE without storage. (Backend not relevant.)

Note: the "init" entry point is not part of the "key management" family, but listed here as it's declared and implemented in the same file. With the transparent driver and the libtestdriver1 backend, it calls libtestdriver1_psa_crypto_init(), which partially but not fully ensures that this entry point is called before other entry points in the test drivers. With the opaque driver, this entry point just does nothing an returns success.

The following entry points are defined by the driver interface but missing from our test drivers:

  • "allocate_key", "destroy_key": this is for opaque drivers that store the key material internally.

Note: the instrumentation also allows forcing the output and its length.

Message digests (Hashes)

The following entry points are declared (transparent only):

  • "hash_compute"
  • "hash_setup"
  • "hash_clone"
  • "hash_update"
  • "hash_finish"
  • "hash_abort"

The transparent driver fully implements the declared entry points, and can use any backend: internal or libtestdriver1.

This familly is not part of the opaque driver as it doesn't use keys.

Message authentication codes (MAC)

The following entry points are declared (transparent and opaque):

  • "mac_compute"
  • "mac_sign_setup"
  • "mac_verify_setup"
  • "mac_update"
  • "mac_sign_finish"
  • "mac_verify_finish"
  • "mac_abort"

The transparent driver fully implements the declared entry points, and can use any backend: internal or libtestdriver1.

The opaque driver only implements the instrumentation but not the actual operations: entry points will always return NOT_SUPPORTED, unless another status is forced.

The following entry points are not implemented:

  • mac_verify: this mostly makes sense for opaque drivers; the code will fall back to using "mac_compute" if this is not implemented. So, perhaps ideally we should test both with "mac_verify" implemented and with it not implemented? Anyway, we have a test gap here.

Unauthenticated ciphers

The following entry points are declared (transparent and opaque):

  • "cipher_encrypt"
  • "cipher_decrypt"
  • "cipher_encrypt_setup"
  • "cipher_decrypt_setup"
  • "cipher_set_iv"
  • "cipher_update"
  • "cipher_finish"
  • "cipher_abort"

The transparent driver fully implements the declared entry points, and can use any backend: internal or libtestdriver1.

The opaque driver is not implemented at all, neither instumentation nor the operation: entry points always return NOT_SUPPORTED.

Note: the instrumentation also allows forcing a specific output and output length.

Authenticated encryption with associated data (AEAD)

The following entry points are declared (transparent only):

  • "aead_encrypt"
  • "aead_decrypt"
  • "aead_encrypt_setup"
  • "aead_decrypt_setup"
  • "aead_set_nonce"
  • "aead_set_lengths"
  • "aead_update_ad"
  • "aead_update"
  • "aead_finish"
  • "aead_verify"
  • "aead_abort"

The transparent driver fully implements the declared entry points, and can use any backend: internal or libtestdriver1.

The opaque driver does not implement or even declare entry points for this family.

Note: the instrumentation records the number of hits per entry point, not just the total number of hits for this family.

Key derivation

Not covered at all by the test drivers.

That's a gap in our testing, as the driver interface does define a key derivation family of entry points. This gap is probably related to the fact that our internal code structure doesn't obey the guidelines and is not aligned with the driver interface, see #5488 and related issues.

Asymmetric signature

The following entry points are declared (transparent and opaque):

  • "sign_message"
  • "verify_message"
  • "sign_hash"
  • "verify_hash"

The transparent driver fully implements the declared entry points, and can use any backend: internal or libtestdriver1.

The opaque driver is not implemented at all, neither instumentation nor the operation: entry points always return NOT_SUPPORTED.

Note: the instrumentation also allows forcing a specific output and output length, and has two instance of the hooks structure: one for sign, the other for verify.

Note: when a driver implements only the "xxx_hash" entry points, the core is supposed to implement the psa_xxx_message() functions by computing the hash itself before calling the "xxx_hash" entry point. Since the test driver does implement the "xxx_message" entry point, it's not exercising that part of the core's expected behaviour.

Asymmetric encryption

The following entry points are declared (transparent and opaque):

  • "asymmetric_encrypt"
  • "asymmetric_decrypt"

The transparent driver fully implements the declared entry points, and can use any backend: internal or libtestdriver1.

The opaque driver is not implemented at all, neither instumentation nor the operation: entry points always return NOT_SUPPORTED.

Note: the instrumentation also allows forcing a specific output and output length.

Key agreement

The following entry points are declared (transparent and opaque):

  • "key_agreement"

The transparent driver fully implements the declared entry points, and can use any backend: internal or libtestdriver1.

The opaque driver is not implemented at all, neither instumentation nor the operation: entry points always return NOT_SUPPORTED.

Note: the instrumentation also allows forcing a specific output and output length.

Other cryptographic services (Random number generation)

Not covered at all by the test drivers.

The driver interface defines a "get_entropy" entry point, as well as a "Random generation" family of entry points. None of those are currently implemented in the library. Part of it will be planned for 4.0, see #8150.

PAKE extension

The following entry points are declared (transparent only):

  • "pake_setup"
  • "pake_output"
  • "pake_input"
  • "pake_get_implicit_key"
  • "pake_abort"

Note: the instrumentation records hits per entry point and allows forcing the output and its length, as well as forcing the status of setup independently from the others.

The transparent driver fully implements the declared entry points, and can use any backend: internal or libtestdriver1.

The opaque driver does not implement or even declare entry points for this family.