RSA: Require hashlen to match md_alg when applicable

Signed-off-by: Manuel Pégourié-Gonnard <manuel.pegourie-gonnard@arm.com>
diff --git a/ChangeLog.d/require-matching-hashlen-rsa.txt b/ChangeLog.d/require-matching-hashlen-rsa.txt
new file mode 100644
index 0000000..d1420d1
--- /dev/null
+++ b/ChangeLog.d/require-matching-hashlen-rsa.txt
@@ -0,0 +1,4 @@
+API changes
+   * Functions in the RSA module that accept a hashlen parameter now require
+     it to match the output size of the hash algorithm used, except when
+     signing raw data.
diff --git a/docs/3.0-migration-guide.d/require-matching-hashlen-rsa.md b/docs/3.0-migration-guide.d/require-matching-hashlen-rsa.md
new file mode 100644
index 0000000..75fadd8
--- /dev/null
+++ b/docs/3.0-migration-guide.d/require-matching-hashlen-rsa.md
@@ -0,0 +1,13 @@
+RSA signature functions now require `hashlen` to match the expected value
+-------------------------------------------------------------------------
+
+This only affects users of the low-level RSA API; users of the high-level PK
+API or of the PSA Crypto API are not affected.
+
+All the functions in the RSA module that accept a `hashlen` parameter used to
+ignore it unless the `md_alg` parameter was `MBEDTLS_MD_NONE`, indicating raw
+data was signed. They now require this parameter's value to be equal to the
+output size of the hash algorithm used when signing a hash. (The requirements
+when signing raw data are unchanged.)
+
+The migration path is to pass the correct value to those functions.
diff --git a/include/mbedtls/rsa.h b/include/mbedtls/rsa.h
index c953a89..9a070e7 100644
--- a/include/mbedtls/rsa.h
+++ b/include/mbedtls/rsa.h
@@ -796,13 +796,11 @@
  *                 if \p f_rng doesn't need a context argument.
  * \param md_alg   The message-digest algorithm used to hash the original data.
  *                 Use #MBEDTLS_MD_NONE for signing raw data.
- * \param hashlen  The length of the message digest.
- *                 Ths is only used if \p md_alg is #MBEDTLS_MD_NONE.
+ * \param hashlen  The length of the message digest or raw data in Bytes.
+ *                 If \p md_alg is not #MBEDTLS_MD_NONE, this must match the
+ *                 output length of the corresponding hash algorithm.
  * \param hash     The buffer holding the message digest or raw data.
- *                 If \p md_alg is #MBEDTLS_MD_NONE, this must be a readable
- *                 buffer of length \p hashlen Bytes. If \p md_alg is not
- *                 #MBEDTLS_MD_NONE, it must be a readable buffer of length
- *                 the size of the hash corresponding to \p md_alg.
+ *                 This must be a readable buffer of at least \p hashlen Bytes.
  * \param sig      The buffer to hold the signature. This must be a writable
  *                 buffer of length \c ctx->len Bytes. For example, \c 256 Bytes
  *                 for an 2048-bit RSA modulus. A buffer length of
@@ -830,13 +828,11 @@
  *                 if \p f_rng doesn't need a context argument.
  * \param md_alg   The message-digest algorithm used to hash the original data.
  *                 Use #MBEDTLS_MD_NONE for signing raw data.
- * \param hashlen  The length of the message digest.
- *                 Ths is only used if \p md_alg is #MBEDTLS_MD_NONE.
+ * \param hashlen  The length of the message digest or raw data in Bytes.
+ *                 If \p md_alg is not #MBEDTLS_MD_NONE, this must match the
+ *                 output length of the corresponding hash algorithm.
  * \param hash     The buffer holding the message digest or raw data.
- *                 If \p md_alg is #MBEDTLS_MD_NONE, this must be a readable
- *                 buffer of length \p hashlen Bytes. If \p md_alg is not
- *                 #MBEDTLS_MD_NONE, it must be a readable buffer of length
- *                 the size of the hash corresponding to \p md_alg.
+ *                 This must be a readable buffer of at least \p hashlen Bytes.
  * \param sig      The buffer to hold the signature. This must be a writable
  *                 buffer of length \c ctx->len Bytes. For example, \c 256 Bytes
  *                 for an 2048-bit RSA modulus. A buffer length of
@@ -878,13 +874,11 @@
  *                 if \p f_rng doesn't need a context argument.
  * \param md_alg   The message-digest algorithm used to hash the original data.
  *                 Use #MBEDTLS_MD_NONE for signing raw data.
- * \param hashlen  The length of the message digest.
- *                 Ths is only used if \p md_alg is #MBEDTLS_MD_NONE.
+ * \param hashlen  The length of the message digest or raw data in Bytes.
+ *                 If \p md_alg is not #MBEDTLS_MD_NONE, this must match the
+ *                 output length of the corresponding hash algorithm.
  * \param hash     The buffer holding the message digest or raw data.
- *                 If \p md_alg is #MBEDTLS_MD_NONE, this must be a readable
- *                 buffer of length \p hashlen Bytes. If \p md_alg is not
- *                 #MBEDTLS_MD_NONE, it must be a readable buffer of length
- *                 the size of the hash corresponding to \p md_alg.
+ *                 This must be a readable buffer of at least \p hashlen Bytes.
  * \param saltlen  The length of the salt that should be used.
  *                 If passed #MBEDTLS_RSA_SALT_LEN_ANY, the function will use
  *                 the largest possible salt length up to the hash length,
@@ -935,13 +929,11 @@
  *                 if \p f_rng doesn't need a context argument.
  * \param md_alg   The message-digest algorithm used to hash the original data.
  *                 Use #MBEDTLS_MD_NONE for signing raw data.
- * \param hashlen  The length of the message digest.
- *                 This is only used if \p md_alg is #MBEDTLS_MD_NONE.
+ * \param hashlen  The length of the message digest or raw data in Bytes.
+ *                 If \p md_alg is not #MBEDTLS_MD_NONE, this must match the
+ *                 output length of the corresponding hash algorithm.
  * \param hash     The buffer holding the message digest or raw data.
- *                 If \p md_alg is #MBEDTLS_MD_NONE, this must be a readable
- *                 buffer of length \p hashlen Bytes. If \p md_alg is not
- *                 #MBEDTLS_MD_NONE, it must be a readable buffer of length
- *                 the size of the hash corresponding to \p md_alg.
+ *                 This must be a readable buffer of at least \p hashlen Bytes.
  * \param sig      The buffer to hold the signature. This must be a writable
  *                 buffer of length \c ctx->len Bytes. For example, \c 256 Bytes
  *                 for an 2048-bit RSA modulus. A buffer length of
@@ -972,13 +964,11 @@
  * \param ctx      The initialized RSA public key context to use.
  * \param md_alg   The message-digest algorithm used to hash the original data.
  *                 Use #MBEDTLS_MD_NONE for signing raw data.
- * \param hashlen  The length of the message digest.
- *                 This is only used if \p md_alg is #MBEDTLS_MD_NONE.
+ * \param hashlen  The length of the message digest or raw data in Bytes.
+ *                 If \p md_alg is not #MBEDTLS_MD_NONE, this must match the
+ *                 output length of the corresponding hash algorithm.
  * \param hash     The buffer holding the message digest or raw data.
- *                 If \p md_alg is #MBEDTLS_MD_NONE, this must be a readable
- *                 buffer of length \p hashlen Bytes. If \p md_alg is not
- *                 #MBEDTLS_MD_NONE, it must be a readable buffer of length
- *                 the size of the hash corresponding to \p md_alg.
+ *                 This must be a readable buffer of at least \p hashlen Bytes.
  * \param sig      The buffer holding the signature. This must be a readable
  *                 buffer of length \c ctx->len Bytes. For example, \c 256 Bytes
  *                 for an 2048-bit RSA modulus.
@@ -999,13 +989,11 @@
  * \param ctx      The initialized RSA public key context to use.
  * \param md_alg   The message-digest algorithm used to hash the original data.
  *                 Use #MBEDTLS_MD_NONE for signing raw data.
- * \param hashlen  The length of the message digest.
- *                 This is only used if \p md_alg is #MBEDTLS_MD_NONE.
+ * \param hashlen  The length of the message digest or raw data in Bytes.
+ *                 If \p md_alg is not #MBEDTLS_MD_NONE, this must match the
+ *                 output length of the corresponding hash algorithm.
  * \param hash     The buffer holding the message digest or raw data.
- *                 If \p md_alg is #MBEDTLS_MD_NONE, this must be a readable
- *                 buffer of length \p hashlen Bytes. If \p md_alg is not
- *                 #MBEDTLS_MD_NONE, it must be a readable buffer of length
- *                 the size of the hash corresponding to \p md_alg.
+ *                 This must be a readable buffer of at least \p hashlen Bytes.
  * \param sig      The buffer holding the signature. This must be a readable
  *                 buffer of length \c ctx->len Bytes. For example, \c 256 Bytes
  *                 for an 2048-bit RSA modulus.
@@ -1036,13 +1024,11 @@
  * \param ctx      The initialized RSA public key context to use.
  * \param md_alg   The message-digest algorithm used to hash the original data.
  *                 Use #MBEDTLS_MD_NONE for signing raw data.
- * \param hashlen  The length of the message digest.
- *                 This is only used if \p md_alg is #MBEDTLS_MD_NONE.
+ * \param hashlen  The length of the message digest or raw data in Bytes.
+ *                 If \p md_alg is not #MBEDTLS_MD_NONE, this must match the
+ *                 output length of the corresponding hash algorithm.
  * \param hash     The buffer holding the message digest or raw data.
- *                 If \p md_alg is #MBEDTLS_MD_NONE, this must be a readable
- *                 buffer of length \p hashlen Bytes. If \p md_alg is not
- *                 #MBEDTLS_MD_NONE, it must be a readable buffer of length
- *                 the size of the hash corresponding to \p md_alg.
+ *                 This must be a readable buffer of at least \p hashlen Bytes.
  * \param sig      The buffer holding the signature. This must be a readable
  *                 buffer of length \c ctx->len Bytes. For example, \c 256 Bytes
  *                 for an 2048-bit RSA modulus.
@@ -1069,13 +1055,11 @@
  * \param ctx      The initialized RSA public key context to use.
  * \param md_alg   The message-digest algorithm used to hash the original data.
  *                 Use #MBEDTLS_MD_NONE for signing raw data.
- * \param hashlen  The length of the message digest.
- *                 This is only used if \p md_alg is #MBEDTLS_MD_NONE.
+ * \param hashlen  The length of the message digest or raw data in Bytes.
+ *                 If \p md_alg is not #MBEDTLS_MD_NONE, this must match the
+ *                 output length of the corresponding hash algorithm.
  * \param hash     The buffer holding the message digest or raw data.
- *                 If \p md_alg is #MBEDTLS_MD_NONE, this must be a readable
- *                 buffer of length \p hashlen Bytes. If \p md_alg is not
- *                 #MBEDTLS_MD_NONE, it must be a readable buffer of length
- *                 the size of the hash corresponding to \p md_alg.
+ *                 This must be a readable buffer of at least \p hashlen Bytes.
  * \param mgf1_hash_id      The message digest algorithm used for the
  *                          verification operation and the mask generation
  *                          function (MGF1). For more details on the encoding