docs/design_documents/tfm_cooperative_scheduling_rules.rst

############################
Cooperative Scheduling Rules
############################

:Author: Ashutosh Singh
:Organization: Arm Limited
:Contact: Ashutosh Singh <ashutosh.singh@arm.com>
:Status: Accepted

TF-M Scheduler - Rules
======================

On ArmV8-M CPUs, NSPE and SPE share the same physical processing element(PE). A
TF-M enabled system need to be able to handle asynchronous events (interrupts)
regardless of current security state of the PE, and that may lead to scheduling
decisions. This introduces significant complexity into TF-M. To keep the
integrity of (NSPE and SPE) schedulers and call paths between NSPE and SPE,
following set of rules are imposed on the TF-M scheduler design.

Objectives and Requirements
===========================

1. Decoupling of scheduling decisions between NSPE and SPE
2. Efficient interrupt handling by SPE as well as NSPE
3. Reduce critical sections on the secure side to not block interrupts
   unnecessarily
4. Scalability to allow simplification/reduction of overheads to scale down the
   design for constrained devices
5. Reduce impact on NSPE software design

  a. NSPE interrupt handling implementation should be independent
  b. Impact on the NSPE scheduler should be minimal
  c. No assumptions should be made about NSPE's scheduling capabilities

Scheduler Rules for context switching between SPE and NSPE
==========================================================

To allow coherent cooperative scheduling, following set of rules are imposed on
security state changes.
The switching between SPE and NSPE can be triggered in multiple ways.

`Involuntary security state switch`; when the software has no control over the
switch:

- A NSPE interrupt take control into NSPE from SPE
- A SPE interrupt takes control into SPE from NSPE

`Voluntary security state switch`; when software programmatically makes the
switch:

- A NSPE exception handler returns from NSPE to pre-empted SPE context
- A SPE exception handler returns from SPE to pre-empted NSPE context
- NSPE makes a function call into SPE
- SPE returns a call from NSPE
- SPE makes a function call into NSPE (not covered in current design)
- NSPE returns a call from SPE (not covered in current design)

In order to maintain the call stack integrity across NSPE and SPE, following
rules are imposed on all security state switches.

Rules for NSPE Exception handling
---------------------------------

1. **The NSPE exception handler is allowed to trigger a NSPE context switch**
   **(regardless of security state of the preempted context.**

This is expected behaviour of any (RT)OS.

2. **The NSPE scheduler must eventually 'restore' the preempted (by**
   **exception) context.**

This is expected behaviour of any (RT)OS.

3. **If NSPE exception results in a NSPE context switch, SPM must be informed**
   **of the scheduling decision; this must be done BEFORE the execution of**
   **newly scheduled-in context.**

This is to ensures integrity of the call stack when SPE is ready to return a
previous function call from NSPE.

Rules for SPE Exception handling
--------------------------------

1. **All of the SPE interrupts must have higher priority than NSPE interrupts**

This is rule is primarily for simplifying the SPM design.

2. **The SPE interrupt handler is allowed to trigger a SPE context switch**
   **(regardless of security state of the pre-empted context)**

If the SPE context targeted by the interrupt is not same as current SPE context,
the SPM may choose to switch the current running SPE context based on priority.

3. **SPE scheduler must treat pre-empted context as one of the SPE contexts**

  a. If the pre-empted SPE context is SP1, the TCB for SP1 should be used for
     saving the context. i.e. the context of SP1 should be saved before
     scheduling anything other secure partition.
  b. If SP1 was pre-empted by a NSPE interrupt, and subsequent NSPE execution is
     pre-empted by SPE exception (before NSPE scheduling decision is communicated
     back to SPM) -- SP1 TCB must be used for saving the context
     In this case SPM is not yet aware of the NSPE context switch, from SPM's
     standpoint SP1 is still executing, so SPM assumes that the preempted context
     is SP1.
  c. If SP1 was pre-empted by a NSPE interrupt, and subsequent NSPE execution is
     pre-empted by SPE exception `after` NSPE scheduling decision is
     communicated back to SPM) - a TCB dedicated to NSPE should be used for
     saving the context.

  When NSPE scheduler communicates the scheduling decision to SPM, SPM must save
  the SP1 context, if a SPE interrupt preempts the currently running NSPE context,
  SPM should save the context to a dedicated NSPE TCB.

  d. The SPE scheduler must eventually 'restore' the pre-empted context.
  This is an expected behaviour of any scheduler.

4. **All of the interrupts belonging to a partition must have same priority.**

This serializes ISR execution targeted for same partition.

5. **In case of nested interrupts, all of the ISRs must run to finish before**
   **any service code is allowed to run**

This is an expected behaviour of any scheduler.

6. **If the previously preempted context was a NSPE ISR context, SPE ISR**
   **handler must return to preempted NSPE context.**

This is an expected behaviour of any scheduler to return to preempted context.

Rules for NSPE to SPM function call (and NSPE scheduler)
--------------------------------------------------------

1. Current NSPE context must have been communicated to SPM, otherwise SPM cannot
   guarantee NSPE function calling stack integrity.

Rules for Function Return from SPE to NSPE with result
------------------------------------------------------

1. **The result available on SPE side are for currently active NSPE context.**

To maintain call stack integrity, if SPE is ready to return to NSPE, it can do
function return only if the SPE return path corresponds to currently active NSPE
context.

2. **Last entry into secure world happened programmatically (Voluntary**
   **security state switch into SPE)**

i.e. control is voluntarily given back by NSPE, either through a function call,
or a context restore via 'return to SPE from NSPE'. As opposed to a SPE
interrupt bringing back the execution into SPE.

3. **The current NSPE call stack has not already been returned with SPM_IDLE.**

This rule applies if following optional feature is enabled.

Rules for Return from SPE to NSPE with SPM_IDLE
-----------------------------------------------

This is optional part of the design as it introduces significant complexity on
both sides of the security boundary.
It allows yielding of the CPU to NSPE when SPE has not CPU execution to do but
it has not yet finished the previous request(s) from NSPE; i.e. SPE is waiting
on arrival of a SPE interrupt.

1. **Last entry into secure world happens programmatically (Voluntary**
   **security context switch into SPE)**

i.e. control is voluntarily given back by NSPE, either through a function call,
or a context restore via 'return to SPE from NSPE'. As opposed to a SPE
interrupt bringing back the execution into SPE.

2. **The result for the currently active NSPE entity is not yet available,**
   **the called service is waiting (on interrupt/event).**

SPE request corresponding to currently active NSPE caller is not yet completed
and is waiting on an ISR.

3. **The current NSPE call stack has not already been returned with SPM_IDLE.**

Rules for NSPE pend irq based return from SPE to NSPE
-----------------------------------------------------

This is optional part of the design as it introduces significant complexity on
both sides. This works in conjunction with [Rules for Return from SPE to NSPE
with SPM_IDLE](#rules-for-return-from-spe-to-nspe-with-spm_idle).
In this scenario, when SPE is ready with result for a previous call from NSPE,
it raises a pended IRQ to NSPE instead of returning the function call path.

1. **The SPE has finished a NSPE request.**

2. **The corresponding NSPE context has already been returned with SPM_IDLE.**

Rules for ISR pre-emption
-------------------------

1. **A higher priority NSPE interrupt is allowed to preempt a lower priority**
   **NSPE ISR**

2. **A higher priority SPE interrupt is allowed to preempt a lower priority**
   **SPE ISR**

3. **A SPE interrupt is allowed to preempt NSPE ISR**

4. **A NSPE interrupt is not allowed to preempt SPE ISR**

5. **All interrupts belonging to a service must have same priority**

--------------

*Copyright (c) 2019, Arm Limited. All rights reserved.*