Initial commit for the official OP-TEE documentation
This initial commit brings in all existing documentation found across
various gits making up the documentation for the OP-TEE project. So in
summary what this includes and what has been done is:
- Import all old documentation (optee_os, optee_client, optee_test and
so on).
- Convert everything from markdown (*.md) to reStructuredText (*.rst).
- Add a couple of new pages (optee_docs, disclosure policy to name a
few).
- Add Sphinx configuration.
- Add Travis configuration to enable automatic testing of future
documentation patches.
- Finally, a general clean-up and verification of content,
links as well as verification of build instructions etc has also been
done.
diff --git a/architecture/core.rst b/architecture/core.rst
new file mode 100644
index 0000000..b480086
--- /dev/null
+++ b/architecture/core.rst
@@ -0,0 +1,1041 @@
+.. _core:
+
+####
+Core
+####
+
+.. _interrupt_handling:
+
+Interrupt handling
+******************
+This section describes how :ref:`optee_os` handles switches of world execution
+context based on :ref:`SMC` exceptions and interrupt notifications Interrupt
+notifications are IRQ/FIQ exceptions which may also imply switching of world
+execution context: normal world to secure world, or secure world to normal
+world.
+
+Use cases of world context switch
+=================================
+This section lists all the cases where optee_os is involved in world context
+switches. Optee_os executes in the secure world. World switch is done by the
+cores secure monitor level/mode, referred below as the Monitor.
+
+When the normal world invokes the secure world, the normal world executes a SMC
+instruction. The SMC exception is always trapped by the Monitor. If the related
+service targets the trusted OS, the Monitor will switch to optee_os world
+execution. When the secure world returns to the normal world, optee_os executes
+a SMC that is caught by the Monitor which switches back to the normal world.
+
+When a secure interrupt is signaled by the Arm GIC, it shall reach the optee_os
+interrupt exception vector. If the secure world is executing, optee_os will
+handle straight the interrupt from its exception vector. If the normal world is
+executing when the secure interrupt raises, the Monitor vector must handle the
+exception and invoke the optee_os to serve the interrupt.
+
+When a non-secure interrupt is signaled by the Arm GIC, it shall reach the
+normal world interrupt exception vector. If the normal world is executing, it
+will handle straight the exception from its exception vector. If the secure
+world is executing when the non-secure interrupt raises, optee_os will
+temporarily return back to normal world via the Monitor to let normal world
+serve the interrupt.
+
+Core exception vectors
+======================
+Monitor vector is ``VBAR_EL3`` in AArch64 and ``MVBAR`` in Armv7-A/AArch32.
+Monitor can be reached while normal world or secure world is executing. The
+executing secure state is known to the Monitor through the ``SCR_NS``.
+
+Monitor can be reached from a SMC exception, an IRQ or FIQ exception (so-called
+interrupts) and from asynchronous aborts. Obviously monitor aborts (data,
+prefetch, undef) are local to the Monitor execution.
+
+The Monitor can be external to optee_os (case ``CFG_WITH_ARM_TRUSTED_FW=y``).
+If not, provides a local secure monitor ``core/arch/arm/sm``. Armv7-A platforms
+should use the optee_os secure monitor. Armv8-A platforms are likely to rely on
+an `Trusted Firmware A`_.
+
+When executing outside the Monitor, the system is executing either in the
+normal world (``SCR_NS=1``) or in the secure world (``SCR_NS=0``). Each world
+owns its own exception vector table (state vector):
+
+ - ``VBAR_EL2`` or ``VBAR_EL1`` non-secure or ``VBAR_EL1`` secure for
+ AArch64.
+ - ``HVBAR`` or ``VBAR`` non-secure or ``VBAR`` secure for Armv7-A and
+ AArch32.
+
+All SMC exceptions are trapped in the Monitor vector. IRQ/FIQ exceptions can be
+trapped either in the Monitor vector or in the state vector of the executing
+world.
+
+When the normal world is executing, the system is configured to route:
+
+ - secure interrupts to the Monitor that will forward to optee_os
+ - non-secure interrupts to the executing world exception vector.
+
+When the secure world is executing, the system is configured to route:
+
+ - secure and non-secure interrupts to the executing optee_os exception
+ vector. optee_os shall forward the non-secure interrupts to the normal
+ world.
+
+Optee_os non-secure interrupts are always trapped in the state vector of the
+executing world. This is reflected by a static value of ``SCR_(IRQ|FIQ)``.
+
+.. _native_foreign_irqs:
+
+Native and foreign interrupts
+=============================
+Two types of interrupt are defined in optee_os:
+
+ - **Native interrupt** - The interrupt handled by optee_os (for example:
+ secure interrupt)
+ - **Foreign interrupt** - The interrupt not handled by optee_os (for
+ example: non-secure interrupt which is handled by normal world)
+
+For Arm **GICv2** mode, native interrupt is sent as FIQ and foreign interrupt
+is sent as IRQ. For Arm **GICv3** mode, foreign interrupt is sent as FIQ which
+could be handled by either secure world (aarch32 Monitor mode or aarch64 EL3)
+or normal world. Arm GICv3 mode can be enabled by setting ``CFG_ARM_GICV3=y``.
+For clarity, this document mainly chooses the GICv2 convention and refers the
+IRQ as optee_os foreign interrupts, and FIQ as optee_os native interrupts.
+Native interrupts must be securely routed to optee_os. Foreign interrupts, when
+trapped during secure world execution might need to be efficiently routed to
+the normal world.
+
+Normal World invokes optee_os using SMC
+=======================================
+
+**Entering the Secure Monitor**
+
+The monitor manages all entries and exits of secure world. To enter secure
+world from normal world the monitor saves the state of normal world (general
+purpose registers and system registers which are not banked) and restores the
+previous state of secure world. Then a return from exception is performed and
+the restored secure state is resumed. Exit from secure world to normal world is
+the reverse.
+
+Some general purpose registers are not saved and restored on entry and exit,
+those are used to pass parameters between secure and normal world (see
+ARM_DEN0028A_SMC_Calling_Convention_ for details).
+
+**Entry and exit of Trusted OS**
+
+On entry and exit of Trusted OS each CPU is uses a separate entry stack and runs
+with IRQ and FIQ blocked. SMCs are categorised in two flavors: **fast** and
+**standard**.
+
+ - For **fast** SMCs, optee_os will execute on the entry stack with IRQ/FIQ
+ blocked until the execution returns to normal world.
+
+ - For **standard** SMCs, optee_os will at some point execute the requested
+ service with interrupts unblocked. In order to handle interrupts, mainly
+ forwarding of foreign interrupts, optee_os assigns a trusted thread
+ (`core/arch/arm/kernel/thread.c`_) to the SMC request. The trusted thread
+ stores the execution context of the requested service. This context can be
+ suspended and resumed as the requested service executes and is
+ interrupted. The trusted thread is released only once the service
+ execution returns with a completion status.
+
+ For **standard** SMCs, optee_os allocates or resumes a trusted thread then
+ unblock the IRQ/FIQ lines. When the optee_os needs to invoke the normal
+ world from a foreign interrupt or a remote service call, optee_os blocks
+ IRQ/FIQ and suspends the trusted thread. When suspending, optee_os gets
+ back to the entry stack.
+
+ - **Both** fast and standard SMC end on the entry stack with IRQ/FIQ blocked
+ and optee_os invokes the Monitor through a SMC to return to the normal
+ world.
+
+.. figure:: ../images/core/interrupt_handling/tee_invoke.png
+ :figclass: align-center
+
+ SMC entry to secure world
+
+Deliver non-secure interrupts to Normal World
+=============================================
+This section uses the Arm GICv1/v2 conventions: IRQ signals non-secure
+interrupts while FIQ signals secure interrupts. On a GICv3 configuration, one
+should exchange IRQ and FIQ in this section.
+
+**Forward a Foreign Interrupt from Secure World to Normal World**
+
+When an IRQ is received in secure world as an IRQ exception then secure world:
+
+ 1. Saves trusted thread context (entire state of all processor modes for
+ Armv7-A)
+
+ 2. Blocks (masks) all interrupts (IRQ and FIQ)
+
+ 3. Switches to entry stack
+
+ 4. Issues an SMC with a value to indicates to normal world that an IRQ has
+ been delivered and last SMC call should be continued
+
+The monitor restores normal world context with a return code indicating that an
+IRQ is about to be delivered. Normal world issues a new SMC indicating that it
+should continue last SMC.
+
+The monitor restores secure world context which locates the previously saved
+context and checks that it is a return from IRQ that is requested before
+restoring the context and lets the secure world IRQ handler return from
+exception where the execution would be resumed.
+
+Note that the monitor itself does not know/care that it has just forwarded an
+IRQ to normal world. The bookkeeping is done in the trusted thread handling in
+Trusted OS. Normal world is responsible to decide when the secure world thread
+should resume execution (for details, see :ref:`thread_handling`).
+
+.. figure:: ../images/core/interrupt_handling/irq.png
+ :figclass: align-center
+
+ IRQ received in secure world and forwarded to normal world
+
+**Deliver a non-secure interrupt to normal world when ``SCR_NS`` is set**
+
+Since ``SCR_IRQ`` is cleared, an IRQ will be delivered using the state vector
+(``VBAR``) in the normal world. The IRQ is received as any other exception by
+normal world, the monitor and the Trusted OS are not involved at all.
+
+Deliver secure interrupts to Secure World
+=========================================
+This section uses the Arm GICv1/v2 conventions: FIQ signals secure interrupts
+while IRQ signals non-secure interrupts. On a GICv3 configuration, one should
+exchange IRQ and FIQ in this section. A FIQ can be received during two different
+states, either in normal world (``SCR_NS`` is set) or in secure world
+(``SCR_NS`` is cleared). When the secure monitor is active (Armv8-A EL3 or
+Armv7-A Monitor mode) FIQ is masked. FIQ reception in the two different states
+is described below.
+
+**Deliver FIQ to secure world when SCR_NS is set**
+
+When the monitor gets an FIQ exception it:
+
+ 1. Saves normal world context and restores secure world context from last
+ secure world exit (which will have IRQ and FIQ blocked)
+ 2. Clears ``SCR_FIQ`` when clearing ``SCR_NS``
+ 3. Sets “FIQ” as parameter to secure world entry
+ 4. Does a return from exception into secure context
+ 5. Secure world unmasks FIQs because of the “FIQ” parameter
+ 6. FIQ is received as in exception using the state vector
+ 7. The state vector handle returns from exception in secure world
+ 8. Secure world issues an SMC to return to normal world
+ 9. Monitor saves secure world context and restores normal world context
+ 10. Does a return from exception into restored context
+
+.. figure:: ../images/core/interrupt_handling/fiq.png
+ :figclass: align-center
+
+ FIQ received when SCR_NS is set
+
+.. figure:: ../images/core/interrupt_handling/irq_fiq.png
+ :figclass: align-center
+
+ FIQ received while processing an IRQ forwarded from secure world
+
+**Deliver FIQ to secure world when SCR_NS is cleared**
+
+Since ``SCR_FIQ`` is cleared when ``SCR_NS`` is cleared a FIQ will be delivered
+using the state vector (``VBAR``) in secure world. The FIQ is received as any
+other exception by Trusted OS, the monitor is not involved at all.
+
+Trusted thread scheduling
+=========================
+**Trusted thread for standard services**
+
+OP-TEE standard services are carried through standard SMC. Execution of these
+services can be interrupted by foreign interrupts. To suspend and restore the
+service execution, optee_os assigns a trusted thread at standard SMCs entry.
+
+The trusted thread terminates when optee_os returns to the normal world with a
+service completion status.
+
+A trusted thread execution can be interrupted by a native interrupt. In this
+case the native interrupt is handled by the interrupt exception handlers and
+once served, optee_os returns to the execution trusted thread.
+
+A trusted thread execution can be interrupted by a foreign interrupt. In this
+case, optee_os suspends the trusted thread and invokes the normal world through
+the Monitor (optee_os so-called RPC services). The trusted threads will resume
+only once normal world invokes the optee_os with the RPC service status.
+
+A trusted thread execution can lead optee_os to invoke a service in normal
+world: access a file, get the REE current time, etc. The trusted thread is
+suspended/resumed during remote service execution.
+
+**Scheduling considerations**
+
+When a trusted thread is interrupted by a foreign interrupt and when optee_os
+invokes a normal world service, the normal world gets the opportunity to
+reschedule the running applications. The trusted thread will resume only once
+the client application is scheduled back. Thus, a trusted thread execution
+follows the scheduling of the normal world caller context.
+
+Optee_os does not implement any thread scheduling. Each trusted thread is
+expected to track a service that is invoked from the normal world and should
+return to it with an execution status.
+
+The OP-TEE Linux driver (as implemented in `drivers/tee/optee`_ since Linux
+kernel 4.12) is designed so that the Linux thread invoking OP-TEE gets assigned
+a trusted thread on TEE side. The execution of the trusted thread is tied to the
+execution of the caller Linux thread which is under the Linux kernel scheduling
+decision. This means trusted threads are scheduled by the Linux kernel.
+
+**Trusted thread constraints**
+
+TEE core handles a static number of trusted threads, see ``CFG_NUM_THREADS``.
+
+Trusted threads are only expensive on memory constrained system, mainly
+regarding the execution stack size.
+
+On SMP systems, optee_os can execute several trusted threads in parallel if the
+normal world supports scheduling of processes. Even on UP systems, supporting
+several trusted threads in optee_os helps normal world scheduler to be
+efficient.
+
+----
+
+.. _memory_objects:
+
+Memory objects
+**************
+A memory object, **MOBJ**, describes a piece of memory. The interface provided
+is mostly abstract when it comes to using the MOBJ to populate translation
+tables etc. There are different kinds of MOBJs describing:
+
+ - Physically contiguous memory
+ - created with ``mobj_phys_alloc(...)``.
+
+ - Virtual memory
+ - one instance with the name ``mobj_virt`` available.
+ - spans the entire virtual address space.
+
+ - Physically contiguous memory allocated from a ``tee_mm_pool_t *``
+ - created with ``mobj_mm_alloc(...)``.
+
+ - Paged memory
+ - created with ``mobj_paged_alloc(...)``.
+ - only contains the supplied size and makes ``mobj_is_paged(...)``
+ return true if supplied as argument.
+
+ - Secure copy paged shared memory
+ - created with ``mobj_seccpy_shm_alloc(...)``.
+ - makes ``mobj_is_paged(...)`` and ``mobj_is_secure(...)`` return true
+ if supplied as argument.
+
+----
+
+.. _mmu:
+
+MMU
+***
+Translation tables
+==================
+OP-TEE uses several L1 translation tables, one large spanning 4 GiB and two or
+more small tables spanning 32 MiB. The large translation table handles kernel
+mode mapping and matches all addresses not covered by the small translation
+tables. The small translation tables are assigned per thread and covers the
+mapping of the virtual memory space for one TA context.
+
+Memory space between small and large translation table is configured by TTBRC.
+TTBR1 always points to the large translation table. TTBR0 points to the a small
+translation table when user mapping is active and to the large translation table
+when no user mapping is currently active. For details about registers etc,
+please refer to a Technical Reference Manual for your architecture, for example
+`Cortex-A53 TRM`_.
+
+The translation tables has certain alignment constraints, the alignment (of the
+physical address) has to be the same as the size of the translation table. The
+translation tables are statically allocated to avoid fragmentation of memory due
+to the alignment constraints.
+
+Each thread has one small L1 translation table of its own. Each TA context has a
+compact representation of its L1 translation table. The compact representation
+is used to initialize the thread specific L1 translation table when the TA
+context is activated.
+
+.. graphviz::
+
+ digraph xlat_table {
+ graph [
+ rankdir = "LR"
+ ];
+ node [
+ fontsize = "16"
+ shape = "ellipse"
+ ];
+ edge [
+ ];
+ "node_ttb" [
+ label = "<f0> TTBR0 | <f1> TTBR1"
+ shape = "record"
+ ];
+ "node_large_l1" [
+ label = "<f0> Large L1\nSpans 4 GiB"
+ shape = "record"
+ ];
+ "node_small_l1" [
+ label = "Small L1\nSpans 32 MiB\nper entry | <f0> 0 | <f1> 1 | ... | <fn> n"
+ shape = "record"
+ ];
+
+ "node_ttb":f0 -> "node_small_l1":f0 [ label = "Thread 0 ctx active" ];
+ "node_ttb":f0 -> "node_small_l1":f1 [ label = "Thread 1 ctx active" ];
+ "node_ttb":f0 -> "node_small_l1":fn [ label = "Thread n ctx active" ];
+ "node_ttb":f0 -> "node_large_l1" [ label="No active ctx" ];
+ "node_ttb":f1 -> "node_large_l1";
+ }
+
+
+Switching to user mode
+======================
+This section only applies with following configuration flags:
+
+ - ``CFG_WITH_LPAE=n``
+ - ``CFG_CORE_UNMAP_CORE_AT_EL0=y``
+
+When switching to user mode only a minimal kernel mode mapping is kept. This is
+achieved by selecting a zeroed out big L1 translation in TTBR1 when
+transitioning to user mode. When returning back to kernel mode the original L1
+translation table is restored in TTBR1.
+
+Switching to normal world
+=========================
+When switching to normal world either via a foreign interrupt (see
+:ref:`native_foreign_irqs` or RPC there is a chance that secure world will
+resume execution on a different CPU. This means that the new CPU need to be
+configured with the context of the currently active TA. This is solved by always
+setting the TA context in the CPU when resuming execution. Here is room for
+improvements since it is more likely than not that it is the same CPU that
+resumes execution in secure world.
+
+.. todo::
+
+ Joakim: Jens? Didn't you do some tweaks here already? I.e., "room for
+ improvements" above?
+
+----
+
+.. _pager:
+
+Pager
+*****
+OP-TEE currently requires >256 KiB RAM for OP-TEE kernel memory. This is not a
+problem if OP-TEE uses TrustZone protected DDR, but for security reasons OP-TEE
+may need to use TrustZone protected SRAM instead. The amount of available SRAM
+varies between platforms, from just a few KiB up to over 512 KiB. Platforms with
+just a few KiB of SRAM cannot be expected to be able to run a complete TEE
+solution in SRAM. But those with 128 to 256 KiB of SRAM can be expected to have
+a capable TEE solution in SRAM. The pager provides a solution to this by demand
+paging parts of OP-TEE using virtual memory.
+
+Secure memory
+=============
+TrustZone protected SRAM is generally considered more secure than TrustZone
+protected DRAM as there is usually more attack vectors on DRAM. The attack
+vectors are hardware dependent and can be different for different platforms.
+
+Backing store
+=============
+TrustZone protected DRAM or in some cases non-secure DRAM is used as backing
+store. The data in the backing store is integrity protected with one hash
+(SHA-256) per page (4KiB). Readonly pages are not encrypted since the OP-TEE
+binary itself is not encrypted.
+
+Partitioning of memory
+======================
+The code that handles demand paging must always be available as it would
+otherwise lead to deadlock. The virtual memory is partitioned as:
+
+ +--------------+-------------------+
+ | Type | Sections |
+ +==============+===================+
+ | unpaged | | text |
+ | | | rodata |
+ | | | data |
+ | | | bss |
+ | | | heap1 |
+ | | | nozi |
+ | | | heap2 |
+ +--------------+-------------------+
+ | init / paged | | text_init |
+ | | | rodata_init |
+ +--------------+-------------------+
+ | paged | | text_pageable |
+ | | | rodata_pageable |
+ +--------------+-------------------+
+ | demand alloc | |
+ +--------------+-------------------+
+
+Where ``nozi`` stands for "not zero initialized", this section contains entry
+stacks (thread stack when TEE pager is not enabled) and translation tables (TEE
+pager cached translation table when the pager is enabled and LPAE MMU is used).
+
+The ``init`` area is available when OP-TEE is initializing and contains
+everything that is needed to initialize the pager. After the pager has been
+initialized this area will be used for demand paged instead.
+
+The ``demand alloc`` area is a special area where the pages are allocated and
+removed from the pager on demand. Those pages are returned when OP-TEE does not
+need them any longer. The thread stacks currently belongs this area. This means
+that when a stack is not used the physical pages can be used by the pager for
+better performance.
+
+The technique to gather code in the different area is based on compiling all
+functions and data into separate sections. The unpaged text and rodata is then
+gathered by linking all object files with ``--gc-sections`` to eliminate
+sections that are outside the dependency graph of the entry functions for
+unpaged functions. A script analyzes this ELF file and generates the bits of the
+final link script. The process is repeated for init text and rodata. What is
+not "unpaged" or "init" becomes "paged".
+
+Partitioning of the binary
+==========================
+.. note::
+ The struct definitions provided in this section are explicitly covered by
+ the following dual license:
+
+ .. code-block:: none
+
+ SPDX-License-Identifier: (BSD-2-Clause OR GPL-2.0)
+
+The binary is partitioned into four parts as:
+
+
+ +----------+
+ | Binary |
+ +==========+
+ | Header |
+ +----------+
+ | Init |
+ +----------+
+ | Hashes |
+ +----------+
+ | Pageable |
+ +----------+
+
+The header is defined as:
+
+.. code-block:: c
+
+ #define OPTEE_MAGIC 0x4554504f
+ #define OPTEE_VERSION 1
+ #define OPTEE_ARCH_ARM32 0
+ #define OPTEE_ARCH_ARM64 1
+
+ struct optee_header {
+ uint32_t magic;
+ uint8_t version;
+ uint8_t arch;
+ uint16_t flags;
+ uint32_t init_size;
+ uint32_t init_load_addr_hi;
+ uint32_t init_load_addr_lo;
+ uint32_t init_mem_usage;
+ uint32_t paged_size;
+ };
+
+The header is only used by the loader of OP-TEE, not OP-TEE itself. To
+initialize OP-TEE the loader loads the complete binary into memory and copies
+what follows the header and the following ``init_size`` bytes to
+``(init_load_addr_hi << 32 | init_load_addr_lo)``. ``init_mem_usage`` is used by
+the loader to be able to check that there is enough physical memory available
+for OP-TEE to be able to initialize at all. The loader supplies in ``r0/x0`` the
+address of the first byte following what was not copied and jumps to the load
+address to start OP-TEE.
+
+In addition to overall binary with partitions inside described as above, three
+extra binaries are generated simultaneously during build process for loaders who
+support loading separate binaries:
+
+ +-----------+
+ | v2 binary |
+ +===========+
+ | Header |
+ +-----------+
+
+ +-----------+
+ | v2 binary |
+ +===========+
+ | Init |
+ +-----------+
+ | Hashes |
+ +-----------+
+
+ +-----------+
+ | v2 binary |
+ +===========+
+ | Pageable |
+ +-----------+
+
+In this case, loaders load header binary first to get image list and information
+of each image; and then load each of them into specific load address assigned in
+structure. These binaries are named with `v2` suffix to distinguish from the
+existing binaries. Header format is updated to help loaders loading binaries
+efficiently:
+
+.. code-block:: c
+
+ #define OPTEE_IMAGE_ID_PAGER 0
+ #define OPTEE_IMAGE_ID_PAGED 1
+
+ struct optee_image {
+ uint32_t load_addr_hi;
+ uint32_t load_addr_lo;
+ uint32_t image_id;
+ uint32_t size;
+ };
+
+ struct optee_header_v2 {
+ uint32_t magic;
+ uint8_t version;
+ uint8_t arch;
+ uint16_t flags;
+ uint32_t nb_images;
+ struct optee_image optee_image[];
+ };
+
+Magic number and architecture are identical as original. Version is increased to
+two. ``load_addr_hi`` and ``load_addr_lo`` may be ``0xFFFFFFFF`` for pageable
+binary since pageable part may get loaded by loader into dynamic available
+position. ``image_id`` indicates how loader handles current binary. Loaders who
+don't support separate loading just ignore all v2 binaries.
+
+Initializing the pager
+======================
+The pager is initialized as early as possible during boot in order to minimize
+the "init" area. The global variable ``tee_mm_vcore`` describes the virtual
+memory range that is covered by the level 2 translation table supplied to
+``tee_pager_init(...)``.
+
+Assign pageable areas
+---------------------
+A virtual memory range to be handled by the pager is registered with a call to
+``tee_pager_add_core_area()``.
+
+.. code-block:: c
+
+ bool tee_pager_add_area(tee_mm_entry_t *mm,
+ uint32_t flags,
+ const void *store,
+ const void *hashes);
+
+which takes a pointer to ``tee_mm_entry_t`` to tell the range, flags to tell how
+memory should be mapped (readonly, execute etc), and pointers to backing store
+and hashes of the pages.
+
+Assign physical pages
+---------------------
+Physical SRAM pages are supplied by calling ``tee_pager_add_pages(...)``
+
+.. code-block:: c
+
+ void tee_pager_add_pages(tee_vaddr_t vaddr,
+ size_t npages,
+ bool unmap);
+
+``tee_pager_add_pages(...)`` takes the physical address stored in the entry
+mapping the virtual address ``vaddr`` and ``npages`` entries after that and uses
+it to map new pages when needed. The unmap parameter tells whether the pages
+should be unmapped immediately since they does not contain initialized data or
+be kept mapped until they need to be recycled. The pages in the "init" area are
+supplied with ``unmap == false`` since those page have valid content and are in
+use.
+
+Invocation
+==========
+The pager is invoked as part of the abort handler. A pool of physical pages are
+used to map different virtual addresses. When a new virtual address needs to be
+mapped a free physical page is mapped at the new address, if a free physical
+page cannot be found the oldest physical page is selected instead. When the page
+is mapped new data is copied from backing store and the hash of the page is
+verified. If it is OK the pager returns from the exception to resume the
+execution.
+
+Paging of user TA
+=================
+Paging of user TAs can optionally be enabled with ``CFG_PAGED_USER_TA=y``.
+Paging of user TAs is analogous to paging of OP-TEE kernel parts but with a few
+differences:
+
+ - Read/write pages are paged in addition to read-only pages
+ - Page tables are managed dynamically
+
+``tee_pager_add_uta_area(...)`` is used to setup initial read/write mapping
+needed when populating the TA. When the TA is fully populated and relocated
+``tee_pager_set_uta_area_attr(...)`` changes the mapping of the area to strict
+permissions used when the TA is running.
+
+----
+
+.. _stacks:
+
+Stacks
+******
+Different stacks are used during different stages. The stacks are:
+
+ - **Secure monitor stack** (128 bytes), bound to the CPU. Only available if
+ OP-TEE is compiled with a secure monitor always the case if the target is
+ Armv7-A but never for Armv8-A.
+
+ - **Temp stack** (small ~1KB), bound to the CPU. Used when transitioning
+ from one state to another. Interrupts are always disabled when using this
+ stack, aborts are fatal when using the temp stack.
+
+ - **Abort stack** (medium ~2KB), bound to the CPU. Used when trapping a data
+ or pre-fetch abort. Aborts from user space are never fatal the TA is only
+ killed. Aborts from kernel mode are used by the pager to do the demand
+ paging, if pager is disabled all kernel mode aborts are fatal.
+
+ - **Thread stack** (large ~8KB), not bound to the CPU instead used by the
+ current thread/task. Interrupts are usually enabled when using this stack.
+
+Notes for Armv7-A/AArch32
+ .. list-table::
+ :header-rows: 1
+ :widths: 1 5
+
+ * - Stack
+ - Comment
+
+ * - Temp
+ - Assigned to ``SP_SVC`` during entry/exit, always assigned to
+ ``SP_IRQ`` and ``SP_FIQ``
+
+ * - Abort
+ - Always assigned to ``SP_ABT``
+
+ * - Thread
+ - Assigned to ``SP_SVC`` while a thread is active
+
+Notes for AArch64
+ There are only two stack pointers, ``SP_EL1`` and ``SP_EL0``, available for
+ OP-TEE in AArch64. When an exception is received stack pointer is always
+ ``SP_EL1`` which is used temporarily while assigning an appropriate stack
+ pointer for ``SP_EL0``. ``SP_EL1`` is always assigned the value of
+ ``thread_core_local[cpu_id]``. This structure has some spare space for
+ temporary storage of registers and also keeps the relevant stack pointers.
+ In general when we talk about assigning a stack pointer to the CPU below we
+ mean ``SP_EL0``.
+
+Boot
+====
+During early boot the CPU is configured with the temp stack which is used until
+OP-TEE exits to normal world the first time.
+
+Notes for AArch64
+ ``SPSEL`` is always ``0`` on entry/exit to have ``SP_EL0`` acting as stack
+ pointer.
+
+Normal entry
+============
+Each time OP-TEE is entered from normal world the temp stack is used as the
+initial stack. For fast calls, this is the only stack used. For normal calls an
+empty thread slot is selected and the CPU switches to that stack.
+
+Normal exit
+===========
+Normal exit occurs when a thread has finished its task and the thread is freed.
+When the main thread function, ``tee_entry_std(...)``, returns interrupts are
+disabled and the CPU switches to the temp stack instead. The thread is freed and
+OP-TEE exits to normal world.
+
+RPC exit
+========
+RPC exit occurs when OP-TEE need some service from normal world. RPC can
+currently only be performed with a thread is in running state. RPC is initiated
+with a call to ``thread_rpc(...)`` which saves the state in a way that when the
+thread is restored it will continue at the next instruction as if this function
+did a normal return. CPU switches to use the temp stack before returning to
+normal world.
+
+Foreign interrupt exit
+======================
+Foreign interrupt exit occurs when OP-TEE receives a foreign interrupt. For Arm
+GICv2 mode, foreign interrupt is sent as IRQ which is always handled in normal
+world. Foreign interrupt exit is similar to RPC exit but it is
+``thread_irq_handler(...)`` and ``elx_irq(...)`` (respectively for
+Armv7-A/Aarch32 and for Aarch64) that saves the thread state instead. The thread
+is resumed in the same way though. For Arm GICv3 mode, foreign interrupt is sent
+as FIQ which could be handled by either secure world (EL3 in AArch64) or normal
+world. This mode is not supported yet.
+
+Notes for Armv7-A/AArch32
+ SP_IRQ is initialized to temp stack instead of a separate stack. Prior to
+ exiting to normal world CPU state is changed to SVC and temp stack is
+ selected.
+
+Notes for AArch64
+ ``SP_EL0`` is assigned temp stack and is selected during IRQ processing. The
+ original ``SP_EL0`` is saved in the thread context to be restored when
+ resuming.
+
+Resume entry
+============
+OP-TEE is entered using the temp stack in the same way as for normal entry. The
+thread to resume is looked up and the state is restored to resume execution. The
+procedure to resume from an RPC exit or an foreign interrupt exit is exactly the
+same.
+
+Syscall
+=======
+Syscall's are executed using the thread stack.
+
+Notes for Armv7-A/AArch32
+ Nothing special ``SP_SVC`` is already set with thread stack.
+
+Notes for syscall AArch64
+ Early in the exception processing the original ``SP_EL0`` is saved in
+ ``struct thread_svc_regs`` in case the TA is executed in AArch64. Current
+ thread stack is assigned to ``SP_EL0`` which is then selected. When
+ returning ``SP_EL0`` is assigned what is in ``struct thread_svc_regs``. This
+ allows ``tee_svc_sys_return_helper(...)`` having the syscall exception
+ handler return directly to ``thread_unwind_user_mode(...)``.
+
+----
+
+.. _shared_memory:
+
+Shared Memory
+*************
+Shared Memory is a block of memory that is shared between the non-secure and the
+secure world. It is used to transfer data between both worlds.
+
+Shared Memory Allocation
+========================
+The shared memory is allocated by the Linux driver from a pool ``struct
+shm_pool``, the pool contains:
+
+ - The physical address of the start of the pool
+ - The size of the pool
+ - Whether or not the memory is cached
+ - List of chunk of memory allocated.
+
+.. note::
+ - The shared memory pool is physically contiguous.
+ - The shared memory area is **not secure** as it is used by both non-secure
+ and secure world.
+
+Shared Memory Configuration
+===========================
+It is the Linux kernel driver for OP-TEE that is responsible for initializing
+the shared memory pool, given information provided by the OP-TEE core. The
+Linux driver issues a SMC call ``OPTEE_SMC_GET_SHM_CONFIG`` to retrieve the
+information
+
+ - Physical address of the start of the pool
+ - Size of the pool
+ - Whether or not the memory is cached
+
+The shared memory pool configuration is platform specific. The memory mapping,
+including the area ``MEM_AREA_NSEC_SHM`` (shared memory with non-secure world),
+is retrieved by calling the platform-specific function ``bootcfg_get_memory()``.
+Please refer to this function and the area type ``MEM_AREA_NSEC_SHM`` to see the
+configuration for the platform of interest. The Linux driver will then
+initialize the shared memory pool accordingly.
+
+.. todo::
+
+ Joakim: bootcfg_get_memory(...) is no longer in our code. Text needs update.
+
+Shared Memory Chunk Allocation
+==============================
+It is the Linux kernel driver for OP-TEE that is responsible for allocating
+chunks of shared memory. OP-TEE linux kernel driver relies on linux kernel
+generic allocation support (``CONFIG_GENERIC_ALLOCATION``) to allocation/release
+of shared memory physical chunks. OP-TEE linux kernel driver relies on linux
+kernel dma-buf support (``CONFIG_DMA_SHARED_BUFFER``) to track shared memory
+buffers references.
+
+Using shared memory
+===================
+From the Client Application
+ The client application can ask for shared memory allocation using the
+ GlobalPlatform Client API function ``TEEC_AllocateSharedMemory(...)``. The
+ client application can also provide shared memory through the GlobalPlatform
+ Client API function ``TEEC_RegisterSharedMemory(...)``. In such a case, the
+ provided memory must be physically contiguous, since OP-TEE core, who does
+ not handle scatter-gather memory, is able to use the provided range of
+ memory addresses. Note that the reference count of a shared memory chunk is
+ incremented when shared memory is registered, and initialized to 1 on
+ allocation.
+
+From the Linux Driver
+ Occasionally the Linux kernel driver needs to allocate shared memory for the
+ communication with secure world, for example when using buffers of type
+ ``TEEC_TempMemoryReference``.
+
+From OP-TEE core
+ In case OP-TEE core needs information from TEE supplicant (dynamic TA
+ loading, REE time request,...), shared memory must be allocated. Allocation
+ depends on the use case. OP-TEE core asks for the following shared memory
+ allocation:
+
+ - ``optee_msg_arg`` structure, used to pass the arguments to the
+ non-secure world, where the allocation will be done by sending a
+ ``OPTEE_SMC_RPC_FUNC_ALLOC`` message.
+
+ - In some cases, a payload might be needed for storing the result from
+ TEE supplicant, for example when loading a Trusted Application. This
+ type of allocation will be done by sending the message
+ ``OPTEE_MSG_RPC_CMD_SHM_ALLOC(OPTEE_MSG_RPC_SHM_TYPE_APPL,...)``,
+ which then will return:
+
+ - the physical address of the shared memory
+ - a handle to the memory, that later on will be used later on when
+ freeing this memory.
+
+From TEE Supplicant
+ TEE supplicant is also working with shared memory, used to exchange data
+ between normal and secure worlds. TEE supplicant receives a memory address
+ from the OP-TEE core, used to store the data. This is for example the case
+ when a Trusted Application is loaded. In this case, TEE supplicant must
+ register the provided shared memory in the same way a client application
+ would do, involving the Linux driver.
+
+----
+
+.. _smc:
+
+SMC
+***
+SMC Interface
+=============
+OP-TEE's SMC interface is defined in two levels using optee_smc.h_ and
+optee_msg.h_. The former file defines SMC identifiers and what is passed in the
+registers for each SMC. The latter file defines the OP-TEE Message protocol
+which is not restricted to only SMC even if that currently is the only option
+available.
+
+SMC communication
+=================
+The main structure used for the SMC communication is defined in ``struct
+optee_msg_arg`` (in optee_msg.h_). If we are looking into the source code, we
+could see that communication mainly is achieved using ``optee_msg_arg`` and
+``thread_smc_args`` (in thread.h_), where ``optee_msg_arg`` could be seen as the
+main structure. What will happen is that the :ref:`linux_kernel` driver will get
+the parameters either from :ref:`optee_client` or directly from an internal
+service in Linux kernel. The TEE driver will populate the struct
+``optee_msg_arg`` with the parameters plus some additional bookkeeping
+information. Parameters for the SMC are passed in registers 1 to 7, register 0
+holds the SMC id which among other things tells whether it is a standard or a
+fast call.
+
+----
+
+.. _thread_handling:
+
+Thread handling
+***************
+OP-TEE core uses a couple of threads to be able to support running jobs in
+parallel (not fully enabled!). There are handlers for different purposes. In
+thread.c_ you will find a function called ``thread_init_primary(...)`` which
+assigns ``init_handlers`` (functions) that should be called when OP-TEE core
+receives standard or fast calls, FIQ and PSCI calls. There are default handlers
+for these services, but the platform can decide if they want to implement their
+own platform specific handlers instead.
+
+Synchronization primitives
+==========================
+OP-TEE has three primitives for synchronization of threads and CPUs:
+*spin-lock*, *mutex*, and *condvar*.
+
+Spin-lock
+ A spin-lock is represented as an ``unsigned int``. This is the most
+ primitive lock. Interrupts should be disabled before attempting to take a
+ spin-lock and should remain disabled until the lock is released. A spin-lock
+ is initialized with ``SPINLOCK_UNLOCK``.
+
+ .. list-table:: Spin lock functions
+ :header-rows: 1
+ :widths: 1 5
+
+ * - Function
+ - Purpose
+
+ * - ``cpu_spin_lock(...)``
+ - Locks a spin-lock
+
+ * - ``cpu_spin_trylock(...)``
+ - Locks a spin-lock if unlocked and returns ``0`` else the spin-lock
+ is unchanged and the function returns ``!0``
+
+ * - ``cpu_spin_unlock(...)``
+ - Unlocks a spin-lock
+
+Mutex
+ A mutex is represented by ``struct mutex``. A mutex can be locked and
+ unlocked with interrupts enabled or disabled, but only from a normal thread.
+ A mutex cannot be used in an interrupt handler, abort handler or before a
+ thread has been selected for the CPU. A mutex is initialized with either
+ ``MUTEX_INITIALIZER`` or ``mutex_init(...)``.
+
+ .. list-table:: Mutex functions
+ :header-rows: 1
+ :widths: 1 5
+
+ * - Function
+ - Purpose
+
+ * - ``mutex_lock(...)``
+ - Locks a mutex. If the mutex is unlocked this is a fast operation,
+ else the function issues an RPC to wait in normal world.
+
+ * - ``mutex_unlock(...)``
+ - Unlocks a mutex. If there is no waiters this is a fast operation,
+ else the function issues an RPC to wake up a waiter in normal world.
+
+ * - ``mutex_trylock(...)``
+ - Locks a mutex if unlocked and returns ``true`` else the mutex is
+ unchanged and the function returns ``false``.
+
+ * - ``mutex_destroy(...)``
+ - Asserts that the mutex is unlocked and there is no waiters, after
+ this the memory used by the mutex can be freed.
+
+ When a mutex is locked it is owned by the thread calling ``mutex_lock(...)``
+ or ``mutex_trylock(...)``, the mutex may only be unlocked by the thread
+ owning the mutex. A thread should not exit to TA user space when holding a
+ mutex.
+
+Condvar
+ A condvar is represented by ``struct condvar``. A condvar is similar to a
+ ``pthread_condvar_t`` in the pthreads standard, only less advanced.
+ Condition variables are used to wait for some condition to be fulfilled and
+ are always used together a mutex. Once a condition variable has been used
+ together with a certain mutex, it must only be used with that mutex until
+ destroyed. A condvar is initialized with ``CONDVAR_INITIALIZER`` or
+ ``condvar_init(...)``.
+
+ .. list-table:: Condvar functions
+ :header-rows: 1
+ :widths: 1 5
+
+ * - Function
+ - Purpose
+
+ * - ``condvar_wait(...)``
+ - Atomically unlocks the supplied mutex and waits in normal world via
+ an RPC for the condition variable to be signaled, when the function
+ returns the mutex is locked again.
+
+ * - ``condvar_signal(...)``
+ - Wakes up one waiter of the condition variable (waiting in
+ ``condvar_wait(...)``).
+
+ * - ``condvar_broadcast(...)``
+ - Wake up all waiters of the condition variable.
+
+ The caller of ``condvar_signal(...)`` or ``condvar_broadcast(...)`` should
+ hold the mutex associated with the condition variable to guarantee that a
+ waiter does not miss the signal.
+
+.. _core/arch/arm/kernel/thread.c: https://github.com/OP-TEE/optee_os/blob/master/core/arch/arm/kernel/thread.c
+.. _optee_msg.h: https://github.com/OP-TEE/optee_os/blob/master/core/include/optee_msg.h
+.. _optee_smc.h: https://github.com/OP-TEE/optee_os/blob/master/core/arch/arm/include/sm/optee_smc.h
+.. _thread.c: https://github.com/OP-TEE/optee_os/blob/master/core/arch/arm/kernel/thread.c
+.. _thread.h: https://github.com/OP-TEE/optee_os/blob/master/core/arch/arm/include/kernel/thread.h
+
+.. _ARM_DEN0028A_SMC_Calling_Convention: http://infocenter.arm.com/help/topic/com.arm.doc.den0028b/ARM_DEN0028B_SMC_Calling_Convention.pdf
+.. _Cortex-A53 TRM: http://infocenter.arm.com/help/topic/com.arm.doc.ddi0500j/DDI0500J_cortex_a53_trm.pdf
+.. _drivers/tee/optee: https://github.com/torvalds/linux/tree/master/drivers/tee/optee
+.. _Trusted Firmware A: https://github.com/ARM-software/arm-trusted-firmware
diff --git a/architecture/crypto.rst b/architecture/crypto.rst
new file mode 100644
index 0000000..034b028
--- /dev/null
+++ b/architecture/crypto.rst
@@ -0,0 +1,185 @@
+.. _cryptographic_implementation:
+
+############################
+Cryptographic implementation
+############################
+This document describes how the TEE Cryptographic Operations API is implemented,
+how the default crypto provider may be configured at compile time, and how it
+may be replaced by another implementation.
+
+Overview
+********
+There are several layers from the Trusted Application to the actual crypto
+algorithms. Most of the crypto code runs in kernel mode inside the TEE core.
+Here is a schematic view of a typical call to the crypto API. The numbers in
+square brackets ([1], [2]...) refer to the sections below.
+
+.. code-block:: none
+
+ - some_function() (Trusted App) -
+ [1] TEE_*() User space (libutee.a)
+ ------- utee_*() ----------------------------------------------
+ [2] tee_svc_*() Kernel space
+ [3] crypto_*() (libtomcrypt.a and crypto.c)
+ [4] /* LibTomCrypt */ (libtomcrypt.a)
+
+[1] The TEE Cryptographic Operations API
+****************************************
+OP-TEE implements the Cryptographic Operations API defined by the GlobalPlatform
+association in the :ref:`tee_internal_core_api`. This includes cryptographic
+functions that span various cryptographic needs: message digests, symmetric
+ciphers, message authentication codes (MAC), authenticated encryption,
+asymmetric operations (encryption/decryption or signing/verifying), key
+derivation, and random data generation. These functions make up the TEE
+Cryptographic Operations API.
+
+The Internal API is implemented in tee_api_operations.c_, which is compiled into
+a static library: ``${O}/ta_arm{32,64}-lib/libutee/libutee.a``.
+
+Most API functions perform some parameter checking and manipulations, then
+invoke some *utee\_\** function to switch to kernel mode and perform the
+low-level work.
+
+The *utee\_\** functions are declared in utee_syscalls.h_ and implemented in
+utee_syscalls_asm.S_ They are simple system call wrappers which use the *SVC*
+instruction to switch to the appropriate system service in the OP-TEE kernel.
+
+[2] The crypto services
+***********************
+All cryptography-related system calls are declared in tee_svc_cryp.h_ and
+implemented in tee_svc_cryp.c_. In addition to dealing with the usual work
+required at the user/kernel interface (checking parameters and copying memory
+buffers between user and kernel space), the system calls invoke a private
+abstraction layer: the **Crypto API**, which is declared in crypto.h_. It serves
+two main purposes:
+
+ 1. Allow for alternative implementations, such as hardware-accelerated
+ versions.
+
+ 2. Provide an easy way to disable some families of algorithms at
+ compile-time to save space. See `LibTomCrypt` below.
+
+[3] crypto_*()
+**************
+The ``crypto_*()`` functions implement the actual algorithms and helper
+functions. TEE Core has one global active implementation of this interface. The
+default implementation, mostly based on LibTomCrypt_, is as follows:
+
+.. code-block:: c
+ :caption: File: core/crypto/crypto.c
+
+ /*
+ * Default implementation for all functions in crypto.h
+ */
+
+ #if !defined(_CFG_CRYPTO_WITH_HASH)
+ TEE_Result crypto_hash_get_ctx_size(uint32_t algo __unused,
+ size_t *size __unused)
+ {
+ return TEE_ERROR_NOT_IMPLEMENTED;
+ }
+ ...
+ #endif /*_CFG_CRYPTO_WITH_HASH*/
+
+.. code-block:: c
+ :caption: File: core/lib/libtomcrypt/tee_ltc_provider.c
+
+ #if defined(_CFG_CRYPTO_WITH_HASH)
+ TEE_Result crypto_hash_get_ctx_size(uint32_t algo, size_t *size)
+ {
+ /* ... */
+ return TEE_SUCCESS;
+ }
+
+ #endif /*_CFG_CRYPTO_WITH_HASH*/
+
+As shown above, families of algorithms can be disabled and crypto.c_ will
+provide default null implementations that will return
+``TEE_ERROR_NOT_IMPLEMENTED``.
+
+Public/private key format
+*************************
+crypto.h_ uses implementation-specific types to hold key data for asymmetric
+algorithms. For instance, here is how a public RSA key is represented:
+
+.. code-block:: c
+ :caption: File: core/include/crypto/crypto.h
+
+ struct rsa_public_key {
+ struct bignum *e; /* Public exponent */
+ struct bignum *n; /* Modulus */
+ };
+
+This is also how such keys are stored inside the TEE object attributes
+(``TEE_ATTR_RSA_PUBLIC_KEY`` in this case). ``struct bignum`` is an opaque type,
+known to the underlying implementation only. ``struct bignum_ops`` provides
+functions so that the system services can manipulate data of this type. This
+includes allocation/deallocation, copy, and conversion to or from the big endian
+binary format.
+
+.. code-block:: c
+ :caption: File: core/include/crypto/crypto.h
+
+ struct bignum *crypto_bignum_allocate(size_t size_bits);
+
+ TEE_Result crypto_bignum_bin2bn(const uint8_t *from, size_t fromsize,
+ struct bignum *to);
+
+ void crypto_bignum_bn2bin(const struct bignum *from, uint8_t *to);
+ /*...*/
+
+
+[4] LibTomCrypt
+***************
+Some algorithms may be disabled at compile time if they are not needed, in order
+to reduce the size of the OP-TEE image and reduces its memory usage. This is
+done by setting the appropriate configuration variable. For example:
+
+.. code-block:: bash
+
+ $ make CFG_CRYPTO_AES=n # disable AES only
+ $ make CFG_CRYPTO_{AES,DES}=n # disable symmetric ciphers
+ $ make CFG_CRYPTO_{DSA,RSA,DH,ECC}=n # disable public key algorithms
+ $ make CFG_CRYPTO=n # disable all algorithms
+
+Please refer to `core/lib/libtomcrypt/sub.mk`_ for the list of all supported
+variables.
+
+Note that the application interface is **not** modified when algorithms are
+disabled. This means, for instance, that the functions ``TEE_CipherInit()``,
+``TEE_CipherUpdate()`` and ``TEE_CipherFinal()`` would remain present in
+``libutee.a`` even if all symmetric ciphers are disabled (they would simply
+return ``TEE_ERROR_NOT_IMPLEMENTED``).
+
+Add a new crypto implementation
+*******************************
+To add a new implementation, the default one in `core/lib/libtomcrypt`_ in
+combination with what is in `core/crypto`_ should be used as a reference. Here
+are the main things to consider when adding a new crypto provider:
+
+ - Put all the new code in its own directory under ``core/lib`` unless it is
+ code that will be used regardless of which crypto provider is in use. How
+ we are dealing with AES-GCM in `core/crypto`_ could serve as an example.
+
+ - Avoid modifying tee_svc_cryp.c_. It should not be needed.
+
+ - Although not all crypto families need to be defined, all are required for
+ compliance to the GlobalPlatform specification.
+
+ - If you intend to make some algorithms optional, please try to re-use the
+ same names for configuration variables as the default implementation.
+
+.. Source files
+.. _core/crypto: https://github.com/OP-TEE/optee_os/blob/master/core/crypto
+.. _crypto.c: https://github.com/OP-TEE/optee_os/blob/master/core/crypto/crypto.c
+.. _crypto.h: https://github.com/OP-TEE/optee_os/blob/master/core/include/crypto/crypto.h
+.. _core/lib/libtomcrypt: https://github.com/OP-TEE/optee_os/blob/master/core/lib/libtomcrypt
+.. _core/lib/libtomcrypt/sub.mk: https://github.com/OP-TEE/optee_os/blob/master/core/lib/libtomcrypt/sub.mk
+.. _tee_api_operations.c: https://github.com/OP-TEE/optee_os/blob/master/lib/libutee/tee_api_operations.c
+.. _tee_svc_cryp.c: https://github.com/OP-TEE/optee_os/blob/master/core/tee/tee_svc_cryp.c
+.. _tee_svc_cryp.h: https://github.com/OP-TEE/optee_os/blob/master/core/include/tee/tee_svc_cryp.h
+.. _utee_syscalls.h: https://github.com/OP-TEE/optee_os/blob/master/lib/libutee/include/utee_syscalls.h
+.. _utee_syscalls_asm.S: https://github.com/OP-TEE/optee_os/blob/master/lib/libutee/arch/arm/utee_syscalls_asm.S
+
+.. Other links:
+.. _LibTomCrypt: https://github.com/libtom/libtomcrypt
diff --git a/architecture/device_tree.rst b/architecture/device_tree.rst
new file mode 100644
index 0000000..d3db8c9
--- /dev/null
+++ b/architecture/device_tree.rst
@@ -0,0 +1,76 @@
+.. _device_tree:
+
+###########
+Device Tree
+###########
+OP-TEE core can use the device tree format to inject platform configuration
+information during platform initialization and possibly some run time contexts.
+
+Device Tree technology allows to describe platforms from ASCII source files
+so-called DTS files. These can be used to generate a platform description binary
+image, so-called DTB, embedded in the platform boot media for applying expected
+configuration settings during the platform initializations.
+
+This scheme relaxes design constraints on the OP-TEE core implementation as most
+of the platform specific hardware can be tuned without modifying C source files
+or adding configuration directives in the build environments.
+
+Secure and Non-Secure Device Trees
+**********************************
+There can be several device trees embedded in the target system and some can be
+shared across the boot stages.
+
+ - Boot loader stages may load a device tree structure in memory for all boot
+ stage to get platform configuration from. If such device tree data are to
+ be accessed by the non-secure world, they shall be located in non-secure
+ memory.
+
+ - Boot loader stages may load a device tree structure in secure memory for
+ the benefit of the secure world only. Such device tree blob shall be
+ located in secure memory.
+
+ - OP-TEE core can also embedded a device tree structure to describe the
+ platform.
+
+ - Non-secure world can embed a device tree structure and/or rely on a device
+ tree structure loaded by the secure world, being an early boot stage
+ and/or OP-TEE core.
+
+Obviously the non-secure world will not be able to access a device tree image
+located on a secure memory which non-secure world as no access to.
+
+Early boot device tree argument
+*******************************
+The bootloader provides arguments to the OP-TEE core when it boots it. Among
+those, the physical memory base address of a device tree image accessible to
+OP-TEE core.
+
+When OP-TEE core is built with ``CFG_DT=y`` this device tree is accessed by
+OP-TEE core to get some information: console configuration, main memory size.
+
+OP-TEE will also try to add the description of the OP-TEE resources for the
+non-secure world to properly communicate with OP-TEE. This assumes the image is
+located in non-secure memory.
+
+Modifications made by OP-TEE core on the non-secure device tree image provided
+by early boot and passed to non-secure world are the following:
+
+ - Add an OP-TEE node if none found with the related invocation parameters.
+
+ - Add a reserved memory node for the few memory areas that shall be reserved
+ to the secure world and non accessed by the non-secure world.
+
+ - Add a PSCI description node if none found.
+
+Early boot DTB located in non-secure memory can be accessed by OP-TEE core only
+during its initialization, before non-secure world boots.
+
+Embedded Secure Device Tree
+***************************
+When OP-TEE core is built with configuration directive ``CFG_EMBED_DTB=y``
+directive ``CFG_EMBED_DTB_SOURCE_FILE`` shall provide the relative path of the
+DTS file inside directory ``core/arch/$(ARCH)/dts`` from which a DTB is
+generated and embedded in a read-only section of OP-TEE core.
+
+In this case the device tree address passed to the OP-TEE entry point by the
+bootloader is ignored, only the embedded device tree is accessible.
diff --git a/architecture/file_structure.rst b/architecture/file_structure.rst
new file mode 100644
index 0000000..3cca3a6
--- /dev/null
+++ b/architecture/file_structure.rst
@@ -0,0 +1,186 @@
+.. todo::
+
+ Joakim: Question is whether this should be moved under the "optee_os"
+ specific page instead?
+
+.. _file_structure:
+
+File structure
+##############
+This page describes what different folders in :ref:`optee_os` contains.
+
+Top level directories
+*********************
+.. list-table:: Top level directories
+ :header-rows: 1
+ :widths: 1 5
+
+ * - Directory
+ - Description
+
+ * - /core
+ - Files that are only used building TEE Core
+
+ * - /lib
+ - Files that are used both when building TEE Core and TAs
+
+ * - /ta
+ - Files that are only used when building TAs
+
+ * - /mk
+ - Makefiles supporting the build system
+
+ * - /tmp-stuff
+ - Temporary stuff that will be removed before the final commit is made
+
+ * - /scripts
+ - Helper scripts for miscellaneous tasks
+
+ * - /out
+ - Created when building unless a different out directory is specified with
+ ``O=...`` on the command line
+
+/core
+*****
+.. list-table:: Structure of /core
+ :header-rows: 1
+ :widths: 1 5
+
+ * - Directory
+ - Description
+
+ * - /arch
+ - Architecture and platform specific files
+
+ * - /include
+ - Header files of resources exported by the core
+
+ * - /lib
+ - Generic libraries that are likely to be replaced in a final product
+
+ * - /mm
+ - Generic memory management, currently empty
+
+ * - /tee
+ - Generic TEE files
+
+/core/arch
+**********
+.. list-table:: Structure of /core/arch
+ :header-rows: 1
+ :widths: 1 5
+
+ * - Directory
+ - Description
+
+ * - /arm
+ - ARMv7 and Aarch32 specific architecture and platform specific files
+
+/core/arch/arm
+**************
+.. list-table:: Structure of /core/arch/arm
+ :header-rows: 1
+ :widths: 1 5
+
+ * - Directory
+ - Description
+
+ * - /dts
+ - Device tree source files
+
+ * - /include
+ - Include files used in rest of TEE core but not in any supporting
+ libraries
+
+ * - /kern
+ - Low level and core parts of TEE Core
+
+ * - /mm
+ - Memory management
+
+ * - /tee
+ - TEE files
+
+ * - /sm
+ - Secure Monitor
+
+ * - /plat-foo
+ - Specific files for the ``foo`` platform
+
+/core/arch/arm/include
+**********************
+.. list-table:: Structure of /core/arch/arm/include
+ :header-rows: 1
+ :widths: 1 5
+
+ * - Directory
+ - Description
+
+ * - /kern
+ - Include files exposing API for /core/arch/arm/kern files
+
+ * - /kta
+ - Include files exposing the KTA API that is mainly used by kernel TAs
+
+ * - /mm
+ - Include files exposing API for /core/arch/arm/mm files
+
+ * - /rom
+ - Old ROM files that should be removed before going public
+
+ * - /sm
+ - Include files exposing API for Secure Monitor
+
+/core/include
+*************************
+.. list-table:: Structure of /core/include
+ :header-rows: 1
+ :widths: 1 5
+
+ * - Directory
+ - Description
+
+ * - /drivers
+ - Include files exposing API for /core/drivers files
+
+ * - /dt-bindings
+ - Include files for the device tree bindings
+
+/core/lib/lib{crypto,sla}
+*************************
+.. list-table:: Structure of /core/lib/lib{crypto,sla}
+ :header-rows: 1
+ :widths: 1 5
+
+ * - Directory
+ - Description
+
+ * - /
+ - Source files for the library
+
+ * - /include
+ - Include files exposing the API of the library
+
+/lib/libutils
+*************
+.. list-table:: Structure of /lib/libutils
+ :header-rows: 1
+ :widths: 1 5
+
+ * - Directory
+ - Description
+
+ * - /
+ - Source file for the library
+
+ * - /arch
+ - Architecture specific source files
+
+ * - /arch/arm
+ - ARMv7 and Aarch32 specific source files
+
+ * - /arch/arm/include
+ - ARMv7 and Aarch32 specific include files
+
+ * - /include
+ - Include files exposing the API of the library
diff --git a/architecture/globalplatform_api.rst b/architecture/globalplatform_api.rst
new file mode 100644
index 0000000..9b4d366
--- /dev/null
+++ b/architecture/globalplatform_api.rst
@@ -0,0 +1,797 @@
+.. _globalplatform_api:
+
+##################
+GlobalPlatform API
+##################
+Introduction
+************
+GlobalPlatform_ works across industries to identify, develop and publish
+specifications which facilitate the secure and interoperable deployment and
+management of multiple embedded applications on secure chip technology. OP-TEE
+has support for GlobalPlatform TEE Client API Specification_ v1.0 (GPD_SPE_007)
+and TEE Internal Core API Specification v1.1.2 (GPD_SPE_010).
+
+
+.. _tee_client_api:
+
+TEE Client API
+**************
+The TEE Client API describes and defines how a client running in a rich
+operating environment (REE) should communicate with the TEE. To identify a
+Trusted Application (TA) to be used, the client provides an UUID_. All TA's
+exposes one or several functions. Those functions corresponds to a so called
+``commandID`` which also is sent by the client.
+
+TEE Contexts
+============
+The TEE Context is used for creating a logical connection between the client and
+the TEE. The context must be initialized before the TEE Session can be created.
+When the client has completed a job running in secure world, it should finalize
+the context and thereby also release resources.
+
+TEE Sessions
+============
+Sessions are used to create logical connections between a client and a specific
+Trusted Application. When the session has been established the client has opened
+up the communication channel towards the specified Trusted Application
+identified by the ``UUID``. At this stage the client and the Trusted Application
+can start to exchange data.
+
+
+TEE Client API example / usage
+==============================
+Below you will find the main functions as defined by GlobalPlatform and are used
+in the communication between the client and the TEE.
+
+.. code-block:: c
+
+ TEEC_Result TEEC_InitializeContext(
+ const char* name,
+ TEEC_Context* context)
+
+ void TEEC_FinalizeContext(
+ TEEC_Context* context)
+
+ TEEC_Result TEEC_OpenSession (
+ TEEC_Context* context,
+ TEEC_Session* session,
+ const TEEC_UUID* destination,
+ uint32_t connectionMethod,
+ const void* connectionData,
+ TEEC_Operation* operation,
+ uint32_t* returnOrigin)
+
+ void TEEC_CloseSession (
+ TEEC_Session* session)
+
+ TEEC_Result TEEC_InvokeCommand(
+ TEEC_Session* session,
+ uint32_t commandID,
+ TEEC_Operation* operation,
+ uint32_t* returnOrigin)
+
+In principle the commands are called in this order:
+
+.. code-block:: c
+
+ TEEC_InitializeContext(...)
+ TEEC_OpenSession(...)
+ TEEC_InvokeCommand(...)
+ TEEC_CloseSession(...)
+ TEEC_FinalizeContext(...)
+
+It is not uncommon that ``TEEC_InvokeCommand(...)`` is called several times in
+a row when the session has been established.
+
+For a complete example, please see chapter **5.2 Example 1: Using the TEE Client
+API** in the GlobalPlatform TEE Client API Specification_ v1.0.
+
+
+.. _tee_internal_core_api:
+
+TEE Internal Core API
+*********************
+The Internal Core API is the API that is exposed to the Trusted Applications
+running in the secure world. The TEE Internal API consists of four major parts:
+
+ 1. Trusted Storage API for Data and Keys
+ 2. Cryptographic Operations API
+ 3. Time API
+ 4. Arithmetical API
+
+Examples / usage
+================
+Calling the Internal Core API is done in the same way as described above using
+Client API. The best place to find information how this should be done is in the
+TEE Internal Core API Specification_ v1.1.2 which contains many examples of how
+to call the various APIs. One can also have a look at the examples in the
+optee_examples_ git.
+
+
+.. _extensions:
+
+Extensions
+**********
+In addition to what is stated in :ref:`tee_internal_core_api`, there are some
+non-official extensions in OP-TEE.
+
+Trusted Applications should include header file ``tee_api_defines_extensions.h``
+to import the definitions of the extensions. For each extension, a configuration
+directive prefixed ``CFG_`` allows one to disable support for the extension when
+building the OP-TEE packages.
+
+Cache Maintenance Support
+=========================
+Following functions have been introduced in order to allow Trusted Applications
+to operate with the data cache:
+
+.. code-block:: c
+
+ TEE_Result TEE_CacheClean(char *buf, size_t len);
+ TEE_Result TEE_CacheFlush(char *buf, size_t len);
+ TEE_Result TEE_CacheInvalidate(char *buf, size_t len);
+
+These functions are available to any Trusted Application defined with the flag
+``TA_FLAG_CACHE_MAINTENANCE`` sets on. When not set, each function returns the
+error code ``TEE_ERROR_NOT_SUPPORTED``. Within these extensions, a Trusted
+Application is able to operate on the data cache, with the following
+specification:
+
+.. list-table::
+ :widths: 10 60
+ :header-rows: 1
+
+ * - Function
+ - Description
+
+ * - ``TEE_CacheClean()``
+ - Write back to memory any dirty data cache lines. The line is marked as
+ not dirty. The valid bit is unchanged.
+
+ * - ``TEE_CacheFlush()``
+ - Purges any valid data cache lines. Any dirty cache lines are first
+ written back to memory, then the cache line is invalidated.
+
+ * - ``TEE_CacheInvalidate()``
+ - Invalidate any valid data cache lines. Any dirty line are not written
+ back to memory.
+
+In the following two cases, the error code ``TEE_ERROR_ACCESS_DENIED`` is
+returned:
+
+ - The memory range has not the write access, that is
+ ``TEE_MEMORY_ACCESS_WRITE`` is not set.
+ - The memory is **not** user space memory.
+
+
+You may disable this extension by setting the following configuration variable
+in ``conf.mk``:
+
+.. code-block:: make
+
+ CFG_CACHE_API := n
+
+
+.. _rsassa_na1:
+
+PKCS#1 v1.5 RSASSA without hash OID
+===================================
+This extension adds identifer``TEE_ALG_RSASSA_PKCS1_V1_5`` to allow signing and
+verifying messages with RSASSA-PKCS1-v1_5, in `RFC 3447`_, without including the
+OID of the hash in the signature. You may disable this extension by setting the
+following configuration variable in ``conf.mk``:
+
+.. code-block:: make
+
+ CFG_CRYPTO_RSASSA_NA1 := n
+
+The TEE Internal Core API was extended with a new algorithm descriptor.
+
+.. list-table::
+ :widths: 10 60
+ :header-rows: 1
+
+ * - Algorithm
+ - Possible Modes
+
+ * - TEE_ALG_RSASSA_PKCS1_V1_5
+ - TEE_MODE_SIGN / TEE_MODE_VERIFY
+
+.. list-table::
+ :widths: 10 60
+ :header-rows: 1
+
+ * - Algorithm
+ - Identifier
+
+ * - TEE_ALG_RSASSA_PKCS1_V1_5
+ - 0xF0000830
+
+
+.. _concat_kdf:
+
+Concat KDF
+==========
+Support for the Concatenation Key Derivation Function (Concat KDF) according to
+`SP 800-56A`_ (*Recommendation for Pair-Wise Key Establishment Schemes Using
+Discrete Logarithm Cryptography*) can be found in OP-TEE. You may disable this
+extension by setting the following configuration variable in ``conf.mk``:
+
+.. code-block:: make
+
+ CFG_CRYPTO_CONCAT_KDF := n
+
+**Implementation notes**
+
+All key and parameter sizes **must** be multiples of 8 bits. That is:
+
+ - Input parameters: the shared secret (``Z``) and ``OtherInfo``.
+ - Output parameter: the derived key (``DerivedKeyingMaterial``).
+
+In addition, the maximum size of the derived key is limited by the size of an
+object of type ``TEE_TYPE_GENERIC_SECRET`` (512 bytes). This implementation does
+**not** enforce any requirement on the content of the ``OtherInfo`` parameter.
+It is the application's responsibility to make sure this parameter is
+constructed as specified by the NIST specification if compliance is desired.
+
+**API extension**
+
+To support Concat KDF, the :ref:`tee_internal_core_api` v1.1 was extended with
+new algorithm descriptors, new object types, and new object attributes as
+described below.
+
+**p.95 Add new object type to TEE_PopulateTransientObject**
+
+The following entry shall be added to **Table 5-8**:
+
+.. list-table::
+ :widths: 10 60
+ :header-rows: 1
+
+ * - Object type
+ - Parts
+
+ * - TEE_TYPE_CONCAT_KDF_Z
+ - The ``TEE_ATTR_CONCAT_KDF_Z`` part (input shared secret) must be
+ provided.
+
+**p.121 Add new algorithms for TEE_AllocateOperation**
+
+The following entry shall be added to **Table 6-3**:
+
+.. list-table::
+ :widths: 10 60
+ :header-rows: 1
+
+ * - Algorithm
+ - Possible Modes
+
+ * - TEE_ALG_CONCAT_KDF_SHA1_DERIVE_KEY
+ TEE_ALG_CONCAT_KDF_SHA224_DERIVE_KEY
+ TEE_ALG_CONCAT_KDF_SHA256_DERIVE_KEY
+ TEE_ALG_CONCAT_KDF_SHA384_DERIVE_KEY
+ TEE_ALG_CONCAT_KDF_SHA512_DERIVE_KEY
+ TEE_ALG_CONCAT_KDF_SHA512_DERIVE_KEY
+ - TEE_MODE_DERIVE
+
+**p.126 Explain usage of HKDF algorithms in TEE_SetOperationKey**
+
+In the bullet list about operation mode, the following shall be added:
+
+ - For the Concat KDF algorithms, the only supported mode is
+ ``TEE_MODE_DERIVE``.
+
+**p.150 Define TEE_DeriveKey input attributes for new algorithms**
+
+The following sentence shall be deleted:
+
+.. code-block:: none
+
+ The TEE_DeriveKey function can only be used with the algorithm
+ TEE_ALG_DH_DERIVE_SHARED_SECRET.
+
+The following entry shall be added to **Table 6-7**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Algorithm
+ - Possible operation parameters
+
+ * - TEE_ALG_CONCAT_KDF_SHA1_DERIVE_KEY
+ TEE_ALG_CONCAT_KDF_SHA224_DERIVE_KEY
+ TEE_ALG_CONCAT_KDF_SHA256_DERIVE_KEY
+ TEE_ALG_CONCAT_KDF_SHA384_DERIVE_KEY
+ TEE_ALG_CONCAT_KDF_SHA512_DERIVE_KEY
+ TEE_ALG_CONCAT_KDF_SHA512_DERIVE_KEY
+ - TEE_ATTR_CONCAT_KDF_DKM_LENGTH: up to 512 bytes. This parameter is
+ mandatory: TEE_ATTR_CONCAT_KDF_OTHER_INFO
+
+**p.152 Add new algorithm identifiers**
+
+The following entries shall be added to **Table 6-8**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Algorithm
+ - Identifier
+
+ * - TEE_ALG_CONCAT_KDF_SHA1_DERIVE_KEY
+ - 0x800020C1
+
+ * - TEE_ALG_CONCAT_KDF_SHA224_DERIVE_KEY
+ - 0x800030C1
+
+ * - TEE_ALG_CONCAT_KDF_SHA256_DERIVE_KEY
+ - 0x800040C1
+
+ * - TEE_ALG_CONCAT_KDF_SHA384_DERIVE_KEY
+ - 0x800050C1
+
+ * - TEE_ALG_CONCAT_KDF_SHA512_DERIVE_KEY
+ - 0x800060C1
+
+**p.154 Define new main algorithm**
+
+In **Table 6-9** in section 6.10.1, a new value shall be added to the value
+column for row bits ``[7:0]``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Bits
+ - Function
+ - Value
+
+ * - Bits [7:0]
+ - Identifiy the main underlying algorithm itself
+ - ...
+
+ 0xC1: Concat KDF
+
+The function column for ``bits[15:12]`` shall also be modified to read:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Bits
+ - Function
+ - Value
+
+ * - Bits [15:12]
+ - Define the message digest for asymmetric signature algorithms or Concat KDF
+ -
+
+**p.155 Add new object type for Concat KDF input shared secret**
+
+The following entry shall be added to **Table 6-10**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Identifier
+ - Possible sizes
+
+ * - TEE_TYPE_CONCAT_KDF_Z
+ - 0xA10000C1
+ - 8 to 4096 bits (multiple of 8)
+
+**p.156 Add new operation attributes for Concat KDF**
+
+The following entries shall be added to **Table 6-11**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Value
+ - Protection
+ - Type
+ - Comment
+
+ * - TEE_ATTR_CONCAT_KDF_Z
+ - 0xC00001C1
+ - Protected
+ - Ref
+ - The shared secret (``Z``)
+
+ * - TEE_ATTR_CONCAT_KDF_OTHER_INFO
+ - 0xD00002C1
+ - Public
+ - Ref
+ - ``OtherInfo``
+
+ * - TEE_ATTR_CONCAT_KDF_DKM_LENGTH
+ - 0xF00003C1
+ - Public
+ - Value
+ - The length (in bytes) of the derived keying material to be generated,
+ maximum 512. This is ``KeyDataLen`` / 8.
+
+
+.. _hkdf:
+
+HKDF
+====
+OP-TEE implements the *HMAC-based Extract-and-Expand Key Derivation Function
+(HKDF)* as specified in `RFC 5869`_. This file documents the extensions to the
+:ref:`tee_internal_core_api` v1.1 that were implemented to support this
+algorithm. Trusted Applications should include
+``<tee_api_defines_extensions.h>`` to import the definitions.
+
+Note that the implementation follows the recommendations of version 1.1 of the
+specification for adding new algorithms. It should make it compatible with
+future changes to the official specification. You can disable this extension by
+setting the following in ``conf.mk``:
+
+.. code-block:: make
+
+ CFG_CRYPTO_HKDF := n
+
+**p.95 Add new object type to TEE_PopulateTransientObject**
+
+The following entry shall be added to **Table 5-8**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Object type
+ - Parts
+
+ * - TEE_TYPE_HKDF_IKM
+ - The TEE_ATTR_HKDF_IKM (Input Keying Material) part must be provided.
+
+**p.121 Add new algorithms for TEE_AllocateOperation**
+
+The following entry shall be added to **Table 6-3**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Algorithm
+ - Possible Modes
+
+ * - TEE_ALG_HKDF_MD5_DERIVE_KEY
+ TEE_ALG_HKDF_SHA1_DERIVE_KEY
+ TEE_ALG_HKDF_SHA224_DERIVE_KEY
+ TEE_ALG_HKDF_SHA256_DERIVE_KEY
+ TEE_ALG_HKDF_SHA384_DERIVE_KEY
+ TEE_ALG_HKDF_SHA512_DERIVE_KEY
+ TEE_ALG_HKDF_SHA512_DERIVE_KEY
+ - TEE_MODE_DERIVE
+
+**p.126 Explain usage of HKDF algorithms in TEE_SetOperationKey**
+
+In the bullet list about operation mode, the following shall be added:
+
+ - For the HKDF algorithms, the only supported mode is TEE_MODE_DERIVE.
+
+**p.150 Define TEE_DeriveKey input attributes for new algorithms**
+
+The following sentence shall be deleted:
+
+.. code-block:: none
+
+ The TEE_DeriveKey function can only be used with the algorithm
+ TEE_ALG_DH_DERIVE_SHARED_SECRET
+
+The following entry shall be added to **Table 6-7**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Algorithm
+ - Possible operation parameters
+
+ * - TEE_ALG_HKDF_MD5_DERIVE_KEY
+ TEE_ALG_HKDF_SHA1_DERIVE_KEY
+ TEE_ALG_HKDF_SHA224_DERIVE_KEY
+ TEE_ALG_HKDF_SHA256_DERIVE_KEY
+ TEE_ALG_HKDF_SHA384_DERIVE_KEY
+ TEE_ALG_HKDF_SHA512_DERIVE_KEY
+ TEE_ALG_HKDF_SHA512_DERIVE_KEY
+ - TEE_ATTR_HKDF_OKM_LENGTH: Number of bytes in the Output Keying Material
+
+ TEE_ATTR_HKDF_SALT (optional) Salt to be used during the extract step
+
+ TEE_ATTR_HKDF_INFO (optional) Info to be used during the expand step
+
+**p.152 Add new algorithm identifiers**
+
+The following entries shall be added to **Table 6-8**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Algorithm
+ - Identifier
+
+ * - TEE_ALG_HKDF_MD5_DERIVE_KEY
+ - 0x800010C0
+
+ * - TEE_ALG_HKDF_SHA1_DERIVE_KEY
+ - 0x800020C0
+
+ * - TEE_ALG_HKDF_SHA224_DERIVE_KEY
+ - 0x800030C0
+
+ * - TEE_ALG_HKDF_SHA256_DERIVE_KEY
+ - 0x800040C0
+
+ * - TEE_ALG_HKDF_SHA384_DERIVE_KEY
+ - 0x800050C0
+
+ * - TEE_ALG_HKDF_SHA512_DERIVE_KEY
+ - 0x800060C0
+
+## p.154 Define new main algorithm
+
+In **Table 6-9** in section 6.10.1, a new value shall be added to the value column
+for row ``bits [7:0]``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Bits
+ - Function
+ - Value
+
+ * - Bits [7:0]
+ - Identifiy the main underlying algorithm itself
+ - ...
+
+ 0xC0: HKDF
+
+The function column for ``bits[15:12]`` shall also be modified to read:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Bits
+ - Function
+ - Value
+
+ * - Bits [15:12]
+ - Define the message digest for asymmetric signature algorithms or HKDF
+ -
+
+**p.155 Add new object type for HKDF input keying material**
+
+The following entry shall be added to **Table 6-10**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Identifier
+ - Possible sizes
+
+ * - TEE_TYPE_HKDF_IKM
+ - 0xA10000C0
+ - 8 to 4096 bits (multiple of 8)
+
+**p.156 Add new operation attributes for HKDF salt and info**
+
+The following entries shall be added to **Table 6-11**:
+
+.. list-table::
+ :widths: 40 10 10 10 40
+ :header-rows: 1
+
+ * - Name
+ - Value
+ - Protection
+ - Type
+ - Comment
+
+ * - TEE_ATTR_HKDF_IKM
+ - 0xC00001C0
+ - Protected
+ - Ref
+ -
+
+ * - TEE_ATTR_HKDF_SALT
+ - 0xD00002C0
+ - Public
+ - Ref
+ -
+
+ * - TEE_ATTR_HKDF_INFO
+ - 0xD00003C0
+ - Public
+ - Ref
+ -
+
+ * - TEE_ATTR_HKDF_OKM_LENGTH
+ - 0xF00004C0
+ - Public
+ - Value
+ -
+
+.. _pbkdf2:
+
+PBKDF2
+======
+This document describes the OP-TEE implementation of the key derivation
+function, *PBKDF2* as specified in `RFC 2898`_ section 5.2. This RFC is a
+republication of PKCS #5 v2.0 from RSA Laboratories' Public-Key Cryptography
+Standards (PKCS) series. You may disable this extension by setting the following
+configuration variable in ``conf.mk``:
+
+.. code-block:: make
+
+ CFG_CRYPTO_PBKDF2 := n
+
+**API extension**
+
+To support PBKDF2, the :ref:`tee_internal_core_api` v1.1 was extended with a new
+algorithm descriptor, new object types, and new object attributes as described
+below.
+
+**p.95 Add new object type to TEE_PopulateTransientObject**
+
+The following entry shall be added to **Table 5-8**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Object type
+ - Parts
+
+ * - TEE_TYPE_PBKDF2_PASSWORD
+ - The TEE_ATTR_PBKDF2_PASSWORD part must be provided.
+
+**p.121 Add new algorithms for TEE_AllocateOperation**
+
+The following entry shall be added to **Table 6-3**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Algorithm
+ - Possible Modes
+
+ * - TEE_ALG_PBKDF2_HMAC_SHA1_DERIVE_KEY
+ - TEE_MODE_DERIVE
+
+**p.126 Explain usage of PBKDF2 algorithm in TEE_SetOperationKey**
+
+In the bullet list about operation mode, the following shall be added:
+
+ - For the PBKDF2 algorithm, the only supported mode is TEE_MODE_DERIVE.
+
+**p.150 Define TEE_DeriveKey input attributes for new algorithms**
+
+The following sentence shall be deleted:
+
+.. code-block:: none
+
+ The TEE_DeriveKey function can only be used with the algorithm
+ TEE_ALG_DH_DERIVE_SHARED_SECRET
+
+The following entry shall be added to **Table 6-7**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Algorithm
+ - Possible operation parameters
+
+ * - TEE_ALG_PBKDF2_HMAC_SHA1_DERIVE_KEY
+ - TEE_ATTR_PBKDF2_DKM_LENGTH: up to 512 bytes. This parameter is
+ mandatory.
+
+ TEE_ATTR_PBKDF2_SALT
+
+ TEE_ATTR_PBKDF2_ITERATION_COUNT: This parameter is mandatory.
+
+**p.152 Add new algorithm identifiers**
+
+The following entries shall be added to **Table 6-8**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Algorithm
+ - Identifier
+
+ * - TEE_ALG_PBKDF2_HMAC_SHA1_DERIVE_KEY
+ - 0x800020C2
+
+**p.154 Define new main algorithm**
+
+In **Table 6-9** in section 6.10.1, a new value shall be added to the value
+column for row ``bits [7:0]``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Bits
+ - Function
+ - Value
+
+ * - Bits [7:0]
+ - Identifiy the main underlying algorithm itself
+ - ...
+
+ 0xC2: PBKDF2
+
+The function column for ``bits[15:12]`` shall also be modified to read:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Bits
+ - Function
+ - Value
+
+ * - Bits [15:12]
+ - Define the message digest for asymmetric signature algorithms or PBKDF2
+ -
+
+**p.155 Add new object type for PBKDF2 password**
+
+The following entry shall be added to **Table 6-10**:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Identifier
+ - Possible sizes
+
+ * - TEE_TYPE_PBKDF2_PASSWORD
+ - 0xA10000C2
+ - 8 to 4096 bits (multiple of 8)
+
+**p.156 Add new operation attributes for Concat KDF**
+
+The following entries shall be added to **Table 6-11**:
+
+.. list-table::
+ :widths: 40 10 10 10 40
+ :header-rows: 1
+
+ * - Name
+ - Value
+ - Protection
+ - Type
+ - Comment
+
+ * - TEE_ATTR_PBKDF2_PASSWORD
+ - 0xC00001C2
+ - Protected
+ - Ref
+ -
+
+ * - TEE_ATTR_PBKDF2_SALT
+ - 0xD00002C2
+ - Public
+ - Ref
+ -
+
+ * - TEE_ATTR_PBKDF2_ITERATION_COUNT
+ - 0xF00003C2
+ - Public
+ - Value
+ -
+
+ * - TEE_ATTR_PBKDF2_DKM_LENGTH
+ - 0xF00004C2
+ - Public
+ - Value
+ - The length (in bytes) of the derived keying material to be generated,
+ maximum 512.
+
+
+.. _GlobalPlatform: https://globalplatform.org
+.. _optee_examples: https://github.com/linaro-swg/optee_examples
+.. _TZC-400: http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0504c/index.html
+.. _RFC 2898: https://www.ietf.org/rfc/rfc2898.txt
+.. _RFC 3447: https://tools.ietf.org/html/rfc3447#section-8.2
+.. _RFC 5869: https://tools.ietf.org/html/rfc5869
+.. _Specification: https://globalplatform.org/specs-library/?filter-committee=tee
+.. _SP 800-56A: http://csrc.nist.gov/publications/nistpubs/800-56A/SP800-56A_Revision1_Mar08-2007.pdf
+.. _UUID: https://en.wikipedia.org/wiki/Universally_unique_identifier
diff --git a/architecture/index.rst b/architecture/index.rst
new file mode 100644
index 0000000..62713bd
--- /dev/null
+++ b/architecture/index.rst
@@ -0,0 +1,17 @@
+Architecture
+############
+
+.. toctree::
+ :maxdepth: 2
+
+ core
+ Crypto <crypto>
+ device_tree
+ file_structure
+ globalplatform_api
+ libraries
+ porting_guidelines
+ secure_boot
+ secure_storage
+ trusted_applications
+ virtualization
diff --git a/architecture/libraries.rst b/architecture/libraries.rst
new file mode 100644
index 0000000..664941b
--- /dev/null
+++ b/architecture/libraries.rst
@@ -0,0 +1,27 @@
+.. _libraries:
+
+#########
+Libraries
+#########
+
+.. _libutee:
+
+libutee
+*******
+The :ref:`tee_internal_core_api` describes services that are provided to Trusted
+Applications. **libutee** is a library that implements this API.
+
+libutee is a static library the Trusted Applications shall statically link
+against. Trusted Applications do execute in non-privileged secure userspace and
+libutee also aims at being executed in the non-privileged secure userspace.
+
+Some services for this API are fully statically implemented inside the libutee
+library while some services for the API are implemented inside the OP-TEE core
+(privileged level) and libutee calls such services through system calls.
+
+.. _libmpa:
+
+libmpa
+******
+Now deprectated, used to the the BigNum library in OP-TEE.
+
diff --git a/architecture/porting_guidelines.rst b/architecture/porting_guidelines.rst
new file mode 100644
index 0000000..0744064
--- /dev/null
+++ b/architecture/porting_guidelines.rst
@@ -0,0 +1,408 @@
+.. _porting_guidelines:
+
+##################
+Porting guidelines
+##################
+This document serves a dual purpose:
+
+* Serve as a base for getting OP-TEE up and running on a new device with initial
+ xtest validation passing. This is the first part of this document (section 2).
+
+* Highlight the missing pieces if you intend to make a real secure product, that
+ is what the second part of this document is about.
+
+We are trying our best to implement full end to end security in OP-TEE in a
+generic way, but due to the nature of devices being different, NDA etc, it is
+not always possible for us to do so and in those cases, we most often try to
+write a generic API, but we will just stub the code. This porting guideline
+highlights the missing pieces that must be addressed in a real secure consumer
+device. Hopefully we will sooner or later get access to devices where we at
+least can make reference implementations publicly available to everyone for the
+missing pieces we are talking about here.
+
+.. _add_a_new_platform:
+
+Add a new platform
+******************
+The first thing you need to do after you have decided to port OP-TEE to another
+device is to add a new platform device. That can either be adding a new platform
+variant (``PLATFORM_FLAVOR``) if it is a device from a family already supported,
+or it can be a brand new platform family (``PLATFORM``). Typically this initial
+setup involve configuring UART, memory addresses etc. For simplicity let us call
+our fictive platform for "gendev" just so we have something to refer to when
+writing examples further down.
+
+core/arch/arm
+=================
+In ``core/arch/arm`` you will find all the currently supported devices. That is
+where you are supposed to add a new platform or modify an existing one.
+Typically you will find this set of files in a specific platform folder:
+
+.. code-block:: bash
+
+ $ ls
+ conf.mk main.c platform_config.h sub.mk
+
+So for the gendev platform it means that the files should be placed in this
+folder:
+
+.. code-block:: bash
+
+ core/arch/arm/plat-gendev
+
+**conf.mk**
+
+This is the device specific makefile where you define configurations unique to
+your platform. This mainly comprises two things: - OP-TEE configuration
+variables (``CFG_``), which may be assigned values in two ways. ``CFG_FOO ?=
+bar`` should be used to provide a default value that may be modified at compile
+time. On the other hand, variables that must be set to some value and cannot be
+modified should be set by: ``$(call force,CFG_FOO,bar)``. - Compiler flags for
+the TEE core, the user mode libraries and the Trusted Applications, which may be
+added to macros used by the build system. Please see `Platform-specific
+configuration and flags`_ in the build system documentation.
+
+It is recommended to use a existing platform configuration file as a starting
+point. For instance, `core/arch/arm/plat-hikey/conf.mk`_.
+
+The platform ``conf.mk`` file should at least define the default platform flavor
+for the platform, the core configurations (architecture and number of cores),
+the main configuration directives (generic boot, arm trusted firmware support,
+generic time source, console driver, etc...) and some platform default
+configuration settings.
+
+.. code-block:: make
+
+ PLATFORM_FLAVOR ?= hikey
+
+ include core/arch/arm/cpu/cortex-armv8-0.mk
+
+ $(call force,CFG_TEE_CORE_NB_CORE,8)
+ $(call force,CFG_GENERIC_BOOT,y)
+ $(call force,CFG_PL011,y)
+ $(call force,CFG_PM_STUBS,y)
+ $(call force,CFG_SECURE_TIME_SOURCE_CNTPCT,y)
+ $(call force,CFG_WITH_ARM_TRUSTED_FW,y)
+ $(call force,CFG_WITH_LPAE,y)
+
+ ta-targets = ta_arm32
+ ta-targets += ta_arm64
+
+ CFG_NUM_THREADS ?= 8
+ CFG_CRYPTO_WITH_CE ?= y
+ CFG_WITH_STACK_CANARIES ?= y
+ CFG_CONSOLE_UART ?= 3
+ CFG_DRAM_SIZE_GB ?= 2
+
+**main.c**
+
+This platform specific file will contain power management handlers and code
+related to the UART. We will talk more about the information related to the
+handlers further down in this document. For our gendev device it could look like
+this (here we are excluding the necessary license header to save some space):
+
+.. code-block:: c
+
+ #include <console.h>
+ #include <drivers/serial8250_uart.h>
+ #include <kernel/generic_boot.h>
+ #include <kernel/panic.h>
+ #include <kernel/pm_stubs.h>
+ #include <mm/core_mmu.h>
+ #include <platform_config.h>
+ #include <stdint.h>
+ #include <tee/entry_fast.h>
+ #include <tee/entry_std.h>
+
+ static void main_fiq(void)
+ {
+ panic();
+ }
+
+ static const struct thread_handlers handlers = {
+ .std_smc = tee_entry_std,
+ .fast_smc = tee_entry_fast,
+ .nintr = main_fiq,
+ .cpu_on = cpu_on_handler,
+ .cpu_off = pm_do_nothing,
+ .cpu_suspend = pm_do_nothing,
+ .cpu_resume = pm_do_nothing,
+ .system_off = pm_do_nothing,
+ .system_reset = pm_do_nothing,
+ };
+
+ const struct thread_handlers *generic_boot_get_handlers(void)
+ {
+ return &handlers;
+ }
+
+ /*
+ * Register the physical memory area for peripherals etc. Here we are
+ * registering the UART console.
+ */
+ register_phys_mem(MEM_AREA_IO_NSEC, CONSOLE_UART_BASE, SERIAL8250_UART_REG_SIZE);
+
+ static struct serial8250_uart_data console_data;
+
+ void console_init(void)
+ {
+ serial8250_uart_init(&console_data, CONSOLE_UART_BASE,
+ CONSOLE_UART_CLK_IN_HZ, CONSOLE_BAUDRATE);
+ register_serial_console(&console_data.chip);
+ }
+
+**platform_config.h**
+
+This is a mandatory header file for every platform, since there are several
+files relaying upon the existence of this particular file. This file is where
+you will find the major differences between different platforms, since this is
+where you do the memory configuration, define base addresses etc. we are going
+to list a few here, but it probably makes more sense to have a look at the
+already existing ``platform_config.h`` files for the other platforms. Our
+fictive gendev could look like this:
+
+.. code-block:: c
+
+ #ifndef PLATFORM_CONFIG_H
+ #define PLATFORM_CONFIG_H
+
+ /* Make stacks aligned to data cache line length */
+ #define STACK_ALIGNMENT 64
+
+ /* 8250 UART */
+ #define CONSOLE_UART_BASE 0xcafebabe /* UART0 */
+ #define CONSOLE_BAUDRATE 115200
+ #define CONSOLE_UART_CLK_IN_HZ 19200000
+
+ /* Optional: when used with CFG_WITH_PAGER, defines the device SRAM */
+ #define TZSRAM_BASE 0x3F000000
+ #define TZSRAM_SIZE (200 * 1024)
+
+ /* Mandatory main secure RAM usually DDR */
+ #define TZDRAM_BASE 0x60000000
+ #define TZDRAM_SIZE (32 * 1024 * 1024)
+
+ /* Mandatory TEE RAM location and core load address */
+ #define TEE_RAM_START TZDRAM_BASE
+ #define TEE_RAM_PH_SIZE TEE_RAM_VA_SIZE
+ #define TEE_RAM_VA_SIZE (4 * 1024 * 1024)
+ #define TEE_LOAD_ADDR (TZDRAM_BASE + 0x20000)
+
+ /* Mandatory TA RAM (external less secure RAM) */
+ #define TA_RAM_START (TZDRAM_BASE + TEE_RAM_VA_SIZE)
+ #define TA_RAM_SIZE (TZDRAM_SIZE - TEE_RAM_VA_SIZE)
+
+ /* Mandatory: for static SHM, need a hardcoded physical address */
+ #define TEE_SHMEM_START 0x08000000
+ #define TEE_SHMEM_SIZE (4 * 1024 * 1024)
+
+ #endif /* PLATFORM_CONFIG_H */
+
+This is minimal amount of information in the ``platform_config.h`` file. I.e,
+the memory layout for on-chip and external RAM. Note that parts of the DDR
+typically will need to be shared with normal world, so there is need for some
+kind of memory firewall for this (more about that further down). As you can see
+we have also added the UART configuration here, i.e., the ``DEVICE0_xyz`` part.
+
+Official board support in OP-TEE?
+=================================
+We do encourage everyone to submit their board support to the OP-TEE project
+itself, so it becomes part of the official releases and will be maintained by
+the OP-TEE community itself. If you intend to do so, then there are a few more
+things that you are supposed to do.
+
+**Update platforms supported**
+
+There is a section at the :ref:`platforms_supported` page that lists all devices
+officially supported in OP-TEE, that is where you also shall list your device.
+It should contain the name of the platform, then composite ``PLATFORM`` flag and
+whether the device is publicly available or not. If there is a product page on
+the internet for the device, please also create a link when writing the device
+name.
+
+**Update .shippable.yml**
+
+Since we are using Shippable to test pull requests etc, we would like that you
+also add your device to the `.shippable.yml
+<https://github.com/OP-TEE/optee_os/blob/master/.shippable.yml>`_ file, so that
+it will at least be built when someone is doing a pull request. Add a line at
+the end of file:
+
+.. code-block:: xml
+
+ - _make PLATFORM=<platform-name>_
+
+**Maintainer**
+
+If you are submitting the board support upstream and cannot give Linaro
+maintainers a device, then we are going to ask you to become the maintainer for
+the device you have added. This means that you should also update the
+MAINTAINERS.md_ file accordingly. By being a maintainer for a device you are
+responsible to keep it up to date and you will be asked every quarter as part of
+the OP-TEE release schedule to test your device running the latest OP-TEE
+software.
+
+**Update build.git and manifest.git**
+
+This isn't strictly necessary, but we are trying to create and maintain OP-TEE
+developer builds that should make it easy to setup, build and deploy OP-TEE on
+various devices. We encourage all maintainers to do the same for the boards they
+are in charge of. Therefore please consider creating a new :ref:`manifest` (and
+a new ``*.mk`` in :ref:`build`) for the device you have added to OP-TEE.
+
+.. _hardware_unique_key:
+
+Hardware Unique Key
+*******************
+Most devices have some kind of Hardware Unique Key (HUK) that is mainly used to
+derive other keys. The HUK could for example be used when deriving keys used in
+secure storage etc. The important thing with the HUK is that it needs to be well
+protected and in the best case the HUK should never ever be readable directly
+from software, not even from the secure side. There are different solutions to
+this, crypto accelerator might have support for it or, it could involve another
+secure co-processor.
+
+In OP-TEE the HUK **is** just **stubbed** and you will see that in the function
+called ``tee_otp_get_hw_unique_key(...)`` in
+`core/include/kernel/tee_common_otp.h`_. In a real secure product you **must**
+replace this with something else. If your device lacks the hardware support for
+a HUK, then you must at least change this to something else than just zeroes.
+But, remember it is not good secure practice to store a key in software,
+especially not the key that is the root for everything else, so this is not
+something we recommend that you should do.
+
+Secure Clock
+************
+The Time API in GlobalPlatform Internal Core API specification defines three
+sources of time; system time, TA persistent time and REE time. The REE time is
+by nature considered as an unsecure source of time, but the other two should in
+a fully trustable hardware make use of trustable source of time, i.e., a secure
+clock. Note that from GlobalPlatform point of view it is not required to make
+use of a secure clock, i.e., it is OK to use time from REE, but the level of
+trust should be reflected by the ``gpd.tee.systemTime.protectionLevel`` property
+and the ``gpd.tee.TAPersistentTime.protectionLevel`` property (100=REE
+controlled clock, 1000=TEE controlled clock). So the functions that one needs to
+pay attention to are ``tee_time_get_sys_time(...)`` and
+``tee_time_get_ta_time(...)``. If your hardware has a secure clock, then you
+probably want to change the implementation there to instead use the secure clock
+(and then you would also need to update the property accordingly, i.e.,
+``tee_time_get_sys_time_protection_level()`` and the variable
+``ta_time_prot_lvl`` in ``tee_svc.c``).
+
+Root and Chain of Trust
+***********************
+To be able to assure that your devices are running the (untampered) binaries you
+intended to run you will need to establish some kind of trust anchor on the
+devices.
+
+The most common way of doing that is to put the root public key in some read
+only memory on the device. Quite often SoC's/OEM's stores public key(s) directly
+or the hash(es) of the public key(s) in OTP_. When the boot ROM (which indeed
+needs to be ROM) is about to load the first stage bootloader it typically reads
+the public key from the software binary itself, hash the key and compare it to
+the key in OTP_. If they are matching, then the boot ROM can be sure that the
+first stage bootloader was indeed signed with the corresponding private key.
+
+In OP-TEE you will not find any code at all related to this and this is a good
+example when it is hard for us to do this in a generic way since device
+manufacturers all tend to do this in their own unique way and they are not very
+keen on sharing their low level boot details and security implementation with
+the rest of the world. This is especially true on ARMv7-A. For ARMv8-A it looks
+bit better, since Arm in Trusted Firmware A have implemented and defined how a
+abstract the chain of trust (see auth-framework.rst_).
+We have successfully verified OP-TEE by using the authentication framework from
+Trusted Firmware A (see :ref:`secure_boot` for the details).
+
+Hardware Crypto IP
+******************
+By default OP-TEE uses a software crypto library (currently mbed TLS and
+LibTomCrypt) and you have the ability to enable Crypto Extensions that were
+introduced with ARMv8-A (if the device is capable of that). Some of the devices
+we have in our hands do have hardware crypto IP's, but due to NDA's etc it has
+not been possible to enable it. If you have a device capable of doing crypto
+operations on a dedicated crypto block and you prefer to use that in favor for
+the software implementation, then you will need to implement relevant functions
+defined in `core/include/crypto/crypto.h`_, the Crypto API, and write the low
+level driver that communicates with the device. Our
+:ref:`cryptographic_implementation` page describes in detail how the Crypto API
+is integrated. Since the communication with crypto blocks tends to be quite
+different depending on what kind of crypto IP you have, we have not written
+how that should be done. It might be that we do that in the future when get hold
+of a device where we can use the crypto block.
+
+By default OP-TEE is configured with a software PRNG. The entropy is added to
+software PRNG at various places, but unfortunately it is still quite easy to
+predict the data added as entropy. As a consequence, unless the RNG is based on
+hardware the generated random will be quite weak.
+
+Power Management / PSCI
+***********************
+In the :ref:`add_a_new_platform` section where we talked about the file
+``main.c``, we added a couple of handlers related to power management, we are
+talking about the following lines:
+
+.. code-block:: c
+
+ .cpu_on = cpu_on_handler,
+ .cpu_off = pm_do_nothing,
+ .cpu_suspend = pm_do_nothing,
+ .cpu_resume = pm_do_nothing,
+ .system_off = pm_do_nothing,
+ .system_reset = pm_do_nothing,
+
+The only function that actually does something there is the ``cpu_on`` function,
+the rest of them are stubbed. The main reason for that is because we think that
+how to suspend and resume is a device dependent thing. The code in OP-TEE is
+prepared so that callbacks etc from Trusted Firmware A will be routed to OP-TEE,
+but since the function(s) are just stubbed we will not do anything and just
+return. In a real production device, you would probably want to save and restore
+CPU states, secure hardware IPs' registers and TZASC and other memory firewall
+related setting when these callbacks are being called.
+
+Memory firewalls / TZASC
+************************
+Arm have defined a system IP / SoC peripheral called TrustZone Address Space
+Controller (TZASC, see TZC-380_ and TZC-400_). TZASC can be used to configure
+DDR memory into separate regions in the physcial address space, where each
+region can have an individual security level setting. After enabling TZASC, it
+will perform security checks on transactions to memory or peripherals. It is not
+always the case that TZASC is on a device, in some cases the SoC has developed
+something equivalent. In OP-TEE this is very well reflected, i.e., different
+platforms have different ways of protecting their memory. On ARMv8-A platforms
+we are in most of the cases using Trusted Firmware A as the boot firmware and
+there the secure bootloader is the one that configures secure vs non-secure
+memory using TZASC (see plat_arm_security_setup_ in TF-A). The takeaway here is
+that you must make sure that you have configured whatever memory firewall your
+device has such that it has a secure and a non-secure memory area.
+
+.. _core_pub_priv_keypair:
+
+Trusted Application private/public keypair
+******************************************
+By default all Trusted Applications (TA's) are signed with the pre-generated
+2048-bit RSA development key (private key). This key is located in the ``keys``
+folder (in the root of optee_os.git) and is named ``default_ta.pem``. This key
+**must** be replaced with your own key and you should **never ever** check-in
+this private key in the source code tree when in use in a real product. The
+recommended way to store private keys is to use some kind of HSM_ (Hardware
+Security Module), but an alternative would be temporary put the private key on a
+computer considered as secure when you are about to sign TA's intended to be
+used in real products. Typically it is only a few number of people having access
+to this type of key in company. The key handling in OP-TEE is currently a bit
+limited since we only support a single key which is used for all TA's. We have
+plans on extending this to make it a bit more flexible. Exactly when that will
+happen has not been decided yet.
+
+.. _core/arch/arm/plat-hikey/conf.mk: https://github.com/OP-TEE/optee_os/blob/master/core/arch/arm/plat-hikey/conf.mk
+.. _core/include/crypto/crypto.h: https://github.com/OP-TEE/optee_os/blob/master/core/include/crypto/crypto.h
+.. _core/include/kernel/tee_common_otp.h: https://github.com/OP-TEE/optee_os/blob/master/core/include/kernel/tee_common_otp.h
+
+
+.. _auth-framework.rst: https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/auth-framework.rst
+.. _HSM: https://en.wikipedia.org/wiki/Hardware_security_module
+.. _MAINTAINERS.md: https://github.com/OP-TEE/optee_os/blob/master/MAINTAINERS
+.. _OTP: https://en.wikipedia.org/wiki/Programmable_read-only_memory
+.. _plat_arm_security_setup: https://github.com/ARM-software/arm-trusted-firmware/search?utf8=%E2%9C%93&q=plat_arm_security_setup&type=
+.. _Platform-specific configuration and flags: build_system.md#platform-specific-configuration-and-flags
+.. _TZC-380: http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0431c/index.html
+.. _TZC-400: http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.100325_0001_02_en/index.html
diff --git a/architecture/secure_boot.rst b/architecture/secure_boot.rst
new file mode 100644
index 0000000..ea10f20
--- /dev/null
+++ b/architecture/secure_boot.rst
@@ -0,0 +1,50 @@
+.. _secure_boot:
+
+###########
+Secure boot
+###########
+
+Armv8-A - Using the authentication framework in TF-A
+****************************************************
+This section gives a brief description on how to enable the verification of
+OP-TEE using the authentication framework in Trusted Firmware A (TF-A), i.e.,
+something that could be used in an Armv8-A environment.
+
+According to user-guide.rst_, there is no additional specific build options for
+the verification of OP-TEE. If we have enabled the authentication framework and
+specified the ``BL32`` build option when building TF-A, the BL32 related
+certificates will be created automatically by the cert_create tool, and then
+these certificates will be verified during booting up.
+
+To enable the authentication framework, the following steps should be followed
+according to user-guide.rst_. For more details about the authentication
+framework, please see auth-framework.rst_ and trusted-board-boot.rst_.
+
+ - Check out a recent version of the `mbed TLS`_ repository and then switch
+ to tag mbedtls-2.2.0
+
+ - Besides the normal build options, add the following build options for TF-A
+
+ .. code-block:: bash
+
+ MBEDTLS_DIR=<path of the directory containing mbed TLS sources>
+ TRUSTED_BOARD_BOOT=1
+ GENERATE_COT=1
+ ARM_ROTPK_LOCATION=devel_rsa
+ ROT_KEY=<TF-A-PATH/plat/arm/board/common/rotpk/arm_rotprivk_rsa.pem>
+
+Above steps have been tested on FVP platform, all verification steps are OK and
+xtest runs successfully without regression.
+
+Armv7-A systems
+***************
+Unlike for Armv8-A systems where one can use a more standardized way of doing
+secure boot by leverage the authentication framework as described above, most
+device manufacturers have their own way of doing secure boot. Please reach out
+directly to the manufacturer for the device you are working with to be able to
+understand how to do secure boot on their devices.
+
+.. _auth-framework.rst : https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/auth-framework.rst
+.. _mbed TLS: https://github.com/ARMmbed/mbedtls.git
+.. _user-guide.rst: https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/user-guide.rst
+.. _trusted-board-boot.rst: https://github.com/ARM-software/arm-trusted-firmware/blob/master/docs/trusted-board-boot.rst
diff --git a/architecture/secure_storage.rst b/architecture/secure_storage.rst
new file mode 100644
index 0000000..b600404
--- /dev/null
+++ b/architecture/secure_storage.rst
@@ -0,0 +1,419 @@
+.. _secure_storage:
+
+##############
+Secure storage
+##############
+
+Background
+**********
+Secure Storage in OP-TEE is implemented according to what has been defined in
+GloblaPlatform's :ref:`tee_internal_core_api` (here called Trusted Storage).
+This specification mandates that it should be possible to store general-purpose
+data and key material that guarantees confidentiality and integrity of the data
+stored and the atomicity of the operations that modifies the storage (atomicity
+here means that either the entire operation completes successfully or no write
+is done).
+
+There are currently two secure storage implementations in OP-TEE:
+
+ - The first one relies on the normal world (REE) file system. It is
+ described in this document and is the default implementation. It is
+ enabled at compile time by ``CFG_REE_FS=y``.
+
+ - The second one makes use of the Replay Protected Memory Block (RPMB)
+ partition of an eMMC device, and is enabled by setting ``CFG_RPMB_FS=y``.
+ It is described in [secure_storage_rpmb.md](secure_storage_rpmb.md).
+
+It is possible to use the normal world file systems and the RPMB implementations
+simultaneously. For this, two OP-TEE specific storage identifiers have been
+defined: ``TEE_STORAGE_PRIVATE_REE`` and ``TEE_STORAGE_PRIVATE_RPMB``. Depending
+on the compile-time configuration, one or several values may be used. The value
+``TEE_STORAGE_PRIVATE`` selects the REE FS when available, otherwise the RPMB FS
+(in this order).
+
+REE FS Secure Storage
+*********************
+.. figure:: ../images/secure_storage/secure_storage_system_architecture.png
+ :figclass: align-center
+
+ Secure Storage System Architecture
+
+**Source Files in OP-TEE OS**
+
+.. list-table:: Secure storage files
+ :header-rows: 1
+ :widths: 1 6
+
+ * - Source file
+ - Purpose
+
+ * - `core/tee/tee_svc_storage.c`_
+ - TEE trusted storage service calls
+
+ * - `core/tee/tee_ree_fs.c`_
+ - TEE file system & REE file operation interface
+
+ * - `core/tee/fs_htree.c`_
+ - Hash tree
+
+ * - `core/tee/tee_fs_key_manager.c`_
+ - Key manager
+
+ * - `lib/libutee/`_
+ - GlobalPlatform Internal API library
+
+Basic File Operation Flow
+=========================
+When a TA is calling the write function provided by GP Trusted Storage API to
+write data to a persistent object, a corresponding syscall implemented in TEE
+Trusted Storage Service will be called, which in turn will invoke a series of
+TEE file operations to store the data. TEE file system will then encrypt the
+data and send REE file operation commands and the encrypted data to TEE
+supplicant by a series of RPC messages. TEE supplicant will receive the messages
+and store the encrypted data accordingly to the Linux file system. Reading files
+are handled in a similar manner.
+
+GlobalPlatform Trusted Storage Requirement
+==========================================
+Below is an excerpt from the specification, listing the most vital requirements:
+
+.. code-block:: none
+
+ 1. The Trusted Storage may be backed by non-secure resources as long as
+ suitable cryptographic protection is applied, which MUST be as strong as
+ the means used to protect the TEE code and data itself.
+
+ 2. The Trusted Storage MUST be bound to a particular device, which means
+ that it MUST be accessible or modifiable only by authorized TAs
+ running in the same TEE and on the same device as when the data was
+ created.
+
+ 3. Ability to hide sensitive key material from the TA itself.
+
+ 4. Each TA has access to its own storage space that is shared among all the
+ instances of that TA but separated from the other TAs.
+
+ 5. The Trusted Storage must provide a minimum level of protection against
+ rollback attacks. It is accepted that the actually physical storage
+ may be in an insecure area and so is vulnerable to actions from
+ outside of the TEE. Typically, an implementation may rely on the REE
+ for that purpose (protection level 100) or on hardware assets
+ controlled by the TEE (protection level 1000).
+
+ (see GP TEE Internal Core API section 2.5 and 5.2)
+
+If configured with ``CFG_RPMB_FS=y`` the protection against rollback is
+controlled by the TEE and is set to 1000. If ``CFG_RPMB_FS=n``, there's no
+protection against rollback, and the protection level is set to 0.
+
+TEE File Structure in Linux File System
+=======================================
+OP-TEE by default uses ``/data/tee/`` as the secure storage space in the Linux
+file system. Each persistent object is assigned an internal identifier. It is an
+integer which is visible in the Linux file system as ``/data/tee/<file
+number>``.
+
+A directory file, ``/data/tee/dirf.db``, lists all the objects that are in the
+secure storage. All normal world files are integrity protected and encrypted, as
+described below.
+
+Key Manager
+***********
+Key manager is an component in TEE file system, and is responsible for handling
+data encryption and decryption and also management of the sensitive key
+materials. There are three types of keys used by the key manager: the Secure
+Storage Key (`SSK`), the TA Storage Key (`TSK`) and the File Encryption Key
+(`FEK`).
+
+Secure Storage Key (SSK)
+========================
+SSK is a per-device key and is generated and stored in secure memory when OP-TEE
+is booting. SSK is used to derive the TA Storage Key (TSK).
+
+SSK is derived by
+
+ SSK = HMAC\ :sub:`SHA256` (HUK, Chip ID || "static string")
+
+The functions to get :ref:`hardware_unique_key` (HUK) and chip ID depends on the
+platform implementation. Currently, in OP-TEE OS we only have a per-device key,
+SSK, which is used for secure storage subsystem, but, for the future we might
+need to create different per-device keys for different subsystems using the same
+algorithm as we generate the SSK; An easy way to generate different per-device
+keys for different subsystems is using different static strings to generate the
+keys.
+
+Trusted Application Storage Key (TSK)
+=====================================
+The TSK is a per-Trusted Application key, which is generated from the SSK and
+the TA's identifier (UUID). It is used to protect the FEK, in other words, to
+encrypt/decrypt the FEK.
+
+TSK is derived by:
+
+ TSK = HMAC\ :sub:`SHA256` (SSK, TA_UUID)
+
+File Encryption Key (FEK)
+=========================
+When a new TEE file is created, key manager will generate a new FEK by PRNG
+(pesudo random number generator) for the TEE file and store the encrypted FEK in
+meta file. FEK is used for encrypting/decrypting the TEE file information stored
+in meta file or the data stored in block file.
+
+Hash Tree
+*********
+The hash tree is responsible for handling data encryption and decryption of a
+secure storage file. The hash tree is implemented as a binary tree where each
+node (``struct tee_fs_htree_node_image`` below) in the tree protects its two
+child nodes and a data block. The meta data is stored in a header (``struct
+tee_fs_htree_image`` below) which also protects the top node.
+
+All fields (header, nodes, and blocks) are duplicated with two versions, 0 and
+1, to ensure atomic updates. See `core/tee/fs_htree.c`_ for details.
+
+Meta Data Encryption Flow
+=========================
+.. figure:: ../images/secure_storage/meta_data_encryption.png
+ :figclass: align-center
+
+ Meta data encryption
+
+A new meta IV will be generated by PRNG when a meta data needs to be updated.
+The size of meta IV is defined in `core/include/tee/fs_htree.h`_, likewise are
+the data structures of meta data and node data are defined in fs_htree.h as
+follows:
+
+.. code-block:: c
+
+ struct tee_fs_htree_node_image {
+ uint8_t hash[TEE_FS_HTREE_HASH_SIZE];
+ uint8_t iv[TEE_FS_HTREE_IV_SIZE];
+ uint8_t tag[TEE_FS_HTREE_TAG_SIZE];
+ uint16_t flags;
+ };
+
+ struct tee_fs_htree_meta {
+ uint64_t length;
+ };
+
+ struct tee_fs_htree_imeta {
+ struct tee_fs_htree_meta meta;
+ uint32_t max_node_id;
+ };
+
+ struct tee_fs_htree_image {
+ uint8_t iv[TEE_FS_HTREE_IV_SIZE];
+ uint8_t tag[TEE_FS_HTREE_TAG_SIZE];
+ uint8_t enc_fek[TEE_FS_HTREE_FEK_SIZE];
+ uint8_t imeta[sizeof(struct tee_fs_htree_imeta)];
+ uint32_t counter;
+ };
+
+Block Data Encryption Flow
+==========================
+.. figure:: ../images/secure_storage/block_data_encryption.png
+ :figclass: align-center
+
+ Block data encryption
+
+A new block IV will be generated by PRNG when a block data needs to be updated.
+The size of block IV is defined in `core/include/tee/fs_htree.h`_.
+
+Atomic Operation
+****************
+According to GlobalPlatform Trusted Storage requirement of the atomicity, the
+following operations should support atomic update::
+
+ Write, Truncate, Rename, Create and Delete
+
+The strategy used in OP-TEE secure storage to guarantee the atomicity is
+out-of-place update.
+
+.. _rpmb:
+
+RPMB Secure Storage
+*******************
+This document describes the RPMB secure storage implementation in OP-TEE, which
+is enabled by setting ``CFG_RPMB_FS=y``. Trusted Applications may use this
+implementation by passing a storage ID equal to ``TEE_STORAGE_PRIVATE_RPMB``, or
+``TEE_STORAGE_PRIVATE`` if ``CFG_REE_FS`` is disabled. For details about RPMB,
+please refer to the JEDEC eMMC specification (JESD84-B51).
+
+The architecture is depicted below.
+
+.. code-block:: none
+
+ | NORMAL WORLD : SECURE WORLD |
+ :
+ U tee-supplicant : Trusted application
+ S (rpmb.c) : (secure storage API)
+ E ^ ^ : ^
+ R | | : |
+ ~~~~~~~ ioctl ~~~~~~~|~~~~~~~~~~~~:~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~~~~~~~~~
+ K | | : OP-TEE
+ E v v : (tee_svc_storage.c)
+ R MMC/SD subsys. OP-TEE driver : (tee_rpmb_fs.c, tee_fs_key_manager.c)
+ N ^ ^ : ^
+ E | | : |
+ L v | : |
+ Controller driver | : |
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~~~~~~~~~~~~~|~~~~~~~~~~~~~~~~~~~~
+ v v
+ Secure monitor / EL3 firmware
+
+For information about the ``ioctl()`` interface to the MMC/SD subsystem in the
+Linux kernel, see the Linux core MMC header file `linux/mmc/core.h`_ and the
+mmc-utils_ repository.
+
+The Secure Storage API
+======================
+This part is common with the REE-based filesystem. The interface between the
+system calls in `core/tee/tee_svc_storage.c`_ and the RPMB filesystem is the
+`tee_file_operations`, namely ``struct tee_file_ops``.
+
+The RPMB filesystem
+===================
+The FS implementation is entirely in `core/tee/tee_rpmb_fs.c`_ and the RPMB
+partition is divided in three parts:
+
+ - The first 128 bytes are reserved for partition data (``struct
+ rpmb_fs_partition``).
+
+ - At offset 512 is the File Allocation Table (FAT). It is an array of
+ ``struct rpmb_fat_entry`` elements, one per file. The FAT grows
+ dynamically as files are added to the filesystem. Among other things, each
+ entry has the start address for the file data, its size, and the filename.
+
+ - Starting from the end of the RPMB partition and extending downwards is the
+ file data area.
+
+Space in the partition is allocated by the general-purpose allocator functions,
+``tee_mm_alloc(...)`` and ``tee_mm_alloc2(...)``.
+
+All file operations are atomic. This is achieved thanks to the following
+properties:
+
+ - Writing one single block of data to the RPMB partition is guaranteed to be
+ atomic by the eMMC specification.
+
+ - The FAT block for the modified file is always updated last, after data
+ have been written successfully.
+
+ - Updates to file content is done in-place only if the data do not span more
+ than the "reliable write block count" blocks. Otherwise, or if the file
+ needs to be extended, a new file is created.
+
+Device access
+=============
+There is no eMMC controller driver in OP-TEE. The device operations all have to
+go through the normal world. They are handled by the ``tee-supplicant`` process
+which further relies on the kernel's ``ioctl()`` interface to access the device.
+``tee-supplicant`` also has an emulation mode which implements a virtual RPMB
+device for test purposes.
+
+RPMB operations are the following:
+ - Reading device information (partition size, reliable write block count).
+
+ - Programming the security key. This key is used for authentication
+ purposes. Note that it is different from the Secure Storage Key (SSK)
+ defined below, which is used for encryption. Like the SSK however, the
+ security key is also derived from a hardware unique key or identifier.
+ Currently, the function ``tee_otp_get_hw_unique_key()`` is used to
+ generate the RPMB security key.
+
+ - Reading the write counter value. The write counter is used in the HMAC
+ computation during read and write requests. The value is read at
+ initialization time, and stored in ``struct tee_rpmb_ctx``, i.e.,
+ ``rpmb_ctx->wr_cnt``.
+
+ - Reading or writing blocks of data.
+
+RPMB operations are initiated on request from the FS layer. Memory buffers for
+requests and responses are allocated in shared memory using
+``thread_rpc_alloc_payload(...)``. Buffers are passed to the normal world in
+a ``TEE_RPC_RPMB_CMD`` message, thanks to the ``thread_rpc_cmd()`` function.
+Most RPMB requests and responses use the data frame format defined by the JEDEC
+eMMC specification. HMAC authentication is implemented here also.
+
+Encryption
+==========
+The FS encryption routines are in `core/tee/tee_fs_key_manager.c`_. Block
+encryption protects file data. The algorithm is 128-bit AES in Cipher Block
+Chaining (CBC) mode with Encrypted Salt-Sector Initialization Vector (ESSIV),
+see CBC-ESSIV_ for details.
+
+ - During OP-TEE initialization, a 128-bit AES Secure Storage Key (SSK) is
+ derived from a :ref:`hardware_unique_key` (HUK). It is kept in secure
+ memory and never written to disk. A Trusted Application Storage Key is
+ derived from the SSK and the TA UUID.
+
+ - For each file, a 128-bit encrypted File Encryption Key (FEK) is randomly
+ generated when the file is created, encrypted with the TSK and stored in
+ the FAT entry for the file.
+
+ - Each 256-byte block of data is then encrypted in CBC mode. The
+ initialization vector is obtained by the ESSIV algorithm, that is, by
+ encrypting the block number with a hash of the FEK. This allows direct
+ access to any block in the file, as follows:
+
+ .. code-block:: none
+
+ FEK = AES-Decrypt(TSK, encrypted FEK);
+ k = SHA256(FEK);
+ IV = AES-Encrypt(128 bits of k, block index padded to 16 bytes)
+ Encrypted block = AES-CBC-Encrypt(FEK, IV, block data);
+ Decrypted block = AES-CBC-Decrypt(FEK, IV, encrypted block data);
+
+
+SSK, TSK and FEK handling is common with the REE-based secure storage, while the
+AES CBC block encryption is used only for RPMB (the REE implementation uses
+GCM). The FAT is not encrypted.
+
+REE FS hash state
+=================
+If configured with both ``CFG_REE_FS=y`` and ``CFG_RPMB_FS=y`` the REE FS will
+create a special file, ``dirfile.db.hash`` in RPMB which hold a hash
+representing the state of REE FS.
+
+Important caveats
+*****************
+.. warning::
+ Currently **no OP-TEE platform** is able to support retrieval of the
+ Hardware Unique Key or Chip ID required for secure operation. For all
+ platforms, a constant key is used, resulting in no protection against
+ decryption, or Secure Storage duplication to other devices. This is because
+ information about how to retrieve key data from the SoC is considered
+ sensitive by the vendors and it is not publicly available.
+
+ In OP-TEE, there are APIs for reading keys generically from
+ One-Time-Programmable (OTP) memory. But there are no existing platform
+ implementations.
+
+To allow Secure Storage to operate securely on your platform, you must define
+implementations in your platform code for:
+
+.. code-block:: c
+
+ void tee_otp_get_hw_unique_key(struct tee_hw_unique_key *hwkey);
+
+ int tee_otp_get_die_id(uint8_t *buffer, size_t len);
+
+These implementations should fetch the key data from your SoC-specific e-fuses,
+or crypto unit according to the method defined by your SoC vendor.
+
+References
+**********
+For more information about secure storage, please see SFO15-503, LAS16-504,
+SFO17-309 at :ref:`presentations` and the :ref:`tee_internal_core_api`
+specification.
+
+.. _CBC-ESSIV: https://en.wikipedia.org/wiki/Disk_encryption_theory#Cipher-block_chaining_(CBC)
+.. _linux/mmc/core.h: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/include/linux/mmc/core.h
+.. _mmc-utils: http://git.kernel.org/cgit/linux/kernel/git/cjb/mmc-utils.git
+
+.. _core/tee/tee_svc_storage.c: https://github.com/OP-TEE/optee_os/blob/master/core/tee/tee_svc_storage.c
+.. _core/tee/tee_ree_fs.c: https://github.com/OP-TEE/optee_os/blob/master/core/tee/tee_ree_fs.c
+.. _core/tee/fs_htree.c: https://github.com/OP-TEE/optee_os/blob/master/core/tee/fs_htree.c
+.. _core/include/tee/fs_htree.h: https://github.com/OP-TEE/optee_os/blob/master/core/include/tee/fs_htree.h
+.. _core/tee/tee_fs_key_manager.c: https://github.com/OP-TEE/optee_os/blob/master/core/tee/tee_fs_key_manager.c
+.. _core/tee/tee_rpmb_fs.c: https://github.com/OP-TEE/optee_os/blob/master/core/tee/tee_rpmb_fs.c
+.. _lib/libutee/: https://github.com/OP-TEE/optee_os/blob/master/lib/libutee/
diff --git a/architecture/trusted_applications.rst b/architecture/trusted_applications.rst
new file mode 100644
index 0000000..57662e1
--- /dev/null
+++ b/architecture/trusted_applications.rst
@@ -0,0 +1,221 @@
+.. _trusted_applications:
+
+####################
+Trusted Applications
+####################
+There are two ways to implement Trusted Applications (TAs), Pseudo TAs and user
+mode TAs. User mode TAs are full featured Trusted Applications as specified by
+the :ref:`globalplatform_api` TEE specifications, these are simply the ones
+people are referring to when they are saying "Trusted Applications" and in most
+cases this is the preferred type of TA to write and use.
+
+.. _pta:
+
+Pseudo Trusted Applications
+***************************
+These are implemented directly to the OP-TEE core tree in, e.g.,
+``core/arch/arm/pta`` and are built along with and statically built into the
+OP-TEE core blob.
+
+The Pseudo Trusted Applications included in OP-TEE already are OP-TEE secure
+privileged level services hidden behind a "GlobalPlatform TA Client" API. These
+Pseudo TAs are used for various purposes such as specific secure services or
+embedded tests services.
+
+Pseudo TAs **do not** benefit from the GlobalPlatform Core Internal API support
+specified by the GlobalPlatform TEE specs. These APIs are provided to TAs as a
+static library each TA shall link against (the ":ref:`libutee`") and that calls
+OP-TEE core service through system calls. As OP-TEE core does not link with
+:ref:`libutee`, Pseudo TAs can **only** use the OP-TEE core internal APIs and
+routines.
+
+As Pseudo TAs runs at the same privileged execution level as the OP-TEE core
+code itself and that might or might not be desirable depending on the use case.
+
+In most cases an unprivileged (user mode) TA is the best choice instead of
+adding your code directly to the OP-TEE core. However if you decide your
+application is best handled directly in OP-TEE core like this, you can look at
+``core/arch/arm/pta/stats.c`` as a template and just add your Pseudo TA based on
+that to the ``sub.mk`` in the same directory.
+
+.. _user_mode_ta:
+
+User Mode Trusted Applications
+******************************
+User Mode Trusted Applications are loaded (mapped into memory) by OP-TEE core in
+the Secure World when something in Rich Execution Environment (REE) wants to
+talk to that particular application UUID. They run at a lower CPU privilege
+level than OP-TEE core code. In that respect, they are quite similar to regular
+applications running in the REE, except that they execute in Secure World.
+
+Trusted Application benefit from the GlobalPlatform :ref:`tee_internal_core_api`
+as specified by the GlobalPlatform TEE specifications. There are several types
+of user mode TAs, which differ by the way they are stored.
+
+TA locations
+************
+Plain TAs (user mode) can reside and be loaded from various places. There are
+three ways currently supported in OP-TEE.
+
+.. _early_ta:
+
+Early TA
+========
+The so-called early TAs are virtually identical to the REE FS TAs, but instead
+of being loaded from the Normal World file system, they are linked into a
+special data section in the TEE core blob. Therefore, they are available even
+before ``tee-supplicant`` and the REE's filesystems have come up. Please find
+more details in the `early TA commit`_.
+
+.. _ree_fs_ta:
+
+REE filesystem TA
+=================
+They consist of a cleartext signed ELF_ file, named from the UUID of the TA and
+the suffix ``.ta``. They are built separately from the OP-TEE core boot-time
+blob, although when they are built they use the same build system, and are
+signed with the key from the build of the original OP-TEE core blob.
+
+Because the TAs are signed, they are able to be stored in the untrusted REE
+filesystem, and ``tee-supplicant`` will take care of passing them to be checked
+and loaded by the Secure World OP-TEE core. Note that this type of TA isn't
+encrypted.
+
+.. _secure_storage_ta:
+
+Secure Storage TA
+=================
+These are stored in secure storage. The meta data is stored in a database of all
+installed TAs and the actual binary is stored encrypted and integrity protected
+as a separate file in the untrusted REE filesystem (flash). Before these TAs can
+be loaded they have to be installed first, this is something that can be done
+during initial deployment or at a later stage.
+
+For test purposes the test program xtest can install a TA into secure storage
+with the command:
+
+.. code-block:: bash
+
+ $ xtest --install-ta
+
+
+.. _ta_properties:
+
+TA Properties
+*************
+This section give a more in depth description of the TA properties (see
+:ref:`build_trusted_applications` also).
+
+1. GlobalPlatform properties
+============================
+Standard TA properties must be defined through property flag in macro
+``TA_FLAGS`` in ``user_ta_header_defines.h``
+
+1.1 Single Instance
+===================
+``"gpd.ta.singleInstance"`` is a boolean property of the TA. This property
+defines if one instance of the TA must be created and will receive all open
+session request, or if a new specific TA instance must be created for each
+incoming open session request. OP-TEE TA flag ``TA_FLAG_SINGLE_INSTANCE`` sets
+to configuration of this property. The boolean property is set to ``true`` if
+``TA_FLAGS`` sets bit ``TA_FLAG_SINGLE_INSTANCE``, otherwise the boolean
+property is set to ``false``.
+
+1.2 Multi-session
+=================
+``"gpd.ta.multiSession"`` is a boolean property of the TA. This property defines
+if the TA instance can handle several sessions. If disabled, TA instance support
+only one session. In such case, if the TA already has a opened session, any open
+session request will return with a busy error status.
+
+.. note::
+
+ This property is **meaningless** if TA is **NOT** SingleInstance TA.
+
+OP-TEE TA flag ``TA_FLAG_MULTI_SESSION`` sets to configuration of this property.
+The boolean property is set to ``true`` if ``TA_FLAGS`` sets bit
+``TA_FLAG_MULTI_SESSION``, otherwise the boolean property is set to ``false``.
+
+1.3 Keep Alive
+==============
+``"gpd.ta.instanceKeepAlive"`` is a boolean property of the TA. This property
+defines if the TA instance created must be destroyed or not when all sessions
+opened towards the TA are closed. If the property is enabled, TA instance, once
+created (at 1st open session request), is never removed unless the TEE itself is
+restarted (boot/reboot).
+
+.. note::
+
+ This property is **meaningless** if TA is **NOT** SingleInstance TA.
+
+OP-TEE TA flag ``TA_FLAG_INSTANCE_KEEP_ALIVE`` sets to configuration of this
+property. The boolean property is set to ``true`` if ``TA_FLAGS`` sets bit
+``TA_FLAG_INSTANCE_KEEP_ALIVE``, otherwise the boolean property is set to
+``false``.
+
+1.4 Heap Size
+=============
+``"gpd.ta.dataSize"`` is a 32bit integer property of the TA. This property
+defines the size in bytes of the TA allocation pool, in which ``TEE_Malloc()``
+and friends allocate memory. The value of the property must be defined by the
+macro ``TA_DATA_SIZE`` in ``user_ta_header_defines.h`` (see
+:ref:`build_ta_properties`).
+
+1.5 Stack Size
+==============
+``"gpd.ta.stackSize"`` is a 32bit integer property of the TA. This property
+defines the size in bytes of the stack used for TA execution. The value of the
+property must be defined by the macro ``TA_STACK_SIZE`` in
+``user_ta_header_defines.h`` (see :ref:`build_ta_properties`).
+
+2. Property extensions
+======================
+2.1 User Mode Flag
+==================
+``TA_FLAG_USER_MODE`` is a bit flag supported by ``TA_FLAGS``. This property
+flag is currently meaningless in OP-TEE. It may be set or not without impact on
+TA execution. All OP-TEE TAs are executed in user mode/level. Because of this we
+**do not** recommend to use this flag.
+
+2.2 DDR Flag
+============
+``TA_FLAG_EXEC_DDR`` is a bit flag supported by ``TA_FLAGS``. This property flag
+is currently meaningless in OP-TEE. Nevertheless it shall be set. It is a legacy
+property flag that aimed at targeting location for the TA execution, internal
+RAM or external DDR. Therefore all TAs must set ``TA_FLAG_EXEC_DDR`` in
+``TA_FLAGS`` in their ``user_ta_header_defines.h`` header file (see
+:ref:`user_ta_header_defines_h`).
+
+.. note::
+
+ This flag will soon be deprecated.
+
+2.3 Secure Data Path Flag
+=========================
+``TA_FLAG_SECURE_DATA_PATH`` is a bit flag supported by ``TA_FLAGS``. This
+property flag claims the secure data support from the OP-TEE OS for the TA.
+Refer to the OP-TEE OS for secure data path support. TAs that do not set
+``TA_FLAG_SECURE_DATA_PATH`` in the value of ``TA_FLAGS`` will **not** be able
+to handle memory reference invocation parameters that relate to secure data path
+buffers.
+
+2.4 Remap Support Flag
+======================
+``TA_FLAG_REMAP_SUPPORT`` is a bit flag supported by ``TA_FLAGS``. This property
+flag is currently meaningless in OP-TEE and therefore we recommend to not use
+this flag.
+
+.. note::
+
+ This flag will soon be deprecated.
+
+2.5 Cache maintenance Flag
+==========================
+``TA_FLAG_CACHE_MAINTENANCE`` is a bit flag supported by ``TA_FLAGS``. This
+property flag claims access to the cache maintenance API for the TA:
+``TEE_CacheXxxx()``. Refer to the OP-TEE to check if cache API support is
+enabled. TAs that do not set ``TA_FLAG_CACHE_MAINTENANCE`` in the value of their
+``TA_FLAGS`` will not be able to call the cache maintenance API.
+
+.. _ELF: https://en.wikipedia.org/wiki/Executable_and_Linkable_Format
+.. _early TA commit: https://github.com/OP-TEE/optee_os/commit/d0c636148b3a
diff --git a/architecture/virtualization.rst b/architecture/virtualization.rst
new file mode 100644
index 0000000..b12f94c
--- /dev/null
+++ b/architecture/virtualization.rst
@@ -0,0 +1,170 @@
+##############
+Virtualization
+##############
+OP-TEE have experimental virtualization support. This is when one OP-TEE
+instance can run TAs from multiple virtual machines. OP-TEE isolates all
+VM-related states, so one VM can't affect another in any way.
+
+With virtualization support enabled, OP-TEE will rely on a hypervisor, because
+only the hypervisor knows which VM is calling OP-TEE. Also, naturally the
+hypervisor should inform OP-TEE about creation and destruction of VMs. Besides,
+in almost all cases, hypervisor enables two-stage MMU translation, so VMs does
+not see real physical address of memory, instead they work with intermediate
+physical addresses (IPAs). On other hand OP-TEE can't translate IPA to PA, so
+this is a hypervisor's responsibility to do this kind of translation. So,
+hypervisor should include a component that knows about OP-TEE protocol internals
+and can do this translation. We call this component "TEE mediator" and right now
+only XEN hypervisor have OP-TEE mediator.
+
+Configuration
+*************
+Virtualization support is enabled with ``CFG_VIRTUALIZATION`` configuration
+option. When this option is enabled, OP-TEE will **not** work without compatible
+a hypervisor. This is because the hypervisor should send
+``OPTEE_SMC_VM_CREATED`` SMC with VM ID before any standard SMC can be received
+from client.
+
+``CFG_VIRT_GUEST_COUNT`` controls the maximum number of supported VMs. As OP-TEE
+have limited size of available memory, increasing this count will decrease
+amount of memory available to one VM. Because we want VMs to be independent,
+OP-TEE splits available memory in equal portions to every VM, so one VM can't
+consume all memory and cause DoS to other VMs.
+
+Requirements for hypervisor
+***************************
+As said earlier, hypervisor should be aware of OP-TEE and SMCs from virtual
+guests to OP-TEE. This is a list of things, that compatible hypervisor should
+perform:
+
+ 1. When new OP-TEE-capable VM is created, hypervisor should inform OP-TEE
+ about it with SMC ``OPTEE_SMC_VM_CREATED``. ``a1`` parameter should
+ contain VM id. ID 0 is defined as ``HYP_CLNT_ID`` and is reserved for
+ hypervisor itself.
+
+ 2. When OP-TEE-capable VM is being destroyed, hypervisor should stop all
+ VCPUs (this will ensure that OP-TEE have no active threads for that VMs)
+ and send SMC ``OPTEE_SMC_VM_DESTROYED`` with the same parameters as for
+ ``OPTEE_SMC_VM_CREATED``.
+
+ 3. Any SMC to OP-TEE should have VM ID in ``a7`` parameter. This is either
+ ``HYP_CLNT_ID`` if call originates from hypervisor or VM ID that was
+ passed in ``OPTEE_SMC_VM_CREATED`` call.
+
+ 4. Hypervisor should perform IPA<->PA address translation for all SMCs. This
+ includes both arguments in ``a1``-``a6`` registers and in in-memory
+ command buffers.
+
+ 5. Hypervisor should pin memory pages that VM shares with OP-TEE. This
+ means, that hypervisor should ensure that pinned page will reside at the
+ original PA as long, as it is shared with OP-TEE. Also it should still
+ belong to the VM that shared it. For example, the hypervisor should not
+ swap out this page, transfer ownership to another VM, unmap it from VM
+ address space and so on.
+
+ 6. Naturally, the hypervisor should correctly handle the OP-TEE protocol, so
+ for any VM it should look like it is working with OP-TEE directly.
+
+Limitations
+***********
+Virtualization support is in experimental state and it have some limitations,
+user should be aware of.
+
+Platforms support
+=================
+Only Armv8 architecture is supported. There is no hard restriction, but
+currently Armv7-specific code (like MMU or thread manipulation) just know
+nothing about virtualization. Only one platform has been tested right now and
+that is QEMU-V8 (aka qemu that emulates Arm Versatile Express with Armv8
+architecture). Support for Rcar Gen3 should be added soon.
+
+Static VMs guest count and memory allocation
+============================================
+Currently, a user should configure maximum number of guests. OP-TEE will split
+memory into equal chunks, so every VM will have the same amount of memory. For
+example, if you have 6MB for your TAs, you can set ``CFG_VIRT_GUEST_COUNT`` to 3
+and every VM would be able to use 2MB maximum, even if there is no other VMs
+running. This is okay for embedded setups when you know exact number and roles
+of VMs, but can be inconvenient for server applications. Also, it is impossible
+to configure amount of memory available for a given VM. Every VM instance will
+have exactly the same amount of memory.
+
+Sharing hardware resources and PTAs
+===================================
+Right now only HW that can be used by multiple VMs simultaneously is serial
+console, used for logging. Devices like HW crypto accelerators, secure storage
+devices (e.g. external flash storage, accessed directly from OP-TEE) and others
+are not supported right now. Drivers should be made virtualization-aware before
+they can be used with virtualization extensions.
+
+Every VM will have own PTA states, which is a good thing in most cases. But if
+one wants PTA to have some global state that is shared between VMs, he need to
+write PTA accordingly.
+
+No compatibility with "normal" mode
+===================================
+OP-TEE built with ``CFG_VIRTUALIZATION=y`` will not work without a hypervisor,
+because before executing any standard SMC, ``OPTEE_SMC_VM_CREATED`` must be
+called. This can be inconvenient if one wants to switch between virtualized and
+non-virtualized environment frequently. On other hand, it is not a big deal in a
+production environment. Simple workaround can be made for this: if OP-TEE
+receives standard SMC prior to ``OPTEE_SMC_VM_CREATED``, it implicitly creates
+VM context and uses it for all subsequent calls.
+
+Implementation details
+======================
+OP-TEE as a whole can be split into two entities. Let us call them "nexus" and
+TEE. Nexus is a core part of OP-TEE that takes care of low level things: SMC
+handling, memory management, threads creation and so on. TEE is a part that does
+the actual job: handles requests, loads TAs, executes them, and so on. So, it is
+natural to have one nexus instance and multiple instances of TEE, one TEE
+instance per registered VM. This can be done either explicitly or implicitly.
+
+Explicit way is to move TEE state in some sort of structure and make all code to
+access fields of this structure. Something like ``struct task_struct`` and
+``current`` in linux kernel. Then it is easy to allocate such structure for
+every VM instance. But this approach basically requires to rewrite all OP-TEE
+code.
+
+Implicit way is to have banked memory sections for TEE/VM instances. So memory
+layout can look something like that:
+
+.. code-block:: none
+
+ +-------------------------------------------------+
+ | Nexus: .nex_bss, .nex_data, ... |
+ +-------------------------------------------------+
+ | TEE states |
+ | |
+ | VM1 TEE state | VM 2 TEE state | VM 3 TEE state |
+ | .bss, .data | .bss, .data | .bss, .data, |
+ +-------------------------------------------------+
+
+This approach requires no changes in TEE code and requires some changes into
+nexus code. So, idea that Nexus state resides in separate sections
+(``.nex_data``, ``.nex_bss``, ``.nex_nozi``, ``.nex_heap`` and others) and is
+always mapped.
+
+TEE state resides in standard sections (like ``.data``, ``.bss``, ``.heap`` and
+so on). There is a separate set of this sections for every VM registered and
+Nexus maps them only when it receives call from corresponding VM.
+
+As Nexus and TEE have separate heaps, ``bget`` allocator was extended to work
+with multiple "contexts". ``malloc()``, ``free()`` with friends work with one
+context. ``nex_malloc()`` (and other ``nex_`` functions) were added. They use
+different context, so now Nexus can use separate heap, which is always mapped
+into OP-TEE address space. When virtualization support is disabled, all those
+``nex_`` functions are defined to point to standard ``malloc()`` counterparts.
+
+To change memory mappings in run-time, in MMU code we have added a new entity,
+named "partition", which is defined by ``struct mmu_partition``. It holds
+information about all page-tables, so the whole MMU mapping can be switched by
+one write to ``TTBR`` register.
+
+There is the default partition, it holds MMU state when there is no VM context
+active, so no TEE state is mapped. When OP-TEE receives ``OPTEE_SMC_VM_CREATED``
+call, it copies default partition into new one and then maps sections with TEE
+data. This is done by ``prepare_memory_map()`` function in ``virtualization.c``.
+
+When OP-TEE receives STD call it checks that the supplied VM ID is valid and
+then activates corresponding MMU partition, so TEE code can access its own data.
+This is basically how virtualization support is working.