Add build test runner: b-test.
Test runner can be used to do build testing. The main intention is to
provide a simple and robust way to capture build commands.
For details please check the documentation.
Change-Id: Icab22320a86dcfdb882a36c1c8c2a3380c40d30c
Signed-off-by: Gyorgy Szing <gyorgy.szing@arm.com>
diff --git a/tools/b-test/.gitignore b/tools/b-test/.gitignore
new file mode 100644
index 0000000..bde4006
--- /dev/null
+++ b/tools/b-test/.gitignore
@@ -0,0 +1,4 @@
+#Ignore instaniated template.
+run.sh
+# Ignore user specific setting script
+user.env
diff --git a/tools/b-test/Makefile b/tools/b-test/Makefile
new file mode 100644
index 0000000..1a4a321
--- /dev/null
+++ b/tools/b-test/Makefile
@@ -0,0 +1,62 @@
+# Copyright (c) 2020, Arm Limited and Contributors. All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+#
+
+.SUFFIXES:
+.SUFFIXES: .j2
+
+# Disable parallel execution for this makefile.
+.NOTPARALLEL:
+
+TEMPLATES:=$(wildcard *.j2)
+OUTPUTS:=$(TEMPLATES:%.j2=%)
+
+all:
+ make config
+ bash run.sh
+
+config: ${OUTPUTS}
+
+list:
+ make config
+ bash run.sh help
+
+# run a command of the generated script
+r-%:
+ make config
+ bash run.sh "$*"
+
+define help_msg
+*************************************************************
+Available targets:
+ all - generate and run test script
+ config - run script generation only
+ clean - remove intermediate files
+ list - run config and list test cases
+ p-<variable> - print value of make variable (for debugging)
+ r-<test> - run a command of the generated script. Allow
+ executing a single test case. Use
+ make r-help
+ to get the list of tests.
+*************************************************************
+endef
+
+export help_msg
+help:
+ @echo "$$help_msg"
+
+# remove intermediate output
+clean:
+ifneq ($(wildcard ./run.sh),)
+ bash run.sh clean
+endif
+ -rm -rf ${OUTPUTS}
+
+# Convert template to shell script
+%.sh : %.sh.j2 test_data.yaml
+ yasha -v test_data.yaml $<
+
+# print variable value (i.e. make p-TEMPLATES)
+p-%:
+ @echo $*=${$*}
diff --git a/tools/b-test/Readme.rst b/tools/b-test/Readme.rst
new file mode 100644
index 0000000..4100e91
--- /dev/null
+++ b/tools/b-test/Readme.rst
@@ -0,0 +1,107 @@
+Build test runner
+=================
+
+This directory captures build test case definitions and a tool to execute the test based on
+the data. The tool combines the power of shell scripting with the power of structured data
+(|yaml|). The bridge between the two technologies is provided by |jinja2| template engine
+and |yasha|.
+
+Dependencies
+------------
+
+|Jinja2| and |yasha| are python tools and python3 is needed to run the tests. Please install
+the following tools into your build environment:
+
+ - python3
+ - pip3
+
+After this please install further pip packages listed in ``requirements.txt`` as:
+
+.. code:: shell
+
+ pip3 install -r requirements.txt
+
+.. note::
+
+ This document lists the dependencies of this tool only. To be able to successfully run the
+ build tests further tools are needed. Please refer to the ``Trusted Services``
+ documentation for details.
+
+Files
+-------
+
+.. uml::
+
+ @startsalt
+ {
+ {T
+ + b-test
+ ++Makefile | generate and run tests
+ ++Readme.rst | this document
+ ++requirements.txt | pip package dependencies
+ ++run.sh.j2 | shell script template
+ ++test_data.schema.json | JSON schema for test case data
+ ++test_data.yaml | test case data
+ ++//user.env// | optional user specific settings
+ }
+ }
+ @endsalt
+
+Design
+------
+
+The project needs a convenient way to define and execute "build tests". This test aims to ensure
+all build configurations are in a good working condition. Testing is done by executing build
+of all supported build configurations. In order to make the testing robust and easy to use a
+"data driven" approach is the best fit. With this test cases are described by pure data and this
+data is processed by some tool which is responsible for test execution.
+
+For command execution the bash shell is a good candidate. It provides portability between OSs, is
+widely adopted and well tested. Unfortunately shells are not good on handling structured data.
+To address this shortcoming templating is utilized or "code generation" is used. The shell script
+to execute the command is generated based on a template file and the test data.
+
+Since python is already a dependency of Trusted Services we selected the |Jinja2| template engine
+to go with, and to decrease maintenance cost, we use it trough |yasha|.
+
+.. uml::
+
+ @startuml
+ [test_data.yaml] --> [yasha]
+ [run.sh.j2] --> [yasha]
+ [//user.env//] -right-> [run.sh]
+ [yasha] --> [run.sh]
+ @enduml
+
+Usage
+-----
+
+There are two "entry points" to the tests. If the intention is to run all tests, issue ``make``.
+
+Makefile
+""""""""
+The makefile is responsible to provide a high level "API". It allows executing the script generation
+process and to run the tests. It ensures all components are fresh before being executed.
+
+Issue ``make help`` to get a list of supported commands.
+
+run.sh
+""""""
+
+``run.sh`` is the test runner. It is responsible to execute the needed builds in a proper way and
+thus validate the build definitions.
+
+Execute ``run.sh help`` to get further details.
+
+
+--------------
+
+*Copyright (c) 2020, Arm Limited and Contributors. All rights reserved.*
+
+.. |yasha| replace:: `yasha`_
+.. |jinja2| replace:: `Jinja2`_
+.. |yaml| replace:: `yaml`_
+
+.. _Jinja2: https://palletsprojects.com/p/jinja
+.. _yasha: https://github.com/kblomqvist/yasha
+.. _yaml: https://yaml.org
diff --git a/tools/b-test/requirements.txt b/tools/b-test/requirements.txt
new file mode 100644
index 0000000..1d0415e
--- /dev/null
+++ b/tools/b-test/requirements.txt
@@ -0,0 +1,6 @@
+#
+# Copyright (c) 2020, Arm Limited and Contributors. All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+yasha==4.4
diff --git a/tools/b-test/run.sh.j2 b/tools/b-test/run.sh.j2
new file mode 100644
index 0000000..32d5e83
--- /dev/null
+++ b/tools/b-test/run.sh.j2
@@ -0,0 +1,174 @@
+#!/bin/bash
+#
+# Copyright (c) 2020, Arm Limited and contributors. All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+#
+
+# Exit with error if any command fails
+set -e
+
+# Read in user specific environment settings
+if [ -e user.env ]
+then
+ echo "Reding user specific settings from user.env"
+ source user.env
+fi
+
+# Default to non-verbose mode.
+VERBOSE=${VERBOSE:-0}
+
+# Get root of TS repo.
+TS_ROOT=${TS_ROOT:-$(git rev-parse --show-toplevel)}
+
+# Number of threads to use in parallel
+NUMBER_OF_PROCESSORS=${NUMBER_OF_PROCESSORS:-$(( $(nproc) * 2 )) }
+
+# Convert test name to build directory
+function name-to-bdir() {
+ printf "./build-%s" "$1"
+}
+
+# Wrap cmake to allow verbose vs non-verbose mode
+function _cmake() {
+ log_file=$1
+ shift
+ if [ "$VERBOSE" != "0" ]
+ then
+ cmake "$@" 2>&1 | tee -a "$log_file"
+ return ${PIPESTATUS[0]}
+ else
+ cmake "$@" >>"$log_file" 2>&1
+ fi
+}
+
+{% for config in data %}
+# Run build test "{{config.name}}"
+{{config.name}}() {
+ echo "##################################################################"
+ echo "########################## {{config.name}} started."
+
+ {% if config.os_id is defined %}
+ if [ "$OS_ID" != "{{config.os_id}}" ]
+ then
+ echo "Test case is not supported on this host."
+ echo "########################## $COLOR_YELOW {{config.name}} skipped $COLOR_RESET"
+ echo "##################################################################"
+ return
+ fi
+ {% endif %}
+ b_dir=$(name-to-bdir "{{config.name}}")
+ log_file=$b_dir/build.log
+ rm -rf "$b_dir"
+ mkdir -p "$b_dir"
+
+ _cmake "$log_file" -S {{config.src}} -B "$b_dir" {% for param in config.params %} "{{param}}" {%endfor%} || {
+
+ echo "For details see: $log_file"
+ echo "########################## $COLOR_RED {{config.name}} failed $COLOR_RESET"
+ return
+ }
+
+ if _cmake "$log_file" --build "$b_dir" -j ${NUMBER_OF_PROCESSORS}
+ then
+ if _cmake "$log_file" --install "$b_dir" --prefix ./install
+ then
+ echo "########################## $COLOR_GREEN {{config.name}} passed $COLOR_RESET"
+ else
+ echo "For details see: $log_file"
+ echo "########################## $COLOR_RED {{config.name}} failed $COLOR_RESET"
+ fi
+ else
+ echo "For details see: $log_file"
+ echo "########################## $COLOR_RED {{config.name}} failed $COLOR_RESET"
+ fi
+ echo "##################################################################"
+}
+
+{% endfor %}
+
+# Clean intermediate files
+do_clean() {
+ {% for config in data %}
+ b_dir=$(name-to-bdir "{{config.name}}")
+ if [ -d "$b_dir" ]
+ then
+ echo "Removing $b_dir"
+ rm -rf "$b_dir" || true
+ fi
+ {% endfor %}
+}
+
+# Print usage info
+do_help() {
+ cat <<END_HELP
+Build test runner
+=================
+
+Invocation::
+ ``$0 <command>``
+
+The file "user.env" is sourced from the current directory. Use it to set
+environment specific defaults. For config variables see the start of this script
+and any "$<XXXX>" in the "params" array of any command in test_data.yaml
+Some variables to note
+ - VERBOSE : make the script output more info.
+ VERBOSE=$VERBOSE
+ - TS_ROOT : Root directory of the TS repo.
+ TS_ROOT=$TS_ROOT
+ - NUMBER_OF_PROCESSORS: number of processors in the system. Used for setting the number of
+ parallel processes during build
+ NUMBER_OF_PROCESSORS=$NUMBER_OF_PROCESSORS
+ - SP_DEV_KIT_DIR : location of OP-TEE OS SPDEVKIT export.
+ SP_DEV_KIT_DIR=$SP_DEV_KIT_DIR
+
+Available commands:
+ "" - no command/default -> run all test cases
+ clean - remove build directories
+ help - print this text
+ <test case> - run a single build
+ available test cases:
+ {% for config in data %}
+ {{config.name}}
+ {% endfor %}
+END_HELP
+}
+
+#################### Entry point ###################################
+
+OS_ID=$(uname -o )
+
+if [ -n $(which tput) -a -t ]
+then
+ COLOR_YELOW=$(tput setaf 3)
+ COLOR_RESET=$(tput sgr0)
+ COLOR_RED=$(tput setaf 1)
+ COLOR_GREEN=$(tput setaf 2)
+else
+ COLOR_YELOW=
+ COLOR_RESET=
+ COLOR_RED=
+ COLOR_GREEN=
+fi
+
+case $1 in
+ {% for config in data %}
+ {{config.name}})
+ {{config.name}}
+ ;;
+ {% endfor %}
+ clean)
+ do_clean
+ ;;
+ help)
+ do_help
+ ;;
+ "")
+ {% for config in data %}
+ {{config.name}}
+ {% endfor %}
+ ;;
+ *)
+ do_help
+ ;;
+esac
diff --git a/tools/b-test/test_data.schema.json b/tools/b-test/test_data.schema.json
new file mode 100644
index 0000000..da41e96
--- /dev/null
+++ b/tools/b-test/test_data.schema.json
@@ -0,0 +1,18 @@
+{
+ "$schema" : "http://json-schema.org/schema#",
+ "$id" : "Schema for build test description file",
+ "type": "object",
+ "properties" : {
+ "data" : {
+ "type" : "array",
+ "properties" : {
+ "name" : {"type" : "string"},
+ "src" : {"type" : "string"},
+ "params" : {
+ "type" : "array",
+ "items" : [ {"type" : "string"} ]
+ }
+ }
+ }
+ }
+}
diff --git a/tools/b-test/test_data.yaml b/tools/b-test/test_data.yaml
new file mode 100644
index 0000000..3c8c916
--- /dev/null
+++ b/tools/b-test/test_data.yaml
@@ -0,0 +1,13 @@
+#
+# Copyright (c) 2020, Arm Limited and Contributors. All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+#
+
+$schema: ./test_data.schema.json,
+data:
+ - name: "libsp-optee-arm"
+ src: "$TS_ROOT/deployments/libsp/opteesp"
+ params:
+ - "-GUnix Makefiles"
+ - "-DSP_DEV_KIT_DIR=$SP_DEV_KIT_DIR"