Visualize CI runs by test type and job category
This visualization is a stacked bar chart with each bar representing a
job category and each colored segment represesenting a test type.
Change-Id: I4b8c470cef40266bf3dabd1f35a135592619a12c
diff --git a/script/graphs/README.rst b/script/graphs/README.rst
new file mode 100644
index 0000000..10724cb
--- /dev/null
+++ b/script/graphs/README.rst
@@ -0,0 +1,21 @@
+Scripts that Generate Graphs
+============================
+
+This directory contains scripts that generate graphs. Each script is run with
+bash and may require additional tools. All of the scripts require gnuplot.
+
+All scripts produce a PNG graph on stdout and the data on stderr.
+
+Test Runs by category
+---------------------
+
+The script `categorize-tests.bash`, and its associated awk and plot scripts,
+generate a stacked bar chart with bars representing groups of tests (l1 l2,
+etc.) and segments of the bars representing types. `categorize-tests.bash`
+accepts an argument to filter the tests included with grep.
+
+For example, the following will produce a graph of the juno-specific tests:
+
+ bash categorize-tests.bash juno > juno-tests.png 2> juno-tests.txt
+
+*Copyright (c) 2021, Arm Limited. All rights reserved.*
diff --git a/script/graphs/categorize-tests.awk b/script/graphs/categorize-tests.awk
new file mode 100644
index 0000000..b590c65
--- /dev/null
+++ b/script/graphs/categorize-tests.awk
@@ -0,0 +1,50 @@
+#
+# Copyright (c) 2021 Arm Limited. All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+#
+#!/usr/bin/env awk
+#
+# This is a script to categorize tests within this repo by type. This script is
+# intended to be run with the output of `find group -type f`, run from within
+# the root directory of this repo piped into it. See the bash script with the
+# same name for an usage example.
+
+BEGIN {
+ # We're breaking records upon the "/" character so that we can have an
+ # aggregation that's keyed by test group, if we want to.
+ FS = "/";
+}
+
+# Here we filter out any records without exactly 3 fields (i.e. 3-level paths)
+# and categorize the rest.
+NF == 3 {
+ if (/-l1/) category = "\"l1 - Every Patch\"";
+ else if (/-l2/) category = "\"l2 - Risky or Big Patches\"";
+ else if (/-l3/) category = "\"l3 - Daily\"";
+ else if (/-manual/ || /-release/ ) category = "\"remainder - Every Release\"";
+ else if (/-unstable/) category = "\"unstable - Never Run\"";
+ else category = "\"remainder - Every Release\"";
+ cats[category] = 1
+ # Each of these categorizes a test into a category, based on a regular
+ # expression. When you add another test category, you should also add
+ # printing to the print group loop below.
+ if (/linux/ || /uboot/ || /edk2/ || /:fvp-([a-z0-9.]-)*spm/ || /:juno-([a-z0-9.]-)*scmi/) integration[category] += 1;
+ else if (/tftf/) component[category] += 1;
+ else if (/coverity/ || /misra/ || /scan_build/) static[category] += 1;
+ else if (/:nil/ || /norun/) build[category] += 1;
+ else print $0 " No test category; excluding from data" >> "/dev/stderr";
+}
+
+
+END {
+ for (name in cats)
+ # This prints a single test group, by name. When you add another
+ # category (with another map), add another field to this print.
+ printf("%s %d %d %d %d\n",
+ name,
+ build[name],
+ static[name],
+ component[name],
+ integration[name]);
+}
diff --git a/script/graphs/categorize-tests.bash b/script/graphs/categorize-tests.bash
new file mode 100644
index 0000000..92d286c
--- /dev/null
+++ b/script/graphs/categorize-tests.bash
@@ -0,0 +1,64 @@
+#
+# Copyright (c) 2021 Arm Limited. All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+#
+#!/usr/bin/env bash
+set -euo pipefail
+
+# This script plots the categories of tests by group. It does this by combining
+# the awk script and gnuplot script of the same name. This script accepts an
+# argument, in the format of a grep expression, that allows the tests to be
+# filtered before categorization. This script produces the plot as a png on
+# stdout.
+
+# Variables
+# ^^^^^^^^^
+#
+# We are located in a specific location with this repo, so we can take
+# advantage of that to avoid any issues with running this from an unexpected
+# directory.
+rootdir=$(realpath $(dirname $(realpath $0))/../..)
+
+# I would like to use process-substitution for this, so that we can avoid
+# making a file on disk and keep everything in memory, removing the need to
+# clean anything up on exit and preventing any chance of polluting the user's
+# filesystem. However, when gnuplot is asked to plot from the same file more
+# than once, it will seek to the start of the file for every subsequent plot
+# after the first. Unix Pipes do not support this operation, and plotting fails
+# under these circumstances. Instead, we use an intermediate file, which is
+# removed on success.
+categories=$(mktemp "XXXXXXX-test-categories.dat")
+
+# We change a portion of the title for our graph based on the argument passed
+# to this script.
+title=$(if [[ $# -ge 1 ]] ; then echo $1 ; else echo "All Tests" ; fi)
+
+# Generate Data into the $categories file
+# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#
+# The following pipeline is the heart of the implementation, and has four
+# stages: find, ???, awk, and sort. The ??? stage of the pipeline is determined
+# by the bash if statement, which switches between a filter, when an argument
+# is passed, and a passthrough, implemented as `cat -`, when no filter
+# argument is passed.
+#
+# Note that the env -C before the find is to enforce that it produces
+# directories relative to the $rootdir, so that it does not trip up the awk
+# script.
+env -C $rootdir find group -type f |\
+ if [[ $# -ge 1 ]] ; then
+ grep -e "$1" -
+ else
+ cat -
+ fi | awk -f ${0%bash}awk | sort > $categories
+
+# Generate a Plot (on stdout)
+gnuplot -e "subtitle='$title'" -c ${0%bash}plot $categories
+
+# Dump data to stderr
+echo name build static component inegration 1>&2
+cat $categories 1>&2
+
+# Clean up temporary files
+rm $categories
diff --git a/script/graphs/categorize-tests.plot b/script/graphs/categorize-tests.plot
new file mode 100644
index 0000000..05fe5b2
--- /dev/null
+++ b/script/graphs/categorize-tests.plot
@@ -0,0 +1,15 @@
+#
+# Copyright (c) 2021 Arm Limited. All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+#
+set terminal png enhanced font ",18" size 1920, 1080
+set style data histograms
+set style histogram rowstacked
+set boxwidth 0.5 relative
+set style fill solid 1.0 border -1
+set title "Incremental Tests Enabled at each CI level for ".subtitle
+plot ARG1 using 2:xtic(1) title "Build-only",\
+ '' using 3 title "Static (MISRA, etc.)",\
+ '' using 4 title "Component",\
+ '' using 5 title "Integration (boot Linux, etc.)"