aboutsummaryrefslogtreecommitdiff
path: root/tools/tf_fuzz/regression/README
blob: d061dbacc8a5d0b311303bb300926ff5f2150ac2 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
.../tf_fuzz/tf_fuzz_regression directory contents:

 000001_set_sst_uid_data_expect_pass
 000002_set_sst_name_data_expect_nothing
 000003_set_sst_name_data
 000004_set_sst_name_rand_data
 000005_set_sst_rand_name_rand_data
 000006_set_sst_multi_name_rand_data
 000007_set_sst_multi_uid_rand_data
 000008_set_sst_name_rand_data_read_check_wrong
 000009_set_sst_name_rand_data_read_check_var_read_print
 000010_read_nonexistent_sst_check_string
 000011_read_nonexistent_sst_check_string_expect_pass
 000012_read_nonexistent_sst_check_string_expect_other
 000013_set_sst_name_rand_data_remove_twice
 000014_set_sst_name_rand_data_remove_other
 add_these_tests
 README
 regress
 regress_lib

--------------------------------------------------------------------------------

This is the beginnings of a regression suite for TF-Fuzz.  That is, tests to
make sure that tf_fuzz is still functioning properly after making changes.

This is neither complete nor working yet, but is close to its first-pass
implementation.

Here's the basic scheme of it all:

*  "bash regress" from this directory runs regression.  It will fail with an
   error if a problem is found.  If it runs to completion, then regression has
   passed.

*  Each test is in its own sub-directory containing these files, by name (always
   same name):

  *  template:  The test-template file to be run though the TF-Fuzz under test,
     here called "the DUT TF-Fuzz" here.

  *  exp_stdout_stderr:  The *expected*, combined stdout and stderr from running
     TF-Fuzz in verbose mode (-v).  This file contains wildcard expressions to
     be checked (more on that below).

  *  exp_test.c:  The *expected* output C code.  This file also contains
     wildcard expressions to be resolved against the DUT TF-Fuzz output (again,
     more on that below).

  *  stdout_stderr (if present):  The combined stdout and stderr from running
     the DUT TF-Fuzz in verbose mode (-v), during regression testing.

  *  test.c (if present):  The output C code generated from running the DUT
     TF-Fuzz in verbose mode (-v), during regression testing.

  *  diff_stdout_stderr (if present):  The "diff" of stdout_stderr against
     exp_stdout_stderr.  As mentioned above, there are wildcards in some lines
     of exp_stdout_stderr that have to be resolved against stdout_stderr.

  *  diff_test.c (if present):  The "diff" of test.c against exp_test.c.  As
     mentioned above, there are wildcards in some lines of exp_test.c that have
     to be resolved against test.c.

  *  check.py:  This Python script doesn't yet, but will, resolve the wildcard
     differences in diff_stdout_stderr and diff_test.c.  Each test directory
     has its own script customized to the needs of that particular test, but
     they all draw from functions in the regress_lib directory.

To explain how check.py checks -- or will check! -- a regression test, consider
one example of the ./000005_set_sst_rand_name_rand_data/diff_test.c file, below:

    45,46c45,46
    <     static uint8_t gibberish_data[] = "Gof jav ofdomviv wazauyicef zoc xut rus sekneiqiv eidzai yefrabxiyob abjie pah jashui ziuven qetuvraqwiu omwid xenmav fipwiy meftofc.";
    <     int gibberish_data_size = 133;
    ---
    >     static uint8_t gibberish_data[] = "@@002@10@@********";
    >     int gibberish_data_size = ********;
    51,52c51,52
    <     /* Creating SST asset "gibberish," with data "Gof jav of...". */
    <     sst_status = psa_ps_set(8617, gibberish_data_size, gibberish_data, PSA_PS_FLAG_NONE);
    ---
    >     /* Creating SST asset "gibberish," with data "@@002@10@@...". */
    >     sst_status = psa_ps_set(@@@001@@@, gibberish_data_size, gibberish_data, PSA_PS_FLAG_********);
    60c60
    <     psa_ps_remove(8617);
    ---
    >     psa_ps_remove(@@@001@@@);

So, by this process, a simple "diff" itself checks the vast majority of the
correctness of the result.  check.py has only to resolve the wildcards, and in
the process will also immediately see cases where "diff" discovered disparities.

The wildcards in the exp_stdout_stderr and exp_test.c files are of three basic
natures, using the examples shown above (please reference them above to
clearly understand the ideas here):

  ******** (exactly 8 *s):
      This denotes any pattern of characters, until the expected and actual
      character streams re-converge (sync up) again.  The characters in the
      output from stdout_stderr or test.c from the DUT TF-Fuzz, that
      correspond to the ******** in the exp_stdout_stderr or exp_test.c files
      are *not* checked;  anything in the output from the DUT TF-Fuzz at this
      position is acceptable.

  @@@001@@@ ("@@@", a pattern number, "@@@"):
      This denotes a particular pattern of characters, until the expected and
      actual character streams re-sync again.  The important thing, however,
      is that what this wildcard stands for *must be consistent* throughout the
      comparison!  In this case above, @@@001@@@ in the exp_test.c must consis-
      tently correspond to "8617" everywhere throughout the test.c file.  The
      number between the two "@@@"s in the wildcard designates which pattern
      must consistently match.

  @@002@10@@ ("@@", a pattern number, "@", a pattern size, "@@"):
      This is a slight variant upon the previous wildcard, in which a specific
      match length is required.  In lines 45 and 46 above, random data generated
      consists of 10 characters (thus the ...@10@@ in the wildcard) "Gof jav of"
      followed by other characters we don't care about;  they can be anything.
      Thus "@@002@10@@********" in line 45 of exp_test.c:  the "@@002@10@@"
      denotes a pattern number 002 for a length of 10 characters that must match
      "Gof jav of" in this case, followed by some arbitrary number of characters
      we don't care about, thus ******** in exp_test.c, line 45.

After the check.py capability -- resolving these wildcards -- for this purpose
is fleshed out, we shall have to figure out how to address "shuffle" and
"2 to 5 of {}" randomizations.

--------------------------------------------------------------------------------

The add_these_tests directory contains regression-test information of the above
nature that the regression framework is not currently able to address.

--------------

*Copyright (c) 2019-2020, Arm Limited. All rights reserved.*