This framework is for our own internal use. We decided to release it but, at least in short term, we won't give any help or support about it.
We need a source of truth which is describing all the tests and can be used to generate code, format output etc ...
We have lots of tests. We need to be able to organize them in a hierarchical way
We need a way to identify in an unique way each test to ensure traceability and enable to create history of test results and benchmark.
It is important to keep traceability.
For benchmarking, we may need to vary some dimensions of the tests (like input length). The tests may depend on several parameters (width, height etc ...) We need to be able to specify how those parameters are varied.
For instance, if our test is dependent on a vector size, we may want to compute a linear regression to know how the performances are dependent on the cycle size.
But, our test may also depend on another parameter B which are not interesting us in the regression. In that case, the regression formula should not take into account B. And we would have several regression formula for each value of the parameter B.
The parameters of the tests would be Vector Size and B but the Summary parameter only Vector Size.
A test suite is a set of tests using the same data.
For following requirements, we define a device under tests (DUT) as the place where the function to test is executed. But the test itself (to check that the execution has been successful could be running on the DUT or on a host like a PC).
A test should start (as far as possible) in a clean state. There should not be interferences between the tests.
(CSV, HTML, Text etc ...)
The design is a consequence of all the requirements.
A test description file is defined with a specific syntax to support R1 to R8.
group Root {
class = Root
group DSP Test {
class = DSPTest
folder = DSP
suite Basic Tests {
class = BasicTests
folder = BasicMaths
The tests are organized in a hierarchy. For each node of thee hierarchy, a C++ class is specified. The script processTest.py is generating C++ codee for the group. For the test suite, the script is generating a partial implementation since a test suite is containing tests and you need to add the test themselves.
The patterns, output of tests, parameters are also following a hierarchical structure. But they do not need to be organized in exactly the same way. So, the folder property of a node is optional.
A folder can be reused for different nodes. For instance, you may have a suite for testing and one for benchmarking and both may use the same pattern folder.
A test suite is more complex since it contains the descriptino of the tests and related information.
The simplest suite is just containing functions:
suite Basic Tests {
class = BasicTests
folder = BasicMaths
Functions {
Test arm_add_f32:test_add_f32
}
}
A function is described with some text and followed by the name of the function in the C++ class. The text is used when reporting the results of the tests.
The same function can be used for different tests in the suite. The tests will be different due to different input data or parameters.
A test is requiring input patterns, reference patterns and outputs (to be compared to the reference). Since the test must not know where is the data and how to get it, this information is provided in the test descriptino file.
So, the test suite would be:
suite Basic Tests {
class = BasicTests
folder = BasicMaths
Pattern INPUT1_F32_ID : Input1_f32.txt
Pattern INPUT2_F32_ID : Input2_f32.txt
Pattern REF_ADD_F32_ID : Reference1_f32.txt
Output OUT_SAMPLES_F32_ID : Output
Functions {
Test arm_add_f32:test_add_f32
}
}
A pattern or output description is an ID (to be used in the code) followed by a filename.
The file is is the folder defined with the folder properties of the group / suites.
The root folder for pattern and output is different.
A benchmark will often have to be run with different length for the input. So we need a way to communicate arguments to a function.
We make the assumption that those arguments are integers. In the benchmark results, we may want to generate a CSV (or any other format) with different columns for those arguments.
And we may want to compute a regression formula using only a subset of those arguments.
So, we have the possibility in the suite section to add a parameter section to describe all of this.
suite Complex Tests {
class = ComplexTests
folder = ComplexMaths
ParamList {
A,B,C
Summary A,B
Names "Param A", "Param B"
Formula "A*B"
}
Pattern INPUT1_F32_ID : Input1_f32.txt
In above example we declare that thee functions of the suite are using 3 parameters named A,B and C. We declare that a regression formula will use only A and B. So for each C value, we will get a different regression formula.
We list the names to use when formatting the output of benchmarks. We define a regression formula using R syntax. (We do not write cycles ~ AB but only AB)
Once parameters have been described, we need a way to feed parameter values to a test.
There are 2 ways. First is a parameter file. Problem of a parameter file when it has to be included in the test (C array) is that it may be big. So, we also have a parameter generator. It is less felxible but enough for lot of cases.
Those parameters values, when specified with a file, are described with:
Output OUT_SAMPLES_F32_ID : Output
Params PARAM1_ID : Params1.txt
They follow the outputs and use similar syntax.
When the parameter is specified with a generator then the syntax is :
Params PARAM3_ID = {
A = [1,3,5]
B = [1,3,5]
C = [1,3,5]
}
This generator will compute the cartesian product of the 3 lists.
To use parameters with a function the syntax is:
Functions {
Test A:testA -> PARAM3_ID
} -> PARAM1_ID
PARAM1_ID is the default applied to all functions. In this example we decide to use PARAM3_ID for the testA function.
Pattern files have the following format:
W 128 // 1.150898 0x3f93509c ...
First line if the word size (W,H or B) Second line is the number of samples Then for each samples we have an human representation of the value: // 1.150898
and an hexadecimal representation 0x3f93509c
Output files are only containing the hexadecimal values.
Parameters files have the following format:
81 1 1 1 1 1 3 ...
First line is the number of samples. Then the samples.
First line must be a multiple of the number of parameters. In our above example we have 3 parameters A,B,C. So, the number of possible run must be a multiple of 3 since we need to specify values for all parameters.
Any node (Group, Suite or Function) can be disabled by using disabled { ...}.
A disabled group/suite/test is not executed (and its code not generated fro group/suite). Using disabled for tests is allowing to disable a test without changing the test ID of following tests.
Memory manager is coming from requirement R9 Its API is defined by virtual class Memory. An implementation ArrayMemory is provided which is using a buffer. The details of the APIs are in Test.h
A memory manager can provide new buffer, free all the already allocated buffers and give a generation number which is incremented each time all buffer are released.
According to R13 , the test may be controlled on the DUT or from an external host. It is implemented with a Runner class. The only implementation provided is IORunner,
A Runner is just an implementation of the visitor pattern. A runner is applied to the tree of tests. In case of the IO runner, the IO mechanism and the memory manager must be provided.
According to R12 and R15, tests do not know how to access patterns. It is a responsiblity implemented with the IO, Pattern and PatternMgr.
IO is about loading patterns and dumping output. It is not about IO in general. We provide 2 IO implementations : Semihosting and FPGA.
FPGA is when you need to run the tests in a constrained environment where you only have stdout. The inputs of tests are in C array. The script processTest.py will generate those C arrays.
Patterns is the interface to patterns and output from the test point of view. They will return NULL when a pattern is still referencing a generation of memory older than the current one.
PatternMgr is the link between IO and Memory and knows how to load a pattern and save it into memory.
According to R10 and R11, one must be able to disable tests done on the DUT and dump the output so that the test itself can be done on the host. When instantiating a runner, you can specify the running mode with an enum. For instance Testing::kTestAndDump. There are 3 modes, Test only, Dump only, Test and dump.
For R14, we have a python script which will process the result of tests and format it into several possible formats like text, CSV, Mathematica dataset.
pip install pyparsing
If you want to compute summary statistics with regression:
pip install statsmodels pip install numpy pip install panda
cd Testing python PatternGeneration\BasicMaths.py
mkdir build cmake -DCMAKE_PREFIX_PATH="path/to/tools" -DCMAKE_TOOLCHAIN_FILE=../../armcc.cmake -DARM_CPU="cortex-a5" -DPLATFORM="FVP" -G "Unix Makefiles" ..
cd .. python processTests.py -f desc.txt
You can pass a C++ class to specifiy that you want to generate tests only for a specific group or suite.
python processTests.py -f desc.txt BasicTests
You can add a test ID to specify that you wan to run only a specific test in the suite:
python processTests.py -f desc.txt BasicTests 4
Folder Output/BasicMaths should exist
cd build make VERBOSE=1 "C:\Program Files\ARM\Development Studio 2019.0\sw\models\bin\FVP_VE_Cortex-A5x1.exe" -a Testing > result.txt
cd .. python processResult.py -f desc.txt -r build\result.txt
The result parsing may have generated some statistics in FullBenchmark folder.
The script summaryBench can parse those results and compute regression formula.
python summaryBench.py -f desc.txt -r build\result.txt
The output of this script may look like:
"ID","CATEGORY","Param C","Regression","MAX" 1,"DSP:ComplexMaths",1,"225.3749999999999 + A * 0.7083333333333606 + B * 0.7083333333333641 + A*B * 1.3749999999999876",260
Each test is uniquely identified with the CATEGORY and test ID (ID in the suite). The MAX column is the max of cycles computed for all values of A and B which were used for this benchmark.
To convert some benchmark to an older format. The PARAMS must be compatible between all suites which are children of AGroup
python convertToOld.py -f desc.txt -e AGroup
To add a to sqlite3 databse:
python addToDB.py -f desc.txt -e AGroup
The database must be created with createDb.sql before this script can be used.
In FPGA mode, it is slightly different. The script processTests and processResult must be used with additional option -e
testmain.cpp must be modified
This line
Client::Semihosting io("../TestDesc.txt","../Patterns","../Output");
must be replaced with
Client::FPGA io(testDesc,patterns);
testDesc and patterns are char* generated by the script processTests
To dump the output of the tests, the line
Client::IORunner runner(&io,&mgr,Testing::kTestOnly);
Must be replaced by
Client::IORunner runner(&io,&mgr,Testing::DumoOnly);
or
Client::IORunner runner(&io,&mgr,Testing::kTestAndDump);
and of course, the test must have a line to dump the outputs.
To start the tests you need to:
For a test suite MyClass, the scripts are generating an include file MyClass_decl.h
You should create another include MyClass.h and another cpp file MyClass.cpp
MyClass.h should contain:
#include "Test.h" #include "Pattern.h" class MyClass:public Client::Suite { public: MyClass(Testing::testID_t id); void setUp(Testing::testID_t,std::vector<Testing::param_t>& params,Client::PatternMgr *mgr); void tearDown(Testing::testID_t,Client::PatternMgr *mgr); private: #include "MyClass_decl.h" // Definitions of the patterns you have in the test description file // for this test suite Client::Pattern<float32_t> input1; Client::Pattern<float32_t> input2; Client::LocalPattern<float32_t> output; // Reference patterns are not loaded when we are in dump mode Client::RefPattern<float32_t> ref; };
Then, you should provide an implementation of setUp, tearDown and of course your tests.
So, MyClass.cpp could be:
#include "MyClass.h" #include "Error.h" // Implementation of your test void MyClass::test_add_f32() { // Ptr to input patterns, references and output. // Input and references have been loaded in setUp const float32_t *inp1=input1.ptr(); const float32_t *inp2=input2.ptr(); float32_t *refp=ref.ptr(); float32_t *outp=output.ptr(); // Execution of the tests arm_add_f32(inp1,inp2,outp,input1.nbSamples()); // Testing. // Warning : in case of benchmarking this will be taken into account in the // benchmark. So a benchmark should not contain tests. ASSERT_NEAR_EQ(ref,output,(float)1e-6); }
Then setUp should load the patterns:
void MyClass::setUp(Testing::testID_t id,std::vector<Testing::param_t>& params,Client::PatternMgr *mgr) { Testing::nbSamples_t nb=MAX_NB_SAMPLES; // We can load different pattern or length according to the test ID switch(id) { case MyClass::TEST_ADD_F32_1: nb = 3; ref.reload(MyClass::REF_ADD_F32_ID,mgr,nb); break; } input1.reload(BasicTests::INPUT1_F32_ID,mgr,nb); input2.reload(BasicTests::INPUT2_F32_ID,mgr,nb); output.create(input1.nbSamples(),BasicTests::OUT_SAMPLES_F32_ID,mgr); }
In tearDown we have to clean the test. No need to free the buffer since the memory manager will do it in an automatic way. But if other allocations were done outside of the memory manager, then the clean up should be done here.
It is also here that you specify what you want to dump if you're in dump mode.
void MyClass::tearDown(Testing::testID_t id,Client::PatternMgr *mgr) { output.dump(mgr); }