Unit Testing of Workers

OpenCPI supports unit testing where a <component>.test directory in a component library is created to hold a test suite (called a component unit test suite) for all the workers in the library that implement the same spec (OCS). The workers that are tested could be written to different authoring models or languages or simply be alternative source code implementations written to the same authoring model.

E.g. if a library contained fft.hdl and fft.rcc and fft_xilinx_dsp.hdl workers that all implemented the fft-spec.xml OCS file in the library’s specs/ directory, a single fft.test directory would be created to hold a test suite that tested them all. Each <component>.test directory is associated with a single OCS and has an XML file (called an OpenCPI Test Suite Description (OTSD)) named <component>-test.xml. This XML file is described below in Unit Test Description XML File.

The OpenCPI unit test framework manages multiple dimensions of worker testing, with automation to minimize test design and preparation efforts. The dimensions are:

Test cases (individual parameterized tests, possibly using different runtime values)
Platforms (HDL hardware and simulation platforms, RCC/OCL Platforms)
Workers (different source code implementations, different models)
Worker build configurations (compiled in vs. runtime settable property values)

The unit test framework allows complex test scenarios while providing layered complexity to keep simple test cases very simple to define and execute. Test inputs and outputs can be pre-prepared data files (i.e. test vectors), or be developer-provided scripts for input data generation and output data verification.

Unit testing proceeds in five phases, although it is most common to group them into two steps: build (phases 1-2) and run (phases 3-5). The phases are:

Generate — generate testing artifacts after finding the spec and the workers to test
Build — building HDL bitstream/executable artifacts for testing (if any HDL workers are found to be tested)
prep — examine available built workers and available runtime platforms, and create execution scripts to execute all feasible tests.
run — execute tests for all workers, configurations, test cases and platforms
Verify — verify results from the execution of test cases on workers and platforms

In most cases, ocpidev build is used to perform the generate and build phases, while ocpidev run, performs the prep, run, and verify phases.

The rest of this section will describe:

The terminology used in this testing framework
The phases of unit testing in detail
What is required and possible in the test description XML file: <component>-test.xml
How to support data generation and verification
How to execute test cases and verify their results

A test suite, in a <component>.test directory, can be created and initially populated using this command:

ocpidev create test <component>

This will create the directory and the one necessary file: <component>-test.xml.

Unit Test Concepts and Terminology

Here are the terms used in the OpenCPI unit test framework:

Component unit test suite (aka test suite) – as embodied in the <component>.test directory, is a suite of test cases for testing all workers implementing a component spec across all available platforms for which the workers have been built.

Test matrix —the virtual multidimensional space of testing, across:
Workers — the different source code or authoring model implementations
Worker configurations — different parameter value sets for worker builds
Initial property values — runtime property value configurations
Platforms — possible runtime environments
User-defined test cases — with property values, inputs and outputs

Test case – a parameterized test
Using a defined set of inputs or generation scripts
Using a defined set of outputs or verification scripts
Using a defined matrix of property values

Test subcase – a very specific test
Defined by and generated from a test case
Using a specific worker build configuration
Using a specific set of property values
Not bound to a specific platform or artifact

Generator — a script to create input data files for ports, or property value files

Called for a subcase, with all property values supplied. Fixed input files can be provided that do not require generation scripts.

Verifier — a script to verify test output data produced by output ports

Called for a subcase, supplied with each port’s output data file, with all property values supplied, both initial and final (volatile). Fixed “golden” output files can be used in place of verification scripts.

Viewer — a script to view the results of a subcase execution. (e.g. plot).

Default Test Case — the case that is automatically created when none are specified
Tests all parameter combinations as derived from all worker parameter/build configurations or all workers
Developer can supply runtime property settings with multiple values for each, resulting in the cross-product of subcases
One generation script (input) and one verification script (output), per port, parameterized by subcase property values

The Phases of the Unit Test Process

The generate(1) phase performs the following tasks automatically without any developer involvement other than to prepare the unit test XML file:

Discovers the OCS associated with this test directory.
Discovers workers in the same library that implement that OCS.
Discovers the build configurations (parameter values) for each worker.
Derives a baseline for test cases based on all the build configurations that have been used on any worker.
Derives the actual tests appropriate for all parameter combinations vs. the actual worker build configurations they apply to.
Generates XML applications (OAS files) that perform unit tests on all workers
Generates HDL test assemblies (subdirectories and OHAD files) that can be built for HDL platforms (hardware and simulation).
Creates a report in the gen/cases.txt file listing the parameter values used in all test cases and subcases.

Although there are many options described below, the default generated unit test applications feed specified input data to input ports of the worker being tested and cause EOF to be sent to input ports after the input data is sent.

Generated test applications are considered finished when either:

The component being tested has a single output port and an EOF is received from that port.
The component being tested has multiple output ports and an EOF is received from the first one.
The port designated as the finishPort and an EOF is received from that port.
The worker itself is designated as the “finish” worker for the application and it enters the finished state.
A specified time duration for the test has been exceeded.

After the test completes, the output data from all output ports as well as the final values of all properties are available to the verification process for assessment.

All the above tasks in the generate(1) phase are done “off-line”, without regard to:

Which worker artifacts are built
Which platforms are available on which to execute tests
The availability of any build-related tools (compilers or FPGA synthesis tools or simulators).

The build(2) phase is used for testing HDL workers, and builds the generated HDL test assemblies for whichever platforms (including simulators) are specified. When building for hardware HDL platforms, this phase takes the longest. There must be tools available to build the specified platforms, but this phase does not require those platforms be available for execution. This phase does nothing when only RCC workers are available. Running ocpidev build on a unit test suite performs both the generate phase and the build phase.

The prepare(3) phase does the necessary work to prepare to actually execute test cases and perform associated verification. In this phase the unit test framework automatically does:

Discovery of available execution platforms, local and remote (reachable via network)
Discovery of available built artifacts that can be executed on available platforms (including generated HDL test assemblies generated in the generate(1) phase)
Generation of test scripts to perform all feasible tests on all available platforms.

After preparation, the developer invokes the execute(4) and verify(5) phases. These phases can be sequential (all executions followed by all verifications), or interleaved (each test subcase is executed and verified before executing the next one).

During execution and verification, there are various filtering capabilities to subset which tests are run, which platforms should be tested, and whether a test case failure should immediately stop the testing process.

OpenCPI Test Suite Description (OTSD)

The <component>-test.xml is the OpenCPI Test Suite Description (OTSD), which specifies test cases and the defaults that apply to all test cases. As with all OpenCPI XML files, element names and attribute names are case insensitive. The top-level element in the file is <tests>, with the possible child elements being:

<input> to define an input file or generator script usable by any test case

<output> to define an output file or verifier script usable by any test case

<property> to define property values for all test cases

<case> to define a non-default test case when needed

If no <case> element is defined, the default test case is used. This is a common situation since the default test matrix is based on the parameters that workers are built with, and the available workers that implement this component spec. Here is an example file using the default test case for a component:

<tests>
  <input port='in' script='generate.py 16'/>
  <output port='out' script='verify.py 16384 16' view='view.sh'/>
  <property name='phs_inc' values='-4096'/>
  <property name='enable' values='0,1'/>
</tests>

It specifies that the default test case should be used (no <case> elements), the “generate.py 16” command should be issued to generate test data for port in, the “verify.py 16384 16” command should be issued to verify output data from port out, the phs_inc property should always be tested set to -4096, and the enable property should be tested with values 0 and 1. All scripts are run per subcase and have access to the parameter properties as well as the runtime properties of the subcase being tested.

Several attributes described below refer to scripts that will be executed by the unit test framework. In all cases, scripts must properly return process/shell exit status, with zero indicating success and non-zero indicating failure. This is true regardless of the language used in the script.

Attributes for the Top-level `<Tests>` Element

The valid attributes for the top-level tests element apply to all test cases and are spec, UseHdlFileIO, ExcludeWorkers, OnlyWorkers, ExcludePlatforms, OnlyPlatforms, DoneWorkerIsUUT, TimeOut, and Duration. All are optional and are specified in special situations.

Spec Attribute of the Top-level `<Tests>` Element

Normally the spec (OCS) for all the workers being tested is inferred from the name of the <component>.test directory, and found in the file:

../specs/<component>-spec.xml

When this is not the case, this spec attribute can specify the name of the spec file for this test suite, much like the same attribute can be used in a worker’s OWD.

UseHdlFileIO Attribute of the Top-level `<Tests>` Element

This boolean attribute applies only when HDL workers are being tested on simulation platforms. When true, it indicates that file I/O between the worker being tested and the input and output test files is done in the simulator using VHDL/Verilog file operations directly. When false (the default), the file I/O is being done by file reading and writing RCC workers running outside the simulator, with the data flowing in and out of the simulator. Both settings can be useful, but the true setting generally results in faster simulation times since less logic is being simulated for this file I/O.

ExcludeWorkers Attribute of the Top-level `<Tests>` Element

This string attribute specifies a list of comma-separated workers (e.g. fft.hdl) that should not be tested, even if they implement the spec of this test suite.

OnlyWorkers Attribute of the Top-level `<Tests>` Element

This string attribute specifies a list of comma-separated workers that should be the only ones tested. Any others found to implement the same spec will be ignored.

ExcludePlatforms Attribute of the Top-level `<Tests>` Element

This string attribute specifies a comma-separated list of platforms that should not be tested. Wildcard patterns may be used for any name. E.g. “*sim” would exclude any platform that ended with the letters sim. Any other available platforms that have built artifacts will be used.

OnlyPlatforms Attribute of the Top-level `<Tests>` Element

This string attribute specifies a comma-separated list of platforms that should be the only ones tested. Wildcard patterns may be used for any name. E.g. “*sim” would test only platforms that ended with the letters sim. Any other available platforms will be ignored.

Duration Attribute of the Top-level `<Tests>` Element

Normally unit test cases execute until the test subcase application is done, which is usually when an EOF is asserted on the first or only output port of the worker being tested. See the EOF Assertion section.

When this default EOF behavior is not viable or desirable (e.g. when the workers being tested have no output ports), the duration attribute can be set to an amount of time the application should run before being considered successfully done.

The duration value is in seconds, and represents wall clock time for the execution. After this amount of clock time, the execution is stopped, and the execution is considered successful and any outputs or final property values are provided for verification.

This attribute is separate from the timeout attribute, which indicates when the execution should be considered failed. Only one of these attributes may be set.

Timeout Attribute of the Top-level `<Tests>` Element

The timeout attribute indicates an amount of wall clock time in seconds after which the execution of a test subcase is considered a failure. Setting this value prevents an execution from hanging and preventing completion (but failure) of the subcase. This is especially useful for non-interactive scripted regression testing. Only one of the duration or timeout attributes may be set. This attribute has a default of 3600 (one hour).

FinishPort Attribute of the Top-level `<Tests>` Element

The finishPort attribute specifies which of the component’s output ports is used to indicate that the test case is finished. The default behavior of the unit test framework is to designate the file_write component connected to the first output port listed in a component’s spec as the done worker in all the unit test applications. Setting finishPort will change the done worker to the file_write connected to the designated output port. The value of the finishPort attribute is the name of the output port. This top-level setting becomes the default for all test cases, but this setting can be overridden for any test case.

<tests finishPort='out3'>

FinishedWorkerIsUUT Attribute of the Top-level `<Tests>` Element

The finishedWorkerIsUUT boolean attribute indicates that, as a default for all test cases, the worker being tested (the “unit under test” or UUT) is the finished worker in unit test applications. The finished worker is the worker which, when finished, indicates that the application is finished. The default is that the file_write component connected to the first output port is the finished worker. If the component being tested has no output ports, then the default is in fact as if this attribute is set to true. I.e., this attribute is only needed when the component does have output ports, but for some reason they should not be used to indicate when the unit test applications are finished. This global setting can be overridden for any test case.

<tests finishedWorkerIsUUT='true'>

Input Element of Top-level `<Tests>` Element

An <input> element as a direct child of the top-level tests element specifies a source of input data that can be used by test cases. It is not specific to a test case but may be used by any test case for any input port. Its allowable attributes are: name, port, file, script, messageSize, messagesInFile, suppressEOF, and stressorMode.

Name Attribute of the Input Element

This optional string attribute specifies the name of this input source, so it can be referenced by test cases that use it, by name. If it applies to all cases, it doesn’t need a name. If it applies only to a specific port, the port attribute can be set, which is more common. One of name and port must be specified.

Port Attribute of the Input Element

This optional string attribute specifies the name of the port that this input source will always apply to. If there is only one input source for a port, it will be used for all cases. One of name and port must be specified.

Script Attribute of the Input Element

This string attribute indicates a command to execute to produce data. When data is generated for a subcase and for a port, this command will be issued. The attribute value is not just the name of a file to execute, but of a command, so it can have a command name followed by some command arguments. When the command is executed in order to produce data, it will be appended with the name of the file to be written into; i.e. the script’s job is to write into the file whose name is at the end of the command. Thus if the value of this attribute was:

echo hello >

then the source of data would always be a line of text containing hello since the actual command executed would be:

echo hello > <output-file-supplied-by-unit-test-framework>

The way these scripts become more useful is that all parameter and initial runtime property values are supplied to the script as environment variables. Thus this script is parameterized by these values for the subcase being generated. Accessing environment values is easy for the scripts, whether they are written as shell scripts, python, or C. When a script is executed for a subcase (and for a port), the value of each parameter and runtime property is the value of an environment variable named:

OCPI_TEST_<prop>

So, if the property’s name was myprop:

In C or C++, the value (as a string) would be: getenv("OCPI_TEST_myprop")

In python, it would be: os.environ.get("OCPI_TEST_myprop")

In bash/shell, it would be: $OCPI_TEST_myprop

Only parameter properties or runtime properties that are initial or writable are present. Using scripts based on these values normally means one script can be applied to all test cases.

The command is executed by the shell in the <component>.test directory, and must have execute permissions.

File Attribute of the Input Element

This string attribute specifies the name of a file to be used as the source of data. It is not affected by any property values and is thus a “constant”. This is useful if the same input data should be used for a port for all test cases, or if the file is not easily generated by a script, but is used for some test cases.

MessageSize Attribute of the Input Element

This positive integer attribute specifies the size of messages to be supplied to the port of the worker under test when this data source is being used. Since data flowing between ports always consists of messages, this determines their size. The data from this input source is split into messages of this size, in bytes. This attribute is the same as the property of the file_read worker.

MessagesInFile Attribute of the Input Element

This boolean attribute indicates that the data produced by this input source has message boundaries and opcodes embedded in the data. Each message in the input file is preceded by a header consisting of two 32 bit unsigned integers (little endian), with the first being the length of the message in bytes, and the second being the opcode (with only the low-order 8 bits used). This attribute is the same as the property of the file_read or file_write workers.

TestOptional Attribute of the Input Element

This boolean attribute indicates that an input is to be unconnected, to test its optionality. The default is that all optional ports will be connected to test components. Setting testOptional to true will force the port to be left unconnected. It can also be used in case elements.

SuppressEOF Attribute of the Input Element

This boolean attribute indicates no EOF indication should be provided to the input port after all the data (messages) has been provided as input. This attribute is the same as the property of the file_read worker and is only valid for inputs. The default behavior is to provide such EOF indications.

StressorMode Attribute of the Input Element

This attribute only applies when testing HDL workers and is only valid for input ports. The generated test benches for HDL workers include a stressor function at each input port which varies the range of input behavior to provide coverage of what the worker might see at its port in applications. This optional enumeration attribute specifies the stressor behavior at this port. The term metadata is used here to indicate the signals indicating message boundaries and whether the boundaries are separate from or coincident with, the first or last data word of a message. These signals are som, eom, and valid.

There are four modes available: bypass, throttle, metadata, and full. The meanings are:

bypass
all messages are passed through to the worker under test without change or delay.

throttle
the metadata of all received messages is passed through to the worker under test without change, but data within the message will be withheld at random intervals to create intra-message data starvation.

metadata
the stressor at this port will cycle through a pattern of all valid metadata combinations, but the data will not be throttled.

full
the stressor will cycle through a pattern of all valid metadata combinations, data within the message will be throttled at random intervals to create data starvation, and idle cycles will be inserted between messages.

The default mode is bypass.

Output Element of `<Tests>` Top-level Element

An output element as a direct child of the top-level tests element specifies how the output data from a port of the worker being tested may be verified for correctness. The valid attributes are: name, port, file, script, stopOnEOF, view and disableBackpressure. It may 1) be applied to all test cases, 2) be used as a default for test cases that do not mention an output element for a port, or 3) be referred to by name by some test cases.

It is very similar to the input element:

The name attribute allows this element to be referred to in test cases.
The port attribute specifies that this element should be used for a particular port.
The file attribute specifies an existing file to compare the output data to for correctness.
The script attribute specifies a command that takes a file name as the data to verify.

Script Attribute of the Output Element

This attribute is similar to the script attribute of input and property elements. The major difference is that there are multiple arguments appended to the command instead of one. The first is an input file that contains the output of the given output port as a result of executing the worker in a subcase. After that first file name argument there are file name arguments for each input port of the component that contain the input data supplied to that port, in the order the ports are declared in the OCS. This allows the script to not only access the resulting output data from an output port, but also access the data supplied to each input port (if needed for the verification).

For example, if the component had input ports in1 and in2, and an output port name out, and a script attribute value of <command>, the actual command executed would be:

<command> <output-from-port-out> <input-to-in1> <input-to-in2>

A second important difference for the output script vs. an input script is that the final values of writable and volatile properties are available in the environment in addition to the initial values of all other properties. For generated properties (those with a script attribute in its property element), the name of the generated property file is placed in the environment variable named OCPI_TESTFILE_<property-name>, while the final value is still in the OCPI_TEST_<property-name> environment variable.

The name of the test case is in the OCPI_TESTCASE environment variable and the name (which is numeric) of the subcase is in the OCPI_TESTSUBCASE environment variable. E.g. if the subcase being run was case43.03, the case name is case43 and the subcase name is 03.

As with all other scripts, a process/shell exit status of zero indicates success, while a non-zero exit status indicates failure. The script may write other informational messages about the failure to stderr which will be logged. The script should not write simple success and failure (PASS/FAIL) messages since the unit test framework does that already, using green/red colors for PASS/FAIL, based on the exit status.

View Attribute of the Output Element

This optional string attribute operates similar to the script attribute, but has a different purpose. It provides a convenient way for the developer to ask for a “view” of the data for the port. Taking all the same arguments as the verification script (in the script attribute), it is expected to present the data in some useful way during test development, typically in some viewing or plotting window.

TestOptional Attribute of the Output Element

This boolean attribute indicates if an output is to be unconnected, to test its optionality. The default is that all optional ports will be connected to test components. Setting testOptional to true will cause the port to be unconnected during the test case.

StopOnEOF Attribute of the Output Element

This boolean attribute indicates that reception of an EOF indication should stop the recording of output data from an output port. The default value is true (which is unusual for a boolean attribute). This attribute is the same as the property of the file_write worker and is only valid for outputs. The reception of EOF indications on the first or only output port is used to indicate the end of execution of a test subcase unless the duration attribute is set for the case (or as a top-level default for all cases).

DisableBackpressure Attribute of the Output Element

This boolean attribute only applies to output ports of HDL workers under test and specifies the backpressure (i.e. flow control) behavior the worker will see at this output port during testing. Setting disableBackpressure to true will disable the backpressure applied to this port. When false (the default) the test bench will apply random backpressure to this port.

Property Element of `<Tests>` Top-level Element

This element specifies the default set of values for a property for all test cases, unless overridden in particular test cases. When multiple values are specified, the implication is that subcases should be generated that test each of the specified values.

For parameter properties, where the potential set of test values is normally derived from the values used to build the workers being tested, the values specified in this element act as a filter or subset of those values, since no tests can be performed for parameter values that are not used in any worker’s build configuration.

The allowable attributes for the property element are: name (required), test, value, values, valueFile, valuesFile, generate, add, only, and exclude. Exactly one of the value* attributes must be specified. The textual syntax for property values is used, as described in the Property Value Syntax and Ranges topic.

Name Attribute of the Property Element

This required string attribute identifies a property defined in the OCS of the test suite or in the OWD for a worker being tested. The values specified in other attributes are applied to this specified property during testing (except when the test attribute is true – see below).

Value Attribute of the Property Element

This attribute specifies a single value to be tested.

Values Attribute of the Property Element

This attribute specifies a comma-separated sequence of values to be tested.

ValueFile Attribute of the Property Element

This attribute specifies the name of a file containing a single value to be tested. Multiple lines in the file are considered elements of a sequence or array value. In fact, new-lines in the file are equivalent to commas, so anywhere in the value syntax that commas occur, they can be newlines in the file.

Property value syntax is described in the Property Value Syntax and Ranges topic.

ValuesFile Attribute of the Property Element

This attribute specifies the name of a file containing multiple values to be tested. Multiple lines in the file are considered separate values to be tested. Multiple values can also be specified on a single line in the syntax of a sequence of values of the type of the property. E.g., if the type is Ulong, the ValuesFile file could contain a single line of 1,2,3,4 or four lines containing the four values.

Generate Attribute of the Property Element

This attribute specifies a command to execute to create a file containing a value to be tested. An argument is added to the specified command that is the name of the file to be written. All parameter and initial runtime property values are available to the script as environment variables. This feature is convenient when a property value depends on others in a complex way. Note that expressions can be used in the value attributes of property elements, so scripts are not necessary to perform simple arithmetic based on other parameters.

The generated file should be in the format used for the valueFile attribute, as described above in the ValueFile Attribute of the Property Element topic.

Test Attribute of the Property Element

This optional boolean attribute, when true, indicates that this property is a test property that is not a property of the workers being tested. It is a property whose value is available to all input generation, output verification, output viewing and property generation scripts. Its name must be different than all property names in the OCS or in any of the workers’ OWDs.

Values assigned to a test property are used to generate other test cases not defined simply by the values of worker properties. When this attribute is true, other data type attributes used in an OCS property element, such as type or arrayLength, may be applied to this property element since it is in fact defining a property.

Exclude/Only/Add Attributes of the Property Element

These attributes limit certain values to certain platforms. The exclude and only attributes provide value-specific restrictions on platforms similar to the effect of the top-level excludePlatforms and onlyPlatforms attributes: they exclude certain platforms from testing certain values or specify that some values should only be tested on certain platforms. The add attribute specifies that the values should be added to the set of tested values only for certain platforms.

The syntax of these attributes is the same as mentioned in the ExcludePlatforms Attribute of the Top-level <Tests> Element topic.

`<Set>` Child Element of the `<Property>` Element for Delayed Values

Normally any non-parameter initial property values are set before the test case application is started (after initialize and before start). This means that such values are stable during the life of the test. In some cases it is useful to set writable property values during execution.

The set child element can be used to “schedule” the setting of a property value at some time after the application is started. Delayed property settings in applications are described in the OpenCPI Application Development Guide, and the syntax here is similar: the set child element can occur more than once, and have delay and value attributes to specify the delay after start (in seconds, floating point), and a value to set at that time (relative to when the application is started). A delay of zero causes the value to be set immediately after the application is started. The following example:

<property name='myctl'/>
  <set delay='1' value='10'/>
  <set delay='1.001' value='20'/>
</property>

sets the myctl property to 10 one second after start, and sets the value to 20 one millisecond later.

Case Element of `<Tests>` Top-level Element

The case element defines a non-default test case when the default test cases are not sufficient. It is necessary when the automatic parameterization of the default test case is insufficient for testing the worker(s).

The allowed attributes of a case element are: Name, OnlyWorkers, ExcludeWorkers, OnlyPlatforms, ExcludePlatforms, DoneWorkerIsUUT, Duration and Timeout. All but Name have the same function as previously defined for the top level tests elements, but only apply to this case.

If ExcludePlatforms is set at the top level, setting it again at the case level adds to the list of platforms that will not be tested. Setting OnlyPlatforms at the case level if ExcludePlatforms is set at the top level is valid, but OnlyPlatforms at the case level cannot list a platform that has been excluded at the top level.

If OnlyPlatforms is set at the top level, setting it again at the case level narrows the list of platforms that will be tested further, but it is not valid to set OnlyPlatforms at the case level to a platform not in the top level OnlyPlatforms list. It is valid to set ExcludePlatforms at the case level if OnlyPlatforms is set at the top level, but excluding a platforms not in the top level list has no effect.

ExcludeWorkers and OnlyWorkers at the top and case levels interoperate in the same way as ExcludePlatforms and OnlyPlatforms.

DoneWorkerIsUUT at the top level will apply the attribute to all cases, but can be set at the case level to apply to only select cases.

The allowed child elements under a case element are: input, output, and property.

Each case can override or use the default inputs and outputs for each port, and each case can override the property values tested for each property. If no input or output is defined for an input/output port, then the default input/output is used (the input/output defined for the port under the top-level tests element). If no property element is present for a property under a case element, then the values defined at the top level are used. For parameter properties, the default values tested are derived from the values defined in all the workers’ build configurations, but this automatic default can still be overridden (limited) by a property element at the top level or under a case element.

Name Attribute of Case Elements

This optional string attribute specifies the name of the test case. If not present, the name of the case is case followed by a case number starting at zero, with at least 2 digits (i.e. the second case would be case01, and the 101^st case would be case100). The name of a case is listed in various reports, and can be used when specifying that only certain cases (rather than all cases) should be executed or verified.

Input Element under Case Elements

This element specifies how input data is generated for a port, in a test case. If not specified, the default input source for the port specified at the top level is used. The port attribute of the input element specifies the port this input element applies to. The name attribute, when present, indicates that a specifically named input source defined at the top level should be used for this port. If the named input source at the top level already has a port attribute, no port attribute need be supplied for this input element.

When the name attribute is specified, none of the file, or script attributes are allowed. If there was a top-level input element like this:

<input name='pulsegen' port='in' script='mygen.py'/>

then a case element could simply have:

<input name='pulsegen'/>

Similarly, it the top-level input element was this (with no port attribute, allowing it to be used for different ports):

<input name='pulsegen' script='mygen.py'/>

then a case element could have:

<input name='pulsegen' port='in'/>

Output Element under Case Elements

The output element for a test case acts the same as the input elements. They refer to a named output element at the top level or override the default, per port.

Property Element under Case Elements

The property element for a test case acts the same as the property elements at the top level: it specifies values to be used for the named property for this test case. If a property is not mentioned in a case element, the default top level values are used.

A single test case can have multiple values for any property. Subcases are automatically generated for all combinations of property values specified for the test case whether specified at the top level as default sets of values, or specified for the test case in the case element.

Individual values for a property for a test case may be restricted to certain platforms as described above in the Exclude/Only/Add Attributes of the Property Element topic.

Preparing Unit Test Inputs

An input element must be specified for each input port of the component, either at the top level as a default for all cases, or for specific cases. It either specifies a pre-existing data file to use (using the file attributes) or a command to execute to generate the input data which can depend on the property settings for the specific subcase (using the script attribute). While a specified input file applies to all subcases regardless of property settings, a generator script can generate input data for each subcase that depends on all its property settings.

The format of the data in the (possibly generated) file is a series of message payloads as defined by the protocol for the port, as described in the Message Payloads on Data Ports topic. The data must respect the setting of the ocpi_endian built-in parameter property, whose value is available to all data generation scripts. All platforms currently supported use only little-endian data layout, but to test a worker that might be built for different endian systems, the layout of the data must match this parameter value.

The generator scripts for input ports are run for each subcase, with all property values for the subcase available to the script. The script is responsible for writing a file whose name is provided on the command line. Since the script command is executed as it would be on the shell command line, it can be written in any language, such as python, bash, or even compiled C or C++. It is executed in the context of the development system (not the target, potentially embedded system), so it can depend on any tools installed on the development system. However, scripts that depend on tools not installed or required as part of OpenCPI will make the project as a whole less portable.

An example input generator script written in the python language is below, for a FIR filter component. The script depends on two properties. The first COEFF_WIDTH_p is a parameter specifying the bit-width of samples. The second NUM_TAPS_p is the number of taps in the filter. The script generates an impulse, with a maximum value followed by zeroes. The file is binary 16 bit signed fixed point data.

#!/usr/bin/env python3
import sys, os, struct
max_tap  = pow(2,int(os.environ.get("OCPI_TEST_COEFF_WIDTH_p"))-1)-1
num_taps = int(os.environ.get("OCPI_TEST_NUM_TAPS_p"))
fo       = open(sys.argv[1], 'wb')
for j in range(num_taps):
    fo.write(struct.pack('h', max_tap))
    for i in range(1,num_taps*2):
    fo.write(struct.pack('h', 0))

If the script was in the local file generate.py and made executable (e.g. with chmod a+x generate.py), and the input port was named in, then the input specification that used this script would be:

<input port='in' script='generate.py'/>

An input generation script must return exit status of zero/non-zero for success/failure.

Preparing for Unit Test Output Verification

An output element must be specified for each output port of the component, either at the top level as a default for all cases, or for specific cases. It either specifies a pre-existing file to compare test output data against (using the file attribute), or a command that examines the data produced at an output port to decide whether the execution of the subcase was successful (using the script attribute). Output verification scripts have access to the output data produced, the input data provided to all input ports, and the final values of all properties at the end of execution (in the environment).

If the component had input ports in1 and in2, and an output port name out, and a script named <command>, in the script attribute, the actual command executed would be:

<command> <output-from-port-out> <input-to-in1> <input-to-in2>

The three filename arguments would be added by the unit test framework to run this output verification script for a given subcase, providing the file names for the data associated with the subcase (input and output). The script would run in the development environment and not in the environment of a potentially embedded target platform.

As with input data, the message payload formats will comply with the lay out as described in the Message Payloads on Data Ports topic, and also respect the value of the built-in ocpi_endian parameter property.

An output verification script must return exit status of zero/non-zero for success/failure.

Summary of Steps Prior to Test Execution and Verification

After creating the test suite in the <component>.test directory using the ocpidev create test command, the following steps are taken prior to running any tests.

Add the elements (input, output, property, case) to the <component>-test.xml file.
Prepare any input data files or input data generator scripts.
Prepare any output data files (for comparison) or output verification scripts.
Run the generate phase (see below)
Examine the report created in gen/cases.txt to see the generated subcases.
If any workers are HDL workers (which must be built for some HDL platforms), run the build phase to build the bitstream/executables required for testing.

After these steps, all applications, HDL assemblies, input data sets and verification scripts have been generated (in the gen/ directory) and any required HDL assemblies have been built for the desired HDL platforms. The generate phase does not depend on which platforms any of the workers being tested have been built for. Prior to the build phase no compilers or other build tools are required or used. The build phase does require any HDL workers to have been built for the desired platforms.

Building an OpenCPI Unit Test Suite

`ocpidev build test`

Build an OpenCPI Test.

Usage:

$ ocpidev build test [OPTIONS] [Test]

Arguments:

[Test]

Options:

-l, --library Library: Specify the Library that the asset resides in.
-p, --project Project: Specify the Project that the asset resides in.
--rcc-platform RCC Platform: RCC Platform(s) to apply the command for.
--hdl-platform HDL Platform: HDL Platform(s) to apply the command for.
--doc-only: Only build docs.
--no-doc: Do not build docs.
--artifacts-only: Clean up intermediate build files.
--report-format [junit]: Format of report(s). [default: junit]
--report-path FILE: Path of file to write report(s) to.
--builder [make]: Specify the Builder to use to run the asset(s). [default: make]
--help: Show this message and exit.

This method can be useful when debugging the XML file and checking the generated results, i.e. in the gen/cases.txt file.

When the build phase is invoked, the generate phase may be re-invoked based on file dependencies. The ocpidev options --hdl-platform, --only-platform, ‑‑exclude-platform may be used on the command line for building to modify defaults specified in the project or library.

Phases for Test Execution and Verification

After the off-line steps described above are performed and any remote systems are enabled, there are three phases that relate to actually executing tests and performing verification, and these phases are based on available local and remote platforms on which to execute tests.

The ocpidev run command on a test suite normally runs the three execution-related phases: prep, run and verify. To run the phases individually, use the ‑‑phase option to indicate which phases to run.

Running an OpenCPI Unit Test Suite

`ocpidev run test`

Run an OpenCPI Unit Test.

Usage:

$ ocpidev run test [OPTIONS] [Test]

Arguments:

[Test]

Options:

-p, --project Project: Specify the Project that the asset resides in.
-l, --library Library: Specify the Library that the asset resides in.
--exclude-platform Platform: Platform(s) to apply the command for.
--only-platform Platform: Platform(s) to apply the command for.
--accumulate-errors: Report execution or verification errors as they occur rather than ending the test on the first failure detected.
--keep-simulations: Keep HDL simulation files regardless of verification results.
--view: Run the view script for the test at the conclusion of the test execution.
--case TEXT: Specify the test case(s) to be executed/verified. May be used more than once.
--remote TEXT: Specify the remote system to run the tests.
--phase [prep|run|verify|prep,run,verify|prep,run|run,verify]: Specify which non-building phase of unit testing to run. [default: prep,run,verify]
--report-format [junit]: Format of report(s). [default: junit]
--report-path FILE: Path of file to write report(s) to.
--runner [make]: Specify the Runner to use to run the asset(s). [default: make]
--help: Show this message and exit.

Examples:

$ ocpidev run --phase prep,run # stop before verify
$ ocpidev run --phase run # rely on previous prepare, don't verify
$ ocpidev run --phase run --verify # use previous prepare
$ ocpidev run --phase verify # just verify previous execution

Skipping the prep phase has the benefit of avoiding some setup overhead and time, but runs the risk of something changing in the environment (e.g. a remote system becoming unavailable).

Preparing for Execution: Discovery and Execution Script Generation

The prep phase, by itself, is invoked by the ocpidev run --phase prepare command. It considers all available platforms, local and remote, including available RCC, HDL hardware and HDL simulators.

This phase also considers which RCC built artifacts are available as well as which HDL bitstream/executable artifacts have been built locally from generated HDL test assemblies in the build phase. From the combination of available platforms and available artifacts, it determines which subcases can be run on which platforms, and generates execution scripts accordingly. These execution scripts are generated in the run/ subdirectory of the test suite. The gen/ subdirectory captures the results of the generate and build phases, and the run/ subdirectory captures the results of the prep and run phases. Both subdirectories are removed by ocpidev clean.

When determining available artifacts for unit test execution, it does not look at the prevailing setting of the OCPI_LIBRARY_PATH, but forces a new environment value that limits the artifact search to local artifacts in the component library, the local gen/assemblies directory and the built-in ocpi.core project.

After determining available and appropriate test execution platforms, this phase generates per-platform scripts that run all feasible subcases on that platform. These scripts are placed in a subdirectory of the run/ directory named after the platform.

Executing Tests on Available Platforms: the Execute Phase

The run phase executes, for each available platform, test applications for all possible cases and subcases. The scope of execution is filtered by the OnlyPlatforms and ExcludePlatforms XML attributes, and the ‑‑only‑platform, ‑‑exclude‑platform, and ‑‑case options to the ocpidev run command.

The run phase iterates through platforms (sequentially), executing all subcases on each platform in turn, and recording the results for later verification. The recorded results are:

The output data from output ports
The final values of all properties, including volatile ones
A log of the actual execution of the tests.

All these results are recorded in the platform’s subdirectory of the run/<platform>/ directory with different files for different subcases.

After the execution of each test on a given platform, a console message will indicate whether the execution succeeded or failed. If an execution fails, the execution of all tests is stopped, and the outputs can be examined for the cause of the failure.

Command Options that Control Execution of Tests

The ‑‑accumulate-errors option to ocpidev run can be used to continue execution of test cases even if some fail, with the final exit status of the ocpidev run command indicating whether any tests failed. This option also applies to during verification of test cases. Not setting this option is useful to solve each problem as it occurs, while setting it allows all errors to occur and be analyzed or examined later. There is no option that will accumulate errors across multiple test directories in a library or project.

The ‑‑verbose (‑v) option to ocpidev run puts all detailed execution logging in the console output (as well we into the subcase log file).

The ‑‑keep-simulations option can be set to cause the runtime logging of simulator execution (in the run/<simulator-platform>/*.simulation directories) to be retained after successful executions on simulation platforms. Successful verification for a platform normally causes the associated simulation logging directory to be removed immediately. Keeping simulation output may use lots of file system space (100s of GBs in some extreme cases). This option is useful when examining simulation traces of successful subcase test runs.

The ‑‑case option (which can be indicated more than once) is a wildcard pattern indicating which cases/subcases should be executed or verified. Subcases are named <casename>.<subcase#>, so this option may be set to patterns that affect certain cases or subcases. Subcase numbers are listed in the report in gen/cases.txt produced by the generate phase. The default case name is case00. For example, to only run subcases that end in 3, you could specify: ‑‑case='case*.*3'. Multiple patterns are allowed, such as: ‑‑case=case00.01 ‑‑case=case00.03.

Verifying Test Results

The verify phase relies on the previous execution of appropriate subcases for all platforms, and performs verification using the results recorded previously in the run phase.

If both run and verify phases are indicated (or prep, run and verify — the default), the run and verify steps are interleaved by subcase, with each subcase being verified immediately after it has been executed. Note that the verify phase can be run without access to the runtime platforms, and is sometimes conveniently done later, even on a different system, since most verification scripts are written in Python.

The command:

$ ocpidev run --phase verify

performs only verification.

The verify phase by itself does not involve any execution or access to local or remote execution platforms and thus can be performed offline. Whether the verify phase is executed with the run phase or by itself, the view option will run the defined view scripts with verification.

If the --accumulate-errors option is not set, the verify phase will immediately stop and return an error if any subcase fails by returning non-zero exit status. If the option is set, all subcases will be verified with failures reported as they occur. The output of the verify phase makes it clear whether each subcase PASSED or FAILED. The individual verification scripts are required to return a standard shell command exit status of zero for success and non-zero for failure.

The ‑‑case option described above applies to standalone verification as well.

Viewing Test Result Data

For interactive debug purposes, the output data can be “viewed” by running the view script associated with each output port defined in the OpenCPI test suite description (OTSD). Viewing is enabled one of two ways:

With the ‑‑view option, which simply adds viewing to whatever phases are being performed by ocpidev run.
By specifying viewing as a phase, which only performs viewing. For example:
```
$ ocpidev run --phase view
```

Viewing is only possible after an execution that produces output data from a port.

When verification is specified, viewing happens per subcase execution before each verification. View scripts typically put up a window to display the data, and then they may wait for user input or simply return, allowing the data to be displayed while verification proceeds.

In this latter case, i.e., when view scripts run their associated data display programs in the background, a problem may arise when such scripts are used with test suites that have a large number of test cases. Running the data display program in the background allows the view script to finish and testing to proceed but leaves the display program running until the user explicitly closes it. Processing a large number of test cases can leave a large number of display programs running simultaneously, which can seriously degrade system performance. View script developers should consider carefully whether display programs invoked by the script should be run in the background (i.e., use the ampersand (&) character).

Testing on Remote Systems

The default behavior of the unit test framework is to test on locally available platforms. This usually means using the local CPU and any locally attached GPUs or FPGAs. However, there are two methods to including non-local, network-reachable systems in the set of available platforms for testing.

The first and more common method is to use the remote container feature of OpenCPI that generally supports OpenCPI applications executing on a mix of local and remotely accessible platforms. This feature is fully described in the OpenCPI Application Development Guide. In order to have the unit test execution framework consider remote systems as test platforms, the remote and local systems must be enabled as described in that document. Using remote containers this way requires no common network mounts between the local and remote systems and prior download of OpenCPI software, but has some firewall-related limitations in some environments.

The second method is specific to unit testing and involves defining possible remote systems using the OCPI_REMOTE_TEST_SYSTEMS environment variable. This method is described in the next section.

Defining Remote Systems for Executing Tests without Remote Containers

When not using the simpler (no NFS) remote containers described above, the following alternative method may be used.

In order for the unit test framework to execute tests on platforms that are not available on the development system, remote systems with additional platforms must be specified. Such platforms are not discovered automatically but are specified in the OCPI_REMOTE_TEST_SYSTEMS environment variable.

Remote systems are accessed using the ssh remote execution command, with an ssh server capability required to be enabled on the remote system. This environment variable is a colon separated list of remote system specifications, and each remote system is specified by five fields separated by ‘=’. These fields are:

Remote host name/IP address
SSH user name
SSH password (yes, this is not a secure solution)
Project directory NFS mount path as seen on remote system
SSH version (optional)

The remote system must meet the following requirements:

SSH access from the development host, using the username and password in the first three (and optional fifth) fields. If the remote system is set up for public/private key access control from the development system, the password is not used, but must still be non-empty.

Then, after a successful SSH login from the development system

The project’s directory on the development system must be mounted (NFS or equivalent) for access from the remote system, using the path of the fourth field.
The OpenCPI Linux kernel driver must be loaded on the remote system
The OCPI_CDK_DIR environment must be set up properly consistent with the development host. An OpenCPI CDK installation is assumed, with the remote system and development system using the same OpenCPI release.

The project directory mount, the OpenCPI CDK installation, and the kernel driver may either be established at boot time or at SSH login time on the remote system.

A remote system may support multiple platforms (e.g. both HDL and RCC). Whatever platforms are available on the remote system will be discovered when the remote system is contacted. Like local platforms, these discovered platforms are subject to the filters specified by OnlyPlatforms and ExcludePlatforms.

Remote systems are frequently embedded systems which do not host a development environment, but any system can be a remote system. E.g. if the development system is running RockyLinux9, the remote system could be Ubuntu 22.04 to run tests on that system also. Of course, the Ubuntu 22.04 system will only run RCC artifacts built for Ubuntu 22.04.

Remote systems may be defined in the environment so that they are available for all test suites in the project, e.g.:

$ export OCPI_REMOTE_TEST_SYSTEMS:=10.0.1.16=root=root=/mnt/myproj

Summary of Unit Test Phases and ocpidev Options

The ocpidev “verbs” that apply to unit testing are:

create, delete, build, clean, and run

The ocpidev build test command is used to perform the generate and build phases, and the ‑‑generate option performs the generate phase without building. The ‑‑hdl‑platform option is used to cause HDL test assemblies to be built for HDL platforms.

The ocpidev clean test command performs as expected, removing all files resulting from generation, building, and running of tests.

The ocpidev run test command is used to perform the prep, run, and verify phases, with the ‑‑phase option specified to be more selective as to which of those phases are actually performed. The ‑‑accumulate-errors option allows all tests to be run, even if some fail. The ‑‑keep-simulations option preserves simulation output even when test execution succeeds. The ‑‑only-platforms and ‑‑exclude‑platforms options can filter which of the available platforms are actually used for executing tests. The ‑‑cases option can limit which test cases are executed and/or verified.