The OpenCPI System Configuration File

The OpenCPI system configuration file is an XML file that allows settings to be specified that affect the runtime system. Its normal location is:

$OCPI_ROOT_DIR/system.xml

but if it is not present (that is, not customized for the system), the default file is used, which is:

$OCPI_CDK_DIR/default-system.xml

Its location may also be overridden by the OCPI_SYSTEM_CONFIG environment variable. An example file is:

<opencpi>
    <container>
        <ocl load='0'/>
        <rcc load='1'/>
        <hdl load='1'>
            <device name="PCI:0000:05:00.0" esn="000013C1E1F401"/>
            <device name="PCI:0000:04:00.0" esn="91d28408"/>
        </hdl>
        <remote load='1'/>
    </container>
    <transfer smbsize='128K'>
        <pio load='1'/>
        <!-- <dma load='1'/> -->
        <socket load='1'/>r
    </transfer>
</opencpi>

The top-level elements are categories of modules (plugins) and the next level is individual plugins. The two top-level categories are:

  • container: for the plugins that support different types of runtime environments

  • transfer: for the plugins that support different data transport mechanisms.

Each plugin has a boolean load attribute that specifies whether the module should be loaded (and thus enabled) in the runtime environment. Otherwise all attributes are plugin-specific. These OpenCPI plugins are like device drivers which know how to discover and manage certain devices.

At installation time, this file is normally only modified for specific processing device attributes. Note that in the container plugin category, the term (and XML element) <device> is in fact the processor that supports the OpenCPI runtime environment, which is an instance of an OpenCPI platform.

Configuration Elements for Container Plugins

The <device> child element can appear under any of the container plugin elements and specifies attributes specific to individual processing devices (of some platform type) supporting that container type. In the example above, an esn attribute is applied to two HDL devices and indicates the electronic serial numbers of the JTAG cables attached to specific PCIe-based FPGAs used as OpenCPI processors (when they are present in the system).

These <device> elements can contain <instance> child elements that can provide device-specific default property values when certain workers are used on that device in applications or loaded artifacts (e.g. FPGA bitstreams). To understand how and when these property values get applied to the underlying workers, two concepts must be introduced: 1) how artifacts are used for execution, and 2) the particular case of load-time workers. These two concepts are introduced in the next sections.

How Artifacts are Used when Applications are Executed

When an application is run, for each instance in the application the following three deployment decisions are made:

  1. Which container (and underlying processing device) the instance will run on.

  2. Which artifact will be used/loaded on that container to support the instance.

  3. Which built worker in that artifact will be used (when there is more than one built worker in the artifact).

To prepare for application execution, the artifact chosen (in decision #2) is loaded onto the container (like a shared object/library is loaded onto a CPU, or an FPGA bitstream is loaded onto an FPGA). An artifact may contain multiple compiled/built workers, so choice #3 is needed when there are multiple compiled/built workers in the chosen artifact that all implement the component spec indicated in the application <instance> (using the component attribute).

We call these compiled/built workers artifact workers. Note that there may be artifact workers in the artifact that are not used by the application at all. They may exist in the artifact, but are not used or mentioned by the application (XML).

The artifact workers in an artifact may be reentrant, which means that the artifact worker can support multiple application instances at the same time. This is normal and common in software artifacts (shared objects/libraries). I.e. if the application has two instances using the component file_read, which are both running in a software/RCC container, that read two different files at the same time, both of those instances will use the same artifact worker in the file_read artifact.

When the application is started, based on the deployment decisions above and after the chosen artifacts are loaded, runtime workers are created in the container based on the chosen artifact worker in the loaded chosen artifact. At that time, the runtime worker is initialized and the property settings indicated in the application are applied to that runtime worker, all prior to the application being started. So the artifact worker is used to create a runtime worker to implement the instance in the application.

Remember the distinction between the artifact worker in the artifact (a chunk of binary code), and the runtime worker that is a runtime object that runs the code in/from the artifact worker, in a container. The term worker is used three ways: 1) the source code, 2) the compiled/built worker code in an artifact, and 3) the runtime object that “does work” executing in a container.

Any property settings are made on the runtime worker and do not affect the binary code of the artifact worker in the artifact. Hence the artifact worker can be used for more than one runtime worker (application instance) at the same time (if it is reentrant), and can also be serially reused in later applications for new workers in those new applications (whether it is reentrant or not), without the artifact being reloaded.

At this time, artifact workers in HDL/FPGA artifacts are never reentrant.

Load-time Workers

Normally, runtime workers created for applications have a lifecycle that is the same as the application they are part of. They are created (from artifact workers), initialized, configured, and started when the application starts and are released and destroyed when the application ends.

Artifact workers may be labeled as having a different lifecycle, where runtime workers are created when the artifact is loaded, independent of applications, and destroyed when (just before) the artifact is unloaded. Such workers are called load-time workers since they are created/initialized/configured/started when the artifact is loaded, which is usually when the artifact is first needed by some application. They have a lifecycle that persists across multiple applications being executed that re-use the same loaded artifact. They are created whether or not they are mentioned in any application. Workers that are slaves to proxies mentioned in the application are considered “mentioned in the application”.

Artifact workers are labeled as being load-time by setting the loadtime attribute to true in either the worker’s OWD or when a device is declared as part of a platform (or card) in the XML file that defines the platform or card (in the <device> elements there). So a platform developer may designate a device worker to be load-time for that platform (or card) even if it is not declared as being load-time in its OWD.

In HDL artifacts, the platform worker and the worker that keeps time are always considered to be load-time workers.

If a load-time worker is specified in the application, only writable properties are allowed to be specified in the application, since the load-time worker is started before the application is started.

Providing Default Values for Worker Properties in system.xml

One of the things that can be specified in the system.xml file for a container are default property settings for workers that run in that container. These property settings are applied to runtime workers when they are created, before any property settings specified by the application are applied. If the indicated worker is a load-time worker, these properties are applied when the worker is created at the time the artifact is loaded. Otherwise they are applied when the runtime worker is created for the application.

These property settings can thus serve as device-specific, default values. This is accomplished by inserting <instance> elements in the <device> elements under the <container> element in the system.xml file, e.g.:

<opencpi>
    <container>
        <hdl>
            <device name='myfpga'>
                <instance worker='foo'>
                    <property name='pname' value='pvalue'/>
                </instance>
            </device>
            .. other HDL devices/containers
        </hdl>
        ... other models
    </container>
</opencpi>

The above example indicates: whenever you encounter a foo.hdl worker in an application where its runtime worker will run on this device, apply this property setting before applying any settings in the application. Thus the attributes of the system.xml <instance> elements (e.g. worker) are used to match against any instances in applications, after the above deployment decisions have been made, to see whether these settings in the system.xml apply to any of the instances (and thus runtime workers) in the application.

The <instance> elements in system.xml use a subset of the valid attributes for <instance> elements in application XML (OAS). Note the authoring model suffix for worker names here is implicit and thus is not needed (although it is accepted).

The worker attribute thus may have a fully qualified worker name (including package prefix and model) or just the worker name, the worker name and model suffix and may or may not use a build configuration ID (after a hyphen, before the model suffix).

At this time, property values for parameter property values are not allowed in system.xml and are thus not used in this matching process. The property values in system.xml are thus only used for initial or writable properties.

The system.xml <instance> elements can unambiguously specify device workers for particular devices in the device’s platform, using the device attribute (which is also valid in <instance> elements in applications). Its value is matched against the artifact worker.

Properties set this way must either be initial properties or writable properties.

Specifying load-time worker properties in system.xml for a processing device is typically used to specify static processing-device-specific properties for device workers on the platform. Such properties are typically not specified in portable applications. Here is an example where a load-time worker, which corresponds to a particular device worker in the artifact, is configured:

<opencpi>
    <container>
        <hdl>
            <device name='myfpga'>
                <instance device='ethermac1'>
                    <property name='macaddr' value='1122334455566'/>
                </instance>
            </device>
        </hdl>
    </container>
</opencpi>

The above example indicates that the macaddr property of the device worker that supports the ethermac1 device should be set to 1122334455566, when that device is present and used in an artifact (bitstream). The use of the word “device” is overloaded here. Plugins manage devices whose meaning is specific to the plugin, but container plugins support setting properties on workers that represent specific lower-level devices attached to the underlying platform.

Configuration Elements for Transfer Plugins

The smbsize attribute to the top-level transfer element specifies a default value for the size in bytes of the buffer memory pool for each data transfer endpoint. It can also be applied to each transfer plugin module below it, which overrides the default.

Discovery of Devices Managed by Plugins

Most plugins act as drivers for a certain type of device. They manage and discover the devices they support. Container plugins manage processing devices that can support containers for running workers. Transfer plugins manage interconnect devices that can support connections between containers (to support connections between workers running on different containers). Other plugin types are used internal to the system.

When plugins are loaded and used, they find their devices to be managed by a discovery process. This involves scanning the system environment for candidate devices. The <device> elements under a plugin’s XML element are used to provide extra attributes for the devices once they are discovered. These are attributes that are not automatically determined during the discovery process. I.e. they are only necessary when the plugin cannot determine them when a device is discovered. Thus no <device> element is necessary for discoverable devices that require no extra attributes.

Some devices are in fact not discoverable and must be explicitly described using the <device> element with attributes. These are termed static devices, which are not discoverable devices. When a plugin is asked to discover its devices, it first looks for <device> elements with a static attribute set to true, and tries to use them explicitly. The plugin then looks for discoverable devices (that are not already indicated as static).

This discovery process (use static devices, then find discoverable ones), can be controlled by the discovery attribute of the plugin, which is an enumeration of these values:

  • all – the default value indicating use of both static and discoverable devices

  • none – do not do discovery for this plugin

  • static – only use statically defined devices

  • dynamic – only use dynamically discovered devices (avoid static ones)