OpenCPI Systems and Platforms

OpenCPI provides a consistent model and framework for component-based application development and execution on various combinations of (“heterogeneous”) processing technologies, focusing mostly on embedded systems.

An OpenCPI system is a collection of processing elements that can be used together as resources for running component-based applications. OpenCPI considers each processor part of some hardware subsystem. These subsystems are wired together using some interconnect technologies (e.g. networks, buses, fabrics, cables).

We call each available processor and its surrounding directly-connected hardware a platform. Most commonly, a platform is a “card” or “motherboard” housing a processor and associated memory and I/O devices. The data paths that allow platforms to communicate with each other are called interconnects. The most common interconnects for OpenCPI systems are PCI Express or Ethernet, although others are also supported and used for some platforms (e.g. the AXI system interconnects on Xilinx Zynq-based SoC-based systems).

The scope of a system can be a small embedded system that fits in a pocket, or racks full of network-connected hardware that act as a “system of systems”. Since this “system” definition is somewhat broad, the focus is on efforts to enable running OpenCPI at each platform and interconnect within a system. Hence platform development is enabling a platform, and enabling a system is primarily enabling whatever platforms and interconnects are in the system.

Our most common and simple example system is the ZedBoard from Digilent (zedboard.org), which is based on the Xilinx Zynq chip. This chip is called a “system on chip” or SoC, and indeed has two processing elements connected with an interconnect, all on one chip: 1) a dual-core ARM general-purpose processor, and 2) an FPGA. They are connected via an on-chip fabric based on the AXI standard, and each is connected to some I/O devices. Thus the ZedBoard system consists of two platforms that happen to reside in the same chip, with an AXI interconnect between them.

Another common example system is a typical PC, which has a multicore (1-12) Intel or AMD x86 processor on a motherboard. If cards are plugged into slots on the PC’s motherboard, and those cards have processors (e.g. GPP, FPGA, GPU) on them, then those cards can act as additional platforms in that system.

We consider multi-core GPPs as single “processors” since they generally run a single operating system and act as a single resource that can run multiple threads concurrently.

The final defined element of an OpenCPI system is devices, which are locally attached to some platform to act as source or sink of data to enter or exit the system. Thus devices are distinct from interconnects.

A system consists of platforms connected by interconnects, and platforms can have local devices, either permanently attached (e.g. on the motherboard) or on optional cards in the platform’s slots.

Inside an OpenCPI Platform

As mentioned above, a platform consists of a processor (GPP, FPGA, GPU, etc.) attached to interconnects allowing it to communicate with other platforms. The processor may have multiple cores, and may even consist of multiple processor chips (such as a dual-socket motherboard where two Intel X86 processors act together). There are two other elements that make up a platform: devices and slots.

Devices are hardware elements that are locally attached to the processor of the platform. They are controlled by special workers called device workers (analogous to “device drivers”), and usually act as sources or sinks for data into or out of the system, and thus can be used for inputs and outputs for a component-based application running on that system.

When a device is hard-wired to the platform, it is defined as part of the platform. It is also common for platforms to be optionally configured with add-on cards that provide additional devices for the platform. To allow for this, platforms can have slots, which are an intrinsic part of the platform, and enable cards to be plugged in that make additional devices accessible to the platform. Such cards may be plugged in to any platforms that have compatible slots.

The OpenCPI concepts of devices and slots currently only applies to HDL/FPGA platforms. On software (GPP) platforms, the (usually Linux) operating system is used for device support and thus no OpenCPI device support is included.

An example system is the ZedBoard, which has a Xilinx Zynq SoC part on its motherboard that has a dual-core ARM processor as well as an FPGA inside. This board thus has two platforms, 1) an ARM-CPU-based platform attached to a variety of external peripherals (Ethernet, USB, etc.) as well as 2) an FPGA-based platform attached to some external peripherals (devices) as well as an attached external FMC slot into which different cards may be plugged.

Thus the devices available on a platform are either a permanent part of the platform, or are available on defined cards when they are plugged into one of the platform’s slots.

Platforms can have any number of devices, and may have multiple devices of the same type. When cards are plugged into a platform’s slots, they make additional devices available to the platform.

  • Systems have platforms and interconnects.

  • Platforms have slots and devices, and are attached to interconnects.

  • Cards plug into slots and have devices.

Enabling Development for, and Execution on, an OpenCPI Platform

To make a platform usable for OpenCPI one must enable:

  • development – installing and configuring tools to create executables targeting it, on some development host.

  • execution – providing the runtime infrastructure to execute applications (including component unit tests) on it.

Development is the process of producing “executable binaries”, and execution is the process of running those binaries on the platform, as part of the execution of a component-based application on a system. Enabling development targeting an embedded platform typically involves installing a cross-compilation tool chain on some development host that can produce binaries for the targeted platform.

Development activities are typically said to be done at build-time; execution activities are typically said to be done at run-time.

OpenCPI uses the term “binary artifact” as a technology-neutral term for the binary file that executes on various processing technologies. On GPPs, they are typically “shared object files” or “dynamic libraries”. On FPGAs, they are sometimes called “bitstreams”. On GPUs, they are sometimes called “graphics kernels”.

Enabling development targeting a platform includes procuring, installing, configuring and integrating the various tools necessary to enable the developer to design and create binary artifacts from source code, for the platform. Some adaptations, scripts or wrappers are typically required to enable such tools to operate in the OpenCPI development context. OpenCPI does not preclude or require GUI-based IDEs in the component or application development process.

For most embedded systems, the development tools do not run on the system itself, but instead run on a separate “development host”, typically a (possibly virtual) PC. The unusual, but still possible, case where the tools run on the targeted embedded system itself, is termed “self-hosted development”. When the target platform is embedded and development tools run on a separate “development host”, that is termed “cross development”, using “cross-tools” (e.g. cross-compilers). OpenCPI focuses on “cross development”, except when targeting the development host itself.

All development hosts also act as execution platforms since any host capable of running development tools can act as an OpenCPI execution platform for the GPP/processor of that system. OpenCPI contains tools that must be compiled on the development host and will always run on the development host. Thus a development host is established to execute tools used to create binaries for itself and for other target platforms (cross development). These tools include both OpenCPI’s own tools as well as target-specific tools from third parties.

Enabling New Systems for OpenCPI

Enabling systems for OpenCPI implies enabling the platforms and interconnects in the system for OpenCPI, and thus enabling the processors and devices on the platforms and the devices on any cards used in the system. The first step in the enabling process is to establish the inventory of these elements in the system.

Since specific interconnects, processors, and devices are frequently found in many different systems, it is likely that support is already in OpenCPI. So the enabling process is really for the elements without OpenCPI support. Furthermore, there may be devices in the system that do not require OpenCPI support, either because they are not going to be used, or there is no benefit given the scope of what OpenCPI does.

The system inventory for OpenCPI consists of:

  • Processors (attached to interconnects and devices)

  • Interconnects (among processors)

  • Devices (attached to FPGAs or on cards)

For each, if there is no existing support in OpenCPI, support modules must be developed. For some, there may be existing partial support that must be extended for the intended usage. For new classes of processor (beyond GPP, FPGA, GPU) or new interconnects, the core framework of OpenCPI must be enhanced. Otherwise individual support modules can be developed without modifying the core infrastructure of OpenCPI. Later sections of this document describe the requirements and process of enabling each type of element in the system. Processors must be supported for both development (e.g. compilation tool-chain) and runtime (e.g. device driver changes). Devices and interconnects are normally enabled for runtime only (i.e. their support does not involve changing the build process or tools).

Since OpenCPI is open source software, it is very desirable to contribute new or extended support modules back to the community since most such modules will likely be used in other systems. OpenCPI has a built-in project, in the projects/platform directory, which contains such sharable support assets.

A key aspect of supporting new system elements is obtaining the necessary technical information and tools, and in some cases performing a variety of experiments to assess feasibility and derive otherwise unavailable information. When the information is not available from public sources (such as data sheets for device ICs), NDAs or other confidentiality agreements may be required. Vendors may not supply the necessary information, leaving reverse engineering the only option, subject to legal constraints.

OpenCPI, especially on FPGAs, operates directly on the FPGAs and usually interacts directly with attached devices. Thus either the vendor must supply the required device workers or supply the information required to develop them. Preexisting FPGA modules for attached devices that were not written with OpenCPI in mind are usually unsuitable as is, and must be modified or replaced. In some cases they can be “wrapped”, although this may result in the addition of undesired overhead. Wrapping involves using existing code as a primitive module inside an OpenCPI device worker.

Process Template for Enabling OpenCPI on a New System

Here is a list of steps normally required to enable a new system for OpenCPI. The first four steps are used to establish a clear base of information to estimate and plan the effort. This section may be useful for managers planning the effort, and includes non-technical aspects of the process.

System Inventory

Collect information to determine the rough scope of the effort.

This is the basic inventory of the relevant parts of the system, each of which needs to be considered in planning to enable the system.

Develop the list of processors, interconnects, and devices in the system that are relevant to OpenCPI applications, establishing the system breakdown. This is usually a “block diagram” and “data sheet” exercise.

The time required for this activity depends to a large extent on whether complete information about the target system is available and not restricted.

Processor, Interconnect and Device Assessment

For each, evaluate the state of support currently within OpenCPI and identify additional technical efforts likely to be required. These assessments establish a rough level of effort, without necessarily establishing feasibility. A ROM (rough order or magnitude) LOE (level of effort) can be established.

This effort requires matching technical support requirements with the current state of support in OpenCPI, and thus would require either gaining some familiarity with the OpenCPI supported hardware universe, or engaging with a group that is already familiar with it.

For each processor, determine the level of support within OpenCPI:

  • Currently supported (e.g. Zynq ARM processor, Intel AMD x86 processor)

  • Variant supported (e.g. Virtex6 vs. Virtex 7 FPGA)

  • New processor of existing type (e.g. PPC CPU vs. ARM CPU)

  • New class of processor (e.g. Adapteva Multicore). This requires new work usually outside the scope of “enabling a new system”

For each processor, determine the level of tool chain support within OpenCPI:

  • Currently supported by OpenCPI

  • Requires a version upgrade/downgrade/variant of what is currently supported

  • Not currently supported by OpenCPI

For each interconnect, determine level of support required for each processor type that is attached to it:

  • Currently supported

  • Requires updating or enhancements (e.g. PCI-E gen3x8, vs. PCI-E gen2x4)

  • Not currently supported for processor type (e.g. Ethernet L2 on FPGAs)

For each device attached to an FPGA or on a required card, determine the level of support within OpenCPI and identify the additional technical efforts likely to be required the each device.

  • Currently supported

  • Requires updating or enhancement

  • Not currently supported, but similar devices are supported as a model

  • Not supported and different from any existing supported devices

Assemble the Technical Data Package

For all elements requiring updated code or new support, collect information necessary to perform the enabling technical development, and to establish more detailed work estimates. The information required here is more comprehensive than the basic information required above: it must be sufficient to perform the needed development. This effort will establish the availability of appropriate information to perform the technical development. It will also find any roadblocks to obtaining the information (vendor unwillingness, legacy unavailability).

In the particular case of device support, some vendors may not expose sufficient information to support their devices in the absence of their own “drivers” that embed their ICD (interface control document), information they consider a trade secret. Such positioning by vendors may make optimal support for OpenCPI challenging or impossible.

Required information for processors/FPGAs include:

  • Tool requirements (which tools and versions, which settings, cost)

  • Connectivity technical details (e.g. how interconnects, devices and slots are connected, including pin-outs etc.)

Required information for devices include:

  • Device data sheets or equivalent functional and interface documentation

  • Programming/application guides

  • Appropriate/relevant existing OpenCPI support modules

  • How the devices are attached to the processors. (e.g. ICD)

As part of this effort, any additional required feasibility experiments, reverse engineering or other design/analysis tasks, are identified. These are tasks to fill in the information gaps in order to have high confidence work estimates and plans.

Experiments to Establish Feasibility and Missing Information

There are typically uncertainties and gaps in technical documentation, and in some cases the information is unavailable due to proprietary restrictions. In any case, as a final step to enable accurate work breakdowns and time/cost estimates, there are usually a set of tasks involving hands-on experience with the target system prior to the actual technical development. Such experiments are derived from the process of the above tasks (i.e. discovering knowledge gaps), and may include:

  • Identify, install, and configure vendor-provided tools and hardware.

  • Verify and/or establish functional or performance capabilities of the system that are missing or questionable from the information obtained earlier.

  • Reverse-engineer missing ICD aspects (assuming no legal impediments).

These hands-on tasks establish the final information to plan and budget for the level of effort and scope of enabling a new system for OpenCPI. This activity depends on access to a real system. When the vendor is performing the work for their own hardware, these tasks may be unnecessary.

Planning and Specification

This phase of enabling a system is specifying the technical capabilities to be achieved for OpenCPI on the target system, and planning the tasks to develop and verify the required functionality. The specifications are mostly based on achieving functionality that already exists on other systems, so they are mostly references to other existing documents, with particular options, exceptions, or limitations highlighted.

Every system element not currently supported requires a development task, while all system elements, including those that are already supported, should still have a verification task planned.

In some cases, a system may have desirable functionality that is not yet well defined or standardized in OpenCPI. In this case some proposed changes to the OpenCPI framework itself may be warranted along with a discussion with OpenCPI maintainers on gitlab (via gitlab issues) or email.

The tasks defined here will be of types described more fully in other sections of this document, where the various aspects of platform development are described in detail.

Technical Development

The various technical development tasks are of the types corresponding to specific sections of this document. Each type in the following list may or may not be required to enable a given system. Small updates or enhancements to existing modules are common.

  • Enabling a new development system (including native development and execution)

  • New GPP development tools

  • New FPGA tools

  • New FPGA platform

  • New FPGA devices

  • New FPGA cards

  • New GPP platform

  • New interconnect for GPP platforms

  • New interconnect for FPGA platforms

Verification

Verification requirements may be very project-specific, but for each system element, a baseline verification should be defined, including specific characterization, both manual and automated.

For processors, running all OpenCPI tests and the unit tests in built-in projects is a baseline. Similarly, for FPGAs, building all built-in projects and running unit tests is a baseline. All such tests can run in the OpenCPI Continuous Integration (CI) system, on gitlab. if a dedicated test system is made available.

Contribution

To reduce all efforts at system enablement for OpenCPI, it is strongly encouraged (or in some cases required by licensing), that enhancements to existing support modules, and new support modules for widely available processors, interconnects and devices be contributed back to the appropriate repositories. Usually this consists of merge requests to the built-in platform project.

While assembling the data package as described in the Assemble the Technical Data Package topic, the developer should maintain a complete list of licensed tools, items and associated repositories in order to ensure changes are contributed appropriately back to the sources.