The OpenCPI Linux Kernel Device Driver

OpenCPI uses a Linux kernel device driver for OpenCPI container discovery and for communication between CPUs and FPGAs on certain types of OpenCPI systems. On Linux, this device driver is a loadable kernel module. This kernel module is built against the Linux kernel headers that correspond to the running kernel it will be used with. Arranging for this build is part of platform development for the Linux RCC platform, and the build happens via the ocpiadmin install platform command for the platform.

The next sections describe when the driver is required, how to load it, driver configuration options, and how to allocate additional DMA memory for it when required by OpenCPI applications.

When the Linux Kernel Device Driver is Required

Whether or not an OpenCPI system requires the Linux kernel device driver depends on two things:

  • The platforms on the system that are enabled for use by OpenCPI during application execution

  • The interconnect technology used in the system that wires the platforms together

Systems with platforms connected to a system bus, such as PCI Express or the Xilinx Zynq AXI interfaces between the CPU (PS) and the FPGA (PL) parts of the SoC, require the device driver when the bus or fabric is used for communication between the host CPU platform and another platform. Systems that do not require this type of communication do not need the driver.

Loading the Linux Kernel Device Driver

When the Linux kernel device driver is required, it must be loaded. For embedded systems, including PCI Express-based systems that are set up as runtime-only systems, loading is done automatically by the scripts that establish the modes of operation for OpenCPI (server, network, or standalone) during OpenCPI installation and setup (see the OpenCPI Installation Guide for details).

If the system has not been set up with one of the modes of operation for OpenCPI (or the mode-of-operation script used to set up the system was changed to disable automatic driver loading) and the system is a development host, driver loading must be done explicitly using the ocpidriver.

Loading the device driver with ocpidriver can be done either:

  • When needed by OpenCPI applications that are to run on the target platform(s), where a user manually runs the ocpidriver load command

  • When the system is booted, where the appropriate operating system startup script runs the ocpidriver load command

Loading the device driver manually when needed loads it in a generic way that is common to all Linux systems, whereas loading the device driver at boot time means that it is specific to the Linux operating system in use on the development host (Rocky, Ubuntu, etc.).

Configuring the Linux Kernel Device Driver

The setting for the OCPI_DMA_CACHE_MODE environment variable, which controls the Linux caching mechanism on platforms that support cacheable DMA buffers, applies when using the Linux kernel device driver.

The default setting for OCPI_DMA_CACHE_MODE is 1. In this mode, DMA buffers use normal caching when accessed by the CPU and the caching of a buffer’s contents is explicitly invalidated when new buffers are received by the CPU and explicitly flushed after new buffers are filled (written) by the CPU.

A setting to mode 0 disables the CPU caching for the DMA buffers, while a setting to mode 2 enables cache-coherent DMA between the CPU and other (OpenCPI platform) DMA masters on the bus/fabric.

All platforms support mode 0, most platforms support mode 1, and few platforms support mode 2. Using the default mode is usually the best policy.

Reserving Additional Memory for the Linux Kernel Device Driver

When OpenCPI communicates over a system bus or fabric to other platforms on the bus/fabric, it uses the OpenCPI Linux kernel device driver for OpenCPI container discovery and DMA-based communication, which requires local (reserved) DMA memory resources. DMA memory resources must be allocated or reserved on the CPU-side memory that is accessible to both the CPU (via the local mmap system call) and OpenCPI’s DMA engines, which are used on all DMA-capable system busses and which allow the FPGA to directly access (read and write) system memory.

On Linux, OpenCPI uses a default of 128 KB of memory for the OpenCPI Linux kernel device driver. However, OpenCPI applications may have buffering requirements that necessitate additional DMA memory resources.

OpenCPI requires a physically contiguous DMA memory area. Some Linux systems make this easy and dynamic (that is, possible when applications are running), and some do not. For those that do not, DMA memory reservation is required such that the DMA memory required by OpenCPI is reserved at boot time. Consequently, the requirement for DMA memory reservation is derived from 1) how much DMA memory is required by the applications that run, and 2) whether the Linux kernel requires DMA memory allocations above a certain size to be reserved at boot time.

This section describes how to use the Linux kernel parameter memmap to reserve 128 MB of additional DMA block memory from the Linux kernel at boot time for DMA-based communication on a PCI Express-based system. While this parameter supports many formats, the following usage has proven to be sufficient:

memmap=size!start

Where size is the number of bytes to reserve in either hexadecimal or decimal, and start is the physical address in hexadecimal bytes. All addresses and sizes must be on even boundaries (0x1000 or 4096 bytes).

The OpenCPI PCI DMA engine currently requires that the user accessible-mode DMA memory pool be in a 32 or 64-bit memory range; due to the way that Linux manages memory, it is recommended that the address be higher than the first 24 bits.

With these requirements, the first step to reserve additional memory for the example here is to find a usable contiguous memory range by examining the BIOS physical RAM map as reported by dmesg.

Run dmesg and filter on BIOS to review the physical RAM map:

dmesg --notime | grep BIOS

The output should look similar to the following:

BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x0000000000097bff] usable
BIOS-e820: [mem 0x0000000000097c00-0x000000000009ffff] reserved
BIOS-e820: [mem 0x00000000000ce000-0x00000000000cffff] reserved
BIOS-e820: [mem 0x00000000000dc000-0x00000000000fffff] reserved
BIOS-e820: [mem 0x0000000000100000-0x00000000bfecffff] usable
BIOS-e820: [mem 0x00000000bfed0000-0x00000000bfefefff] ACPI data
BIOS-e820: [mem 0x00000000bfeff000-0x00000000bfefffff] ACPI NVS
BIOS-e820: [mem 0x00000000bff00000-0x00000000bfffffff] usable
BIOS-e820: [mem 0x00000000f0000000-0x00000000f7ffffff] reserved
BIOS-e820: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved
BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
BIOS-e820: [mem 0x00000000fffe0000-0x00000000ffffffff] reserved
BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable

Select a “(usable)” section of memory and reserve a subsection of that memory. Once the memory is reserved, the Linux kernel will ignore it. In this case, there are four usable sections to consider:

BIOS-e820: [mem 0x0000000000000000-0x0000000000097bff] usable
BIOS-e820: [mem 0x0000000000100000-0x00000000bfecffff] usable
BIOS-e820: [mem 0x00000000bff00000-0x00000000bfffffff] usable
BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable

On close inspection of the usable regions, the first range is too small and is below the first 24 bits, the third range is simply too small, and the fourth range is beyond the 32 bit limit. Fortunately, the second address space meets the address range requirement (between 24 and 32 bits) and it is large enough to reserve several hundred megabytes of memory.

The starting address for the memmap parameter can be set anywhere in this range as long as it occurs on an even page boundary of 4KB. Once the memmap parameter has been calculated, it will need to be added to the kernel command line in the boot loader. For Rocky 9, the grubby utility can be used to add the parameter to all kernels in the startup menu. Ensure that a backslash is used to escape the “!”:

sudo grubby --update-kernel=ALL --args=memmap="128M\!0x100000"

To verify that the current kernel has the argument set:

sudo -v
sudo grubby --info $(sudo grubby --default-kernel)

Note that the memmap argument has been added without the backslash:

args="ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet $tuned_params memmap=128M!0x100000"

If no longer desired, the parameter can also be removed with the following command:

sudo grubby --update-kernel=ALL --remove-args=memmap

Reboot the system to apply the new configuration for memory reservation. Once the system has finished booting, examine the state of the physical RAM map to confirm that the desired memory has been reserved:

dmesg --notime | more Linux version 5.14.0-503.23.2.el9_5.x86_64 (mockbuild@iad1-prod-build001.bld.equ.rockylinux.org) (gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5), G
NU ld version 2.35.2-54.el9) #1 SMP PREEMPT_DYNAMIC Thu Feb 13 09:52:14 UTC 2025
The list of certified hardware and cloud instances for Enterprise Linux 9 can be viewed at the Red Hat Ecosystem Catalog, https://catalog.redhat.
com.
Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-503.23.2.el9_5.x86_64 root=/dev/mapper/rl-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:51
2M resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet memmap=128M!0x100000
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x0000000000097bff] usable
BIOS-e820: [mem 0x0000000000097c00-0x000000000009ffff] reserved
BIOS-e820: [mem 0x00000000000ce000-0x00000000000cffff] reserved
BIOS-e820: [mem 0x00000000000dc000-0x00000000000fffff] reserved
BIOS-e820: [mem 0x0000000000100000-0x00000000bfecffff] usable
BIOS-e820: [mem 0x00000000bfed0000-0x00000000bfefefff] ACPI data
BIOS-e820: [mem 0x00000000bfeff000-0x00000000bfefffff] ACPI NVS
BIOS-e820: [mem 0x00000000bff00000-0x00000000bfffffff] usable
BIOS-e820: [mem 0x00000000f0000000-0x00000000f7ffffff] reserved
BIOS-e820: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved
BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
BIOS-e820: [mem 0x00000000fffe0000-0x00000000ffffffff] reserved
BIOS-e820: [mem 0x0000000100000000-0x000000023fffffff] usable
NX (Execute Disable) protection: active
APIC: Static calls initialized
user-defined physical RAM map:
user: [mem 0x0000000000000000-0x0000000000097bff] usable
user: [mem 0x0000000000097c00-0x000000000009ffff] reserved
user: [mem 0x00000000000ce000-0x00000000000cffff] reserved
user: [mem 0x00000000000dc000-0x00000000000fffff] reserved
user: [mem 0x0000000000100000-0x00000000080fffff] persistent (type 12)
user: [mem 0x0000000008100000-0x00000000bfecffff] usable
user: [mem 0x00000000bfed0000-0x00000000bfefefff] ACPI data
user: [mem 0x00000000bfeff000-0x00000000bfefffff] ACPI NVS
user: [mem 0x00000000bff00000-0x00000000bfffffff] usable
user: [mem 0x00000000f0000000-0x00000000f7ffffff] reserved
user: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved
user: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
user: [mem 0x00000000fffe0000-0x00000000ffffffff] reserved
user: [mem 0x0000000100000000-0x000000023fffffff] usable

A new “persistent (type 12)” area is shown in the “user-defined physical RAM map” section. Now, when the ocpidriver load command is run, it will detect the new reserved area and pass that data to the Linux kernel device driver.

Erroneous CMA Region Memory Allocation Warning

The Contiguous Memory Allocator (CMA) is a feature available in the most recent Linux kernels and in some older kernel versions. When the Linux kernel supports CMA, the OpenCPI Linux kernel device driver will attempt to make use of the CMA region for direct memory access. In use cases where many memory allocations are made, you may receive the following kernel message:

alloc_contig_range test_pages_isolated([memory start], [memory end]) failed

This is a kernel warning, but does not indicate that a memory allocation failure occurred, only that the CMA engine could not allocate memory in the first pass. Its default behavior is to make a second pass, and if that succeeds, you should not see any more error messages. This message cannot be suppressed, but can be safely ignored. An actual allocation failure will generate unambiguous error messages.