Using Remote Containers: Network-Connected Processors

When an application is run, containers are found and used where workers (based on artifacts) may execute. The set of containers considered are those that are part of, or directly attached to, the local system. This includes RCC (software) containers based on the local CPU, and FPGAs attached to the system’s bus or fabric such as PCI Express or the local interconnect between CPU and FPGA on Zynq SoCs.

OpenCPI’s remote containers feature adds containers available in other systems on the network to the set of containers considered for execution. Remote containers are containers offered for use by these other network-accessible systems. The local system where the application is launched acts as a network client and the systems offering remote containers act as network servers.

There are several reasons for using remote containers:

  • Scaling the available resources by using all the containers on a collection of network-connected systems (e.g. clusters, multiple radios)

  • Heterogeneous testing, control, and experimentation allowing the client to perform application control and testing, using many disparate platforms

  • Support containers which are only controllable via networks, e.g.:

    • plug-in processor cards controllable via network over backplane connectors

    • virtual machines running other OSs or licensed simulators

  • Support embedded server nodes which only require a small server software configuration to run and nothing else (minimal configuration and footprint)

An OpenCPI system may offer its containers to other systems for use as remote containers by running the ocpiserve command. This command says: make my local containers available for use by other systems acting as network clients. The ocpiserve command runs a “container server” serving up local containers as remotely accessible containers to network clients running ocpirun or ACI-based applications. These clients are where the application is initiated and controlled (launched).

Since ocpiserve does not run, initiate or control entire OpenCPI applications, it requires fewer resources than ocpirun or ACI executables. No OpenCPI artifacts are necessary on the server system. Instead, the artifacts are downloaded, then cached, from the client to the servers, on demand.

The ocpiremote command, run on a network client or development system, can be used to provision and control a system acting as a container server. This command eliminates all requirements to manually:

  • copy any files to the server system

  • run setup scripts on the server system

  • have any file system network mounting (NFS) between client and server systems.

  • install or run the ocpiserve command on the server system

See the man page for ocpiremote at Manual Page for ocpiremote. The ocpiserve command is fully described in its man page at: Manual Page for ocpiserve.

Enabling Remote/Embedded Systems to be Container Servers

Using ocpiremote to Enable and Manage Container Servers

This command is used separate from application execution or container server discovery to conveniently manage a remote container server from a client system. It loads the OpenCPI server software onto the remote system, and starts the server. Other than specifying the remote system’s IP address (and optionally, the port, and SSH user name and password), no other action is necessary to use the remote system.

The ocpiremote command can use the same environment variables as ocpirun and the ACI to know the addresses of available servers.

Using ocpiserve to Offer Containers to Remote Clients

When not using ocpiremote, the ocpiserve command is run manually on a server system to offer its local containers to remote clients. When it starts, ocpiserve discovers all local network interfaces and prepares to be contacted by clients for all of them. When given the -d (--discoverable) option, it prepares to receive multicast queries from clients on all network interfaces. It is a fully multi-homed server, using all network interfaces for normal usage (via TCP) and discovery (via multicast UDP).

Running the ocpiserve command manually on the server system is not necessary when you use the ocpiremote command.

The ocpiserve command does not need or use the OCPI_LIBRARY_PATH environment variable since the client-server protocol automatically downloads (from client to server) each artifact needed by the application using remote containers. This automatic artifact downloading operates as a cache. When any client requests execution using any of ocpiserve’s containers, they also indicate which artifacts should be used in each container. If these artifacts have been previously downloaded, they are reused. If not, they are downloaded from client to server.

The ocpiserve command maintains this cache in a directory called artifacts (unless overridden by an option). The artifact cache is normally maintained after ocpiserve exits (e.g. via control-C). An option indicates whether it should be removed when ocpiserve exits.

Server Side Remote Container Setup Requirements

On the server side, the ocpiserve executable must be present and there should be enough disk space to hold the artifact cache. The ocpiremote command puts ocpiserve where it needed to be, or it can be part of a complete SD-card based OpenCPI installation.

If it is desirable to allow the server to be discovered via multicast, its local firewall must allow inbound UDP multicast packets from the local network or at least selected clients. To open the firewall to local network traffic the following command can be used (on RockyLinux when it is a server):

sudo firewall-cmd --zone=trusted --change-interface=ens160

Of course more fine-grained permissions may be appropriate.

Example Setup Script for Enabling a Server

Here is a sequence of steps done on a client system to setup a server with the ocpiremote tool and environment variables. The IP/TCP port being used at the server is 12345, and the explicitly specified network address of the server is 10.0.0.86.

# set the server address and port for both ocpiremote and ocpirun
export OCPI_SERVER_ADDRESSES=10.0.0.86:12345
ocpiremote --password root\
           reload \
           --rcc-platform xilinx24_1_aarch32\
           --hdl-platform zed
ocpiremote --password root\
           start -b

At this point the server is ready to be used by a client. An example of running an application using a server set up this way is in the Example Script for Using a Container Server topic.

Using Remote Containers from Client/Development Systems

When applications run, if remote containers are available, some of the components in the application may be deployed onto those remote containers (provided by ocpiserve running on remote systems). The remote containers are added to the set of places where components may execute. Use of remote containers is configured, initiated and controlled from a client system (sometimes the development system).

Using remote containers is transparent: applications use the additional containers when considering where to run the component instances in the application.

Whether an application is started from within a C++ main program using the ACI, or using the ocpirun command, it uses the remote containers it knows about, and finds out about available remote containers by contacting systems running container servers, which run the ocpiserve command.

Container servers are found by two methods: locally specified and discovered.

Locally Specified Container Servers

Locally specifying servers avoids any necessity of discovery or multicast, but requires a manual method to obtain the IP address of the server system and provide that address locally on the client system.

When the addresses of server systems are known in advance, they are specified in one of three ways:

  • environment variables containing addresses or specifying filenames where addresses are found

  • command line options to the ocpirun command to specify server addresses.

  • ACI functions that can specify server addresses

The OCPI_SERVER_ADDRESSES environment variable provides a list of the IP address and port number of available servers. The list is comma-separated, and each address is followed by a port number after a colon. For example:

export OCPI_SERVER_ADDRESSES=10.1.2.3:12345,192.168.1.2:54332

specifies two servers, with different IP addresses and port numbers. These servers will be contacted when an application is started, whether via ocpirun or the ACI. A third optional field (after IP address and port) can contain ocpiremote options that should be used for all ocpiremote commands. I.e. if the password of the remote system was admin, then specifying the server address as:

export OCPI_SERVER_ADDRESSES="10.1.2.3:12345:-p admin"

would mean the -p admin option would be inserted for all ocpiremote commands.

A similar environment variable, OCPI_SERVER_ADDRESSES_FILE can refer to a file containing the same type of server addresses, one per line.

The -S (--server) option to ocpirun may also specify the IP address (and port) of a server system, and that option can be supplied multiple times.

Finally there is an ACI function that can be used to specify a server to access remote containers. The function:

void OA::useServer(const char *server);

is the ACI equivalent to the ocpirun -S option.

The OCPI_SERVER_ADDRESSES_FILE variable can be used when the client and server both have the same file system mounted since the ocpiserve command can be told to put its address(es) into a named file. This is a tradeoff between the overhead and complexity of having a common mounted file system (e.g. when the server has an NFS mount to the client or vice versa), vs. the need to know the server’s IP address, which could be dynamically set using DHCP.

When not using server discovery via multicast (see below), and there is a common NFS mounted directory, using OCPI_SERVER_ADDRESSES_FILE is a convenient way to avoid knowing or typing or copy-pasting IP addresses. If you cannot use discovery and do not want or have a common mounted file system, using OCPI_SERVER_ADDRESSES on the client side (or the --server option to ocpirun) may be best. The ocpiserve -v option causes ocpiserve to print out its IP addresses.

Discovered Container Servers

When not locally specified, remote containers may be discovered using a multicast technique to find the systems running ocpiserve, and retrieve from them the list of available containers on each such system.

The OCPI_ENABLE_REMOTE_DISCOVERY variable can be set to 1 to enable the multicast discovery of servers running ocpiserve.

The ocpirun command has an option for this same purpose. The -R (--remote) option tells ocpirun to “discover” and use remote containers via multicast discovery. The command:

ocpirun -R -C

runs ocpirun simply to list available containers, including any remote ones discovered via multicast.

Similarly, there is a -d (--discoverable) option to ocpiserve that says: make ocpiserve discoverable via multicast. This enables clients using the -R option to find the server. Without the -d (--discoverable) option, the ocpiserve command must be contacted using its explicit IP address (e.g. using the --server option to ocpirun).

The ACI function:

void OA::enableServerDiscovery();

is the ACI equivalent to the ocpirun -R option, and enables remote container discovery. The ACI function:

bool OA::isServerSupportAvailable();

is a function used to determine whether remote container support is enabled in the current environment. This function can be called early in the ACI program to decide whether the other two (OA::useServer or OA::enableServerDiscovery) should be called at all. It essentially indicates whether the remote container plug-in is loaded via being specified in the system.xml file on the client system.

Client Side Remote Container Setup Requirements

Client systems must be enabled for remote containers. The first requirement is that the remote container OpenCPI plug-in must be loaded, which is specified in the system.xml file for the client system. E.g. the load attribute of the remote container plug-in must be set to one:

<opencpi>
  <container>
    <rcc load='1'/>
    <remote load='1'/>
    ...
  </container>
  ...
</opencpi>

This setting is normally already set. The second requirement, which is optional, is to enable clients to automatically discover servers via multicast UDP. This requirement is to provide a proper multicast route to the local network interface on the client system. This is normally accomplished by a command such as:

route -n add -net 224.0.0.0 netmask 240.0.0.0 dev enp2s0

This assumes the normal default network interface for the client is enp2s0.

If multicast discovery is not being used (i.e. explicit server addresses are specified), this route setup requirement is unnecessary.

The client system also must be contactable (as a IP/TCP server) when data stream connections (from worker to worker) are made from the server back to the client. This may require that the firewall on the client system allow connections back from the server. On RockyLinux this means a command such as this is necessary:

sudo firewall-cmd --zone=trusted --change-interface=ens160

The OCPI_SOCKET_INTERFACE variable is needed when the host/client system running ocpirun or ACI or unit test, has multiple active network interfaces. When this is the case, it must be set to the network interface name that should be used for contacting container servers running ocpiserve. I.e. access to remote container servers is currently limited to using a single network interface. The names of network interfaces are obtained using the standard Linux or MacOS ifconfig or ip addr command. This variable should be set to the interface with the same network (IP address anded with the netmask) as the servers being used. Using this variable has no affect on firewall configurations. The ocpihdl ethers command shows this information in a different format.

It is also possible to use remote containers with a less permissive active firewall. The OCPI_TRANSFER_PORT variable can be set to the TCP port that should be used for contacting remote containers. Opening the specified port should then allow a successful execution by ocpirun or ACI. This can be done on RockyLinux with a command such as the following, using port 9001 as an example:

sudo firewall-cmd --zone=public --add-port=9001/tcp

In summary, when servers are set up (see previous section), they are made usable by:

  • enabling the client systems for discovering servers or specifying their addresses

  • enabling data connections from server back to the client system

  • running ocpirun or an ACI program that then uses the remove containers

Example Script for Using a Container Server

Here is a sequence of steps done on a client system to run an application using remote containers setup previously. The network interface on the client system that is used to reach the server system is eth1, the IP/TCP port being used at the server is 12345, and the explicitly specified network address of the server is 10.0.0.86.

# set the server address for both ocpiremote and ocpirun
export OCPI_SERVER_ADDRESSES=10.0.0.86:12345
PDIR=$OCPI_ROOT_DIR/projects
export OCPI_LIBRARY_PATH=$PDIR/assets/artifacts:$PDIR/core/artifacts
# needed if there is more than one interface with an IP address
export OCPI_SOCKET_INTERFACE=eth1
cd $PDIR/assets/applications
ocpirun -C # output should confirm that server can be reached
ocpirun -v -m bias=hdl testbias.xml