Running Ember with Xilinx Onload TCP Bypass
Author: Valery Svistunou
Summary
This article explores the impact of using TCP bypass on the order-to-tick latency metric of the Ember Matching Engine. In essence, the Onload TCP bypass results in approximately a 2.4-time improvement in market data latency for the matching engine benchmark.
Preface
Onload is a part of Xilinx / AMD (former Solarflare) software stack which allows to accelerate networking for certain applications without source code modification. Working with compatible network adapters it provides compliant TCP, UDP and multicast sockets with full kernel bypass.
This article describes how to configure host OS and build Ember container, which might be accelerated with Onload. This setup has been tested with Ubuntu 22.04, but it might be adopted for other Linux distributions as well.
Host OS configuration
Onload should be installed to host OS first. The latest version could be downloaded from the official site. Note, that Onload version should be the same on a host and in a container.
To install Onload on a host:
- Install packages, required for Onload building:
sudo apt-get update
sudo apt-get install ca-certificates coreutils curl ethtool gcc kmod libpcap0.8-dev libtool make autoconf automake binutils gawk gettext glibc-source libcap-dev libtool sed libelf-dev g++ linux-libc-dev linux-headers-$(uname -r)
- Unpack Onload distribution and run the installation script:
unzip SF-109585-LS-42-OpenOnload-Release-Package.zip onload-8.1.1.17.tgz
tar -xzf onload-8.1.1.17.tgz
./onload-8.1.1.17/scripts/onload_install
- Make sure, that correct driver is used with your NIC:
ethtool -i PUT_NETWORK_INTERFACE_NAME_HERE
If the installation is correct, version from the output will not match with your Linux kernel version:
version: 5.3.14.1019
firmware-version: 4.7.1.1001 rx1 tx1
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: yes
Note: network interfaces' names might be changed after driver's installation. Don't forget to update network configuration in /etc/netplan/network.yaml and run sudo netplan apply to apply the changes.
- Verify user-space tools by running
onloadwithout any arguments. The output should not contain any error/warning messages and look like the example below:
$ onload
Onload 8.1.1.17
Copyright (c) 2002-2023 Advanced Micro Devices, Inc.
Built: Sep 25 2023 11:29:29 (release)
Build profile header: <ci/internal/transport_config_opt_extra.h>
Kernel module: 8.1.1.17
... trunkated ...
Building Ember container with Onload
To run Ember with Onload, a custom Docker image should be built. The image should contain both Ember and user-space Onload files. The example of Dockerfile following shows how to do it:
# Builder container
FROM ubuntu:22.04 as builder
ARG KERNEL=linux-5.15.72-tickless
ARG ONLOAD=onload-8.1.1.17
ADD *.tgz /src
RUN dpkg -i /src/$KERNEL/linux-headers-5.15.72-tickless_5.15.72-tickless-1_amd64.deb && \
dpkg -i /src/$KERNEL/linux-libc-dev_5.15.72-tickless-1_amd64.deb
RUN apt-get update && apt-get install -y ca-certificates coreutils curl ethtool gcc kmod \
libpcap0.8-dev libtool make autoconf automake binutils gawk gettext glibc-source \
libcap-dev libtool sed libelf-dev g++
RUN /src/$ONLOAD/scripts/onload_build
# Ember container
FROM registry.deltixhub.com/deltix.docker/anvil/deltix-ember:1.14.54-ubuntu as ember
ARG ONLOAD=onload-8.1.1.17
USER root
COPY --from=builder /src/$ONLOAD /src/$ONLOAD
RUN apt-get update && \
apt-get install -y python3 python3-distutils gcc libpcap0.8 && \
/src/$ONLOAD/scripts/onload_install --nobuild --userfiles && \
apt-get purge -y gcc && \
apt-get autoremove -y && \
rm -rf /var/lib/apt/lists/*
WORKDIR /var/lib/emberwork
USER deltix
ENTRYPOINT [ "onload","/opt/deltix/ember/bin/ember" ]
Note:
- The example above assumes that there is a couple of TGZ archives within the same directory, as Dockerfile. One archive contains Openonload, and another one contains some Linux packages. If you don't use any custom build kernel, linux-libc- and linux-headers- packages could be installed from the system repositories instead.
A new image could be built from the Dockerfile using the following command:
docker build -t ember-onload .
Running Ember with Onload
Additional changes should be done to existing docker-compose file to use the new image:
- Set Ember-based services to use the new image:
services:
aeron:
image: "ember-onload:latest"
... skipped ...
ember:
image: "ember-onload:latest"
- Pass Onload devices to Ember container:
ember:
image: "ember-onload:latest"
environment:
- JAVA_OPTS=
- EF_POLL_USEC=100000
- EF_STACK_PER_THREAD=1
devices:
- "/dev/onload:/dev/onload"
- "/dev/onload_epoll:/dev/onload_epoll"
- Make sure that all containers are configured to use host network:
ember:
image: "ember-onload:latest"
environment:
- JAVA_OPTS=
- EF_POLL_USEC=100000
- EF_STACK_PER_THREAD=1
devices:
- "/dev/onload:/dev/onload"
- "/dev/onload_epoll:/dev/onload_epoll"
network_mode: "host"
Start Docker compose. Ember will accept connections on all host's interfaces, but Onload acceleration will work only on supported ones (Solarflare/AMD). Note: since Onload implements its own network stack, some network tools, like tcpdump, will not work with accelerated traffic. Use the tools shipped with Onload instead (with names prefixed with onload_).
Onload's influence on Ember's performance
We ran a series of tests to verify influence of Onload on performance of Ember's market data gateway. During the test we ran a group of trading clients producing 20000 of trade messages per second, and a group of 10 market data consumers. Our goal was to measure latency between a trading message coming to Ember and a corresponding market data message. An Ember instance was running on an x86-server equipped with Solarflare SFC9120-based 10G ethernet controller, all client instances were running on a separate VM instance. Both the server and the VM were connected to the same L2 network. The latency was calculated based on capture files of network interactions between the clients and the server.
Baseline latency (no Onload):
-------- Latency IN -> OUT --------
Percentile : Microseconds : Event count
MIN : 24.0 : 7
50.0% : 78.0 : 15043515
90.0% : 154.0 : 26504300
99.0% : 243.0 : 29125181
99.9% : 359.0 : 29385785
99.99% : 10520.0 : 29412233
99.999% : 31085.0 : 29414882
Latency with Onload-accelerated Ember:
-------- Latency IN -> OUT --------
Percentile : Microseconds : Event count
MIN : 18.0 : 98
50.0% : 32.0 : 15937095
90.0% : 59.0 : 27130091
99.0% : 99.0 : 29827540
99.9% : 158.0 : 30091933
99.99% : 1638.0 : 30118459
99.999% : 35753.0 : 30121170
With Onload, the median latency had dropped more than two times, which might be considered as a good result. Note, that only Ember’s image had been rebuild, whereas nothing had been done with Ember itself.
Please be aware that we utilized a lower-end Solarflare card for this experiment. We intend to implement the Onload approach in our production setup, where we have high-end Solarflare cards. Stay tuned for updates on our results. April 2024 Update: See PROD results here.