Tuning Ubuntu 22.04 for Ember

Author: Valery Svistunou

September 2023

Preface

This document describes how to tune Ubuntu Server 22.04 (Ubuntu) for better Ember performance. Some advices from this article might be applicable to other Debian-based distributions as well.

Kernel selection and configuration

We strongly recommend to use a tickless kernel. Such a kernel allows to reduce system noise on particular CPU cores by disabling periodic system's timers on it. For that purpose, you may either re-build the default kernel, which is shipped with Ubuntu, or install a real-time kernel, which is available via linux-realtime package. If you prefer your own kernel, make sure that it has been built with CONFIG_NO_HZ_FULL=y parameter.

Whatever kernel is selected, it should be properly configured via boot parameters. The table below show recommended parameters:

Parameter	Description
isolcpus=CPUS_LIST	Isolate a list of CPUs from the general schedule (these cores will be used by Ember).
nohz_full=CPUS_LIST	Specifies a list of CPUs whose tick will be stopped whenever possible. Should match with isolcpus parameter.
transparent_hugepage=never	Disable THP.
intel_pstate=disable	Do not enable intel_pstate as the default scaling driver for the supported processors.
intel_idle.max_cstate=0	Disable intel_idle if the driver is intel_idle. (Run cat /sys/devices/system/cpu/cpuidle/current_driver to check.)
processor.max_cstate=0	Limits processor to the maximum C-state.
mce=ignore_ce	Ignore corrected errors in machine checks
nosoftlockup=0	Disable the soft-lockup detector.
audit=0	Disable the audit-subsystem.
idle=poll	Don't do power saving in the idle loop using HLT, but poll for rescheduling event.
nmi_watchdog=0	Turn hardlockup detector in nmi_watchdog off
mitigations=off	Disable all optional CPU mitigations. This improves system performance, but it may also expose users to several CPU vulnerabilities.
apparmor=0	Disable AppArmor at boot time
rcu_nocbs	TBD...

Note: proper CPU isolation should include both virtual cores, which share the same physical core (for CPUs with HyperThreading enabled). CPU core's topology could be retrieved with lscpu -p command.

To set kernel's parameters:

1. Edit the file called /etc/default/grub.

   sudo vi /etc/default/grub

2. Find or add the line GRUB_CMDLINE_LINUX_="" and add one or more parameters described below (space separated). If something goes wrong, boot in rescue mode and remove the bad changes.

   GRUB_CMDLINE_LINUX_="transparent_hugepage=never intel_pstate=disable intel_idle.max_cstate=0 processor.max_cstate=0 mce=ignore_ce nosoftlockup=0 audit=0 idle=poll nmi_watchdog=0 mitigations=off isolcpus=10-23,34-47 nohz_full=10-23,34-47"

3. To apply changes, run the following command:

   sudo update-grub

4. Reboot.

5. After reboot, verify that the settings were applied using the command cat /proc/cmdline.

Runtime parameters configuration

Some kernel parameters might be configured at runtime. Our advice is to change the parameters following:

net.core.rmem_max = 2097152
net.core.wmem_max = 2097152
vm.stat_interval = 3600
vm.mmap_min_addr = 4096

To persist the parameters between system reboots, save the content of the snipped above in a conf-file into /etc/sysctl.d directory. To apply changes without reboot, run sudo sysctl --system command.

Note: there is a file /etc/sysctl.d/10-zeropage.conf, which may override vm.mmap_min_addr parameter. It should be either removed or adjusted to a new value.

Tuned profile

Additional parameters might be implicitly configured using tuned utility. Use the next commands to install and configure tuned:

sudo apt-get install tuned
sudo systemctl start tuned
sudo systemctl enable tuned
sudo tuned-adm profile latency-performance

This will configure tuned to use latency-performance profile, which is our choise for the best latency.

Disable unused services

Ubuntu Server 22.04 comes with moderate set of services out of the box. Nevertheless, some of them should be disabled to improve latency. We recommend disabling AppArmor and UFW services:

sudo systemctl stop ufw
sudo systemctl stop apparmor
sudo systemctl disable ufw
sudo systemctl disable apparmor

Disable swap

Permanently disable swap. To do that, edit /etc/fstab and comment out swap line (usually the last entry).

sudo vi /etc/fstab

For example:

#/dev/mapper/cl-swap     swap                    swap    defaults        0 0