Real Time Setup¶

Since a constant update rate of the RobotBackend is important for stable control of the robot, a real-time capable operating system should be used.

For this we are using a standard Linux distribution (Ubuntu in our case) but install a kernel with better real-time capabilities. There are two options: the “low-latency” kernel which is easy to install but doesn’t give full real-time guarantees, and the preempt_rt kernel, which has better real-time performance but is more tedious to install.

Below are instructions for both of them and on how to configure the system to use them.

Installation: Low-latency vs PREEMPT_RT Kernel¶

We recommend one of the following Linux kernel variants:

low-latency:

Fewer latency guarantees as with PREEMPT_RT but often good enough. Latency can increase if the system load is high, though. It can be useful to assign separate CPU cores for the real-time critical processes.
Can be used with Nvidia drivers.
Easy to install. On Ubuntu it’s just:
```
sudo apt install linux-lowlatency
```

PREEMPT_RT patch:

Better real-time guarantees.
Officially not supported by Nvidia drivers (but may still kind of work, see Using Nvidia Drivers with the PREEMPT_RT Kernel).
Difficult to install as it has to be built from source. See Build and Install the PREEMPT_RT Linux Kernel.

Historically we used the PREEMPT_RT kernel (and still do on many of our robots) but lately also used the low-latency kernel in some projects without problems (you have to be a bit more careful to not overload the CPU, though). Since the low-latency Kernel is much easier to install, you may want to try with that one first and only move to the PREEMPT_RT version if you notice timing-related issues.

The further system configuration described in the following is the same in both cases.

Boot with the real-time Kernel¶

Once the real-time kernel is installed, you need to reboot and select the corresponding kernel (“preempt-rt” or “low-latency”) kernel in the GRUB menu (go to “Advanced options for Ubuntu”, it should be listed there).

When the system is running, you can check which kernel is running with uname -a. It should print something containing “PREEMPT_RT”/”lowlatency”.

Select real-time Kernel by Default¶

You can configure GRUB to automatically select a specific kernel when booting the computer. We provide a script to semi-automatically update the configuration (option 1), but you may also edit it manually (option 2).

Option 1: Semi-automatic¶

We provide a script grub_select_default_kernel.py for this. Download it and execute with

sudo python3 grub_select_default_kernel.py

It will automatically parse the GRUB configuration and list the available boot options. Select the desired kernel and press enter. It will then show a diff of the change it intends to apply to /etc/default/grub. Review the diff carefully and only confirm if it looks good! It should only modify the line starting with GRUB_DEFAULT, i.e. like this:

--- current
+++ new
@@ -3,7 +3,7 @@
 # For full documentation of the options in this file, see:
 #   info -f grub -n 'Simple configuration'

-GRUB_DEFAULT=0
+GRUB_DEFAULT="gnulinux-advanced-cc71b1fb-b530-4694-a839-aecf600f1f49>gnulinux-5.15.0-112-generic-advanced-cc71b1fb-b530-4694-a839-aecf600f1f49"
 GRUB_TIMEOUT_STYLE=hidden
 GRUB_TIMEOUT=0
 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`

If all is good and you confirmed, you still need to run

sudo update-grub

to apply the change. Then reboot and verify that the correct kernel is used.

Option 2: Manually¶

If you don’t want to use the script mentioned above, you may also edit the GRUB configuration manually.

For this, first the identifier of the kernel needs to be determined. Open a terminal and run

cat /boot/grub/grub.cfg | grep -w -e menuentry -e submenu

It should print something like this:

menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-1a26991b-b045-48dd-bb12-064a2725b80b' {
submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' {
    menuentry 'Ubuntu, with Linux 5.4.93-rt51-preempt-rt' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.93-rt51-preempt-rt-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' {
    menuentry 'Ubuntu, with Linux 5.4.93-rt51-preempt-rt (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.93-rt51-preempt-rt-recovery-1a26991b-b045-48dd-bb12-064a2725b80b' {
    menuentry 'Ubuntu, with Linux 5.4.0-65-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.0-65-generic-advanced-1a26991b-b045-48dd-bb12-064a2725b80b' {
    menuentry 'Ubuntu, with Linux 5.4.0-65-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.0-65-generic-recovery-1a26991b-b045-48dd-bb12-064a2725b80b' {

For GRUB_DEFAULT the path full submenu/menuentry is needed, using the id of each step (the last part in the line, “gnulinux-…”), separated by “>”. In this specific case, the setting for starting the rt-kernel would be:

GRUB_DEFAULT = "gnulinux-advanced-1a26991b-b045-48dd-bb12-064a2725b80b>gnulinux-5.4.93-rt51-preempt-rt-advanced-471e9718-013f-4cbb-91a7-d22635173b70"

After saving the changes in /etc/default/grub you need to run the following command for the changes to become active:

sudo update-grub

Then reboot and verify that the correct kernel is used.

System Configuration¶

Normally root permission is needed to run real-time processes. To allow non-root users to run real-time processes, apply the configuration described in this section.

Create the “realtime” Group and Add Users¶

If you used our install script to install the PREEMPT_RT kernel, the “realtime” group is already created automatically. Otherwise, you can create it manually with this command:

sudo groupadd realtime

Then add users that should be able to run real-time applications to the group:

sudo usermod -aG realtime <username>

Set rtprio and memlock limits¶

Note: This step can be skipped if you used our install script to install the PREEMPT_RT kernel.

Create a file /etc/security/limits.d/99-realtime.conf with the following content to increase the rtprio and memlock limit for users in the realtime group:

@realtime   -   rtprio  99
@realtime   -   memlock unlimited

Set Permissions of /dev/cpu_dma_latency¶

The following command needs to be run after each reboot to allow non-root users to set the CPU DMA latency:

sudo chmod 0666 /dev/cpu_dma_latency

Alternatively you can set up a udev rule to do this automatically. For this, simply create a file /etc/udev/rules.d/55-cpu_dma_latency.rules with the following content:

KERNEL=="cpu_dma_latency", NAME="cpu_dma_latency", MODE="0666"

Building robot_interfaces for the Real-Time System¶

To be able to run the robot backend in a real-time thread, some options need to be set at build time. This is automatically done when building on a system that is running the PREEMPT_RT or lowlatency kernel. However, if you previously built while running a different kernel, you’ll need to rebuild.

You can also explicitly build with real-time settings on any kernel with the following command:

colcon build --cmake-args -DOS_VERSION=preempt-rt

Note

If you see the following output during initialisation of the robot, this means you are running a non-real-time build.

Warning this thread is not going to be real time.