Skip to main content

EmberRTOS Headless RT

EmberRTOS Headless RT is the real-time edition of EmberRTOS. It is the same immutable, transactionally updated foundation described in EmberRTOS Headless, running a real-time kernel by default and the tuning that goes with it. Use it where a workload has a hard deadline: motion control, high-rate deterministic I/O, and PLC or drive timing.

It comes as an install ISO for x86_64 industrial PCs and as a flashable image for the Raspberry Pi 4 and 5. The two behave the same once they are up, and the Raspberry Pi page covers what is different about the image.

This page covers what you see on the machine, what real-time means here, how to turn it on and off, and how to check it on your own hardware. When you have read it, open the Real-time screen of the setup app on your box.

What you see on the machine​

On an x86_64 PC you install from a USB stick or virtual media. On a Raspberry Pi you write the image to a card. Either way the machine starts the installed system, and the screen goes straight to the setup app. Nobody has to log in to bring a box up.

Boot menu​

The boot menu lists both kernels on the machine, the real-time kernel and the general-purpose kernel, in one menu. The real-time kernel is the one selected, and a countdown starts it. Use the up and down keys to choose the general-purpose kernel instead, which is the fallback.

The EmberRTOS boot menu, with the real-time kernel selected and the general-purpose kernel below it

Boot splash​

The EmberNet mark shows while the system starts, in place of scrolling boot messages.

The EmberRTOS boot splash

Status screen​

The setup app's Status screen shows the kernel the node is running. A yes next to Real-time kernel means the node booted the real-time kernel, with the kernel release beside it.

The Status screen of the EmberRTOS setup app, showing the real-time kernel

Console login​

A console login (a keyboard on the machine, or SSH) shows the EmberRTOS mark, the edition, and where the documentation is. fastfetch prints the full picture of the box.

What real-time means here​

Real-time is not about being fast on average. It is about a bounded worst case. A control loop that is usually quick but occasionally stalls for milliseconds still misses its deadline, and that can damage equipment or trip a safety function. So we look at the worst wake-up, not the average one, and we look at it under load.

The real-time kernel​

Real-time support is part of the mainline Linux kernel, so a real-time kernel is mainline kernel source built with real-time options. EmberRTOS builds and ships its own, so the real-time capability is ours to tune and support end to end. The x86_64 and the Raspberry Pi 4 and 5 (arm64) builds are separate packages of the same kernel, with the general-purpose kernel beside them.

The real-time kernel does not include the Intel graphics driver. Under it the display is drawn from the firmware framebuffer, without graphics acceleration. The setup app does not need acceleration. The general-purpose kernel in the boot menu includes the full graphics driver.

After boot, confirm which kernel is running. This is what a Headless RT box reports:

uname -r                           # a version ending in -rt
uname -v # the line includes a real-time marker on a real-time build
cat /etc/emberrtos/kernel-flavor # rt

Tuning​

Real-time behavior comes from tuning as much as from the kernel. EmberRTOS RT applies this set:

  • Deep idle states disabled. They add wake-up latency, and this is the biggest single gain in determinism.
  • CPU isolation. The first cores handle housekeeping. The remaining cores are reserved for real-time work, and interrupts and general load are kept on the housekeeping cores.
  • Timer jitter removed, which takes out a periodic source of jitter.
  • Maximum performance mode and no real-time throttling, so there is no frequency scaling or throttling on the hot path.

The tuning is active on a fresh install, so a node boots with it applied. On a 4-core machine the housekeeping cores are 0 and 1, and cores 2 and 3 are kept for real-time work.

The Real-time screen of the setup app shows what is running on your box: the kernel, whether the tuning is active, which cores are kept for real-time work, and which are left for everything else.

The Real-time screen of the EmberRTOS setup app, with the real-time kernel, the tuning active, cores 2 and 3 kept for real-time work, and interrupts kept off them

Turning real-time off and on​

Both kernels are always on the box, and the Real-time screen switches between them. On boots the real-time kernel with its tuning. Off boots the general-purpose kernel with no tuning. A change takes effect after a restart, and a banner across the top of every screen reminds you until you restart.

On a cluster, control planes restart one at a time. A box will not restart while another node is not Ready, so restart them in turn and wait for each one to come back.

Checking real-time on your box​

cyclictest is installed on the node. It wakes a thread on a fixed interval and records how late each wake-up was. Run it while the node carries its real load:

sudo cyclictest -m -p99 -i200 -D 300     # five minutes, one thread, 200 µs interval

Read the Max column. That is the worst wake-up on this machine while the test ran. To test the cores kept for real-time work, add -a 2-3 -t 2, which pins one thread to each of cores 2 and 3 on a 4-core machine. For a longer run, raise -D: rare events show up over hours, not minutes.

Our measurements​

These are our own latency test reads, each with its hardware, duration, and load. The full method, every thread, and what none of them show are in the white paper Real-Time Latency, Measured on Real Hardware. The 24 hour Raspberry Pi 5 run is also written up on the EmberNet blog. Each read applies to the board it was taken on, so we never compare across boards. For your hardware, use the check above.

HardwareDurationLoadWorst-case wake-up
Raspberry Pi 5 (BCM2712, Cortex-A76), real-time cores 1 to 324 hours, about 627 million samplesa system load generator, two CPU-burn loops, and a tmpfs memory streamer, pinned to the housekeeping core23 µs (17, 20, and 23 µs on the three threads)
Raspberry Pi 5 (BCM2712, Cortex-A76), real-time cores 1 to 35 minutes, about 2.2 million samplesa system load generator and a CPU-burn loop, pinned to the housekeeping core12 µs (10, 12, and 7 µs)
Commodity 4-core Intel desktop, real-time cores 1 to 35 minutes, about 2.2 million samplesa system load generator and a CPU-burn loop, pinned to the housekeeping core42 µs (36, 42, and 31 µs)

How each was run: one high-priority test thread per real-time core, at 200, 700, and 1200 µs intervals, with deep idle states off, interrupts on the housekeeping core, and the performance governor. These bench runs kept one core, core 0, for housekeeping and used cores 1 to 3 for real-time work, with the load pinned to core 0, so the real-time cores stayed isolated. An installed node splits its cores in half instead: on a 4-core machine cores 0 and 1 do housekeeping and cores 2 and 3 do real-time work, as described above. It is not an all-core stress test. The 24 hour Pi 5 run started at 30 °C and peaked near 60 °C with no throttling. The average on every thread was 1 µs.

When to use Headless RT​

Deploy EmberRTOS Headless RT when the node must respond within a bounded worst-case time:

  • Motion control and servo or drive loops
  • High-rate deterministic data acquisition
  • PLC and PAC runtimes, such as CODESYS, with cycle-time requirements

For everything else, including gateways, telemetry, container hosts, and general edge compute, use EmberRTOS Headless. It does not need isolated cores or disabled idle states.

References​