Skip to main content
EmberNetEmberNetWhite paper

MicroVMs on EmberNet: a full operating system per workload at the edge

Fireball Industries

Abstract. Containers share one kernel, and for most industrial software that is the right choice. Some workloads need their own kernel, their own real-time scheduling, or a harder wall between them and their neighbors. The EmberRTOS MicroVM is a lightweight virtual machine that boots EmberRTOS inside a cluster you already run. You install it from the EmberNet App Store, the cluster schedules it like any other app, and uninstalling it removes it cleanly. This paper covers what a MicroVM is on EmberNet, how it differs from a container, why we built it, how it fits the platform, what it cannot do yet, and what it means for the people who run a plant.

1. What a MicroVM is here​

A MicroVM on EmberNet is a virtual machine that your cluster runs beside its containers. It boots from an operating system image that is packaged and distributed the same way a container image is. The image is an EmberRTOS build, produced by the same EmberRTOS release that produces our node images.

From the cluster's point of view, the VM is a workload. It is scheduled onto a node, it has a lifecycle, and it can be started, stopped, and removed. From the inside, it is a complete EmberRTOS machine with its own kernel.

"Micro" describes how it is handled, not a lack of capability. There is no installer to walk through and no hardware to rack. The image is pulled, the machine boots, and it is running.

2. How a MicroVM differs from a container​

A container is a process, or a group of processes, with a private view of the filesystem and the network. Every container on a node runs on that node's kernel. That makes containers fast to start and cheap to run, and it is why most of the App Store is containers.

Containers share the node kernel; a MicroVM carries its ownOn the left, three containers sit on one shared node kernel. On the right, a MicroVM holds its own EmberRTOS kernel and workload, running on the node.ContainersAppAppAppOne shared kernelthe node kernelNode hardwareMicroVMEmberRTOS MicroVMYour workloadIts own kernelVirtual machine support on the nodeNode hardware
Containers share the kernel of the node. A MicroVM brings its own.

A MicroVM carries its own kernel. That changes three things.

Its own kernel. Software inside the VM sees a kernel that you chose, not the kernel of the node. A workload that needs a particular kernel build, or kernel settings that would be wrong for everything else on the node, can have them.

Its own real-time scheduling. Real-time behavior is a property of the kernel and how it is tuned. A MicroVM can run a real-time kernel and hold CPU cores for itself while the host and the other workloads around it run normally. Containers inherit whatever the host kernel does.

A stronger boundary. A container escape is a question about one shared kernel. A VM adds the hypervisor boundary between the workload and the host, and between one tenant's workload and another's. For software you trust less, or that must be kept apart for contractual or safety reasons, that extra wall matters.

The cost is weight. A VM boots a whole operating system and holds memory for it. That is why the rule of thumb is simple.

Use a container whenUse a MicroVM when
The workload runs fine on the node's kernelThe workload needs its own kernel or kernel settings
It has no hard deadline, or the node is already tuned for oneIt needs real-time scheduling inside a cluster that is not dedicated to real-time work
The usual container isolation is enoughYou want a hypervisor boundary around it
You want the smallest footprintYou want a stand-in for a whole edge node

Most of what runs on EmberNet should stay a container. The MicroVM is for the cases in the right-hand column.

3. Why EmberNet uses MicroVMs​

3.1 The EmberRTOS kernel inside an existing cluster​

EmberRTOS has two editions, a general-purpose one and a real-time one, covered in the EmberRTOS manual. Its real-time edition runs a real-time kernel with isolated cores and tuning for deterministic control. Until now, getting that meant dedicating a whole machine to it.

The MicroVM RT edition brings the same kernel and tuning into a cluster that also runs other things. On a node set aside for real-time work, the VM gets dedicated CPUs and memory, so its cores are its own. The rest of the node keeps doing its general-purpose job.

3.2 A virtual edge node​

A MicroVM boots the same kind of EmberRTOS image that runs on a physical node, and it is built by the same release. It can enroll in Flux the way a physical node does, and it can run a small cluster of its own. That makes it a virtual edge node: something that behaves like an EmberRTOS box without being one.

That is useful in three places. Staging a change before it reaches production hardware. Testing an integration without borrowing a machine. And onboarding, where a site can be brought onto EmberNet and tried out before its hardware ships.

3.3 Isolation when a container is not enough​

Some workloads should not share a kernel with anything else. Putting one in a MicroVM gives it a kernel of its own and a hypervisor boundary, while it stays managed like every other app on the cluster.

3.4 Nodes from the Store, with no hand edits​

You install a MicroVM from the App Store, set its options on the install form, and remove it from the App Store when you are done. There is no manifest to edit and no command to run on the node. Uninstalling removes the VM, its enrollment secret, and its state disk, so nothing is left behind to tidy.

4. How it fits with the rest of EmberNet​

How a MicroVM fits into EmberNetThe App Store installs a MicroVM on a cluster node. The MicroVM enrolls in Flux with a credential the Store issues. Its local cluster stays inside the VM and is not registered with Clusters.App Store: install, set the switches, uninstallCluster nodeEmberRTOS MicroVMState disk keeps identityLocal cluster (switch), local to the VMFlux enrollment (switch)Fluxenrollment credentialClustersno registration
The VM enrolls in Flux with a credential. Its local cluster never registers with Clusters.

The App Store. The MicroVM is an app, listed under EmberRTOS. The Store offers two: Headless, on the general-purpose kernel, and Headless RT, on the real-time kernel. Both are headless, so you reach them over the network rather than a screen. The app page, EmberRTOS MicroVM, has the install options.

Flux. Flux is a switch on each VM, set at install. When it is on, the Store issues an enrollment credential for the VM and keeps it safe for your organization. The VM uses it to enroll itself. When you uninstall, the VM is deregistered. See Flux for what the fabric does.

Clusters. A local cluster is also a switch. A MicroVM can run a small cluster of its own, and that cluster stays local to the VM. It is never registered with Clusters, so a virtual node you spin up for a test does not appear in your fleet's cluster inventory.

Identity and state. The VM keeps its identity on a small state disk, so a restart brings back the same node and does not need a new credential. The switches hold across reboots too: one that is off stays off.

Updates. The images are built by the EmberRTOS release, the same release that builds our node images. A MicroVM is a new image away from the next EmberRTOS version, instead of a long-lived machine that drifts.

5. Proven so far​

We tested the MicroVM on our own cluster, pulling from the real registry. We ran Headless with Flux on, where it enrolled and stayed enrolled through restarts. We ran Headless with Flux off and the local cluster on. And we ran Headless RT, unmodified, installed through the Store.

6. Limits today​

We would rather you hear these from us.

  • The Plasma firewall is off in this version. The install form says so. A MicroVM does not give you Plasma firewall protection yet.
  • amd64 only. The images are built for amd64 today. arm64 follows as its own release.
  • Real-time needs real-time nodes for its full guarantees. The RT edition gets dedicated CPUs and memory on a node set aside for real-time work. On any other node the Store picks values that let the VM schedule, and you can override them on the install form, but you should not expect the full real-time behavior there.
  • It needs virtual machine support. The cluster has to have it, on nodes with hardware virtualization.

7. Where it is going​

Two things are next. arm64 images, so the MicroVM can run on arm64 nodes, will ship as their own release after the amd64 one. And the Plasma firewall will be turned on in the MicroVM once its key is wired. We will update this paper and the app page when each one is live.

8. What it means for a plant​

If you run a plant, a MicroVM turns a hardware decision into a software one. You can try a new edge node in an afternoon, with no purchase order and nobody on a ladder. You can put a deadline-driven workload on the cluster you already have, and give a workload you trust less a wall of its own. When a test is over, you remove it, and it is gone.

The real-time work still belongs on tuned hardware that you have qualified for it, and today the firewall protection is not in the MicroVM. Inside those limits, you get a full EmberRTOS machine from the App Store, and it is yours to start and stop.

Next steps​