Creating a Virtual Machine that can be instantly resumed or duplicated using migration

3 min readApr 11, 2019

Overview

QEMU supports migration of a virtual machine across machine over the network. It also has the capability to save the state of a virtual machine and resume from that state.

It is described in detail in https://www.linux-kvm.org/page/Migration

This capability can be leveraged to create an instant resume virtual machine image if the virtual machine is crafted with care.

Requirements

The virtual machine should run purely from memory
The virtual machine should save any persistent state to ram based storage
The virtual machine should not rely on any host network devices (like tap, blk, 9p)
The state (including networking state) should be completely contained in the QEMU process itself (i.e. SLIRP)

How to create the Virtual machine

Use kernel + initramfs + qemu + slirp networking to create the base image

Building the Kernel

The default kernel configuration has to support initrd CONFIG_BLK_DEV_INITRD=y
This will provide you with the bzImage needed

Building the initramfs

Copied from https://gist.github.com/pacalet/660fa6472e0bcd53f71fc31f167c628f)
Use a busybox based rootfs for ease of us

git clone https://github.com/mirror/busybox
  cd busybox
  make defconfig
  make menuconfig (enable Busybox Settings ---> Build Options ---> Build BusyBox as a static binary (no shared libs) ---> yes)
  make
  make install

Create the initramfs

  mkdir $HOME/rootfs
  cd $HOME/rootfs
  mkdir -p bin sbin etc proc sys usr/bin usr/sbin
  cp -a $BUSYBOX_BUILD/_install/* .
  rm linuxrc

Create an init binary with appropriate contents with executeable perms

#!/bin/sh
  mount -t proc none /proc
  mount -t sysfs none /sys
  echo -e "Hello World\n"
  exec /bin/shcd $HOME/rootfs
  find . -print0 | cpio --null -ov --format=newc | gzip -9 > $HOME/initramfs.cpio.gz

Booting the base image

Launch the VM with monitor enabled

sudo ~/qemu/x86_64-softmmu/qemu-system-x86_64 \
         -machine pc,accel=kvm,kernel_irqchip \
         -cpu host \
         -kernel ./bzImage \
         -initrd ./initramfs.cpio.gz \
         -nographic \
         -nodefaults \
         -no-user-config \
         -m 256,maxmem=1G,slots=2 \
         -smp 2 -rtc base=utc,driftfix=slew \
         -global kvm-pit.lost_tick_policy=discard \
         -append 'console=hvc0 single iommu=false root=/dev/ram0 dhcp' \
         -netdev user,id=mynet0 \
         -device virtio-net-pci,netdev=mynet0 \
         -device virtio-serial-pci,id=virtio-serial0 \
         -chardev stdio,id=charconsole0 \
         -device virtconsole,chardev=charconsole0,id=console0     \
         -monitor telnet:127.0.0.1:1234,server,nowait

Capturing its state

telnet localhost 127.0.0.1 1234
stop
migrate_set_speed 4095m                                            
migrate "exec:gzip -c > STATEFILE.gz"   
cont

Launching a new VM based on this VM’s saved state

sudo ~/qemu/x86_64-softmmu/qemu-system-x86_64 \
         -machine pc,accel=kvm,kernel_irqchip \
         -cpu host \
         -kernel ./bzImage \
         -initrd ./initramfs.cpio.gz \
         -nographic \
         -nodefaults \
         -no-user-config \
         -m 256,maxmem=1G,slots=2 \
         -smp 2 -rtc base=utc,driftfix=slew \
         -global kvm-pit.lost_tick_policy=discard \
         -append 'console=hvc0 single iommu=false root=/dev/ram0 dhcp' \
         -netdev user,id=mynet0 \
         -device virtio-net-pci,netdev=mynet0 \
         -device virtio-serial-pci,id=virtio-serial0 \
         -chardev stdio,id=charconsole0 \
         -device virtconsole,chardev=charconsole0,id=console0     \
         -incoming "exec: gzip -c -d STATEFILE.gz" \
         -monitor telnet:127.0.0.1:1235,server,nowait \

Resume it

As the VM was in a stopped state you need to resume it. This VM will resume instantly in exactly the same state as the previous one. As this is a pure RAM based VM there is no additional setup needed on host or guest side. The external network connectivity is also instantly restored.

Also you can launch as many VM’s as you need from this single image

telnet localhost 127.0.0.1 1235
cont

Note

Using cpu=host means that image needs to be created on the same type of machine as the target