Creating a Virtual Machine that can be instantly resumed or duplicated using migration
Overview
QEMU supports migration of a virtual machine across machine over the network. It also has the capability to save the state of a virtual machine and resume from that state.
It is described in detail in https://www.linux-kvm.org/page/Migration
This capability can be leveraged to create an instant resume virtual machine image if the virtual machine is crafted with care.
Requirements
- The virtual machine should run purely from memory
- The virtual machine should save any persistent state to ram based storage
- The virtual machine should not rely on any host network devices (like tap, blk, 9p)
- The state (including networking state) should be completely contained in the QEMU process itself (i.e. SLIRP)
How to create the Virtual machine
- Use kernel + initramfs + qemu + slirp networking to create the base image
Building the Kernel
- The default kernel configuration has to support initrd
CONFIG_BLK_DEV_INITRD=y
- This will provide you with the bzImage needed
Building the initramfs
- Copied from https://gist.github.com/pacalet/660fa6472e0bcd53f71fc31f167c628f)
- Use a busybox based rootfs for ease of us
git clone https://github.com/mirror/busybox
cd busybox
make defconfig
make menuconfig (enable Busybox Settings ---> Build Options ---> Build BusyBox as a static binary (no shared libs) ---> yes)
make
make install
Create the initramfs
mkdir $HOME/rootfs
cd $HOME/rootfs
mkdir -p bin sbin etc proc sys usr/bin usr/sbin
cp -a $BUSYBOX_BUILD/_install/* .
rm linuxrc
- Create an init binary with appropriate contents with executeable perms
#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
echo -e "Hello World\n"
exec /bin/shcd $HOME/rootfs
find . -print0 | cpio --null -ov --format=newc | gzip -9 > $HOME/initramfs.cpio.gz
Booting the base image
Launch the VM with monitor enabled
sudo ~/qemu/x86_64-softmmu/qemu-system-x86_64 \
-machine pc,accel=kvm,kernel_irqchip \
-cpu host \
-kernel ./bzImage \
-initrd ./initramfs.cpio.gz \
-nographic \
-nodefaults \
-no-user-config \
-m 256,maxmem=1G,slots=2 \
-smp 2 -rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-append 'console=hvc0 single iommu=false root=/dev/ram0 dhcp' \
-netdev user,id=mynet0 \
-device virtio-net-pci,netdev=mynet0 \
-device virtio-serial-pci,id=virtio-serial0 \
-chardev stdio,id=charconsole0 \
-device virtconsole,chardev=charconsole0,id=console0 \
-monitor telnet:127.0.0.1:1234,server,nowait
Capturing its state
telnet localhost 127.0.0.1 1234
stop
migrate_set_speed 4095m
migrate "exec:gzip -c > STATEFILE.gz"
cont
Launching a new VM based on this VM’s saved state
sudo ~/qemu/x86_64-softmmu/qemu-system-x86_64 \
-machine pc,accel=kvm,kernel_irqchip \
-cpu host \
-kernel ./bzImage \
-initrd ./initramfs.cpio.gz \
-nographic \
-nodefaults \
-no-user-config \
-m 256,maxmem=1G,slots=2 \
-smp 2 -rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-append 'console=hvc0 single iommu=false root=/dev/ram0 dhcp' \
-netdev user,id=mynet0 \
-device virtio-net-pci,netdev=mynet0 \
-device virtio-serial-pci,id=virtio-serial0 \
-chardev stdio,id=charconsole0 \
-device virtconsole,chardev=charconsole0,id=console0 \
-incoming "exec: gzip -c -d STATEFILE.gz" \
-monitor telnet:127.0.0.1:1235,server,nowait \
Resume it
As the VM was in a stopped state you need to resume it. This VM will resume instantly in exactly the same state as the previous one. As this is a pure RAM based VM there is no additional setup needed on host or guest side. The external network connectivity is also instantly restored.
Also you can launch as many VM’s as you need from this single image
telnet localhost 127.0.0.1 1235
cont
Note
Using cpu=host means that image needs to be created on the same type of machine as the target