KVM Nested virtualization.

For my current project, I needed to perform nested virtualization on KVM using Ubuntu 12.04, and I encountered the VM would pause itself after showing the bios.

That error dragged me to the logs where this line called my attention:

KVM: entry failed, hardware error 0x80000021
 
If you're running a guest on an Intel machine without unrestricted mode
support, the failure can be most likely due to the guest entering an invalid
state for Intel VT. For example, the guest is maybe running in a real big mode, which is not supported on less recent Intel processors.
 
EAX=00000011 EBX=00000000 ECX=00000030 EDX=00007baa
ESI=c007deba EDI=00038110 EBP=00000000 ESP=00007baa
EIP=00000370 EFL=00010006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0030 0009aef0 ffffffff 0000f300 DPL=3 DS16 [-WA]
CS =9a60 0009a630 0000ffff 00009b00 DPL=0 CS16 [-RA]
SS =0000 00000000 0000ffff 00009300 DPL=0 DS16 [-WA]
DS =0030 0009aef0 ffffffff 0000f300 DPL=3 DS16 [-WA]
FS =0030 0009aef0 ffffffff 0000f300 DPL=3 DS16 [-WA]
GS =0030 0009aef0 ffffffff 0000f300 DPL=3 DS16 [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT=     0009af30 00000037
IDT=     00000000 0000ffff
CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=66 0f 01 16 40 00 66 0f 01 1e 78 00 0f 20 c0 0c 01 0f 22 c0 <66> ea a4 00 00 00 08 00 0f 20 c0 24 fe 0f 22 c0 ff 2e 7e 00 2e a1 14 08 8e d8 8e c0 8e e0
handle_dev_stop: stop

And digging, I found out it’s a bug known issue with the 12.04 kernel. After that, 400 patches came along and cleared the whole issue, and now I got to perform nested virtualization without issues. Updating the kernel fixed it, but how did we do it?:

1. In the KVM host, make sure you have nested VT enabled.
2. Reboot, and make sure when the machine comes back, the change is still there.
3. Test nested virtualization, and if it’s your case, like mine, three levels of virtualization, then perform the same changes to the second host.
4. Reboot the 2nd level host.
5. Check the changes are still there.
6. If so, you got your 3d level of hardware-assisted VT.

cat /sys/module/kvm_intel/parameters/nested 
Y
systool -m kvm_intel -v   | grep -i nested
---Not a command---
Otherwise, you can add that to modprobe...
---Not a command---
echo "options kvm-intel nested=y" > /etc/modprobe.d/dist.conf
or
echo "options kvm-intel nested=y" > /etc/modprobe.d/kvm.conf
---Not a command---
For AMD
---Not a command---
echo "options kvm-amd nested=0" > /etc/modprobe.d/kvm.conf

With that, you can now run your second host with the following params to make sure he gets to see he has VT.

–enable-kvm –cpu host