KVM - The Linux Kernel-Based Virtual Machine

KVM-79 Released: pci device assignment, pci device hot plug

Haydn Solomon — Thu, 13 Nov 2008 01:41:34 +0000

Today, kvm-79 was released featuring full support for host pci device assignment and hot-plugging host pci devices. In order to use these features at this time, linux 2.6.28 is required on the host. This release also includes another qemu merge which recently got kvm support.

New PCI device assignment support

The new pci device assignment allows you to assign a device on your host to a guest. There is a new command line option –pcidevice added for this support. As an example, if you have a pci device sitting at 04:08.0 ( pci bus:dev.fn) you can use the following option when starting kvm guest to assign the device.

-pcidevice host=04:08.0

A couple of things to note when using this new feature.

The host driver for the device, if any, needs to be removed first or assignment will fail.
Currently, you cannot assign a device that shares irq with another host device
The raw_io capability is needed for this to work
This works only with the in-kernel irqchip method. In order to use the userspace irqchip, a kernel module (irqhook) and some extra changes are needed.

In order to hot plug host pci devices into your guest , you have to issue the command from your monitor using the pci_add command.

Changelog

The official changelog from the kvm maintainers are as follows:
Changes from kvm-78:
- merge qemu-svn
- fix qcow2 problems with scsi
- 'info chardev' monitor command
- device assignment userspace (Amit Shah, Muli Ben Yehudah, Ben-Ami Yassour,
Weidong Han, Or Sagi, Nir Peleg, Glauber Costa, Xiantao Zhang)
- multiple processor infrastructure for ppc (Hollis Blanchard)
- fix error handling in eventfd() emulation (Mark McLoughlin)
- handle large mtu with virtio-net (Mark McLoughlin)
- move x86 specific device assignment code to x86 files (Christian Ehrhardt)
- fix 'pci_add' command descriptor (Weidong Han)
- enable pci function level reset for device assignment (Sheng Yang)
- fix incorrect handling of aliases gfns (Izik Eidus)
- ppc optimizations (Hollis Blanchard, Christian Ehrhardt)
- disallow guests from setting memory type when using EPT (Sheng Yang)
- fix cpuid leaf 11 loop termination (Nitin A Kamble)
- fix cpuid multiple leaf iteration (Nitin A Kamble)
- fix ia64 uniprocessor build (Xiantao Zhang)
- fix ppc Kbuild constraints (Hollis Blanchard)
- fix pit initialization memory leak

Kvm-79 can be downloaded here

KVM-78 Released: Bug Fixes, qemu merge

Haydn Solomon — Mon, 03 Nov 2008 20:56:46 +0000

Yesterday kvm-78 was released featuring mostly bug fixes and a qemu merge. One major fix includes booting paravirt Linux kernels which was broken after kvm-77. The qemu merge features usb improvements and live migration fixes.

The official changelog from the development team is as follows:

Changes from kvm-77:
- fix paravirt kernels due to missing oos flush (Marcelo Tosatti)
- code cleanups (Sheng Yang)
- fix guest shared interrupts for device assignment (Sheng Yang)
- avoid stale asm-offsets.h on ia64 (Xiantao Zhang)
- fix ia64 halt emulation (Xiantao Zhang)
- extend slot bitmap on i386 (Sheng Yang)
- move private memory slots to avoid conflict with device assignment
(Sheng Yang)
- future-proof device assignment ABI
- add emulation and unit tests for add acc, imm instruction
(Guillaume Thouvenin)
- fix nmi watchdog emulation (Jan Kiszka)
- optimize nmi watchdog delivery (Jan Kiszka)
- require irq ack notifier to be used with in-kernel irqchip (Sheng Yang)
- don't halt non-boot cpu if using in-kernel irqchip (Gleb Natapov)
- add cscope definitions (Sheng Yang)
- merge qemu-svn
- new live migration implementation
- usb improvements
- migration fixes
- read UUID from qemu (Gleb Natapov)
- increase max cpu count to 255 for ia64 (Xiantao Zhang)
- handle INIT before SIPI (Gleb Natapov)
- update ppc for nmi changes (Christian Ehrhardt)
- fix external module generation unifdef problem (Sheng Yang)
- set 'g' bit of segment descriptor on AMD when necessary (Amit Shah)
- set 'busy' bit of task descriptor on AMD when necessary (Amit Shah)
- fixes some AMD->Intel live migration failures
- use emulator to adjust rip of pio string instructions (Guillaume Thouvenin)
- increase rmap preallocation (Marcelo Tosatti)
- fixes oops on mmu intensive smp workloads
- handle mmio in big real mode (Guillaume Thouvenin)
- reorganize ia64 guest data area (Xiantao Zhang)
- fix ia64 guest signal mask (Xiantao Zhang)
- reduce locking when vcpu is halted (Xiantao Zhang)
- update e820 table for ept real mode identity map page (Sheng Yang)
- fix external module build with kvmtrace
- fix ppc xer access (Christian Ehrhardt)
- fix ppc threading (Hollis Blanchard)
- implement libcflat for ppc 44x (Hollis Blanchard)
- set smp limit for ppc (Hollis Blanchard)
- fix ppc memory setup (Christian Ehrhardt)

Kvm-78 can be downloaded here

Multiplexing your monitor and serial port

Haydn Solomon — Fri, 24 Oct 2008 17:13:20 +0000

There’s a feature in kvm which allows you to multiplex your serial and monitor port over a single connection. This is another one of the many features inherited from the qemu tree and was added around kvm-57 which is probably not well known. I think this is one feature that may not be used much but I think it’s still good to know thy options.

The Syntax

The syntax to enable the multiplexing of your monitor port is as follows:

mon:dev_string

Your monitor is accessed using a key sequence of Ctrl-A then c. This key sequence will toggle between your monitor and serial port. For example, if you wanted to multiplex your monitor onto a telnet server listening on port 4444, you would start your kvm guest with the following option.

qemu-system-x86_64  -hda  windows.img  -serial  mon:telnet::4445,server,nowait

For more on accessing your monitor via telnet, see this post here on accessing your monitor across the network.

An example

For a concrete example, start your kvm guest using a syntax similar to the one described in the previous section, for example.

qemu-system-x86_64  -hda  windows.img  -m 400 -serial  mon:telnet::4445,server,nowait

Now type the following in a terminal window on your kvm host

telnet localhost 4445

When you do this you will get an output similar to the following.

Note that you will not get any response from your keyboard in this session. This is because when you initially connect, you’ll be connecting to the serial port via the telnet server. Now type Ctl-a then c and hit enter from within the same session. You’ll now see your kvm monitor come up as shown below.

Using this key sequence you can toggle between the serial and monitor sessions using your telnet server. To break out of the telnet server, issue the Ctrl-] and you’ll get the telnet prompt as shown below from which you can then quit.

The multiplexing your monitor and serial port is another one of those features that is probably not very popular but you never know when you might have to pull it out of your bag of tricks. Can you think of an application of this feature?

Share your thoughts and drop a comment if you can think of how you might use this feature.

KVM-77 Released: -std-vga deprecated, Improved disk performance

Haydn Solomon — Mon, 13 Oct 2008 14:16:10 +0000

Yesterday kvm maintainers released kvm-77 which includes a fix for the –std-vga regression described here after the last release.

Also included in this release are performance improvement for disk and a new syntax for vga options pulled from upstream qemu. The previous syntax of –std-vga has been deprecated for a new –vga option. Below is an excerpt from documentation describing the new change.

New vga option

Instead of having three command line switches ie -std-vga, -cirrusvga and -vmwarevga, you now provide one -vga switch which takes an argument, so that:

qemu -std-vga becomes qemu -vga std
qemu -cirrusvga becomes qemu -vga cirrus
qemu -vmwarevga becomes qemu -vga vmware

Changelog

The official changelog for kvm-77 released is as follows:

Changes from kvm-76:

- merge bochs-bios-cvs
- merge qemu-svn
- more -cpu options
- faster disk emulation (esp. with scsi/virtio)
- improved NMI support (Jan Kiszka)
- improve >4GB memory support (Alex Williamson)
- memory alias cleanups (Glauber Costa)
- fix kvmtrace segfault (Ryota OZAKI)
- make external module compile on split source/object configs (Alexander Graf)
- allows compiling on opensuse
- fix -std-vga regression
- fix migration failure at end of migration protocol
- map mmio pages for device assignment (Weidong Han)
- silence lapic kernel messages (Jan Kiszka)
- fix vcpu reset (Gleb Natapov)
- fix missed invlpg on EPT-enabled machines with EPT disabled (Marcelo Tosatti)
- device assignment on ia64 (Xiantao Zhang)
- memory type support on EPT (Sheng Yang)

KVM-77 can be downloaded here

Convirt 0.9.5 Released: Adds drag and drop live migration for KVM

Haydn Solomon — Thu, 02 Oct 2008 14:10:02 +0000

An alternative GUI management tool for KVM that was previously posted here just released version 0.9.5 yesterday adding drag and drop live migration for KVM.

The official changelog for the latest version is:

Drag and Drop Live Migration for KVM : Similar to Xen Live migration, now you can use drag and drop to migrate KVM virtual machines without downtime. In addition you can save a running KVM virtual machine state and restore it. KVM Management is also tested against common distros like Cent OS 5.2 and Ubuntu 8.0.4.
Storage management : This release adds support for shared storage management associated with a Server Pool. The users can declare their iSCSI, AOE and NFS shared storage and select Luns. ether disks and mount points at the time of provisioning. ConVirt gives Storage Utilization overview at Server Pool, Server or Virtual Machine level.

You can find out more about convirt here: www.convirt.net

The new uuid option

Haydn Solomon — Tue, 30 Sep 2008 12:08:00 +0000

Beginning with kvm-76, you can now assign a unique machine identifier to your kvm guest machine during startup using the –uuid option. This will probably be used by management tools based on libvirt to manage a database of already existing kvm guest machines since libvirt already uses the uuid format to identify virtual machines. The uuid format is in hexadecimal and of the format 8x-4x-4x-4x-12x where x is a digit so 8x is 8 hex digits and 4x is 4 hex digits etc. The new option allows you to do two things:

specify a uuid during startup and
a monitor command for displaying the uuid of your kvm guest machine.

The syntax for the uuid startup option is as follows:

qemu-system-x86_64 –hda windows.img –m 400 –uuid 12345678-1234-1234-1234-123456789abc

From within the monitor, you can now issue the info uuid to display the unique identifier for your kvm guest machine. For example, to display the uuid of the guest started above:

QEMU 0.9.1 monitor - type 'help' for more information

(qemu) info uuid
12345678-1234-1234-1234-123456789abc

KVM-76 Released: Performance Improvments, Qemu Merge, Kernel support for device assignment

Haydn Solomon — Mon, 29 Sep 2008 12:08:00 +0000

Yesterday, KVM-76 was released featuring another qemu merge and performance improvements for hosts without NPT or EPT. Only kernel support for device assigment has been added in preparation for pci passthrough but no userspace component has been added yet. The qemu merge brings with it improved support for scsi and usb support. You can now use a “real” scsi tape with qemu using the following syntax

-drive /dev/sgX,if=scsi

This option allow you to decode correctly the transfer length when the type of device is a tape. However, there are still issues when the application reading the tape goes beyond the end of the stream.

The usb support now addes for removing devices by host address and improved auto filter syntax. These features will be useful for libvirt since there’s no easy way to to find the guest usb address of a device. Below is an excerpt from the patch comments explaining the new features.

USB patch comments

usb: Support for removing device by host addr, improved auto filter syntax (Max Krasnyansky)

This patch adds support for removing USB devices by host address which is usefull for things like libvirtd because there is no easy way to find guest USB address of the host device. In other words you can now do:

       usb_add host:3.5
       ...
       usb_del host:3.5

Before the patch 'usb_del' did not support 'host:' notation.

----
Syntax for specifying auto connect filters has been improved.

Old syntax was
host:bus.dev
host:pid:vid

New syntax is
host:auto:bus.dev[:pid:vid]
In both the cases any attribute can be set to "*".

New syntax is more flexible and lets you do things like

host:3.*:5533:* /* grab any device on bus 3 with vendor id 5533 */

It's now possible to remove auto filters. For example:

usb_del host:auto:3.*:5533:*

Active filters are printed after all host devices in 'info usb' output. Which now looks like this:

Device 1.1, speed 480 Mb/s
        Hub: USB device 1d6b:0002, EHCI Host Controller
      Device 1.4, speed 480 Mb/s
        Class 00: USB device 1058:0704, External HDD
      Auto filters:
        Device 3.* ID *:*

Issues

There does seem to be one regression with the –std-vga option. You get the following error repeated in your console when using the option and your windows shows up black after initial boot screen.

kvm: get_dirty_pages returned -2

If you use kvm with full screen mode, this will affect you as only -std-vga can support the resolutions to do full screen with native resolution. In this case you'll either have to go back to kvm-75 or use rdp if full screen is important to you. I use a windows xp kvm guest for work and decided to revert to kvm-75 because of this issue. I suspect that a patch might be released for this before the next release.

Changelog

The official changelog from the mailing list is as follows:

- merge qemu-svn
- improved usb support
- vnc improvements
- pass uuid to guest
- scsi improvements
- change virtio-blk to guest-endian configuration (Liu Yu)
- add 'call near absolute' to x86 emulator and testsuite (Mohammed Gamal)
- drop support for kernel memory allocation (Glauber Costa)
- avoid fiddling with ISA mappings (Jan Kiszka)
- improves debugging support
- fix external module build on 2.6.22 hosts
- make kernel and userspace memory slot count agree (Sheng Yang)
- add 'cmp' to x86 emulator testsuite (Mohammed Gamal)
- allow 'make sync' from toplevel directory
- external module compatibility for get_user_pages_fast() api (Jan Kiszka)
- correct mtrr setup with >= 4GB RAM (Alex Williamson)
- fix ia64 boot with acpi enabled (Xiantao Zhang)
- initialize userspace kvm context (Uri Lublin)
- fix ia64 with >= 3GB RAM (Xiantao Zhang)
- avoid allocating translated code buffer when using kvm
- simplify memory registration (Glauber Costa)
- support kvmtrace with external module (Eduardo Habkost)
- ia64 init code simplification (Jes Sorensen)
- ia64 external module compatibility with older kernels (Xiantao Zhang)
- fix live migration due to slot number getting lost (Uri Lublin)
- prevent live migration if a device is missing on the target (Uri Lublin)
- fix include-compat/asm symlinks (Uri Lublin)
- do not execute halted vcpus (Marcelo Tosatti)
- fix global tlb flushes with NPT (Joerg Roedel)
- fix preemption error on real mode emulation
- fix page aging with EPT (Sheng Yang)
- simplify locking around ppc shadow tlb page access (Hollis Blanchard)
- cleanup redundant vmcs read (Jan Kiszka)
- fix 64-bit jmp instruction emulation
- move vmx msr definitions to common code
- better support for instructions that reference the accumulator
(Guillaume Thouvenin)
- add cmp acc, imm instruction emulation (Guillaume Thouvenin)
- kernel support for device assignment
(Allen Kay, Amit Shah, Weidong Han, Ben-Ami Yassour)
- switch to using get_user_pages_fast() (Marcelo Tosatti)
- nice speedup on 2.6.27+ hosts
- build fix for ia64 (Jes Sorensen)
- code cleanup (Harvey Harrison)
- avoid entering guest after a startup IPI (Gleb Natapov)
- simplify 'push reg' instruction emulation (Guillaume Thouvenin)
- fix error handling on vcpu creation failure (Glauber Costa)
- support Tukwilla ia64 processors (Xiantao Zhang)
- fix tlb flush when converting a large page to small pages (Marcelo Tosatti)
- out-of-sync shadow (Marcelo Tosatti)
- reduce inter-processor interrupt generation when using the PIC
(Marcelo Tosatti)

KVM-76 can be downloaded here.

Using bridged networking with Virt-manager

Haydn Solomon — Fri, 19 Sep 2008 12:08:00 +0000

In previous posts you can find instructions on how to configure bridged networking when managing your kvm guests by command line. However for many users experimenting with kvm, the virt-manager GUI tool is the preferred method of managing kvm guests in which case bridged networking configuration might pose a challenge. If you fall into the camp of users who prefer virt-manager and have ever wondered how to configure bridged networking , you’ll be able to use bridged networking with virt-manager after reading this post.

1. Configure your bridge

Your libvirt installation will provide default bridges for use with xen, qemu and other types of domains. For qemu/kvm, the default bridge should be called vnet0 and you can verify this by issuing the following command on your system.

# brctl show

bridge name     bridge id               STP enabled     interfaces
pan0            8000.000000000000       no
virbr0          8000.000000000000       yes
vnet0           8000.000000000000       yes

Th first thing you need to do is add a network script for your vnet0 bridge. On Fedora this script will be located under /etc/sysconfig/network-scripts/ but may differ for other linux distributions. So go ahead and create a new script at /etc/sysconfig/network-scripts/ifcfg-vnet0 as root user and edit the contents so that it looks like the following.

DEVICE=vnet0
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes

This completes set up of your vnet0 bridge.

2. Add your physical interface to the bridge

The next step is to add the physical interface that your kvm guest will be sharing to the vnet0 bridge configured above. For this post, we’ll assume that you’ll be using eth0 so to add eth0 to vnet0 bridge, issue the following command.

# brctl addif vnet0 eth0

You can verify that your eth0 was added to the vnet0 bridge issuing the brctl show command. Your output should look similar to the following.

# brctl show

bridge name     bridge id               STP enabled     interfaces
pan0            8000.000000000000       no
virbr0          8000.000000000000       yes
vnet0           8000.0019b97ec863       yes             eth0

Now you need to configure your network script for eth0. You will already have a script for eth0, but you’ll need to modify it by adding one line as BRIDGE=vnet0 so that it looks similar to the following script.

# Broadcom Corporation BCM4401-B0 100Base-TX
DEVICE=eth0
BRIDGE=vnet0
BOOTPROTO=dhcp
HWADDR=00:19:b9:7e:c8:63
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
IPV6INIT=no
PEERDNS=yes
NM_CONTROLLED=no
…
…

3. Restart your network services

In order for all the network script modifications to take effect, you need to restart your network services. Your command may differ slightly but for most linux distributions the following command will restart your network services.

# /etc/rc.d/init.d/network restart

Shutting down interface vnet0:  [  OK  ]
Shutting down interface eth0:  [  OK  ]
Shutting down loopback interface:  [  OK  ]
Bringing up loopback interface:  [  OK  ]
Bringing up interface eth0:  [  OK  ]
Bringing up interface vnet0:  
Determining IP information for vnet0... done.
[  OK  ]

My experience is that the dhcp on the bridge device usually takes a little longer than the other scripts so don’t worry if the line that says “determining ip information for vnet0” sits there for a min or two. Once this script completes, you’re finished with your system network configurations. Now, onto actually using bridged networking with virt-manager.

4. Configure Bridged Networking in Virt-Manager

Now comes the easy part. Configure your kvm guest in virt-manager as you normally would. When you get to the part that asks about your network type, choose to share with a physical device and you will be able to select the eth0 on bridge vnet0. Below is a screenshot of what you should see. You don’t need to specify a specific mac address unless you need to for some kind of static mapping.

Boot up your kvm guest which will be now connected to your local network. Your brctl show command will now show your kvm guest network added to your vnet0 bridge. Below is a sample output.

# brctl show

bridge name     bridge id               STP enabled     interfaces
pan0            8000.000000000000       no
virbr0          8000.000000000000       yes
vnet0           8000.0019b97ec863       yes             eth0
                                                        vnet1

Summary

Bridged networking with virt-manager is simpler than configuring bridged networking when managing kvm guests with command line since you don’t have to worry about things like qemu network scripts and generating random mac addresses. Virt-manager takes care of these things behind the scenes. What will be nice to see someday would be GUI tools for managing the bridges which would make the procedure even simpler.

Virtual Machine Manager 0.6.0-1 Release

Haydn Solomon — Tue, 16 Sep 2008 12:08:00 +0000

The latest release of the virtual machine manager software tagged as 0.6.0-1 has been released a few days ago adding new features for managing your kvm guest machines. You should expect to see packages coming to your favourite linux distribution in the not too distant future so watch out for it.

There are many new small features but the major changes are listed below from the official changelog.

• Add libvirt storage management support
• Basic support for remote guest installation
• Merge VM console and details windows
• Poll avahi for libvirtd advertisement
• Hypervisor autoconnect option
• Add sound emulation when creating new guests

The download can be found here: http://virt-manager.org/download.html

Monitor your KVM guests with nagios-virt

Haydn Solomon — Mon, 15 Sep 2008 12:08:00 +0000

Have you ever considered how you will monitor your production kvm guest machines? You could monitor them just like any other physical machine but that is not considered best practice in the virtualization industry and in fact touted as a shortcoming in virtual machine management practice. Well..there is a tool being developed by Redhat that relies on the libvirt interface for monitoring virtual machines based on kvm or any other hypervisor supported by the libvirt interface. This means that it obviously will only work for kvm guests being managed by tools based on libvirt such as virt-manager, virtsh etc. and will not work for your kvm guests being managed by command line.

What is Nagios?

Nagios is one of the most popular open source monitoring systems and considered the grandfather of opensource monitoring on linux. It is available by default in all major linux distributions and provides a solid, reliable framework for systems and network monitoring. One of the best features of nagios is how easy you can plugin your custom scripts to extend it’s capabilities. This is where nagios-virt comes in.

More information can be found on nagios here: Nagios Monitoring

The nagios-virt tool

Nagios-virt is an open source nagios plug-in monitor maintained by redhat designed for using the libvirt interface to monitor your virtual machines. The nagios-virt tool can be easily integrated with your current nagios installation if you already have one or can be used to setup a new nagios installation.

The project pages for nagios-virt can be found here: Nagios support for virtualization. It provides documentation for installation and basic instructions for getting it up and running. At the time of this writing it is only at version 0.2.0 so still a young project but development is happening at a rapid case as is usually the case for young projects. If you manage your kvm guests using libvirt tools, you should consider giving this a try if monitoring is something you need to do.