KVM - The Linux Kernel-Based Virtual Machine
News, Blogs and Resources on the Linux (KVM) Kernel-Based Virtual Machine

Get a performance boost by backing your KVM guest with hugetlbfs

Support for hugetlbfs was added with the release of kvm-62 and it can give your kvm guest machine a performance boost anywhere up to 10%. KVM provides you with an option of using hugetlbfs by using the –mem-path option when starting your kvm guest machine. Just as a sidenote, the mem-path option also supports tmpfs filesystems.

 

 

What is hugetlbfs and why would you want to use it?

It is common nowadays for architectures to support more than one page size. As an example, x86 allows you to choose 4kb or 2MB pages. In the past, Linux traditionally used large pages for supporting the actual kernel image only but in kernel 2.6 linux started allowing processes to use huge pages. The whole purpose for huge page support is performance and you can get up to a 10% performance boost based on the application you’re running. Huge pages tend to give a bigger performance boost to applications that are memory and compute intensive. In the following sections, I’ll walk through the procedure for backing your kvm guest with huge pages.

 

1. Check that your kernel has support for Huge Pages

First thing you need to do is check that your kernel has support for huge pages. If you’re running a 2.6 kernel then you most likely have huge page support already. You can confirm by running the following command.

# grep HUGETLB /boot/config-`uname -r`
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y

I ran this command on a Fedora Linux host but it should be similar for your distribution. Your output should look similar if your kernel has hugetlbfs support enabled.
 

2. Calculate the number of huge pages you require

On x86, large pages are 2MB in size so you need to find out how many 2MB pages you need to back your kvm guest. Let’s say you want to use 512 MB for your guest machine then divide 512 by 2 to get 256 pages. Add a few extra pages for additional memory requirements, lets say 40 pages to give a total of 296 pages. Record this value as you’ll need it later on.

 

3. Setup your hugetlbfs filesystem

Now you need to setup your hugetlbfs filesystem on your host. As root, create a directory called hugepages like so.

mkdir /hugepages

Next add an entry to your /etc/fstab so that when you reboot your computer, your hugetlbfs will mount automatically. Your entry should look like the following.

hugetlbfs       /hugepages  hugetlbfs       defaults        0 0

Because it is difficult to reserve large pages on a busy system, it’s always best to reserve your huge pages shortly after restart.

 

4. Reboot and reserve your huge pages

As mentioned in the previous section, reserving huge pages is best done right after a reboot as it is difficult to reserve large pages on a busy system. Right after your computer has rebooted and using the value you calculated above for the number of huge pages you want to reserve, execute the following command.

echo 296 >  /proc/sys/vm/nr_hugepages

If you want this to be a permanent setup , you can also add it to your /etc/rc.local script so that it is always reserved on startup. To verify that you have large pages reserved inspect the contents of the /proc/meminfo file as follows.

# tail -n 5 /proc/meminfo 
HugePages_Total:   296
HugePages_Free:    296
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB

Now you are ready to start backing your virtual machine with huge pages.

 

5. Start your kvm guest with huge page support

You have verified everything to this point and ready to back your virtual machine with large pages. Start your virtual machine with using the –mem-path option as follows:

qemu-system-x86_64 –hda windows.img –m 512 –mem-path /hugepages

You will probably notice a performance increase in your virtual machine. You can check that you’re actually using huge pages by inspecting the /proc/meminfo file again. For example, this is what mine looked like while running a kvm guest backed with the values in this post.

# tail -n 5 /proc/meminfo 
HugePages_Total:   296
HugePages_Free:     39
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB

 That concludes the procedure for backing your guest with huge pages. It doesn't take that long to setup so give it a try and see if you get a performance boost.

See Also

Comments

sysctl.conf

Nice post Hadyn - I didn't know about the hugelfbfs option at all.

The best way of permanently setting these kernel parameters is via /etc/sysctl.conf, of course, rather than via /etc/rc.local - most distros (including RedHat- and debian-derived ones) support this. So for your example you'd just add:

    vm.nr_hugepages = 296

to /etc/sysctl.conf.

 

EPT and NPT

Its because of this I like your site

  • Say, does this also work if I have a processor that support EPT or NPT?
  • Also, I guess hugetlbfs is so effective because virtual machines are just processes on the host and the process is big in size? Bigger than normal processes.
  • "Huge pages tend to give a bigger performance boost to applications that are memory and compute intensive." Are you refering to virtual machines here? The article does not make that clear.
  • Does anything in KVM stop to work if we are using hugetlbfs? Like migration or ballooning?

Re: EPT and NPT

Thanks for the comments Blinkiz,

You have some good questions and I'll try to answer them as best as I could.

  1. Does it work with EPT/NPT?  This is a good question. I will actually post this as a question on the mailing list to make sure there are no issues. However, in my opinion there shouldn't be any issues.
  2. Is hugetlbfs effective for virtual machines due to virtual machine process size, bigger than normal processes?  Yes.
  3. Do virtual machines fall into category of memory and compute intensive? Yes.
  4. Hugetlbfs affect features like ballooning or migration? I don't think there should be any issues but I will also post this as a question on the mailing list to be certain.

 

Hugetlbfs

Blinkiz,

I had a discussion on the irc chat channel with some of the developers who were very helpful. Your question did generate some discussion. Turns out that all operations ie live migration, EPT/NPT and ballooning will work with hugepages. The only thing to be mindful about with ballooning is that since the balloon driver is set to work with 4k pages, it's unlikely that the guest will try to balloon a 2MB page due to fragmentation so ballooning will not be effective with huge pages.

 

So is it possible to use this

So is it possible to use this even if I start my machines with libvirt?

Re: hugepages with libvirt

Currently it is not possible to use the hugepages option if managing your virtual machine with libvirt.

I see. Maybe simple "alias"

I see. Maybe simple "alias" feature of bash can solve this?

libvirt is looking after /bin/qemu-kvm, so, alias qemu-kvm='qemu-system-x86_64 -mem-path /hugepages' can work. It worked on my laptop when I tested. But that was not with libvirt. Probably this alias command can solve a lot of problem when libvirt don't have a feature but the command line qemu-system-x86_64 has.

4096kb vs 2048kb hugetlbfs size

On my laptop (ubuntu 8.04 desktop) I have a page size of 4096 kb. Is that bad? Should I try to lower it to 2048kb? How can I do this?

Re: alias

Blinkiz,

Yes, that is a hack that could work but it will mean all your guest machines will attempt to start using hugepages and you don't really have the option. Now if you're only running one machine or two and account for the hugepages memory that is ok.  From libvirt's point of view, there's no option to build the -mem-path into the command line yet. However, your hack is creative and is ok as long as the user understands what he/she is doing.

 

Re: 4096kb vs 2048kb hugetlbfs size

Blinkiz,

If that is the case, it's not bad at all, you might actually get better performance.  You don't have to lower it just make sure you calculate the right number of pages when setting it up.

 

I've configured KVM with huge

I've configured KVM with huge pages. When i start more than 2 instances of qemu-kvm, it crashes with the following error:

"Could not allocate dynamic translator buffer"

KVM-75
2GB = 1,5GB HugePages / 512MB NormalPages
HugePages = 15% in use
NormalPages = 12% in use

Can you paste the output of

Can you paste the output of your /proc/meminfo even if your virtual machines are not running? Also paste the startup commands for your two virtual machines.

Without VMs

Without VMs running:
MemTotal: 2052552 kB
MemFree: 190404 kB
Buffers: 340 kB
Cached: 35828 kB
SwapCached: 0 kB
Active: 17588 kB
Inactive: 25672 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 7136 kB
Mapped: 1944 kB
Slab: 10764 kB
SReclaimable: 1912 kB
SUnreclaim: 8852 kB
PageTables: 628 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 130276 kB
Committed_AS: 34648 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 10636 kB
VmallocChunk: 34359726843 kB
HugePages_Total: 875
HugePages_Free: 875
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

With 2 VMs:
MemTotal: 2052552 kB
MemFree: 18860 kB
Buffers: 616 kB
Cached: 63432 kB
SwapCached: 0 kB
Active: 52688 kB
Inactive: 28540 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 17220 kB
Mapped: 14848 kB
Slab: 12720 kB
SReclaimable: 1780 kB
SUnreclaim: 10940 kB
PageTables: 956 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 65764 kB
Committed_AS: 385600 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 11980 kB
VmallocChunk: 34359725811 kB
HugePages_Total: 938
HugePages_Free: 680
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB

CommandLine:
qemu-kvm -smp 1 -m 256 -boot n -netnic,macaddr=00:16:3e:00:0:01,model=rtl8139 -net tap,ifname=tap0 -std-vga -usb -usbdevice tablet -localtime -mem-path /hugepages -monitor unix:/var/run/kvm/example/monitor,server,nowait -vnc:0 -daemonize

I've startet memtest86 within the VMs to consume all available memory.

ErrorMessage: Could not allocate dynamic translator buffer

From your output, it looks

From your output, it looks like you should have more than enough memory on your hugepages filesystem for your two vms. The only thing that looks strange is that the "HugePages_Total" is different with and without vms. That should not change. I will test this scenario myself and see if I have any issues and let you know.

I changed the ammount of

I changed the ammount of HugePages, during my tests. But a second try shows, that it does not matter if HugePages_Total is the same or not. There is enough normal memory free (79Megs).

There are a lot of hints out there about /dev/shm for the qemu/kvm vms. How is /dev/shm related to huge pages? Are huge pages a replacement for this /dev/shm tuning?

By the way: thanks for this blog!

Ok. I tested the hugepages

Ok. I tested the hugepages with 3 vms and had no problems so it should work. We'll have to investigate why you're getting that error message.

As far as shm is concerned, it is not related to hugepages but rather related to tmpfs which you can also use with the -mem-path option. I'll do a post on that procedure soon.

Thank you for your interest in the blog and your participation.

I've investigated the problem

I've investigated the problem a little bit further:

QEmu/KVM needs round about 16-20Megs for itself when using hugepages, espacially during startup it seems that KVM uses even more memory.

This 16-20Megs are related to inodes and direntries in the fs cache (have a look at the Cached/Active values). Cache pages which are active can not be recalimed by the kernel.
This inodes and direntries are produced by the hughetblfs, because it's a fs.

So you need 10% of the vm memory additional, to be able to start the vm.

Hmm, the old game of IT: memory versus speed ;-)

This 16-20Megs, which are

This 16-20Megs, which are related to inodes and direntries in the fs cache, are stored in normal memory pages and not in hugepages! This was the source of the problem, because assigning 90% of the memory to hugepages has shorten the ammount of free normal pages to much.

Good research Anonymous5.

Good research Anonymous5.

That information will be useful for anyone else experimenting with hugepages.

no more than 3.5 GB for vm?

Hi,
just tried hugetables on a server. Gave 44GB memory for hugetables. Works fine for vms up to 3.5 GB - if given more, the vm (Debian stable, 64 bit) crashes during boot:

unhandled vm exit: 0x31 vcpu_id 0
rax 0000000000000080 rbx ffff81011fc01000 rcx ffffffff8064ef00 rdx ffff810000000000
rsi 0000000000000009 rdi 0000000000000286 rsp ffffffff80573e08 rbp 0000000000000080
r8 ffffffff8064efab r9 000000000001ffff r10 0000000000000000 r11 0000000000000002
r12 ffffffff80505380 r13 ffff81011fc01000 r14 ffffffff805ac7c0 r15 0000000000000000
rip ffffffff80295888 rflags 00010202
cs 0010 (00000000/ffffffff p 1 dpl 0 db 0 s 1 type b l 1 g 1 avl 0)
ds 0018 (00000000/ffffffff p 1 dpl 0 db 1 s 1 type 3 l 0 g 1 avl 0)
es 0018 (00000000/ffffffff p 1 dpl 0 db 1 s 1 type 3 l 0 g 1 avl 0)
ss 0018 (00000000/ffffffff p 1 dpl 0 db 1 s 1 type 3 l 0 g 1 avl 0)
fs 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
gs 0000 (ffffffff8053b000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
tr 0040 (ffff810001024300/00002087 p 1 dpl 0 db 0 s 0 type b l 0 g 0 avl 0)
ldt 0000 (00000000/ffffffff p 0 dpl 0 db 0 s 0 type 0 l 0 g 0 avl 0)
gdt ffffffff80574000/80
idt ffffffff805e9000/fff
cr0 8005003b cr2 0 cr3 201000 cr4 6a0 cr8 0 efer d01
Aborted

Anything i can do about that?
thanks,
Kai

Re: hugepages bug

Kai,

I know that since kvm-86 some patches were added for hugepages bug that might solve your problem. You can try testing again after next release. You can see discussion at the following link and test the patch if you are familiar with applying patches.

http://www.mail-archive.com/kvm@vger.kernel.org/msg16069.html

invalid option

Hayden, I'm running gentoo 2.6.29-hardened kernel and kvm-85-r2. When I try to use -mem-path I get an "invalid option". Searching the web I haven't had any hits on this error which usually means I'm doing something fundamentally and obviously wrong but for whatever reason just unable to spot what it is. Everything seems to be set up correctly, do you know of any reason I would be getting an error trying to use this option?
The error:

# kvm -mem-path /hugepages -hda gentoo-i386.img -m 256 -cdrom install-amd64-minimal-2008.0-r1.iso -boot d -net nic,vlan=0,macaddr=52:54:00:21:43:65 -net tap,vlan=0
kvm: invalid option -- '-mem-path'

My settings:

# equery l kvm
* Searching for kvm ...
[IP-] [ ~] app-emulation/kvm-85-r2 (0)

# grep HUGETLB /boot/config-`uname -r`
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y

# cat /proc/sys/vm/nr_hugepages
2048

# tail -n 5 /proc/meminfo
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 5824 kB
DirectMap2M: 8382464 kB

# grep hugep /etc/mtab
hugetlbfs /hugepages hugetlbfs rw 0 0

# sysctl vm.nr_hugepages
vm.nr_hugepages = 2048

# uname -r
2.6.29-hardened

There was a spelling error

There was a spelling error with kvm-85 in that -mem-path was changed to -mempath. Note that this was corrected in kvm-86. See link below.

http://www.linux-kvm.com/content/mempath-option

Thanks Hayden

I've really learned a lot from your site

Hi, you can pre-allocate

Hi, you can pre-allocate memory on system boot with kernel parameter: hugepages=N, where 'N' = the number of requested huge pages requested.

Not working for me

Hi Hayden,

Following your instruction, I tried with setting hugetblfs for KVM guest. However, it does not seem to use the hugepage.

I am running Gentoo 2.6.30-r8.

# grep HUGETLB /boot/config-`uname -r`
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y

# tail -n 7 /proc/meminfo
HugePages_Total: 148
HugePages_Free: 148
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 4096 kB
DirectMap4k: 36856 kB
DirectMap4M: 872448 kB

Also the pagesize seems to be 4MB rather than 2MB. Is this normal?

Re: Not working for me

Hugepage size on x86_32 is 4MB so that is fine. As far as kvm not using them this sounds like it might be an issue with kvm running on 32-bit host but I'm not sure. I'll have to investigate.

VM does not boot

Hi Haydn, I followed your instruction and it seemed everything was fine except my kvm virtual did not fired up event huge pages was used

[sysadmin@vir1 /]$ tail -n 5 /proc/meminfo
VmallocChunk: 34359467183 kB
HugePages_Total: 600
HugePages_Free: 86
HugePages_Rsvd: 9
Hugepagesize: 2048 kB

What seem to be wrong here?
Thanks

Re: VM does not boot

Are you getting any error messages? How much memory does your host have in total?

VM does not boot

Thank you for quick reply. The reason WM does not boot probably cause by network error. The VM is view through java viewer which requires the VM's IP address. I am currently looking for a way to optimize qemu-kvm. I have about 6 servers,each is capable to host 8 VMs concurrently, connect through the same gateway. Each has 12GB of memory. The problem I have is when I connect to the network from outside to open any VM, I have really bad screen redraw problem doesn't matter the guest OS is Winxp, 2003, Fedora 11-12. I tried to use para-virtual driver for the video and network with no help. If I connect to them locally, all the Vm were much better...
That why I think I should give hugepages a shot. Any idea?

Thank you

plz solve my problem!! If you

plz solve my problem!!
If you have an existing system you need to install xen-hypervisor-3.4-i386 and purge xen-hypervisor-3.2-1-i386 as the older Xen hypervisor won’t boot the newer kernel. This also requires installing xen-utils-3.4 and removing xen-utils-3.2-1 as the utilities have to match the kernel. You don’t strictly need to remove the old hypervisor and utils packages as it should be possible to have dual-boot configured with old and new versions of Xen and matching Linux kernels. But this would be painful to manage as update-grub doesn’t know how to match Xen and Linux kernel versions so you will get Grub entries that are not bootable – it’s best to just do a clean break and keep a non-Xen version of the older kernel installed in case it doesn’t initially boot.
regards
156-215 dumps

You are right. It isn't

You are right. It isn't booting properly because of a network error. You need to make sure that the VM has an IP address assigned to it. What type of VMWare are you using? Do your servers have enough memory?

Post new comment

The content of this field is kept private and will not be shown publicly.
Type the characters you see in this picture. (verify using audio)
Type the characters you see in the picture above; if you can't read them, submit the form and a new image will be generated. Not case sensitive.