KVM - The Linux Kernel-Based Virtual Machine
News, Blogs and Resources on the Linux (KVM) Kernel-Based Virtual Machine

Using KSM (Kernel Samepage Merging) with KVM

Kernel SamePage Merging is a recent linux kernel feature which combines identical memory pages from multiple processes into one copy on write memory region. Because kvm guest virtual machines run as processes under linux, this feature provides the memory overcommit feature to kvm so important to hypervisors for more efficient use of memory. So if you need to run multiple virtual machines on a host where memory is a constraint, then KSM is your solution. Even in cases where memory is not a constraint you might have a particular application that you want separated onto individual virtual machines and want maximum scalability. One case that might come to mind is webhosting where you might want to maximize scalability. Redhat claims that tests with KSM were able to achieve 600 vms on a host with 48 cores and 256 GB RAM.

In this post I’ll be walking through the basics of using KSM with kvm and most of it will apply to other applications as well. This feature has been accepted for inclusion in linux kernel 2.6.32 release but is already backported for testing in some kernel builds. Fedora 12 is one distribution I know where you can start testing KSM but note that Fedora 12 is considered unstable at this time so only use it for testing. For other distributions you’ll have to check with the corresponding maintainers.

 

Verfiy Kernel KSM Support

First thing you want to do is verify that KSM is available and enabled in your kernel. Future releases of the kernel will enable this by default but you should verify this at least for now. Run the following command to verify that KSM is enabled in your kernel.

grep KSM /boot/config-`uname –r`

Below is a snapshot of what you should see if KSM is enabled.

You will also see a subdirectory for KSM under /sys/kernel/mm/KSM filesystem showing in-kernel values related to KSM.

By default KSM is limited to 2000 kernel pages which can be seen by inspecting the max_kernel_pages file under the /sys/kernel/mm/KSM filesystem.

# cat /sys/kernel/mm/KSM/max_kernel_pages
2000

Once you have verified that your kernel has KSM support, the next step is to verify that your version of qemu-kvm supports samepage merging.

 

Verify KVM support for Samepage Merging

In order for your KVM guests to take advantage of KSM, your version of qemu-kvm must explicitly request from the kernel that identical pages be merged using the new madvise interface. The patch for this feature was added to the kvm development tree just recently following the kvm-88 release. If you’re compiling kvm yourself you can verify whether your version of kvm will support KSM by inspecting exec.c source file for the following lines of code.

#ifdef MADV_MERGEABLE
        madvise(new_block->host, size, MADV_MERGEABLE);
#endif

If you don’t see these lines in your exec.c file then your kvm process will still run fine but but it won’t take advantage of KSM. If you do have KSM support in your version kvm then you can expect to see much more efficient use of memory on your host when running multiple kvm guests.

 

Run Multiple Similar Guests

Once you’ve verified support for KSM in your kernel and in kvm as described above you simply need to run multiple similar guests to take advantage of kernel samepage merging. It doesn’t matter whether you use command line, virt-manager or any other management tool as long as your underlying kvm process supports KSM. With multiple virtual machines running, you can verify that KSM is working by inspecting the following file to see how many pages are being shared between your kvm guests.

cat /sys/kernel/mm/KSM/pages_sharing

Once this value is greater than zero, you’re taking advantage of KSM.

 

Conclusion

One of the most popular applications of server virtualization technology is consolidating physical servers into virtual machines on a single server. Most enterprise hypervisors provide memory overcommit features in order to gain even more usage out of the same physical memory resources and thereby more dollar savings. KSM in the linux kernel adds this important feature which allows kvm to also provide these savings so important to enterprises. Kernel Samepage Merging has been accepted for inclusion in the 2.6.32 linux kernel so this is a feature you can expect to benefit from very soon. If you can get your hands on a earlier kernel backported to KSM, give it a try.

 

Additional Resources

KSM has been in development for some time now. At the KVM 2008 forum, a presentation was made by Izik Eidus who developed the kernel feature explaining some of the architecture behind KSM. I’ve attached the slides from the presentation below for reference in case you haven’t seen it yet.
 

AttachmentSize
KvmForum2008_KSM.pdf145.92 KB

See Also

Comments

Does it support hugepages?

This is a great feature. I love seeing this coming into the kernel. So so great!!

Does anyone know if this is going to work with hugepages active at the same time? I saw at fedora 12 feature page that they are going to bring these two features at the same time. But is this only for fedora or for all 2.6.32 kernels?

small note: cat

small note: cat /sys/kernel/mm/KSM/pages_shared is the wrong place to look at (this will show how many pages with DIFFERENT CONTENT were shared), the right place to look is:
/sys/kernel/mm/KSM/pages_sharing

Re: pages_sharing

Thanks,

I've updated the post to show this.

In addition it worth to note

In addition it worth to note that it wont do much as far as /sys/kernel/mm/KSM/max_kernel_pages will stay at 2000 (this mean that only 2000 pages with different values can be merged)

To get full effect of it you probably want to put there 0 (unlimited kernel pages that can be allocated) or bigger number of 2000

Re: does it support hugepages

blinkiz,

KSM will not support hugepages. See the following link for discussion on mailing list.

http://www.mail-archive.com/kvm@vger.kernel.org/msg24230.html

#defines

Note that you need to make sure that
#define MADV_MERGEABLE 12
#define MADV_UNMERGEABLE 13

Are defined by your libc, or if your libc isn't new enough, add them somewhere

Guido

Project Page/Driver download?

Hi,

is the patch/driver available somewhere I can download to patch my kernel (from kernel.org)?

thanks a lot,
James

Re: ksm project page/driver download

James,

See the following KSM feature page for latest Fedora release. Under the Documentation Section you'll find links to various patches for KSM.

https://fedoraproject.org/wiki/Features/KSM

I've installed today the

I've installed today the 2.6.32-rc8 kernel within ubuntu and added ksm.

I've too installed qeme-kvm from git source and checked if exec.c includes the ksm stuff.

After that I started some VM's:

cat /sys/kernel/mm/ksm/pages_sharing
0

cat /sys/kernel/mm/ksm/pages_shared
0

cat /sys/kernel/mm/ksm/max_kernel_pages
507512

cat /sys/kernel/mm/ksm/pages_unshared
0

Where can be the problem? It seems that ksm is not used.

Thanks
magy

Re: ksm not running

Fedora provides a startup script which sets some values and starts up ksm. I don't know if ubuntu provides such a script so check to make sure. In any case, in order to start up ksm issue the following command.

echo 1 > /sys/kernel/mm/ksm/run

Now check your pages_sharing, pages_shared values and you should see a value.

Thanks for the reply. Tested

Thanks for the reply.

Tested it with /sys/kernel/mm/ksm/run set to 1.

/sys/kernel/mm/ksm/pages_shared
/sys/kernel/mm/ksm/pages_sharing

are still showing a zero.
It seems that the ubuntu (debian) glibc has no support for it.

I'm gona test it later when I found time.

KSM works great! I'm runing

KSM works great!

I'm runing 50 copys of WinXP installations at my server.
With out KSM memory usage was ~9200Mb, after starting KSM memory usage was ~3200Mb. So about 6000Mb dedublicated data, it's great!

About starting of deduplication in qemu:

In exec.c does't included header where defined MADV_MERGEABLE, so dedublication didn't works.
Fo testing purposes i used:

//#ifndef MADV_MERGEABLE
madvise(new_block->host, size, 12);
//#endif

So KSM starts work.

Regards, Alexander.

node0 ~ # while [ 1 ]; do cat

node0 ~ # while [ 1 ]; do cat /sys/kernel/mm/ksm/pages_shared; sleep 1; done
13522
13523
13519
13518
13520
13520
13528

HugeTlb vs KSM

It is possible to use KSM with HugeTlb?

Re: HugeTlb vs KSM

 You can only use KSM with 4k pages. You can find short discussion on that at the following link

http://www.mail-archive.com/kvm@vger.kernel.org/msg24230.html

Beginner

I am new to KSM.
PLease elaborate more on it.

Working with Centos/RHEL 5.4

It's not built into the kernel -- it's just a kmod kernel module. Hence none of the above instructions work under Centos. Instead, type:

ksmctl start 50 5000

and it will start its magic. Did some simple testing on 2 Windows VMs supposed to take up 2.5GB of memory and after a while, the footprint was reduced to 1.6GB. I will try more denser configs later on but this stuff is looking good.

Awesome feature!

I think 600 vms may be stretching it a bit, but it is an amazing feature nonetheless. I've seen as high as 540, which is still absolutely incredible. http://www.neverfailgroup.com
Carly S.

food for thought

i'm using a similar config, but i had a problem.
i'm using the 2.6.32 from backport
qemu-kvm 0.12.3 also from backport
but ksm is not working.
the strange thing is that i previously used qemu-kvm 0.11.1 also from backport and ksm worked fine.
Any idea why with the new qemu package KSM stopped working ?

In a recent product demo I

In a recent product demo I have seen it as high as 580~, which in my opinion is a phenomenal achievement.

Re:

I too am wondering the same thing. Have you figured out if KSM is compatible withe HugeTib? We (Web Dept.@ RSF) are currently involved with a project along the same lines as posted & would like further information before we proceed.

Excellent post. Very

Excellent post. Very informative. Will be bookmarking very short. =) test

cat

cat /sys/kernel/mm/KSM/pages_sharing

should be
cat /sys/kernel/mm/ksm/pages_sharing
(lower case ksm)

Post new comment

The content of this field is kept private and will not be shown publicly.
Type the characters you see in this picture. (verify using audio)
Type the characters you see in the picture above; if you can't read them, submit the form and a new image will be generated. Not case sensitive.