bad network performance with 10Gbit
Hello,
I have trouble with the network performance inside my virtual machines.
My KVM-Host machine is connected to a 10Gbit Network. All interfaces are configured to a mtu of 4132. On this host I have no problems and I can use the full bandwidth:
# iperf -c 10.10.80.100 -w 65536 -p 12345 -t 60 -P4
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-60.0 sec 18.8 GBytes 2.69 Gbits/sec
[ 5] 0.0-60.0 sec 15.0 GBytes 2.14 Gbits/sec
[ 6] 0.0-60.0 sec 19.3 GBytes 2.76 Gbits/sec
[ 3] 0.0-60.0 sec 15.1 GBytes 2.16 Gbits/sec
[SUM] 0.0-60.0 sec 68.1 GBytes 9.75 Gbits/sec
Inside a virtual machine don't reach this result:
# iperf -c 10.10.80.100 -w 65536 -p 12345 -t 60 -P 4
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 5.65 GBytes 808 Mbits/sec
[ 4] 0.0-60.0 sec 5.52 GBytes 790 Mbits/sec
[ 5] 0.0-60.0 sec 5.66 GBytes 811 Mbits/sec
[ 6] 0.0-60.0 sec 5.70 GBytes 816 Mbits/sec
[SUM] 0.0-60.0 sec 22.5 GBytes 3.23 Gbits/sec
I only can use 3,23Gbits of 10Gbits. I use the virtio driver for all of my vms, but I have also tried to use the e1000 nic device instead.
With starting the iperf performance test on multiple vms simultaneously I can use the full bandwidth of the kvm host's interface. But only one vm can't use the full bandwith. Is this a known limitation, or can I improve this performance?
Does anyone have an idea how I can improve my network performance? It's very important, because I want to use the network interface to boot all vms via AOE (ATA over Ethernet).
If I mount a harddisk via AOE inside a vm I get only this results:
Write |CPU |Rewrite |CPU |Read |CPU
102440 |10 |51343 |5 |104249 |3
On the KVM Host I get those results on a mouted AOE Device:
Write |CPU |Rewrite |CPU |Read |CPU
205597 |19 |139118 |11 |391316 |11
If I mount the AOE Device directly on the kvm-host and put a virtual harddisk-file in it I got the following results inside a vm using this harddisk-file:
Write |CPU |Rewrite |CPU |Read |CPU
175140 |12 |136113 |24 |599989 |29
Here more details about my environment:
KVM Version:
QEMU PC emulator version 0.12.3 (qemu-kvm-0.12.3), Copyright (c) 2003-2008 Fabrice Bellard
0.12.3+noroms-0ubuntu9
KVM Host Kernel:
2.6.32-22-server #36-Ubuntu SMP Thu Jun 3 20:38:33 UTC 2010 x86_64 GNU/Linux
KVM Host OS:
Distributor ID: Ubuntu
Description: Ubuntu 10.04 LTS
Release: 10.04
Codename: lucid
KVM Guest Kernel:
2.6.32-22-server #36-Ubuntu SMP Thu Jun 3 20:38:33 UTC 2010 x86_64 GNU/Linux
KVM Guest OS:
Distributor ID: Ubuntu
Description: Ubuntu 10.04 LTS
Release: 10.04
Codename: lucid
I would feel very happy if someone can help to solve my problem!
Hello everybody,
I now have tested a few things:
I have upgraded to a new kernel with vhost_net support (2.6.35-6) and have compiled the new qemu-kvm version from git://git.kernel.org/pub/scm/linux/kern ... mu-kvm.git (0.12.50). But I still got the same "bad" results.
Someone told me to have a look at my cpu usage during this test. But in my opinion it seems to be ok:
KVM-Guest
top - 08:50:55 up 11:25, 1 user, load average: 0.43, 0.14, 0.05
Tasks: 81 total, 1 running, 80 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.1%us, 0.4%sy, 0.0%ni, 98.5%id, 0.8%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu1 : 0.2%us, 0.4%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 1023532k total, 597276k used, 426256k free, 70772k buffers
Swap: 1049592k total, 0k used, 1049592k free, 447928k cached
KVM-Host
Tasks: 319 total, 1 running, 318 sleeping, 0 stopped, 0 zombie
Cpu0 : 9.6%us, 9.9%sy, 0.0%ni, 64.2%id, 0.0%wa, 0.0%hi, 16.2%si, 0.0%st
Cpu1 : 6.4%us, 12.4%sy, 0.0%ni, 63.0%id, 0.3%wa, 0.0%hi, 17.9%si, 0.0%st
Cpu2 : 4.6%us, 2.3%sy, 0.0%ni, 92.8%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu3 : 4.3%us, 3.0%sy, 0.0%ni, 92.1%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st
Cpu4 : 0.3%us, 3.3%sy, 0.0%ni, 95.4%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%st
Cpu5 : 1.0%us, 0.3%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.4%us, 1.6%sy, 0.0%ni, 97.5%id, 0.0%wa, 0.0%hi, 0.4%si, 0.0%st
Cpu7 : 1.0%us, 1.3%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8 : 0.0%us, 1.4%sy, 0.0%ni, 98.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 0.3%us, 3.9%sy, 0.0%ni, 95.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 0.7%us, 1.9%sy, 0.0%ni, 96.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 : 0.7%us, 0.7%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu12 : 1.3%us, 1.3%sy, 0.0%ni, 97.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu13 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu14 : 10.2%us, 1.1%sy, 0.0%ni, 88.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu15 : 28.8%us, 5.1%sy, 0.0%ni, 66.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 33005428k total, 5292376k used, 27713052k free, 120588k buffers
Swap: 4194296k total, 0k used, 4194296k free, 828588k cached
I had problems with 10Gbe using HP Virtual Connect and bnx2x. Had to disable the TCP offload "options bnx2x disable_tpa=1" in the modprobe.conf. The VM's cannot access these features.
Hello MarkF,
thanks for your post. I not have a broadcomm interface, but I will try to find this option for my intel card.
best regards
René
Hello MarkF,
I have searched for an option for disabling tpa with my Intel 10Gbit Card. I had only found a way to disable TPA (Transparent Packet Aggregation) for Broadcom cards, for intel I didn't found this option.
best regards
René
You may have better performance mounting the AOE device in the host and then using it as block device via virtio in the guest.
Best regards.
Post new comment