KVM - The Linux Kernel-Based Virtual Machine
News, Blogs and Resources on the Linux (KVM) Kernel-Based Virtual Machine
Low network performance even with virtio
August 10, 2010 - 3:39pm
I have this config (short):
Host:
- Baseboard Intel S5000VSA with 1Gbit adapters
- RHEL 5.4 x64
- bridge:
[root@vmserver-rhel5 samba]# brctl show bridge name bridge id STP enabled interfaces br0 8000.001517186320 no vnet1 vnet0 eth0 virbr0 8000.000000000000 yes
Guest:
- RHEL 5.4 x64
- Network model virtio (-net nic,macaddr=54:52:00:58:b7:c8,vlan=0,model=virtio -net tap,fd=21,script=,vlan=0,ifname=vnet1)
- lspci shows 00:03.0 Ethernet controller: Qumranet, Inc. Virtio network device
I did not install any additional drivers for netcard because there are virtio drivers in RHEL 5.4.
iperf from the host machine shows:
[root@vmserver-rhel5 samba]# iperf -c 172.16.1.10 -t 30 ------------------------------------------------------------ Client connecting to 172.16.1.10, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 172.16.1.202 port 35162 connected with 172.16.1.10 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-30.0 sec 2.49 GBytes 714 Mbits/sec
dstast at this moment shows:
usr sys idl wai hiq siq| read writ| recv send| in out | int csw 0 0 100 0 0 0| 156k 23k| 0 0 | 0 0 | 276 1171 0 1 95 0 0 3| 0 208k|4139k 176M| 0 0 |5045 7880 1 1 94 0 0 4| 0 0 |4133k 175M| 0 0 |5033 8643 0 1 94 0 0 4| 0 0 |4140k 175M| 0 0 |5043 7785 1 1 93 0 0 5| 0 146k|4128k 175M| 0 0 |5089 8926 0 1 95 0 0 3| 0 20k|4156k 176M| 0 0 |5062 7973
Low CPU utilization and good network speed
BUT!
iperf on guest shows:
[root@vm-rhel5 ~]# iperf -c 172.16.1.10 -t 30 ------------------------------------------------------------ Client connecting to 172.16.1.10, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 172.16.1.212 port 33138 connected with 172.16.1.10 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-30.0 sec 1.91 GBytes 547 Mbits/sec
dstat results at this moment on host:
usr sys idl wai hiq siq| read writ| recv send| in out | int csw 0 0 100 0 0 0| 156k 23k| 0 0 | 0 0 | 276 1172 2 12 84 0 0 2| 0 0 | 67M 70M| 0 0 |5030 30k 3 12 84 0 0 2| 0 44k| 68M 70M| 0 0 |5043 30k 3 11 85 0 0 2| 0 176k| 67M 70M| 0 0 |5031 29k 3 11 83 0 0 2| 0 0 | 67M 70M| 0 0 |5033 30k 2 10 85 0 0 3| 0 0 | 67M 70M| 0 0 |5035 29k 3 10 84 0 0 3| 0 0 | 66M 68M| 0 0 |4978 30k
dstat on guest
usr sys idl wai hiq siq| read writ| recv send| in out | int csw 0 0 98 1 1 0| 87k 189k| 0 0 | 0 0 |1120 87 0 4 60 0 18 19| 0 0 |1773k 65M| 0 0 |5503 869 1 5 60 0 16 16| 0 0 |1779k 65M| 0 0 |5595 869 0 4 61 0 19 14| 0 1376k|1773k 65M| 0 0 |5592 874 0 4 62 0 21 14| 0 0 |1794k 66M| 0 0 |5589 880 0 4 60 0 24 11| 0 220k|1793k 65M| 0 0 |5546 887 1 4 60 0 22 14| 0 0 |1787k 65M| 0 0 |5561 888
LOWER network performance and higher CPU utilization.
Is it normal? If not, what drivers, options etc. to look at?
I'm using kvm actively now for 3 years,
and the networking has been working for me always as expected:
virtio (when it was available) was faster
- mostly at 850-950 MBit
than
e1000
- in the range 550-600
than
rtl8139 (100MBit)
- sorry cant remember any numbers :-)
Since now all of those servers i built have been running on
"standard" Hardware (well chosen high end consumer mainboards).
All sytems named here are running debian lenny with more or less
recent kernels... (2.6.30-2.6.35) and qemu-kvm 0.12.4
The last two servers i've built:
System 1
Tyan S8005 with Opteron 1389
System 2
SuperMicro H8SCM-F with Opteron 4184
Both Systems are equipped with two Intel 82574 GBit LAN ports.
Both System performing as expected when testing the host to network
connection.
Both Systems show unexpected behaviour when testing network performance guest 2 host or guest 2 LAN with iperf:
System 1:
Guest to Host (virtio)
aprox. 400MBit
Guest to Host (e1000)
aprox. 200MBit
System 2:
Guest to Host (virtio)
aprox. 300MBit
Guest to Host (e1000)
aprox. 600MBit
Weird, isn't it ?
My Reference System (Gigabyte MA790FXT-UD5P with Phenom II X3 720BE)
get these numbers:
Guest Kernel:
2.6.30-bpo.1-amd64
------------------------------------------------------------
Client connecting to ls1, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.3.120 port 59101 connected with 192.168.3.100 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.55 GBytes 1.33 Gbits/sec
Has anybody out there an idea where to dig for the solution?
What i think is that it must be either a bios setting (and the bios
options of that server boards differ a bit from what i'm used to),
or that the opterons / chipsets (SR5650) require some special
kernel options active / disabled?
Well, i'd be really nice if someone has a solution right out of the box
( tweaking the nics with ethtool - tx rx checksumming and such, some timer
thing, etc.) or can give me a hint what to test next - since i'm totally clueless
in this case!
Thnx for reading until here! ;-)
joern
This night i found that using the debian lenny stock kernel
(2.6.26-2-amd64) on both host and guest (talking about the two opteron
system of my former post) the whole thing rocks!
Instead of the low performance i wrote about, using 2.6.26 i get
1.2 - 1.5 GBits/sec
Are some kernel gurus reading this and have an idea what ist causing
this massive difference?
Because i'd really have to have both: speedy network and features like ksm!
Thnx,
joern
I have noticed low performance using fedora binary drivers on w2008 guest and ubuntu host... just like there was some buffer filling up and after that traffic stops for a few seconds. After that it goes on and again stops. Very disruptive.
Hi Neikius
Did you get any improvements in the setup?
Checkout my report in another thread
http://www.linux-kvm.com/content/windows-2008-paravirtualized-virtio-driver
cheers
Artem.
--Never forget to thank KVM developers for excellent work
Post new comment