We recently upgraded our LAN network from 1G to 10G but the Oracle OVS servers network cards showed only 1G link even though the advertised link was 10G.
[root@ovs1 ~]# ethtool em1 Settings for em1: Supported ports: [ FIBRE ] Supported link modes: 1000baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Advertised link modes: 1000baseT/Full Advertised pause frame use: No Advertised auto-negotiation: Yes Link partner advertised link modes: 1000baseT/Full 10000baseT/Full Link partner advertised pause frame use: Symmetric Link partner advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: FIBRE PHYAD: 1 Transceiver: internal Auto-negotiation: on Supports Wake-on: g Wake-on: d Current message level: 0x00000000 (0) Link detected: yes
And the OVS had the famous bnx2x Broadcom driver which we have had issues earlier too.
[root@ovs1 ~]# ethtool -i em1 driver: bnx2x version: 1.712.10 firmware-version: FFV7.10.17 bc 7.10.11
So when I tested the speed using iperf I got the following:
[root@ovs1 ~]# ./iperf64 -c 10.10.10.12 ------------------------------------------------------------ Client connecting to 10.10.10.12, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.10.10.11 port 45871 connected with 10.10.10.12 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.10 GBytes 942 Mbits/sec
So only 1G link speed as expected when the card is showing 1000Mb/s speed.
After trying few different things the solution was to upgrade to latest kernel and latest bnx2x drivers. Current kernel was:
[root@ovs1 ~]# uname -a Linux ovs1.local 3.8.13-68.3.3.el6uek.x86_64 #2 SMP Tue Jun 23 23:43:15 PDT 2015 x86_64 x86_64 x86_64 GNU/Linux
And to upgrade it we added the public yum ovs repo to yum.repos.d
[root@ovs1 ~] cat /etc/yum.repos.d/ovm.new: [3.3_ovs_repo] gpgcheck = 0 baseurl = http://public-yum.oracle.com/repo/OracleVM/OVM3/3x_latest/x86_64/ name = 3.3_ovs_repo enabled = 1
After running yum update kernel and rebooting host new kernel is in use:
[root@ovs1 ~]# uname -a Linux ovs1.local 3.8.13-118.16.3.el6uek.x86_64 #2 SMP Wed Feb 8 17:40:13 PST 2017 x86_64 x86_64 x86_64 GNU/Linux
And the drivers are updated as well:
[root@ovs1 ~]# ethtool -i em1 driver: bnx2x version: 1.713.01 firmware-version: FFV7.10.17 bc 7.10.11
However the link shows up quite weird even though speed is 10000Mb/s with autonegotiation on. It doesn’t have advertised speed anymore as 10000base-T!
[root@ovs1 ~]# ethtool em1 Settings for em1: Supported ports: [ FIBRE ] Supported link modes: 1000baseKX/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Advertised link modes: Not reported Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Link partner advertised link modes: 1000baseKX/Full Link partner advertised pause frame use: Symmetric Link partner advertised auto-negotiation: Yes Speed: 10000Mb/s Duplex: Full Port: FIBRE PHYAD: 1 Transceiver: internal Auto-negotiation: on Supports Wake-on: g Wake-on: d Current message level: 0x00000000 (0) Link detected: yes
But now running the same iperf test gives new results:
[root@ovs1 ~]# ./iperf64 -c 10.10.10.12 ------------------------------------------------------------ Client connecting to 10.10.10.12, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.10.10.11 port 57350 connected with 10.10.10.12 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 5.34 GBytes 4.58 Gbits/sec
So lot better throughput! I’m not so keen recommending the Broadcom NIC to anyone as we have had lot of issues on setting them up with Oracle VM / OVS. Compared to Intel based 10Gb NIC which has been lot easier to manage.
There is also note on MOS on using iperf: How to Use Iperf to Test Network Performance (Doc ID 1507397.1)
Why the card still doesn’t show correctly the advertised and supported speed I don’t know but need to figure out.