Oracle OVS, Broadcom Netextreme II and 10G network

We recently upgraded our LAN network from 1G to 10G but the Oracle OVS servers network cards showed only 1G link even though the advertised link was 10G.

[root@ovs1 ~]# ethtool em1
Settings for em1:
 Supported ports: [ FIBRE ]
 Supported link modes: 1000baseT/Full
 Supported pause frame use: Symmetric Receive-only
 Supports auto-negotiation: Yes
 Advertised link modes: 1000baseT/Full
 Advertised pause frame use: No
 Advertised auto-negotiation: Yes
 Link partner advertised link modes: 1000baseT/Full
 10000baseT/Full
 Link partner advertised pause frame use: Symmetric
 Link partner advertised auto-negotiation: Yes
 Speed: 1000Mb/s
 Duplex: Full
 Port: FIBRE
 PHYAD: 1
 Transceiver: internal
 Auto-negotiation: on
 Supports Wake-on: g
 Wake-on: d
 Current message level: 0x00000000 (0)

Link detected: yes

And the OVS had the famous bnx2x Broadcom driver which we have had issues earlier too.

[root@ovs1 ~]# ethtool -i em1
driver: bnx2x
version: 1.712.10
firmware-version: FFV7.10.17 bc 7.10.11

So when I tested the speed using iperf I got the following:

[root@ovs1 ~]# ./iperf64 -c 10.10.10.12
------------------------------------------------------------
Client connecting to 10.10.10.12, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.11 port 45871 connected with 10.10.10.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.10 GBytes 942 Mbits/sec

So only 1G link speed as expected when the card is showing 1000Mb/s speed.

After trying few different things the solution was to upgrade to latest kernel and latest bnx2x drivers. Current kernel was:

[root@ovs1 ~]# uname -a
Linux ovs1.local 3.8.13-68.3.3.el6uek.x86_64 #2 SMP Tue Jun 23 23:43:15 PDT 2015 
x86_64 x86_64 x86_64 GNU/Linux

And to upgrade it we added the public yum ovs repo to yum.repos.d

[root@ovs1 ~] cat /etc/yum.repos.d/ovm.new:
[3.3_ovs_repo]
gpgcheck = 0
baseurl = http://public-yum.oracle.com/repo/OracleVM/OVM3/3x_latest/x86_64/
name = 3.3_ovs_repo
enabled = 1

After running yum update kernel and rebooting host new kernel is in use:

[root@ovs1 ~]# uname -a
Linux ovs1.local 3.8.13-118.16.3.el6uek.x86_64 #2 SMP Wed Feb 8 
17:40:13 PST 2017 x86_64 x86_64 x86_64 GNU/Linux

And the drivers are updated as well:

[root@ovs1 ~]# ethtool -i em1
driver: bnx2x
version: 1.713.01
firmware-version: FFV7.10.17 bc 7.10.11

However the link shows up quite weird even though speed is 10000Mb/s with autonegotiation on. It doesn’t have advertised speed anymore as 10000base-T!

[root@ovs1 ~]# ethtool em1
Settings for em1:
 Supported ports: [ FIBRE ]
 Supported link modes: 1000baseKX/Full
 Supported pause frame use: Symmetric Receive-only
 Supports auto-negotiation: Yes
 Advertised link modes: Not reported
 Advertised pause frame use: Symmetric Receive-only
 Advertised auto-negotiation: Yes
 Link partner advertised link modes: 1000baseKX/Full
 Link partner advertised pause frame use: Symmetric
 Link partner advertised auto-negotiation: Yes
 Speed: 10000Mb/s
 Duplex: Full
 Port: FIBRE
 PHYAD: 1
 Transceiver: internal
 Auto-negotiation: on
 Supports Wake-on: g
 Wake-on: d
 Current message level: 0x00000000 (0)

Link detected: yes

But now running the same iperf test gives new results:

[root@ovs1 ~]# ./iperf64 -c 10.10.10.12
------------------------------------------------------------
Client connecting to 10.10.10.12, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.11 port 57350 connected with 10.10.10.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 5.34 GBytes 4.58 Gbits/sec

So lot better throughput! I’m not so keen recommending the Broadcom NIC to anyone as we have had lot of issues on setting them up with Oracle VM / OVS. Compared to Intel based 10Gb NIC which has been lot easier to manage.

There is also note on MOS on using iperf: How to Use Iperf to Test Network Performance (Doc ID 1507397.1)

Why the card still doesn’t show correctly the advertised and supported speed I don’t know but need to figure out.