10 Gbps network for less than $60

I needed a fast connection between my NAS and my VM server. So I set on to the mission of doing 10 Gbps network connectivity for as cheap as possible.

This is mainly to run iSCSI for VMs. I had looked at Infiband as well. However driver support was poor and things aren’t guaranteed to work. IPoIB is also very processor intensive.

Things you will need

1. NIC cards – There are two options to do 10 Gbps for cheap here.
   1. Mellanox Connectx-2 – These are quite old cards with support only on GNU/Linux. However they are pretty cheap.
   2. Chelsio 110-1088-30 – These have dual SFP+ interfaces and have working drivers for FreeBSD and GNU/Linux but could be a little expensive. I picked up a pair of them for $37.36, inclusive of shipping.
      

   I have not investigated if either of them have Windows or Mac OS support. I got these through a friend in the US. If you live outside of US, you can ask Amazon to ship it to your country or use a parcel forwarding company like PPOBOX.

   Update 13/03/2017: The cost of the Chelsio NICs has shot up on Amazon. They are much cheaper on Ebay. As pointed out by Robert in the comments, Brocade BR-1020 with Brocade firmware flashed DAC/transceivers are also a good option. You can read tips about 10 Gbps networking on his blog post here.

2. Optics/DAC cable – Fiberstore is one of the best for this. The packaging and support for even the smallest orders is excellent and they ship around the world. For connecting two NICs together, get one of the Direct Attach Cables(DACs). I ended up getting this for $21.70, inclusive of shipping to India.
   

   DAC cable from FS.com

   For distances >5m pickup the active version, otherwise passive should be fine. You can also get transceivers and fiber from Fiberstore.

3. Switch – This is not necessary if you want to connect NICs directly. However if you need one with SFP+ slots, I highly recommend the Mikrotik 326-24G-2S+RM. Make sure you get transceivers flashed with the brand your switch is. The other end should be a generic version. You can also buy a bunch of NICs and do bridging in software if you are fine with a less performant and messy setup.

So this is what I ended up getting:

Testing it all out

Connecting it all up was pretty much straight forward. I had to put in the card in a PCIe x8 slot and attach the DAC cable. PCIe 2.0 x8 can do speeds of up to 24 Gbps so bottlenecking of PCIe lanes shouldn’t be an issue. Both GNU/Linux and FreeNAS(uses FreeBSD under the hood) detected the NICs out of the box. I added both of them up in a 192.168.5.0/30 subnet and confirmed I could ping each host from the other.

The Chelsio card sitting pretty in my server

I used the open source network benchmarking tool iperf to test out the network.

[root@freenas] ~# iperf -t 60 -c 192.168.5.2
------------------------------------------------------------
Client connecting to 192.168.5.2, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  3] local 192.168.5.1 port 25118 connected with 192.168.5.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  65.7 GBytes  9.40 Gbits/sec

From the other side:

root@proxmox:~# iperf -t 60 -c 192.168.5.1
------------------------------------------------------------
Client connecting to 192.168.5.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.5.2 port 33846 connected with 192.168.5.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  61.9 GBytes  8.86 Gbits/sec

As you can see it can push almost 9 Gbps over TCP. Throughput over UDP should be higher. There was very less increase in CPU power while pushing through a lot of data.

And a nice graph:

Network graph from my Proxmox server

You can run LACP/MPIO if you need more throughput.

I’m super happy with my setup. I hope this has been useful for you. Please do leave any experiences that you may have had with cheap 10 Gbps gear in the comments below. 🙂