10 Gbps network for less than $60

By | 12th March 2017

I needed a fast connection between my NAS and my VM server. So I set on to the mission of doing 10 Gbps network connectivity for as cheap as possible.

This is mainly to run iSCSI for VMs. I had looked at Infiband as well. However driver support was poor and things aren’t guaranteed to work. IPoIB is also very processor intensive.

Things you will need

  1. NIC cards – There are two options to do 10 Gbps for cheap here.
      1. Mellanox Connectx-2 – These are quite old cards with support only on GNU/Linux. However they are pretty cheap.
      2. Chelsio 110-1088-30 – These have dual SFP+ interfaces and have working drivers for FreeBSD and GNU/Linux but could be a little expensive. I picked up a pair of them for $37.36, inclusive of shipping.

    I have not investigated if either of them have Windows or Mac OS support. I got these through a friend in the US. If you live outside of US, you can ask Amazon to ship it to your country or use a parcel forwarding company like PPOBOX.

    Update 13/03/2017: The cost of the Chelsio NICs has shot up on Amazon. They are much cheaper on Ebay. As pointed out by Robert in the comments, Brocade BR-1020 with Brocade firmware flashed DAC/transceivers are also a good option. You can read tips about 10 Gbps networking on his blog post here.

  2. Optics/DAC cableFiberstore is one of the best for this. The packaging and support for even the smallest orders is excellent and they ship around the world. For connecting two NICs together, get one of the Direct Attach Cables(DACs). I ended up getting this for $21.70, inclusive of shipping to India.

    DAC cable from FS.com

    For distances >5m pickup the active version, otherwise passive should be fine. You can also get transceivers and fiber from Fiberstore.

  3. Switch – This is not necessary if you want to connect NICs directly. However if you need one with SFP+ slots, I highly recommend the Mikrotik 326-24G-2S+RM. Make sure you get transceivers flashed with the brand your switch is. The other end should be a generic version. You can also buy a bunch of NICs and do bridging in software if you are fine with a less performant and messy setup.

So this is what I ended up getting:

Components cost
2x Chelsio 110-1088-30 $37.36
1x 3m passive DAC cable(inc. shipping to India) $21.70
Total $59.06

Testing it all out

Connecting it all up was pretty much straight forward. I had to put in the card in a PCIe x8 slot and attach the DAC cable. PCIe 2.0 x8 can do speeds of up to 24 Gbps so bottlenecking of PCIe lanes shouldn’t be an issue. Both GNU/Linux and FreeNAS(uses FreeBSD under the hood) detected the NICs out of the box. I added both of them up in a 192.168.5.0/30 subnet and confirmed I could ping each host from the other.

The Chelsio card sitting pretty in my server

I used the open source network benchmarking tool iperf to test out the network.

[root@freenas] ~# iperf -t 60 -c 192.168.5.2
------------------------------------------------------------
Client connecting to 192.168.5.2, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  3] local 192.168.5.1 port 25118 connected with 192.168.5.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  65.7 GBytes  9.40 Gbits/sec

From the other side:

root@proxmox:~# iperf -t 60 -c 192.168.5.1
------------------------------------------------------------
Client connecting to 192.168.5.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.5.2 port 33846 connected with 192.168.5.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  61.9 GBytes  8.86 Gbits/sec

As you can see it can push almost 9 Gbps over TCP. Throughput over UDP should be higher. There was very less increase in CPU power while pushing through a lot of data.

And a nice graph:

Network graph from my Proxmox server

You can run LACP/MPIO if you need more throughput.

I’m super happy with my setup. I hope this has been useful for you. Please do leave any experiences that you may have had with cheap 10 Gbps gear in the comments below. 🙂

14 thoughts on “10 Gbps network for less than $60

  1. Srijit Banerjee

    Hi,
    Throughput is great! What is the H/W specs for your servers? Also, is there any custom related issues while ordering from FiberStore / Amazon?

    Reply
    1. varunpriolkar Post author

      >What is the H/W specs for your servers?
      Both of them are dual CPU Xeons with 24-32 GBs of ECC RAM. One system has 8 cores, while the other has 12 cores, 24 threads.

      >Also, is there any custom related issues while ordering from FiberStore / Amazon?
      None for Fiberstore. Didn’t have to pay a dime. For Amazon, I ordered it to a friend’s US address and he carried it to India for me so didn’t have to pay anything. You can use a parcel forwarding service like the one mentioned in the post or ask Amazon to deliver it to India for you. Both of options would ask you to pay the customs charges, but it shouldn’t be much.

      Reply
  2. Robert Novak

    Good (and timely) article… was having a conversation about 10gig connectivity on Twitter about the time Fiberstore retweeted your post.

    There are some Brocade CNAs that are viable as well, although you have to get Brocade-specific cables (or appropriately flashed ones maybe). I put BR-1020 cards in my Dell C6100 at an eBay best offer cost of $50 each, and the cables from Hong Kong were about $25-30 each.

    Reply
    1. varunpriolkar Post author

      Thanks for letting me know. Have updated the post accordingly.

      Reply
  3. FS.COM

    Thank you for your feedback, we are more than happy to know that our customers feel satisfied with our services and products.

    FS.COM will continue to make improvement and achieve a qualitative leap with cost-effective products and solutions.

    Reply
  4. Issac

    I had no issues with Mellanox Connectx-2 on Win2016, worked out of the ebay box a few weeks ago.

    Reply
  5. Pingback: Как Получить Сеть 10G по Цене Менее 60$? — Fiberstore

  6. Pingback: 10 Gbps network for less than $60 | Varun Priolkar – Moynahan

  7. Pingback: Как Получить Сеть 10G по Цене Менее 60$? — fiberopticnetwork

  8. Pingback: Links de Segunda #02 | Homelaber Brasil

  9. Frank

    I also have very good experiences with the supplier from Germany. If someone needed fast DAC cables in Germany, I was very satisfied. The costs were also about 25usd including freight https://www.netram.de/dac

    Reply

Leave a Reply