b>j)΄!Pԫ&;"kB޶}pSVT(wę!j x;-m@JnQ+պכ7MajfJͱ4jѲ撆RxZMz7vIW/dٞТזcZM~ji ߒsQzԠDW3Den"M+/B:-uIJ7j委9p='mANޭ=/B:-n&nUfqxZM~c Ϲ+,&ᾺܢF[(1*" ϒ"Jԧ<;b" "jܢF[x ,!q қ*]/؝27SMcs"ޭDQ/应ܢF_! :s" 7`F+SVTn"IJnQ/应B 4 wD"IJ׭-`S9DrjiEJ߅gJ应矁[xZM~n"IB؃!'Тѕ+(mIKʭ/|ϐܢF[xZMzG %嬩/c[[ networking – Varun Sinai Priolkar https://varunpriolkar.com My Space on the Internet Wed, 25 Mar 2026 15:54:57 +0000 en-GB hourly 1 https://wordpress.org/?v=7.0 https://varunpriolkar.com/wp-content/uploads/2016/10/cropped-cartoon-blowfish-512-263078-1-150x150.png networking – Varun Sinai Priolkar https://varunpriolkar.com 32 32 117415536 Building a Mostly IPv6 Only Home Network https://varunpriolkar.com/2026/03/building-a-mostly-ipv6-only-home-network/ https://varunpriolkar.com/2026/03/building-a-mostly-ipv6-only-home-network/#respond Mon, 23 Mar 2026 21:20:18 +0000 https://varunpriolkar.com/?p=7649 Read More »]]> I wanted to switch my home network to IPv6 only design. However the reality is still that there are lot of devices today that only support IPv4 and large parts of Internet are still IPv4 only.

Because of this, many networks still deploy dual stack everywhere. However, I believe there is a case to be made for deploying IPv6 only in your network. Technologies like NAT64, DNS64 and 464XLAT make it possible for IPv6 only networks to access IPv4 services through translation. In this post I explore these technologies and the practical aspects of designing such a IPv6 only home network.

Getting a /48 over dedicated tunnel

My upstream Telus gives me multiple /64s using DHCP-PD. However the issue is that these are dynamic prefixes with a 10 minute lease timer. So if I turn off my router for more than that, there is good chance that the prefix could change. I wanted static prefix for my LAN and I also wanted more prefixes that I could then delegate to other things like Docker and VPN.

So I decided to use Free Range Cloud service provider to lease a /48, which cost me around C$10/year. I also used their tunnel service to then route this over to me. They have a POP in Vancouver, Canada and since I am in Kelowna, this is around 15 ms of latency away. However since most traffic flows through Vancouver anyways, this didn’t add much in terms of latency in reality.

Setting up Wireguard for tunnel was simple

I added the Wireguard details on OPNSense firewall and it started working. I decided to use IPv6 on tunnel and got a IPv6 address from WAN to connect from. This way I am not reliant on IPv4 for connectivity and don’t need to worry about MTU and fragmentation and can just let IPv6 PMTU take care of it.

PBR on OPNSense for LAN traffic

I then added a simple Policy Based Routing(PBR) rule on LAN interface to send traffic through the tunnel. Since there is no rdns setup, I used All Knowing DNS Server to setup rdns for the IP block. I have talked about how to set this up in this post.

IPv6 Addressing – SLAAC and DHCPv6

I decided to use a /64 out of the /48 for my main /48 subnet for my LAN. I then ran SLAAC(Router Advertisements) in Assisted mode with M+O+A flags set. This way clients can get IPv6 address in a stateless way, but I also ran Kea DHCPv6 server which would hand out addresses in stateful way. Unbound then picked up the static leases and then ensured that DNS resolution worked.

I also used /56 on every Docker host, with /64 for each Docker network that was then advertised to my firewall, but I’ll talk about this setup in later section.

  • Main subnet ➡ 2602:fed2:7e02::/48
  • LAN subnet ➡ 2602:fed2:7e02:69::/64
  • Example IP for Docker host on LAN ➡ 2602:fed2:7e02:69::22
  • /56 for Docker host(/64 for each network) ➡ 2602:fed2:7e02:2200::/56

I also used static addressing for each physical host so that I would not have issues during an outage.

Labeling each device with static address and adding it in authoratative DNS server

At this stage, I am still running a DHCPv4 server to give private IPv4 address to devices. I moved all my servers to IPv6 only, but my printer, vacuum robot and laptop still gets a IPv4 address using DHCPv4 server. My phone still got an IPv4 address at this stage, but this will change in the later section.

IPv6 test results

After setting all of this up, going to a IPv6 test website should give an output like this.

Accessing IPv4 Services over IPv6 – DNS64 and NAT64

There is a way to statefully access IPv6 services over IPv4. The process involves allocating a /96 from the IPv6 pool to hold the entire IPv4 address space. Both of these hold 32 bits of address space so can be used for translation. I used the default 64:ff9b::/96 address pool to keep things simple.

Diagram showing how NAT64+DNS64 is setup

I used a software called Jool on a VM to do the translation. There are other options too like Tayga64, but it works in userspace, unlike Jool that is a kernel module. I passed through an interface bridged to my WAN to this VM so that I could get a public IPv4 address to avoid double NAT. There was also a default route with higher metric to OPNSense as a fallback. You need DNS64 server that can translate DNS records with only A record to AAAA records. Unbound on OPNSense can do this or you can also use NAT64 version of public resolvers. I also changed lowest-ipv6-mtu to 1470 – the value of CLAT interface on Android.

The other thing I had to do was use FRR routing to advertise this prefix to OPNSense. You could get away with static route, but this way I can scale out Jool instances if needed in future.

!
ipv6 route 64:ff9b::/96 Null0
!
router bgp 64420
 bgp router-id 192.168.69.24
 bgp log-neighbor-changes
 no bgp ebgp-requires-policy
 neighbor 2602:fed2:7e02:69::1 remote-as 64420
 neighbor 2602:fed2:7e02:69::1 description OPNSense
 !
 address-family ipv4 unicast
  neighbor 2602:fed2:7e02:69::1 next-hop-self
 exit-address-family
 !
 address-family ipv6 unicast
  network 64:ff9b::/96
  redistribute static
  neighbor 2602:fed2:7e02:69::1 activate
 exit-address-family
exit
!

Once you configure the peer on OPNSense, the route should be visible in the routing table.

FRR injecting 64:ff9b::/96 route into OPNSense routing table

I also had to had a firewall rule above the PBR rule we added above to make it use the system routing table. After this, you can visit any IPv4 website and it should be accessible over IPv6. I use the plugin IPvFoo to know what is going on underneath while visiting the website.

IPvFoo confirming DNS64+NAT64 works

Disabling IPv4 – 464XLAT, PREF64 and DHCPv4 Option 108

Once you have NAT64 and DNS64 working, you could just turn off the DHCPv4 server and IPv4 address on the firewall. However, lot of devices like printer, vacuum robot and IPv4 literal addresses would simply not work. This is where DHCPv4 Option 108 comes into the picture. This simply tells the client that IPv6 only mode is preferred. Clients that do not honor this will get a IPv4 address as normal. Those who do have an option can setup 464XLAT to handle IPv4 literals and disable IPv4 addresses.

Borrowed from Microsoft website – How 464XLAT works

Remember the /96 prefix that we had configured earlier for DNS64? This can be discovered using PREF64 option on SLAAC server. The other way to discover this is by querying ipv4only.arpa domain name. 464XLAT used for handling IPv4 literals has two sides – client side stateless CLAT and stateful NAT64 on server that we setup with Jool above. To summarise, we use the following.

  • 464XLAT – Provides private IP interface on end machine for IPv4 literals. Contains CLAT – Stateless NAT64 on host side and PLAT – Stateful NAT64 on Jool.
  • DHCPv4 Option 108 – Tells client that IPv6 only network is preferred. Client usually enables CLAT if available and disables IPv4 address from DHCPv4. On Linux, you could not enable CLAT and just disable IPv4 if you want.
  • PREF64 – Get the /96 prefix used for DNS64 from Router Advertisement.
  • Query ipv4only.arpa – Used as fallback by some clients to find NAT64 /96 prefix if PREF64 is not available.
PREF64 option in Router Advertisements and Option 108 for DHCPv4 in OPNSense

Support for OS is as follows.

  • Android – CLAT enabled with DHCPv4 Option 108. Uses PREF64 as main DNS64 prefix discovery method. Queries ipv4only.arpa as fallback method. DNS64 can be disabled if PREF64 is enabled.
  • Linux – CLAT support recently merged in Network Manager – link. Requires DHCPv4 Option 108. Requires PREF64 as DNS64 prefix discovery method. Tested on Ubuntu 26.04: CLAT support hasn’t made it yet, but IPv6 only option works with ipv4.dhcp-ipv6-only-preferred.
  • Windows – CLAT not yet enabled, but in preview. Requires DHCPv4 Option 108. Requires PREF64 as DNS64 prefix discovery method. On latest Windows 11, still get IPv4 address.
  • Apple – CLAT activated with DHCPv4 Option 108 and PREF64. iOS v16 or above supports ipv4only.arpa as fallback for activating CLAT. DNS64 is required for iOS and Safari. I don’t have Apple devices so not tested.

IPv6 on Docker

IPv6 support on Docker is greatly improved. I am using Docker Compose type setup on most of my servers and wanted to avoid having any IPv4 IP addresses even for internal networking unless really necessary. I also wanted to route my Docker GUA prefixes without NAT66 and not use ULA addresses. I added this in my /etc/docker/daemon.json file.

{
  "ipv6": true,
  "default-address-pools": [
    { "base": "2602:fed2:7e02:2200::/56", "size": 64 }
  ],
  "default-network-opts": {"bridge":{"com.docker.network.enable_ipv6":"true"}},
  "bridge": "none",
  "experimental" : true,
  "ip6tables" : false
}

The DHCPv6 address this server got from the DHCP server was 2602:fed2:7e02:69::22 so I decided to use 2602:fed2:7e02:2200::/56 for Docker networks, with /64 being allocated for each network. After that, you can use prefixes in your Docker compose like this.

services:
  upstream:
...
    networks:
      blocky:
 
networks:
  blocky:
    driver: bridge
    enable_ipv4: false
    enable_ipv6: true

You can also pick the /64 to use for that particular network and static addresses for each container if you like. Most services worked fine, but I had issues with a couple and I opened GitHub tickets for them. Most notable of these were Flood and Nextcloud.

I then announced the prefixes to OPNSense using FRR.

!
ipv6 prefix-list home_subnet seq 5 deny 2602:fed2:7e02:69::/64
ipv6 prefix-list home_subnet seq 10 permit 2602:fed2:7e02::/48 le 64
!
route-map HOME_SUBNET permit 10
 match ipv6 address prefix-list home_subnet
exit
!
password zebra
!
router bgp 64420
 bgp router-id 192.168.69.23
 bgp log-neighbor-changes
 no bgp ebgp-requires-policy
 neighbor 2602:fed2:7e02:69::1 remote-as 64420
 neighbor 2602:fed2:7e02:69::1 description OPNSense
 !
 address-family ipv4 unicast
  neighbor 2602:fed2:7e02:69::1 next-hop-self
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute connected route-map HOME_SUBNET
  neighbor 2602:fed2:7e02:69::1 activate
 exit-address-family
exit
!
end

The routes should then show up in OPNSense and work. You may have to create a firewall rule to skip the PBR we set earlier for these routes.

That’s it. I don’t have IPv4 addresses on servers, unless they really need it. Exceptions would be a server that makes connections to Speedtest server to get download speed and Torrent server.

Accessing IPv6 Services over IPv4 Internet

I mostly only have IPv6 deployed on servers and no IPv4. I needed a way to get IPv6 services working over IPv4 internet. Easiest way I found was to just use a VM with a IPv4 address and proxy traffic back over IPv6 VPN. I used Zerotier, which provided ULA IPv6 address over SDN VPN. I disabled IPv4. I simply set routes to appropriate devices over this.

Routing global addresses over Zerotier VPN and terminating SSL over VM is a simple solution

I can now use Caddy to terminate SSL on VM edge or use TCP proxy to send traffic from outside world to internal IPv6 network. There are other solutions like using Cloudflare tunnel, but I found this solution simple and it works well.

List of Issues I found

These are the list of issues I’ve found with running IPv6 so far.

  • Slack timeouts with DNS64+NAT64 – Ticket opened and forwarded to internal team for resolution.
  • No IPv6 on Eufy vacuum robot – Ticket opened and forwarded to internal team for resolution.
  • No IPv6 only mode and broken intra subnet IPv6 routing on HP printer – Ticket opened and forwarded to internal team for resolution.
  • Broken IPv6 on Docker containers – Flood(fixed), Nextcloud AIO.
  • Disable IPv4 on default Docker bridge – Issue.

I’ll post issues if they come up in my testing.

]]> https://varunpriolkar.com/2026/03/building-a-mostly-ipv6-only-home-network/feed/ 0 7649 Running 40Gbps Network on Mini PCs For Under $50 https://varunpriolkar.com/2023/12/running-40gbps-network-on-mini-pcs-for-under-50/ https://varunpriolkar.com/2023/12/running-40gbps-network-on-mini-pcs-for-under-50/#respond Sun, 31 Dec 2023 03:59:07 +0000 https://varunpriolkar.com/?p=6606 Read More »]]> I recently setup my Home Lab. However, I realised that the network connecting two of the mini PCs hosting my distributed GlusterFS volume was too slow. I was getting bottlenecked by the network between them. The 2 NVMe SSDs I put in would be no good if the network between them was going to be really slow. The read speed was acceptable because GlusterFS can read from the local disk if the data exists there, however the write speed was pretty poor.

varun@docker-4:/mnt/vol1/bench$ sudo dd if=/dev/zero of=./test1 bs=512k count=2048 oflag=direc
t
2048+0 records in
2048+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 9.32665 s, 115 MB/s

Since you can add network cards to the Lenovo Tiny M920Q Mini PCs hosting my Docker Swarm cluster with a PCIe riser card, that’s what I did. I had 2x Mellanox ConnectX-3 MCX354A-FCBT lying around in my apartment that I decided to try to put to use. These can do upto 56Gbps per port, however you should not expect to get double that in LACP because of PCIe bandwidth limitations. The Mellanox ConnectX-4 or ConnectX-5 are better cards so you should get them if you are in a position to do so. The first step ofcourse was to install the cards in the mini PCs. From my research, these need atleast 10-12W extra power so the 90W power supply is preferable, but it worked for me with the 65W too.

Mellanox Connect X-3 fits perfectly in the Lenovo M920Q

I also got an Active Optical Cable(AOC). A DAC should work too, the only thing to keep in mind is that the ConnectX-3 does not like AOCs that need more than 1.5W of power. 10Gtek QSFP-H40G-AOC1M from Amazon worked well for me.

ItemCost
Mellanox ConnectX-3 MCX354A-FCBT$15/each(EBay or FB Marketplace)
10Gtek QSFP-H40G-AOC1M QSFP AOC/DAC cable$20/each(or less from used market for DAC)
Total$50 for 40Gbps, $72 for 80 Gbps in LACP
Bill of materials for this project

You can get upto 56Gbps per port if you get the right DAC cable, but they were too expensive and hard to find for me. You could also put them in LACP to get 80Gbps(or more) with failover, but I decided to not do that for now.

Preparing the card

My cards came with the Oracle firmware installed, so I had to install the Mellanox firmware to get it running. The first thing I had to do was to install the Mellanox tools for flashing the firmware. Make sure to download the firmware tools and the VPI firmware for Mellanox ConnectX-3. Not all versions of cards can do 40Gbps. Some people have had success with cross flashing to FCBT version, which can do 40/56Gbps. The version 4.22.1-307-LTS of firmware tools worked for me, the other version did not. I managed to install the firmware tools with this.

apt install pve-headers-$(uname -r) proxmox-default-headers gcc make dkms gcc make dkms #Replace first 2 packages with appropriate kernel header package
tar -xvzf mft-4.22.1-307-x86_64-deb.tgz
cd mft-4.22.1-307-x86_64-deb.tgz
bash install.sh

After this, make sure you can query the card. You will need to replace 01:00.0 with the output from the lspci command.

root@proxmox-3:~# mst start
Starting MST (Mellanox Software Tools) driver set
...
root@proxmox-3:~/test/mft-4.22.1-307-x86_64-deb# lspci | grep -i "mellanox"
01:00.0 InfiniBand: Mellanox Technologies MT27500 Family [ConnectX-3]
root@proxmox-3:~/test/mft-4.22.1-307-x86_64-deb# flint -d 01:00.0 q
Image type:            FS2
FW Version:            2.35.5532
FW Release Date:       12.5.2017
...

Backup and Flash Mellanox firmware

Next, we’ll backup the firmware that the card comes with.

flint -d 01:00.0 query full > flint_query.txt
flint -d 01:00.0 hw query > flint_hwinfo.txt
flint -d 01:00.0 ri orig_firmware.mlx
flint -d 01:00.0 dc orig_firmware.ini
flint -d 01:00.0 -vpd > orig_vpd.txt

Next, we will flash the VPI firmware we downloaded earlier.

root@proxmox-3:~/test# flint -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin query full
Image type:            FS2
FW Version:            2.42.5000
...
root@proxmox-3:~/test# flint -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin verify
     FS2 failsafe image. Start address: 0x0. Chunk size 0x80000:
     NOTE: The addresses below are contiguous logical addresses. Physical addresses on
           flash may be different, based on the image start address and chunk size
     /0x00000038-0x0000065b (0x000624)/ (BOOT2) - OK
     /0x0000065c-0x0000297f (0x002324)/ (BOOT2) - OK
...
-I- FW image verification succeeded. Image is bootable.

root@proxmox-3:~/test# flint -d 01:00.0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBo
ot-3.4.752.bin -allow_psid_change burn
    Current FW version on flash:  2.35.5532
    New FW version:               2.42.5000
    You are about to replace current PSID on flash - "ORC1090120019" with a different PSID - "MT_1090120019".
    Note: It is highly recommended not to change the PSID.
 Do you want to continue ? (y/n) [n] : y
Burning FS2 FW image without signatures - OK  
Restoring signature                     - OK

Reboot after this. When you bootup the card should show up as a non-Infiniband card.

Switch to Ethernet mode

We will now need to switch this card to Ethernet mode.

root@proxmox-3:~# mlxconfig -d 01:00.0 q
Device type:    ConnectX3       
Device:         01:00.0         
Configurations:                                      Next Boot
         LINK_TYPE_P1                                VPI(3)          
         LINK_TYPE_P2                                VPI(3)          
...

LINK_TYPE_P1 and LINK_TYPE_P2 are the 2 interfaces of the card. We have 3 options to choose from Infiniband(1), Ethernet(2) and VPI(3) for the two ports.

root@proxmox-3:~# mlxconfig -d 01:00.0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
Configurations:                                      Next Boot       New
         LINK_TYPE_P1                                VPI(3)          ETH(2)          
         LINK_TYPE_P2                                VPI(3)          ETH(2)          
 Apply new Configuration? (y/n) [n] : y
Applying... Done!

Next, reboot and make sure the card is getting detected. There is also an option to get the card in SR-IOV mode, but I couldn’t get it working.

Benchmarking

After getting the ethernet mode working, I simply connected the 2 mini PCs with Active Optical Cable. Next, I added IPs to either ends and made sure it gave me a ping.

10Gtek QSFP-H40G-AOC1M AOC between Mini PCs

iperf with MTU of 6000 gave me the best results.

root@proxmox-3:~# iperf3 -c 192.168.4.103
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  39.4 GBytes  33.8 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  39.4 GBytes  33.8 Gbits/sec                  receiver

Next, I setup a Linux bridge interface with this interface and added it to the VM. You can also passthrough the interface if you don’t need it for other VMs. I resetup my GlusterFS volume to use this link between the nodes, but my Raspberry Pi would use the default interface to connect to these two nodes. I did it using different /etc/hosts files injected using Ansible. The only potential issue would be a link failure so I’m considering doing LACP between the nodes. This is the performance I got.

root@docker-3:/mnt/vol1/bench# dd if=/dev/zero of=./test3 bs=512k count=2048 oflag=direct
2048+0 records in
2048+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.96741 s, 362 MB/s

That is much better and leaves my main link spare for use with uplink. The only caveat is that I need to exclude the Raspberry Pi Docker Swarm node from IO heavy tasks, which is not a concern.

]]> https://varunpriolkar.com/2023/12/running-40gbps-network-on-mini-pcs-for-under-50/feed/ 0 6606 Updates to blog – BGP, IPv6, TLS 1.3 and more https://varunpriolkar.com/2018/12/updates-to-blog-ipv6-tls1-3-and-more/ https://varunpriolkar.com/2018/12/updates-to-blog-ipv6-tls1-3-and-more/#comments Sat, 08 Dec 2018 02:19:37 +0000 https://varunpriolkar.com/?p=2493 Read More »]]>

Just wanted to share a few updates about blog and life.

TLS 1.3 with 0-RTT

TLS 1.3 is a new standard of encryption on the web and I have implemented it! This should result in faster load times and improved security for the users. ECDSA certs are enabled too, along with RSA for better performance. All that has helped the website score A+ on Qualys SSL Labs results. Woosh!

A+ on Qualys SSL Labs test. Yay!

All this is possible because of the fantastic Ondřej Surý’s fantastic packages. 0-RTT and HTTP/2 server push is enabled as well. Thanks to my friend Sunit Nandi for making me do this.

Infrastructure as code

This is by far the most useful and important change. Now the entire hosting infrastructure consisting of LXD containers, a Nginx reverse proxy, BGP is entirely handled by Ansible scripts and Gitlab. I can make a change to the code and Jenkins will auto upload and run the code. Periodic backups are automated as well.

Recently a good samaritan informed me of expired TLS certificate on blog

I have had problems with Let’s Encrypt SSL certificates expiring in the past, inspite of automating them mostly due to Python dependency issues. They are now centrally managed with acme.sh and pushed out periodically. All of this allows me to focus on blogging rather than spend time over managing the infrastructure. I also setup monitoring for the same with Uptime Robot and updown.io.

IPv6 and BGP

I’m using 2a0a:b707:1111::/48 block with my AS132383 for blog

As you may know, I recently got hold of /44 and /48 IPv6 blocks. The blog is up on the IP 2a0a:b707:1111:0:bad:babe:a9ed:22. Announced IPv6 block is 2a0a:b707:1111::/48 block with my AS132383. I’m hosting everything at First Root UG in Germany.

2401:f9c0:1111::/48 is being used for home use

I also enabled IPv6 at home by routing IPv6 traffic over an OpenVPN tunnel and doing BGP/announce at Singapore on a $3.5/month Vultr VPS. The only remaining thing is setting up reverse DNS for both.

WordPress 5.0

WordPress 5.0 was released today and I updated to it! The most notable feature was the new Gutenberg editor, which I was using earlier as well. I must say the usability jump has been immense, although I can see why people are holding off switching over. There’s classic editor(same as the older one) which is supported until 2023.

I had contemplated moving to alternate platforms like Ghost or static site generators like Hugo, Jenkyll but it looks to be too big of a move. I don’t like the fact that WordPress and the community is focusing more on dynamic site builders and custom frameworks, rather than focusing on building a better product for blogging.

I’m going to APRICOT 2019!

APRICOT is the technical event for network admins organised by APNIC

I managed to get hold of the early bird tickets for APRICOT 2019, which is the technical event held annually for network administrators by APNIC. It will be held in Daejeon, South Korea in February 2019. I will be there for two weeks. Hopefully I get my Visa cleared quickly. That’s the only thing that the whole plan is contingent on.

Thanks for reading and hope to meet you there 😀

]]> https://varunpriolkar.com/2018/12/updates-to-blog-ipv6-tls1-3-and-more/feed/ 2 2493 BGP on a shoestring – The definitive guide https://varunpriolkar.com/2018/12/bgp-on-a-shoestring-the-definitive-guide/ https://varunpriolkar.com/2018/12/bgp-on-a-shoestring-the-definitive-guide/#comments Wed, 05 Dec 2018 12:37:42 +0000 https://varunpriolkar.com/?p=2336 Read More »]]>

Virtually every address on the internet is connected with each other with a protocol called as the Border Gateway Protocol(BGP), which dynamically routes traffic on the internet to the closest network. This guide will show you how you can be part of the internet for the cheapest possible cost. This guide is intended for sysadmins and network engineers, as such will not attempt to explain intricacies involved.

There are various reasons why you may want to do this, including better control over traffic routing, cleaner IP range for things like running mail servers, more publicly routable IPs to use etc. So let’s get started!

Creating maintainer objects

Maintainer objects will help you to manage objects(AS numbers and IP blocks) that you will be acquiring.

Function of maintainer objects. Source: RIPE

For this you will need to create an account on one of the Regional Internet Registry(RIR) that you want the resources with, which should be the primary region that you will be using the resources in or the region where you live. Also some RIRs like RIPE will allow you to easily create an account to manage resources, while others like APNIC will need you pay a hefty annual fee just to manage resources; which can get annoying as you will need to ask your Local Internet Registry(LIR) to make changes if needed, instead of doing them yourself. The list of RIRs are:

  • APNIC – Asia Pacific region.
  • AFRINIC – Africa region.
  • LACNIC – Latin America region.
  • RIPE – Europe region.
  • ARIN – North America region.
How my created maintainer objects look like

Note that some LIRs and IP leasing services won’t readily assign maintainers and would prefer to host the services themselves, which was fine for me as I didn’t have to update resources frequently.

Cost: Free.

Apply for an ASN

ASN(Autonomous system number) is an unique number given to IP networks on the internet, with the network having a clearly defined routing policy. You can do BGP with a private ASN and ask the upstream to filter out the ASN before announcing to upstream, but I decided to get my own so that I can be identifiable on the internet.

To get an ASN, you will need to be sponsored by a LIR. The LIR should be a member of one of the Regional Internet Registry(RIR) in which you intend to apply for. Do keep in mind that some RIRs may want you to be multi-homed(have more than one transits or be present in multiple regions). 

The best way to get a list of LIRs which can sponsor you on the linked excel sheet or the follow this LowEndTalk thread. I would highly recommend getting IP space from the same company which offered sponsored your ASN request.

Make sure the WHOIS of the resources(ASN, IP blocks) reflects your info

Make sure the WHOIS reflects the correct information, including your email ID. RADB has a nice WHOIS tool. This would be required later to setup BGP peering. Do note that it is better to get your resources from a reputed LIR, or better yet directly from the RIR if you will be using the IP space for anything professional.

Cost: Around 35 USD once. Could vary heavily depending on sponsoring LIR and region.

Lease IPv4/IPv6 space

You will need IP addresses that you can “announce” on the internet and hence claim to be yours. These can either be bought and transferred to you or you can lease them off an owner of the IP range. Since we are talking about doing BGP on a shoestring, leasing is the only model that works for us. Make sure the IP leasing service assigns the proper maintainer object or assigns proper ROAs, adds the correct WHOIS entry and rDNS.

You will need a minimum of /48 for IPv6 and /24 for IPv4. I ended up skipping IPv4 since it was way too cost prohibitive. Just one IPv4 address given by my host is enough for me.

Cost: Around 10-20 USD/year for a /48 IPv6 block, 60-70 USD/month for a /24 IPv4 block.

Select a transit/VPS provider

There is an awesome spreadsheet which lists all known VPS providers which will do BGP for you for free or for a nominal cost. Many of them are quite happy to do custom things that you may want like connection to an Internet Exchange(IXP) or BGP communities or announcements to selective upstreams. You can also buy IP transit directly for more industrial applications. I’ve heard Hurricane Electric even gives free IPv6 transit at certain locations. I couldn’t confirm if this is true.

Do keep in mind that you will need to have matching upstreams or control routing with BGP communities incase you are planning to do IP anycast. Also you will need enough memory to hold the full internet routing table if you are requesting one. Otherwise you could just ask for the default route.

I decided to go for Vultr and First Root

The awesome thing about Vultr is that they do BGP for even the lowest plan and the whole process is very automated. First Root is recommended as well since they have attractive pricing and great support.

Cost: 5-10 USD/month depending on the resources you want.

Setting up BGP

To announce BGP, you will first need to validate your resources and write a Letter of Authorisation(LOA) to authorise that you own the resources and that you give the host permission to announce them.

This will depend on each particular host, but most hosts ask you to send a LOA from the email on the WHOIS for the resources. Something like this would work.


11 November 2018

To whom it may concern,

This letter serves as authorization for COMPANY NAME with ASXXXXX to announce the following IP address blocks:

2401:f9c0:1111::/48 – AS132383

As the owner of the subnet and ASN, I hereby declare that I’m authorized to represent and sign for this LOA.

Should you have questions about this request, email me at me@varupriolkar.com, or call: +91XXXXXXXXXX.

From,

Varun Priolkar
me@varunpriolkar.com
+91XXXXXXXXXX

After this, the host should setup BGP for you and give you details like which IP to connect to. You would also need to pick if you want the full BGP table or just the default route. Typically the default route should be fine.

To setup BGP, I installed bird and bird-bgp packages on Ubuntu. A basic setup should look like this. Vultr has a nice guide for configuring BGP.

router id 193.25.100.229;

filter out_filter {
    if net = 2a0a:b707:1111::/48 then accept;
    else reject;
}

protocol bgp froot
{
    local as 132383;
    source address 2001:67c:12a0:8010:0:0:0:2;
    import filter all;
    export filter out_filter;
    graceful restart on;
    neighbor 2001:67c:12a0:8010:0:0:0:1 as 41108;
    next hop self;
}

protocol static
{
    route 2a0a:b707:1111::/48 via 2001:67c:12a0:8010:0:0:0:2;
}

protocol device
{
    scan time 5;
}

protocol kernel {
	scan time 20;		# Scan kernel routing table every 20 seconds
	import all;		# Default is import all
	export all;		# Default is export none
}

Then all you have to do is run systemctl start bird/bird6. You will also need to open TCP/179 port on your firewall. There’s a useful guide on NLNOG website for setting up more BGP filters.

Confirming it all works

You will have to use looking glass from the transit providers to check if your prefix is getting propagated correctly. I used Hurricane Electric’s Looking Glass. You can also use Oregan Route Views project to get a more complete picture.

HE’s looking glass output

Hurrican Electric also has a nice BGP view tool through which you can check the route propagation.

Route propagation as viewed from HE’s BGP tool

 Then all you have to do is simply assign an IP to an interface and see if it pings.

Reverse DNS

Reverse DNS zone will need to point to your name servers. You can either set this up yourself or host it on DNS providers like Amazon’s Route 53 or DNS Made Easy. Reverse DNS zone will be the IP in the inverted format followed by the appropriate suffix. For eg. for the IPv4 prefix 1.2.3.0/24 the reverse zone will be 3.2.1.in-addr.arpa. and for the IPv6 address 2401:f9c0:1111::/48 the zone will be 1.1.1.1.0.c.9.f.1.0.4.2.ip6.arpa. You will also need to configure the appropriate PTR record. For IPv4 address this will be 1.3.2.1.in-addr-arpa. for the IPv4 address 1.2.3.4 and 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.1.1.1.0.c.9.f.1.0.4.2.ip6.arpa. for the IPv6 address 2401:f9c0:1111::1. It is preferable that the reverse record matches the forward as well.

It can be difficult to set the reverse record for a huge range of IPv6 addresses. I found this project useful for that.

Cost: 1 USD per hosted zone on Route 53, free if you are self hosting.

Monitoring

 BGPMon is really useful for this and free for monitoring upto five prefixes. It will also send you email alerts incase it notices any flaps/prefix withdrawal. 

BGPMon is excellent for monitoring your announcements

It also has a project called PeerMon which can allow you to peer with BGPMon routers and share your routes with it for it to detect any routing changes. I wouldn’t try it out because they had some capacity issues at the time of writing.

https://twitter.com/bgpmon/status/878270461091631106

Cost: Free for upto 5 prefixes.

Total costs

To recap the total costs should be:

VM: 5-10 USD/month for VMs
IP blocks: 1-2 USD/month for IPv6 blocks, 60-70 USD for IPv6 blocks
ASN: 35 USD once.
rDNS hosting: 1-2 USD/month, free if self hosting.
Total: 35 USD once, 7-15 USD/month recurring for IPv6, 65-80 USD/month for IPv4.

Useful resources

Hope you found this post useful. Don’t forget to subscribe, comment if you did. I’ll have more advanced BGP and internet related posts coming up, which includes IP anycast, BGP communities, building CDNs 🙂

]]> https://varunpriolkar.com/2018/12/bgp-on-a-shoestring-the-definitive-guide/feed/ 7 2336 IPv6 should be working on this blog! https://varunpriolkar.com/2016/10/ipv6-should-be-working-on-this-blog/ https://varunpriolkar.com/2016/10/ipv6-should-be-working-on-this-blog/#respond Sun, 16 Oct 2016 00:51:18 +0000 https://varunpriolkar.com/?p=349 Read More »]]>

IPv6 is now enabled on this blog. Getting it to work wasn’t as straight forward as I initially thought since my hosting provider Contabo gives only a single /64 and I use many LXC containers to host different things. Thankfully my friend Anurag helped me out. You can read his blog post about it here.

The problem was that:

  1. I use internal IPv4 addresses for hosting different things because public IPv4 addresses are expensive so I couldn’t bridge my VM interface with eth0.
  2. Subnetting the the /64 into /112 would not work since IPv6 addresses were allocated with L2 and when I tried using a /112 with one IP on the VM interface and the other on an another container, it allowed for outgoing packets but incoming packets were blocked. This is because the IP lacked an entry in my hosting provider’s NDP table.

The solution? Proxy NDP. I will be going more in detail on how to get it working in this post.

Adding IPv6 addresses to interfaces

Let’s assume I was assigned 2001:db8:1:1::/64 by my hosting provider. 2001:db8:1:1::1/64 sits on eth0. We’ll use 2001:db8:1:1:c:c:c:0/112 subnet for our VMs. That is the default config and your /etc/network/interfaces file would look something like this.

iface eth0 inet6 static
    address 2001:0db8:0001:0001:0000:0000:0000:0001
    netmask 64
    gateway fe80::1 #my providers gateway local-link address. may be different for you
    accept_ra 0
    autoconf 0
    privext 0

Next let’s assign 2001:db8:1:1:c:c:c:1 to our VM interface vmnet. Add up ifconfig vmnet add 2001:db8:1:1:c:c:c:1/112 in your /etc/network/interfaces file. Restart networking. For me this was systemctl restart networking .

Enable proxy NDP for a bunch of IPs

Enable proxy NDP with echo “1” > ‘/proc/sys/net/ipv6/conf/eth0/proxy_ndp’ and enable it for a single IP with ip neigh add proxy 2001:db8:1:1:c:c:c:1 dev eth0.

This is a script to auto do that for you and add proxy NDP entries for 100 IPs. Let’s call it test.sh .

#!/bin/bash
echo "1" > '/proc/sys/net/ipv6/conf/eth0/proxy_ndp'
# Generate proxy arp in bulk

 for i in `seq 1 100`;
        do

        ip neigh add proxy 2001:db8:1:1:c:c:c:$i dev eth0

        done

Make it executable with chmod +x test.sh . Let’s make it run on every reboot. Run crontab -e and put in @reboot /path/to/test.sh. You can run it for now with sh test.sh.

Enable IPv6 on your container

We can use 2001:db8:1:1::c:c:c:2 to 2001:db8:1:1:c:c:c:100 for VMs in this case since we’ve enabled proxy NDP for them. I’m going with 2001:db8:1:1:c:c:c:2 in this case. Add this in your /etc/network/interfaces of your VM and restart networking.

up ifconfig eth0 add 2001:db8:1:1:c:c:c:2/112
up ip -6 route add default via 2001:db8:1:1:c:c:c:1

I use a reverse proxy so it was enough for me to enable IPv6 on it. I use a VPN to access some internal applications on some of my containers and OpenVPN needs a /64 for IPv6 to work correctly so I’m sticking with source NATed IPv4 addresses for now.

That’s it! You can use listen [::]:port; with Nginx to make it listen to any IPv6 addresses it has been assigned on a specific port.

Enabling IPv6 on Cloudfront

I use Amazon S3 plus Cloudfront for images. They recently announced support for IPv6 here. Enabling IPv6 is quite easy to do. Simply head on over to the Cloudfront distribution settings and enable IPv6.

screenshot-from-2016-10-16-07-33-13

However trying to ping cdn.varunpriolkar.com over IPv6 from the VPS itself simply wouldn’t work. I found more info in Amazon’s documentation about this.

We’re deploying IPv6 gradually during October and November of 2016. The deployment is going out to one viewer network at a time. (A viewer network is analogous to your home internet or wireless carrier.) Some viewer networks have excellent IPv6 support, but some viewer networks don’t support IPv6 at all.Amazon documentation

Hopefully Amazon should roll out support within a couple of months to more networks. You can check c-ip column in Cloudfront access logs to determine if visitors are indeed using IPv6. I don’t have logging enabled but maybe I’ll look at it in the future.

Testing it all out

I asked a friend who has IPv6 enabled on his network to test out the connectivity from outside and connectivity from inside worked as expected as well. You can test out whether IPv6 is working on your website here.

screenshot-from-2016-10-16-05-30-04

Yay! It all works

This website will not work in a pure IPv6 network because my authoritative DNS provider NS1 has not yet deployed IPv6. You can have 65k IPv6 addresses for VMs using this method. Enjoy!

So what are you waiting for? Deploy IPv6 on your website today! 🙂

]]> https://varunpriolkar.com/2016/10/ipv6-should-be-working-on-this-blog/feed/ 0 349