I recently read Srijit Banerjee’s excellent article, which brought to light ugly things done by service providers to fool their customers who have grown accustomed to the ping google.com command to judge the quality of the internet.

While he has strayed away from how this is done, I feel that educating people on how this can be done and how to ensure that they are not a victim is much more important.

My setup

My setup for testing looked something like this. Everything was virtualised with Virtualbox. For the router I picked up the CHR(Cloud Hosted Router) images from Mikrotik’s website.

My test setup on Virtualbox

Connections made from 192.168.100.3 through to the internet through the default gateway 192.168.100.2 is also source NATed. The other connection between the Mikrotik CHR VM and the host is also source NATed by Virtualbox.

Let’s do a ping from VM 1 to 8.8.8.8:

Very normal for a tethered 4G connection. All of this faking of pings relies on one simple concept:

Destination NAT (Port forwarding)

Illustration showing how destination NAT works

This essentially allows you to map a single IP or a set of IPs to a single IP or a set of IPs by matching transport layer parameters like ports, protocol used for transimission etc. Traffic can also be directed to a single port. The destination address and port is rewritten to by the router. The same is done for the source packet on return. Think of it as opposite of source NAT. So essentially just matching every IP on the Google’s address list and destination NATing it to a single IP will allow for this. Care has to be taken to do this for only for things you want up ending up at the new server. 

Taking it for a spin

For now let’s do it only for ICMP so that only ICMP traffic ends up at 192.168.200.2.

Destination NATing on my Mikrotik VM

Ping from VM 1 to 8.8.8.8:

Traceroute from VM 1 to 8.8.8.8 with ICMP:

RELATED  Analysing Steam's network in India

Tcpdump running on VM 2 with two pings sent/received from VM 1:

Have a look at the SRC/DEST IP. There is something interesting going on here. The destination IP is being changed by the router for the ICMP packets to 8.8.8.8 to 192.168.200.2 and SRC IP back to 8.8.8.8 from 192.168.200.2 on return. Tcpdump makes this very clear.

UDP traceroutes

Traceroute from VM 1 to 8.8.8.8 with UDP:

As expected, the UDP traceroute will land at the correct place.

RELATED  Google outage due to Cyclone Vardah in India

Manipulating latency

This can be done with the tc GNU/Linux command. Let’s add 15 ms to make it look a little bit more believable.

On VM 2:

On VM 1:

Well there you go! A lot more believable.

Similarly this whole process can be done for a bunch of IPs by importing address lists of a particular ASN/network you want to manipulate the traceroutes/pings of.

Preventing against it

By using lots of tcpdump I found out that traceroutes can be done in four different ways:

  1. ICMP This is how Windows does it. traceroute command on GNU/Linux can do this if you pass the –icmp flag. Can be easily be manipulated
  2. UDP (Unix style) Ports between 33435-33655 are usually used. This is the normal way of doing traceroutes on GNU/Linux and Mac OS. This can be easily marked and traffic manipulated again.
  3. UDP – This works on port 53. This can be used with –udp flag with the traceroute command on GNU/Linux. I am not sure how this can be manipulated.
  4. TCP This works on port 80. This can be used with –tcp flag with the traceroute command on GNU/Linux. I am not sure how this can be manipulated.

If traceroute traffic can be marked, its routing policy can definitely be edited depending on that marking. I wanted to investigate this more but I have lots of work pending to do. The truly best way of preventing this is picking up the phone and screaming at your ISP for doing these stupid illicit things rather than focusing on sell a good service to the customer.

Offending ISPs

Since Srijit’s blog had talked about North Eastern ISPs doing this shady practise, I did a RIPE Atlas tests on the only ISP who’s name I knew. The result was surprising!

Traceroute to 8.8.8.8 from an offending ISP

Surprisingly traceroutes for UDP, ICMP and TCP traffic were similar so they have indeed found a way to manipulate UDP and TCP traceroutes as well.

But the DNS test does give us a clue on the pings and traceroutes being completely wrong.

Fetching A record from 8.8.8.8 from offending ISP’s network

The latency should never be that high to fetch the A record of google.com.

Links to all RIPE Atlas measurements:

https://atlas.ripe.net/measurements/8273219

https://atlas.ripe.net/measurements/8273170

https://atlas.ripe.net/measurements/8273063

https://atlas.ripe.net/measurements/8310700

Finishing thoughts

I think one day when I get enough time I’ll schedule a mass traceroute from the RIPE Atlas project, process the data and tabulate a nice chart of offending ISPs to publicly shame them. I still cannot understand why ISPs in India like to all kinds of stupid things like this, ridiculous amounts of bandwidth manipulation with DPI, retarded OTT services rather than just selling a good service. SMH. People obviously will find out and when they do it isn’t going to be good for them.

I think I am done here. Good night!

If there are any errors here, please do let me know. I have written this in a hurry and don’t have enough time to proofread.

As always this being my personal blog, reflects my personal opinions only and not those of my employers.