Excited by the prospect for Steam for Linux I fired up Steam to download some games. The problem was that finding a mirror with the best speed for seeming problematic. So I decided to do some analysis. 😀
I am testing all of this from AS55836 Reliance Jio Infocomm Limited. I primarily use this for steam since the throughput is nice to have and I don’t use more than 4GB of download on Steam games per day. To choose a mirror goto Steam>Settings>Download>Download region in the menu.
Singapore mirror
Since Reliance Jio peers with their upstream AS64049 Reliance Jio Infocomm Pte Ltd Singapore in Singapore, I decided to try the Singapore mirror first. The results? Not bad. I’ve used iftop command on Linux to see from which IPs the download was being streamed from.
iftop and game download from Singapore
The throughput was decent at 2.5 MB/s.The only problem is that it would be affected badly during peak times, due to heavy strain on the international capacity. As you can see most of the IPs are from 103.10.124.0/24 network, which is announced by AS32590 Valve Corporation in Singapore. Let’s do a traceroute, shall we?
arhue@xubuntu-desktop:~/Downloads$ traceroute 103.10.124.20 traceroute to 103.10.124.20 (103.10.124.20), 30 hops max, 60 byte packets 1 gateway (192.168.42.129) 2.327 ms 2.322 ms 2.390 ms 2 * * * 3 10.71.225.130 (10.71.225.130) 111.569 ms 111.609 ms 111.705 ms 4 172.16.93.209 (172.16.93.209) 96.941 ms 97.030 ms 97.130 ms 5 172.26.31.2 (172.26.31.2) 96.664 ms 96.638 ms 96.805 ms 6 * * * 7 * * * 8 * * * 9 103.198.140.164 (103.198.140.164) 93.141 ms 93.010 ms 92.984 ms 10 103.198.140.27 (103.198.140.27) 186.625 ms 152.113 ms 160.538 ms 11 ldt-au01.ibeo.hgc-intl.com (80.81.194.166) 196.549 ms 191.214 ms 196.720 ms 12 global.hgc.com.hk (218.189.8.141) 249.810 ms 254.026 ms 235.519 ms 13 d1-244-224-143-118-on-nets.com (118.143.224.244) 243.390 ms 253.236 ms 239.243 ms 14 global.hgc.com.hk (218.189.12.234) 238.645 ms 246.932 ms 234.971 ms 15 * * * 16 * * *
It times out after that probably because ICMP is disabled after that. The path taken is AS55836 Reliance Jio Infocomm Limited(India)>AS64049 Reliance Jio Infocomm Pte Ltd Singapore(Singapore)>AS9304 Hutchison Global Communications(Singapore)>???. Luckily AS9304 Hutchison Global Communications has a looking glass for us to peak into. You can find it here. Let’s do ping.
ping 103.10.124.20 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 103.10.124.20, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
And a show ip bgp command.
show ip bgp 103.10.124.20
BGP routing table entry for 103.10.124.0/24, version 321730
Paths: (1 available, best #1, table default)
Not advertised to any peer
9304 32590 32590, (aggregated by 32590 103.10.124.255)
118.143.235.129 from 118.143.235.129 (118.143.225.4)
Origin IGP, localpref 100, valid, external, best
So AS9304 Hutchison Global Communications peers with Valve. The next hop should be AS32590 Valve Corporation directly. The location is most probably Equinix Singapore if you have a look at Valve’s PeeringDB profile. The more important downstream path is hidden since Valve doesn’t have a looking glass. Let’s just assume that AS32590 Valve Corporation‘s router picks path through AS9304 Hutchison Global Communications as the best, which could be likely since they peer with them.
show ip bgp 49.35.248.119
BGP routing table entry for 49.35.128.0/17, version 73468304
Paths: (1 available, best #1, table default)
Not advertised to any peer
9304 15412 3491 64049 55836
118.143.235.129 from 118.143.235.129 (118.143.225.4)
Origin IGP, localpref 100, valid, external, best
Path is AS9304 Hutchison Global Communications>AS15412 Reliance Glocalcom Limited(Singapore)>AS3491 PCCW Global(Singapore)>AS64049 Reliance Jio Infocomm Pte Ltd Singapore(Singapore)>AS55836 Reliance Jio Infocomm Limited(India). The same local preference means AS9304 Hutchison Global Communications is peering with AS15412 Reliance Glocalcom Limited. The long AS path length probably means AS32590 Valve Corporation‘s router may not choose to send traffic this way unless they are not peering with any network which is advertising a shorter one. Also probably AS18101 Reliance Communications Ltd‘s upstream AS15412 Reliance Glocalcom Limited is not announcing AS55836 Reliance Jio Infocomm Limited‘s IP pools in Singapore but is only used for local connectivity.
India(Mumbai) mirror
I am going to consider Mumbai since it is closest to me.
iftop and game download from Mumbai(India)
The throughput was better at about 2.6 MB/s and this would be better at peak times. This test was done at 6 AM so the results would have been different at different times of the day. Let’s do a trace.
arhue@xubuntu-desktop:~/Downloads$ traceroute 115.248.101.129 traceroute to 115.248.101.129 (115.248.101.129), 30 hops max, 60 byte packets 1 gateway (192.168.42.129) 2.866 ms 2.968 ms 3.065 ms 2 * * * 3 10.71.225.162 (10.71.225.162) 89.943 ms 90.042 ms 90.015 ms 4 172.16.93.211 (172.16.93.211) 89.604 ms 89.455 ms 89.675 ms 5 172.26.31.6 (172.26.31.6) 89.275 ms 89.136 ms 89.348 ms 6 * * * 7 * * * 8 * * * 9 115.249.214.165 (115.249.214.165) 1555.076 ms 1555.055 ms 1555.133 ms 10 * * * 11 * * *
It is directly from AS55836 Reliance Jio Infocomm Limited to AS18101 Reliance Communications Ltd. We are more interested in the return tho. AS18101 Reliance Communications Ltd‘s routers do not have looking glass to find out the return path.
Let’s trace to the other IP.
arhue@xubuntu-desktop:~$ traceroute 182.79.231.2 traceroute to 182.79.231.2 (182.79.231.2), 30 hops max, 60 byte packets 1 gateway (192.168.42.129) 1.216 ms 1.286 ms 1.391 ms 2 * * * 3 10.71.225.130 (10.71.225.130) 59.200 ms 59.317 ms 59.416 ms 4 172.16.93.211 (172.16.93.211) 59.152 ms 59.270 ms 59.253 ms 5 172.26.31.6 (172.26.31.6) 58.671 ms 58.656 ms 58.694 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 218.100.48.20 (218.100.48.20) 127.326 ms 85.924 ms 94.720 ms 12 182.79.243.105 (182.79.243.105) 94.810 ms 94.666 ms 99.804 ms 13 * * * 14 * * * 15 * * *
It times out after that. This is super interesting because if you look at NIXI‘s(AS24029 NIXI is an IXP in India) connected network list, 218.100.48.20 is the the AS9498 Bharti Airtel Ltd‘s IP at Delhi(Noida). So why is it flowing through Delhi(Noida) and not Mumbai? Both are connected at both places right? Well it turns out Airtel’s routers only share the regional routes for peering since they do not want to carry the traffic over their backbone. Here are NIXI’s LG results for Delhi(Noida):
show ip bgp 182.79.231.2
Number of BGP Routes matching display condition : 1
Status codes: s suppressed, d damped, h history, * valid, > best, i internal
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop MED LocPrf Weight Path
*> 182.79.231.0/24 218.100.48.20 100 0 9498 ?
Last update to IP routing table: 4d10h26m56s, 1 path(s) installed:
Route is advertised to 19 peers:
218.100.48.9(55644) 218.100.48.40(132453) 218.100.48.10(10029)
218.100.48.12(9583) 218.100.48.13(17439) 218.100.48.15(9829)
218.100.48.17(17426) 218.100.48.24(58640) 218.100.48.25(17754)
218.100.48.27(17488) 218.100.48.28(55410) 218.100.48.29(10201)
218.100.48.30(4755) 218.100.48.32(38625) 218.100.48.34(45528)
218.100.48.37(132323) 218.100.48.36(132215) 218.100.48.26(18101)
218.100.48.31(55836)
Route is to be sent to 3 peers:
218.100.48.21(9498) 218.100.48.23(63829) 218.100.48.6(25152)
And for Mumbai:
show ip bgp 182.79.231.2 BGP4 : None of the BGP4 routes match the display condition
The path taken is AS55836 Reliance Jio Infocomm Limited(Goa)>AS24029 NIXI is an IXP in India(Delhi)>AS9498 Bharti Airtel Ltd(Delhi). It is interesting that AS9498 Bharti Airtel Ltd does not advertise all routes at all locations domestically but does so at international locations. Let’s see how the flow is in the other direction. Trace from AS9498 Bharti Airtel Ltd‘s LG on their Delhi router.
Tue Nov 29 05:40:15 GMT+05:30 2016 traceroute 49.35.248.119 traceroute to 49.35.248.119 (49.35.248.119), 30 hops max, 40 byte packets 1 182.79.201.85 (182.79.201.85) 3.314 ms 182.79.201.89 (182.79.201.89) 0.985 ms 1.133 ms 2 218.100.48.31 (218.100.48.31) 2.090 ms 2.104 ms 1.715 ms 3 172.16.24.7 (172.16.24.7) 33.088 ms 31.330 ms 172.16.24.3 (172.16.24.3) 29.648 ms 4 172.16.24.3 (172.16.24.3) 29.586 ms 172.16.24.7 (172.16.24.7) 31.102 ms 172.16.24.3 (172.16.24.3) 29.648 ms 5 * * * 6 * * *
It times out after that. 218.100.48.31 is AS55836 Reliance Jio Infocomm Limited‘s IP on NIXI’s member list. So return path is similar and flows through NIXI as well. Which is good! show route 49.35.248.119 for AS9498 Bharti Airtel Ltd‘s Delhi router:
Tue Nov 29 05:33:19 GMT+05:30 2016
show route 49.35.248.119 detail | no-more
lookingglass@DEL-ISP-ACC-RTR-50> ...9 detail | no-more
inet.0: 696246 destinations, 1393589 routes (696203 active, 14 holddown, 207 hidden)
Restart Complete
49.35.128.0/17 (2 entries, 1 announced)
*BGP Preference: 170/-491
Next hop type: Indirect
Address: 0x18644e9c
Next-hop reference count: 9422
Source: 203.101.87.68
Next hop type: Router, Next hop index: 1049641
Next hop: 182.79.190.58 via xe-10/1/0.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0xc97a
Next hop: 182.79.217.98 via xe-8/0/1.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x80c
Next hop: 182.79.217.94 via xe-8/0/2.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x80d
Next hop: 182.79.208.118 via xe-8/1/1.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x80e
Next hop: 182.79.208.174 via xe-9/0/0.0, selected
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x90d
Next hop: 182.79.208.122 via xe-9/0/3.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x90f
Next hop: 182.79.208.182 via xe-9/1/0.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x910
Next hop: 182.79.208.186 via xe-9/2/0.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x911
Next hop: 182.79.208.178 via xe-9/3/0.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x912
Next hop: 182.79.203.126 via xe-7/2/0.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0xa246
Next hop: 182.79.203.114 via xe-7/3/2.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0xa247
Next hop: 182.79.190.62 via xe-10/1/3.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0xc97b
Next hop: 182.79.237.210 via xe-1/0/1.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x6df
Next hop: 182.79.255.10 via xe-1/1/0.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x6e1
Next hop: 182.79.190.66 via xe-10/1/7.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0xc97c
Next hop: 182.79.190.46 via xe-10/2/0.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0xc97d
Next hop: 182.79.190.50 via xe-10/2/1.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0xc97e
Next hop: 182.79.190.54 via xe-10/3/5.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0xc97f
Next hop: 182.79.248.45 via xe-1/3/1.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x6e3
Next hop: 182.79.252.57 via xe-0/1/0.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x6c6
Next hop: 182.79.255.250 via xe-0/1/1.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x6c7
Next hop: 182.79.252.209 via xe-0/2/0.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x6c8
Next hop: 182.79.237.206 via xe-0/2/3.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x6c9
Next hop: 182.79.252.213 via xe-0/3/0.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x6ca
Next hop: 182.79.248.49 via xe-0/3/1.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x6cb
Next hop: 182.79.255.246 via xe-0/3/3.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x6cc
Protocol next hop: 203.101.87.74
Indirect next hop: 0x44ea6184 1049631 INH Session ID: 0xc986
State: <Active Int Ext>
Local AS: 9498 Peer AS: 9498
Age: 4d 10:40:13 Metric2: 103
Validation State: unverified
Task: BGP_9498.203.101.87.68+46929
Announcement bits (3): 0-KRT 5-Resolve tree 2 9-RT
AS path: 24029 55836 I (Originator)
Cluster list: 0.0.0.50
Originator ID: 203.101.87.74
AS path: Recorded
Communities: 9498:92 9498:24029 14111:20111 no-export
Accepted
Localpref: 490
Router ID: 203.101.87.68
BGP Preference: 170/-491
Next hop type: Indirect
Address: 0x18644e9c
Next-hop reference count: 9422
Source: 203.101.87.71
Next hop type: Router, Next hop index: 1049641
Next hop: 182.79.190.58 via xe-10/1/0.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0xc97a
Next hop: 182.79.217.98 via xe-8/0/1.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x80c
Next hop: 182.79.217.94 via xe-8/0/2.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x80d
Next hop: 182.79.208.118 via xe-8/1/1.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x80e
Next hop: 182.79.208.174 via xe-9/0/0.0, selected
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x90d
Next hop: 182.79.208.122 via xe-9/0/3.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x90f
Next hop: 182.79.208.182 via xe-9/1/0.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x910
Next hop: 182.79.208.186 via xe-9/2/0.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x911
Next hop: 182.79.208.178 via xe-9/3/0.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x912
Next hop: 182.79.203.126 via xe-7/2/0.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0xa246
Next hop: 182.79.203.114 via xe-7/3/2.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0xa247
Next hop: 182.79.190.62 via xe-10/1/3.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0xc97b
Next hop: 182.79.237.210 via xe-1/0/1.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x6df
Next hop: 182.79.255.10 via xe-1/1/0.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x6e1
Next hop: 182.79.190.66 via xe-10/1/7.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0xc97c
Next hop: 182.79.190.46 via xe-10/2/0.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0xc97d
Next hop: 182.79.190.50 via xe-10/2/1.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0xc97e
Next hop: 182.79.190.54 via xe-10/3/5.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0xc97f
Next hop: 182.79.248.45 via xe-1/3/1.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x6e3
Next hop: 182.79.252.57 via xe-0/1/0.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x6c6
Next hop: 182.79.255.250 via xe-0/1/1.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x6c7
Next hop: 182.79.252.209 via xe-0/2/0.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x6c8
Next hop: 182.79.237.206 via xe-0/2/3.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x6c9
Next hop: 182.79.252.213 via xe-0/3/0.0
Label operation: Push 465296
Label TTL action: no-prop-ttl
Session Id: 0x6ca
Next hop: 182.79.248.49 via xe-0/3/1.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x6cb
Next hop: 182.79.255.246 via xe-0/3/3.0
Label operation: Push 418113
Label TTL action: no-prop-ttl
Session Id: 0x6cc
Protocol next hop: 203.101.87.74
Indirect next hop: 0x44ea6184 1049631 INH Session ID: 0xc986
State: <NotBest Int Ext>
Inactive reason: Not Best in its group - Update source
Local AS: 9498 Peer AS: 9498
Age: 4d 10:40:13 Metric2: 103
Validation State: unverified
Task: BGP_9498.203.101.87.71+23609
AS path: 24029 55836 I (Originator)
Cluster list: 0.0.0.50
Originator ID: 203.101.87.74
AS path: Recorded
Communities: 9498:92 9498:24029 14111:20111 no-export
Accepted
Localpref: 490
Router ID: 203.101.87.71
show route 49.35.248.119 for AS9498 Bharti Airtel Ltd‘s Mumbai router:
Tue Nov 29 05:36:52 GMT+05:30 2016
show route 49.35.248.119 detail | no-
lookingglass@MUM-SC-ISP-IGW-RTR-116> ...9 detail | no-m ore
inet.0: 694924 destinations, 1396880 routes (694870 active, 3 holddown, 7457 hidden)
Restart Complete
49.35.128.0/17 (1 entry, 1 announced)
*BGP Preference: 170/-502
Next hop type: Router, Next hop index: 1985
Address: 0x214dc5b4
Next-hop reference count: 393
Source: 218.100.48.65
Next hop: 218.100.48.82 via xe-0/1/2.0, selected
Session Id: 0xf460
State: <Active Ext>
Local AS: 9498 Peer AS: 24029
Age: 2d 17:13:25
Validation State: unverified
Task: BGP_24029.218.100.48.65+8001
Announcement bits (4): 0-KRT 2-RT 8-Resolve tree 2 9-BGP_RT_Background
AS path: 24029 55836 I
AS path: Recorded
Communities: 9498:92 9498:24029 14111:20111 no-export
Accepted
Localpref: 501
Router ID: 218.100.48.65
AS55836 Reliance Jio Infocomm Limited is announcing 49.35.128.0/17 at both locations at NIXI and AS9498 Bharti Airtel Ltd‘s routers are learning that through them.
Conclusion
This seems to be kind of a stop gap measure by Valve. It would be nice if they start announcing their IPs in India. I think Cloudflare is doing an island POP here, with their upstreams not announcing IPs outside of India. If they peer with most eyeball networks I doubt they would have a pay much for transit. I think there would not be any serious international traffic as well.
I think there is a default option for the download mirror on the client as well but you can’t set it once you change it to something else. So I couldn’t test it out. Also I didn’t have time to test out the other mirrors.
Well that was a fun few hours of poking around. I hope you found this interesting. 🙂
i need to know if AS55836 has a looking glass.
I don’t think so. But I think there are probes on RIPE Atlas project(https://atlas.ripe.net), which will allow you to ping/traceroute from AS55836.
I’ve the ip pools for Mumbai and Chennai servers, Interestingly mumbai server connecting through airtel routes and chennai server connecting through reliance communication routes for every isp available in india.
Mumbai Server : 155.133.233.0/24
Chennai Server : 155.133.232.0/24
Good to know. Thanks
Good work on the assessment . Bring more news and details to top streain services and issues . N one has entered that field yet