Analysing Steam’s network in India

By | 29th November 2016

Excited by the prospect for Steam for Linux I fired up Steam to download some games. The problem was that finding a mirror with the best speed for seeming problematic. So I decided to do some analysis. 😀

I am testing all of this from AS55836 Reliance Jio Infocomm Limited. I primarily use this for steam since the throughput is nice to have and I don’t use more than 4GB of download on Steam games per day. To choose a mirror goto Steam>Settings>Download>Download region in the menu.

Singapore mirror

Since Reliance Jio peers with their upstream AS64049 Reliance Jio Infocomm Pte Ltd Singapore in Singapore, I decided to try the Singapore mirror first. The results? Not bad. I’ve used iftop command on Linux to see from which IPs the download was being streamed from.

screenshot_2016-11-27_06-31-49

iftop and game download from Singapore

The throughput was decent at 2.5 MB/s.The only problem is that it would be affected badly during peak times, due to heavy strain on the international capacity. As you can see most of the IPs are from 103.10.124.0/24 network, which is announced by AS32590 Valve Corporation in Singapore. Let’s do a traceroute, shall we?

arhue@xubuntu-desktop:~/Downloads$ traceroute 103.10.124.20
traceroute to 103.10.124.20 (103.10.124.20), 30 hops max, 60 byte packets
 1  gateway (192.168.42.129)  2.327 ms  2.322 ms  2.390 ms
 2  * * *
 3  10.71.225.130 (10.71.225.130)  111.569 ms  111.609 ms  111.705 ms
 4  172.16.93.209 (172.16.93.209)  96.941 ms  97.030 ms  97.130 ms
 5  172.26.31.2 (172.26.31.2)  96.664 ms  96.638 ms  96.805 ms
 6  * * *
 7  * * *
 8  * * *
 9  103.198.140.164 (103.198.140.164)  93.141 ms  93.010 ms  92.984 ms
10  103.198.140.27 (103.198.140.27)  186.625 ms  152.113 ms  160.538 ms
11  ldt-au01.ibeo.hgc-intl.com (80.81.194.166)  196.549 ms  191.214 ms  196.720 ms
12  global.hgc.com.hk (218.189.8.141)  249.810 ms  254.026 ms  235.519 ms
13  d1-244-224-143-118-on-nets.com (118.143.224.244)  243.390 ms  253.236 ms  239.243 ms
14  global.hgc.com.hk (218.189.12.234)  238.645 ms  246.932 ms  234.971 ms
15  * * *
16  * * *

It times out after that probably because ICMP is disabled after that. The path taken is AS55836 Reliance Jio Infocomm Limited(India)>AS64049 Reliance Jio Infocomm Pte Ltd Singapore(Singapore)>AS9304 Hutchison Global Communications(Singapore)>???. Luckily AS9304 Hutchison Global Communications has a looking glass for us to peak into. You can find it here. Let’s do ping.

ping 103.10.124.20

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 103.10.124.20, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

And a show ip bgp command.

show ip bgp 103.10.124.20
BGP routing table entry for 103.10.124.0/24, version 321730
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  9304 32590 32590, (aggregated by 32590 103.10.124.255)
    118.143.235.129 from 118.143.235.129 (118.143.225.4)
      Origin IGP, localpref 100, valid, external, best

So AS9304 Hutchison Global Communications peers with Valve. The next hop should be AS32590 Valve Corporation directly. The location is most probably Equinix Singapore if you have a look at Valve’s PeeringDB profile. The more important downstream path is hidden since Valve doesn’t have a looking glass. Let’s just assume that AS32590 Valve Corporation‘s router picks path through AS9304 Hutchison Global Communications as the best, which could be likely since they peer with them.

show ip bgp 49.35.248.119
BGP routing table entry for 49.35.128.0/17, version 73468304
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  9304 15412 3491 64049 55836
    118.143.235.129 from 118.143.235.129 (118.143.225.4)
      Origin IGP, localpref 100, valid, external, best

Path is AS9304 Hutchison Global Communications>AS15412 Reliance Glocalcom Limited(Singapore)>AS3491 PCCW Global(Singapore)>AS64049 Reliance Jio Infocomm Pte Ltd Singapore(Singapore)>AS55836 Reliance Jio Infocomm Limited(India). The same local preference means AS9304 Hutchison Global Communications is peering with AS15412 Reliance Glocalcom Limited. The long AS path length probably means AS32590 Valve Corporation‘s router may not choose to send traffic this way unless they are not peering with any network which is advertising a shorter one. Also probably AS18101 Reliance Communications Ltd‘s upstream AS15412 Reliance Glocalcom Limited is not announcing AS55836 Reliance Jio Infocomm Limited‘s IP pools in Singapore but is only used for local connectivity.

India(Mumbai) mirror

I am going to consider Mumbai since it is closest to me.

screenshot_2016-11-27_06-34-57

iftop and game download from Mumbai(India)

The throughput was better at about 2.6 MB/s and this would be better at peak times. This test was done at 6 AM so the results would have been different at different times of the day. Let’s do a trace.

arhue@xubuntu-desktop:~/Downloads$ traceroute 115.248.101.129
traceroute to 115.248.101.129 (115.248.101.129), 30 hops max, 60 byte packets
 1  gateway (192.168.42.129)  2.866 ms  2.968 ms  3.065 ms
 2  * * *
 3  10.71.225.162 (10.71.225.162)  89.943 ms  90.042 ms  90.015 ms
 4  172.16.93.211 (172.16.93.211)  89.604 ms  89.455 ms  89.675 ms
 5  172.26.31.6 (172.26.31.6)  89.275 ms  89.136 ms  89.348 ms
 6  * * *
 7  * * *
 8  * * *
 9  115.249.214.165 (115.249.214.165)  1555.076 ms  1555.055 ms  1555.133 ms
10  * * *
11  * * *

It is directly from AS55836 Reliance Jio Infocomm Limited to AS18101 Reliance Communications Ltd. We are more interested in the return tho. AS18101 Reliance Communications Ltd‘s routers do not have looking glass to find out the return path.

Let’s trace to the other IP.

arhue@xubuntu-desktop:~$ traceroute 182.79.231.2
traceroute to 182.79.231.2 (182.79.231.2), 30 hops max, 60 byte packets
 1  gateway (192.168.42.129)  1.216 ms  1.286 ms  1.391 ms
 2  * * *
 3  10.71.225.130 (10.71.225.130)  59.200 ms  59.317 ms  59.416 ms
 4  172.16.93.211 (172.16.93.211)  59.152 ms  59.270 ms  59.253 ms
 5  172.26.31.6 (172.26.31.6)  58.671 ms  58.656 ms  58.694 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  218.100.48.20 (218.100.48.20)  127.326 ms  85.924 ms  94.720 ms
12  182.79.243.105 (182.79.243.105)  94.810 ms  94.666 ms  99.804 ms
13  * * *
14  * * *
15  * * *

It times out after that. This is super interesting because if you look at NIXI‘s(AS24029 NIXI is an IXP in India) connected network list, 218.100.48.20 is the the AS9498 Bharti Airtel Ltd‘s IP at Delhi(Noida). So why is it flowing through Delhi(Noida) and not Mumbai? Both are connected at both places right? Well it turns out Airtel’s routers only share the regional routes for peering since they do not want to carry the traffic over their backbone. Here are NIXI’s LG results for Delhi(Noida):

show ip bgp 182.79.231.2
Number of BGP Routes matching display condition : 1
Status codes: s suppressed, d damped, h history, * valid, > best, i internal
Origin codes: i - IGP, e - EGP, ? - incomplete
    Network            Next Hop        MED    LocPrf     Weight Path
*>  182.79.231.0/24    218.100.48.20          100        0      9498 ?
       Last update to IP routing table: 4d10h26m56s, 1 path(s) installed:
       Route is advertised to 19 peers:
        218.100.48.9(55644)                      218.100.48.40(132453)                    218.100.48.10(10029)                     
        218.100.48.12(9583)                      218.100.48.13(17439)                     218.100.48.15(9829)                      
        218.100.48.17(17426)                     218.100.48.24(58640)                     218.100.48.25(17754)                     
        218.100.48.27(17488)                     218.100.48.28(55410)                     218.100.48.29(10201)                     
        218.100.48.30(4755)                      218.100.48.32(38625)                     218.100.48.34(45528)                     
        218.100.48.37(132323)                    218.100.48.36(132215)                    218.100.48.26(18101)                     
        218.100.48.31(55836)                     
       Route is to be sent to 3 peers:
        218.100.48.21(9498)                      218.100.48.23(63829)                     218.100.48.6(25152)

And for Mumbai:

show ip bgp 182.79.231.2
BGP4 : None of the BGP4 routes match the display condition

The path taken is AS55836 Reliance Jio Infocomm Limited(Goa)>AS24029 NIXI is an IXP in India(Delhi)>AS9498 Bharti Airtel Ltd(Delhi). It is interesting that AS9498 Bharti Airtel Ltd does not advertise all routes at all locations domestically but does so at international locations. Let’s see how the flow is in the other direction. Trace from AS9498 Bharti Airtel Ltd‘s LG on their Delhi router.

Tue Nov 29 05:40:15 GMT+05:30 2016
 traceroute 49.35.248.119 
traceroute to 49.35.248.119 (49.35.248.119), 30 hops max, 40 byte packets
 1  182.79.201.85 (182.79.201.85)  3.314 ms 182.79.201.89 (182.79.201.89)  0.985 ms  1.133 ms
 2  218.100.48.31 (218.100.48.31)  2.090 ms  2.104 ms  1.715 ms
 3  172.16.24.7 (172.16.24.7)  33.088 ms  31.330 ms 172.16.24.3 (172.16.24.3)  29.648 ms
 4  172.16.24.3 (172.16.24.3)  29.586 ms 172.16.24.7 (172.16.24.7)  31.102 ms 172.16.24.3 (172.16.24.3)  29.648 ms
 5  * * *
 6  * * *

It times out after that. 218.100.48.31 is AS55836 Reliance Jio Infocomm Limited‘s IP on NIXI’s member list. So return path is similar and flows through NIXI as well. Which is good! show route 49.35.248.119 for AS9498 Bharti Airtel Ltd‘s Delhi router:

Tue Nov 29 05:33:19 GMT+05:30 2016
 show route 49.35.248.119 detail | no-more
lookingglass@DEL-ISP-ACC-RTR-50> ...9 detail | no-more                       

inet.0: 696246 destinations, 1393589 routes (696203 active, 14 holddown, 207 hidden)
Restart Complete
49.35.128.0/17 (2 entries, 1 announced)
        *BGP    Preference: 170/-491
                Next hop type: Indirect
                Address: 0x18644e9c
                Next-hop reference count: 9422
                Source: 203.101.87.68
                Next hop type: Router, Next hop index: 1049641
                Next hop: 182.79.190.58 via xe-10/1/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xc97a
                Next hop: 182.79.217.98 via xe-8/0/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x80c
                Next hop: 182.79.217.94 via xe-8/0/2.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x80d
                Next hop: 182.79.208.118 via xe-8/1/1.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x80e
                Next hop: 182.79.208.174 via xe-9/0/0.0, selected
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x90d
                Next hop: 182.79.208.122 via xe-9/0/3.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x90f
                Next hop: 182.79.208.182 via xe-9/1/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x910
                Next hop: 182.79.208.186 via xe-9/2/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x911
                Next hop: 182.79.208.178 via xe-9/3/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x912
                Next hop: 182.79.203.126 via xe-7/2/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xa246
                Next hop: 182.79.203.114 via xe-7/3/2.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xa247
                Next hop: 182.79.190.62 via xe-10/1/3.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xc97b
                Next hop: 182.79.237.210 via xe-1/0/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6df
                Next hop: 182.79.255.10 via xe-1/1/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6e1
                Next hop: 182.79.190.66 via xe-10/1/7.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xc97c
                Next hop: 182.79.190.46 via xe-10/2/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xc97d
                Next hop: 182.79.190.50 via xe-10/2/1.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xc97e
                Next hop: 182.79.190.54 via xe-10/3/5.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xc97f
                Next hop: 182.79.248.45 via xe-1/3/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6e3
                Next hop: 182.79.252.57 via xe-0/1/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6c6
                Next hop: 182.79.255.250 via xe-0/1/1.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6c7
                Next hop: 182.79.252.209 via xe-0/2/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6c8
                Next hop: 182.79.237.206 via xe-0/2/3.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6c9
                Next hop: 182.79.252.213 via xe-0/3/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6ca
                Next hop: 182.79.248.49 via xe-0/3/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6cb
                Next hop: 182.79.255.246 via xe-0/3/3.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6cc
                Protocol next hop: 203.101.87.74
                Indirect next hop: 0x44ea6184 1049631 INH Session ID: 0xc986
                State: <Active Int Ext>
                Local AS:  9498 Peer AS:  9498
                Age: 4d 10:40:13 	Metric2: 103 
                Validation State: unverified 
                Task: BGP_9498.203.101.87.68+46929
                Announcement bits (3): 0-KRT 5-Resolve tree 2 9-RT 
                AS path: 24029 55836 I (Originator)
                Cluster list:  0.0.0.50
                Originator ID: 203.101.87.74
                AS path: Recorded
                Communities: 9498:92 9498:24029 14111:20111 no-export
                Accepted
                Localpref: 490
                Router ID: 203.101.87.68
         BGP    Preference: 170/-491
                Next hop type: Indirect
                Address: 0x18644e9c
                Next-hop reference count: 9422
                Source: 203.101.87.71
                Next hop type: Router, Next hop index: 1049641
                Next hop: 182.79.190.58 via xe-10/1/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xc97a
                Next hop: 182.79.217.98 via xe-8/0/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x80c
                Next hop: 182.79.217.94 via xe-8/0/2.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x80d
                Next hop: 182.79.208.118 via xe-8/1/1.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x80e
                Next hop: 182.79.208.174 via xe-9/0/0.0, selected
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x90d
                Next hop: 182.79.208.122 via xe-9/0/3.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x90f
                Next hop: 182.79.208.182 via xe-9/1/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x910
                Next hop: 182.79.208.186 via xe-9/2/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x911
                Next hop: 182.79.208.178 via xe-9/3/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x912
                Next hop: 182.79.203.126 via xe-7/2/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xa246
                Next hop: 182.79.203.114 via xe-7/3/2.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xa247
                Next hop: 182.79.190.62 via xe-10/1/3.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xc97b
                Next hop: 182.79.237.210 via xe-1/0/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6df
                Next hop: 182.79.255.10 via xe-1/1/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6e1
                Next hop: 182.79.190.66 via xe-10/1/7.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xc97c
                Next hop: 182.79.190.46 via xe-10/2/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xc97d
                Next hop: 182.79.190.50 via xe-10/2/1.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xc97e
                Next hop: 182.79.190.54 via xe-10/3/5.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xc97f
                Next hop: 182.79.248.45 via xe-1/3/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6e3
                Next hop: 182.79.252.57 via xe-0/1/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6c6
                Next hop: 182.79.255.250 via xe-0/1/1.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6c7
                Next hop: 182.79.252.209 via xe-0/2/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6c8
                Next hop: 182.79.237.206 via xe-0/2/3.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6c9
                Next hop: 182.79.252.213 via xe-0/3/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6ca
                Next hop: 182.79.248.49 via xe-0/3/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6cb
                Next hop: 182.79.255.246 via xe-0/3/3.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6cc
                Protocol next hop: 203.101.87.74
                Indirect next hop: 0x44ea6184 1049631 INH Session ID: 0xc986
                State: <NotBest Int Ext>
                Inactive reason: Not Best in its group - Update source
                Local AS:  9498 Peer AS:  9498
                Age: 4d 10:40:13 	Metric2: 103 
                Validation State: unverified 
                Task: BGP_9498.203.101.87.71+23609
                AS path: 24029 55836 I (Originator)
                Cluster list:  0.0.0.50
                Originator ID: 203.101.87.74
                AS path: Recorded
                Communities: 9498:92 9498:24029 14111:20111 no-export
                Accepted
                Localpref: 490
                Router ID: 203.101.87.71

show route 49.35.248.119 for AS9498 Bharti Airtel Ltd‘s Mumbai router:

Tue Nov 29 05:36:52 GMT+05:30 2016
 show route 49.35.248.119 detail | no-
lookingglass@MUM-SC-ISP-IGW-RTR-116> ...9 detail | no-m                      ore 

inet.0: 694924 destinations, 1396880 routes (694870 active, 3 holddown, 7457 hidden)
Restart Complete
49.35.128.0/17 (1 entry, 1 announced)
        *BGP    Preference: 170/-502
                Next hop type: Router, Next hop index: 1985
                Address: 0x214dc5b4
                Next-hop reference count: 393
                Source: 218.100.48.65
                Next hop: 218.100.48.82 via xe-0/1/2.0, selected
                Session Id: 0xf460
                State: <Active Ext>
                Local AS:  9498 Peer AS: 24029
                Age: 2d 17:13:25 
                Validation State: unverified 
                Task: BGP_24029.218.100.48.65+8001
                Announcement bits (4): 0-KRT 2-RT 8-Resolve tree 2 9-BGP_RT_Background 
                AS path: 24029 55836 I
                AS path: Recorded
                Communities: 9498:92 9498:24029 14111:20111 no-export
                Accepted
                Localpref: 501
                Router ID: 218.100.48.65

AS55836 Reliance Jio Infocomm Limited is announcing 49.35.128.0/17 at both locations at NIXI and AS9498 Bharti Airtel Ltd‘s routers are learning that through them.

Conclusion

This seems to be kind of a stop gap measure by Valve. It would be nice if they start announcing their IPs in India. I think Cloudflare is doing an island POP here, with their upstreams not announcing IPs outside of India. If they peer with most eyeball networks I doubt they would have a pay much for transit. I think there would not be any serious international traffic as well.

I think there is a default option for the download mirror on the client as well but you can’t set it once you change it to something else. So I couldn’t test it out. Also I didn’t have time to test out the other mirrors.

Well that was a fun few hours of poking around. I hope you found this interesting. 🙂

5 thoughts on “Analysing Steam’s network in India

  1. R.Basak

    I’ve the ip pools for Mumbai and Chennai servers, Interestingly mumbai server connecting through airtel routes and chennai server connecting through reliance communication routes for every isp available in india.

    Mumbai Server : 155.133.233.0/24
    Chennai Server : 155.133.232.0/24

    Reply
  2. Shashank

    Good work on the assessment . Bring more news and details to top streain services and issues . N one has entered that field yet

    Reply

Leave a Reply