This post is constantly updated with whatever information I can get my hands on. Refresh for updates.
World wide panic!!! Google is down! Grab your routers and run!!!11
Well not really but internet services were massively affected due to the Cyclone Vardah near Chennai.
To my knowledge three major things happened.
- Google(AS15169) had huge problems and traffic was shifted to POPs other than Mumbai.
- Vodafone’s Chennai location had an outage. Traffic was being moved to Mumbai. Enterprise and retail customers had issues.
- There was huge packet loss on most international websites, possibly due to capacity issues.
I am going to talk about the first one.
Google down you say?
Well I was alerted by this post by my friend and unusually slow internet.
Seems like traffic for Google routing to Chennai (instead of Mumbai) for few ISPs in India. Impact of #vardahcyclone ?
— Anurag Bhatia (@anurag_bhatia) December 12, 2016
So let’s explore what went wrong. Inspite of having very high packet loss while accessing RIPE’s website, I could manage to get one traceroute through RIPE Atlas project.
Trace from AS58457, Hyderabad to google.com at around 7 PM
As you can see traffic destined for Chennai is terminated at Singapore, possibly carried there by Google itself cold potato style. Likewise I noticed that I was hitting Delhi rather than Mumbai. So traffic was just moved away from Mumbai to other locations. Likely reason for this would be connectivity issues between Mumbai and Chennai possibly due to fiber cuts.
I continued to monitor the situation some time later. Link to test: https://atlas.ripe.net/measurements/6959487/#!probes
Trace to google.com from probes in India at around 1 AM
As you can clearly see even probes close to Mumbai were hitting Chennai instead of their POP closest to their region. Let’s test out the aspmx.l.google.com domain used for Google Apps which isn’t terminated in India. Link to test: https://atlas.ripe.net/measurements/6960037/#!probes
Trace to aspmx.l.google.com from probes in India at around 1 AM
As you can see the latencies are not good. Also my friend alerted to me a few strange traces like this.
Trace to aspmx.l.google.com from AS9430 at around 12:30 AM
What is happening here is that Google was rerouting traffic constantly and BGP routers were taking a while to respond.
So what exactly happened?
Google has a huge network. They can’t possibly be down, can they?
There is not much Google content actually in India right now. Most of it is either cached by Google itself, while terminating SSL on one of their Indian nodes – Mumbai, Chennai or Delhi. Eyeball networks/ISPs then either directly peer with Google or their upstreams do. Most YouTube content is just cached by GGC nodes inside ISP networks.
Google has limited capacity to Europe through Mumbai and most of Indian traffic is actually served from Taiwan via Singapore through a fat pipe in Chennai.
Moving traffic between sites is normally not a problem but this caused two major problems:
- ISPs were quite used to undersizing on IP transit capacity and would rely on peering with Google. Google is around 50% of their capacity so this does make sense.
- Google’s routes were constantly flapping and routers would be rather slow to react to that. Cheaper routers like Mikrotik would be most susceptible to this.
- Larger transit players and telecos likely had issues of their own.
So what happened was that Google wasn’t actually down. The connectivity from ISPs to Google was hampered because they didn’t have enough transit capacity if the plug was pulled on their peering capacity.
Long term Solutions
Likely solutions would be to avoid this in the future would be:
- Eyeball networks should always try to over build on transit capacity. So incase there is loss in connectivity to Google PNI, traffic can always be served from a different location from transit. If they are large and diverse enough, maybe they can peer at different locations. This would also prevent from carrying traffic over their backbone.
- Better routers for BGP or using something like BIRD would help. Mikrotik, which are very popular for small to medium networks in India can only do BGP on only one core, which can hamper performance.
- Smaller ISPs should have redundant upstreams. Vodafone was badly affected, while networks like Airtel and Tata were less so. Having redundancy helps.
- More content networks like Google should have more content inside the country. Google has plans of setting up a datacenter in Mumabi. More info here.
- Redundant connectivity between sites with different fiber paths would be nice for large networks.
- Outside connectivity with landing stations at multiple locations for transit and larger players would be nice for internet in India in general.
Change your DNS!
If you are using Google DNS(8.8.8.8, 8.8.4.4) move to OpenDNS(208.67.222.222, 208.67.220.220) immediately!
Also what I did was move from Google’s public DNS resolvers to OpenDNS because Google makes their queries from outside India and they are just cached at their POPs here so non-cached queries could have issues. That plus the fact that they use anycast and could hit a node like Mumbai would cause issues. Non-Google services were much better for me after I made the switch.
As I finish this post, I have noticed that connectivity is much better, atleast with my ISP(AS58405) so time to go watch some YouTube streams! 😀
Update 13/12/2016 3 PM:
I have some information that tells me that Google had interconnects from Powergrid(AS132215) that had issues. While connectivity by Powergrid was restored by morning, i2i cable leased by Google possibly has some issues as well and I am seeing a lot of Google traffic taking international route after my ISP’s upstream hands it off outside India. ping google.com is hitting South Carolina, after Airtel(AS9498) hands it off in London.
Update 13/12/2016 5:30 PM:
A friend confirmed that Airtel’s i2i cable indeed has issues.
Dear Customer, the cyclone in Chennai has impacted one of our undersea network cables which may affect your internet speeds. We deeply regret the inconvenience. Our engineers are working to resolve the issue and services will be normalized shortly.Airtel
Trace to Singapore was also going through LINX, London for me.
arhue@xubuntu-desktop:~/Videos/servo$ traceroute www.oneasiahost.com traceroute to www.oneasiahost.com (163.47.176.1), 30 hops max, 60 byte packets 1 gateway (192.168.10.1) 0.198 ms 0.255 ms 0.259 ms 2 10.102.254.1 (10.102.254.1) 26.036 ms 28.425 ms 29.244 ms 3 10.240.2.10 (10.240.2.10) 29.980 ms 30.215 ms 30.248 ms 4 103.65.198.1 (103.65.198.1) 32.776 ms 33.312 ms 33.446 ms 5 10.139.103.1 (10.139.103.1) 33.539 ms 33.543 ms 33.557 ms 6 aes-static-025.51.246.61.airtel.in (61.246.51.25) 47.429 ms 22.547 ms 58.581 ms 7 125.62.187.214 (125.62.187.214) 182.880 ms 182.79.222.17 (182.79.222.17) 186.652 ms 182.79.222.78 (182.79.222.78) 190.037 ms 8 linx-juniper.sg.gs (195.66.226.38) 246.037 ms 246.187 ms 246.149 ms 9 switch-1.v3.uk1.sg.gs (124.6.36.25) 252.934 ms 248.928 ms 263.849 ms 10 ns1.oneasiahost.com (163.47.176.1) 260.528 ms 259.311 ms 261.143 ms
I did an another RIPE Atlas test. Link here: https://atlas.ripe.net/measurements/6960503/#!probes
Trace to google.com from probes in India at around 5:30 PM
As you can see, the map shows a very bleak picture, with most of the traffic going outside the country. Let’s look at the OpenIPMap to see where most of the traces are landing up.
OpenIPMap for trace to google.com from probes in India at around 5:30 PM
As you can see very little traffic is going to Singapore directly via Chennai. This possibly means there is a problem with the i2i cable connecting India to Singapore. Google likely carries whatever traffic lands up in Europe to other places since it’s a cold potato network. It doesn’t prefer to carry it from India itself since the Europe-India pipe that they have isn’t very high capacity to carry all that traffic.
Update 13/12/2016 9:30 PM:
Traffic is back to hitting the Indian nodes. RIPE Atlas test here: https://atlas.ripe.net/measurements/6960735/#probes
Trace to google.com from probes in India at around 9:30 PM
As you can see traffic is hitting even the Mumbai node. I don’t know how these nodes are fed with content as it looks like the i2i cable system still has issues.
Tracerouting to Singapore the traffic all flows through London. For reverse path, I used HostSG(AS24482)’s looking glass.
Router: SG1 (Epsilon) - Edge-2 (Customer, Standard Route)
Command: traceroute inet 43.249.187.200 as-number-lookup source 203.175.175.2
1 edge-1.sg2.sg.gs (203.175.175.101) 0.827 ms 3.495 ms 0.837 ms
2 116.51.23.65 (116.51.23.65) [AS 2914] 3.106 ms 1.203 ms 1.743 ms
3 ae-4.r21.sngpsi05.sg.bb.gin.ntt.net (129.250.4.16) [AS 2914] 1.886 ms 2.267 ms 2.102 ms
MPLS Label=440911 CoS=0 TTL=1 S=1
4 ae-8.r24.londen12.uk.bb.gin.ntt.net (129.250.7.64) [AS 2914] 191.297 ms 188.173 ms 186.029 ms
MPLS Label=617522 CoS=0 TTL=1 S=1
5 ae-8.r02.londen03.uk.bb.gin.ntt.net (129.250.4.22) [AS 2914] 184.094 ms 189.205 ms 189.433 ms
6 flagtelecom-0.r02.londen03.uk.bb.gin.ntt.net (83.231.235.238) [AS 2914] 209.137 ms 200.179 ms 205.002 ms
7 85.95.27.125 (85.95.27.125) [AS 15412] 208.916 ms xe-0-0-1.0-pjr04.mmb004.flagtel.com (85.95.26.158) [AS 15412] 226.170 ms 248.547 ms
8 * * *
9 * * *
10 220.227.118.233 (220.227.118.233) [AS 18101] 215.753 ms 199.318 ms 199.310 ms
11 10.139.103.1 (10.139.103.1) 215.223 ms 215.225 ms 211.806 ms
12 43.249.187.200 (43.249.187.200) [AS 58405] 212.304 ms 214.853 ms 215.016 ms
13 43.249.187.200 (43.249.187.200) [AS 58405] 210.071 ms 212.743 ms 200.098 ms
As you can see NTT(AS2914) is carrying the traffic to UK, before handing it off to Flagtel(AS15412). Most probably Indian telecos have stopped announcements of Indian routes on Singapore routers and are preferring routes learnt from their European routers.
Google services should be better but we’ll have to wait and watch about how the i2i problem plays out. Networks not using the i2i cable should be much better off, as far as connectivity to Singapore is concerned.
Update 13/01/2017 6:30 PM:
i2i cable issues are still not fixed. Airtel is using SE-ME-WE4 to get to Singapore instead of i2i but the Singapore to India capacity is probably not as much as required so they are sharing fewer Indian routes in Singapore. You can observe a lot of traffic going via Europe to India. The forward path from India to Singapore is fine, probably because there is very less traffic from India to Singapore so little choking in general. My ISP uses Reliance communications(AS18101) for return(heavily prepends announcements to Airtel) so I am much better off.
Tata Communications Ltd. has issues with TIC cable as well. This is what they sent to their customers.
You may observe additional latency of 30 -230ms for US West Coast and APAC prefix.
Please be informed that, this is due to under-sea cable fault on TIC submarine cable system. The fault is located in the wet segment in between repeater R58 & R59, this is near Singapore CLS, around 85-95 KMs from CLS.
As per latest update, the TIC cable fault repair is tentatively scheduled from 8th February 2017 and permit process is in progress. Additionally during repair operation Repeater R52 replacement will be carried out. The tentative ETR for the repair completion is 19th February 2017 depending on weather conditions.Tata Communications Ltd.
Fixing i2i and TIC cables could take a few more months, before which latency to Singapore could be affected. Fixing submarine cables is a long process. You need good planning, permits, allocating resources, sending diving teams etc. So we’ll have to be patient
Pingback: Google affected due to Cyclone Vardah in Chennai, India | moomblr
I am in Pune, youbroadband customer, and i have been facing youtube loading issues since 7pm yesterday.
YouTube could be slow because of transits choking as content is still not served out of Mumbai, atleast for me. See if you can watch any popular YouTube content as it will be served by GGC cache your ISP will have in their premises.
To add to that larger transit players also have issues of their own. If you use Google public DNS, switch to OpenDNS. That will help massively.
Thanks for the reply varunpriolkar, indeed the content i was trying to watch on youtube was not regional, it was some corsair power-supply reviews, and i have a 10mbps connection which while not being superduper fast like my fiber friends, is good enough for hd streaming.
Im sure if it was some popular regional content it oc course have been cached by the google CDN or something similar.
The interesting thing is that if you do a speedtest and the server is youbroadband itself, you will be getting 10mbps, and yet websites in the other window werent opening so i didnt get it at first. Now i understand that since the youbroadband server is hosted locally, those lines are up, so ofcourse i will see the full speed in the tests, but for sites with servers that are hosted abroad is where the low speed issue persists. In the dnsleaktest, the DNS im using is youbroadbands own servers. Interesting thing is im a VPN subscriber, so yesterday evening i thought ok i will switch on the VPN to show im coming from the EU, but i was still facing the same issues, so now i understand that regardless of the external ip, the encrypted packets must still be using the Chennai fiber to come to me, so perhaps thats why even with a VPN the youtube issue was replicated. There must be other undersea cables coming in from east and west coast of India right? i will have to check the maps, but in any case, thank you for your article, im glad it was picked up by googles crawlers, because curious people like myself would like to be told in a bit more detail what exactly is happening.
>the content i was trying to watch on youtube was not regional, it was some corsair power-supply reviews
That doesn’t matter. Popular content is usually stored on the caches. You can use iftop or EtherApe to see from where the stream is getting played from.
>The interesting thing is that if you do a speedtest and the server is youbroadband itself, you will be getting 10mbps
Speedtests are completely unreliable and will not tell you the bandwidth you can push through. Many ISPs like to prioritise Speedtest traffic or just host an instannce of it. You can try iperf/ftp with a small VPS in some locations.
>Interesting thing is im a VPN subscriber, so yesterday evening i thought ok i will switch on the VPN to show im coming from the EU
VPN wouldn’t make any difference since IP transit capacity of your ISP is probably choked. Also a lot of traffic would have been directed from Europe to Singapore and that could choke pipes for upstream providers too.
Also VPN will form a tunnel. So if you are using a EU location, packets will come back through the same location as well.
>There must be other undersea cables coming in from east and west coast of India right?
I haven’t looked at the submarine cable maps but from what I know many providers use Airtel’s i2i cable. Jio and Vodafone are in the process of completing the BBG cable to Singapore too.
>but in any case, thank you for your article, im glad it was picked up by googles crawlers, because curious people like myself would like to be told in a bit more detail what exactly is happening.
Not a problem. Right now traffic is back to hitting Chennai(no idea where it is fed from since Singapore traffic is still handed off by Airtel at London and is not using the i2i link). I will continue to monitor the situation.
Ahh you have shed some more light in the muddled waters of my comprehension. I will have to google a couple of things you have said because they are quite technical for a noob like me. Speaking of submarine cables, did a quick lookup on the i2i and wiki says that cables peak throughput can touch 8.4 Tbit/s which is frikkin insane!
I remember a great article on arstechnica this year, describing “How the Internet works: Submarine fiber, brains in jars, and coaxial cables”. They begin with “The secret world of cable landing sites” describing the landing station of Tata’s Atlantic submarine cable.
Its a fascinating indepth article that your readers might appreciate:
http://arstechnica.com/information-technology/2016/05/how-the-internet-works-submarine-cables-data-centres-last-mile/
There is also a cool map of the world’s submarine cables.
That’s a very interesting map indeed. Thank you for sharing.
Not all wavelengths/fibers are lit and capacity is often shared in specific quantities. At scale 8.4 Tbit/s is not insane at all.
Check these graphs out:
https://ams-ix.net/technical/statistics
https://www.linx.net/tech-info-help/traffic-stats
That’s literally a few tbit/s of capacity flowing through an internet exchange(IX).
Hi Varun,
in the AMS-IX traffic graph, what is that dip in the middle where it says “04” ?
Well finally today the times of India has a front page blurb about this:
http://timesofindia.indiatimes.com/india/Vardah-damages-undersea-cable-internet-slows-down/articleshow/55967405.cms
Now, youtube is loading fine, so google has rerouted traffic, but clearly my VPN must still be routing through Airtels i2i cable to tunnel to me, because the speed is abysmal, or maybe its as you say about youbroadband’s IP transit capacity is still choked, so it speaks to the resiliency of the google GGC that it managed to reroute traffic so efficiently, while my ISP is struggling with other traffic.
It means 4 AM at night. Low traffic during that time.
Interesting article.
Google is terminating traffic in India but it is “fed” from somewhere else. Peering and GCC are probably working well in combination. Do a traceroute to Google and check results. Make sure you are actually hitting the Mumbai node. Check for “bom” in reverse DNS.
Ah 4am got it.
yep, in the traceroute 8th hop is bom05s08-in-f174.1e100.net [216.58.199.174]
This account has some interesting updates, cant say i understand his posts either lol.
https://twitter.com/anurag_bhatia
Here is a undersea cable map that you can actually zoom into, thanks to the folks at TeleGeography.
http://www.submarinecablemap.com/
In the morning, via my VPN got an IP in Thiruvananthapuram, which basically in the same vicinity as Trivandrum so i was probably tunelling through the WARF cable, and basically no speed issues after that.
Haha Anurag is my good friend. Check the trace to see if any traffic is flowing to peering.
>In the morning, via my VPN got an IP in Thiruvananthapuram, which basically in the same vicinity as Trivandrum so i was probably tunelling through the WARF cable, and basically no speed issues after that.
Which network was it from? AS6453(Tata Communications Global ASN) is working fine to Singapore from India. Trace from their looking glass:
traceroute to oneasiahost.com (163.47.176.1), 30 hops max, 52 byte packets
1 if-ae-3-3.tcore2.CXR-Chennai.as6453.net (180.87.36.6) 33.104 ms 33.125 ms 33.262 ms
MPLS Label=541511 CoS=0 TTL=1 S=1
2 if-ae-6-2.tcore2.SVW-Singapore.as6453.net (180.87.37.14) 32.745 ms 33.131 ms 32.989 ms
MPLS Label=304704 CoS=0 TTL=1 S=1
3 if-ae-20-2.tcore1.SVQ-Singapore.as6453.net (180.87.96.21) 33.081 ms 33.067 ms 33.074 ms
4 180.87.97.162 (180.87.97.162) 32.910 ms 32.876 ms 33.003 ms
5 core-1.v3.sg3.sg.gs (203.175.175.69) [AS 24482] 45.511 ms 57.352 ms 130.128 ms
6 core-1.v53.hk1.sg.gs (103.14.244.101) [AS 24482] 78.433 ms 80.172 ms core-1.v52.hk1.sg.gs (103.14.244.99) [AS 24482] 79.847 ms
7 switch-1.v3.hk1.sg.gs (124.6.32.25) [AS 24482] 65.691 ms 64.082 ms 66.521 ms
8 ns1.oneasiahost.com (163.47.176.1) [AS 24482] 65.280 ms 65.509 ms 66.900 ms
Trace from Tata’s Indian ASN 4755:
1 192.168.34.1 0.512ms 0.367ms 0.406ms
2 * * *
3 115.114.137.158 115.114.137.158.STATIC-Pune.vsnl.net.in AS4755 3.38ms 3.212ms 3.679ms
4 172.29.251.33 28.866ms 28.957ms 35.885ms
5 116.0.67.121 ix-ae-6-1331.tcore1.HK2-Hong-Kong.as6453.net AS6453 95.492ms 92.92ms 92.863ms
6 123.255.90.102 sggs2-10G.hkix.net 97.018ms 96.9ms 96.953ms
7 124.6.32.25 switch-1.v3.hk1.sg.gs AS24482 97.133ms 96.97ms 97.009ms
8 163.47.176.1 ns1.oneasiahost.com AS24482 93.525ms 93.506ms 93.529ms
So it looks like a i2i cable problem. If you are on a Tata connection, you should be much better off as far as connectivity to Singapore is concerned.
Hi Varun, this should work.
http://pastebin.com/jc4BdeS5
If you have any proper formatting tips to paste tracert messages please let me know.
Oops! That was just Askimet marking your post as spam for some reason. Sorry about that. It’s probably due to all the hosts in the trace. Trace to Google is going to Delhi. The second website is hosted on Akamai CDN so doubt that is accurate. Try http://www.oneasiahost.com instead.
What a coincidence about Anurag, its a small world!
Ahh, now that you mention Akamai, i see it at the last hop, of course….! i mean, its one of the largest CDN’s in the world…
WITH A VPN with Indian IP:
Tracing route to oneasiahost.com [163.47.176.1]
over a maximum of 30 hops:
1 738 ms 727 ms 870 ms 10.28.10.1
2 633 ms 702 ms 683 ms hostdime.IL-core.dimenoc.com [109.73.161.1]
3 694 ms 678 ms 708 ms 14.141.116.21.static-Delhi.vsnl.net.in [14.141.116.21]
4 747 ms 1180 ms 1491 ms 172.31.17.5
5 628 ms 637 ms 655 ms ix-ae-6-1331.tcore1.HK2-Hong-Kong.as6453.net [116.0.67.121]
6 673 ms 638 ms 700 ms sggs2-10G.hkix.net [123.255.90.102]
7 713 ms 803 ms 855 ms switch-1.v3.hk1.sg.gs [124.6.32.25]
8 895 ms 886 ms 891 ms ns1.oneasiahost.com [163.47.176.1]
WITHOUT A VPN:
Tracing route to oneasiahost.com [163.47.176.1]
over a maximum of 30 hops:
1 1 ms <1 ms <1 ms ralink.dlink.com [192.168.0.1]
2 23 ms 12 ms 12 ms 100.68.0.1
3 16 ms 13 ms 18 ms Pune-Core01.youbroadband.in [203.187.215.1]
4 * * * Request timed out.
5 18 ms 19 ms 22 ms 172.31.183.253
6 40 ms 39 ms 56 ms 172.17.169.202
7 184 ms * 183 ms ix-ae-6-1331.tcore1.HK2-Hong-Kong.as6453.net [116.0.67.121]
8 199 ms 227 ms 195 ms sggs2-10G.hkix.net [123.255.90.102]
9 226 ms 203 ms 197 ms switch-1.v3.hk1.sg.gs [124.6.32.25]
10 188 ms 185 ms * ns1.oneasiahost.com [163.47.176.1]
11 186 ms 186 ms 209 ms ns1.oneasiahost.com [163.47.176.1]
Trace complete.
With a VPN with EU IP:
Tracing route to oneasiahost.com [163.47.176.1]
over a maximum of 30 hops:
1 358 ms 364 ms 360 ms 10.57.10.1
2 361 ms 364 ms 362 ms 5.153.233.49
3 * 362 ms 357 ms 5.153.234.253
4 369 ms 362 ms 364 ms 5.157.4.9
5 375 ms 364 ms 362 ms s-b5-link.telia.net [62.115.145.254]
6 394 ms 389 ms 393 ms vsnl-ic-124239-s-b1.c.telia.net [213.248.104.162]
7 393 ms 474 ms 1164 ms if-ge-7-1-0.core2.OS1-Oslo.as6453.net [80.231.89.33]
8 * 394 ms 398 ms if-xe-8-3-1-0.tcore2.AV2-Amsterdam.as6453.net [80.231.152.41]
9 397 ms 398 ms 401 ms if-ae-8-2.tcore2.L78-London.as6453.net [80.231.131.5]
10 394 ms 393 ms 394 ms if-ae-2-2.tcore1.L78-London.as6453.net [80.231.131.2]
11 398 ms 397 ms 503 ms if-ae-17-2.tcore1.LDN-London.as6453.net [80.231.130.130]
12 * 399 ms 395 ms 195.219.83.2
13 395 ms 399 ms * switch-1.v3.uk1.sg.gs [124.6.36.25]
14 * 399 ms 397 ms ns1.oneasiahost.com [163.47.176.1]
If you were to explain/diagnose these results, what would you say?
Those traceroutes look fine. For the second one without a VPN, traffic is moving through AS6453 Tata Communications(Global) to Hong Kong and then to Singapore. Reverse path may be very different. The VPN is adding in a lot of latency, even for the Indian one. Trace through that looks similar.
It’s getting a little cluttered in here. 😛 Would be better to talk directly via email.
What a coincidence about Anurag, its a small world!
Ahh, now that you mention Akamai, i see it at the last hop, of course….! i mean, its one of the largest CDN’s in the world…
Since the Askimet plugin refuses to learn fast enough lol, here is the pastebin link of the 3 tracertroutes.
http://pastebin.com/CDPwMBpe
If you were to explain/diagnose these results, what would you say?
Hi Varun, was preoccupied for a couple of days, but today it seems to me the situation is improved, i dont know if the i2i cable is finally fixed by Airtel, but these results look better dont they?
Tracing route to oneasiahost.com [163.47.176.1]
over a maximum of 30 hops:
1 1 ms <1 ms 1 ms ralink.dlink.com [192.168.0.1]
2 12 ms 10 ms 11 ms 100.68.0.1
3 13 ms 19 ms 13 ms Pune-Core01.youbroadband.in [203.187.215.1]
4 * * * Request timed out.
5 21 ms 11 ms 10 ms 172.31.183.253
6 39 ms 34 ms 40 ms 172.17.169.202
7 133 ms 116 ms 137 ms ix-ae-6-1331.tcore1.HK2-Hong-Kong.as6453.net [116.0.67.121]
8 114 ms 112 ms 132 ms sggs2-10G.hkix.net [123.255.90.102]
9 110 ms 121 ms 125 ms switch-1.v3.hk1.sg.gs [124.6.32.25]
10 124 ms 138 ms 179 ms ns1.oneasiahost.com [163.47.176.1]
That looks OK but I don’t know why it is flowing through Hong Kong for Tata. Connectivity for Airtel is still going through UK for me.
This article was very much informative so thanks for that and am from Delhi, i work as an seo specialist. Even i faced the issue even here and even i received the sms of regretting for inconvenience from my broadband company.
Yet i can see some traffic in my website, so am happy halfly !! 😀
So, as i play Online games, i have been getting high ping to singapore servers since yesterday, getting around 240 ms to SG. So , i know it will be fixed in some time probably, but is there any temporary solution to fix the problem , i tried changing DNS but it barely affects my latency. Is there any other DNS i can try connecting to that will help resolve my issue.
Hi Tanveer,
DNS will not help you here. A lot of Singapore traffic is routed via Europe because of the i2i cable issues. You can try an Indian or an European server for better latency or just wait a few hours/days for it to be fixed.
Well, that’s a bummer, Thank you for replying and helping though. Yes my ping is normal in Indian and european servers, but most of the games have Singapore server (which is the closest in asia). Looks like i have to wait some hours/days to get it fixed.
Pingback: Very Severe Cyclone Vardah : Slows down Internet Connection in India - GBHackers
Nice Stuff..
My ping/latency is all recked can’t play my fav online game how long is this expected to go?
Try an European or an Indian server. I don’t have any idea. Maybe a few more days, depending on the severity.
well my ping is back to normal with airtel but problem is just after 5:00 in evening problem comes back and my lag in game starts
how can i change server?
It still flows through UK I’m afraid. Checked from Airtel’s looking glass GPX1 location. Nothing you can do other then to choose a different server in your game. Changing the server would depend upon which game you are playing.
Appreciate your effort in putting the info out into public domain. Keep rocking.
So, whats the status now? Google seems to be fine here but the upstream bandwidth still seems to be on low capacity.
Are you having the same problem too ?
Sorry for the delayed response. Was a little busy.
Traceroutes look fine but I’m not sure if the i2i cable is being used. My friend was complaining of quality issues on a Whatsapp call during peak time(Whatsapp calls connect to Singapore to proxy traffic between devices). So it could be because some other cable is being used and there is choking on that during peak.
I play Overwatch connecting to the Singapore server and since Jan 4,it has been high ping. This might be the reason. http://www.thedailystar.net/bytes/internet-problems-submarine-cable-glitch-1340770
Read up. They said the maintenance will be done by Jan 20.
Thanks for letting me know.
We’ll have to wait and see if it indeed does get fixed by then.
Well,yeah the promised date is Jan 20 but a friend of mine from Vietnam said that wire glitches usually take some time to stabilize so I’d at least give it a month. Let’s hope for the best because I really want to return to Overwatch as soon as possible. Rusty af.
I play an online game whose servers are located in Singapore, it was fine after the cyclone, but on January my ping increased from 80ms to 300+ ms. It’s nearly impossible to play that game with such high ping. I found from traceroute that it was going from Chennai-Mumbai-London-Singapore. Can you tell me a solution to this issue without changing my ISP (Alliance Broadband)?
Changing ISPs will not fix it. You will have to wait it out till the damaged submarine cables are fixed unfortunately.
Well it is the 8th now. Here is to hoping the work has started.
Hey! Any update on when i2i cables will be fixed and latency to Singapore servers will be back to normal ?