b>j)΄!Pԫ&;"kB޶}pSVT(wę!j x;-m@JnQ+պכ7MajfJͱ4jѲ撆RxZMz7vIW/dٞТזcZM~ji ߒsQzԠDW3Den"M+/B:-uIJ7j委9p='mANޭ=/B:-n&nUfqxZM~c Ϲ+,&ᾺܢF[(1*" ϒ"Jԧ<;b" "jܢF[x ,!q қ*]/؝27SMcs"ޭDQ/应ܢF_! :s" 7`F+SVTn"IJnQ/应B 4 wD"IJ׭-`S9DrjiEJ߅gJ应矁[xZM~n"IB؃!'Тѕ+(mIKʭ/|ϐܢF[xZMzG %嬩/c[[ india – Varun Sinai Priolkar https://varunpriolkar.com My Space on the Internet Sun, 04 Jan 2026 20:52:13 +0000 en-GB hourly 1 https://wordpress.org/?v=7.0 https://varunpriolkar.com/wp-content/uploads/2016/10/cropped-cartoon-blowfish-512-263078-1-150x150.png india – Varun Sinai Priolkar https://varunpriolkar.com 32 32 117415536 How to buy US Equities from India https://varunpriolkar.com/2019/01/how-to-buy-us-equities-from-india/ https://varunpriolkar.com/2019/01/how-to-buy-us-equities-from-india/#comments Sat, 26 Jan 2019 18:14:48 +0000 https://varunpriolkar.com/?p=2778 Read More »]]>

Disclaimer: This post is for information purposes only and not an investment advise. Please consult your investment advisor before doing anything. I am not liable if you lose money or if anything in this post is not factual.

So you’re at a stage where you hold a decent chunk of Indian equities, but would like to diversify your holdings to overseas companies. Everytime you use your phone or shop online, you are making money for non-US companies. There is a substantial part of value globally that is not captured by Indian companies. As such it might be a good idea to diversify your holdings globally.

As a resident Indian citizen you have many options –

  1. Open an overseas trading account offered by Indian brokerages.
  2. Open an account directly with an overseas brokerage firm offering access to US markets.
  3. Invest in a domestic mutual fund holding US equities.

I’ll be talking about each one of these options in detail. You can invest upto 250,000 USD every year overseas. You cannot buy/selling anything on margin, deal with Forex or buy Foreign Currency Convertible Bonds(FCCBs) from Indian companies. The full list of what is allowed and prohibited is available on RBI’s website.

Option offered by Indian brokerages

Kotak Securities, ICICI Direct, India Infoline, Reliance Money and Religare all offer an option of access to overseas securities. However I found that the local ICICI Bank branch had no knowledge about the same and they made no effort to help me out. I also learnt from someone that they charge very high fees.

So I didn’t try this option out, but it’s a good option to have.

Directly from foreign brokerages

I decided to go for TD Ameritrade due to their low costs. Source: Investopedia

There are a few foreign/US brokerages that allow you to buy/sell equities. The prominent ones are:

  1. Saxo bank
  2. Charles Schwab
  3. Interactive Brokers
  4. TD Ameritrade

While some of these have dedicated teams to help you with an application from India, others do not. I chose TD Ameritrade as they had the lowest fees for the volume that I would be using. The per trade transaction cost is around 6.95 USD flat. The full price list for TD Ameritrade is present here. The list of documents I had to give digitally include:

  1. Filling out their form online , printing it and signing it.
  2. Aadhaar card and driving license for ID.
  3. Passport copy.
  4. Bank statements.
  5. W8-BEN form for tax declaration.

It took around 2 weeks for them to verify everything, although that could be because I had submitted everything during the winter holidays. While they do tell you to physically send the documents or fax them, they do accept documents over email too. You have to mail them to international@tdameritrade.com.

Taxation

India has a Double Taxation Avoidance Agreement(DTAA) with the US, which you can find on the Indian income tax website. In short for a non-resident alien:

  1. Capital gains will be taxed in India like debt. Taxes will not be withheld in the US by the broker.
  2. Dividends will be taxed at 25% in the US, which will be income from other sources. You can use deduction for these in India while filing your tax returns.

You may or may not need an International Tax Identification Number(ITIN) from US Internal Revenue Service(IRS). Authorised agents in India charge around ₹8000. I figured I didn’t need it and applied with just my Indian Permanent Account Number(PAN).

My W8-BEN form looked like this

Most brokers should have an international accounts opening team to help you with taxation related queries and how to fill out the requisite forms.

Transferring money

While TD Ameritrade offered many options for transferring money, only Wire Transfer seemed like a practical solution, which most banks offer without questions asked. I found the transfer costs and the spreads for currency exchange to be excessive for ICICI Bank. Do let me know if you know an another option.

International wire transfer form should be filled out something like this

For the transfer you’ll have to fill out the form from the bank, which includes reasons for transfer, if you want like a regular/guaranteed transfer(regular transfer means the receiving bank could deduct some amount for fees, which was 6 USD on a 400 USD transfer for me). The brokerage will give you an account where you can transfer money to. For TD Ameritrade I had to add the For Benefit Of(FBO) number on the receiver’s details(next to the account holder name) and in additional information.

If everything goes well the amount should be reflected in your brokerage checking account

If you filled out everything correctly, you should see the amount in the brokerage account in a few days. It took nearly 5 days for me. It would have been nice if they had an Indian bank account to make the transfer fees lower.

Invest in Mutual Funds holding US equities

This is a relatively easy option, wherein you can buy Mutual Funds in India which hold US equities. Some of these are:

  1. Frankin India Feeder – US Opportunities Fund
  2. DSP US Flexible Equity Fund
  3. ICICI Prudential US Bluechip Equity Fund
  4. PPFAS Long Term Equity Fund  – This fund has a part of it’s portfolio in international stocks but has > 65% of portfolio in Indian equities for getting equities treatment for taxation. It is one of my favorite funds.

If holding of Indian equities is < 65%, these the fund will be taxed like debt. Some of these funds also hedge for currency risks.

Thanks for reading. Do subscribe to my blog if you like the content. I will be covering more finance related topics in the future(along with the usual tech content). 🙂

]]> https://varunpriolkar.com/2019/01/how-to-buy-us-equities-from-india/feed/ 75 2778 Getting a Poland Schengen Visa in India https://varunpriolkar.com/2018/08/getting-a-poland-schengen-visa-in-india/ https://varunpriolkar.com/2018/08/getting-a-poland-schengen-visa-in-india/#respond Wed, 22 Aug 2018 12:30:23 +0000 https://varunpriolkar.com/?p=2215 Read More »]]>

I had to briefly fly to Europe for some purpose so had to get my Schengen Visa done. The process is not as straight forward as I had hoped for it to be and information is often scarce on the internet. I will be posting all resources that helped me

What Visa to apply for?

A Schengen Visa will give you access to all of the Schengen countries, however you should get the Visa using the following criteria:

  • Country where you will be spending the most number of days in, regardless of the port of entry.
  • If you are spending the most amount of days in equivalent in two or more countries, apply to the country of port of entry.

Both of these guidelines are strictly adhered to, both during the Visa process and at the port of entry. There have been cases of people being sent back if you can’t demonstrate this criteria at the port of entry so be careful!

After you know which Visa to apply for, you need to check how to apply for the Visa of the country that you are planning to visit. Many countries use VFS Global to process documents. Poland does not do this for India.

There are three type of Schengen Visas offered:

  • “C” type(Schengen) Visa – which is for a stay up to 90 days within 180 days in the Schengen area.
  • “D” type(National) Visa – Long term visa for a stay over 90 and up to 365 days in Poland. Please note that the national visa allows to stay in other Schengen countries for up to 90 days within the period of 180 days.
  • “A” type transit Visa.

You have to apply for the appropriate one accordingly. Usually “D” type Visas are for things like working, studying etc. in the destination country, while “C” type Visas are for temporary visits.

How to apply

You first have to go to the e-konsulate website, select your language on top right as English, select your country(India) and the consulate you belong to.

  • Mumbai Consulate – Maharashtra, Goa, Kerala, Karnataka, Andhra Pradesh, Tamil Nadu, Gujarat, Daman, Diu ,Pondicherry, Telangana, Andaman and Nicobar islands

Click the Schengen Visa – Register form option and book the appointment after filling out the form. Many people have trouble getting the appointment and travel dates may be approaching. In that case please contact the Embassy/Consulate via Email/phone and they may help you out.

I was told that slots for the Mumbai consulate open on every Monday at 9-10 AM. I however managed to get my appointment at 3 AM on a Friday night. Keep checking every half an hour or so throughout the day. Someone may have cancelled his appointment and you may get it. Using agents is frowned upon. Most people online say that agents are not legitimate. I don’t know anything to comment on this.

Submit the documents with all the documents they have asked for on their website on the appointment date. The following may be a little problematic.

  • Airplane tickets – They ask for a valid reservation without actually paying for the ticket. Yatra provides this.
  • Hotel reservation – They ask you to show bookings for a hotel room. I already had this figured out. Expedia has an option wherein you can book hotels without paying for them until you end up there and it gives you an option to cancel at the last moment.
  • Travel insurance – United India Insurance provides this for a cheap cost online. It was valid for Mumbai at the time of writing this.

I also had to go a day early to do the payment at a different location specified. I was applying from Mumbai consulate.

What if you don’t have a passport?

No need to worry! Goto the Passport Seva website, create account, fill in details and book an appointment on the closest day possible with the Tatkal option. I booked an appointment on the next day and got the passport delivered by Speed Post within 24 hours.

It cost me ₹ 3500 as I used the Tatkal option. The police verification will be done in a few days, wherein you’ll be called to the police station and a few documents will be collected from you. However this is not needed to travel. However make sure you get it done before a month or so. The whole process is very streamlined and there is much to appreciate.

Interview and passport collection

Immediately after submitting the documents, they’ll call you for an interview. If you are going for a legitimate purpose you have nothing to fear. The interviewer quickly asked me about me, what I do, why I am going to Poland, cross questioned me a little to make sure I am not lying, noted something down on the paper and that was the interview done. Took roughly 2-3 minutes. The interviewer was something like this.

Interviewer: What is the purpose of your visit?

Me: …

Interviewer: Tell me more about your sponsoring company?

Me: ...

Interviewer: XYZ seems strange. Can you tell me more about it?

Me: …

Interviewer: Do you intend to go back? Would you be applying for a permanent Visa for Poland later.

Me: Yes. No, this is a solitary visit.

Interviewer: What do you work as?

Me: …

Interviewer: Have you to Europe before?

Me: …

Me: Would it be possible for you to give me a yes or a no today?

Interviewer: Unfortunately we are a little short on resources and have to give priority to people travelling on the same week.

Me: No problem. Thank you.

Interviewer: Thank you.

The Embassy/Consulate can take upto two weeks to take a decision and they do not courier passports. You will have to check the decision status on the Embassy/Consulate website and collect the passport when it personally or through someone you know after it shows due for collection.

While I have heard of cases wherein people have got their Visa on a single day, it took three working days for mine and I got it collected through a friend. You will need extra documents if you are getting someone else to collect it.

I got a 30 day, single entry Visa as requested. Getting a Schengen Visa is a lot of trouble for what you get. Next time you want to travel, you need to do the same process all over again. If your Visa is rejected, you have to re-apply with the same fees again and there is no guarantee of an approval. The cost of applying for a Visa and re-consideration of application on rejection is ₹ 4900 each.

Resources

I found a few resources on the internet which were useful for me.

  • Indian Community of Poland Facebook Group.

PS: Special thanks to my friend Rohan Chaubal for collecting my passport.

]]> https://varunpriolkar.com/2018/08/getting-a-poland-schengen-visa-in-india/feed/ 0 2215 Issues plaguing NIXI internet exchange in India https://varunpriolkar.com/2018/05/issues-plaguing-nixi-internet-exchange-in-india/ https://varunpriolkar.com/2018/05/issues-plaguing-nixi-internet-exchange-in-india/#comments Wed, 23 May 2018 16:54:50 +0000 https://varunpriolkar.com/?p=1115 Read More »]]>

Before proceeding I want to make it absolutely clear that these are my personal views only and nothing to do with my employer or their views.

National Internet Exchange of India(NIXI) is an internet exchange in India, infact one of the few in the country and is fully government owned non-profit entity initially established to provide a neutral Internet Exchange in India. It has come leaps and bounds with it now exchanging more than 100 Gbps of data. Good internet exchanges are vital to the internet infrastructure in the country and helps to keep the traffic local. However there are a few issues which it needs to overcome.

Traffic exchange charges

You can have a look at NIXI tariffs here. While the Currently peered parties have to shell out ₹1/GB for every excess of downstream data that they consume through the exchange as compared to the upstream data that they end up pushing, unless you push 5 times more upload as compared to download. That is then paid to the parties pushing out the data. Assuming charge of ₹/GB and ratio of 5:1 of download to upload, the yearly bill for a 1 Gbps fully utilised circuit(assuming avg. down 1 Gbps, average upload 200 Mbps) will cost ₹322 cr. ($47438217), which is exponentially higher the cost of transit in India. Doesn’t make any sense for an eyeball ISP to connect! They often try to compensate by announcing lesser prefixes to balance traffic.

This has resulted in parties announcing partial routes, while receiving everything. Also since the rates are determined on RX/TX on the L2 switch, the smaller providers can just do a UDP flood to balance traffic.

Keeping flat port charges and allowing parties to exchange traffic for free is a much better solution. I don’t know if this is an arangement to keep the larger transit providers like Tata, Airtel happy.

The requirement of an ISP license

A lot of content providers who want to peer with eyeball networks in India like Google, Microsoft, Amazon etc. simply can’t because of this. Getting an ISP license is not an option for them since getting one is a bureaucratic mess, compliance with DOT regulations and the % of AGR fee charged an even bigger problem. They don’t need one for their operations.

An exchange without content players is much worse a proposition.

Broken L1 connectivity and down members

We can check how many of the connected networks at NIXI are working from the  NIXI looking glass.

  • Routers which are pinging: 88
  • Routers which are down: 36
  • Total routers: 124
  • Routers up percentage: 70.96%

It was good to note that NIXI no longer announces the subnet used for peering globally and keeps it local.

There is also an issue of choking of links. These are IPs of connected AS9498(Bharti Airtel Ltd) and AS45528(Tikona Digital Networks Pvt Ltd) routers graphed as an eg. from Delhi(Noida) location. AS55644(Idea Cellular Limited) and AS9829(Bharat Sanchar Nigam Ltd) routers on the same subnet graphed absolutely fine, while some others were far worse than this.

Ping to 218.100.48.34(Tikona) router from Noida

Ping to 218.100.48.20(Tikona) router from Noida

From Tikona’s graph you can make out that the issue is of choking of downstream traffic during peak hours. For Airtel it was more uniform. The reason why nobody wants to upgrade capacities is because:

  1. Larger ISPs like Airtel/Tata – Better connectivity at exchanges affects their IP transit business. They have a large number of end customers directly connected and make a lot of large chunk of revenue through selling connectivity to them.
  2. Eyeball ISPs – Higher port charges, higher costs due to traffic exchange charges(requestor pays logic). It is cheaper to remove announcements and try to balance upload and download than to upgrade the link. For many ISPs it is very small part of their capacity. It’s running because nobody has cared to remove it.

I have tried to get these issues fixed in my personal capacity but mostly I’m asked to send them traffic via transit or they reduce prefixes announced at NIXI.

Datacenter choice and newer locations

The datacenters where NIXI is present are:

  • Mumbai – Cyquator Technologies Pvt. Ltd.
  • Delhi(Noida) – Netmagic IT Services Private Limited
  • Chennai – Sify Technologies Limited
  • Bangalore – Sify Technologies Limited
  • Hyderabad – CtrlS Datacenters Ltd
  • Kolkata – STPI
  • Ahmedabad – (n) Code Solutions -A Division of GNFC Limited
  • Guwahati – Assam Electronics Development Corporation Ltd

As you can see, the choice of datacenter in a lot of cases is not stellar, where a lot of networks are already present and don’t need to build out separately just for connecting to NIXI.

In 2008, four more locations were added to NIXI mostly from non-internet hub cities. The older locations have done fairly well in terms of traffic, but the newer locations have suffered, largely due to no large player or any content player being present. There are mostly local eyeball ISPs present there. Most of the traffic between them is probably peer to peer or gaming traffic. At the current traffic levels, it doesn’t make sense for anyone to connect there.

Traffic rate through newer locations is pretty bad

As it stands, NIXI is working on expanding to Lucknow and Mohali.

Miscellaneous

  • Lack of bilateral peering option

There is no option for participating networks to bilaterally peer with each other. Providing just the L2 switching infrastructure can solve the problem of costly interconnects for a lot of networks hence allowing to connect to more parties.

  • Route leaks (largely fixed)

As NIXI is an exchange only, nobody should ideally be announcing routes learnt through AS24029(NIXI is an IXP in India) through their transits into the global routing table. However due to incorrect filter application, the routes can end up on the global BGP routing table.

route-views>sh ip bgp regexp _24029_
...                                                                                                                                                                                                                                                                                                                             
     Network          Next Hop            Metric LocPrf Weight Path                                                                                                                                                                                                                                                          
 *   43.224.159.0/24  140.192.8.16                           0 54728 6939 15412 55644 24029 4755 132556 i                                                                                                                                                                                                                    
 *                    207.172.6.20            86             0 6079 15412 55644 24029 4755 132556 i                                                                                                                                                                                                                          
 *                    132.198.255.253                        0 1351 10578 11164 15412 55644 24029 4755 132556 i                                                                                                                                                                                                              
 *                    162.250.137.254                        0 4901 6079 15412 55644 24029 4755 132556 i                                                                                                                                                                                                                     
 *                    207.172.6.1              0             0 6079 15412 55644 24029 4755 132556 i                                                                                                                                                                                                                          
 *                    209.124.176.223                        0 101 101 11164 15412 55644 24029 4755 132556 i                                                                                                                                                                                                                 
 *                    129.250.1.66         17635             0 2914 15412 55644 24029 4755 132556 i                                                                                                                                                                                                                          
 *                    64.71.137.241                          0 6939 15412 55644 24029 4755 132556 i                                                                                                                                                                                                                          
 *                    103.247.3.45                           0 58511 6939 15412 55644 24029 4755 132556 i                                                                                                                                                                                                                    
 *                    91.218.184.60            0             0 49788 12552 6939 15412 55644 24029 4755 132556 i                                                                                                                                                                                                              
 *   116.212.177.0/24 140.192.8.16                           0 54728 6939 9583 24029 4755 55839 i                                                                                                                                                                                                                            
 *                    64.71.137.241                          0 6939 9583 24029 4755 55839 i                                                                                                                                                                                                                                  
 *                    132.198.255.253                        0 1351 6939 9583 24029 4755 55839 i                                                                                                                                                                                                                             
 *   202.62.116.0     140.192.8.16                           0 54728 6939 9583 24029 4755 9941 i                                                                                                                                                                                                                             
 *                    129.250.1.66         18628             0 2914 9583 24029 4755 9941 i                                                                                                                                                                                                                                   
 *                    132.198.255.253                        0 1351 6939 9583 24029 4755 9941 i                                                                                                                                                                                                                              
 *                    64.71.137.241                          0 6939 9583 24029 4755 9941 i                                                                                                                                                                                                                                   
 *   202.122.134.0    140.192.8.16                           0 54728 6939 9583 24029 4755 4755 4755 38615 i                                                                                                                                                                                                                  
 *                    129.250.1.66         18628             0 2914 9583 24029 4755 4755 4755 38615 ?                                                                                                                                                                                                                        
 *                    64.71.137.241                          0 6939 9583 24029 4755 4755 4755 38615 i                                                                                                                                                                                                                        
 *                    132.198.255.253                        0 1351 6939 9583 24029 4755 4755 4755 38615 i                                                                                                                                                                                                                   
 *                    114.31.199.1                           0 4826 9583 24029 4755 4755 4755 38615 i                                                                                                                                                                                                                        
 *   223.31.122.0     194.85.40.15             0             0 3267 43531 9583 24029 18101 18101 18101 132215 ?                                                                                                                                                                                                              
 *                    195.208.112.161                        0 3277 3267 43531 9583 24029 18101 18101 18101 132215 ?

AS9583(Sify Ltd) and AS55644(Idea Cellular Limited) are still leaking routes learnt from peers at NIXI onto the global routing table. This was an issue a major issue with many of the transit players connected to NIXI. The filters would be applied on the customer ASN and not from where the route was learnt. This is now fixed to a large extent.

  • Partial sharing of routes

Due to requestor pays logic and unwillingness of networks to carry traffic over their backbones, they end up sharing partial routes at NIXI, while accepting all routes. This has resulted in issues for root servers connected there and don’t have transit, ending up blackholing traffic. K-root server ended up removing their root servers from NIXI largely because of this.

The silver lining is that there is decent competition that has now come up due to these issues with NIXI in Extreme IX, Mumbai IX and Bharat IX.

]]> https://varunpriolkar.com/2018/05/issues-plaguing-nixi-internet-exchange-in-india/feed/ 1 1115 Lifting after a disc prolapse and SANOG https://varunpriolkar.com/2017/07/lifting-after-a-disc-prolapse-and-sanog/ https://varunpriolkar.com/2017/07/lifting-after-a-disc-prolapse-and-sanog/#respond Sun, 09 Jul 2017 12:13:00 +0000 https://varunpriolkar.com/?p=1247 Read More »]]> As many of those close to me know, I had a bad disc prolapse in my lower back about 1.5 years back. The situation was so bad that I couldn’t walk for a few months, with pain shooting down to my legs. After that I was very out of shape, with a lot of weight gain because even jogging/running would cause the pain to come back. After showing it to an orthopedist, it was diagnosed as a disc prolapse(herniated disc); most probably caused due to lack of strength in my abdominal muscles not being able to support the heavy load on my lower back.

Starting out

Well three months back I started with cardio, started doing abdominal muscles strengthening exercises and slowly started with weight training. I started out with splits and then transitioned over to the excellent Starting Strength program, with accessories added in mostly for the abdominal muscles and substituted the power clean for rows. However I couldn’t continue to ignore the problem which I had in the past and decided to go in for a MRI a couple of weeks back.

My MRI result

The MRI results were very positive. I however am keen to follow up with physiotherapy to increase the flexibility in my lower back. Yoga helps from what people have told me too.

Progress

Right now my lifts are as follows for 3 sets, >5 reps at 100 kg BW:

 Lift  Weight
Bench  60 kg
Squat  70 kg
Deadlift  110 kg
OHP  45 kg

I know that isn’t very impressive but I am happy with the start. The diet needs sorting out, but I barely get much time, trying to balance work and lifting and I am struggling a bit with lower back flexibility and I need to go slow and careful since I absolutely do not want to tweak anything in my lower back again.

I am also trying out high bar squats and front squats for better balance. They seem to be helping me a lot. Trying to do more cardio as well but I barely get time for it.

E2E Networks sending me to SANOG 30!! 😀

SANOG 30 – One of the only decent internet/network related events in India

So I will be at SANOG 30 on all days from 10th July to 18th July which will be held in Gurgaon, which is about an hour away from where I live – Faridabad. I hope to see you there 🙂

I’m sorry if I have still not replied to your mail. I will try to do so at the earliest once I get some time. Anyways gotta run..

]]> https://varunpriolkar.com/2017/07/lifting-after-a-disc-prolapse-and-sanog/feed/ 0 1247 How to protect against DDOS attacks in India https://varunpriolkar.com/2017/05/how-to-protect-against-ddos-attacks-in-india/ https://varunpriolkar.com/2017/05/how-to-protect-against-ddos-attacks-in-india/#comments Sun, 28 May 2017 19:04:40 +0000 https://varunpriolkar.com/?p=1218 Read More »]]>

Whether you may be a large web host, or a small ISP serving a neighborhood; DDOS attacks in India can be extremely problematic and hard hitting. Here are some of the ways you can ensure that you aren’t affected by a lot. All of these solutions assume that you are running BGP.

Preparing for an attack

Do either one of the following things:

  1. DDOS protection from transit provider – Get DDOS protected IP transit from Tata Communications Ltd.(AS4755). You can use it with other transit providers. This may be expensive by about 30% over normal IP transit but will save your business when you get an attack.When they detect an attack, they will originate the attacked IP pool at all of their edge locations with Arbor filtering devices and run a GRE tunnel from there. Tata has enough capacity to bear 300-400 Gbps of attack. They also have a Chennai location for traffic for attacks originating inside India. Other transit players also offer similar services, but do keep in mind that they may not have the capacities to bear a huge attack. If you frequently deal with attacks, then announcing more specific on DDOS protected upstream(s) can be a good idea.
  2. Scrubbing solution providers – A cheaper option is to start originating your IP pool during an attack with one of providers offering scrubbing services outside India where transit is cheap and run a GRE tunnel from there. This can often be cheaper. Using BGP communities can help.

Other general tricks

  1. Null routing with BGP communities – Ask your transit player for support for null routing with BGP communities. After that if you want to blackhole traffic for even a /32 from the source network itself, you can do so by tagging the announcement with [upstreamASN]:666 community. This is great if the attack is on a single customer and it is costing way more for you to bear the attack than you get from the customer.
  2. Stop announcing your pool – If you are a retail ISP and everything else fails, then you can simply NAT all your public IPs behind the IP on your side from the /30 provided to you from the transit provider. Stop the announcements. Attacks will stop.

As always, automate any solution that you will be going for. You can also try visiting the nearest police station. 😛

I hope that was useful. I am sorry about the lack of posts on this blog. I don’t get enough spare time anymore. 🙁

]]> https://varunpriolkar.com/2017/05/how-to-protect-against-ddos-attacks-in-india/feed/ 4 1218 How to fake pings and traceroutes https://varunpriolkar.com/2017/04/how-to-fake-pings-and-traceroutes/ https://varunpriolkar.com/2017/04/how-to-fake-pings-and-traceroutes/#respond Thu, 20 Apr 2017 22:18:49 +0000 https://varunpriolkar.com/?p=1142 Read More »]]> I recently read Srijit Banerjee’s excellent article, which brought to light ugly things done by service providers to fool their customers who have grown accustomed to the ping google.com command to judge the quality of the internet.

While he has strayed away from how this is done, I feel that educating people on how this can be done and how to ensure that they are not a victim is much more important.

My setup

My setup for testing looked something like this. Everything was virtualised with Virtualbox. For the router I picked up the CHR(Cloud Hosted Router) images from Mikrotik’s website.

My test setup on Virtualbox

Connections made from 192.168.100.3 through to the internet through the default gateway 192.168.100.2 is also source NATed. The other connection between the Mikrotik CHR VM and the host is also source NATed by Virtualbox.

Let’s do a ping from VM 1 to 8.8.8.8:

arhue@ubuntu:~$ ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=313 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=79.9 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=63.1 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 63.133/152.072/313.158/114.111 ms

Very normal for a tethered 4G connection. All of this faking of pings relies on one simple concept:

Destination NAT (Port forwarding)

Illustration showing how destination NAT works

This essentially allows you to map a single IP or a set of IPs to a single IP or a set of IPs by matching transport layer parameters like ports, protocol used for transimission etc. Traffic can also be directed to a single port. The destination address and port is rewritten to by the router. The same is done for the source packet on return. Think of it as opposite of source NAT. So essentially just matching every IP on the Google’s address list and destination NATing it to a single IP will allow for this. Care has to be taken to do this for only for things you want up ending up at the new server. 

Taking it for a spin

For now let’s do it only for ICMP so that only ICMP traffic ends up at 192.168.200.2.

Destination NATing on my Mikrotik VM

Ping from VM 1 to 8.8.8.8:

arhue@ubuntu:~$ ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=63 time=1.89 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=63 time=1.98 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=63 time=2.14 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2014ms
rtt min/avg/max/mdev = 1.896/2.009/2.149/0.105 ms

Traceroute from VM 1 to 8.8.8.8 with ICMP:

arhue@ubuntu:~$ sudo traceroute --icmp 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  google-public-dns-a.google.com (8.8.8.8)  0.627 ms  0.530 ms  0.638 ms
 2  google-public-dns-a.google.com (8.8.8.8)  3.570 ms  3.520 ms  3.451 ms

Tcpdump running on VM 2 with two pings sent/received from VM 1:

arhue@ubuntu-spoofer:~$ sudo tcpdump -i enp0s8 icmp
[sudo] password for arhue: 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s8, link-type EN10MB (Ethernet), capture size 262144 bytes
02:25:50.197457 IP 192.168.100.3 > 192.168.200.2: ICMP echo request, id 1633, seq 1, length 64
02:25:50.197518 IP 192.168.200.2 > 192.168.100.3: ICMP echo reply, id 1633, seq 1, length 64
02:25:51.201668 IP 192.168.100.3 > 192.168.200.2: ICMP echo request, id 1633, seq 2, length 64
02:25:51.201732 IP 192.168.200.2 > 192.168.100.3: ICMP echo reply, id 1633, seq 2, length 64

Have a look at the SRC/DEST IP. There is something interesting going on here. The destination IP is being changed by the router for the ICMP packets to 8.8.8.8 to 192.168.200.2 and SRC IP back to 8.8.8.8 from 192.168.200.2 on return. Tcpdump makes this very clear.

UDP traceroutes

Traceroute from VM 1 to 8.8.8.8 with UDP:

arhue@varun-laptop:~$ sudo traceroute --udp 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  192.168.100.2 (192.168.100.2)  0.883 ms  0.834 ms  0.630 ms
 2  192.168.50.1 (192.168.50.1)  1.139 ms  1.492 ms  1.591 ms
 3  * * *
 4  * * *
 5  10.206.233.129 (10.206.233.129)  269.318 ms  269.359 ms  269.342 ms
 6  125.19.2.37 (125.19.2.37)  271.798 ms  271.795 ms  271.695 ms
 7  182.79.224.21 (182.79.224.21)  271.767 ms 182.79.217.33 (182.79.217.33)  271.758 ms 182.79.247.54 (182.79.247.54)  271.751 ms
 8  72.14.220.197 (72.14.220.197)  271.744 ms  79.663 ms  79.537 ms
 9  108.170.248.161 (108.170.248.161)  76.414 ms 108.170.248.193 (108.170.248.193)  74.400 ms 108.170.248.209 (108.170.248.209)  74.383 ms
 10  google-public-dns-a.google.com (8.8.8.8)  109.846 ms  75.681 ms  110.936 ms

As expected, the UDP traceroute will land at the correct place.

Manipulating latency

This can be done with the tc GNU/Linux command. Let’s add 15 ms to make it look a little bit more believable.

On VM 2:

arhue@ubuntu-spoofer:~$ sudo tc qdisc add dev enp0s8 root netem delay 15ms
[sudo] password for arhue:

On VM 1:

arhue@ubuntu:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=63 time=16.9 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=63 time=18.0 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=63 time=18.5 ms

Well there you go! A lot more believable.

Similarly this whole process can be done for a bunch of IPs by importing address lists of a particular ASN/network you want to manipulate the traceroutes/pings of.

Preventing against it

By using lots of tcpdump I found out that traceroutes can be done in four different ways:

  1. ICMP – This is how Windows does it. traceroute command on GNU/Linux can do this if you pass the –icmp flag. Can be easily be manipulated
  2. UDP (Unix style) – Ports between 33435-33655 are usually used. This is the normal way of doing traceroutes on GNU/Linux and Mac OS. This can be easily marked and traffic manipulated again.
  3. UDP – This works on port 53. This can be used with –udp flag with the traceroute command on GNU/Linux. I am not sure how this can be manipulated.
  4. TCP – This works on port 80. This can be used with –tcp flag with the traceroute command on GNU/Linux. I am not sure how this can be manipulated.

If traceroute traffic can be marked, its routing policy can definitely be edited depending on that marking. I wanted to investigate this more but I have lots of work pending to do. The truly best way of preventing this is picking up the phone and screaming at your ISP for doing these stupid illicit things rather than focusing on sell a good service to the customer.

Offending ISPs

Since Srijit’s blog had talked about North Eastern ISPs doing this shady practise, I did a RIPE Atlas tests on the only ISP who’s name I knew. The result was surprising!

Traceroute to 8.8.8.8 from an offending ISP

Surprisingly traceroutes for UDP, ICMP and TCP traffic were similar so they have indeed found a way to manipulate UDP and TCP traceroutes as well.

But the DNS test does give us a clue on the pings and traceroutes being completely wrong.

Fetching A record from 8.8.8.8 from offending ISP’s network

The latency should never be that high to fetch the A record of google.com.

Links to all RIPE Atlas measurements:

https://atlas.ripe.net/measurements/8273219
https://atlas.ripe.net/measurements/8273170
https://atlas.ripe.net/measurements/8273063
https://atlas.ripe.net/measurements/8310700

Finishing thoughts

I think one day when I get enough time I’ll schedule a mass traceroute from the RIPE Atlas project, process the data and tabulate a nice chart of offending ISPs to publicly shame them. People obviously will find out and when they do it isn’t going to be good for them.

I think I am done here. Good night!

If there are any errors here, please do let me know. I have written this in a hurry and don’t have enough time to proofread.

As always this being my personal blog, reflects my personal opinions only and not those of my employers.

]]> https://varunpriolkar.com/2017/04/how-to-fake-pings-and-traceroutes/feed/ 0 1142 10 Gbps network for less than $60 https://varunpriolkar.com/2017/03/10-gbps-network-for-less-than-60-dollars/ https://varunpriolkar.com/2017/03/10-gbps-network-for-less-than-60-dollars/#comments Sun, 12 Mar 2017 11:57:27 +0000 https://varunpriolkar.com/?p=1045 Read More »]]> I needed a fast connection between my NAS and my VM server. So I set on to the mission of doing 10 Gbps network connectivity for as cheap as possible.

This is mainly to run iSCSI for VMs. I had looked at Infiband as well. However driver support was poor and things aren’t guaranteed to work. IPoIB is also very processor intensive.

Things you will need

  1. NIC cards – There are two options to do 10 Gbps for cheap here.
      1. Mellanox Connectx-2 – These are quite old cards with support only on GNU/Linux. However they are pretty cheap.
      2. Chelsio 110-1088-30 – These have dual SFP+ interfaces and have working drivers for FreeBSD and GNU/Linux but could be a little expensive. I picked up a pair of them for $37.36, inclusive of shipping.

    I have not investigated if either of them have Windows or Mac OS support. I got these through a friend in the US. If you live outside of US, you can ask Amazon to ship it to your country or use a parcel forwarding company like PPOBOX.

    Update 13/03/2017: The cost of the Chelsio NICs has shot up on Amazon. They are much cheaper on Ebay. As pointed out by Robert in the comments, Brocade BR-1020 with Brocade firmware flashed DAC/transceivers are also a good option. You can read tips about 10 Gbps networking on his blog post here.

  2. Optics/DAC cable – Fiberstore is one of the best for this. The packaging and support for even the smallest orders is excellent and they ship around the world. For connecting two NICs together, get one of the Direct Attach Cables(DACs). I ended up getting this for $21.70, inclusive of shipping to India.

    DAC cable from FS.com

    For distances >5m pickup the active version, otherwise passive should be fine. You can also get transceivers and fiber from Fiberstore.

  3. Switch – This is not necessary if you want to connect NICs directly. However if you need one with SFP+ slots, I highly recommend the Mikrotik 326-24G-2S+RM. Make sure you get transceivers flashed with the brand your switch is. The other end should be a generic version. You can also buy a bunch of NICs and do bridging in software if you are fine with a less performant and messy setup.

So this is what I ended up getting:

Components cost
2x Chelsio 110-1088-30 $37.36
1x 3m passive DAC cable(inc. shipping to India) $21.70
Total $59.06

Testing it all out

Connecting it all up was pretty much straight forward. I had to put in the card in a PCIe x8 slot and attach the DAC cable. PCIe 2.0 x8 can do speeds of up to 24 Gbps so bottlenecking of PCIe lanes shouldn’t be an issue. Both GNU/Linux and FreeNAS(uses FreeBSD under the hood) detected the NICs out of the box. I added both of them up in a 192.168.5.0/30 subnet and confirmed I could ping each host from the other.

The Chelsio card sitting pretty in my server

I used the open source network benchmarking tool iperf to test out the network.

[root@freenas] ~# iperf -t 60 -c 192.168.5.2
------------------------------------------------------------
Client connecting to 192.168.5.2, TCP port 5001
TCP window size:  129 KByte (default)
------------------------------------------------------------
[  3] local 192.168.5.1 port 25118 connected with 192.168.5.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  65.7 GBytes  9.40 Gbits/sec

From the other side:

root@proxmox:~# iperf -t 60 -c 192.168.5.1
------------------------------------------------------------
Client connecting to 192.168.5.1, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.5.2 port 33846 connected with 192.168.5.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  61.9 GBytes  8.86 Gbits/sec

As you can see it can push almost 9 Gbps over TCP. Throughput over UDP should be higher. There was very less increase in CPU power while pushing through a lot of data.

And a nice graph:

Network graph from my Proxmox server

You can run LACP/MPIO if you need more throughput.

I’m super happy with my setup. I hope this has been useful for you. Please do leave any experiences that you may have had with cheap 10 Gbps gear in the comments below. 🙂

]]> https://varunpriolkar.com/2017/03/10-gbps-network-for-less-than-60-dollars/feed/ 14 1045 Google outage due to Cyclone Vardah in India https://varunpriolkar.com/2016/12/google-outage-due-to-cyclone-vardah-in-india/ https://varunpriolkar.com/2016/12/google-outage-due-to-cyclone-vardah-in-india/#comments Mon, 12 Dec 2016 21:55:34 +0000 https://varunpriolkar.com/?p=791 Read More »]]> This post is constantly updated with whatever information I can get my hands on. Refresh for updates.

World wide panic!!! Google is down! Grab your routers and run!!!11

Well not really but internet services were massively affected due to the Cyclone Vardah near Chennai.

To my knowledge three major things happened.

  1. Google(AS15169) had huge problems and traffic was shifted to POPs other than Mumbai.
  2. Vodafone’s Chennai location had an outage. Traffic was being moved to Mumbai. Enterprise and retail customers had issues.
  3. There was huge packet loss on most international websites, possibly due to capacity issues.

I am going to talk about the first one.

Google down you say?

Well I was alerted by this post by my friend and unusually slow internet.

So let’s explore what went wrong. Inspite of having very high packet loss while accessing RIPE’s website, I could manage to get one traceroute through RIPE Atlas project.

Trace from AS58457, Hyderabad to google.com at around 7 PM

As you can see traffic destined for Chennai is terminated at Singapore, possibly carried there by Google itself cold potato style. Likewise I noticed that I was hitting Delhi rather than Mumbai. So traffic was just moved away from Mumbai to other locations. Likely reason for this would be connectivity issues between Mumbai and Chennai possibly due to fiber cuts.

I continued to monitor the situation some time later. Link to test: https://atlas.ripe.net/measurements/6959487/#!probes

Trace to google.com from probes in India at around 1 AM

As you can clearly see even probes close to Mumbai were hitting Chennai instead of their POP closest to their region. Let’s test out the aspmx.l.google.com domain used for Google Apps which isn’t terminated in India. Link to test: https://atlas.ripe.net/measurements/6960037/#!probes

Trace to aspmx.l.google.com from probes in India at around 1 AM

As you can see the latencies are not good. Also my friend alerted to me a few strange traces like this.

Trace to aspmx.l.google.com from AS9430 at around 12:30 AM

What is happening here is that Google was rerouting traffic constantly and BGP routers were taking a while to respond.

So what exactly happened?

Google has a huge network. They can’t possibly be down, can they?

There is not much Google content actually in India right now. Most of it is either cached by Google itself, while terminating SSL on one of their Indian nodes – Mumbai, Chennai or Delhi. Eyeball networks/ISPs then either directly peer with Google or their upstreams do. Most YouTube content is just cached by GGC nodes inside ISP networks.

Google has limited capacity to Europe through Mumbai and most of Indian traffic is actually served from Taiwan via Singapore through a fat pipe in Chennai.

Moving traffic between sites is normally not a problem but this caused two major problems:

  1. ISPs were quite used to undersizing on IP transit capacity and would rely on peering with Google. Google is around 50% of their capacity so this does make sense.
  2. Google’s routes were constantly flapping and routers would be rather slow to react to that. Cheaper routers like Mikrotik would be most susceptible to this.
  3. Larger transit players and telecos likely had issues of their own.

So what happened was that Google wasn’t actually down. The connectivity from ISPs to Google was hampered because they didn’t have enough transit capacity if the plug was pulled on their peering capacity.

Long term Solutions

Likely solutions would be to avoid this in the future would be:

  1. Eyeball networks should always try to over build on transit capacity. So incase there is loss in connectivity to Google PNI, traffic can always be served from a different location from transit. If they are large and diverse enough, maybe they can peer at different locations. This would also prevent from carrying traffic over their backbone.
  2. Better routers for BGP or using something like BIRD would help. Mikrotik, which are very popular for small to medium networks in India can only do BGP on only one core, which can hamper performance.
  3. Smaller ISPs should have redundant upstreams. Vodafone was badly affected, while networks like Airtel and Tata were less so. Having redundancy helps.
  4. More content networks like Google should have more content inside the country. Google has plans of setting up a datacenter in Mumabi. More info here.
  5. Redundant connectivity between sites with different fiber paths would be nice for large networks.
  6. Outside connectivity with landing stations at multiple locations for transit and larger players would be nice for internet in India in general.

Change your DNS!

If you are using Google DNS(8.8.8.8, 8.8.4.4) move to OpenDNS(208.67.222.222, 208.67.220.220) immediately!

Also what I did was move from Google’s public DNS resolvers to OpenDNS because Google makes their queries from outside India and they are just cached at their POPs here so non-cached queries could have issues. That plus the fact that they use anycast and could hit a node like Mumbai would cause issues. Non-Google services were much better for me after I made the switch.

As I finish this post, I have noticed that connectivity is much better, atleast with my ISP(AS58405) so time to go watch some YouTube streams! 😀

Update 13/12/2016 3 PM:

I have some information that tells me that Google had interconnects from Powergrid(AS132215) that had issues. While connectivity by Powergrid was restored by morning, i2i cable leased by Google possibly has some issues as well and I am seeing a lot of Google traffic taking international route after my ISP’s upstream hands it off outside India. ping google.com is hitting South Carolina, after Airtel(AS9498) hands it off in London.

Update 13/12/2016 5:30 PM:

A friend confirmed that Airtel’s i2i cable indeed has issues.

Dear Customer, the cyclone in Chennai has impacted one of our undersea network cables which may affect your internet speeds. We deeply regret the inconvenience. Our engineers are working to resolve the issue and services will be normalized shortly.Airtel

Trace to Singapore was also going through LINX, London for me.

arhue@xubuntu-desktop:~/Videos/servo$ traceroute www.oneasiahost.com
traceroute to www.oneasiahost.com (163.47.176.1), 30 hops max, 60 byte packets
 1  gateway (192.168.10.1)  0.198 ms  0.255 ms  0.259 ms
 2  10.102.254.1 (10.102.254.1)  26.036 ms  28.425 ms  29.244 ms
 3  10.240.2.10 (10.240.2.10)  29.980 ms  30.215 ms  30.248 ms
 4  103.65.198.1 (103.65.198.1)  32.776 ms  33.312 ms  33.446 ms
 5  10.139.103.1 (10.139.103.1)  33.539 ms  33.543 ms  33.557 ms
 6  aes-static-025.51.246.61.airtel.in (61.246.51.25)  47.429 ms  22.547 ms  58.581 ms
 7  125.62.187.214 (125.62.187.214)  182.880 ms 182.79.222.17 (182.79.222.17)  186.652 ms 182.79.222.78 (182.79.222.78)  190.037 ms
 8  linx-juniper.sg.gs (195.66.226.38)  246.037 ms  246.187 ms  246.149 ms
 9  switch-1.v3.uk1.sg.gs (124.6.36.25)  252.934 ms  248.928 ms  263.849 ms
10  ns1.oneasiahost.com (163.47.176.1)  260.528 ms  259.311 ms  261.143 ms

I did an another RIPE Atlas test. Link here: https://atlas.ripe.net/measurements/6960503/#!probes

Trace to google.com from probes in India at around 5:30 PM

As you can see, the map shows a very bleak picture, with most of the traffic going outside the country. Let’s look at the OpenIPMap to see where most of the traces are landing up.

OpenIPMap for trace to google.com from probes in India at around 5:30 PM

As you can see very little traffic is going to Singapore directly via Chennai. This possibly means there is a problem with the i2i cable connecting India to Singapore. Google likely carries whatever traffic lands up in Europe to other places since it’s a cold potato network. It doesn’t prefer to carry it from India itself since the Europe-India pipe that they have isn’t very high capacity to carry all that traffic.

Update 13/12/2016 9:30 PM:

Traffic is back to hitting the Indian nodes. RIPE Atlas test here: https://atlas.ripe.net/measurements/6960735/#probes

Trace to google.com from probes in India at around 9:30 PM

As you can see traffic is hitting even the Mumbai node. I don’t know how these nodes are fed with content as it looks like the i2i cable system still has issues.

Tracerouting to Singapore the traffic all flows through London. For reverse path, I used HostSG(AS24482)’s looking glass.

Router: SG1 (Epsilon) - Edge-2 (Customer, Standard Route)
Command: traceroute inet 43.249.187.200 as-number-lookup source 203.175.175.2


 1  edge-1.sg2.sg.gs (203.175.175.101)  0.827 ms  3.495 ms  0.837 ms
 2  116.51.23.65 (116.51.23.65) [AS  2914]  3.106 ms  1.203 ms  1.743 ms
 3  ae-4.r21.sngpsi05.sg.bb.gin.ntt.net (129.250.4.16) [AS  2914]  1.886 ms  2.267 ms  2.102 ms
     MPLS Label=440911 CoS=0 TTL=1 S=1
 4  ae-8.r24.londen12.uk.bb.gin.ntt.net (129.250.7.64) [AS  2914]  191.297 ms  188.173 ms  186.029 ms
     MPLS Label=617522 CoS=0 TTL=1 S=1
 5  ae-8.r02.londen03.uk.bb.gin.ntt.net (129.250.4.22) [AS  2914]  184.094 ms  189.205 ms  189.433 ms
 6  flagtelecom-0.r02.londen03.uk.bb.gin.ntt.net (83.231.235.238) [AS  2914]  209.137 ms  200.179 ms  205.002 ms
 7  85.95.27.125 (85.95.27.125) [AS  15412]  208.916 ms xe-0-0-1.0-pjr04.mmb004.flagtel.com (85.95.26.158) [AS  15412]  226.170 ms  248.547 ms
 8  * * *
 9  * * *
10  220.227.118.233 (220.227.118.233) [AS  18101]  215.753 ms  199.318 ms  199.310 ms
11  10.139.103.1 (10.139.103.1)  215.223 ms  215.225 ms  211.806 ms
12  43.249.187.200 (43.249.187.200) [AS  58405]  212.304 ms  214.853 ms  215.016 ms
13  43.249.187.200 (43.249.187.200) [AS  58405]  210.071 ms  212.743 ms  200.098 ms

As you can see NTT(AS2914) is carrying the traffic to UK, before handing it off to Flagtel(AS15412). Most probably Indian telecos have stopped announcements of Indian routes on Singapore routers and are preferring routes learnt from their European routers.

Google services should be better but we’ll have to wait and watch about how the i2i problem plays out. Networks not using the i2i cable should be much better off, as far as connectivity to Singapore is concerned.

Update 13/01/2017 6:30 PM:

i2i cable issues are still not fixed. Airtel is using SE-ME-WE4 to get to Singapore instead of i2i but the Singapore to India capacity is probably not as much as required so they are sharing fewer Indian routes in Singapore. You can observe a lot of traffic going via Europe to India. The forward path from India to Singapore is fine, probably because there is very less traffic from India to Singapore so little choking in general. My ISP uses Reliance communications(AS18101) for return(heavily prepends announcements to Airtel) so I am much better off.

Tata Communications Ltd. has issues with TIC cable as well. This is what they sent to their customers.

You may observe additional latency of 30 -230ms for US West Coast and APAC prefix.

Please be informed that, this is due to under-sea cable fault on TIC submarine cable system. The fault is located in the wet segment in between repeater R58 & R59, this is near Singapore CLS, around 85-95 KMs from CLS.

As per latest update, the TIC cable fault repair is tentatively scheduled from 8th February 2017 and permit process is in progress. Additionally during repair operation Repeater R52 replacement will be carried out. The tentative ETR for the repair completion is 19th February 2017 depending on weather conditions.Tata Communications Ltd.

Fixing i2i and TIC cables could take a few more months, before which latency to Singapore could be affected. Fixing submarine cables is a long process. You need good planning, permits, allocating resources, sending diving teams etc. So we’ll have to be patient

]]> https://varunpriolkar.com/2016/12/google-outage-due-to-cyclone-vardah-in-india/feed/ 38 791 How to start an Internet Service Provider in India https://varunpriolkar.com/2016/12/how-to-start-an-internet-service-provider-in-india/ https://varunpriolkar.com/2016/12/how-to-start-an-internet-service-provider-in-india/#comments Sat, 03 Dec 2016 05:57:52 +0000 https://varunpriolkar.com/?p=689 Read More »]]>

So I was planning to start an ISP in India but a few things happened and I gave on the idea. I did however do some research which I can share with you and hopefully you’ll find that interesting and helpful. 🙂

An Internet Service Provider or an ISP is basically an organisation or a company which provides users with internet access. I will be exploring most of the needed things needed to start one, the regulations and a few other things. I know that a lot of people would have the idea of starting one and hopefully I can help you out. While this post is primarily geared towards the Indian ISPs, this should help people trying to start one in other countries as well.

Firstly I highly recommend you to view this YouTube video. Skip to 54:30.

The last mile

For a residential/commercial provider, this should be your major focus point. Most of the time you’ll be scrambling to splice up fibers or putting in an Ubiquiti device, not managing a bunch of devices at the NOC. You will need to be a hardcore last mile guy or have someone in your company/business to help you out on that front. I didn’t go forward with my idea exactly for that reason. Having a good last mile will make or break your company, not trying to have cheaper bandwidth. There are two options you can look at for the last mile.

Wireless

Many providers like to start out on wireless or aspire to be a fully wireless provider. The problem? Telecom Regulatory Authority of India(TRAI) only allows for usage between 5.825 Ghz and 5.875 Ghz for WISPs. 50 Mhz is very cramped up for an ISP to operate in, especially when it will share that spectrum with many of it’s competitors. You can always use other bands illegally, but so do your competitors and overall there is too much noise here. It can be very problematic and there can be numerous issues which can occur. Site to site P2P links for commercial usage can workout fine but in P2MP(Point to Multipoint) scenarios, sharing so little spectrum among so many users is very problematic. PPPoE can also be an issue on wireless and /30 VLANs can be problematic if you don’t want to NAT. A huge wireless bridge is an even bigger problem. I will be talking about this in later sections. Wireless is very nice for corporate/business customers or linking far away sites together for P2P links. Otherwise it’s a massive pain.

Does it make sense:

  • In India to every customer? Absolutely not. Laying fiber is cheaper usually.
  • Other countries where last mile laying of fiber is expensive and spectrum is more liberal for WISPs and less choked? Yes. Also explore Millimeter wave radios if you fall in this category.

Also do remember that for wireless Ubiquiti is usually better than Mikrotik, albeit a little more expensive.

Fiber

pon_vs_aon

Two different topologies – Active Ethernet and Passive Optical Network

This would probably be better. The problem is that last mile access and right of way is badly screwed up in India. It would have been great if the govt. had run conducts(essentially pipes) along electricity poles or underground installations and leased them out to private players. Find out about the cost of laying fiber legally on poles in your area. In Goa this is super cheap at about ₹1000/km/month. Other places like NCR it is prohibitively expensive. You can always do it illegally and usually nobody would care, but you of course run the risk of future action being taken. Also fiber gets cut. A lot. So make sure you have a nice team set up who can take care of the fiber and lay new fiber and get a splicer, unless you can lease it from someone for a good price. For fiber you again have two options:

Active ethernet

This basically is running one core of fiber either to each apartment/customer or using active devices like switches to essentially split fiber to be used by more customers. A lot of people like to use media converters and regular cheap unmanaged switches downstream. Most of media converters are crap and this tends to cause a lot of issues. Active ethernet is very hard to do in a country like India where the cost of fiber a lot compared to the cost of laying it. Also you need uninterrupted power at all locations, which can be a huge headache to manage and is not at all scalable. Thankfully there is a better solution.

Passive Optical Network

ftth-network-splitting-level-design

4×8 splitting arrangement for 32 ONUs. You can split however you like

Passive Optical Network or PON allows you to essentially split a single strand of fiber to upto 128 customers, although 32 or 64 is usually preferred. The PLC splitters are extremely cheap and do not need power. You will need an ONU or an Optical Network Unit on the customers end, which can be had for around ₹1500-2000(for GEPON) and OLT, which costs around ₹70000-80000 for 4 ports expandable for GEPON. As for technologies you can use GEPON(Gigabit Ethernet Passive Optical Network) or GPON(Gigabit Passive Optical Network). GEPON is pure ethernet, while GPON encapsulates ethernet packets. GPON will give you 2.5 Gbps downstream per fiber, while GEPON is restricted to 1 Gbps. There is 10-EPON/10-GPON too but as of writing this article, they are prohibitively expensive. 10-EPON is backwards compatible with GEPON, which is gaining more traction compared to the 10-GPON so keep that in mind. With GEPON you can also do a hybrid by putting in active devices or POE powered ONUs so keep that in mind. GPON supports direct TV content, without relying on IPTV. Both use Wavelength Division Multiplexing (WDM) to have uplink and downlink over one fiber strand and Time-division multiplexing (TDM) to divide upstream data. Generally GEPON devices are cheaper as compared to GPON.

Spiliting and power levels is something you will have to be aware of. Get a nice power meter to check for power levels. OTDR may be also needed. You can get one later if it is too expensive for you at this stage. Explaining how power levels work may be beyond the scope of this article. Hopefully the above slide may help you out. AFAIK PON equipment is mostly SC/PC and you will need single mode fiber. RX/TX of either ONU or OLT added onto the opposite of the other will give you the total power. Eg. for upstream if TX of ONU is 3 dBm and RX sensitivity of OLT is -27 dBm, then you have 30 dBm of power to use max. People usually prefer to do 1:32 split for <= 5 kms and 1:64 for <= 20 kms.

Not doing last mile at all

This is what most providers end up doing since the last mile is a major pain. Instead you can just sell your bandwidth and billing services to smaller ISPs or LCOs(Local Cable Operators) and they will do the last mile for you and take a cut. However to get to that stage you will need to have a network of your own since without major investment or cashflow, this is just difficult to do. Running an ISP needs money and cash to flow in since this is a business where economies of scale really matter. You can use a NLD(National Long Distance) link from a teleco operator or simply a wireless P2P link to deliver the circuit to a smaller provider if the distance is large. For a smaller distance you can use fiber. A healthy mix of both is nice, since the smaller providers can always buy the circuit directly from a teleco if they grow large enough.

The middle mile

This is a place where wireless really can be helpful. Ubiquiti sells Airfiber radios which are very nice for these scenarios. Yes you should really try to build out fiber but over long distance this can simply be not possible. Millimeter wave radios are nice but they are illegal in India and not easily obtainable.

You can also use NLD links for this. These are basically point to point links usually sold by a telecom operator like Airtel, Tata, Reliance Communications, Vodafone. Unless you have serious distance between which you want to transport data, do not have a look at this. Powergrid also sells these in India for a good price. The pricing? ₹200-300, depending on capacity you want and how you negotiate. Always negotiate. Everything in this business depends on your negotiating skills to get the best price possible. Everyone is ready to budge. You however will not be needing any kind of NLDs in the beginning, unless you are selling bandwidth to someone else.

Network Operations Center

noc

What your NOC may look like!

The Network Operations Center or the NOC will be the central point in your network. It is where the bandwidth will be coming in and you will be doing the billing or shaping traffic. I will be using this topic to talk about these things in a fairly detailed way. Overtime you will need smaller Point of Presences(POPs) away from your main NOC. You can use teleco buildings to keep your equipment to keep your equipment in. You can pull fiber from there.

Bandwidth and buying IP space

You will need to get bandwidth from the nearest teleco site from where it is delivered to your NOC. I would recommend using two fiber installations with different paths or fiber plus wireless for redundancy for this. I will talk about regulations and from where to buy the bandwidth from in later sections.

About the choice of routing protocols at the core, you can either run BGP or use static routing. For anything serious, BGP is a must. You can then announce your own IP space and can have multiple uplink providers and peer with content providers. Peering essentially means you exchange traffic for free. About 40-50% of your bandwidth would be to Google. Wouldn’t it be amazing that you can get all that for free and get better latency for your users? Well you can by peering with Google. You will need to get a NLD link to one of their POPs and peer. My friend has written an excellent article on the content providers to peer with here. Do note that you will need some capacity before the other party can agree. For Google that is 100-150 Mbps. Also do note that the datacenter provider that any company you want to peer is present that may charge you for keeping your router/switch there and for the interconnect. GGC(Google Global Cache) may be better in the beginning for caching content. Redundancy for NLD link may be problematic too. You can also have a look at NIXI for peering.

All I’m saying is that running BGP gives you so much more flexibility and self reliance than just depending on someone else for IPs and bandwidth. You can say run Airtel, Tata links for IP transit(essentially internet connectivity) and peer for regional routes so essentially you have to pay for less for bandwidth and get better routing and redundancy.

For IP space you will have to get this from your upstream if you are doing static routing. However it is simply a better idea to get this from the local internet registry – APNIC for India. The costs however are prohibitively expensive. Luckily we have IRINN for India. The costs are a very low ₹25000 initial application fee and about ₹45000/year for a /22 IPv4 block, with free IPv6 blocks. That’s 1024 public IPv4 addresses. So ₹3.66 per month. That’s cheap! They are still giving out /22 IPv4 block for new applicants and /32 IPv6 block if you try hard enough. You will need an ISP license and a Pvt. Ltd. comapny to get IPs from IRINN. For dynamic BGP routing, you can announce them yourself or for static routing your upstream will have to announce them for you. If you need to run NAT for more IPv4 addresses, do make sure you keep logs since that’s mandated in your ISP license. More on that later.

Contention Ratio (or over subscription ratio) is the number of users sharing the same data capacity. Many people try to think too much about contention. Don’t. What you should be looking at is the peak bandwidth usage. Whatever you do, do NOT oversell your line. Less bandwidth is a good problem to have. It means you can start getting more and buying more is cheaper. While overselling may get you more customers, they will likely not stay. The relationship is very clear:

Less capacity>Buy more bandwidth>Cheaper prices with more capacity and better contention>More profits.

Also as you grow, keep one DDOS protected circuit from a transit provider. They usually cost about 30% more but can come in handy.

Billing software

You will need to have some kind of billing/subscriber management/IP address allocation/payment collection management service. Most of the solutions use RADIUS to interact with the router. You can ofcourse build your own with FreeRADIUS as a base, but that is too cumbersome when there are ready made solutions available for not a lot of price. Synnefo, Log2Space and SSRMS from Height8 are some of the more reputable and larger ones. Synnefo and SSRMS also provide a cloud hosted solutions. Otherwise you would have to host it yourself on a server. They also are RADIUS based, unlike the other meaning they can directly interact with your router. Pricing is usually ₹10-15/customer/month depending on how you negotiate and what kind of features/services you may want. I do have Synnefo’s PDF with me. You can have a look at it here. Ask for a demo before making your choice.

Router/switch choice

fb_mikrotik

Mikrotik is an excellent choice for routers

For router Mikrotik is an excellent choice for a small-medium sized ISPs. You can run BGP, PPPoE(for subscriber management), VLANs etc. all on the same box. Do remember to use PPPoE and VLANs for keeping broadcast domains different since putting everything in a bridge is a recipe for issues as you grow. Get something from the Mikrotik’s CCR line. You can also run your own x86 hardware or a VM to run Router OS.

For the switch, any managed switch would do. However Cisco’s SG300 switches are super nice and reliable. Also keep spare equipment and cheaper 10/100/1000 unmanaged switches in hand just in case you may need them some day. Doesn’t hurt to have them.

For NIC cards in servers always use Intel whenever possible. They work very reliably and are not very expensive. I will be playing around with some cheap cheap Chelsio and Mellanox ConnectX2 cards for 10G. I’ll keep this updated with the results for that. But generally you shouldn’t be needing 10G at this stage.

Traffic monitoring and DNS

I am a big fan of using open source software whenever possible. The software is higher quality, there are no costs other than managing it and overall the experience is much better. You can either run a DNS recursor to send queries to the root server or run it in forwarding mode to say Google. Whatever you do do, do not use third party non-upstream DNS servers. Most DNS servers do not have EDNS support and that can break CDN support. You can run Unbound or PowerDNS for the DNS server. You can use Pi Blocker to block off ads for customers who want it, for convenience and saving money off your bandwidth bill.

You will need some kind of traffic analysis to get an idea of what and how much the traffic in your network is. There is again an open source solution available. You will simply have to port mirror from your managed switch to the appliance that you build or a VM. Ntop is a nice solution. I believe their ntopng can work out nicely for you, which also has nDPI for deep packet inspection, which can tell you what quantity is of traffic is flowing where.

LibreNMS is something you can use for monitoring all of your devices. Uptime Robot is nice for uptime status pages.

Web/torrent caching servers

Web caches are pretty much obsolete these days. There is not much unencrypted traffic which can be cached by a HTTP transparant proxy. Do not fall in the trap of torrent caches since most of them heavily rely on PBR or Policy Based Routing. Without a router that can do routing decisions on ASIC, this can lead to CPU spikes on your router and overall is a terrible solution. If you do want to proceed, I’ll be listing out the vendors for such caches in later sections. Instead peering is simply a better solution.

For caching Steam downloads, Steamcache is an excellent solution that you can look at. It is completely free and doesn’t rely on PBR. For torrents, rolling out your own solution with BEP22 may be a good idea. Not only does this not rely on PBR, it can cache encrypted torrents as well. Integrating all of this with your billing software could be a challenge.

Also do not do much traffic shaping. This can generally lead to poorer performance and you are essentially trying to solve a non-technical problem with a technical solution. The solution is simply to buy more bandwidth.

Power

Redundant power at all sites will be very important. You can either get an inverter with sufficient capacity, or get an Online UPS, which will isolate and protect your equipment as well. Make sure you have enough capacity to last atleast 10 hours in case of a power outage. I have seen people run huge infra without inverters at places with reliable power so YMMV.

Regulations

It’s no secret that over regulation is hurting India and this industry is no exception. Your first priority should be getting a license. This makes your operation more legitimate and means that you can get cheap IPs and cheaper bandwidth since you can deal with the more legitimate players and teleco companies directly, instead of doing shady dealings. Only reselling someone elses pre made plans is allowed if you do not have an ISP license. Smaller upstream players can budge though, however you are less likely to get a good deal.

Getting your ISP license

The initial cost for getting a license is described as below.

Service Min. Equity Min. Networth Entry Fee PBG FBG Pro. fee
ISP “A” Nil Nil 0.3 2 0.1 0.005
ISP “B” Nil Nil 0.02 0.1 0.01 0.0015
ISP “C” Nil Nil 0.002 0.005 0.001 0.001

All prices are in ₹ crores. ISP A license is a PAN India license. ISP B is a Telecom circle/Metro Area license, which usually means for the entire state. ISP C is a SSA(Secondary Switching Area) license. For getting the license you will either have to file the application yourself(talk to the Department of Telecom(DOT) guys) or find a middleman who can help you out for a small cut. You may have to pay 8-12% of your AGR as fees every year, although that is up in court. The later is usually preferred since the process can be tedious.

This will be a huge chunk of your work starting up. After this you can go apply for IP resources from IRINN and get cheaper bandwidth.

NAT logs and lim devices

DOT needs you to keep NAT logs so make arrangement for this. Further there is a requirement for lim devices, but this is often not enforced. From what I know, they will require you to run a machine or a VM with their monitoring software and port mirror to that. So keep this in mind.

Importing

Importing things is a huge hassle in India. Buy things locally as far as possible. Importing in personal capacity upto ₹50000 shouldn’t be a problem, however for companies you need a license. DHL, Fedex and some other cargo carriers can do customs clearance on behalf of you so do explore that as well.

Vendors

Getting where to get stuff from for a good price will be crucial. I have a list of vendors which I will be sharing. If you are a vendor and reading this, let me know and maybe I can add you to the list.

  • Fiber: Aksh Optifiber, Sterlite or any good provider near your area. Transporting fiber is expensive. Do keep that in mind. I got a quote of ₹7.8/m for 4 strand, ₹8.4/m for 6 strand, ₹9/m for 8 strand and ₹10.5/m for 12 strand from a quality local supplier.
  • Fiber splicer: Simimoto or Fujikura are good options. Cost: ₹1.5-2.5 lakhs. I don’t remember the name of the vendor in India.
  • PON equipment/general Mikrotik and Ubiquiti products: Tara Consulting Pvt. Ltd., JRS communications and Multilink Computers Pvt. Ltd.. Tara is better with the pricing initially, while JRS is better if you can negotiate. Tara’s OLT/ONUs are very good from what a friend told me. The pricing is nice too. Syrotech also deals with GEPON equipment. For Huawei OLT/ONUs importing would be your only good option.
  • IP transit: Smaller players for upto 25 Mbps will cost you like ₹1000-1100/Mbps if you negotiate well. With a license STM-1(155 Mbps) should cost you ₹700-800/Mbps. The pricing is dropping all the time. If you do have a license, Airtel and Vodafone are good options for a few STMs of capacity. You may get a better deal with other Class A ISPs. Don’t bother with Tata unless you have >1 Gbps of capacity. As you grow, keep two pipes from large players and get rest of the capacity from peering. If you can’t find a good deal, do contact me. Maybe I can help.
  • IP addresses: IRINN or APNIC. Maybe your upstream provider can give you some but generally this is not a good practice.
  • NLD: Powergrid is cheap. Also talk to the telecos. Cost: ₹200-300.
  • Peering locations: GPX Mumbai, if you are close to Mumbai. Otherwise look for something regional. NIXI is a good option too. Iifon is operating an exchange at Agartala I think.
  • Torrent caches: Xtraband’s XCache2 and Extreme Peering. Cost: For XCache2: ₹60000/year for 40 Mbps of capacity. I also have XCache2’s brochure here. For Extreme Peering: ₹300/Mbps for 100 Mbps, ₹128/Mbps for 1 Gbps of “effective saving” per month. Either is not recommended if you do not have a router with ASIC to do forwarding.
  • Server equipment: Build your own from commodity hardware or buy ready made hardware. Some AMD motherboards support ECC RAM so that should be nice. You can also try importing from Ebay as well. Run VMs on something like Proxmox to save power and money.
  • Billing: Synnefo, Log2space and SSRMS. SSRMS is what most larger players use.
  • ISP license: I think Multilink can help you out in this. I don’t know about anyone else.

Some tips

Do keep in mind that larger telecos have plans of outsourcing last mile to LCOs and making plans/doing billing themselves. Also you need to have good financing since this is a business where you have to pour in a lot of money for building the network, before you start to see any real kind of revenue flowing in.

For India last mile ISPs are very important. The remaining infra is already there. The landing stations are in place and the telecos have laid fiber till the city. Now getting fiber to the homes is the next step. As more people use the internet and more capacity is being used, bandwidth prices will keep dropping.

Also I would suggest you to use some forums for reference:

  • Ubiquiti forums: The business talk section is excellent for asking questions/looking at what problems people are solving.
  • DSL reports forum: Another excellent site for generally exploring. Their ISP section is nice.

Well that’s it for me. This ended up being longer than I expected. Happy network building to you! 😀

Feel free to contact me here if you have any queries or simply leave a comment down below.

]]> https://varunpriolkar.com/2016/12/how-to-start-an-internet-service-provider-in-india/feed/ 37 689 Analysing Steam’s network in India https://varunpriolkar.com/2016/11/analysing-steams-network-in-india/ https://varunpriolkar.com/2016/11/analysing-steams-network-in-india/#comments Tue, 29 Nov 2016 01:37:30 +0000 https://varunpriolkar.com/?p=624 Read More »]]>

Excited by the prospect for Steam for Linux I fired up Steam to download some games. The problem was that finding a mirror with the best speed for seeming problematic. So I decided to do some analysis. 😀

I am testing all of this from AS55836 Reliance Jio Infocomm Limited. I primarily use this for steam since the throughput is nice to have and I don’t use more than 4GB of download on Steam games per day. To choose a mirror goto Steam>Settings>Download>Download region in the menu.

Singapore mirror

Since Reliance Jio peers with their upstream AS64049 Reliance Jio Infocomm Pte Ltd Singapore in Singapore, I decided to try the Singapore mirror first. The results? Not bad. I’ve used iftop command on Linux to see from which IPs the download was being streamed from.

screenshot_2016-11-27_06-31-49

iftop and game download from Singapore

The throughput was decent at 2.5 MB/s.The only problem is that it would be affected badly during peak times, due to heavy strain on the international capacity. As you can see most of the IPs are from 103.10.124.0/24 network, which is announced by AS32590 Valve Corporation in Singapore. Let’s do a traceroute, shall we?

arhue@xubuntu-desktop:~/Downloads$ traceroute 103.10.124.20
traceroute to 103.10.124.20 (103.10.124.20), 30 hops max, 60 byte packets
 1  gateway (192.168.42.129)  2.327 ms  2.322 ms  2.390 ms
 2  * * *
 3  10.71.225.130 (10.71.225.130)  111.569 ms  111.609 ms  111.705 ms
 4  172.16.93.209 (172.16.93.209)  96.941 ms  97.030 ms  97.130 ms
 5  172.26.31.2 (172.26.31.2)  96.664 ms  96.638 ms  96.805 ms
 6  * * *
 7  * * *
 8  * * *
 9  103.198.140.164 (103.198.140.164)  93.141 ms  93.010 ms  92.984 ms
10  103.198.140.27 (103.198.140.27)  186.625 ms  152.113 ms  160.538 ms
11  ldt-au01.ibeo.hgc-intl.com (80.81.194.166)  196.549 ms  191.214 ms  196.720 ms
12  global.hgc.com.hk (218.189.8.141)  249.810 ms  254.026 ms  235.519 ms
13  d1-244-224-143-118-on-nets.com (118.143.224.244)  243.390 ms  253.236 ms  239.243 ms
14  global.hgc.com.hk (218.189.12.234)  238.645 ms  246.932 ms  234.971 ms
15  * * *
16  * * *

It times out after that probably because ICMP is disabled after that. The path taken is AS55836 Reliance Jio Infocomm Limited(India)>AS64049 Reliance Jio Infocomm Pte Ltd Singapore(Singapore)>AS9304 Hutchison Global Communications(Singapore)>???. Luckily AS9304 Hutchison Global Communications has a looking glass for us to peak into. You can find it here. Let’s do ping.

ping 103.10.124.20

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 103.10.124.20, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

And a show ip bgp command.

show ip bgp 103.10.124.20
BGP routing table entry for 103.10.124.0/24, version 321730
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  9304 32590 32590, (aggregated by 32590 103.10.124.255)
    118.143.235.129 from 118.143.235.129 (118.143.225.4)
      Origin IGP, localpref 100, valid, external, best

So AS9304 Hutchison Global Communications peers with Valve. The next hop should be AS32590 Valve Corporation directly. The location is most probably Equinix Singapore if you have a look at Valve’s PeeringDB profile. The more important downstream path is hidden since Valve doesn’t have a looking glass. Let’s just assume that AS32590 Valve Corporation‘s router picks path through AS9304 Hutchison Global Communications as the best, which could be likely since they peer with them.

show ip bgp 49.35.248.119
BGP routing table entry for 49.35.128.0/17, version 73468304
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  9304 15412 3491 64049 55836
    118.143.235.129 from 118.143.235.129 (118.143.225.4)
      Origin IGP, localpref 100, valid, external, best

Path is AS9304 Hutchison Global Communications>AS15412 Reliance Glocalcom Limited(Singapore)>AS3491 PCCW Global(Singapore)>AS64049 Reliance Jio Infocomm Pte Ltd Singapore(Singapore)>AS55836 Reliance Jio Infocomm Limited(India). The same local preference means AS9304 Hutchison Global Communications is peering with AS15412 Reliance Glocalcom Limited. The long AS path length probably means AS32590 Valve Corporation‘s router may not choose to send traffic this way unless they are not peering with any network which is advertising a shorter one. Also probably AS18101 Reliance Communications Ltd‘s upstream AS15412 Reliance Glocalcom Limited is not announcing AS55836 Reliance Jio Infocomm Limited‘s IP pools in Singapore but is only used for local connectivity.

India(Mumbai) mirror

I am going to consider Mumbai since it is closest to me.

screenshot_2016-11-27_06-34-57

iftop and game download from Mumbai(India)

The throughput was better at about 2.6 MB/s and this would be better at peak times. This test was done at 6 AM so the results would have been different at different times of the day. Let’s do a trace.

arhue@xubuntu-desktop:~/Downloads$ traceroute 115.248.101.129
traceroute to 115.248.101.129 (115.248.101.129), 30 hops max, 60 byte packets
 1  gateway (192.168.42.129)  2.866 ms  2.968 ms  3.065 ms
 2  * * *
 3  10.71.225.162 (10.71.225.162)  89.943 ms  90.042 ms  90.015 ms
 4  172.16.93.211 (172.16.93.211)  89.604 ms  89.455 ms  89.675 ms
 5  172.26.31.6 (172.26.31.6)  89.275 ms  89.136 ms  89.348 ms
 6  * * *
 7  * * *
 8  * * *
 9  115.249.214.165 (115.249.214.165)  1555.076 ms  1555.055 ms  1555.133 ms
10  * * *
11  * * *

It is directly from AS55836 Reliance Jio Infocomm Limited to AS18101 Reliance Communications Ltd. We are more interested in the return tho. AS18101 Reliance Communications Ltd‘s routers do not have looking glass to find out the return path.

Let’s trace to the other IP.

arhue@xubuntu-desktop:~$ traceroute 182.79.231.2
traceroute to 182.79.231.2 (182.79.231.2), 30 hops max, 60 byte packets
 1  gateway (192.168.42.129)  1.216 ms  1.286 ms  1.391 ms
 2  * * *
 3  10.71.225.130 (10.71.225.130)  59.200 ms  59.317 ms  59.416 ms
 4  172.16.93.211 (172.16.93.211)  59.152 ms  59.270 ms  59.253 ms
 5  172.26.31.6 (172.26.31.6)  58.671 ms  58.656 ms  58.694 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  218.100.48.20 (218.100.48.20)  127.326 ms  85.924 ms  94.720 ms
12  182.79.243.105 (182.79.243.105)  94.810 ms  94.666 ms  99.804 ms
13  * * *
14  * * *
15  * * *

It times out after that. This is super interesting because if you look at NIXI‘s(AS24029 NIXI is an IXP in India) connected network list, 218.100.48.20 is the the AS9498 Bharti Airtel Ltd‘s IP at Delhi(Noida). So why is it flowing through Delhi(Noida) and not Mumbai? Both are connected at both places right? Well it turns out Airtel’s routers only share the regional routes for peering since they do not want to carry the traffic over their backbone. Here are NIXI’s LG results for Delhi(Noida):

show ip bgp 182.79.231.2
Number of BGP Routes matching display condition : 1
Status codes: s suppressed, d damped, h history, * valid, > best, i internal
Origin codes: i - IGP, e - EGP, ? - incomplete
    Network            Next Hop        MED    LocPrf     Weight Path
*>  182.79.231.0/24    218.100.48.20          100        0      9498 ?
       Last update to IP routing table: 4d10h26m56s, 1 path(s) installed:
       Route is advertised to 19 peers:
        218.100.48.9(55644)                      218.100.48.40(132453)                    218.100.48.10(10029)                     
        218.100.48.12(9583)                      218.100.48.13(17439)                     218.100.48.15(9829)                      
        218.100.48.17(17426)                     218.100.48.24(58640)                     218.100.48.25(17754)                     
        218.100.48.27(17488)                     218.100.48.28(55410)                     218.100.48.29(10201)                     
        218.100.48.30(4755)                      218.100.48.32(38625)                     218.100.48.34(45528)                     
        218.100.48.37(132323)                    218.100.48.36(132215)                    218.100.48.26(18101)                     
        218.100.48.31(55836)                     
       Route is to be sent to 3 peers:
        218.100.48.21(9498)                      218.100.48.23(63829)                     218.100.48.6(25152)

And for Mumbai:

show ip bgp 182.79.231.2
BGP4 : None of the BGP4 routes match the display condition

The path taken is AS55836 Reliance Jio Infocomm Limited(Goa)>AS24029 NIXI is an IXP in India(Delhi)>AS9498 Bharti Airtel Ltd(Delhi). It is interesting that AS9498 Bharti Airtel Ltd does not advertise all routes at all locations domestically but does so at international locations. Let’s see how the flow is in the other direction. Trace from AS9498 Bharti Airtel Ltd‘s LG on their Delhi router.

Tue Nov 29 05:40:15 GMT+05:30 2016
 traceroute 49.35.248.119 
traceroute to 49.35.248.119 (49.35.248.119), 30 hops max, 40 byte packets
 1  182.79.201.85 (182.79.201.85)  3.314 ms 182.79.201.89 (182.79.201.89)  0.985 ms  1.133 ms
 2  218.100.48.31 (218.100.48.31)  2.090 ms  2.104 ms  1.715 ms
 3  172.16.24.7 (172.16.24.7)  33.088 ms  31.330 ms 172.16.24.3 (172.16.24.3)  29.648 ms
 4  172.16.24.3 (172.16.24.3)  29.586 ms 172.16.24.7 (172.16.24.7)  31.102 ms 172.16.24.3 (172.16.24.3)  29.648 ms
 5  * * *
 6  * * *

It times out after that. 218.100.48.31 is AS55836 Reliance Jio Infocomm Limited‘s IP on NIXI’s member list. So return path is similar and flows through NIXI as well. Which is good! show route 49.35.248.119 for AS9498 Bharti Airtel Ltd‘s Delhi router:

Tue Nov 29 05:33:19 GMT+05:30 2016
 show route 49.35.248.119 detail | no-more
lookingglass@DEL-ISP-ACC-RTR-50> ...9 detail | no-more                       

inet.0: 696246 destinations, 1393589 routes (696203 active, 14 holddown, 207 hidden)
Restart Complete
49.35.128.0/17 (2 entries, 1 announced)
        *BGP    Preference: 170/-491
                Next hop type: Indirect
                Address: 0x18644e9c
                Next-hop reference count: 9422
                Source: 203.101.87.68
                Next hop type: Router, Next hop index: 1049641
                Next hop: 182.79.190.58 via xe-10/1/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xc97a
                Next hop: 182.79.217.98 via xe-8/0/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x80c
                Next hop: 182.79.217.94 via xe-8/0/2.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x80d
                Next hop: 182.79.208.118 via xe-8/1/1.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x80e
                Next hop: 182.79.208.174 via xe-9/0/0.0, selected
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x90d
                Next hop: 182.79.208.122 via xe-9/0/3.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x90f
                Next hop: 182.79.208.182 via xe-9/1/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x910
                Next hop: 182.79.208.186 via xe-9/2/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x911
                Next hop: 182.79.208.178 via xe-9/3/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x912
                Next hop: 182.79.203.126 via xe-7/2/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xa246
                Next hop: 182.79.203.114 via xe-7/3/2.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xa247
                Next hop: 182.79.190.62 via xe-10/1/3.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xc97b
                Next hop: 182.79.237.210 via xe-1/0/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6df
                Next hop: 182.79.255.10 via xe-1/1/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6e1
                Next hop: 182.79.190.66 via xe-10/1/7.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xc97c
                Next hop: 182.79.190.46 via xe-10/2/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xc97d
                Next hop: 182.79.190.50 via xe-10/2/1.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xc97e
                Next hop: 182.79.190.54 via xe-10/3/5.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xc97f
                Next hop: 182.79.248.45 via xe-1/3/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6e3
                Next hop: 182.79.252.57 via xe-0/1/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6c6
                Next hop: 182.79.255.250 via xe-0/1/1.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6c7
                Next hop: 182.79.252.209 via xe-0/2/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6c8
                Next hop: 182.79.237.206 via xe-0/2/3.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6c9
                Next hop: 182.79.252.213 via xe-0/3/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6ca
                Next hop: 182.79.248.49 via xe-0/3/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6cb
                Next hop: 182.79.255.246 via xe-0/3/3.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6cc
                Protocol next hop: 203.101.87.74
                Indirect next hop: 0x44ea6184 1049631 INH Session ID: 0xc986
                State: <Active Int Ext>
                Local AS:  9498 Peer AS:  9498
                Age: 4d 10:40:13    Metric2: 103 
                Validation State: unverified 
                Task: BGP_9498.203.101.87.68+46929
                Announcement bits (3): 0-KRT 5-Resolve tree 2 9-RT 
                AS path: 24029 55836 I (Originator)
                Cluster list:  0.0.0.50
                Originator ID: 203.101.87.74
                AS path: Recorded
                Communities: 9498:92 9498:24029 14111:20111 no-export
                Accepted
                Localpref: 490
                Router ID: 203.101.87.68
         BGP    Preference: 170/-491
                Next hop type: Indirect
                Address: 0x18644e9c
                Next-hop reference count: 9422
                Source: 203.101.87.71
                Next hop type: Router, Next hop index: 1049641
                Next hop: 182.79.190.58 via xe-10/1/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xc97a
                Next hop: 182.79.217.98 via xe-8/0/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x80c
                Next hop: 182.79.217.94 via xe-8/0/2.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x80d
                Next hop: 182.79.208.118 via xe-8/1/1.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x80e
                Next hop: 182.79.208.174 via xe-9/0/0.0, selected
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x90d
                Next hop: 182.79.208.122 via xe-9/0/3.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x90f
                Next hop: 182.79.208.182 via xe-9/1/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x910
                Next hop: 182.79.208.186 via xe-9/2/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x911
                Next hop: 182.79.208.178 via xe-9/3/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x912
                Next hop: 182.79.203.126 via xe-7/2/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xa246
                Next hop: 182.79.203.114 via xe-7/3/2.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xa247
                Next hop: 182.79.190.62 via xe-10/1/3.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xc97b
                Next hop: 182.79.237.210 via xe-1/0/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6df
                Next hop: 182.79.255.10 via xe-1/1/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6e1
                Next hop: 182.79.190.66 via xe-10/1/7.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xc97c
                Next hop: 182.79.190.46 via xe-10/2/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xc97d
                Next hop: 182.79.190.50 via xe-10/2/1.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0xc97e
                Next hop: 182.79.190.54 via xe-10/3/5.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0xc97f
                Next hop: 182.79.248.45 via xe-1/3/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6e3
                Next hop: 182.79.252.57 via xe-0/1/0.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6c6
                Next hop: 182.79.255.250 via xe-0/1/1.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6c7
                Next hop: 182.79.252.209 via xe-0/2/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6c8
                Next hop: 182.79.237.206 via xe-0/2/3.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6c9
                Next hop: 182.79.252.213 via xe-0/3/0.0
                Label operation: Push 465296
                Label TTL action: no-prop-ttl
                Session Id: 0x6ca
                Next hop: 182.79.248.49 via xe-0/3/1.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6cb
                Next hop: 182.79.255.246 via xe-0/3/3.0
                Label operation: Push 418113
                Label TTL action: no-prop-ttl
                Session Id: 0x6cc
                Protocol next hop: 203.101.87.74
                Indirect next hop: 0x44ea6184 1049631 INH Session ID: 0xc986
                State: <NotBest Int Ext>
                Inactive reason: Not Best in its group - Update source
                Local AS:  9498 Peer AS:  9498
                Age: 4d 10:40:13    Metric2: 103 
                Validation State: unverified 
                Task: BGP_9498.203.101.87.71+23609
                AS path: 24029 55836 I (Originator)
                Cluster list:  0.0.0.50
                Originator ID: 203.101.87.74
                AS path: Recorded
                Communities: 9498:92 9498:24029 14111:20111 no-export
                Accepted
                Localpref: 490
                Router ID: 203.101.87.71

show route 49.35.248.119 for AS9498 Bharti Airtel Ltd‘s Mumbai router:

Tue Nov 29 05:36:52 GMT+05:30 2016
 show route 49.35.248.119 detail | no-
lookingglass@MUM-SC-ISP-IGW-RTR-116> ...9 detail | no-m                      ore 

inet.0: 694924 destinations, 1396880 routes (694870 active, 3 holddown, 7457 hidden)
Restart Complete
49.35.128.0/17 (1 entry, 1 announced)
        *BGP    Preference: 170/-502
                Next hop type: Router, Next hop index: 1985
                Address: 0x214dc5b4
                Next-hop reference count: 393
                Source: 218.100.48.65
                Next hop: 218.100.48.82 via xe-0/1/2.0, selected
                Session Id: 0xf460
                State: <Active Ext>
                Local AS:  9498 Peer AS: 24029
                Age: 2d 17:13:25 
                Validation State: unverified 
                Task: BGP_24029.218.100.48.65+8001
                Announcement bits (4): 0-KRT 2-RT 8-Resolve tree 2 9-BGP_RT_Background 
                AS path: 24029 55836 I
                AS path: Recorded
                Communities: 9498:92 9498:24029 14111:20111 no-export
                Accepted
                Localpref: 501
                Router ID: 218.100.48.65

AS55836 Reliance Jio Infocomm Limited is announcing 49.35.128.0/17 at both locations at NIXI and AS9498 Bharti Airtel Ltd‘s routers are learning that through them.

Conclusion

This seems to be kind of a stop gap measure by Valve. It would be nice if they start announcing their IPs in India. I think Cloudflare is doing an island POP here, with their upstreams not announcing IPs outside of India. If they peer with most eyeball networks I doubt they would have a pay much for transit. I think there would not be any serious international traffic as well.

I think there is a default option for the download mirror on the client as well but you can’t set it once you change it to something else. So I couldn’t test it out. Also I didn’t have time to test out the other mirrors.

Well that was a fun few hours of poking around. I hope you found this interesting. 🙂

]]> https://varunpriolkar.com/2016/11/analysing-steams-network-in-india/feed/ 5 624