 So yeah, good evening everyone. I think everyone is tired and already left, but it helps The stream is being recorded. So hello to people on the international space station and So today I'll be talking about automate your IX routes of a config and How many people in the audience are familiar with an internet exchange, can you raise hand? So it's not everyone. I'll just quickly explain what an internet exchange is An internet exchange is a place where networks interconnect the autonomous networks interconnect So an autonomous network can be any ISP or a content player These are the places which are actually responsible for for interconnecting and and that's where a large number of data flows from Content networks towards the eyeball ISP networks it's common to call ISP networks as eyeballs because that's where eyeballs are and In India, there are multiple exchanges. So in back then in 2004 Nixie was started, which is an international exchange of India partially government then in last four five years, there have been number of smaller exchanges which have which have come over and The reason for those exchanges to exist is because they are facilitating an interconnect of content players with the with the eyeballs Near to Bangalore, I can imagine would be probably Mumbai and Chennai. So Chennai you have You know, you have Nixie Chennai. You have extreme IX and in Mumbai. There are lots and lots of exchanges. So it's Nixie. It's extreme It's Bharat IX where I'm contributing as a volunteer and You have you have decades you have M6 India and so on. So today's talk is about how you Automate your route servers config. I'll give an introduction. What an route server is and and how it works So first how how networks interconnect at internet exchanges? So imagine a network like say Google or Microsoft or or or LinkedIn Interconnecting with eyeball networks like say Airtel or ACT and so on at an exchange So there are two possible options one is bilateral peering where members all connect on the same switch Which is the which is a layer to switch by design. It has to be layer 2. It should not be layer 3 so members connect on the layer to switch and Then they just start creating BGP sessions So you have say two members you they both create BGP sessions with each other a bilateral BGP session and they start exchanging routes So that's how it works. The other option is multilateral peering in multilateral peering members still connect to the shared peering fabric But instead of creating BGP sessions with each other they create BGP session with a route server It helps by reducing the number of sessions because now if you are say 10th person at IX You don't have to manage creating of BGP sessions with all other nine members You can just create with one one route server or possibly two route servers for redundancy and and get get going So what a route server does it's important feature when you peer with route server route server reflects you all the routes So imagine an internet exchange with around 10 members A route server will reflect all nine members to you and your routes to those nine members It will not change the next hope next hope Next hope IP. So whether you have bilateral session or you have Have multilateral session via route server. The traffic flow is exactly the same There's no difference in traffic flow the real traffic never flows through the route server traffic always flows through the switches It's just that route service facilitating the exchange of exchange of data exchange of routing data It is often desired that Exchanges route transparently so you should not see the route server AS number in the in the in the AS path It varies based on exchanges. I think probably if I have to put a guess probably 80 to 90 percent exchanges Follow this so you will not see their AS number in the AS path There are certain large exceptions as well. For instance, HKIX Hong Kong internet exchange It's route server does appears in the AS path In case of India Nixie's ASN does appears in the AS path as per their policy For all the newer exchanges which have come over they do it transparently and it has a effect if you have Route servers AS number appearing in the AS path. You are unnecessarily Extending the length of ASNs and now as AS path let's length is increased the path may not be followed while it should be ideally So so it has a effect Again often desired feature of a route server. They should do filtering based on IRR If you are not sure what IRR, maybe you can check out my yesterday's talk in detail about routing security that covers IRR in detail So it is desired that route servers do filtering based on based on route objects and AS sets And again as a last feature it is often desired that route server offers BGP community where one can control route announcement This is especially true with super large content players who are very much worried about announcing their routes to a route server as The announcement can go beyond a certain point and they want to control it They don't want a scenario where someone is is just exporting those routes in a different region So they want that control and I can personally tell you the fourth was the reason we started working on this project Because other two options I can still make them work without automating, but Fourth option is really painful IRR is painful, but still one can somehow manage so The challenge as I said, how do you generate filters based on IRR? And this is not a one-time thing you have to do it at a regular basis So it should be either few hours or at max a day You should regenerate the whole config of filtering with that and the other thing of course to offer the the peer specific export rules By design an exchange route server Accepts and has import policies specific to peer So if if if I am the route server I would have a specific policy for pure one specific for peer two specific for peer three on the import side But I would have a same common policy for the export, but if you want to offer The the BGP communities control this can't be true anymore You have to have separate export policies and it becomes very tedious to do that by hand Solution We use a tool called a route server. That's a open source tool The developer of the tool is one of the engineers at Cloudflare and he has written this very excellent tool On and it's there on GitHub along with the documentation It supports generating the route server config for both bird and open BGPD Open BGPD is still relatively new while bird is is quite old and advanced. It's it's really stable package And lots and lots of exchanges that at large scale use utilize bird So here's how it works you you are supposed to set up the package and initialize it then you just add the peer details in the client dot eml and You generate the route server given config so you define whether you are using birds It will generate the syntax for bird whether if you want it for for for the open BGPD It will generate the syntax for open BGPD and so on And of course it checks the config and then then you push it So just some basic syntax for for you know commands installing on DB and you went to in our case I can probably take a minute to explain in our case. We are running LXC containers for running a route server. So it's two different set of hardwares running LXC containers for for the a route server and some other things including including looking glass and so on and The config is generated on locally on the laptops for admin So it's two admins involved right now one myself and then another my colleague So we just generate the config by hand and then just push it push it by using Ansible is utilizing an ansible role for it Here's how clients dot eml look like it's extremely simple easy to read files. So all you define is ASN So for an example here, we have put AS 22 you define AS set Which is the set of ASNs behind it if you are aware So if we have a member ISP joining in the exchange saying my AS set is AS hyphen My my name, please allow all the prefixes from that. We would just put it here Although as a practice, we are trying we are trying our best not to not to do it and instead make the package learn that from the peering DB because then it stays up to date all the time and If you don't define a set by default it will pull it from the peering DB Which is essentially a very big database of all the all the interconnection world if you want to see which Network interconnects where you can probably look at peering DB. It's it's good data with respect to ASNs internet exchanges as well as the facilities You define the you define the IP address and it supports IPv6, of course And you can also define certain other other things in the syntax So here's an example of generating the config just to air out server and here you write the demo name which you're going to use IP version and then the Destination where you want where you want the config. So we are just using the destination as the as the as the location of bird config in this case if you want you can use a separate location do a diff before before committing it on on production and Same thing for for IPv6. So you can set the same thing up with a cron job and that again that again helps by by Automatically generating the filters at this point of time in Bharat IX. We are running a cron job at every hour on On the on both the route servers. So both the route servers Run regenerate their entire config and that's at a gap of 30 minutes So one one route server would you know just do it at the zeroth minute while the other one will do it 30th minute We'll rewrite the whole config and commit it and then proceed and there are checks and checks and balances in place that if just in case Assuming that internet connectivity is broken or something else is broken where packages not able to generate config It will it will hold the process So the key advantage Besides route filtering here is the BGP community support BGP community as you might have learned over time is these are simply the tags which are transitive in nature So using these tags one can define certain things So I can announce a route with a BGP community tag and the other network can interpret that tag based on their routing policy So at ISP level it's very common to use certain communities like a black hole community, which is often 666 So if you are connected to an ISP say AS 1 2 3 It it is quite common that they would have implemented 1 2 3 colon 666 where if you originator out with that tag They would just drop the traffic if you are under DDoS. So that helps There are a lot of other use cases for BGP community Especially with the fact that you can limit your announcement. So ISP may give you a community saying Tag your routes with community X and we'll make sure that we keep your routes only in India We do not announce them outside India or we keep them in Asia and the same thing on the other side as well ISPs also tag the routes they learn and they announce you. So they may say whatever Routes have a tag XY. We are learning in Europe Whatever tag has whatever route has a tag XY to we are learning in Asia and so on So now you can make automated policy where you prefer ISP 1 for sending traffic to Asia ISP 2 for sending traffic to Europe and so on So here's an example and this is totally out of box from from the from the air out server We we didn't manually edit any of these things. So it will generate these communities to work It will generate these communities to work on the on the project on its own So there are basic options. So you have the do not announce to any client So you can announce a route to route server and if you are testing for some time you can tag Tag it with that then you have option of do not announce to peer where you can tag your route with zero column peer So imagine exchange as as I was giving in the example with 10 members You want to announce your route to all other eight members except one the ninth member So you can say zero column nine and now the ninth member would not receive your route So it it significantly helps us now you can control your route announcements And it's common to use these communities if you know that one member is leaking your route or one member has a Has a port capacity issue at an exchange. They have other options including prepending So you can prepend once twice or thrice and and that again that again helps to control and limit the announcement At this point of time in India It's not that common to use communities and that has more to do with the Indian ISPs not not not training communities not implementing them for the customers with but it is it is it is It is picking up over time on the exchange side It's it's really important and much easier to deploy against say Deploying it for a large IX ISP Some of the other features of the open source package which are worth mentioning it has support for out-of-box support for ROAs So we have a full RPK implementation in Bharat IX which is resource PKI So anyone any route we receive we validate it for ROAs if we see an invalid ROA Which is route authorization resource authorization object? We are going to just deny it if if ROA is fine, or if there is a missing ROA will accept it So that helps it has support for graceful shutdown of BGP sessions, which has been a new RFC which supports Signaling when you are shutting down, you know shutting down your your BGP demon because Other side may be a fully loaded router which may take a few minutes for to reflect back and maybe black pulling routes for meanwhile So instead of instead of heartbreaking you can always do a graceful shutdown It has integration with peering DB for asset for pulling the asset I just mentioned that and This is this is a preferred way We we try to force all the members to update their peering DB records before connecting to our IX Often it has been our experience that members do have peering DB record, but they miss putting a asset So we always always suggest them guide them to do that It has integration with IXP manager, which again is a is a open-source project coming from INIX exchange in Dublin and It's it's one of the very advanced and good projects where INIX was putting up softwares for Managing their exchange and eventually they ended up in open sourcing it. So it's it's there on GitHub and We overall we tried to use that But it just didn't make much sense at this point of time with a small scale since our IX is just in one data center One single rack. We didn't need it complication to start with but for larger exchanges. It makes much more sense to use it It supports full RS config generation based on peering DB record What essentially it means is exchanges do list themselves on peering DB So if you are one of the exchanges who is not using this software and wants to start using it All you can do is just point the software towards the peering DB record And it will see what peers are there at your exchange and will generate the required config So it's very this this option makes it extremely easy to migrate to the the route server Although you will still have certain challenges here, which would be if you have Passwords on your BGP session. You still have to update your config file to add them And it has an Ansible role for configuring it via Ansible We utilize that heavily at this point of time So we just have clients or TML sitting on our laptop We make the changes we commit it push it on the git and the server automatically pulls the change and then generates the config within 32 60 minutes 60 minutes time So it would be either 0th minute or the 30th minute as per the cron job Quick references so a route server project. That's the name of the Pier key is the name of the engineer who is who is who has who has wrote it The documentation is aware on read the docs Ansible role is given over here IXP manager if you want to try something more advanced it IXP manager includes all the feature of a route server Along with a web UI along with along with lots and lots of things including Port graphing of the members etc. And of course the link to Bharat Ajax if you're interested if you're running an IP network Please do contact me, you know, feel free to peer at the exchange We are we are trying to make it work at a community level That's all right. Thank you any questions I'm I'm trouble. I'm a member of the internet community What happens if you can't reach peering DB? If you route server for some reason doesn't have a route to peering DB. It's so We have we put a custom script in place Which will just hold the process of generating the config for that time It will wait for the cron job to run again And since the cron job is running every hour, it's usually not a problem when you're making changes on the config It's often not at all urgent because you are adding a new member So new member while has waited for few weeks to get his cross-connects another thing done can of course wait for another hour Or two while the internet connection goes back online or the peering DB comes back online Related question. Can you trust peering DB? Well, uh, if beyond peering DB you may also ask the same question. Can you trust IRR as itself? So anyone can go and register anything in IRR. So it's just that uh In absence of these systems it becomes a total chaos. So you can have those systems in place The if you relate the number of route leaks again one has to do it in a loose term But I think a large part of route leaks or the problems which are there in say routing table are not documented on IRR in the same way I've yet to come across a case where Where people write something crazy on peering DB to influence influence the networks Plus of course when you are connecting it in IX We always, you know, we'll look at the company. We'll look at whether it is a real network or not We'll look at their their domain name their email addresses from where it's coming if it's a totally strange email Requesting for for a totally unrelated AS number. We would we would request them to send mail This has been one of the cases where there was a large multimillion dollar ISP who were operating with different names and different AS numbers So they contacted us with one AS number and when we see on the documentation the email was from a totally unrelated domain So so we cannot trust but I think we can we can add some some intelligence on top of it Any other questions All right, thank you Thanks a lot and that probably ends day two at rootcon. I hope everybody had a great time