 Hello everybody. I would like to welcome Peter Hessler, who has learned how to use and abuse both BGPD and routing domains for BSD. I'm going to talk about the routing domains and routing tables this time. Everybody, give Peter a hand. So thank you. Yes, I am Peter Hessler and I am part of the OpenBSD project. And so this talk was not originally on the schedule, so I had to throw it together since last night. So please forgive me if I'm a bit rusty on some of these things. So we'll just start off. And so there's always a question of what is an R table and what is an R domain? So a routing table was the first thing that we added, and this was support for an alternate routing table that would be used. This could be just simply an overlay on top of what was already in the system. You could not have conflicting IP addresses in this. It was only for routes to different places over different links. Primarily this would be used for like a policy-based routing system where you want to have all of your IP telephones come in on one interface and always use a specific interface that was low latency but slow, whereas you want your web traffic to go over a different interface that would be faster but possibly high latency. After we added routing domain support, then you can have multiple routing tables belonging in a single R domain. Then so what is an R domain? Our domain is a completely independent routing instance, and this is a virtualized router essentially in your machine. You can have conflicting IP addresses. You can do several hundred if you want to. An interface, however, can only be assigned to one routing domain at a time. This is also part of how you know when a packet comes into the system which routing domain does it belong to. Routing domains always contain at least one routing table, and this is the normal routing table in your system. Our domains was first added in OpenBSD 4.9 that was released in late 2009. Originally it was for IPv4 only, and sadly there was some work but it was not finished to include IPv6 support, and that was finally finished in time for the 5.5 release of OpenBSD earlier this year. There are related terminology that is commonly used in the networking field for this. There is what is called VRF light and VRF. VRF stands for virtual routing framework. This is very common amongst the Cisco and Juniper terminology. VRF light is multiple routing tables and multiple routing domains. This is primarily done by hand. It is entirely local to the system that you are running it on. Any system that you connect to would not know and have no way of knowing if that system is running routing domains or not. That is primarily where I spent most of my time with the VRF framework is implementing VRF light. Then you have VRF which is also commonly known as MPLS. This is a technique that uses BGP and LPD. Most commonly large networks are doing this. For example, you can imagine that a nationwide cable, television and internet provider would use this to connect up all of their routers over leased links between sites. This would give you more routing choices or simpler routing choices for a lot of those links. There are a couple of things that you need to be aware of when you want to use routing domains in OpenBSD. You need to have a valid route. My recommendation is having a default route in the routing domain even if the routing domain is only used to receive traffic from. In the OpenBSD network stack, we check the packets when they arrive for a variety of things. The checksum valid is an actually valid IP packet, etc. One of the checks that we do is we check, do we have a route to send this packet to? Can we send it to its destination? We check this extremely early. We check it before PF so you cannot use PF to steal the packet and move it to another routing domain. At a previous job, I was professionally supporting this with the Vantronic systems. I would say about 80-90% of my support calls for routing domains was a missing default route. That should probably be one of the first things that you do on this. Otherwise, the packet arrives and you have no idea where it goes. Debugging can be painful because it is a different routing domain. It may not always be where you think it is. When you run route commands or other interface commands, they by default operate on routing domain zero. They can also operate on the... The routing domain can also be applied to processes, which I will show later on. So it's how do you know which routing domain that you're in? You need to explicitly remember this. When you build a mental image of the routing table, you need to know which routing table it actually belongs to so you know where it goes. Sometimes you want to keep the routing tables completely separate and the routing domains completely separate from each other and just do normal routing through that. That is the default use case and that works just perfectly fine through OpenBSD. But occasionally you want to steal some traffic from one routing domain and send it to another one. In that case, you would need to use a classifying engine and we use the PF framework to classify and to move the packets between the different routing domains. So I will give a very simple setup here so you can see just how it is used and kind of build up a small network in your mind as I go through the other slides. So probably the key is the R domain 1 will assign the routing domain number 1 to the interface RE0. You need to do that to create a routing domain. Once the routing domain is created, then you can start adding IP addresses and networks and routes and things like this. When you change the routing domain of a network interface, this will remove all configuration data from it. So ordinarily when you make changes to an interface, it leaves all the existing IP addresses on it. However, 10.0.0.10, it may have a very important meaning on a different routing domain and so we've decided that we just flush all the configurations of this interface and flip it over. So it's recommended that you do this first to simplify your configurations and you don't think that you have things that you don't really. And then we create a local host on LO1. That's just an arbitrary loopback interface you can use whichever number you want. To make things easier for myself, I tend to use the same number as the routing domain. It makes it much easier to imagine things. And then I add a default route here pointing to 10.0.0.1 and I add this to the routing domain number 1. And then because I want to be able to log in with SSH on this, we extended route to have an exec option to execute commands similar in the style of sudo, except this runs it actually in that routing domain. And so this is a slightly cleaned up example of what it looks like. So you see here you have the routing domain 1 has been changed, the rest of it is normal, same thing here. And then also in netstat to take a look at your routing table, you add the dash t1 option and that will show you the routing table only for that specific routing domain. As I mentioned earlier, if you run just regular netstat, it will show you for the default routing domain, which is also known as routing domain 0. And here's an example from PF to make decisions and things with routing domains. So this anchor block here is any packet that arrives on routing domain number 15 will have this rule set applied. It's a fairly simple rule set. Default to block, allow ICMP and we have a web server there apparently. This line here will, similar to this rule, pass in traffic from that arrives on routing domain number 2 and it will retag it and send it out and routing domain number 4. And then this line here allows us to do a net, a net, so any traffic coming from 10.0 slash 16 going outside of the network, we net to the egress interface and will be put out on our table 20. So as you can see you can add quite a few different things with PF. You can use it as a selector and as a classifier so you can move the packets. So what are some things that you would... So in a very common scenario in VRF Lite is that you would have a shared infrastructure. You would be, for example, an ISP. You would have a variety of customers connecting into your network directly and they, like all customers, use 10 slash 8 as their IP address range. And so they would want to come in and not have their traffic conflict with each other. But you have backup servers that all of them use. You have syslog servers that all of them use. You would have whatever services that you're providing to them that all these customers would use. So you would need to do... So you would use PF configurations to allow these sort of things in the classify based on your needs. You can have... And as the net two example shows, you can have many, many, many clients of yours go through multiple routing domains and then be sent out another one so they can share the same bandwidth pipe. So, for example, this net two line was actually used by one of our customers. I've, of course, anonymized it. But that was used by one of our customers to provide internet services for, I believe, 10 client systems or 10 client routing domains. And they all went out the same IP address. And, however, all the traffic was fully separated in the routing domains on the inside of their network. Of course, you can put monitoring servers and whatever else you would need to share an infrastructure. Then you get to the full VRF or MPLS. It requires two different things. It requires a label distribution protocol and BGP to actually distribute the networks. You would use LDPD to label each of the endpoints so it knows which network it belongs to and they distributed this across the network. And then on top of that, you would have BGP that would distribute the networks aiming at each of the endpoint tunnels. That will be basically all of the full VRF for now. Because, as I mentioned, I spent most of my time in VRF Lite. And I simply don't remember all those details right now. So as we put this into production, we discover a lot of interesting things that happen to us as we're going through this. The first one was the Routexec. And that was originally just a hack tool for us just to set up things and just get it working really fast without having to add routing domain support to each of the individual daemons that we wanted to add to it. It turns out to be now the officially recommended way to start multiple services in this. When you have, for example, a web server and it is listening on ports, it's bound normally to the default routing domain, a routing domain zero. What if you also want it to listen on routing domain 20? Well, either you can use PF to take the traffic and send it across as necessary, or you can start up a completely second instance, possibly different configurations, possibly pointing to a different area with the Routexec. This is the primary thing that we use now. There are a few things, basically network tools, things that set routes, need to know natively about which routing domain they are. And there's a limited number of other daemons that need to know this, which I'll get to in a few minutes. Yeah, so adding our domain to an interface, as I mentioned, it does erase the IP address configuration. So it's highly recommended that you do this first thing before you start setting up the rest of it. You also get into the question of what if you have your 10 gigabit link into your switch and you want to have multiple VLANs on different routing domains connecting over the same link? There's no issue there, provided that the child and the parent relationship is actually fairly split, as it is with VLANs. So there's no issues with that. You want to have native untagged traffic beyond routing domain 5 and then a VLAN sitting on top of that beyond routing domain 20. And then you run into Carp. Carp is actually, it's mostly a full interface, but there's enough that bleeds over to its parent. And so the Carp and the parent interface for Carp must be in the same routing domain. However, you can do Carp on VLAN on physical, the Carp and VLAN can be on one routing domain, and then you can have another set, another set, another set. There's no issues there. FTP Proxy, we ran into a problem with this. Originally, FTP Proxy only allowed you to change one of only the destination routing domain. Turns out that this customer wanted nothing at all to run on the default routing domain. FTP added a lot of very, very interesting things that we had to add in support, similar to what Ted was talking about in his talk about hostile environments. If you can change something, force it to change and see what breaks. So they wanted to do FTP from and to different routing domains, so we had to add support for this. Yeah, so the source and destination routing domains do matter for your thing. NTP became a very interesting problem for us because, as I mentioned before, normally you would use route exec to run the daemon again. Well, what happens when you run multiple NTP daemons on your machine? They start fighting each other and time gets wildly out of sync. On my testing laptop, I had about five different NTP daemons running and after 30 minutes of wall time, my clock was about five months ahead. Yeah, so that gets very, very ugly. You don't want to do that on a machine that is either pretending to use time or pretending to serve time. So I added quite a bit of configuration options to NTPD that allow you to, for each of the individual listen, server, peer, whatever, to individually select which routing domain that it's using. So you just run one NTPD and then you set up all your configurations. So you want to listen only on the external on Routing Domain 20, but you want to have individual services on Routing Domain 3, 4, 5, and 6. And then, as I showed earlier, you have the on our domain option and that was added after the initial release so we discovered that we just wanted to match on all packets that arrived in on this Routing Domain that allowed us to radically simplify a lot of these rules that we were writing for them. So yeah, so best practices when you're running this is you definitely, definitely want default routes or a full routing view in each Routing Domain. This will bite you in the ass. It will be very, very painful and it's a very, very common mistake. As I said, 80 to 90% of all of our support calls at this company were due to missing routes and it will confuse the heck out of you. So just do that. Take a look at what tricks are available in pf.conf and see what you can do to simplify your configuration and to allow you to get a more accurate view of what you're trying to do. And definitely spend as much time as you can in the planning stages both to minimize how many Routing Domains that you actually need to use and so you have a good solid understanding of what your network will look like after you have started deploying this. It is complex. A lot of people really want this feature because it is pushed quite heavily by some vendors and by some consultants who think it is a very neat idea, which it is, and they think that your network needs it, which likely does not. If you have control over all of the IP addresses in your network, you very likely don't need it. You can do everything else with normal traditional firewalling and V-Lanning and segmentation. But if you have multiple links to third parties and you don't have control over what IP addresses they use, especially if they're using internal RFC-19, 18-style addresses, then Routing Domains could be a good solution for this problem. Okay, so I ran faster than I thought. I do want to give special thanks. So Henning, a browser from OpenBSD project, he added the initial multiple routing table support to the system. He actually did that originally not for this feature, but so he can get rid of the Route 2 option in PF. Unfortunately, he doesn't solve all of his problems, so he still has to keep it. Claudio Yecker, who is actually giving a talk right now about iSCSI, he actually wrote pretty much most of the code and explained it all, did all the translations of the Cisco speak to what we called it and then answered all of my assigned questions as we were getting up and going, and he's the one who really beat into my head that we need to have all the proper routes. And I definitely need to thank Reich Flotter, who actually did a lot of the work on integrating this into OpenBSD, did a lot of the testing with us as well. He also funded this with the company that he owned at the time. So, any questions? No questions? Everyone understands? Nobody understands. Excellent. I'm of two minds with routing domains. It is really cool and that we needed to add support for IPv6, which I finished up at a recent hackathon probably about a year ago and because we need to make it full-featured. On the other hand, for most networks that people have, this is overkill. You don't need it. It's very similar to running a full BGP installation for your house. You don't have a dynamic routing at your... For you! OSPF is one thing, full BGP at your house is another. Most people simply don't have that type of networking at their location so the technology doesn't make a lot of sense to be used. But when you do need this sort of thing, then we want to make it as powerful as possible and as useful as possible for you. Sorry, we have a quick question. Stop clapping! It's possible to see all our routes for different domains in one command not to check everyone. No. You would have to enumerate each of them separately, yes. Okay, thank you. But when you do run the ifconfig command, that will tell you all of the interfaces that have a routing domain attached to them. I'm sure I understand correctly first. So you essentially have the normal writing table, everything normally, and then you create the traffic domain which are completely isolated to traffic, correct? Basically, yes. So when you start the machine and you start the networking layer, you get a routing domain for yourself and by default it's routing domain number zero. So that's the default that it starts up with. If you do nothing, then nothing changes and everything behaves as you would normally expect. When you start adding new routing domains, what happens is that that interface gets removed from the default routing domain. It gets placed into its own separate one. And so the traffic cannot cross over unless you use a technique like PF, or you can route exec, or you can even do a loopback from your machine out to the switch and then back in on a different routing domain if you want. It makes sense sometimes. You have to do that sort of thing on occasion. Yeah. You use OpenBSD, but what's implemented in Citrix Netscare which is based on BSD? Yeah. Which I don't know, they maintain that. I'm not familiar with it. I haven't used it myself, but it's probably very similar to what Cisco calls Nites. They're the ones who wrote all the standards for this and wrote all the original documentation. So most people try and behave in similar ways to what Cisco did. Okay. So Citrix called that traffic domain. Yeah. The same thing. I am guessing so, yes. I don't know, but what I was surprised is if you look at the BSD version, it's free BSD version seven on Cisco, on the Netscare, and I have no idea what they meant. So the question actually is, what about those vendors and maybe the other OSs? Can you just give a brief overview if they implement the same thing? Yeah. So as I mentioned, so most of the hardware router vendors do this sort of thing. They call it VRF. And so Cisco certainly does it. Juniper does it. 3Com does it. I think Alcatel does it. And all the major ones do it. As far as free software goes, FreeBC has it. I don't remember what it's called. I think so. It's related to fibs. It's multiple kernel fibs is important. And that is what we call a routing an art table is a different fib. It's part of it. I just don't remember the name. It's not very well known. And Linux certainly has it, but I just don't remember what it is because I avoid using that working on Linux whenever possible. Any questions there on the side? One more? One more? Actually, any questions? Yeah, just to get clarification on the thing, a traffic exchange between routing domains. You said the only way to do this is using PF, right? Yeah. Okay. Did you choose to use PF to have stateful routing? While PF gives us stateful routing and that makes things a lot easier, the main reason is because we did not want to create a separate classification engine. PF already knows all of this, and PF would already have to deal with moving packets through a routing domain even if it never changed routing domains. And so it does not make sense to us to add yet another thing. Because for example, we moved all of the Alt-Q classification into PF and we moved away from the separate NAT table into PF. And so we're using PF to tag the packets and to make those decisions for us. Okay, I understand. Because I had a similar setup some time ago with Junos, and they call it routing incentives, but they bragged it badly because they don't have that stateful routing. So in the Junos case, you have to create a route back and forth between the arguments which makes them practically unusable. Yeah, exactly. And then we wanted to definitely simplify our lives and we figured that since PF knows so much about the network, then we just do it in there. Okay, makes totally sense to me. Okay, other questions? All right, thank you.