 Hello, everyone. I'm Rupa from Cumulus Networks and for those of you who don't know what Cumulus is, Cumulus is a derivative of Debian for network switches and I belong to the engineering team in Cumulus. I work on the kernel and other networking packages in user space. Happy to be here, happy to be seeing the Canadian summer actually. I've been here multiple times, numerous conferences and it's all been in winter. Okay, to set the tone of the talk, I wanted to just give a high-level overview. So this talk is about open networking revolution that happened or is still happening in about six years, around six years now, and how Debian has been at the center of this revolution. Thinking of Debian just not as a server OS but also a network OS. There are a bunch of changes in the two operational models, but fundamentally, we have tried to bring Linux to the networking world and shown that you can run a server OS on a networking box. And it's about a journey of building a network operating system with Debian and also how extending Linux hardware acceleration model to network switches. I'm not sure how many people are know or know about open networking, so I just put this slide up to give some context. So this is a sample modern-day data center network topology. It's called a class topology. Basically you have racks of servers and then the Tor switches and on top of that the spine switches and there's many more things around it. But so when I talk about closed network devices, I'm talking about the Tor's and the spine switches. So these have been always been the Cisco's and Arista's closed box switches and routers which can do terabits or packet switch and switch packets in hardware. These have specialized switch ASICs which are capable of doing this. And the server racks, as you know, they run Debian and any other Linux distribution. So this talk is about Debian moving up the rack, basically running Debian not only on your servers but also on your Tor switches and spine. So all this started with the open networking movement and cumulus networks was very much a part of it and kind of started it. Basically networking boxes with these high-end chips were mainly closed boxes with the CLI and nobody could access anything other than the CLI. You program the network using the CLI. So the open networking revolution actually brought desegregation just like what happened to server world basically separating the hardware with the OS and then you have an open box where you can run networking apps on. So basically bringing the Linux ecosystem to networking. It was about leveraging lessons from the server world, open bootloaders, getting more platforms adopted in an easier way, open Linux ecosystem and Linux API which would help your apps move to different Linux devices. So in the open networking landscape today, the NOS architectures. There are two types of NOS architectures. Again, this slide is again laying context for what I'm going to talk next. So with the open software model, desegregated model, you have two types of architectures, one where it like what cumulus does, basically use native Linux kernel and accelerate that to hardware. For example, you program routes in the Linux kernel like you do on a normal Linux box or a server and there's a driver which offloads that to hardware, Switch A6. And there is another architecture which is actually run by most of the other vendors who are now pressurized to join the open networking movement to not lose control over their software or their SDK to keep some control over the closed parts of their SDK. And this other architecture actually provides a hardware offload API which apps can, networking apps can write to. So there are pros and cons to both architectures. Well, the native Linux architecture actually wins overall. You basically your applications written to a Linux API, like for example, Quagga or LLDPD for that matter, which is using the native Linux API or API to program network interfaces can work on a server or any other networking box. On the other hand, the closed or the software stack that is being pushed by other vendors is basically using hardware offload API apps directly write to it and that makes your networking app non-portable and you will have to rewrite the app to other network chips and hardware. So advantages of using the Linux networking API is basically, as shown in the picture, you basically run your same applications, your Quagga routing suite or your LLDPD or DHCPD, the same application runs on your server or your switches or any other network device. With this, you're actually bringing the server and the networking communities together as well, uniform Linux networking models. You don't want to create a bond or a bridge in a different way on a server and a different way on a tour. This unifies all networking. It minimizes operational problems. You can use your same monitoring and debugging tools everywhere to monitor your servers and your TORs and other network devices and package management. Package management, as we all know, it's not a simple problem. In this picture, we show servers and all these use the same code base and the same package management and it's Debian everywhere. So now, the next few slides are, I don't dive deep into a lot of things, but it's a high level rundown of all the packages and all the features that we touched for Debian and to make it a successful network operating system. So why Debian? As some of us, there are other cumulus folks here. They started with the ODM PowerPC switches from Taiwan and, yeah, they were PowerPC and Debian was the only architecture at that point or only distribution that supported PowerPC at that point. And Debian has the largest ecosystem of networking applications. You can actually search for any networking protocol or any networking package and it is there. And it has a growing and strong community. Then came the need for a open networking NAS installer to standardize NAS loading across open hardware. And this was critical because if you don't get this in and there will be a lag in movement. So this is another project that cumulus worked on. It's called ONI. It's the open networking install environment, which is today almost a de facto standard for open networking boxes. It is open. It is an open compute foundation. And, yeah, now most of these open networking boxes from or desegregated boxes from Taiwan or any of the ODMs, they come prepackaged with ONI, which gives them a choice to run any network operating system. Kernel and platform bring up. Again, these were closed boxes, no hardware specs, very difficult to port Linux to these in an open way, like any, like pick up Debian and port to it. So it was not, I've not personally worked on a platform bring up, but there are many people who have in the company have worked on this and yeah, it was a laborious process, getting specs, reverse engineering and so on. So this says that this industry needs a lot of standardization in this area about hardware standardization to get platforms integrated quicker and early. And there is an effort going on, which is to extend ACPI for network hardware. And this is again bringing, leveraging whatever was done for the server world to the open networking industry. So there's a spec, there's a link to a spec describing this. Netlink. I had to add a slide on this because, so Netlink is the protocol that we know, which is a protocol between kernel and user space, and it's kind of becoming the de facto network configuration API for Linux. Every new kernel API that you add for Linux networking has, is Netlink. So, and we have reused Netlink to a great extent. We, it for us, it's also an offload API, basically offload Linux networking to our switch ASICs. It's configuration monitoring. It's also become a very powerful tool or powerful API to add monitoring capabilities to, to Debian or Linux in general, and for network troubleshooting as well. So in the course, we did enhance, we had to enhance a lot of Netlink libraries for, say, scale, like Libnl. And we did take these packages from Debian, and we had to enhance them for scale. And we also had to, we actually ended up writing another Python Netlink manager or library. So networking bring up. So since we have modeled a switch ASIC as a Linux server, or a Linux, or any general Linux operating system, we represent every switch port as a network device. And so now your switch looks like a Debian server with 128 Nick ports, 128 or more Nick ports. And these Nick ports are now 100 gig, 40 gig, or whatever these high-end switches are capable of. So there are many switch features that have been deployed by other vendors, like port ganging, splitting ports, and so on. All these became necessary when we brought necessary elements to Linux, when we got this on the open hardware. So the challenges here were again scale. When we started picking up packages from Debian and started running on these, now your 120 Nick ports and LLDPD on servers runs on two or three Nick ports, ETH0, ETH1. So some of these challenges with scale we fixed LLDPD and many other packages, Libnl, DHCPD, move them to Netlink or upgrade them to latest packages where they were already fixed. L1 config. Yeah, there's lots of SFPs. I'm not an expert in SFPs, but there's lots of SFPs and lots of other parameters for link settings, 100 gig, 40 gig, FPC settings, and so on. So that also is critical for a NAS. And tons of networking attributes on switch ports for bridging, for STP, lots of timers, tuning of timers, and so on, a lot of networking attributes that need to be added. Now, offloading all this to the Switch ASIC, most of these vendors still have proprietary SDKs, which mostly run in user space. So we had to map the Linux API to the SDK API. And for example, a bridge FTP add which adds a forwarding entry into a bridge had to be, yeah, mapped to an SDK API which programmed the hardware. So the user basically sees the Linux API. You do everything what you did on a server, exactly on a network hardware, and we seamlessly offload the networking functions to hardware. Switch ship initialization. This is something different and it's required in some of these vendor SDKs or APIs are proprietary and closed. So we have written a bunch of tools and scripts and so on to do it in a Linux way, a file-based config file. And this is also being worked upstream to include this natively in the Linux kernel, in the Switch Dev project. And the Switch Dev project is basically natively in the kernel handlers for offloading to hardware. So yeah, this API is called DevLink. It's again a generic Netlink API. It's available in latest IP route two versions. Tunables, network parameters. So on a server, many of these parameters are default to the Linux defaults. Now on a Switch, it's very critical to change some of these defaults because, yeah, because Switch is forwarding and also something like ignore routes with link down. This is, yeah, this is critical. Basically Linux by default does not forwards or routes through ports with link down. And so this is something that we got in the kernel to, because it was critical, you cannot, you have to ignore routes with link down interfaces. Networking demons. Like I said, Debian has, Debian ecosystem repos has network, all sorts of network. There's a ton more. I've not added here, like PTP and so on. So yes, the initial days was of porting Debian to a networking hardware was getting all these demons, testing them for interface scale mainly and interrupt. And most challenges that we faced were on scale because it's not only the number of physical ports, but now when you do bridging and VLAN in a Linux native way, you end up creating network interfaces on the ports. So we could easily scale to hundreds and thousands of network interfaces. Network interface configuration. I've talked about this in DevCon before, and it's a hard problem. It's again the sheer scale and a scale of not only the number of net devs, but it's also the number of networking attributes that you have to now configure on each port, STP parameters and STP. There are tons of STP parameters. I don't even know half of them. And in the same time, you have to handle differences between servers and switches. So the pressures here, pressure or challenges here that we faced were basically from the users. Of course, we are now trying to sell this to people who have used closed boxes with CLIs, selling them a config file and telling them to configure thousands of interfaces or interface attributes in a flat file. Yeah, that did not go that well. So, yeah, and people do, there is a lot of people who love Linux and the files and config files and all that, but an automation, but there's also a group of people who do not want to move from their CLI. So, yes, we ended up doing two things. One is rewriting IF up down to IF up down two, just for scale, dependency tracking, because people didn't want to do dependency, interface dependency on their own. So IF up down two, it's in Debian now. And there are many differences between how you set CCTLs on Debian and the default CCTLs on Debian and servers. So what we have done is we have controlled these differences via policy files, so a user can drop in a policy to change the defaults. And that's what Cumulus does. We have a bunch of Cumulus default policy files to tailor to a network operating system. And, yes, there was a very great need of a CLI or many people wanting a CLI or people not wanting to change SNMPD config in a file or an LLDPD and interfaces on another file. So, yes, we now have actually a unified Linux file editor and manager, which basically translates edits files and appends to files and so on. Kernel networking. So we exercise a lot of kernel network features, bridging, routing, routing to a great extent, ACLs, and networking tools, IP route to bridge you tools, ETH tool. So we end up back quoting a lot of latest... We started from Debian and some of these we have moved to the latest upstream versions, or some of them we have kept the Debian version, but we back put a lot of patches from upstream. We do work closely with upstream, the Linux kernel community to add many of these enhancements to Linux. So talking about Linux networking features, some of these are important to a NOS and I've listed them here to complete the picture on what it takes to build a NOS. Worf. Worf is a virtual routing and forwarding instance. It's like a VLAN for L3 or for your IP addresses, isolation. It provides IP addresses, isolation. It's a very important feature in a network operating system. For example, there is something called Management Worf, which is a default feature in a networking operating system, isolating your ETH0 port from your switch ports. So ETH0 carries all management traffic and the IP address and everything, the L3 context is completely isolated from your switch port traffic. And so Linux kernel now has the Worf feature and the newer kernels, the newer Debian releases already have it. I've done two has support for provisioning Worf, also provisioning management Worf. And we have, by our defaults, the NOS defaults, we manage by I've done two policy files. Then MPLS. MPLS is another protocol, label switching protocol, sometimes in the data center or other environments. So we have enhanced the kernel data path, MPLS data path and I've done two also has support to configure it or to enable it per port. Then some of the other features I've just listed is QNQ, multicast routing, and lightweight tunnels. Lightweight tunnels is again a way to to configure your network tunnel interfaces by not creating a net device. Again, this was done to solve the net dev scale problem because on a NOS, yeah, creating 1000 tunnel endpoints, VXLan tunnel endpoints, you end up creating 1000 VXLan devices and that won't scale. So with this LWT, you can move to a single VXLan device. MPLS tunnels are no more a device. So this is something that is catching on. There are many tunnel net devices that are being eliminated or moved to a single device or you don't need a network interface at all. Routing. Routing, we started with the Quaga package from Debian and we had a lot of enhancements. We have contributed to the Quaga community and now there is a fork of Quaga which is called FRR which hopefully will be in Debian also. Overlays. Overlays is another net NOS feature, VXLan tunnel endpoints. So overlays is a good example or the Linux model of deploying overlays is a, it's a good example to prove the Linux native hardware acceleration model that I was talking about earlier. You can do VXLan overlays the same way that you did on servers or hypervisors, the exact same way you do it on a switch and you get the benefit of hardware accelerating it with a switch ASIC. Monitoring. Monitoring applications. Again, net SNMP, S-Flow and I'm pretty sure there are many others. We have picked up many of these from Debian as well. Again, I mean everything worked right off the box. Just the issues that we have had is with scale and number of interfaces which we have fixed or in some cases in net SNMP, I think the latest version in Debian moved to Netlink and that helped a lot. S-Flow is another sampling protocol. It's, yeah, now we have an upstream Linux API to actually do S-Flow on using TC on the servers and which can be offloaded seamlessly on a network switch. SystemD. We do use SystemD for most of our monitoring of our network services. SystemD worth enhancements is a recent addition. These are not upstream yet but at some point we do want to get some of our worth packages into Debian. The worth is an isolation, L3 isolation for a, you can isolate a process into a particular L3 context. For example, you can run multiple instances each having their own isolated L3 context. SystemD unit files kind of helped here a bit. The worth uses C groups. C groups to attach a particular L3 process in a, put a service in an L3 context and we use the SystemD C group mechanism to actually do this. Futures. We do want to push some of these enhancements. Some are already upstream. Some are, the latest version of Debian has already picked them up. And, yeah, new tools and managers that we have written to make things easier for deploying in a NOS environment. We would like to push it to Debian at some point slowly over the years. Yep, that's all my talk. Thank you. Thanks, everyone. Any questions? Hi. Well, I'm a happy user of Communist Linux for more than a year. So thank you for this product. I love it. I have a few questions regarding the relationship between Communist and Debian. For instance, when you look at the GitHub repository of Communist and you look at the issues and stuff, there are mentions of, okay, we will make a refresh from the master repo or we plan to, but there are usually no deadlines and I would like to have some clarification on, okay, how does that work? Do you have a private repo? And then periodically, when does that happen? Do you contribute your code to the open source? Yeah, so it depends on the project. So most of the projects we work directly with upstream, like for the kernel, we have everything going in the kernel directly, the upstream kernel, and then Debian picks it up. The one package that we are still trying to move to an open development model is I have done too, which we, yeah, so yeah, that's just purely because of no cycles right now, but that's the intention. And we want to move to an open development model. But yeah, we don't mind pushing code every day there is just that it's easier for us to right now push into a release and then when we have cycles move. Because one thing that we want to make sure is the lag is mainly because one thing we want to make sure that the changes we have done work on Debian, or what we do is immediately when we know that it breaks, we have to wrap into a policy file so that it's only invoked on an OS. So it just requires some cycles and that's why it's delayed, but we'll soon, we'll soon be having an open repo. I can elaborate slightly with up down to it is also in the released version that comes with every release is available as a Debian source package. So you can get the source that way as well. The definitely understand the point about wanting to have new features go into the public repo first, but you know, that's, that's a different thing. But all of the, all of the source for the actual release is available. Okay, thanks. And what about PTMD? I never saw any Debian package for that. And actually, I wanted to use it for our servers. Yeah, I don't think there is a Debian package for that, but we will probably be putting it soon. Yeah. Okay. And one last question then. Well, apart from, you know, for instance, imagine I'm using if up down to, but not on community switches, but on my server and stuff, where should I report bugs that I find? Where should I contribute patches and stuff like that? Should I do that on GitHub? GitHub is better right now. GitHub or Debian, we both, we monitor both places. So yeah, GitHub, you can send a pull model. You can use the pull request to actually send us a patch or yeah, or just email us. So there is, we have Julian here in there who's monitoring the GitHub website and thank you. Can you speak a little about how the cumulus kernel and ASIC drivers and other things are structured packaged and delivered? Sure. So the offload itself or just the distribution of the ASIC drivers? Okay. So you support a lot of different hardware vendors boxes, I guess, and with a variety of ASICs. And are these proprietary drivers and are using your own kernel package or are they structured as out-of-tree drivers? Yeah. So our kernel contains a lot of GPL platform drivers, additional drivers that sometimes we get from the vendor and most of them are GPL, but they are not upstream. Some of them are upstream. So you'll find them in our kernel package. We do ship a user space offload driver that is offloads to hardware. And that interfaces with the vendor SDKs. In most cases, those are closed. But yeah, soon we'll have, I guess, a switch dev implementation also, which will be based on a pure, completely open driver. Yeah, it depends on the vendor. And Melanox right now is the one vendor which has an open source driver. Thank you. I was a little bit late from your talk. Sorry for that. So I may be totally mistaken, but why didn't you mention OpenV switch, the project I mean? OpenV switch? Yeah. Why didn't I mention? So we, as OpenV switch to offload flows, because we don't, it's not an flow offload hardware. And we don't offload flows to hardware? No, I mean the software project. It has a similar scope at least partly with what you do. It supports some offload switching engines and ASICs probably. I'm not, I'm no expert, but I find it surprising that you do something very similar and still I can't hear a single word of mention. Okay. So I know there are OpenV switch, there are distributions which offload OpenV switch directly to hardware. So what we have, we do have an implementation where we take OpenV switch flows and they're translated into Linux forwarding elements. And then that gets offloaded to hardware. It's kind of the same thing. But if you want an obvious front end and you have a controller and so on, you can still use our box to offload it to hardware. But our default model, it runs with just Linux commands and the Linux API and we offload to hardware. Okay. Thank you. Hi. I do discover the product like a month ago. I was so happy to see that in the market. But I'm always curious about how do you get success as a company? It means because I know it's a lot of work and there's, I think, a lot of developers that you pay for that work. So my question is about the business model, if you want to share it, because it's always like good to know who people can go inside the open source ecosystem, especially when you are going more in the hardware place that is hardest that juice the software part. So the business model is we do sell support. We do sell, it's like any other Linux distribution, like our Red Hat is. We do license support. And yeah, that's basically the business model. So I'm asking a couple of questions from people watching an IRC. First of all, question one, how far are you with the VRF? Can I run a routing protocol, for example, BGP? Wurf and BGP. Is that what the question is? Yeah, we do have BGP support for Wurf and it should already be in the FRR repo or the Quagga repo. Okay, cool. And the second one, could you provide any ideas on performance for top of whack, as compared to, say, a Coen Cesco Nexus? No, I don't know. But I can get that information. I don't know on top of my head. Does anybody else? You probably know Nolan? Nolan is the founder. Yeah, hi. So the performance of like a Cesco Nexus, like a 3064, for example, will be strictly identical to any cumulus platform that has the same forwarding asic. And in which this case is a Broadcom Trident Plus. Yeah, Trident Plus. And so all of the actual forwarding is being offloaded, is being run in the hardware. So the CPU is just running things like BGP and all the various protocols. And most of these things have fast multi-core x86 chips in them. So there's plenty of extra CPU around and memory around if you want to run applications on it as well. So performance, they're mostly idle, usually. How logical are some of these pieces like you just mentioned that you have a Quagga branch or something like that? Would it be possible to run another software like Bird as an alternate routing protocol? Yeah, definitely. We have people doing that as well. Because it's an open box, you are. And because Bird talks Linux to the kernel again, it uses the same API. So why do you have a separate Quagga branch that you're talking about? So the FRR, the Quagga FRR repo, the FRR project. So the FRR project is again not a closed repo or a closed branch. It's again a community-driven fork of Quagga because of various other reasons. Because patches were not going in as. So but it's not a closed thing. It's again a foundation which is built around FRR. And I think it's part of Linux Foundation now. Yeah, okay. I misunderstood. I thought it was a cumulus specific branch of Quagga. No, it was again a community fork. Okay. Thank you.