 This is our last presentation of the day, and it's great to see you're out in this hall. Toby is going to be talking about moving from closed source to open source switch A6. So, hi everyone. I'm Toby, coming from Berlin, working at this company. How many of you deal with actual hardware switches like A6? That's quite a few people. How many run open source software on them? Run. So you get packets through the pipeline in a way that you define it. So, the agenda, I want to narrow down the scope a little bit in the beginning, and then let's have a look at what needs to be done by a networking operating system, and then let's have the whole ten year through the history via the APIs and noses, what's out there, what is actually open source and what claims to be open source, but maybe it's not. That's why this scope is as open source as possible. At least you should have an API, and it's not needing to sign any NDAs to get actually the stuff running. Because most of the things are nowadays still like having an open API, which is quite great, like a huge step from that point of view in terms of access to A6, but even more still require an NDA that you get the actual software that you can build your stuff yourself, and that's really not what we want to have here, right? So, from what you need in terms of nosing greens, you need first of all the hardware. This is the best thing you want to have, you should build it your own maybe, have the Gerber files, have everything to solder it together if you like to, and actually that's possible nowadays. There's the OCP, the open compute project that got handed in a lot of specifications, and you can go there on their specifications website and you can download it. The OCP is mainly driven by Facebook, but then later Google and several other big companies jumped on to that and brought their ideas in, specifications, and actually then some OEM switching vendors got into this as well and delivered the hardware. Like Facebook has its wedge switch, which is like fully documented there, you can get the files, you can even build it your own if you like to. Then the second part which is like the part where I'm more or mostly talking about today, the open source software part is actually like what do you need to actually run the system, you need drivers, you need the APIs at least, or maybe some SDKs out there, we'll see. So cutting a switch over or looking at a switch and then you find some key components on this, like very boring like power supplies, fans, and you have some temperature sensors on that, the switching AC itself, some embedded system where nowadays even though some Intel x86 CPU which resides in there, you have a lot of ports, some management networks and some management LEDs. You want to configure everything and everything should work nicely together that your switch doesn't burn because it's getting too hot, which may happen. One other thing is like that came up in the last year, it's desegregation, it's a huge topic sort of like, if you're former times you had your switch driver which was mostly closed, so each and every control plane software was basically very close coupled to that SDK. If you want to move to another hardware vendor, you would integrate this really, really specific to the other one as well. Nowadays we want to talk more about common interfaces. We want to really see that we actually can use one control plane software on as many as possible hardware, A6. That's actually happening and you even can take this common interface and have it in a way that the control plane software runs remote and then your NOS may contain like switch like local software that's running on the embedded system and your maybe centralized network operating system controller that's remote. So where this sort of happened, well there was SNMP at some point, some people may still use it but actually like more detailed APIs were starting around 2001 when a working group at the ITF started the forces working group. It's just like, it was very clean design, I'm not sure how many of you know forces? Two, three. So it's like they started with some informational drafts like two or three in the very beginning and then there was a long silence or actually not happening much in public I'd say. And in between there was like something happening in Stanford in the US. There was like the clean slate program found it and there were as well several researchers thinking the internet is broken and we need to fix it and deal with like some things and they wanted to open up actually the hardware because they wanted to solve the problem very, very low down there. And one thing is very nice to look at is the ethane project. It was a very, very starting point for the clean slate program where people actually dealing with like 20s which is a central controller kind of control plane and it was somehow a starting point you already ready like for open flow. And like with ethane and some actually vendors out there like Cisco, Tosh Telecom Labs and NEC there was siding as well and some others in the very beginning that supported this idea likely dropped some money as well and the whole clean slate program started and one of the major outcomes was open flow. Which you may like or not but it was like the first sort of thing that was trying to figure out how we can program the control plane, the data path from remote or like you can run a controller as well on the switch if you'd like to but the idea was to centralize everything to get actually something like use of the centralized knowledge and make things better. So let's have a look at the APIs. Openflow was one of the first of them. It actually sort of got off in 2008 then like with a paper from these researchers basically and the percentage of ACM sitcom of the openflow enabling innovation in campus networks because they came from and started in campus networks to really deal with actual people running their traffic through this network so they got complaints and everything and they had to make it running properly. The openflow 1.0 release was made in 2009 and then closely afterwards there was a release of Indigo V1 which is like the openflow 1.0 agent running on the switch and mainly dealing with the TCAMs only. TCAM is like the most powerful kind of hardware that can do everything sort of down there but it's quite limited so it's like maybe a couple of thousands entries if you can, you can have a bigger one if you pay more but like normally you have like a couple thousand entries maybe and excuse me, in 2012 there was openflow 1.3 released and Google announced hey we're running our network on openflow so they made it actually working for them. It may not be like this Indigo control thing that was running there but you get the idea, they took out the whole components running on the switches they took the control plane to the central point they made it high available obviously because otherwise Google couldn't run and then they presented this to the world. At least the idea, right? In 2014 a Broadcom released OFDPA, it's the openflow data plane abstraction it's like I think one of the things that sort of came out of this Google relationship which was based on the TTP, the table type patterns that came out as well alongside of openflow which you're looking at here right now it's the representation of the pipeline of the Switch A6 here and this specific thing it was Broadcoms and that's actually what was running on Google's hardware back then it got rid of some bugs and I think mainly focused on layer 3 parts I'm not trying to be sure about that but this actually is nowadays as well on GitHub you find some binaries that run on several switches you have the full API, several header files there and you can actually run your own openflow switch with this Come back to this later a little bit then in 2014 like really open source there was Switch Dev introduced Switch Dev is actually Linux kernel's hardware framework for switching A6 so you have a common infrastructure that's residing in kernel and you have the hardware specific driver that maps actually the patterns you want to have implemented to the A6 and the south so you're showing F4 routing there on top of Switch Dev oh sorry what's the status of reality? basically nowadays you can run anything on top what you can run onto Linux interfaces so in the user space you can run free range routing you can run manually your IP commands basically what's happening here is that Switch Dev exposes some interfaces like you would create some tap interfaces you just tap there Switch Dev specific so everything what's happening there is really mapped down to the A6 I kind of get that for something simple like I have to bridge the main maybe something like I think for the 6th routing exactly but if you get into something that's a little bit more edge like something like failover or HJA or ECMP or UCMP does it have that degree of support yet? I think ECMPs now they support it as well I think I read something with VRP these things a lot of these things are actually in the kernel the problem is just floating them having the drivers that actually understand what you're trying to say that was more so my question maybe sorry I should repeat this but does the Switch Dev have all of that kind of support to support the full richness of something like the free range routing? yes you recently actually done like device events you know somebody so in that sense like Switch Dev is just giving you the interface and everything what's happening up here in user space like what is configured that's part of free range routing so you don't offload anything off free range routing down here so there's no logic map down here this sort of static state there's no intelligence in that sense unless you have maybe something like ECMP then you have some hashing some very basic intelligence that does like what you actually want to have and yeah from the application point of view this is like you write a netlink mapper or something like that it's more advanced because you actually can run like each tool with this so the actual interfaces are mapped down as well as long as the driver supports this obviously I think Switch Dev was around kernel version 4.8 introduced sorry 42 and sort of usable at least for like one of the major driver or vendors that has this is Melanox at least for the bigger Broxies there's just some smaller Switching Asics support from Broccom and several others but the really bigger ones are like Melanox specific I think they're quite usable around kernel version 4.8 so let's go ahead and okay half time SAIs, which abstraction interface was introduced in 2014 anybody know this? it's getting less it's a common set of interfaces it's as well in the open compute project it was contributed by Microsoft mainly and it's sort of where health started freezing in the last couple of years because they really got open and it started with 14 header files as of version 0.9 and now has 44 header files to actually like try to deliver an abstract interface that can be used to configure any ASIC below obviously needs some implementation and in 2018 there was SAI Flex introduced the major reason is because there's around the time where flexible ASICs were becoming more and more popular because there was this company in the US that drove this P4 so Barefoot Indian their ASICs are fully programmable so this point of view of a static interface which actually it was in the very beginning it doesn't hold with standards anymore I'm really interested how this will look like in terms of switch def what we saw before because that's like as well quite static from at least the pipeline aspects and the primitives that you can call downwards to the ASIC so then in 2015 there was Open NSL introduced again it's Broadcom that's opening up some other APIs actually it's mainly their SDK it's sort of driven by Facebook again because they're using this I'll show you later but they're using this API heavily it's as well where Broadcom later on some next slides delivers the SAI but this is as well just a set of headers very very specific to Broadcom's ASIC it's as well on GitHub you find the binaries, the header files etc and you can program the ASIC with that now we're getting closer to today it's 2017 and there was introduced P4 Rancho I just mentioned there's like this whole bunch of how I can program ASIC now kind of thing actually like this it was introduced P4 and what Barefoot does with their Tafines it was introduced a little bit earlier because P4 the programming language what you will use is like very powerful and you can write your complete pipeline in that and in the very beginning there was like a Thrift based API and some other tools that you could use but then a couple of years later like in 2017 again Google came and said like wow that's cool but we want to have it slightly different and they presented together with them the P4 Runtime P4 Runtime is not something that actually deals only with this flexible ASICs it's just an API it's a little bit confusing that the P4 name is there because that implies sort of you can deal only deal with the flexible parts of the pipeline but actually you can actually deal with the static pipelines with this as well you just have to model your pipeline correctly and then you just deal with the static one because you will hand it hand to the P4 Runtime you will hand some parts which is in the P4 Infoblock I'm not an expert on P4 but this is like it comes out basically of your P4 compiler there's like the binary blob but defines the ASIC and there's the P4 Info which is sort of the contract between your data path and your control plane and that's actually used in P4 Runtime and so if you model like your if you have your model correct for the static pipeline you can use this as well on static ASICs this is then in 2018 last year Broadcom put SDKLT on GitHub SDKLT is the first complete open source software from Broadcom and you can deal with their Tomahawk chipsets it's just for the Tomahawks but it's completely there everything what you need is on GitHub it's like if you look at this this is like really crazy what they thought out they took a bit of a step to bring something up that's sort of comparable to the P4 because actually what you see here that's coming like from your logical tables API there's a transaction matter and the logical tables itself the logic tables in the end are something that are like preparing for a programmable Broadcom ASIC in that sense because like actually their ASICs are at least to some extent already programmable but they don't give you that out yet unless you're very very big I guess but this interface what you see there on GitHub will like their next SDKs will be using that interface and they're coming up with something that's called network programming language which is their counterpart to the P4 one and this will define the pipeline that is used here almost through the APIs because this year this is a little bit of a look ahead there will be Stratum the Stratum project will announce or open source its works which is currently happening in a sort of close source pro close source manner on the GitHub to actually harmonize everything bring in some ASIC drivers etc like SDKLT is there there's currently written an adaption to that and this part here what you see in the middle which is partly the P4 runtime you saw before and then there's GNMI and GNUI which is coming from the open config project which was as well started from Google a couple of years ago which defines a lot of young sets and models that actually then define a DRPC interface that you can use to configure your pipeline or your switch in the end so I think it's mid of the year if I'm not wrong when this will be completely open source anybody's welcome to join the project it's sort of like you contribute manpower and then you get access so what's used anywhere here like in terms of network operating system and if you remember there's something like you have fans, you have power supplies etc that you actually want to control as well and in 2013 so we jump a little bit back in history like we're done with the APIs there was open network links release and this picture actually coming from BigSwitch because they contributed that partially it's now in GitHub, you can build your NOS you have your platform extraction layers, you have your kernel drivers etc, everything is there I think around 14 vendors that contribute their drivers and everything to that GitHub repository that's as well in the open compute project you can compile it and you get an image that you actually can install anybody aware of ONI I should have maybe explained it here before but that's like the boot loader it's sort of pixie that you use to install an image to the Switch itself it's coming as well out of the open compute project so the image what is produced here is an ONI compatible image and you can run it on the listed vendors switching A6, what you don't get there is actually the most important part like fdpa or open nsl or like any library that deals with the switching silicon you just get the framework to run your Linux on the Switch it's bedabian-based so you can run anything what you want but hopefully you don't get the important part from that project itself but as I mentioned before on the Broadcom side there are some binaries you actually can drop there you just have to make sure that the version they mentioned maybe is correct in terms of what open network Linux uses plus there's like all the others just to name two like Edgecore, Delta and they're having the drivers as well that you can put onto open network Linux so you could actually start with something and write your own controller or network control application then I have here two examples of remote controllers like Open Daylight it was like I think shown here before I'm not sure if it was today anyone showing anything about Open Daylight and there was Onos presentation today I'm for sure and these two are really remote controllers so you would run open network Linux on the switches to like deal with the fans etc and these are actually dealing with your network management so I rushed through them because I'm running late and then there's FWOS which I mentioned before this is Facebook's open switching system you get the agent, it deals with an open NSL API so you need like open NSL from the one side and FWOS is on GitHub, you can compile it yourself you drop it on your open network Linux or whatever you want to use and run Facebook system there it was introduced 2015, has a thrift API and on top of that nowadays there's as well a netlink listener which used to be integrated there was a pull request that wanted to integrate that but they put it outside and actually used the thrift API then Open Switch was I think presented in the last couple of years here as well Open Switch was started from HP in 2015 HP I'm not sure what it was but they went away from Open Switch they still sort of contributed there but they handed it over to Snap, Rout and Dell but actually lies now at Linux Foundation it's still developed, it has as well a Linux integration which is in that sense just a netlink listener, it's sort of like again like switch dev but you may create just like a tab ports, I'm not sure I'm not familiar with that in detail but you get like 48 ports in your Linux that you can configure and Open Switch takes care of applying this down to the ASIC it runs on top of the SAI what you've seen before and it mainly supports Dell which is right now Sonic, that's the part where as well when the hell froze that's Microsoft's operating system it's the software for open networking in the cloud it runs as well on the SAI it has a very interesting architecture, there's some object libraries here that alongside Red's bus and you have applications on top that you can choose to run or not and that's very interesting I don't go into details here look on the github page, they have a very huge amount of documentation it's very interesting 2017 sort of, you run any distro with switch dev again like for Melonix ASICs in particular but like Fedora25 you could run on the switch and have a network operating system or like, I'm not sure if anyone knows a flat car that was last year Twithend 18 that was a major request to enable the parts that actually enable switch dev so you have the possibility to run your container Linux on the switch and deal with the network, right it's sort of crazy but a very very clean approach actually I just listed these two distributions but WN Ubuntu and anyone else I'm pretty sure they have support for this as well so you can run actually for a certain subset of all these ASICs you can run your own software we actually do our software as well we have a Yocto-based system we take parts of OpenNetwork Linux we started with this when OpenNetwork Linux was not there yet and we mainly did some like POCs and these things but actually we're now seeing as well like actually we want to have an open source system so you can look on the facebook.org side based on the OFDPI code that's not open source we have a netlink listener as well so you can run free-range routing on top you can run it integrated you can run it remote or you can run on us if you like that way more just take it we have several images there that run on switches we support a few right now but we extend this in the next year as well one more heads up at the ONF Connect there was Denos shown it's basically an AT&T draw check that still lies internal has to go through some legalization but it's in principle a rewrite from scratch of the Viata people's code and the same people actually doing this they're now at AT&T and want to open source Denos this year so this will have like a full in that sense more telcosanthric applications on board but it looks very very promising as well so a little bit of conclusion there's like many ways to start things like you just have to find the right pieces you can come to me and write me a mail I can show you some things likely but it's actually obviously two things are there that are fully open source which is like Broadcom never thought this ten years ago that Broadcom will actually open source something like that for this Homo Occhips it's like wow it's sort of crazy and switch stuff really a really amazing thing one second and what I think is a little bit problematic is like these huge golden players we had heard Facebook, we had Google we heard Microsoft they're all doing their own thing how would it be like if they would do something more together in that sense but it's not happening they even have like open compute project, open network foundation the organization is even separated I hopefully in the next ten years we see something happening to grow these things together but yeah let's see so now questions I mentioned SDKLTS fully open source, yes no it's fully open source, it's everything there you don't need any of these closed source parts, it's fully open source yeah but in this case no it's as well something that was really like sort of shocking me as well it's really everything there that you need to run on a Homo Occhip and some open source part for the ethics management I just know rumors so I don't want to discuss this in detail but I don't think so that this will happen soon because Cumulus Linux, so the question was is Cumulus Linux embracing some parts of this I think they're in a state where they actually have a good running software so it's hard to change this but because it's managed by existing guys so maybe that is different system but they have investors so it may be hard to change this to a fully open source system so it's just guessing it's really just guessing but I would really like to see that we'll have you put it there and if you have more questions for Tobi maybe catch me afterwards or maybe at the Mike Peace Cafe Yes, I'll be there Thank you so much for hanging out at the SDN room today and if you could talk to me Thank you