 Let me introduce myself. I joined Toradex in 2011 so more than 10 years ago I spearheaded there the embedded Linux adoption. I introduced an upstream-first policy and at times I was top-10 U-boot and kernel-arm-sock contributor and our latest embedded Linux platform. It's an industrial platform called Toryzen and it's fully built based on mainline technology with mainline kernel, mainline U-boot, with Distro-boot, with KMS DRM graphics, with Ednavif or Nouveau and we also have ODIR update with OS3 and we are using Docker respectively Podman who run the application stuff in the containers. What I'm talking about today, I will give a quick introduction to DSA. Then I'll look a little bit at the history, what has been there before, what's happening with DSA. Then we're gonna look a little bit at how a commercial switch even looks like and then of course also how one can run mainline Linux on those. Then another topic I want to cover is Power Over Ethernet. That's basically one reason that you can run, you know, open-source stuff on it, then you have full control over things like Power Over Ethernet, you can control how many watts you want to give on a port and all these kind of stuff. I give an overview of that and I also show how we can run that with OpenWRT, what kind of, you know, platform we have there and at the end I conclude with a live demo. I have a little Netgear switch here where I can run today's Linux next stuff on it. Very good. Let's look at DSA. So this is booted switch architecture. It's basically a protocol for managing switch chips. So if you look at the graphics, I mean a regular PC or also an embedded board usually has like an Ethernet controller, but as soon as you have more than one port, usually there is not separate controllers involved in that sense, but rather some kind of a switch fabric or an Ethernet switch chip and that is basically what DSA helps us to manage. One basically differentiates between data plane and control plane. So the data plane, that's basically where the regular data stuff, you know, flows through your switch. So usually you have like a CPU port that basically allows the control processor or your regular SOC to actually get some of the data to, you know, for certain protocols, you might want to analyze those packets and then depending on the content of the packet, you might want to do certain things with the switch fabric. That's basically the CPU port and that goes to the SOC and that is also referred to as the master interface. And then of course you have also the regular switch ports, those are called the slave interfaces. So in this case here it's an eight port switch and basically from zero to seven, the regular slave ports and the port eight is towards the management SOC. And the whole idea of DSA is basically that the obstruction in Linux really also covers all these slave ports. So you basically have, every port really shows up if you do, you know, IP commands and stuff like that, they all show up there. And then to actually manage the switch, there is also a control plane involved and that's the orange stuff here that, you know, can be regular MDIO stuff or it can be memory managed or memory mapped basically inside an SOC even. So from an obstruction level, these blocks can be all in one chip or it can be a separate SOC, a separate ethernet controller or that can be in one that there are different combinations really possible. And therefore also various of this control path kind of passes SPI also or I2C, all kinds of stuff is possible. And that basically allows the CPU to actually configure the switch so it can, you know, access registers just like it does that just with files, but in this case, you have more registers that also allow you to really configure the switch fabric stuff. And especially in the embedded space, often basically you also have integrated ethernet files or even some kind of abstracted internal such MDIO bus. And an alternative is of course also you can have external files and there or even SFP as SFP plus cages that usually there they actually communicate with those with I2C. And then why is it called distributed? Of course, we can basically continue that concept and it can even be cascaded the whole switch fabric. So there can be multiple switch chips that you chain them. And if you chain them, that's just then the concept of basically you have one CPU port going to the kind of the management processor or the CPU and you then have also special DSA ports that are basically then the ones that are chained. So basically when traffic from switch to here needs to travel to the CPU, it basically needs to go through switch one and switch zero through this special kind of DSA port. Yeah and then basically it's about switching network packets between all those ports and providing kind of you know higher level services like bridging or you know some spanning tree protocols or you can do like Mac filtering, IGMP snooping and of course the whole VLAN tagging and tagging stuff usually in hardware in those switches. And there is also another framework besides DSA which is called switch def and that is basically the framework that allows you to really offload into hardware such functionality. That's what switch def is used for. Okay then let's look a little bit at the history of DSA and I start kind of what was before or what was there before. So in Linux before DSA there was no common framework or subsystem handling switches, network switches and a lot of commercial switches. I mean even like this one that I will use in the demo. While they actually run Linux on them it's totally proprietary driver user space, it is nothing, no common subsystem or anything. Then how about open WRT? I mean most Wi-Fi routers for example have couple of ports usually like up to five ports, it's kind of common. And of course real switches like this one that I use here later that wasn't really something that was supported in open WRT until like very recently. So open WRT basically adopted switch def SW config not to be confused with that later switch def that I talked about before that is basically nowadays in mainline the framework to offload stuff. And the way this SW config stuff works is basically they use a VLAN to differentiate what the traffic, how it's going to be routed not the segregation of the traffic. So basically usually you have a VAM port for your internet connection and that is usually on one VLAN and then you have all the other ports are usually kind of LAN ports where you hook up your computer or stuff like that and they are in another VLAN and that is how that was done. And one disadvantage of that is that to the actual system that the actual ports weren't even abstract that Linux didn't know or open WRT didn't really know that there is how many ports there are. Well it's also not true that of course it knew it through this SW config thing but it's not a standardized solution basically and you can also not use any standardized commands then. If you do IP or route and stuff with those commands don't know nothing about that. So it really required special user space to do that. And there of course was also an attempt made to actually upstream that but in the discussions that happened at the time it kind of was clear that this is not doesn't seem to be the best solution. And actually that discussion later led to the development of the later switch dev which is now in mainline. But of course it was better than nothing for a long time we just carried kind of out of three patches in open WRT for the longest time and only now with the just released the 2102 version at least partially it's now migrated to to using the USA. It actually depends kind of on the target certain target architectures were migrated certain others not yet. Okay but that's basically a work in progress and I expect like you know the next kind of stable open WRT I expect that everything will be migrated over. Okay but then how about DSA itself it basically is not enough something new in that sense it actually started in 2008 and back then I actually that graphics here is from the original kind of commit message from back then it was basically an attempt primary design to support the Marvel you know the kind of the Soho switches the Link Street product line and yeah of course the abstraction wasn't fully there yet at the beginning it was more or less a little bit targeted to that but it then yeah since evolved but at the beginning for the longest time it was rather dormant nothing no development happened and until like mid-end 2014 where there was more push towards you know supporting switches and especially it was about the hardware offloading more people were you know getting interested how can we offload that we don't you know want to use some CPU cycles to to route packets and stuff that need to be a more intelligent way to do that and then basically one of the first drivers outside of the Marvel Soho stuff was then the Procom added support for the Starfighter two switches then there was some infrastructure around it added for example a lot of switches also allow to have a kind of a configuration year prom that that you know at power up it has some kind of a default configuration that it reads from there and the regular kind of year prom subsystem in Linux kernel was basically used and allows now to go through the switch also to read and even write that year prom then another thing in a similar sense is for example temperature sensors a lot of this you know switch fabric stuff of course they get warm as well you want to make sure you know the terminals are right and that was integrated with regular hwmon then other things that were added was energy efficient isn't it support then also like wake on land support that you know you can use any of these ports to also do wake on land functionality then there was better filip integration and of course the also the marvel there were further switches in further families basically supported and then also the device tree binding was basically crafted remember back when basically the arm architecture basically transition because a lot of switches or well a lot of switches meaning the device they use some kind of an arm processor as the control processor or like I mentioned earlier there are even switch socs that basically combine an arm core or multiple arm cores plus the switch fabric in just one chip and then of course also support for doing the bridging really in hardware I mean it doesn't make sense if you have a switch fabric but but every packet need to travel to the CPU to actually do the bridging and then travels out again so that was one of the first thing that was supported kind of to offload the bridging stuff then also more work happened on the whole control plane side of things at the beginning it was limited to mdio that was just what most of the marvel so stuff used but that got refactored to allow also i2c or spi or also memory mapped stuff to be used then procom added the b53 driver qualcomm qca 8k switches support then further marvel families the second generation of the procom starfighter and then also support for the media tech switches that basically in some of the of the quite popular media tech chips that are used in in for example wi-fi routers and then also more acceleration support got added features like port mirroring that is quite common in switches when you want to do some kind of analysis network analysis stuff you want to basically mirror all the traffic on another port support for that or some other tco floating and further family support here then something unique that i'm actually also using on this switch later is is that support for external mdio bus so basically what that is is you have an mdio bus from the soc to the switch but the switch is basically also master again on another mdio bus where you can have further files or switch chips and this support was integrated as well i will show that at the demo as well then further families got supported the further media tech support then microchip as some of you know they were on a shopping spread so they acquired s m sc so also their land 9303 port then microchips or that i think came from microl which they also acquired the ksc kind of families then further qualcom families further procom support i think this one is also one with integrated switch fabric then also with this got acquired basically by micro semi and later by microchip so their vsc 73x or some 5x lines got supported then the real tech kind of switches then broadcom omega that is also an soc with internal switch support further microl ksc stuff and also the max linear stuff that is actually what started as lantic later got acquired by intel and lately then it's now max linear so the vrx 200 quite common in dsl routers actually so that also has a dsa driver nowadays further families then for the marvel one the whole series support got an overhaul that basically means that if you have external files or also cages sfp sfp plus that usually you you know use the service stuff then further support for those ksc ones also including that you can hook them up by isker c not just the mdi then further my vsc family the so called bellix from the ocelot family that's basically uh uh yeah it's ip that microchip has which is then used for example in certain nxp uh chips some of the nxp chips use that ip inside then the hirschman hellcreek support arrow speed chips and last but not least the further ksc the 88xx family so it's yeah pretty lot going on lots of important stuff nowadays i try to summarize here the dsa drivers that are that are available now the r o speed chips pro com b 53 and of course the starfighter the hellcreek this max linear actually take that ip they call it gsw ip then of course the the original marvel stuff that got extended to many more families a medic media tech then microchip like i said they they were on a shopping spree so that there are the kind of the micro ksc types once then the smsc former once the line and the vsc with this one and some of those exist as standalone switches or switch chips as well as there are a kind of soc integrated ones then another contender is the nxp sja 1105 that is kind of very common in the automotive automotive and of course qualcomm qca 8k then the real tech switches which basically got reverse engineered from some leak data sheets how about commercial switches of course there are many a household brand not i'm not going to you know count all that here of course there are managed as well as anime managed switches but for our talk here i mean only to manage stuff is interesting not the others don't really run any uh you know any linux or anything on it then often of course those vendors just kind of copy some reference platform so that also means that often even different brand of switches might more or less have the exact same inside so the same chips the same way hooked up everything that's just because they kind of well just engineers copy stuff if it works so a lot of these reference platforms can be basically recognized again in the actual real commercial switches then like i mentioned before there are dedicated management socs or also with the standalone switch chips as well as fully integrated ones where you have it all in one chip then of course from the storage for the for the program stuff there is nand versus nor usually they have at least like 64 meg of ram so plenty to run linux on it and uh yeah often as it is with this kind of embedded stuff it's a big question is what and how flexible open is the bootloader because if you don't have a kind of a decent bootloader then you really have to start a level earlier and you would have to do a bring up even of some kind of a bootloader like uboot or so and that requires quite more skills and and like jtag access or at least some kind of way to how to write that that nand or nor flash and things like that but to replace bootloader is of course a much more tricky kind of topic okay now we look at some basically some uh mainline available uh switch drivers and what kind of switches actually make use of such platform so i start with the pro com b 53 one there is actually used what is used in the sysco meraki mx 64 or mx 65 i have a picture here and you kind of can see quite nicely how that actually is kind of done you see the the fat chip up there with some memory chips that's basically the soc and then lower you actually see two more chips and those are actually the switch chips so in this case uh this the soc is actually a pro command this strata gx and that one already has a five port switch integrated which uses this b 53 driver and then at least on mx 65 which is the one with i think 10 or 12 ports that that one basically has an additional two qualcom switch chips so this is a really nicely basically a the distributed cascaded kind of way of how to do a switch not and yeah there is a some guy who worked on that there is a merge request pending in open wrt so if you have such a switcher would like to run you know linux on that one you can find it with the merge requests there another platform which actually uses the same soc uh or basically the same switch from b 53 family is the lambo r1 it's kind of a banana pie you know kind of one of these clones and it has the that same switch chips but mdi are connected basically and i have a here actually the device three where you can see so it has two of these qualcom switch chips and they basically are connected through different mdi buses basically so that control plane uh this is goes through different bus basically then another one which i also will talk a little bit more because that is the one that i also use for the demo is is the marvel kind of the original driver basically uh there are kind of there is the solid run clear fog which is basically a nice kind of development platform which is you can buy and that is fully mainline kernel and open wrt supported so that was kind of why i picked that one it's kind of really nice you have like a reference and you can see how they did it and then try to tweak it that it actually runs on your one and then the actual target i picked here is a netgear gs 110 e mx that one actually uses an armada a 381 so this 88 f 68 11 so c and it combines it with an 88 e 6 3 90 x switch so there is also one without the x the x just means that's kind of for the 10 gigabit stuff not so it has really support uh yeah it has there those ports which are then hooked up to two further uh 10 gig files in this case the 88 x 33 10p so basically the two 10 gig ports here are regular rj 45 but they go through a separate buy chip this one basically so it's an eight port gigabit switch so the those eight ports come from the switch chip and on the third s you have two ports 10 multi gig or basically usually for uplink purpose not so you can have your nas or something like that on this one yet and when i got that one of course you kind of take it apart have a look and you see some kind of headers amazingly mostly they usually are even assembled so you have you know some pins you hook some ftdi to it like like i have it here not and then you start to see whether you can actually get some output in this case here it's an ancient downstream marvel you put running on it luckily it's it's not kind of locked down or anything and it even has support for ethernet right in the boot loader which is of course very convenient if you want to tinker with it so you don't have to kind of uh you know use some flash program or something to even get some code on it then the stock switch actually runs linux kernels 3 10 70 so some you know really ancient stuff of course with lots of vulnerabilities and things like that i mean netgear has has been in the media with with you know millions of devices exposed to whatever vulnerabilities basically horrible luckily netgear is kind of a at least from a gpl point of view they're a very compliant company you can just download all the gpl kind of source stuff however like i mentioned initially the whole interesting part about the switching is not really open source they use some proprietary stuff so you don't know what exactly they're doing basically it's not handled in kernel or anything they don't need to give you any driver you don't know here is actually that board and you can also see the header it's kind of on the in the middle on the left side and you can also kind of see how that is done there is in the middle kind of the switch chip and on the left side there is some ram and that so c basically they're armada so c and you also see those kind of separate two ports on the lower right which are the 10 gig ones which basically have a little uh yeah their own magnetics of course and on further up you see the two five 10 gig five chips yeah first when i played with that one i struggled a little bit with the ur pinout i mean i don't really remember why that just didn't want to work at the beginning somehow i was able to stop it from booting but i didn't get any output once i settled that i got the u boot prompt and i was also able to verify that ethernet was actually operational so you know it basically had even tftp commanded stuff working one problem was that with my regular setup it it just always you know returned with this unsupported option error and i googled a little bit and tried a few things with my regular setup but i couldn't get that to work and then i found this kind of python based pt fdp thing which has a special dash dash rfc 1350 switch which seems to make it happy so that made me happy as well so it actually accepted code and like i said because it's very similar to that clear fork thing it just kind of started off with that one you know build a kernel with that and tried to kind of load that in but it didn't really work out because from the addressing it didn't match and then i compared kind of a little bit output from that with the configuration that the clear fork guys use and i found out okay it looks like the load address is kind of different for whatever reason that downstream marvel u boot thing uses some weird other way how they mapped out the memory basically so i modified that and i got a little bit further of course there were also various kernel configuration options that i needed to enable like of course the dsa one and because i used some init rd based stuff i had to enable that kind of stuff and of course i also saw that for the 10 gfi there is a separate driver i enabled that and just the whole squash fs and friends stuff and with that i basically got the first mainline kernel output but it was hanging at the ahci driver but it i mean that port doesn't even use it so it just disabled it then i stripped that device tree down to a bare minimum and with that i was able to get the first prompt and it then is basically just a matter of distilling the proper device tree not and yeah with try and error i figured out the whole mdi configuration you know all the addresses there and which bus is now connected how and like i mentioned i then found out that the those external 10 gfi they they go through the actual switch chip mdi or bus so that the switch chip is the master on those buses again and of course need a little fiddling with that until you have that all in the device tree that is really accepted then another thing that you need to be aware of if you play with with dsa stuff is that of course if you just run like mainline and kind of a very naked user space you also need some level of configuration user space configuration so basically at the beginning none of these ports will actually really do anything so you you actually need to configure that and the most common thing of course is to kind of do a bridge so to basically tell the switch ship well all these ports are just you know in the same network and just you know bridge all the packages between each other not and that is basically the setup i'm showing here luckily in open wrt basically it already knows how to do that of course if you kind of set it to set that up properly but that for example if you just copy the clear folk stuff they already have that in there i just for this one i of course extended it even it's kind of intelligently done they have kind of a loop so you just say okay i have land one two ten and and then it will add all that stuff then to debug whether for example you know if you plug something in whether you get all these notifications things like that i actually use the mii tool and mdi tools i will also quickly show that in the demo then basically reverse engineering the the gpios the soc has some gpios this switch doesn't really have much it it has like a reset button so i just kind of hooked up you know gpio keys to all the the gpios that that the soc and theory has and just kind of press that button and then i saw ah this one is actually that button and then the last thing i'm still kind of tweaking is is the this tank g files they now actually get detected and it looks very good just somehow i haven't seen any traffic really going through and one thing is that those files actually run some kind of a firmware which maybe also even can be updated somehow it's not so fully clear so they they for sure have an e-prom as well on on the board where where they right now they load that thing from there and when i google for that i see that most people have much newer firmware but that switch seems to have a very old firmware and and i'm now a little bit concerned maybe there is something in the driver that kind of works a little different in this older firmware that this one seems to be running so i'm i'm still investigating there okay but let's look a little bit further so on this marvel family there are also further commercial switches there are the syscos the sg 2v 250 line they also use this kind of a marvel pristera switches there are for example the d-link dgs 1210 family actually only the a1 and b1 hardware revisions the later revisions actually run the real tech stuff i also have one of those which runs beautifully i will come to that and i have kind of the bigger brother of this one the netgear ms 5 10 tx tx pp that is also one that i'm playing with but unfortunately that one has a locked down bootloader kind of the same u-boot but but it's kind of locked down so i'm trying to figure out how to kind of unlock that that one will be a true multi gig switch so it has several ports with several different 10 g 5 g 2 and a half g and whatever port then the the microchip vsc 74 xx unfortunately that one doesn't really have mainline support yet so that could be interesting to figure out because the 73 xx or the 75 xx are supported so i'm hoping that maybe that could be tweaked to to also make this family work those are used in quite a few you know popular switches like the maraki ms 220 line like the older hb 1810 planet or this zyxel gs 1910 or xgs 1910 unfortunately by default they run some kind of a red boot and that doesn't have vsnet support so also there it's rather inconvenient to to really play with it let's see then like i said that vsc 75 xx family unfortunately here i really tried hard but so far i could not find out any commercial switch that actually uses that i mean micro semi now microchip they have this development platform which of course costs a fortune and it's actually what bootlin upstream support for that so that would have beautiful driver support but i just haven't found any commercial switch that uses it then the real tech ones like i said lots of older or current model of commercial switches uses that there is kind of a lower end 500 megahertz mips and the higher end 700 megahertz and their code named maple or cypress and those are actually the first switches that are officially supported in open wrt then there is also a later line the real tech rtl 93 xx that's basically real for for also for 10 10 cheese switches and there is work going on there to also support that there is the zyxel xgs 1210 a 12 port multi gigs which kind of the similar thing like like this one here that that is basically also supported but also there they they struggle with the the 10 gsv plus support it's not fully clear how exactly that that all works i think it also needs some kind of a firmware and and it's all not so fully clear but basic support is also experimental support in open wrt is there now i quickly want to talk about power or reason net basically it allows passing electric power along with the data there are various standards i'm not going to go in in much detail here and one talks basically about either power sourcing equipment or power device so those are the two sides not and usually of course the switch actually could be both so it would allow to have a smaller switch that could be powered by a bigger switch or of course the switch can power than other devices like cameras or wi-fi access points and stuff like that then yeah that that's basically that so power over is net switches that is kind of an interesting topic there are various of the switch controllers that that are used to to actually provide that kind of functionality it's it's basically about you know switching some fate which will put the power on the on those lines and then usually they also have some kind of chance which allow to actually measure how much power now the device really uses oftentimes they have some kind of mc use that runs on proprietary power over isn't it firmware and usually those connect and via just uart and it's kind of a fairly easy protocol then in open wrt for the real tech switches somebody actually worked on that and there is a kind of a experimental package available that that would allow you to do that i also list here you can then you boost call call basically poi info and it will show you you know port so-and-so how much consumption and things like that that's fairly nice that's basically in see written a little demon right now and there is also a proof of concept of of a tiny demon just a draft for now so you're welcome to join the discussion you know how we could evolve this and maybe of course one of the question is user space versus kernel space does it even make sense to integrate something like that with a kernel driver or or is that the right way to go just to have some kind of a user space thing for it there are also other such controllers that talk i square c rather than uart things like that so maybe we could do it a little bit modular that we could switch the communication thing something like that okay then i get to the demo let's have a look basically that's the picture here a little bit that you see this is just an f d d i thing by usb connects to my pc here of course i have an isn't a cable and and i have a power cable and this kind of diagram here shows kind of how that switches yeah what it contains it has on the left it starts with the so c or with the ram and flash and then the so c then there is the switch chip which has these eight ports and the two lowest one are basically serdes which have another five and then the other rj45 ports let me switch over here i have kind of just you know terminal to the turn that off i hope you can read it so that's basically the bootloader i can even start over again so you can see the full output that's basically what you get when you power it on and i need to have my cheat sheet so we can actually yeah in in one terminal i'm running this the tfdp server thing and i can now for example you know set an ip address set the server ip address and i can then kind of load some code here first i load the the full kind of init rd open wrd thing uh that is actually kind of master from i don't know 10 days or two weeks ago master open wrd right now is 510 kernel based and that is basically running the the full thing and and there so this like i said it's 510 61 based that's a little bit old to september 6 and you see now it actually did this user space configuration i talked about of course it takes a couple seconds until all the demon everything is kind of running and one can see all the ports here that's basically how they're called uh the lan x so 1 to 10 and at eth0 that's just the convention that is used because that is basically the actual real easernet control behind it and one interesting thing that i can show you here is i can run the mdi oposs command and that one now lists really all the mdi oposses that are there and you see here that i have basically well we can even start there is a fixed one as well so that is basically how the the sock one is just basically hard connected to the switch that is this one and then i basically also i have this uh that is basically the mdi o one that goes to the switch which then has these eight ports and as you can see one of the ports is up the first one and the others are down and then like i said the other the 10 gig ports are on another mdi oposs that is basically this mdi o external so that basically routes through the switch chips goes out again and then do the vise out there that is basically this then i can reboot it and i can also show you how we can even run basically the same user space but i can really run cutting edge uh mainline on it let me set the ip address stuff again and then i have some more i can then load the z image basically separately of course i also sometimes it locks up i don't know that's this old u boot has some kind of a flow control problem i assume not uncommon but not much well maybe one time i try mainline u boot as well but for now i'm happy that it kind of works so i separately load also the squash fs stuff and i also loaded the uis tree up there as you can see and then i can give it some boot arguments and then we can go ahead and start the whole thing up actually there is some kind of a crash as well i haven't even looked but that is with you know live demo cutting edge two days next well i mean what can happen mainline it should just work not so let's see how that works actually it's still kind of starting up but whatever so we can see this is from today who now 2021 oh 9 29 yeah dirty because i had to kind of hack something well and here we can see also it also this one talks eth 0 that one has the neck out the neck disabled not that's the one that is fixed then i can also do that with with some port here not and we can see so the first line one is connected gigabit flow control link okay if i for example take the second one it will say okay no link makes sense and i can also take like nine or ten that goes towards actually there it seems strange it also seems to have some uh i yet have to investigate what what exactly goes wrong it seems that it's not fully happy with it but yeah no link right now okay that's about it what i wanted to show here let me go back to the slide i don't know whether anybody has uh i have some references here uh questions answers anybody has a question yeah go ahead sure yeah yeah i think it uses the regular exactly yeah sure yeah yeah i haven't really played much with it but if you're interested in the best is in the open wrt that those real tech guys like i said that that those are really supported already in in regular open wrt and there is a lot of discussion about this kind of stuff going on so i would suggest you can chime in there i can i'm also usually watching it a little bit but i don't know the full detail to be honest okay any other questions otherwise thank you very much having you here today thank you guys