 All right everybody So I know it's Sunday it's the end of the conference. We're all tired, but bear with me. They should be a fun talk At least it was for me So we are going to talk about stango with system D And the reason for this title is that in as in every dance such as tango or any other First you have to learn the moves to dance and at first you kind of step on each other's toe You fumble and it's not going to go great Then you learn and then then it's great and you kind of look like this Let's first address the font of this slide. It's coming sense. I know deal with it So first we have to go in up. We have to go back a little bit The project I'm going to talk about starts in 2013 and we chose to use system D back then which was the version 207 So all of the feature that I'm going to talk about are a little bit dated But as I go along with the slides we updated our version of system D. Even if we kept doing the same mistakes at the time system D was not really popular There was still a lot of tall of heat in the community. I'm sure many of you remember that Was not really a really great time And especially you did not speak about system D embedded devices You used busy box or system V but not system D. It was huge. It was a lot of tools. It was Unmaintainable. It was horrible But yet we chose to use it. Was it madness? Maybe at the time today not so much So that's my preface. You can see that's my name complicated my mail much easier to remember I Work at this company, which is called De Vialet. It's a French company. We make this product. It's a hi-fi speaker with a lot of Connected services, Spotify, Alplay, Bluetooth, you name it. So it runs Linux and system D Let's tango First so first date. So as I said, we started with system D 207 So some of the feature that I'm gonna talk about were not implemented or Not as useful as they are today First journal D. So Even back then it's the way to store your log. It's It's the way to keep you all your logs with system D But we did not use it Worse we really went against it For example In order to make it boot after a mount point Riddle and read write was created on our systems We hacked the service and I do not mean we acted using a drop-ins like you should do we hacked it like a patch in the code of system D and As this system is a very basic one it basically slowed our whole boot by order of seconds We did not use the API either. In fact, we didn't even knew there was one We even designed our own logger because with stuff. Why not? See time sink D. I know a great tool we could have used But why use something embedded with your own in its system if you can use something else, right? So we use open NTP D open NTP is a very good piece of software. I'm not saying that it's not good That's not what I'm talking about at all But it has a few drawbacks, especially on an embedded device when you don't have interfaces available for some reasons Open NTP will take a lot of time before trying to sync again And when your user is trying to listen to music series online music services You won't you don't want him to wait for 10 minutes. So open NTP Re-sync again, so we patched open NTP Your product starts you don't have a DNS Again open NTP takes a lot of time before it sinks again Again add a patch fix the problems You don't have we don't have an RTC in our In our products. So we created a small Script shell which basically is a demon that Every five seconds will touch a file in the system in some read-write partition ring a bell Again network D So a network D was introduced in mid-October 2013 at the time. It wasn't really useful for us So we did not use it We rolled our own as I said in the beginning of the talk remember that As time goes by we keep update our system D I don't remember what version we were using at this time But I'm pretty sure network D was more useful than when it was just introduced But still we did not use it We rolled our own again and To try to detect the interfaces at boot you had to use a shell script, which basically was While loop Which was looking for C's class net Insert the name of my interface here for all the interfaces that we used and once all the interfaces were detected by this script shell our network manager could run Because it was not dynamic the interfaces if one interface is was not available It would crash and then you were kind of screwed you basically had to restart the whole process So and yeah, the name of the interfaces were hard-coded in the program and So it was a lot of line of code. I think I looked at it before it was 10k line of codes because well, you have to handle WPS applicants host APD if config root DHCP server clients MDNS whatever so basically That's us trying to use system D We tried but kind of missed the mark so what we did is that we took system D and We really beat him up until we shaped him Into something that worked as we expected Not as you should work with it So it was not a great experience of at first But honestly, it was our fault because we didn't really try to understand how it worked to put this in some context the whole project the whole product and by the product, I mean there's the product the Test software the manufacturing software and everything that goes with a product was all done by a team of five people So we didn't really at the time Also, I did not speak of this, but why have why only one in it when you can have two Two is better than one, right? What do I mean by two in its systems Basically, we had system D run all our services and at the end of the boot it would run another program Demon which from various inputs from users would start stop some services or some target which is exactly what system D do and Because to start services or targets you have to be root but you don't want your demon to be roots because it's a security issue and Then we created some root launchers. What is a root launcher? Basically, it's a small program that only takes a few arguments and only allow you to start specific binary with specific arguments and it's Set with set you ID not so great and Because of all all the things that I talked about our boot was almost sequential So we did not take any advantage of the parallel capabilities of system D There was a lot of contention. So a really slow boots a lot of code from That has to be maintained by us tested debugged and Many many many me warriors It would last all day if I would list them all Here is an idea of the boot chart that we had in our product As you can see there is a few really big red line That's not not really great Then we had some epiphany Yeah Much better the second time around especially when you have time So a journal D Yes, you should use it There's an API for that you can store blob. There is compression. It's easy to add context. You can have some fields It's really easy to insert into your demon. I mean just use it. It's easy. So as I said Binary log format great. It's great to compress. It's easy to send to remote server Nice to use It's also under all the cordon which is kind of nice We don't really use it because we have some very specific needs regarding the the the cordon And so we have a small strip cell, but it's only 20 lines. So that's a bad And Also, you have to learn to use the common line. I Would say that journal CTL common line and mini system D tools To me kind of remind me of git When you first start Really hard to understand once you get the hang of it. It's great. You can do great thing with it Time seeing D You should use it too. It's nice It works along with network D, which is which is really great because when you lose an interface or it's disconnect reconnect You want to lose track of the time It will try to re-sync when the new interface arrives when there is if there's a route to internet or anything Nice Maintains a clock file for you as a fake RTC. Nice. We can get rid of this ugly shell script It works even if you don't have DNS we there's There's this bug with ETC resolve conf that if it's empty and your daemon is starting and and It uses the leap see Basically the leap see will never reopen ETC resolve conf even if it changes So you're kind of stuck That's also one of the reason that we use system D resolve D I'm not talking about this here, but it's also a really great tool even for embedded devices And yeah work with of DNS because if It's hard cut in the source that you can use the Google DNS You can change it at compile time if you want if you don't want to use Google DNS It's also pretty fast because instead of implementing the whole NTP protocol It's only implemented if SNMP protocol SNTP. Sorry protocol Network D. So actually we are using version 234 so we are pretty up-to-date looking forward to use 235 Again, good tool dynamically detect interfaces much better than what we had Using network files or link file you can set up whatever you want on your interfaces and to you the MAC addresses You name it It's really good for simple network like you could have on your laptop or even maybe your server And even not so simple network like you can have on such a product because I like show you We create a speaker, but you can have many speaker in a room. So it's a distributed system that runs So the network configuration can be kind of tricky One of the drawbacks is that there is no Wi-Fi AP support as far as I know So we still had to wrote some code to handle WPS supplicant and host APD so In conclusion, I would say that this time around We have seen some real some real gain from the boot time because finally it was paralyzed We removed most of our crappy code and so now we have maintained and a taste test cut base and Something that is documented that we can contribute to It's much better for everyone and also It's not our job to make an in-system our job is to make software that creates feature for our users So if we can use system D great drop-ins there are One of the features that we really really overlooked Which are really awesome for us because it's it enables us to create our system in In in the manner as of an OEM if you would Because if we were to say license our system, it's easy to do we can just give our our kind of distro To a vendor and it would just have to put some files Into the directory that it wants some drop-ins of its own So to troubleshoot there is a lot of a lot of tool added by system D There is the System D Delta there is the cat which is pretty useful Analyze really useful too and many more Debuss interface also use it. It's good Finally we can have a stateless system instead of having a system which would Evolve over time, which is not really what you want in an embedded system Each time your users boot your product you want it in a pristine state so that you know that it is what you designed Not something that's wrought over time or you don't really know what state it's in Also, I've heard a lot of time that system D is was really huge and so you cannot really use it on embedded devices Maybe it's true if you have a really really tiny storage Requirements, but nowadays it's getting pretty rare if you're using emmc or whatever. They are pretty pretty large Also, you have to remember that even if system system V is maybe small, but there's a lot of dependencies on a lot of tools And you can really really chop system D down with a lot of compiled flags. Just have to look them up To build our distro we are using build routes and it exposes a lot of flags To disable a lot of feature in system D So you can easily get to a system D that I think the whole system D package and so all of the tools are Around five megabytes. So Not so bad Great support for read-only root file system. There's the watchdogs also that are great TMP files instead of doing some overlay hacks that we did What we did in the first version of our system is that we would R-sync files over the root FS once it was created In order to replace some services or some configuration file Not good. Do not do this at home TMP files great. We also are using dynamic users Also a great feature Previously we had many many users that were not really used or they were used only once and then Not dropped because they were in the system With dynamic users you can avoid that A lot of security capabilities that we now use protect system protect home. You can define capabilities for you demon so you can use C groups you can define RT preo and One thing that is really great. You don't even need to modify files that are in slash sleep slash system D You can put them in run so in the end system D really was the right choice for us because it Allowed us to really focus on the features that our user wanted and not something that is not our job We gained a lot in boot time We gained in security because of all the security features that are available in system D and really easy to implement It's easier for us to manage a single firmware for multiple products because I only show you one product, but we have others because you can use some drop-ins or To overwrite things you don't have to do some ugly patches And there is many many many many more features available in system D that Surely we don't know about yet Now we try when we need we need something from system D now We really try to look it up in the manual or in Some forum IRC or anything because usually it already exists and Also, I have kind of an habit to read the news file now There's a lot of gems that are available in there that are not Widely discussed so you maybe won't know them if you don't look So yeah, we use system D now for what it's good at and which is building and managing the system and We do what we are good at which is audio speakers And I don't know why it doesn't work Yeah Thanks Sure have any question Nope Nobody ah here's one Do you have the time and diagram for booting the system D on the second tense? I did not tweak it. I can give you an estimated time first version was around 40 second and Right now we are at 18 Pretty cool, and still we haven't really there is a lot of stuff that we can do to optimize So I'm feeling confident that we could go Way quicker Anything else? Nope, no one really you're sleeping wake up All right, that's all for me. Thank you