 Test all right. I think we'll get started here. Welcome everyone to the Sunday morning developer track I appreciate your enthusiasm at the end of scale We're starting off with a Modernized millenics desktop with Christian here. I won't waste any time. I'll let him get right to it Go ahead and start the show Bring the mic a little closer My primary focus is working upstream on GNOME and trying to make it easier for new people to contribute as well as our existing members To have the tooling they need to be able to get more work done a little bit faster, so So if you kind of like go back to desktop Linux development in the late 90s early 2000 pretty much everything was still in C C plus plus there was a couple reasons for that one was the C plus plus compiler was not very good back then and You have lots of crashes and just like writing code in C plus plus on desktop Linux like it was frustrating As for C the compiler had pretty much advanced at that point because Linux kernel had been running for so long And the new project had been going for so long that things worked pretty well But we're still using the build systems that were created from like early 1990s for most people that have built any software on Linux are familiar with auto tools and we just kind of stuck with that for a long time and none of us were really happy with it But it worked and it compiled on all these different unixes like AI x and hbu x and all the different systems that Commercial companies needed to support, but the rest of us really don't care about And It's been like this basically from the beginning that every desktop that we've ever put together our personal laptops or desktops They're all really bespoke like we Install from packages and the packages that I have installed on my system are completely different from the packages you have on your system And when I run into a problem from some incompatibility on your system There's a point where I'm just like, you know, I can't replicate your problem So I really can't do anything but close this bug. I you know, there's nothing more I can do to help you other than you helping yourself and trying to figure it out So that's one of the things that we've had in our system and in some ways that's made users more Educated and in other ways, it's made them incredibly more frustrated than just left Today for the most part applications are still shipped on the operating systems release cycle So if you go to Mac or Windows, this is for Android or iOS is a really unfamiliar concept the idea that Let's say Debbie and stable has a lot of users and the applications that are available on Debbie and stable We're probably released two years good two years ago because they go through such a long stabilization process now imagine that you're an upstream application developer and you're writing a feature and You know that your users are going to start using this feature in about two years After you know of which that time you have no idea how many bugs you've created or fixed or whether or not they're gonna have a problem with it and There's just a really really long cycle. That's just kind of something we dealt with and just pretended that People want newer software. They'll figure out a little compiler from source or whatever But when your users do run into problems, you know, we need to go and debug it But we don't really have a way to replicate the system because everybody's system is so different And I think the key insight for me here is that we really do value our independence a lot of us Are like we're just in that mindset We care about being able to have my system the way I want it But the other side of that is the strong amount of fragmentation That's something that we have to deal with and try to find that balance between that individuality versus the Fragmentation of the ecosystem and making it easier for people to write software so this talk is about modernizing and In the process of modernizing it always helps to say like why do we want to modernize because that will keep us Honest in the trajectory that we do and changing the platform and One of the things I've noticed while trying to teach people our platform and teaching the program and teaching the program specifically targeting desktop Linux is It's really just too difficult to get started. It is a huge amount of work I'm not saying that you don't learn things from that effort. They're definitely valuable skill sets you gain in that but We're losing so many people Because they start getting their workstation set up and they're like, this is too hard. I can't deal with this. I'm gonna Go to a movie or something. I'm gonna do anything else with this because it's really painful And that's a problem for us because we're a very small group of people like Desktop Linux is largely made by a couple thousand people that's trying to distribute around the world So if we don't make those people as efficient with their time as we can there's just no way we can keep this project going indefinitely As I mentioned earlier the fragmentation Is somewhat an issue and the reason it's an issue It's because you only have so much time to work on your project And you either spend it polishing the features you have or trying to deal with incompatibilities between all these different operating systems out there all the different distributions of Linux and if you have the time to polish it you're going to have fewer bugs, I mean that's part of the process but You're gonna make it more polished in the process things are gonna work smoother. They'll look nicer Part of one of the experiences I had working in the GNOME project is I try to teach a bunch of people to program One of it was about four or five years ago I tried to start this project called the GNOME University And I was gonna just teach a few people how to program because that's what I had free time for and I was really overwhelmed In about a hundred and fifty people that asked me to teach them how to program C. This is a huge effort And so I started writing training materials and whatnot how to like write modern desktop software in a fairly archaic language And you know it showed me like how much of the tools and each of these steps that we came across that people had problems with And so I've kind of taken that into account While working on a new project of you know, this is how many people I lost at this stage This is how many people I lost at this stage and like how do we? Increase the success rate each of those steps to ultimately get the most people coming in and joining us in the community And then also for people like me if we have better tools, I can do more in less time. That's always a great thing So I have a series of questions here Can we maintain our independence that we all value so much? while reducing fragmentation and Can we separate the applications from the operating system without losing that long-term stability that we value? from you know Projects like devian stable or red hat enterprise Linux who say enterprise etc. By separating them, can we still have that stability without the ancient versions of software and then can we Allow independent vendors people that are just hobbyists or you know actually trying to create a career writing software Can we help them ship more software? higher quality in less time and I think that kind of goes in with that can we reduce fragmentation and Then something that's important for me is I like trying these really ambitious ideas with desktop Linux Can we allow for these ambitious ideas? Without sacrificing the platform, so we're gonna take a step back here thinking about like why is it so hard to get started? Obviously we all know it's lacking documentation not many people like writing documentation We like to bring people in and say oh you're gonna join the community. Why don't you document this pain points you have? In theory, it's great in reality almost nobody does it We have this dependency and compatibility between the applications and our platform itself. So if we have Let's say I'm using a library to Communicate to a web server to download something to write something off the internet, right? The version that I may need in the application that has a feature that I want specifically May not be available on an operating system that I want to target say Debian has too old of a version The version I need is in the distributions have been released in the last six months Do I? Reimplement that myself and try to ship it everywhere which probably has broken code and won't get security updates at the same rate or You know do I just not use that feature and try to come up with some problem Or some solution to that problem Another interesting thing here is this wide-scale app distribution the just the distributions Have the ability to ship a lot of code to a lot of people at a very low cost because they have these public mirrors That will give access to everyone and it's no cost to use a developer But if you want to start distributing your application You need to go through a huge amount of bandwidth cost potentially you have a popular application That's a that's a good amount of money a good idea and a good example is the Linux vendor firmware service a colleague of mine at Red Hat started this project and it ships firmwares for people to update their laptops automatically or Mines and keyboards and I think the bill recently was a couple hundred dollars a month And that's just a hobby thing on his side You know do we want our individuals that are hosting services for millions of users paying money every month just to do that Can we come up with these ways for us to share that burden and like I said workstation set up is a huge bar And as an upstream app developer, I have to think about does this fragmentation Make me have to be really restrictive in what I'm willing to support say I'm going to only support the last distributions made in the last year or only you know the major Debbie and open to say Fedora Right, and if you're not on those you are best basically out of luck and you have to do everything on your own shifting priorities between projects is also a huge issue because we all because of that independence we all crave and we all value We're all moving in different directions and those different directions that we're all moving in just makes it really difficult for someone coming in To say like oh, what are we gonna do like what what can I work on that really helps and that's always well You know it would be really nice if we could do this and then you know They go off and try to do that and then some maintain every other projects like no, I don't want that I'd rather it do this other thing So now we have these new contributors that are really excited They're just ping-ponging between people and and not really able to contribute and it's of no fault of their own Because there's nobody's you know moving in the same direction And I think that kind of fits into the overlapping concerns here. It's just You know without that Charismatic leader we're not really So the most common question I get on IRC when people are coming up and asking how to get you know, whatever project started is What distribution do you use? Because they tried they tried to build it and they failed at building it They got into some dependency issue. They don't know where to find it and you know So what distribution are you using I'm just gonna do what you do and I'll start from there. I Don't think that's okay You know, I like my distribution, but I'm not zealotry about it I don't need you to use the distribution that I use I the software doesn't require that you use the distribution I use and The fact that that's that first question. I think it's problematic for the health of the ecosystem also one one piece of I Guess it's just a truth It's inevitability of development is that you're always using these like newer just newer dependencies Then the system has software continually evolves, which means we're always going to have something that's newer than what we had So if we have a system that the dependencies that our application is built upon are moving quicker Then the OS inevitably we need to deal with that drift We need a way to install something that overrides what the system has it needs to be able to be Updated on a higher frequency and we need to be able to kind of like separate that from the operating system So we just kind of take a step back here and think about like how is this ecosystem changed in the last? I don't know 18 19 years really since the the turn of the millennium Continue integration continuous integration is pretty ubiquitous now people are using it all over the place We're doing a lot better at finding systems bugs before they get out to users You know get one that's been significantly helpful Containers are everywhere now whether or not we like them some people do some people don't But the truth is they're everywhere We're much more diverse on language choice People are writing desktop applications in Python and JavaScript rust etc. C sharp We have more diversity in the interfaces we're using to laptops generally work really well now touch screens work mobile works We're seeing Internet of Things devices come around Our cross-tool chains have been fixed to make those Internet of Things devices work And a huge one is that we can rely on the graphics drivers If you told me in 2002 that you know, we're gonna eventually have drivers that just pretty much work everywhere GL works Vulcan works and the number of bugs on them are pretty low I would have been surprised and with that in mind like where exactly is our platform going? It seems very likely at least to me that we're gonna be ending up on an immutable base OS you see Things like Chrome OS and silver blue have kind of like pioneered this this idea that if we want true device security and true privacy We need a platform that can give that to us that means secure boot With kernels signed by people we trust it means a user space that we can trust that when it boots It's running the code we expect that it's verifying we can verify that that code is true and we it can verify that we're authenticated and It's safe from attack at runtime If we do that all of a sudden you can't really install new dependencies on your platform, so that makes containers almost necessary for us to continue doing development going forward and Part of this you know while we're doing containers that also kind of implies sandboxing You don't have to do containers with sandboxing, but it goes so well together that you just it comes with sandboxing generally We're gonna see even more types of computing devices You know one thing we might lament about that is the more computing devices We see the less general purpose computers we see and that may not be a great thing for how we think about stuff But it's an inevitability, so we need to take that into account in our platform if we're gonna continue to stay relevant and Then we have new models of privileged access so instead of saying Like using sudo to run some command essentially with root privileges We are we're gonna start seeing more things like portals which are provided by a flat pack and other Distribution integrations what we'll say okay. I want to open this file in this application But I don't actually want to give acts the application access to the real file because that could it could do some nefarious things So or I don't want it to see the file system So instead of giving the application access to the file system it may go through this intermediary This says hey, I need to show a file browser And allow the user select a file and they select the file that file browser Implementation provided by the operating system will open the file and give a virtual file back to the application The application never saw the real file system. It doesn't know the real file name on disk necessarily But it has the ability to read and write from that disk without Exposing that into the sandbox so those are going to take a lot of new design by all of us as the community to build these safe privilege escalations and This more language diversity is OS vendors are less involved that what I mean by that is Usually enabling a language on an operating system is a lot of work So you want to like enable Java on the operating system. There's a lot of integration points that happen there but with Containers coming in that matters a whole lot less because it doesn't interact with the host system so much you can Pretty much do whatever you want and as long as you're in that container. You're pretty well good to go That means I can run some bleeding and bleeding edge Java or Mono or Python inside the container even though the system may be I don't know five six seven years old Now the thing that I'm really here to talk about the part that I spend most of my day working on is an IDE called builder. I Think it's probably the first container native IDE and by that. I mean all of the abstractions for how you run processes how you discover what's supported on the system and How do you execute your application and profile it and debug it they all happen in containers and so you don't need to Find these really weird setups with the right environment variables to put into say like visual studio code or mono develop or any of These like Vim etc. You just it knows about the containers from the lowest APIs inside of builder It's very memory conscientious I'm a Long-time C programmer and I've worked on databases run times hypervisors file systems all of these things. I care about My software not taking over my system I know a lot of people that use atom and visual studio code complain about running out of memory or Their system got slow because it's taking two gigabytes. We're more in the like 100 megabyte realm while providing an entire IDE experience All of our API's are asynchronous. You don't have that locking up of the IDE as you use it the editor is Based on a 1992 design that still today is very fast. It's a B tree with a rope-based editor and It has super smooth scrolling even with high DPI. We can demo all this later, but it's a very slick editor We have integrated UI designers You can extend builder in a lot of different ways Everything's kind of implemented with plugins and you can do that and see C++ Python Valor and probably the next couple months rust Unit test integration debuggers profilers completion terminals Supports for lots support for a lot of build systems right now. We have about 10. I expect a couple more Hopefully from people here that have interesting build systems You can install it in a couple clicks using flat pack Flat pack is the main container system we support, but we support a couple others now as well and What I think is really great about it is getting those newcomers those people that you want to get into your project It really is only a couple minutes set up They need to with builder installed if they open up your project and it has a manifest describing how to build your project It will automatically download all the SDKs. It will set up the containers for you And basically all you need to do is click run and it'll build everything and set up debuggers and all of that We have code index code indexers and really fast fuzzy search based on my work on full-text search engines previously and We're very non-opinionated on developer language choice I know that you know a lot of people are like, oh, I use this idea because I use Python or I use this other IDE because these JavaScript and it's really good at it We've taken the Stance that everybody's language choice is their own personal choice that we're not going to enforce that on you We support lots of languages. I'm also the maintainer of gtk source view. We have support for 50 or 60 different language Specifications so you get highlighting in all sorts of languages We have multi-monitor support now Semantic indentation code formatting highlighting all those important IDE features I guess just recently while I was working on support for Purism's Libram 5 phone We added support to be able to do cross architecture execution. So if you use this really neat project from QEMU QEMU user static is compiled in a very interesting way in that There is no dynamic links in the QEMU's virtual machine emulator And what that allows us to do is to open up in your container QEMU With access to your all the files in your container which are possibly compiled in a completely different architecture like an arm device and Then execute it on your x86 developer workstation What that means is I can take an application I can pile it for the Libram phone as an art as a ARH 64 and just run it and it just runs and works as if I'm doing it and any other because we have the container system set up plus this Virtual machine emulator, which is really just doing what's called binary translation but it is Pretty fantastic way to get supported With basically no works if you're doing stuff on phones or whatever we can Help you get started with that, but we do have support for that built-in We have device abstraction So if you're working on bringing up a new device if you want help bringing up a new IoT device that your company is working on or whatever we can help you with that as well make it really seamless and We also have deployment APIs as part of the builder plug-in architecture So that might be if you have an external device that is over USB You might need to be able to send an operating system image that your build system produces and send it to the USB device and then reboot it we can provide you the APIs you need to wire all that up to This is something that's important to me because I've been involved for GNOME for so long But when I improve builder it improves the platform. So a lot of times I'm going in writing features that go into either GTK or G-Lib libraries or you know the the various ecosystem So this project is important to me because it it makes everything better for all of us Now to keep myself honest like am I Making contributing easier with with builder or am I just like scratching my own niches? What I think is really interesting about this is we've provided a way to preserve that independence you can bring any distribution you use and Thanks to flat pack. We have all your SDKs and containers. It doesn't pollute your host system Your host system can be a read-only file system for example But we allow you to reduce that fragmentation because flat pack uses this mount technology Mount namespaces that allow us to make the container look the same on everybody's OS that's running your application. So they're all going to see this very consistent file system layout and In doing so we've really reduced that fragmentation of like where do I find files? What libraries are going to be available? That that fragmentation goes away, but we've preserved that independence You can keep your SDKs updated easily from GNOME software and This idea of a shared tool chain between your Your project is really useful because if someone says on GCC 7.9 And someone else is on GCC 8 and you get this like weird different warnings and you have you know people curious about what was going on It's much nicer when just everybody has the same consistent tool chain I don't know if you if you worked in large engineering shops one of the first things you do when you join an engineering team is like, okay, you got to get your tool chain set up and That's no longer an issue. It's just it's just handled for you by the manifest of your project We've made it easier to contribute because it's just one click to clone a GNOME app and get started and just another click to run or debug it and For us that reduces the time we get someone providing us their initial patch and I think that is the most important thing in terms of Getting someone to stick around the sooner they get their first patch in the sooner they see their own change in front of them The more likely they are to stick around so for me this was a really fun process to create but there were a lot of challenges because the container ecosystem is so Relatively fresh if you've ever used a terminal the concept of like that the interaction model they're they're called pseudo terminals and Pty's for short and the container systems have this concept of a pid name space and What that is is when you you know P yet run PS you see the list of processes in their IDs That may be different than what it is inside a container and then furthermore the access to the the device that is your your terminal device May look different between those they may not even be available in the same space So that causes a problem for some of the shells like ZSH Which are trying to coordinate between your terminals and merge things like history because they may try to find it and find each other and They just don't find each other Or they think they found each other and it wasn't so that one is fairly problematic and I think we're still waiting on some work upstream by the ZSH folks to make it work a little bit better, but That one is definitely problematic Being able to pass file descriptors just to a terminal process that you want to run in another container Has proved difficult and the reason why is you need to set up What's called like the controlling Pty for these sub processes and that requires code to run So you need to cross this container boundary and you need to set up file descriptors in such a way on the other side of it And in some ways that requires coordinating code We found some ways around that with flat pack, but whenever we integrate a new container system That's one of the first things we have to look at is how do we get? You know the I octals to run on the other side of the container in such a way that we can actually use it If you try to start a shell for the user and you just run bash That's gonna anger some users that don't have bash so you need to discover what their preferred shell is But you're running in a container yourself So how do you guess what that is the container may have? username spaces translating The user ID mappings, you don't even know what their user ID is You don't have access to the Etsy past WD filed it is to sniff what the shell is So you have to come up with a way to like execute a little bit of code on the host To really discover that shell again. We had to put code in flat pack to be able to do this And other container technologies would need to do the same and then furthermore with the terminals users Expect to be able to have a terminal in their actual host system They expect to have one in their build system in case something goes wrong with their build system They expect to have something in their runtime system when they're actually executing the applications and these all may be different I guess I covered FD passing One interesting thing we use the the FD passing for is when you're working across containers They may not have access to the same shared memory segments and shared memories usually how we'll do high performance message passing between coordinating processes But since you don't have that you need to have like a way to Set up this high bandwidth communication so you can use a file descriptor you can pass it across the container and then you can What's called mMap memory map these file descriptors into a range of memory and then you can use them like shared memory They may even be shared memory, but They wouldn't be available to you otherwise When you're in different containers, you have like I mentioned earlier you have these different mountain namespaces What the world looks like to you in that process space is completely different? so that means something in Slash home slash, you know user ID whatever may not look like that inside the container and maybe in like slash var or wherever and You can't just Try to open that file. Let's say you're clicking on a Diagnostic or something you can't just open up that file at the path in which you thought it was inside the container So you have to translate these paths to and from containers and that requires a little bit of knowledge of where things are going to be on both sides And it's I guess it's not totally It's not totally obvious what it's going to be for example in flat pack It has this user runtime user app split where the the runtime the container that the application uses Is in user and then everything else is in app or all your dependencies are so you might need to like sniff a little bit of The path there So we do have debugging support even in the containers debugging We just use gdb at the moment But we could implement per language if someone wants to do pdb for Python or the Java debugger, etc We can definitely help with that We have to use pty's to control gdb And the reason for this is we don't know where gdb is necessarily going to come from your container May have a completely different architecture So we may not just be able to run the host gdb inside that we can't just copy the binary into your container and try to run There and we may not be able to access it from outside the container because it may be in a completely different Pid name space so we don't know what the process ID is So the way we do things today is we expect the build runtime to provide us a debugger that we can execute And then we use all the fd passing and like gdb remote controlling on top of that But again when we break and we get a break point it tells us the file you're on that may be Completely different than what we see in the host ID. So we have to again go through that path translation for those Furthermore if we're going to decode what the symbol is let's say all we have is an instruction pointer from the back trace Normally what you would do is you look up the instruction pointer and you see what file what library was mapped in that address range That overlaps that instruction pointer and then you say okay the offset is 5,000 bytes off the beginning of the file that's mapped here So then you go find that file that dot so the executable and you look in the header of that and say okay Where's your debugging symbols for this file? Then you have to jump to that file and then you have to say okay What symbol was at the five thousandth byte offset of this file? And there's some facilities to do that in the dwarf in elf formats But again each of those have to do path translations all the way there So it gets a little bit tricky to continually Unwrap that enigma Profiling is similar to debugging because you have to do that same sort of symbol expansion But on top of it oftentimes in containers Two important features are disabled and their security features You have p trace which pretty much all the debuggers are built upon and what p trace lets you do is say Go execute this next piece of code and then stop and then go execute this next piece of code and stop So some profilers are implemented using that and then other profilers are implemented using sampling using the Linux kernel feature called of Perf It's a sample-based profiler that will say what's my system doing every you know hundred nanoseconds or something and because those are You can use those to control the system. You might be able to use them to get access to Like if you get access to the perf data stream, you might be able to see what code is executing as the root user And that's not really ideal So the container systems often disable that so we need a way to get access to that so we need a way to Basically elevate what we can do either from the containers or outside the containers what we do and build are currently is we have a a Suid helper on the host that will let us get access to that even though our container is still fairly locked down If we're going to execute code though, we need to be able to execute it inside that container Really convenient way to do that is to use username spaces to set up this new container We're gonna run in and then execute some code However, if our application itself is running in a container We can't do that because again There's a lot of CVE's that have been associated with username spaces and containers and so a lot of distributions disable it by default Also a lot of the container systems will set up their username spaces and then Remove that capability from themselves and then continue executing so you can never regain that capability as part of the security model In doing so that prevents us from being able to do what we want so again We need to help her outside the system that can help us Set up these containers and and whatnot in a fashion That will work for all of the container systems Currently we just use we have some flat pack helpers They'll let us do that because we ship builders a flat pack and that gets us most the way there in The last couple years a lot of editors and IDEs have moved to this out-of-process coordination helper these things called language servers so instead of making an IDE that has Really rich Python support in the IDE and they move all that Python support into what's called a language server It's this external process that coordinates with the IDE and that allows for lots of good code reuse because you can coordinate that That Python specific feature set between editors, so if you use sublime or VS code pie charm You know builder you can We don't all have to re-implement that same bit of code What's also nice about it is we can run it inside your container potentially and that means it's in your build container It sees all of the things your project sees you don't have symbols coming from some other project That's installed on your host system. Whereas The symbols that you have in your project specifically are newer these newer dependencies and completely different APIs But we have this really unique feature by using flat pack to ship builder and for writing applications using flat pack is that we can use SDK extensions to provide language servers into your Build container without polluting your container of your application So they're just kind of there and then builder knows where to find them and say you want the rust language server It's like, okay, that's in this place. We'll just spawn it for you inside your container And when you ship your application when you bundle it it all disappears But we have some problems with the language servers. They're all using JSON they're passing megabytes of text back and forth on a very high rate and That all has to be parsed so You get into the situation where you're spending a lot of your heap memory Just parsing these little JSON strings or very large base 64 encoded file content in JSON strings and we Like I said earlier, we're very memory conscientious and builder. We care about being an efficient editor So for the language servers that we ship as part of builder We use this other format called G variant. Some people will know it more as the D bus encoding format it's very similar to that and It allows us to pass data between processes in a shared memory space without any parsing the format itself already includes The fallback mechanisms to avoid having to do like a beginning to end parts of the format additionally includes some really nice features like trailing null bytes and strings so we can reference the strings directly in the memory zone rather than having to parse the data and copy them onto the heap etc and kind of It's not really People don't really talk about it that much but the language servers that are out there also really terrible They break in all sorts of ways and they're all on different versions of the specification So it's kind of a nightmare. It's getting better, but I don't I'm not really comfortable saying the language servers today are the best solution But it is what we have so we try to support it if you're gonna be building applications. You need an SDK So for us the main SDK management is flat pack an OS tree Flat pack is built on OS tree, but I also see in our future using OS tree to do operating system development We will be able to synchronize operating system images and build simulators and all sorts of stuff with that I mentioned SDK extensions a moment ago. They let you take an SDK and add new bits to it So what makes OS tree really interesting for those of you that aren't familiar with it it's basically get for binaries for operating systems and It lets you do incremental diffs between two versions in time So if you think of like every change to your operating system being a commit like a get repository You may want to say what was different between last week and today and then give me those changes only That is essentially how OS tree doesn't update But you can also do this thing called a static delta where you can say find me the version The difference is between these two versions and can pre-compile that into a single file that can be downloaded very quickly Pre-compress it look for any extra changes. I think it also uses BS diff for looking for Binary changes that can be shifted rather than saying there's lots of changes It'll say oh this this section of the elf was actually over here and now it's over here. Just change it And then so you may use that to say what your previous release was from your current release And your two releases ago from the current release so users get really small downloads We care about user file system usage So it's all content address just like get for automatic de-deplication and it's built on a hardlink farm Which basically just means we have this checkout data in this other area where we're going to deploy the application I just use the hard links on the file system to the checkout and that's just a really efficient way of Saying these two things reference the same data And it works basically on any POSIX compliant file system It's what powers Fedora silver blue and fedora or what was project atomic. It's now Fedora core OS I believe and then again flatback it builds on top of that and includes a sandbox It's built on top of OS tree. It uses all the different new container technologies It's a little bit stricter in which ones we'll use based on like how often we see CVEs associated with it So the way it uses username spaces is fairly conservative But one of the neat things we have is this runtime versus an application split So the runtime may be shared like we say we have like maybe 30 or 40 GNOME applications And they may all share a runtime Because they all have like the same libraries in need it has like GTK and D boss, etc But the dependencies the applications use where they differ We don't want to bundle those all in the runtimes for in case you install an application that doesn't need that you end up Downing more data and then you get angry because you have two gigs of crap on your file system So you can have some of the data bundled in your app But share a great deal of data between applications share the CVE overhead and all of that work that takes to maintain essentially a mini operating system and then we use portals for the safe privilege escalation and You can use it on top of any read-only base OS live CD, etc As I mentioned earlier, there's some SDK extensions I think currently we have ones for like GCC rust Java Mono, etc It makes it very easy like most applications aren't written in Java for the desktop But we do want the people that are writing applications in Java to share that toolchain because it's a lot of work so the SDK extension allows us to Bring that toolchain into the SDK but not require everybody to download it only the people that are developing applications for it And as I said language servers are a really good thing to put in those From the builder side will automatically discover what SDK extensions you need based on your app manifest So if you say you need the rust extension because you're writing an application to rust We'll just automatically download that toolchain for you set it up and you're good to go And as soon as it's installed you can build run debug, etc The interesting side effect that happened that was not on purpose is that by getting more people sharing these Extensions sharing these tool chains. We find they're actually getting closer to reproducible builds Then we anticipated at the beginning Because so much of it is not just the way your system was set up in the way the operating system looked to the compiler But also the version of the compilers that you're using so when you kind of put all of that together All of a sudden you get this massive jump forward and reproducible builds Which means I can ensure that what you distributed to me was what I expected and it helps me audit, you know the Distribution cycle for people that don't necessarily trust the security models of where they're consuming their software Builder like all the other IDEs has plugins tons of extension points I'm not going to go through them all because it would take too long But basically every part of builder is implemented with plugins and you can extend it And you can write those plugins in C C plus plus valid Python rust Only a couple files are necessary. It's really easy to Start adding them. You just subclass one of the plugin interfaces And this is what a simple plugin that extends the the buffer editor buffer looks like and you Overriding the function starts with do in this case. We have a file loaded and a safe file. You can do something when that occurs and In this case you put that with a plugin metadata file and you're good to go And I guess I don't really have too much time to do much demos, but let's see if we can So this entire time I've been running events, which is the PDF viewer for GNOME. I've been running it from inside a builder Being debugged and all I did is I'll show you here when you start builder you get this Presentation view of like the recent projects and then some suggested GNOME projects are easy to contribute to and If you click on one of them, it will try It's one that won't download too much It'll take you to the URL to clone the project and if we have enough Wi-Fi here Clone the project and you'll have to bear with me that we're in a thousand twenty four by seven sixty eight screen Which hopefully none of you are subjected to down here on the bottom. It's automatically downloading all of the dependencies that are Not bundled as part of that runtime So you can see right here it's downloading evolution data server Clones that repository starts building that and I don't want to wait too long for it Go, so what is come over To here, so here's where I have events running and after that other one would have completed You can either just click run or you can run with the profiler debugger, etc And here we go So I'm basically on a read-only OS here, but with You know all of this stuff included not to expose my personal files too much there You know here it is running events from bleeding edge get code from this morning and We have our PDF viewer, so if you're interested in contributing canome stuff It doesn't matter what distribution running just install builder clone the project Tester change submit a merge request, and you're pretty much good to go So to give you an idea of where I'm going I want to see more language servers provided via flat pack because it makes it really convenient for us to support the languages That you want so we can continue to be non-opinionated about language choice I want to see simulators coming in especially as the Libram 5 phone starts to ship And as well for GNOME developer builds because the more that we're using it upstream the better it gets I want to see more container support. I'm gonna land land podman support this week probably Podman is a for those not familiar. It's a re-implementation of dockers command line tools without a daemon, so it's very convenient for what we need and In compatible with docker, of course I want to see maintainer tools release management more version control database integration, etc remote server support the device emulation is what I Was mentioning earlier. It's mostly there deployment documentation, etc So I guess at that point if you have any questions for me. I'm happy to answer them and if not, hopefully that means I Already told you. Thank you. Sure. I Can repeat it So the question is why not create a translation layer that that deals with moving the buffers between them is in my yeah, it's the problem is that The way in which you need to do these path translations seems to be different on everything you need to on like on each of these Use cases so for the when you're passing the buffers to and from That's mostly used for the language servers and they don't really care too much about the file name on disk They just want the newest version of the file so that it can say It'll use the the compilers will use the files on disk in most cases But if you have a modified buffer, they want that modified buffer which may not be saved yet to disk so in those ones the path doesn't matter too much in the other case we don't the the Debugging and symbols case it doesn't matter too much because We're looking at those from the outside world and the debugger itself is dealing with them inside That container so there's no real copying between it's just How does the how do we find it from outside? Yeah, no problem any others With that I would like to see if any of you want to come help build things or test it out I'd be more than happy to help you get started. I didn't really demo the features of the IDE too much I'm happy to do that in person at the GNOME booth. I'll be there a little bit more and Cool, thanks for coming All right. Thank you very much All right, I think we got about a 30 minute break and then we'll be back for the next round of talks See you guys then Hi everyone Who's excited to hear my talk and this is working too great? So I'll wait a few minutes to let people come in and it's not 130 yet. Yeah, we'll give people a couple more minutes Yeah, no problem. It's a careful balance of giving as much time as possible to talk But make sure that people actually get a chance to take their seats. Yeah Absolutely Like it I'm just really excited in those I was setting up for the first time The last comments are is that everything just fell apart and I was like, okay, we'll see how this goes like my joke There's always I keep I did um Last week I did GDC a free day. Have you heard that? Yeah We did a panel and I talked about that. I was like you can't depend on anything working like you should in your heart Be able to talk without any like you should hope and like you should prepare for it to be available. Well, like I've had Resolution cut off part of my slides and I'm just like I Feel like I can even talk without slide Like you should be able to give something right if you're completely to pin on your EV You're gonna be real sad when things don't work, especially like little rooms and Slide design is hard. It's like not my it's not my skill Yeah, I Artists yeah Yeah, she said she gets it on not creative marketplace. Okay, and if you do that presentation a couple times Plus it like reduces that stress of like I'm like, can we just like I mean as long as it's like a language I sort of Let me start at the 131 All right, I think we will get started now Welcome back to the developer track everyone And I will let our speaker take it away talking about how to get a raise take it away, michelle I keep coming up better. Yes. Can everyone hear me? I'm too small for the tiny mics. It's like I can't use airpods or anything. They always fall out So I should really know better So hi today we're going to use data to get you a raise which I'm sure you all deserve During the course of this conference. I'm sure you've learned lots of things and I know it can be overwhelming So instead of frankly trying to take notes Here's all the links that you need Everything I'm going to cover today will be on my website and you can also go straight to the example project repo So you can just copy and paste and not have to worry about anything If there's one thing I absolutely want you to remember from today. It's this Do less accomplish more So let's make some cool stuff But not work too hard So I want to make sure I answer all of your questions I really want to make sure everyone understands what i'm talking about today I want you to be comfortable whether you feel like you can raise your hand or not So you can reach out to me on twitter using this hashtag and I'll answer any questions at the end As much as I'll answer the the raise hands question as well So feel free to Raise your hand at any time. I'm happy to take questions I'm here to help you So you're here today to learn about how to use data to help you get a raise Personally, I'm a project based learner So we'll talk about a business challenge and how to solve that using new technology I'm going to discuss my method of collecting and formatting the data And show you how to build and deploy your own tool to track and report the data using aws and python That way you can accomplish two things at once Try out a new technology And have a method for improving your career Because I want you to do less and accomplish more So here's all the different tools i'm going to use to make this app I'm going to go in a lot more detail later, but just so you know what we're covering And if none of these words mean anything to you Great i'm going to explain it all so it's python 36 aws services and all those services from aws So now that you know the tech here's the agenda Why did i choose these tools? A definition of terms What is the app? The data i collected to make the app Actually creating the app And then alternatives and future plans for the app So i'm going to start with why did i choose these tools? To explain why these aws services and serverless apps in general appeal to me I want to give you a bit of my background Eight years ago. I got an extremely practical degree in 3d animation Luckily for me had some experience coding So when a job came up to work in a tech department at a vfx company I jumped at it Working in entertainment technology is very different than other tech companies Technology is not their business art is Technology is just a means to an end Anything that will get the artist producing higher quality or faster images will be used The work cycle is also not in sprints quarters and years But instead is counted in projects and film shots My first job was focused on reducing companies biggest expenditure, which was artist time If you are working on a tool and it was not done fast enough to actually help The project you are working on it was useless This taught me to be practical and put business needs first Not technology If that wasn't my first job I would definitely suffer more of the consequences of what I like to call the engineer mindset Having an engineering mindset means never being able to turn off that part of our brain that says I can tackle this problem even if I don't have any expertise in it How many of you have been waiting at a store or an event and thoughts yourself how an app could make this better I don't want to tell you how many grocery store apps. I've envisioned in line I've never worked in logistics But because I can't turn it off. I want to fix the issue This problem solving mindset is generally a positive trait but can lead to some unintended consequences Let's talk about a common scenario Engineers can face You're given a work assignment Your customers want a big new feature You now have two choices You could either start coding right away or spend a bunch of time researching I know what I'm inclined to do Getting code on the page seems like it'll be faster and is also much more fun But let's think about this By not taking advantage of available tools You're saying that you have a better solution than everyone who has ever had the same problem Sure, you might be smarter than that one person online or even two How about a hundred people iterating together a hundred thousand You can never beat that as an individual Of course Not every problem has been solved and you might have to adjust what you find to fit your specific needs But if you can describe your need as a general technical issue A hundred others probably have as well So Why not let the tech community and I know you all love open source because you're here Help you so you can Do less than accomplish more So you're probably wondering what does this have to do with serverless apps? Or if you're like me a few months ago What exactly is a serverless app? I'll get to that I like to think of serverless apps as someone doing work that I'm not interested in So I can focus on getting great features out there for my clients They're also a great example of letting a thousand people solve my problem Before I started working with serverless apps most of my personal projects met a dead end at the same place I was great at designing apis and functions But no interest in learning the hundred things and different concepts I didn't understand to actually deploy my app It wasn't that I was incapable It's just not what excites me So I did not make the time and got easily distracted into new projects without actually finishing the old ones which Probably familiar to many of you So before I get into the technical details, I'm going to define some terms One of the joys of python which I write in is that there are possibilities of endless The projects that I make and the projects that you make could be vastly different So let's start with the one you've been waiting for What is a serverless app? So here's the definition directly from aws Serverless computing allows you to build and run and applications and services without thinking about servers Serverless applications don't require you to provision scale and manage any servers You can build them for nearly any type of application or back end service And everything required to run and scale your application with high availability is handled for you So you know what that sounds like Doing less and accomplishing more So let's talk about the app itself What are the basic building blocks of an app? I'm sure you could debate me on this and now that you can find me on twitter. You definitely will Uh, but for the purpose of this talk, they have five parts First is the data storage or some way of persistent information Get anything from my sequel to postgres to a csv file some way of taking that data and holding on to it Next is the algorithm doing something with that data doing the calculations or organizing it differently It doesn't it doesn't matter as long as you're changing in some way And then a quick slide here some advice I got when I was in animation school So you probably haven't heard an engineering school is My teacher used to say with animation anything you can imagine you can make just like with tech But if you could just film it It's a lot cheaper. So if you don't need to change anything Don't build a whole app It's a fun some fun advice So after that is the interface this is some way of running the algorithm and accessing the data It could be an api a script or even a graphical front So next is the home somewhere out of story or app So it can be accessed access outside of your computer Finally the deployment how to get your app to its home So hopefully a lot of these concepts are familiar to this audience Uh, but I just want to break it down so I can show all the pieces of the app that i'm envision So now that we know the different parts of the app Let's talk about the app. We're going to make This app is called roi tracker We're going to make an app that tracks your personal return on investment So that at performance review time you have the numbers to make your case I like to use the business term ROI because it puts me in the mindset of Am I making more money from my company than it spends on my salary? Am I making them a lot more? Do they know that and if not I better tell them so they give me more money And how we do that is a formula called success statements The origin of success statements is from a communication failure A manager was given a large team After a year she was called into the office of the vice president and asked What has she done with that entire last year? Well, she knew she had grown her team and had implemented improvements She did not have the numbers to back it up To the vice president it looked like she had hardly done anything all year When she felt like she had gone above and beyond This is when she developed success statements Success statements are a way of translating your accomplishments into the language everyone in business can understand money So the content of a success statement should be these three things 140 characters or less just like classic twitter You should be able to quickly scan through a list of them like a resume They should be written at a high level and easy to comprehend Where someone might not understand what you do They should be quantified In money or time Money is the easiest But since salary can be one of the biggest corporate costs Time can be translated into cost savings And I really want to emphasize the high level because it's important For people who have no idea what your job is to still understand your impact to the company So here's the exact format of the sentence By having this consistent format It's easy to read and keeps you on message I Improved this thing using this method by this measurable amount So reduced storage costs By archiving data saving five dollars a monthly the reduced storage costs is improved this thing By archiving data is the method And the measurable amount is five dollars monthly and what you could have done was much more complicated than that But this is the sentence that anyone can understand of the street Is that I save five dollars monthly And here's a second example this time using time I reduced manual testing By creating automated tests saving 60 minutes weekly You improve this thing by creating the automated test, which is this method The measurable amount is the 60 minutes weekly So now you might be thinking I learned this great communication method And I can't wait to write all my accomplishments down to success statements But why do I need that to track them? You just told me we don't need to write apps for no reason There is a reason So these are only two examples, but at the end of the year I'd have a hundred And then I had to sit sit there and manually comb through them Figure out which ones to include by date and calculate the savings. It was a pain This is why I made ROI tracker So earlier when I went over the parts of application, which you can see again on the Right Taking left from right We'll see you get on the right and then on the left you'll see the tools that I use for each So dynamo DB is a data storage I use It's a non-relational database and it's more like a dumping ground for data instead of an or an icco database However, it's very easy instead of to use. So it's really great for small newer serverless apps And there's lots of other use cases for it. But in terms of what I like it for So the lambda holds the algorithm. Is it the meat of the serverless app? A lambda is simply a function. That's it If you've written any script or tool before you can transfer that into a lambda and run it on a serverless app API gateway is the interface It handles the routing use and schema validations of a traditional web framework like jango or flex S3 is the home S3 is a flexible storage area and connects easily with the other parts And finally, there are many ways to deploy a serverless app for aws For this one, I found sam to be the right fit Sam uses a yaml template or json to tell aws. These are the services. I want to use please set them up There are simpler ways to do this. There's chalice, which is a python library And more robust such as cloud formation, which is directly interfacing the aws But for this app, I decided to take that middle ground of Room to Experiment but not too much complication So one final step before actually making the app I didn't need to know what data I wanted reported. So I landed on two things In a given time period such as the last year I wanted to know the total time saved and the total money saved I wanted to know not just my immediate impact, but over the entire year If I saved $100 on a monthly process, that's $1,200 after a year If I saved 10 minutes a day, that's 43 hours a year And the employee whose time I saved is making $20 an hour It's $860 that year These small changes add up So what information do I need to collect to get these totals? For the total time I collected three things for each innovation The duration in seconds or how much time I saved When I started saving the time and how often it recurred Was it daily yearly or just once? For the total costs, I collected similar information But except costs and cents instead of duration in seconds. So very similar Finally, I wanted to make sure I could give my report entirely in money if necessary I needed to divert time to money For this, I used a number of employees affected by the innovation and their salary Of course, not everyone can get employee salaries and you might have to guess So it's optional, but it's just a nice to have if you can Especially if you're solving for your own salary, how much time and money of your own salary you save is really great So let's make this up So when I explain the different AWS services I use for this app I'm going to keep referring back to this SAM template So I want to go over what that is first What is SAM? According to AWS again You can use SAM to define serverless applications and simple and clean syntax SAM is a helper that creates those cloud formation templates that AWS reads Cloud formation Is that YAML template they use to tell AWS? I want all these services and I want them to connect in this way So if you want to do something that's more complex than what SAM provides such as if you want to set up a vpc Um, you can use cloud formation commands directly in your SAM template or change over to using cloud formation directly So to be perfectly clear SAM uses a YAML file that translates into cloud formation template That aws uses to define your app So this is the top of the YAML file These top two lines are boilerplate and a description of your app The next section Is parameters You will use these in creating names for different parts of the app The stage name parameter is that you can have ROI trackers in different environments Like QAM production and for our normal production pipeline And you can easily see which ones you're working on based on the name By starting all your components with a service name parameter They're easy to find When you have multiple apps So the next step is defining that data storage Let's go to the resources section of the SAM template Resources are the components of the app like the data store and the interface So this is the data store resource DynamoDB is that non-relational database fully managed by Amazon While I would have preferred a relational database like my sql That takes more work to spin up and manage it I opted for ease of use. Let's get this up there and see how it goes You can see where I've started using the variable names That we set up at the top of the SAM template I set the ID as the key And I set the throughput low since I'm the only one using this Actually accessing a database I used the Python SDK After I initialized the client I created the table Then I was able to use the table to get all the CRUD components pretty simply I created an item using put There is no direct there's no dedicated create The dynamoDB will create them the item if it does not already exist Then I can retrieve the item in two different ways As a single item by using the ID Or by running a scan to get the items Now if you have a giant app you obviously don't want to scan your own table to get information But again, this is just a personal app just an ease of use and a way of learning the system So and then for update I use the same put I use recreation and finally A delete to get rid of any of my mistakes and I think it's hilarious that all the rest of the commands then disappear um, so That that's it for dynamoDB Just a bunch of CRUD commands to get the information I need So now let's talk about the interface I use API gateway to define all of the views And API gateway endpoint can activate other aws services In this case, I can add it to a lambda which I'll go over more in the algorithm section So I decided that I needed five endpoints for this project I always start with CRUD which you can see here on the success endpoints. So I wanted to create Uh retrieve update delete all my successes that I put in And then I just added one more Call report to present a summary of that data. So I'll add up all that information for me So in order to define the endpoints and add features such as schema validation, I use swagger Quoting swagger.io swagger allows you to describe the structure of your apis so that machines can read them Swagger is an open source tool that aws supports aws can read that swagger file and define the api gateway accordingly and there's great Interfaces you use they can also pull in that swagger file when you're creating documentation for other parts of your team So I added this swagger file directly in that SAM template, but you can also use a separate file So here's how that looks like and I know that it's completely unreadable So let's zoom in on some of the main features of the template I'll be skipping around but you can also find all of this in the repo um Sorry to lose my voice So we're first defining a resource as an api gateway Again, you can see the variables in the name that we set up before And next we'll jump Oh apparently it's one delayed. So that's what I just explained So Next we'll jump to the start of that swagger definition Right under that we have A sub validator called basic Yeah, basic that says to validate both the body and the parameters Since dino adb is no sequel database. It doesn't inherently allow for limiting what data can go in Uh, but I wanted to so which is why I add the validation on the endpoint So I can still add whatever I want to the database in my code But hopefully and anyone hitting this endpoint won't be able to So after that is the beginning of the paths Here's the initial posts Did I I went too far? Now all my cool animations are going to take forever to get back Yeah, so we're going to the path. Okay, so this is the initial post um Here's where it says to make sure a body is provided and valigates against a schema. I've called success You know the same word I've used everywhere to define my success statements The success statement schema the success schema As all the data we want to collect such as duration and start tape Yeah, so here's the definition of the schema that we're going to have later And finally and it's cut off because I wanted to make this a little more readable But you can find it on the repo this section tells aws connect that lambda that we're about to write To the end point so that when I hit those end points, it actually does something So let's talk about that algorithm Landers of functions that aws runs for you. That's it the definition from amda from amda from amazon Is aws lambda lets you run code without provisioning or managing servers You pay only for the time you consume and there's no charge your code is not running So what does this mean this app is not currently costing me any money? It only costs money when I actually do stuff with it Since i'm the only one using it it rarely costs me any money Uh, but it's also really nice when you have huge apps at scale where you don't have to Keep things up and running all day when something is not hit It's only hit periodically So it's pretty neat I won't go over the lambdas that back up the success statement crud end points Because you saw most of that code when I went over the dynamo db It's all that boiler blade of when you hit that get on success go to dynamo db and get that item and then display Um, so instead i'm going to go over the report lambda So here's what that looks like in the sam template. I'm going to highlight a few sections So the hamler determines where to find the function in in this case. Uh, there's just a file called report lambda.py I'm going to make sure it knows the name of the dynamo db table So I pass that along as an environment variable and then finally Um, I link back to the api resource that Has that swire definition At oh, yeah, there's one more thing which is the policies That give the lambda permission to actually access the database Anyone who's worked with aws knows the iam permission policies and get really complex Luckily for us we have only a few things we need to throw in there We need to make sure that that lambda can access the database and return information So I made a bunch of functions for the lambda to call So remember the totals I wanted to calculate These functions do it all if I name them right they are self explanatory But you can see the actual code as well as the test with a bunch of examples in the repo It's basically standardizing the data adding the numbers up. So I have these cool summations The only remaining step is to actually push this to aws so you can access it and others can see it This requires an s3 bucket to store the actual files Then you can use sam to package and deploy I created two shell scripts to help with this And they're included in the repo Whenever you create a new file or add a new product to your repo you can run the update build It puts the files in the right place for sam to find when you're ready to deploy the stack Run test stack you can find more information about how these shell scripts work and the commands they Contain in the repo read me. They're basically like three to five commands each I just got tired of typing three commands over and over again as we all know And there's since I've written this talk there's been more tools that allow you to do this automatic But it's nice to know what's actually happening to deploy. So if something goes wrong So like I've been using pycharm to actually deploy now. There's literally a button that says like deploy It's awesome sometimes doesn't work. So it's good to know The actual methods behind it That's one of the reasons actually Okay So once it is successfully pushed you can double check everything you made and test through the aws web interface Which was a lot more readable when I made this slide You can actually sidestep the entire sam process that I described And create everything in the web interface It's actually a lot faster, but that's a more burdensome method when making larger projects also You it's not a production ready way to do it. It's basically you're learning through using the interface But that's not how you can do a production project because there's no way to Standardize what is getting deployed? um So I also found lots of tutorials that use the interface basically every time I googled how to do something It was like oh just press this button But that's not what I want to do. I want to create production ready tools So since there are so many less that help with the exact sam syntax I want to make sure you didn't experience the same frustrations I did and Be able to use the sam template. So here's how your api will look so small Uh, so if you go into aws interface You can see highlighted. I don't know if it's actually readable But what it says is here's the total cost saved Here's the time saved and the deploy salary saved over time After I put in these five success statements I just made up five things and put them together and you get your summary so Let's see So yeah, so I posted all the success statements that are actually included in the repo test You can see that in the repo the success statements I use And then when I run the report endpoint, I get this summary ready to bring to my performance review So obviously you don't want to just bring a json output to your performance review You want to do more to it that's kind of In the list of features that'll be nice to have later, but I just want to get the basic structure out there So you want to there's things like securing your endpoints making cool charts from this and just making it ready to go Um, it'd be also nice if I did just you know Make this more available and out there There are also other companies offering services like this such as Microsoft IBM Who are both out there and google? I've heard good things. I haven't used them myself The concepts are all similar those five parts of an app You'll find everywhere you do serverless apps and there'll be services that help you along with it tutorials These companies all want you to use their services so they've all they're all very helpful and getting these started um So if you try it out and it works great or it doesn't work great at all, please let me know I'd love to hear your findings um If you're thinking michelle you're just giving us homework to do and that seems like it'll only benefit you You're right. Uh, I want this presentation to be your jumping off point and I want you to learn cool things So I can use because I want to As you know by now Do less and accomplish more So now I bet you're wondering what happened to that manager How did using success statements help her career? You want the receipts? She became a director and is currently in charge of multiple departments After she taught her team members about success statements. They were able to achieve great things as well like me These include every year 25 of her employees are promoted At her company 18 percent of those receiving company awards And 20 percent of those celebrating 15 years with the company are from her department So I hope this inspires you to build your app gather your data and impress your boss This will show all of your hard work and your bosses will have no other option Then to give you all ways You're welcome Feel free to send me a percentage of your newly acquired wealth That would be great Um anyone who wants to continue this conversation feel free to reach out to me on like din twitter Or one of the many public slash groups for probably all in um Like I said, I still plenty more to do with this. I know there are probably a few of you who might not have questions now But think of something later I'd love to keep the conversation going and twitter is definitely the best way to do that Additionally, I appreciate any feedback on the presentation today um What you liked what you'd like to see more of for future talks and things like that Um, so thank you for your time and now we have plenty of time for questions I'm gonna check twitter too. See if anyone did it. Oh, let's see. Oh, no questions on twitter Well, I try it's a new thing. It's a new thing. I'm trying trying to get questions off of twitter So sometimes all experiments don't work. Are there any questions in the audience? I can answer for you right now any specifics anyone thinking of Success. Oh, what's up? I don't repeat what she says because there's no other mic um, so I haven't used it as much as I'd like to So I've been mostly been talking about it But it it helps me think about it every day since I've made it So I think about every day how am I saving the company money? And then one of the other next steps that I think about a lot is how might make the company money So I've currently made features. I'm like, I know we're getting More money from people from our clients based on these features So I think about that all the time and then I think about like, okay, what can I ask for since I've done all these things anything else Okay, thank you very much for your time and have a great rest of your conference Hello ladies and gentlemen um I would normally wait until the av person is here, but he is not here and it is time to go So I guess we will start and since I'm already set up and I've got the microphone. I'm assuming you can all hear me Wonderful As the talk slide says My name is dr. Josiah Colson and today I'll be presenting eliminating redis snapshot memory overhead In a lot of ways this talk should really be entitled eliminating struct d snapshot memory overhead Because the likelihood of upstream merging is pretty low And I don't want to give the odds of that um And struct d is my redis fork that I've I got the name from Lars up here in the front row who uh, who've been paying attention for domain registrations and There's a huge bro Hooked me up So yeah, so eliminating struct d snapshot memory overhead Let's get into the agenda Uh, so that you know where we're at along the ways Uh, all these slides will be available online after the fact. So don't worry about taking pictures They're also no transitions. And so don't worry if you look at a slide and you're like, is there anything more There's nothing more on the slide. There's no transitions. This is White I was was written on a raspberry pi. It's not like it. You know, I didn't have a whole lot of extra cpu there Um, so yeah, so agenda, I will talk a little bit about who I am then the problem that I'm trying to fix Uh, which ends up ultimately being a thread synchronization problem And we'll talk about what happens in the foreground and background during that And then we'll talk about some of the limitations and caveats to the use of the platform itself And then we'll eventually prove our 99.8 percent reduction in overhead for this operation with math And then at the end we'll have questions. But if you've got questions along the way, I'm sure I can answer them So let's get to it. Who am I? Um for the I guess half of you don't already know me I've got a phd in theoretical computer science from 2008. So that doctor is not like an honorary title. That's a iron that goodness I've got about 20 years of daily or weekly python As of april 13th because that's when python 152 came out and that's the first version I used Um I'm also former core maintainer of some internal python libraries asynchorus and chat and related And I've got about 10 years of daily weekly cc plus plus which is extra applied to redis because hey redis is all see um Got decade of answering questions on mailing lists about just about everything that you might want to ask about on python wx python data compression redis and Other related things and also the author of redis in action a brook on redis Came out about seven years ago. Uh, that was nine months of evenings and weekends while building chow now Which was a great adventure all of it um And also the author of several open source projects in python primarily summons some other languages But you know you live in the python land and you build a bunch of businesses on python. You're gonna do some python So, yeah, so that's who I am. Let's get to it. What's the problem? So problem statement in four stages The big problem is that periodically redis and or struck d needs to take a snapshot of in-memory data And put it somewhere else Sometimes that's on disk and sometimes that's over the network to a new replica Now this happens on new replica connection old replica connection On request with a bg save they call it or a save On a schedule you can set The schedule to save periodically After a certain number of writes over a certain period of time Or it can rewrite the entire representation that it has of what they call a Pend only file which is a snapshot of memory plus a listing of changes And that can you can automatically rewrite that upon reaching a certain Large size So there are a few different situations where a snapshot can happen And in particular we care about background snapshots because background snapshots are what induces A memory problem. Oh wait, what kind of a memory problem? Well To snapshot we produce a fork or we start by forking the process And then the child Process will go and take the memory and it'll dump it all to disk or over a socket And then at the end it will exit the child Now that's not really a problem in and of itself The problem lies in what happens when other stuff is going on during this Like if this is all that happened, this wouldn't be a big deal But the problem is is that as your parent and child process data vary The total memory use of the combination increases Now that's not really a big deal except for when you actually have actual load and real memory Here's an example. I like to bring up So you've got a redis with about 10 gigabytes of data in it 10 gigabytes is not all that much data, but it's enough to get a lot of applications started But at this size you tend to get say a thousand operations a second on in the ballpark or more Sometimes less depending on your needs, but for 10 gigabyte of redis This kind of load is is pretty expected Now what happens after the fork when you receive rights to your parent process That causes Memory pages to differ in fact unique pages shared by both the parent and child process when they're written They get split they go from one page to two pages. That's the copy on right portion of fork So every right you get to one process more or less induces a page copy So you could write one bit and that's 4k now Another bit another 4k another bit 4k another bit 4k another bit 4k You can touch a bunch of pages and induce huge memory use In a situation where say you've got 4,000 write operations a second What you end up doing is you end up touching approximately 4,000 pages per second, which means that During the snapshot your memory use will increase By approximately 16 megabytes per second So you'll see your your line kind of going up into the right your total memory use up into the right Normally up into the right is a good thing, but in this case memory use bad thing And so over the course of say a minute, which is how long it approximately takes for a 10 gigabyte snapshot to create a snapshot You can end up with a gigabyte of extra memory being used for a 10 gigabyte redis And so you have roughly 10 overhead on this redis. Okay If you increase your operations a second to 40,000 operations a second You end up seeing 160 megabytes per second increased memory use And then over that 60 seconds you will hit your peak of approximately 10 gigabytes and then it will flatline And so you'll if you start out with 10 gigabytes of resident redis At the end of that minute you'll have 20 gig a total resident redis for total doubling of your memory use which is painful For for operationalizing that means that if you want to run redis in production You need to have a 20 gig server to run 10 gigabytes of redis Which is kind of painful and kind of silly So our attempt is trying to remove that But the big problem is that If you touch as little as 1 percent of your data you can induce 100 overhead on your snapshot Sometimes this isn't a big deal. Say for example, you've got 10 gigabytes of redis. Maybe you're spending a minute. Maybe it's only 10 10 gigs of extra overhead If you've got a 100 gig redis Now you're talking maybe 10 minutes to snapshot and that 100 gigabytes matters Nobody's really got 100 gigs. You know just just laying around as far as I know, I mean But yeah, so that's tough Really This is what we're trying to fix So this is a graph of total memory use of redis on a platform The we started at zero because we're not running We load a 10 gigabyte resident redis. It takes time to load. That's the straight line up into the right after loading What I've done just to show what this looks like is I've induced a background save Immediately followed by a benchmark operation that randomly touches keys in the key space Just like you would have on a regular application. It would be randomly touching keys more or less And so that's what I did. I started a bg save and started randomly touching keys with the benchmark As you can see almost immediately the memory goes vertical Now in reality, it didn't actually go vertical. That just means my Detection of how much memory use took a little bit of delay But as we can see within about five seconds, we went from 10 gigabytes to about 12 and a half almost 13 gigabytes of usage in about five seconds That's what happens when we have copy on right. We touched a bunch of pages introduced copies And we have additional memory use Now over time as our randomly touching as we're randomly touching things our memory use goes up and it approaches an asymptote Near 20 gigs it turns out that that's actually when the benchmark or when the snapshot completed And so that child process could now exit Reduce memory use and we can continue on with the benchmark Now what i'm trying to do and what this entire talk is about is about getting rid of that Rounded hump on the top because that is complete waste. There's no reason why that should exist except for Uh Not thinking ahead as to the implications of this kind of the thing and i i've worked with companies who have large resident retuses and this kind of thing happens And i'll have a an isolanic note for my first redis experience and and and how we dealt with this When we still had to deal with this Um, so yeah, we're going to get rid of that bump So ultimately if you if you realize that your problem is forking and you're copy on right Then that means you can't fork and if you can't fork you p thread create And if you p thread create now your entire world is thread synchronization And memory races and all the related shenanigans In order to deal with this we Broke this down into three major thread synchronization problems. The first was just a standard mutex The the redis module gill that controls all of your major accessory accessing to Your connections or data structures Everything important to redis that makes redis redis or struck d struck d as it may be um That's the first part You can ignore that because it happens all the time and it mostly is just a it's my turn. Oh, no, it's your turn Then there are two variables which I like to call progress and next progress And in a redis snapshot we've got to traverse the key space and Transform in memory data to on disk data or transform it to send across the wire This progress tells us where we are in that process in redis we have numbered databases and The databases are just hash tables and so if we remember those then we can remit we can We can mark our progress as a database identifier with five bits a 54 bit slot index In the hash table because we've got hash table slots And a five bit slot count The five bit slot count is a power of two so we have power of two hash slots and we've got 54 bits It's good for I think 10 of the 15 total entries. I think we should be okay for now And yeah these these cursors define ranges of keys before progress defines or any key any slot That comes before the progress Cursor is done and processed Everything that comes after the next progress cursor is unprocessed and everything between is processing And that's only important for the next step because those ranges define some things that we can do on keys at different times And yeah Then we also have This thing which I call the dumped set. It's not a great name whatever But it really just represents the list of keys or the list of slots That have been snapshotted by the foreground thread We'll get into that in a second But suffice it to say that there's some operations that you perform in the course of your redis While the background thread should be snapshotting that you may need to do things in the foreground So you can either wait for the background to finish or you can snapshot now So let's get into what the foreground thread does So foreground operations On a command request We check all of our keys against these processing Processed and unprocessed segments Any keys that are living in the unprocessed segment we go ahead and dump if they haven't previously been dumped And we mark them as dumped If anything is in processing we'll go ahead and block the execution of this particular command execution So we can send it off to the side and let some other client do its business But we'll block this one for one cycle of the background thread and then if it just so happens that Any keys are in the processing between those two cursors that we defined Yeah, yeah, we block it and then after we've processed all those things if nothing is in processing Then we go ahead and execute because at the end we would have You know dumped the keys or verified that they've already been dumped And that's it that's pretty much it for the foreground There's a lot of details that are hiding You'll see the details in a minute Now if we've got the foreground that's running commands, we've got a background that's snapshotting Clearly our snapshot should be pretty easy while we're not done dumping We'll acquire the gill to make sure that we're synchronizing properly We'll update our progress and next progress cursors so that the next time our main our foreground thread comes up It knows exactly where it should pick up Then we also merge any of the foreground snapshot data that had been completed We do this based on buffer size. We're not doing it all the time And then we go ahead and unblock all of our Foreground transactions that had been blocked for a cycle and then we release the gill So the main thread can go and start executing its its business And then while the the main thread it's doing its business We'll go ahead and snapshot our data for up to a millisecond Which is to say we'll work on data between progress and next progress for up to a millisecond If we run out of work, we'll finish early If we had too much work to do and we And it would take more than a millisecond. We'll stop at a millisecond now. Why a millisecond? Well because we stalled Incoming requests based on what data was there if there's data that we needed to stall We want that to come back fast and so by delaying only by one Or maybe two or three depending on our progress next progress advancement We can minimize the latency on any incoming commands Okay, now pretty clearly You can see well, we'll come back a second. So one of the ways that we prevent Commands from being blocked for too long is we limit the size of our progress and next progress range And we do that Very simply via one point gradient descent based on how much work we expected to get done and how much work we actually got done So we change that automatically based on how much work we're actually doing And that means that on average Like number one, we finish things when we expect them to and number two We're not blocking too many slots In a particular cycle That are necessary Because if you you know, you can conceivably block out the entire range of keys And then you never have to worry you never need that dump set But in order to minimize your dump set and minimize other other operations we only snapshot for a millisecond Also, you'll notice that because the foreground thread is also Acquiring and releasing the lock every cycle that would induce delays It turns out that all this stuff and like between our acquire and release Over the course of a roughly 60 second snapshot operation The total amount of time waited by the foreground thread between their attempt to acquire and release Is under 150 micro seconds over the course of a minute So we we actually count those things because we thought that was important to pay attention to Overhead and yeah, there's basically no overhead relative to the foreground thread And synchronizing with the background thread for these operations So some caveats obviously small keys are best If you're going to be creating a representation of a key value pair in memory Then small memory that small chunk of memory if it's smaller is better If it's big well, how big Under 10 megs you can usually do pretty well You'll see a few milliseconds latency on individual keys that are accessed on the first time they're accessed during the snapshot But much larger than that and you'll see larger latency spikes There are a few operate optimizations I'm in the process of doing to make all that better stronger faster, but it's not done yet Now I also have some integrated soon big data structures That I've been purpose building and I actually built over a year ago in order to go into this I just Got my order of operations wrong and in integrating And those big data structures are designed for transactions So that you can say prepare a modification or actually do a modification on say a one terabyte hash table And have that latency be under a microsecond and being able to go forward or backward On that structure in microseconds That allows us to do similarly enough snapshot operations on that data to also progress for giant structures In say microsecond latency for foreground That's not in yet. It's coming I'm only one man So, yeah, so How we solve this and how we prove this on a mathematical basis on how we remove 99.8 percent Of the overhead as we start out with actually counting bytes in memory So in redis we're struck d The smallest data that you can represent is represented by these two structures The r object which is a 24 byte object that represents the type of your in-memory structure along pointing to your key and some General information about the the value being stored And then you've got your dict entry which actually points to your r object and actually stores your value Or pointer to the value In this case, if you're really careful, you can actually store an integer to an integer and that can take up 48 bytes That's the smallest data and we're going to ignore the main hash table overhead that points to these things We're going to ignore all of the other overhead. We can prove this using just those two things So start out from there Got 48 bytes per key at least let's do it Version one We used a hash table a plain hash table redis hash table to store The key to a bitmap of databases that the that the foreground threaded dumped to The bitmap was just a 64 bit int in that dict entry pointer And so it was free. We were already storing that information And the key well We actually had to store the key, but if we're again if we're assuming that the keys are so small they evaporated We only need a dict entry so we can go from 48 bytes per entry To 24 bytes per entry and if that's all we think about we go from 10 gigabytes 100 percent worst case To 50 percent worst case here guaranteed. We're never going to be more than 50 percent more because remember Standard redis if we were to store this we'd still have to store the key names right that's still going to be half as much as the other part But again, we pretend like it doesn't exist. So in the worst case we saw 50 guaranteed You remember that graph where it had that nasty Big thing where we saw 100 additional memory use Well in practice running that same test instead of 100 gigs we see anywhere from 50 to 250 megabytes So we're already substantially better in practice even though our worst case Is only two times better So that was the first version that was July august last year Uh version two instead of using a hash table to store keys. Let's use a int Let's use a hash based int set to store integers that represent our slots. We've got that database slot index slot range Let's just store those directly For where keys live. So instead of dumping a key You dump the whole slot everything that was living in that slot you dump on request And by storing that we go from 48 bytes per key to represent to eight bytes per slot And if we guarantee at least one key per slot, which is something we can do Then we have a worst case guarantee from 48 to 8 Of 17 percent. So worst case we have 17 percent overhead But again running it in practice. We see closer to 20 to 30 megs in practice, so we went from 10 gigs To 20 or 30 get megs, which is Not bad as far as i'm concerned You know, but but again not enough I I like in practice, but i'm a theoretical computer science. My doctorate is in theoretical computer science So i'm always looking at the worst case our big o So let's get better So version three we use version two until we hit a critical size And then we'll switch to a bit set that's it. That's the optimization The other parts are that they're this uses c 11 atomic so that we can actually do multi-threading Correctly with like real semantics and not like hoping your volatile reads and writes make it through Incidentally volatile reads and writes with locks don't work In fact locks by themselves with enough memory don't work by the way You have to be explicit about your sequential cst or your other atomics So this is how we do it Instead of 48 bytes per key. We get one bit per slot Okay, and so our worst case Is guaranteed 0.26 percent if we just do that simple conversion and use all our same math But we adjusted it so that it's less than 0.2 percent as implemented very easily And so for a 10 gigabyte resident struck d you would see less than 16 Mibabytes that is two to the 24 Wanted to be precise there Just because that is literally the largest number of slots that can exist in a 10 gigabyte reddit resident reddus And if you've got fewer keys because your data just happens to be bigger on a per key basis That overhead goes down If you've got half as many keys your overhead is half as much A tenth of keys a tenth as much so we keep winning every time You've got more data And so how do we get this down to less than 0.2 percent? Well, it's just a constant it turns out If we assume that we've got 48 bytes per key Turning into one bit per slot All we really need to do is pick a number of keys per slot That changes our denominator on this 48 And so it turns out that 1.31 keys per slot is enough Somewhere between one and two if you do 1.31 you get 0.002 as your ratio And that's it That's how we do it. We just Pick our cursors We define our cursors in advance before we even start the snapshotting process and we say this is the cursor that we're going to use This is the type of cursor we're going to use We define a slot count and that just makes sure we have at least 1.31 keys per slot And that's it and we've got 99.8 percent reduction But because this is just a constant we can change it to 2.62 and get a 99.9 percent reduction All we're doing is twice as much work per slot and Yeah, and in the future it's very likely that i'll adjust that to be like 100 keys per slot because the cost of snapshotting any individual key from the fourth round thread is nothing so eventually this will basically be free like not even just 99.8 or 99.9, but like Basically, you're not going to be able to measure it on any platform with less than 100 or 200 gigs of actual real memory use so yeah And so this is what we see in practice the blue line is struck d the red line is red as five as released As you can our memory is is a little bit pokey whatever But you can see that we didn't have that bump. It's nice and flat that has the exact same We're going to start up redis Let it fill with memory induce a bg save start a benchmark and there we go and Five seconds after finishing the benchmark will die. So it turns out that we actually Loaded the data faster during this process Can anyone guess why? Because copy-on-write is expensive This is the cost of copy-on-write Like that difference right there that 10 seconds 20 seconds or so. That's the cost of copy-on-write during load That's what it costs to let your nmu and let your kernel handle this So yeah Um And here are qps again blue line is struck d red line is is redis um The cost and this is just during the snapshot operation. So we start the snapshot and we start the operations um The benchmark you can see that eventually redis catches up Because it doesn't have to do that bs with all the the pages that it's copying and no longer has to copy a bunch of pages On the way and it can catch up and do well um, yeah, and so There we go questions does my Does my bookkeeping deal with? um rehashing yes, we properly deal with rehashing rehashing is not a problem The big solution that we do is we There's a little line that says like if you're in the middle of a snapshot like don't don't don't rehash I added extra codes. We really don't rehash Because the existing code will actually rehash on you it it lies um And so yeah, we really stop rehashing. We don't need to actually we can There are some steps that we can do to make rehashing possible. We just don't because The long story short is that threaded reads on arbitrary memory regions Are not consistent across cores unless you've explicitly sent data to that core And None of the existing redis data structures have any kind of atomics attached to it And even if you go ahead and attach your your c11 atomic writes um I ran into situations where I was I was writing several and some of them didn't make it Um, but I also have an actual multi core like a multi socket on single cores. I see data communicating on multi socket It doesn't make it across um And i'm not sure if that's an isa problem isa problem I'm not sure if that is a gcc versus g plus plus issue Because c11 atomics versus c plus plus 11 atomics. They're like Different documentation and some of the documentation lies by the way um Yeah, I read a lot of of of Dave preshing And I tried to use some of his structures and not all of his structures worked Um, and they were supposed to work So I I don't know where to go to that go from there If you know stuff that he says should work isn't other questions So why is it unlikely to be merged with redis? um I'll give you three answers. The first answer is that salvatore doesn't really like other people's code Um, I understand that I mostly don't accept pull requests that come into my stuff Um, but I almost always rewrite what they sent me anyways because I like the feature Um, salvatore just doesn't like other people's code. That's number one number two um I've burnt a lot of the relationships that I've had on that side of things because of dissatisfaction We will say No, I I spent a lot of time emailing folks and and helping people use redis um And I was no more likely to get any change in as anyone else Um, and the big third reason is because Um, all of this basically shows that a lot of his design choices were wrong Um, and so the likelihood of somebody accepting a pull request that basically says Your design choice here was wrong I mean there's some zeros there as opposed to nines followed by a one I don't know how many zeros are there and so I mean this will eventually be open source Not today not tomorrow. I mean To be honest, I was hoping to also be releasing paid for hosting today But due to the fact that I've been sick with collection of injuries, um I haven't been able to work And I got two hours last week and I saw solved a bug that had been sitting on for two and a half months I haven't been able to work since um So I would like to have released this so that everybody could see and feel an experience that like Along with the other changes because along with threading Did you know that you can make redis start up faster and shut down faster and create snapshots faster? That was all disabled But if I enabled faster snapshot creation, this line would have ended here um And there are other similar things that I wanted to release But I don't get to Because of being ill Lars so vanilla redis right now uses threads for doing f-sync So if you need to To disk it will create a background i o thread and it will f-sync with that background i o thread it will also use background threads for Deleting files like big files because sometimes that's slow And it will use it. Yeah, there's one other thing. Ah deleting data from memory if If it's big and you don't need to do it in front like there's a There's a flush db command where you can delete all of your data from your database and you can make that Lazy so that actually is executed in a thread in the background as well um, but the vast majority of redis operations are I mean it uses a E pull or kq or whatever selector to to multiplex on connections Then it's just command response and everything else is just timer events on that that event loop There's almost no threading Um So We added threads in that we added threads that did different things other than just plain F-sync or delete the memory or those kinds of things we We snapshot memory in a thread We load data from disk in a thread we Yes, uh Today it means that you will use more cores when you're starting up or shutting down redis So so does the threading change the multicore thing? Um I am I do have a threaded redis execution model. I haven't turned it on because In the past I've had some race conditions unrelated to Existing stuff and so I need to revisit that I haven't recently because I was more concerned about Let's get data to disk and from disk correctly. That was my priority as opposed to faster execution Um faster execution is definitely easier just because now every at least A lot of the bits are at least Multithread read compatible Not so much write you can't multithread write and redis yet Give me a minute But all of that will be open source eventually it just Not there yet and not done yet So vanilla redis versus redis enterprise So the answer there is I have no idea what redis enterprise offers because I've never seen the code and never run it I did write some of their documentation and helped them write some of their documentation, but I've never seen the code I've sat in their booth and been a booth babe But I've never seen their code I tried to work for them. I tried to see the code I have not seen the code and I do not work for them How they do it I can't say They have a lot of stuff on crdts Maybe they're living in crdt land I I try not to worry about them too much That is what the dump set is for So you've got your progress your next progress and whatever As you're running through again any key comes in if it's unprocessed Snapshot it immediately And you mark it and now you'll never touch it again. You'll never resnap shot it whatever What I discovered early on and why I also discovered that One that locks don't work With multi core real multi core multi socket Is trying to you know use the standard redis set with just locks in front and behind To do that and what I discovered is that I was in fact Reading multiple keys, but that's only because the existing redis hash table is not Is not friendly to threads at all And once I switched to something that was better representative of that all those bugs went away And actually when you start up redis it has a When it loads the snapshot It actually has a check does this key already exist And if it does then it will die I had to Add debug because I was going through that and crashing redis while you're debugging that is kind of hard But yeah, so so we've addressed that it's still point in time snapshot You can't change data before it gets to the snapshot We'll snapshot it and shove it in the buffer We will snapshot it first Other questions, okay, well, thank you very much for your time and thank you for coming One note on the on the snapshot side of things A final anecdote My first gig using redis was Uh nine years ago working for a company called Adley. They no longer exist. They got sold whatever We had a 60 gigabyte redis That was in 2010 which was pretty big for the time um What we discovered was we couldn't actually run bg save We couldn't Because if we ran bg save well one we would double memory and we didn't have that memory on that machine because Because aws at the time only offered 64 gig resident instances So we had the biggest box with as much memory as we could use So we ended up having two non background save in the middle of the night part of the reason why was bg save versus save a bg save would have taken 30 minutes Whereas a save took five And the only reason that I could figure is that there's something going on with the write load Copy on write and now I recognize transparent huge pages Which are now a warning you get when you start up redis Transparent huge pages are kind of a problem. You go from a 4k page to a one meg page um And now you're now you're copy on write It was from a 4k copy to a one meg copy went from microseconds to milliseconds And now what? Touch a million pages and you're done So yeah, um Nowadays I just wouldn't even worry about that. Well, actually I would optimize my structures number one to take it down to 30 gigs Like I did and then proceed to switch to some other platform like this where we don't even have the problem So I think that's my final anecdote. Again. Thank you for your time. Have a great day Hello. Oh my god, it's working But I feel bad because there's no sound person and they just kind of walked away cool Actually, we can test can everyone hear me How about now How about now? All right. This is my ice cream cone And I will try not to lick it It's been a long few days y'all They told me that apparently the video is being automated or something I don't know There is a camera, but there's no blinking lights on it But I'm supposed to come with the I'm supposed to start right at the time. So I don't know. All right. It is 430. Welcome everyone to the final session at scale 2019 So, uh, thank you all for being here. Yay As the title reads this is a developer talk about My experience in open source or let's say one of my experiences in open source Who here has contributed to an open source project Who here has not yet contributed to an open source project but would like to Oh, I like how the same people raised their hands. That's awesome Uh, that's actually great because every open source community is different. Um, this is a very specific experience story and I just wanted to take this talk to kind of share what I learned When I tried to contribute my very first Code-based technical pull request to an open source community Um, I don't think that my experience is particularly unique This talk isn't necessarily going to make me look awesome Um But I am giving it because I want people to be a little bit more comfortable with being out in the open and struggling out in the open and trusting that Sometimes the community will come up and meet you at your level So let the story begin first about me Um, I made a character sheet for myself. I am a production engineer at github where I get to work on kubernetes and deployment infrastructure every day and it is really awesome Um Some of my skills are a little weird I often get asked if that's a real owl in the photo. Yes, it is I am carrying that owl But the falconry glove was kind of big so it didn't make it into the picture Um, that's my contact information on the bottom right Oh, yeah, this is mirror this works Um, and that's how you can get in touch with me There's not going to be another contact slide later if you really want to follow along or ask me questions Um, I used to play piano for a living. That's a fun fact about me Um in my day job I work with docker kubernetes Uh goling Sometimes ruby on rails sometimes bash sometimes all sorts of different things Um as you do when you're in infrap Um And this is uh, this is my story So let's talk about my community. Um when I started contributing to open source I sort of nose dived into the kubernetes community and a lot of my early contributions for documentation Which I always recommend as an early entrance to contribution, but Actually, this is not a but this should be an and and I also wanted to start contributing more on the technical side of things Um, mainly I wanted to get better at learning go Because that is what my work at the time Was purportedly using but I didn't get a whole lot of practice in Also, I wanted to give back to the community that gave me a platform and Gave me a lot of mentors and help If anyone here is not familiar with kubernetes or the kubernetes community. That's totally fine. This is a generic talk However, it is one of the examples of a really supportive open source community Um, I already kind of mentioned what I hope that you all will get out of this talk. Um, I want you to take away some Courage to be vulnerable in public Uh, just to go ahead and open that pull request even if it's not ready even if it's not perfect And I want to encourage if you are maintainers of open source projects to sort of put yourself into my shoes and See how you can work together with newer folks in your communities as well um, let's be kind to each other and Let's not be afraid to go try out stuff even if it might mean that Someone sees your code in public All right without further ado, let's start the adventure um My very first challenge was uh finding something to work on I was pretty good about finding Uh documentation fixes just by going through and trying tutorials and being like wait, this is broken Or there's a thing missing here closing bracket or whatever Um, but finding a technical topic that was doable for me Uh proved a little bit of a challenge. So, um, I had I tried to look for options So the options are look for open issues in the community that you are trying to contribute to Or look at the code find something that's broken Or see if you can talk to the community and find out where you can help I tried to look for open issues So there are a fair amount of open issues on kubernetes kubernetes And many of them have really really really different background prerequisites So as somebody who is fairly new to programming at the time this was really really hard So, uh as the story goes I tried reading some of these issues and just kind of my eyes glazed over And my brain wouldn't grab onto the words anymore and I just kind of Didn't really know what to do next So I talked to people And people were pretty great. They said hey, sometimes it's really hard to start in this particular aspect of our community Why don't you go and try in one of the other related repositories and I can help find you an issue So success So a very kind person eric feta said hey, I've been meaning to try to get this feature into our testing tool for a while And it should be a simple one line change So why don't you go ahead and work on that and I said that sounds perfect. I get to look at the code Some people are laughing. This is uh, that's because they know eric um However Hmm I thought this sounded great You know, I'll do a little one line change that probably means I get to copy paste It means I get to read and understand the code base kind of figure out what's going on And that would be a great thing to start on so I can work on something more complicated next time So that was my next challenge Understanding the code base. So I go to Kubernetes test in front in this case. I download the code base onto my machine. I open it with my editor And I'm just kind of really confused and look at lots and lots of places And I'm just I just don't even know exactly what I'm looking at or looking for So Again, there's a couple of options And it depends on kind of where you are it turned out that I didn't know my text editor as well as I could I had downloaded all the supportive tools for my language go But I didn't realize that in visual studio code You can actually just click on functions and then they will tell you a where they're used and b where they come from Uh, this was kind of a cool thing that I had to learn But I only learned that after I stared at the code base and despair for a while Yeah There were a bunch of really weird imports and plugins and Just It was uh, it was difficult So after staring for a little bit more, I learned the thing about my text editor Actually from a co-worker. He said wait, you didn't know this and I said no and he said how did how do you ever get anything done? And I said, I don't know um But it turned out to be an awesome, uh an awesome tip and then also It really helped me for this case to learn about the github api Which is what this tool was interacting with but that is very very Topic specific so I apologize for that But it's often really useful to know not only what is this code base accomplishing, but also What is the bigger environment? What this tool is being used in right if you can find an example of this tool being used elsewhere And see what it does then maybe you can draw parallels to whatever you're trying to work on or trying to fix so um As I initially thought It was pretty great. I found the place where I should change the code base And I implemented the fix and then I opened a pull request or I attempt to open a pull request um So who here has had feelings about opening their very first pull request in a new repository or a new community and kind of feelings of worry and dread and Not quite knowing what to do each community has its own standards, right? And so it's actually really really useful If not important to know that community standards, do I just get to open a pull request? Do I have to find a related issue in my case? You know, it was spelled out what I was supposed to do so that part was not so hard, but You know, have you ever struggled finding reviewers? Yeah, so it kind of it depends from community to community. So, um, I think these are some of the common challenges Uh, knowing the community standards Getting somebody to review your code and I think the last one is a little bit more important I hope you can all see the bottom of the display. Um I think putting your code up in public for a lot of people is just a really really scary thing Especially if you're just trying to learn something and you're there because you want to get better for yourself Or because you want to learn how to contribute Um, you might not know that there's special linting tools that this community uses or Yeah, you might be writing in ruby, but everyone on this code base is really really picky about white space for You know, no reason that you can tell but it's very important to them So it's really nice to know that in advance and it's kind of scary to not This is This is where it's kind of important to to know your community, right? If you know that your community is extremely critical and I don't really I don't want to get too deep in the weeds here But getting something into the linux kernel used to or is is just kind of the sting that's just really really hard And you have to I don't know grow thick skin and get used to lots of criticism Something like that each community is different. So be prepared for that um But also as maintainers We should be aware that this is actually kind of a scary thing for people to do Maybe it was just me. Maybe just maybe I'm just worried about all these things Turns out I rolled a 20 on this one Um because of all my docs contributions I already knew the community pretty well and I knew how to submit a pull request Um, I had talked to people already. So that was the obvious person from whom to solicit a code review um And I was reasonably okay putting my code up in public because I felt that everyone that was going to see my code Knew that I was a relative beginner at this technology So let me leave the presentation for one second. What is happening? Oh, yeah, whoops escape I wanted to show you the four line change Let me scroll through this really quick Right there I had to I had to add one struct And then I had to reference it in another file And then it literally was just adding a couple of options to a github api state It was pretty easy It it wasn't so bad Let's get back to the presentation. However Um wait, so, um I found more than just one reviewer I found a handful of reviewers and they all had different opinions and it turned out what they really wanted was an entirely new sub module for that's for this change So that it could do things that they wanted to do within the bigger context of their code base Um, I'm being vague not just so that this can be relatable to everyone in the room But also because that is literally how I felt at the time I was like, okay. You wanted to do a thing that I okay. I don't know why you wanted to do this thing, but Um, so this can happen Um, so of course, what do you do when you get a code review that says change all the things? I was going to argue About why these changes were needed That did not work Unsurprisingly really So the next logical thing for me to do was just to ignore it Just to not worry about it. It's not that it's it's fine. I tried. Okay. I'm just not gonna Nothing happen nothing to see here moving right along wait. You want me to change things? And that was really not making me feel good so eventually I got over myself And rewrote the code Like they wanted me to rewrite it I at this point you have a couple options too. You can you can you know, you can argue You can argue about whether these changes are needed like I tried It usually helps if you have a little bit more background than I did at the time So I decided to just go along with it Um, and that seemed to work so So I pushed my changes everything compiled Um, it it even passed ci. Yeah, it totally passed ci The reason I passed ci by the way spoiler alert was because I had not written tests. It was great So, um anyway more on that later Uh I wasn't I was already kind of in waiting mode And so the next thing I did was I just kind of waited for my reviewers to get back to it and look at it and see if I did the right thing It took a while You see the particular community that I work in is enormous. Um I don't know if this is universal But there are not that many reviewers around and many of them also live in other parts of the world So, um communication is asynchronous and sometimes laggy And one of the standards that I was now failing to observe was that I should really just sort of Write a comment on my pr or ping people on github or um or on private message channel saying hey I made my changes. Can you take another look please? Um, but of course I didn't do that because I was still kind of in this mode of oh, no Oh, no, I'm not really sure what I'm doing. I kind of don't want anyone to interact with my code Um, so weeks past eventually someone noticed that changes had happened And a lot of discussion happened all at once I kind of want to link you to it. Let's see Oh conference wi-fi so Right here somebody's explicitly holding this somebody else wants this functionality by other projects Then we need to look at gating it. Um, and then This is happening then I apologize because I have really not paid attention for weeks and weeks and weeks and then I asked for an example and Yeah, so That happened Let's get back to the presentation oops The main the main thing that came out of this discussion was please rewrite it again so that we can gate this feature and add unit tests So these were the options that I was contemplating at the time I was trying to hide again Um, I wasn't sure I'd kind of already asked for help. So I wasn't really comfortable with asking for help again And yeah, I don't know. I put that last one for I put that last one there as a joke Thank you for laughing. I appreciate it Sometimes I give tech presentations and I'm trying to be funny and no one's laughing and I'm always really really sad So you guys are great. Thank you This will not be a surprise I hid from it again Because you know if I can't see you you can't see me also kittens are cute Um, eventually someone found me They literally poked me almost a month later. Hey, are there any updates on this? So that's where I broke down and they said I actually know nothing about unit tests and go Can someone help me? And people were actually really nice Again, this is my experience. This might not be your experience in an open source community. However um It's been it is I know this is very easy to say But it is my opinion that if you ask for help in an open source community and people are snooty about it and tell you You should know this already Maybe you should look for another community Just just because it's just not I why why waste time on that? It's just I Your mileage may vary on this one People were also empathetic um In it turns out we have a shrug label in the Kubernetes community that you can put on random issues because just because This the funny thing is that this pull request Happened before I worked at github or even like thought about working for github and um, this is a comment about Maybe the github api is not, you know Stellerly documented So that was one of the struggles I had so but people said hey, you know what? They were actually really great about saying this is just hard. It's not your fault. We can all get through this together um So my reviewers were super duper helpful in just finding the tiniest little things that were going wrong And so we went through a few days of churn um Let's see Oh, yeah So I was now writing unit tests Remember how they succeeded before because I hadn't how ci succeeded before because I didn't have any tests now I had them I learned that there was a pre-submit script in this uh repository so To extract us to general general lessons A lot of times I learned projects have Bootstrap scripts you can often you can find you can look for a scripts folder or bootstrap folder or Or or yes, something like that and they will live in there and you can kind of see how you actually compile and build the tool um, and then there are often also pre-submit flight checks that you can run um and This particular repository had one of those And I did not know because I also did not know how to look And so what happened was I pushed to my open pull request. It failed ci I went to look at the ci error messages and it said Hi, you're you know your go lint failed and I was really embarrassed because I probably should have run the linter beforehand and I just kind of didn't So yeah, that was the first step Now I know there's often a pre-submit check before you even push up anything that you should probably run Uh, it's always a good idea to look for those in any repository you're working on Um, I sort of got ahead of my slides here. That's fine Um, eventually I fixed all the white space again white space I I don't know why but I have slightly more patience for a white space in goling than I do in ruby, but Your mileage again may vary but by this point I had started opening this pull request in february It was now may And so I was still working on all the tiny little fixes and all the little hints that all my reviewers gave me And I said okay, finally finally I fixed all the white space. I'm going to push to my pull request And because it had been open forever I Got an enormous giant merge conflict somebody else had been writing on a very not well in on a related feature in exactly my part of the code base and They'd gotten theirs merged because They were faster than I was and they didn't hide from their pull request for weeks and months So I had to deal with the merge conflict So what do you do when you have a merge conflict? Give up Use some random git magic um And then there's the not recommended option of losing track of everything And using your copy paste board to actually copy giant chunks of code and put them in parts of the git tree Where you think they should be But they aren't for some reason and it's really confusing and weird So that's what I chose to do It took a while I got better at git um, I had regrets for not working on this faster But at this point I'd already gotten I was already all the way in everyone kind of knew what I was working on everyone knew exactly what I knew and didn't know how to do um, and so I was I was kind of past my embarrassment And decided to just roll with it and do my very best to finish this out um, they were cobalt more final fixes A note to how many final fixes It's not every community but my particular set of reviewers were pretty self-aware and they said Hi, thank you for your patience. We are very picky. We know this about ourselves. Thank you so much for sticking with this This was not what I had expected I was I was expecting to just be very thankful for them for being patient with me So this is actually a story of at least the humans Well, except for me But but most of the most of the players in this story are actually well meaning and friendly and I think I think that's really important Because I think next time Well, actually, this is not a thing the next time I open a poll request to this repository. It was I don't know a couple weeks Much better So I did all the final fixes Or so I thought If the if the words are too small it says this is looking great I just requested a few minor changes in another test case. I think this will be ready to merge after that It is now june It turned out this time it was correct and my poll request was finally merged I was really proud. I shared with all my co-workers I may have adhered our slack channel saying i'm really proud of this by the way um space cats our tradition on the kubernetes repositories I was in fact sent one of these on the poll request So takeaways I kind of touched on these already, right? Um There are multiple ways to find Uh an issue or to find code to work on in an open source Uh repository Try the one that works for you and your community If your community is very friendly It might actually be easier to go to a group of people and say hey I would like to help with x or I have these skills. Where can I help? And they might set you up with something you then have the advantage that they already have context for who you are And uh, and they also have an interest in you finishing this work um Other times You might just pick up one of those help wanted issues that I showed in the beginning Uh This kind of depends on your comfort level and your skill level as well um I shouldn't have said skill level. I should have said skill area because again different people know different things um, also Don't be afraid to mess up in public Everyone does if you mess up in public once It means you're going to be better at messing up in public the second time Wait and Then when you mess up the third time you will recover faster and so on and so forth You're not probably going to stop messing up in public, but that's okay because everyone does and Again, there are some communities that might be more or less critical of this and that's hard but I still think That even if people are very critical you can still learn from them It's still an invaluable experience to have you code out and be reviewed by people who are not your co-workers or your friends or something like that um another thing that is That I completely failed on don't be like me is stay on top of your work Please communicate. Please don't let things wait for weeks and weeks and weeks because you're just too terrified of making more mistakes Don't do that because We're trying to we're trying to build code as a community when we're in open source and If you just stop then there is nothing that anyone can do to change that or to improve it or to make anything happen Be aware that people have weekends people have cultural holidays if they work in other parts of the world Um, so be patient with people, but if you don't stay on top of your work for you know, if you If you don't get back to people within reasonable time, it can be really discouraging for everyone. Please don't do that um And the same goes for if you are a maintainer and a reviewer Please stay on top of these things. We all know it's hard So give and take but I would definitely say that sitting on something without communication for a month Is not okay. Don't do that um Oh, no, I ended on a negative note um So before I make everyone super sad. I wanted to say thanks To the kubernetes sick testing Group that's a group of people who were that's the group of people who were the maintainers for the repository. I made the changes on Uh, especially col wagner christoph blacker and eric feta who are my main reviewers and we're so helpful and so patient with me Also, thanks go to scale 17x for having me at this talk And to all of you for being here in the last session just kind of Hanging out until the end of the conference closing it out. So I think We have time If you all want uh for a couple questions And this is how to get in touch with me. Oh, I see a hand up in the back a question Ladies and gentlemen, this is an awesome community in action Thank you for asking the question was how can we help make these things more visible? um Some of the some of the things I tried I mentioned the pre-submit script. I mentioned What else did I mention? That that was kind of the big one Yeah, that one is that what did I try? Um I realize I don't I don't actually I think I talked to somebody and asked them saying like hey, I'm super embarrassed by tests failed Um, can I how do I run tests locally? Um, and They said they actually led me to this other document. Yeah, so um, sometimes open source communities have documentation That's not super easy to detect um having that Right on top of the repo read me the main read me Or having a how to contribute file and just kind of putting in those steps is Super important a lot of a lot of repositories can have just like a standard contributing Markdown file. These are the steps. Here are your build steps. Here are your test steps. Um Run these things before opening a pull request Yeah Yes, so this is what I meant in the beginning by know your community um, because So I was in the lucky position in that I already knew a lot of people in the community Um, in fact, some of them are here today. Um, oh the question was when you said ask a person Do you mean ask on slack? Um Originally I had gotten the issue from someone in person at a conference So I introduced myself at a conference and said hi, I would like to help. I would like to learn. What can I do? This is where I'm at And then for later follow-up questions. We have the open pull request to communicate So github was one of our channels of communication My community also runs its own slack instance kubernetes i o Slack dot kubernetes i o anyway the kubernetes slack Um, and that's where I got in touch with them. Yes Um, so having these knowing where these communication channels are is really really important um Look for them As a hopeful contributor and as a maintainer make sure that they are easily easy to find Don't be a click. Yeah Okay Any other questions is anyone a little bit more excited about going out there and opening their first pull request? Yay. All right. I hope to see you some of you. Maybe Uh kubernetes is always happy to have new contributors if that's your jam if not Bring it on into other communities Change your own Yeah, thank you