 Okay, we're going to talk about carton. Most of what I'm talking about is me using carton. However, I'm also abusing Docker in a really, I think, kind of interesting way and a way that I'm a little ashamed of. And I'm also abusing CircleCI, and so both of those are in the topic. So, good morning, Pearl. Today I want to talk about how you manage dependencies. I have been managing dependencies for various years in various situations. Mostly what I talk about these days is Bugzilla, but I want to mention that I've actually worked on other things before. I've worked on a thing called LAN POS, which, for the French speakers in the room, I think is an amusing name. This was created by an American, and since we're in Belgium, the translation would be this. But for the English speakers, and since we're in English, of course it's called the ASS POS, which is really funny because you might not know, but POS is frequently an acronym for that, so, well, that, a piece of poo. So packaging, I've done, I did packaging when I worked on this, on this point of sale machine. I did Debian packaging, and, oh, there's one more thing. There's some of the worst Pearl code that I have ever seen. This is how it rounded numbers. So I did packaging on this thing. I used DH Make Pearl, and I made, I think that there was something, a couple hundred different packages, things that weren't in Debian at the time and I needed them, and I did a lot of work on this, but this was all for the developer machines because everyone else was using, well, the production machines, they looked like that, and they ran Red Hat 4, not Red Hat Enterprise Linux 4, like older than that, like Red Hat 4. This was in the early 2000s, but the cool thing is there's somebody else worried about RPMs, so I didn't have to worry about that. I knew about C-Pen to RPM, but at the time I was doing this, I was very much in the camp of using the system Pearl and using packaging that the system provides for all the Pearl modules, even if you have hundreds of them. So then I joined Infinity, and I worked with some really cool superheroes, and at Infinity Day, we didn't use the packaging management for most of the projects that I worked on anyway. We just used C-Pen or C-Pen M, and it was really great. When I was installing stuff for this work code, most of the modules came from this man who cannot be here, but like my first day when I'm installing 70 things that are all from one person, it was a bit interesting when I made that correlation. But then I started working for Mozilla, and that was really exciting. I really love working for Mozilla. Mozilla is wonderful. I got to work on Bugzilla, which is the bug tracker, and it's incredibly important, and it's difficult to characterize Mozilla on Bugzilla. It's incredibly difficult. It's a whole complicated situation, but it's been fun. So Bugzilla, I'm going to provide a little bit of background about why using Carton for this was a pill battle. And so I want to talk about my first impressions for using it. My first actual impression for doing Bugzilla four years ago was that I had to learn BZR. It was some kind of source control thing. I had to use it for about a year, but we migrated off of that and we're on Git now. The second thing as a pro programmer that had been working for people that do modern pro stuff, or like Steven, I was this check setup thing that's in the Bugzilla directory, and I was like, what is that? Is it like makefile.pl? No, it's not like makefile.pl. How do we specify dependencies? Bugzilla had its own way of dispecifying dependencies, which was a constant defined in some module, but it was kind of interesting because it specified the package name, the module name, and also the distribution name. As far as I know, usually there's not a one-to-one mapping like in the pause indexer, but most people don't care about the distribution name. They depend on a particular package, and if a package moves between distributions, that's fine. This actually has a problem in Bugzilla when LWP split out some of their modules. So the next headache was install module.pl, which was this thing that the documentation recommended to not use, but they still shipped with it. What it was was a custom cpan client that's actually being generous. It kind of had copy and pasted parts of cpan.pm, and it installed stuff into the directory on the root called lib. So I wanted to point out that the code for Bugzilla, actually still now until Tom submits a patch to me to fix this, is in the root, and everything that we, used to be every third-party dependency in lib. Now that is actually kind of backwards, and that is the code that accomplished that. We had a really fun experience with Moo, because back in an older version of Moo, Moo thought that all of our code that was like a library and all of the code in lib was first-party, and so warnings and things were turned on the wrong way. That got fixed though. So since then, I've been working on creating new problems, and one of my, I think that new problems are good. They're kind of like an investment. You get rid of the old problems and you have new ones, and so the first thing I want to talk about that was a problem that I've solved and created was inconsistent environments, and then we had inconsistent environments for my Bugzilla. The Bugzilla I'm talking about is the install that's bugzilla.mozilla.org to be clear, but it's most of what I talk about is not important to that, but production runs on CentOS 6. Actually, I think for it runs on Red Hat Enterprise Linux 6, but close to that, it uses RPMs and somebody else built the RPMs. As in my career, that's been the case. So I want to actually thank Fubar, who had been working on Bugzilla RPMs before I came there and basically knew a lot about what the dependencies were. I think in some cases, a bit more than the developers did themselves. So then the developers used Fedora, so if you have a VM on your desktop using VirtualBox or a VM or something like that, it would be running Fedora, and this was the other two people that were working on Bugzilla. I think actually sometimes they're using CentOS 7, but Fedora CentOS 7, not the same as production. But it gets better because the way they install modules, sometimes we install RPMs and sometimes we use cPanM. Of course, they use cPanM. I think originally they would have to use cPanM by specifying every module, which you could get by repeatedly running and collecting the modules that it requires. At some point, there was a shim to module build, and at that point they could use cPanM, but it was still kind of really bad. By the way, continuous integration, we've been running continuous integration tests for a long time, and those ran in CentOS 7, which is good to be one version ahead of your production environment. But then a thing happened. There's this repository called version control tools, which is a bunch of tools related to Firefox development and so on, and a co-worker of mine created a test thing. Oh, is there a typo there? No, it's not a typo. Created a testing suite to test an application that we were using called review board against our bugzilla because we had this integration stuff in it. We had a Docker container at this time, but he didn't use that. He went and created his own based on Ubuntu 14.04. At some point, we also had Amazon Linux, although this has gone away. There were about five different systems, and each of them were installing C-Pen packages in a different way, including Debian ones, and that was kind of fun. So, just to relax for a moment, I didn't actually set out to solve the problem of having a bunch of different environments. I didn't want to solve it. I had a different problem. My problem is that laptops don't like me, and I have hardware failure all the time. Not literally all the time, but enough that it's been annoying. So, oh snap, my laptop's broke. That's annoying. How long is it going to take me to reinstall bugzilla? Well, a day or two? About two years ago, this would be a true statement. So, at some point, I started just using, because we had the module build shim on our own dependency specification, I started using that. And then to get it to work, so I didn't have to modify every file, because at one point I was modifying every file to use local lib, and I had to point into the code, which is like every script and every, not every CGI, because we're actually just a modparl application that's pretending to be CGI. But, yeah, so, I tried patching all the files with local lib. That didn't work. I asked, at some point, I do remember asking, can we use local lib? And this is at the time when there were, I was just kind of junior on the team. And I asked, and the answer was pretty much no. So, what I did ask is how about I add the local lib pro 5 directory to the top of every non-module file. And that patch got applied, so I did that. So, we weren't using local lib, but we were loading all of the modules from the local directory, which is the same directory that Carton uses, by the way. I had wanted to use Carton, but I didn't bring it up until much later. So, we're not using local lib, but we have local lib pro 5 in our path. Next thing, I want to bundle. I want to use, I want every module that I'm using nicely packaged, and I want to use Carton. I've been looking forward to doing this. I had used Carton before, previous to bugzilla, and I was excited to get back to having, like, a complete specification of all my dependencies, which Carton, if you do not what Carton provides, we will have a bit of time for Q and A on that. So, the Carton documentation disagrees. The Carton documentation says that, A, you can't use the system Perl if it's on certain systems. This basically means red hat because they split off core modules. It also says that you can't run it without using Carton exec, and this is also not true, and I think this is known and agreed to not be true. However, A is basically true. Like, you really shouldn't use the system Perl to, if you try to use the system Perl with Carton and install stuff, it's going to get confused, and it's going to think modules that are core modules are not core modules, and it's just not going to work. But you can fix this, so what you can do is you can build your own Perl that's the same version as the system one. So this is a script called using Perl build, the script is called build vanilla Perl, which I wrote and it's a little bit hard to see, but this is taking the Perl version number and running Perl build on whatever that version is, with some flags that I think are vaguely compatible with what the distro I was testing this on worked with, multiple distros actually. So then you have your own Perl, you install Carton into that Perl, and then you have Carton install of its dependencies, and you have Carton fat pack itself, that means that the Carton executable gets put into a directory inside the local directory that it manages, and then you just get rid of all of that. We throw away all of our hopes and dreams, and we start using the system Perl again. So we use the system Perl to reinstall Carton. This time, hold on, missing a slide, and this time in the bottom part of this slide, which encapsulates the previous steps, we rerun the Carton install using the fat pack version. Cache deployment basically prevents it from looking at the internet and only looking at what the head has already downloaded, so it has that complete specification that it got from the real Perl in the system, not the real Perl, but the Perl that's not messed up. So this is pretty good. This is a few bit of annoying steps though, and I think that it would be frustrating to do this for five different systems. So I wondered about abstracting it. I kind of think that that we could use something like a function that kind of is a function over the operating system over the Linux distribution and the Perl version and the list of dependencies and it could spit out what it is. So it would be nice to write such a function. However, it's not really possible, but we have Docker, so we could do something similar to writing a function that operates on a system. I don't know how to do slides apparently. There we go. Bad form. There we go. I apparently made sub-slides and sub-slides are a thing I don't understand how it works. So I wanted to modify the Dockerfile language. I thought this would have been fun to create like a Dockerfile function that is parameterized over the Git repo, and then when you have that Dockerfile function, you could have your set up the base system to match whatever production looks like and then run it. But I didn't actually end up doing that. I thought about it, but I did something kind of worse. I kind of wrapped Docker in Perl. Well, I made a DSL that generates a Dockerfile and that lets me use Perl sub-routines to consist of repeatable environments. This is hard to read, but basically the first few lines are all setting up a CentOS 6 environment that matches our production, and then it calls the sub-routine at the bottom to build all of the dependencies. And the Dockerfile you end up with, you can't definitely not read, but I want to show for length is that. The same thing is possible for Ubuntu and then the same thing is possible for CentOS 7. And then, so we have these Perl files that generate Dockerfiles and then of course we want to use Make to do the build process. So this builds Dockerfiles and then it builds a Docker container and then it runs the Docker container and in running the Docker container it produces all of, a tar ball of all of the dependencies, as previously specified. So then I would just have a target make upload that uploads it to an S3 bucket, each of the tar balls for each of the different systems. And I just need an S3 bucket and I asked our Ops guy I said, hey, Fubar, would you never, would you like to never have to make an RPM Perl module again? And what did you know it? Within about 10 minutes he had me an S3 bucket and so we do that. We updated the production push scripts to pull down the tar ball for every deployment and that was cool. Now meanwhile, version control tools, which was using their own thing in Ubuntu, we got them patched to do that. There's the thing there. They made it significantly shorter in lines, but it also made it faster. So the builds for testing the review board stuff the slowest part was bugzilla and the slowest part of that was building all of the dependencies using cpanm. It took, I think, 20 minutes, 30 minutes. Basically I shaved 30, 37 minutes of their build times. And also their build stopped breaking when I added new dependencies, which I had never done, but in theory if I added new dependencies their stuff would break. It also made our tests run faster because I upgraded our testing suite from this sounds weird, but from CentOS 7 to CentOS 6 so that it was actually the same as production because I couldn't update production, but I could make continuous integration, match production. And I think it did Amazon Linux Docker images later after someone else created their, there used to be Docker images that matched Amazon Linux, but now there are. And so we shoved 270 modules into one carton and it fit a bit like a horse in a cup. That's not a metaphor. And that's the first, that was the first battle. The second result was once I could actually use cpan modules we could have smaller battles and have bigger wins. So for instance the time in Moscow was wrong. Our bug tracker is global and we have users from all over the world including in Russia and the time zone change at some point and the fixture that was updating date and time zone. So that was good. I also got to use Moo. I used Moo for generating content security policies and as it is now my co-workers are starting to write new code that's using Moo and bugzilla and they're actually enjoying it which is really good. We were able to pin DVD MySQL to prevent things from raking which happened recently. We upgraded JSON and JSON-XS. I realize that we should probably have maybe be using cpanel JSON-XS which is on my roadmap. All of that's pretty boring but basically I could update stuff and updating stuff was something that I had wanted to do for about this was two years ago so I had been wanting to update stuff for about two years. But more exciting than updating stuff is shipping features. So I had a feature request which was make all the users of bugzilla hate me and by making their passwords harder and so this was a very important high level request from important people that wanted to make sure we were secure. So I asked EIS which is enterprise information security what do you recommend how do you want me to make passwords secure and they recommended a C library not recommended but they said that they were using a C library and I'm like oh cool is there a pearl binding for that and as it turns out there is there's a pearl binding for that it needed an additional feature which I which I patched and I want to thank Sherwin the author of that module because I was able to ship a feature that was important to a lot of people in about a month and that's that's pretty good but you know it's even faster than shipping features faster and having being able to use new modules what's faster than that is having a development environment that you can throw away and recreate as many times as you want. So I at some point created a vagrant install it was pretty easy to get it going because all the dependencies were already packaged for CentOS 6 for production so I just made a vagrant VM that matched production exactly and that was pretty good the stats for like the previous year there were about three three-ish people contributing on average to on a given month to bugzola.mozola.org in the last month we've had 11 and I think that that's actually really good now I'm like a majority of those commits but the second majority of it is like half again as much as well it's a nice it's actually a nice curve what you would expect pretty much so we're actually pretty darn agile now we accept poor request people can set up dev environments we've contributed we've participated in both Outreach and Google Summer of Code which meant hundreds of new people not hundreds but dozens of new people have tried installing and getting a dev environment working so we're really happy. Now the next problem was we have a cloud migration pending so our current production is CentOS 6 on like machines in the data center and that's got to change pretty soon and so there are people working on that and they had some requirements 12 factor yada yada Batman but they needed our app to be a Docker container and they strongly strongly hinted that it should be CircleCI for running our tests and for building a Docker image so I stared at this task and I thought well this is a good chance to clean up some stuff so Docker they were recommending CircleCI the first version and they had experience with that but right as I was doing the CircleCI 2 had come out and had a lot of shiny features including the ability of running Docker containers it's not literally running MSI Docker containers but you can create and build Docker containers from what amounts to a arbitrary Docker container and so the YAML that you have to write because this is all modern and basically Ingy's YAML language is one of the more popular cloud languages at this point I think so here is a bit how we do a build it's notice it sets up a remote Docker environment and it builds Docker everything is fine you probably want to see what that Docker file looks like so there's the entirety of the Docker file nothing to see there very boring I'm kind of hiding the fact that there is a cutoff by the projector there is the thing that says from Mozilla B team BMO Slim so BMO Slim is our base image and I kind of I'm obviously alighting a lot of stuff in there so I should tell you what that looks like that is this hard to read YAML I wish I could have changed the font colors ahead of time but this is the steps that are required to build the dependencies so this is what was previously in the make file the make file by the way is gone we don't use the make file to do any crazy things I'm not generating Docker files from Perl anymore either but these steps run so let's take a closer look so that you can actually see the Docker environment gets to be based on an image which is our base image of CentOS 6 and it runs a set of steps which you can see in closer detail here that does a check out of the code it installs all of the RPMs that the system level RPMs like libc stuff and then it does build prepare build stage one build stage two and then it makes a tar ball so build prepare build prepare is a simple script that's basically doing the downloading cpan sorry downloading Perl build and carton and then it's also compiling those things so then build stage one uses the real Perl the nice one that we just installed to do the previously mentioned step I think the code is a bit cleaner than the old make file stuff build stage two is when we remove everything and do that all again and then this kind of long spiel here is how we package all of that up I'm actually packaging up a bit of metadata about what system libraries are linked to any shared objects that are in the Perl directory and I check for that later to prevent the case of missing of a mismatch in the current case we could have a mismatch between production and the this build environment in the future that will be impossible but at the moment it's still possible so I make that check so this is how we build BMO slim which is once again you can't see very well but it's our own they will be on the slides you it's based on the docker image I'm running a docker image for running docker commands and then inside there it's copying it's tarring up I'm sorry it's untarring the tar ball that it had previously built in the in the other step and it's putting that in a directory so docker workflows have these directories that you can put stuff in and share between build processes so technically I'm still uploading a tar ball to s3 but that is for the existing production environment and if if all goes well I won't have to do that anymore and the entire build pipeline will be building all the dependencies and just putting them in the container and uploading the container to docker hub and everything will be fine so build times there's a bit of a split here that I should explain so there is the stuff that where I'm building and doing the interesting things with carton is in a directory in a repo called BMO systems and this is mostly because it takes a lot of time technically I only need one branch of this for product for the use down here I only need the CentOS 6 to the BMO slim pipeline but that one alone takes 20 minutes so the dependencies don't change that often so the dependencies live outside of the master copy of the dependencies is the repo but there's a cached copy that's in another repo that is generated for each system because the actual requirements can vary between Ubuntu and CentOS and so on like you don't have this exact same set of dependencies also because you don't have the exact same version of Perl I think there are three versions of Perl at play in this arrangement so they're separate but when you as a result of this the build times for everything else are quite small I think the longest test is our Selenium test which is running a full Firefox browser and that's that takes a bit more time because it's actually hitting a few hundreds of pages on the thing to make sure it's not broken our build time though because everything technically already built is just a minute so I kind of like that I would like to show you one of the bits about this circle CI thing that is nice so to spin up our testing environment I can specify any number or some arbitrarily large number of Docker containers that will be running so when this is running we have our code running in one box one container we have MySQL we have memcache we have the Selenium thing having a container run Selenium is really nice because I don't have to worry about the details of updating that we used to have all this run in one container back before we were using circle and that was not good so that that said it's obvious that there's a bit of a man there must be a manual step between these two repositories because if I just add something to the makefile.pl of bugzilla then how does it get into BMO systems and into the multiple files so I want to show you what that looks like it's still a painful process but it's not that bad so for instance we edit the makefile.pl adding in the new dependency and we have to use make so I run make and with which brand and the sets to make variables the branch and the repository that I want it to look at because obviously it can only have the dependencies for a particular repo at a given time which does make deploying this a little bit difficult you have to have some planning but still you run the make it runs and compiles things creates two tar balls and then I do a commit and I have new a new cpan file for each distribution and a new cpan file.snapshot for each distribution which those don't change so the process here is actually feeding back in on itself because the cpan file there's an artifact of carton cpan file.snapshot that gets fed back in which prevents me from upgrading packages unless I explicitly do that which is not shown here there's a separate process for actually upgrading something that's not that bad and it is documented but it's still this is not ideal but having five at this point actually only two different systems is also not ideal so that is the monster but I wanted to also mention that in that last patch I have I'm adding modulations as a dependency for my bugzilla for a couple of reasons and if this interests anyone and they would like to do things I'm all ears and I should have a sufficient amount of time for questions and I suspect that there should be questions and I have all of the code opened in my IDE for those so yes I'm going to use modulations to implement OAuth2 but I'm also going to have as an endpoint that I can move my great features to for instance the next thing I could do was we need to use JWT I would like to use mojo JWT for email confirmations instead of the current token system that we're using so I can I ask myself questions if no one has any questions but yes the orchestration is outside of my so I just make an application that's roughly 12 factory and the orchestrator that the the other part of the team uses is up to them I don't think they're using an orchestrator right now actually but theoretically it could I've actually got this running myself on Kubernetes so it's not hard it's just not it's not in my responsibilities at the moment and if you just speak just yell if you're if I don't appear to be noticing that you're asking a question yes I haven't played with it I've looked into a proof of concept just using mojolicious itself and then I've done another separate proof of concept of tying it to Bugzol's OAuth system so I know that these two things work and I haven't played with them together yet but Lee Johnson's the fact that he has packaged that up for mojolicious and it seems like less work than using the underlying is the reason that I want to depend on mojolicious because I want to I don't want to have to do a lot of work yep are there any questions about I think that it would be instructive and I'll just show you I'm going to show you the entire config file for built for the entire workflow that generates the the carton files and how are we on time what 10 perfect this is exact amount of time I was hoping for a little bit more questions after all of this madness this is fine so the entire we have just under about about 200 lines of YAML the world's best scripting language and and I think that should be visible on the screen how's that and a much larger font look at that that's amazing there we go I wish I had a Linux machine in front of me I would know how to do that then but then I won't be able to see it and now I need to decrease the font size okay so whoops what was not clear when I lied it over this which I did because I was worried about time is that each of the in the current case you have CentOS and Ubuntu CentOS and Ubuntu tasks each run and they deposit something in a workspace the workspace is shared with the upload task and then the upload task just uploads the tar balls to an S3 bucket specified in the bottom of this is the basically the relationship between each of the tasks and as you can see here the tasks are also filtered on only certain branches so we only look at the master branch and the jobs that get executed are as follows or wait that's for that's for builder so we have also builders which is something you don't need to worry about no one should use that that's actually so builders is a hack to make it so that I don't have to always reinstall everything when generating a new cpan file and a new cpan file.snapshot but in an ideal situation you will only have one environment and you won't have different environments to do by the way if you are doing this as of last week CentOS 7 when you are installing it on when you are using it on CircleCI will break because of some bug in the way they're running Docker so if you need to do a yum update in a CentOS 7 environment you should do that and build that name it and upload it to Docker Hub and use that because running the yum update inside CircleCI will just crash but there is everything these I actually kept the existing when I did this I kept the existing CI Docker container around which was called BMOCI I basically had no interruption of service while doing this we just turned off eventually turned off the old CI and turned on the new CI but all of that can go away now as well so there is really just three containers in play three build processes in play in here if there are no other questions and I think I am going to be done literally no other questions no other questions about the Docker file language either