 So on our next talk is by Rudy and Cenk about how CIP gate uses Debbie and another open source software test test Hello, and good morning everyone. Thanks for joining us here My name is Rudolf what I've been working for a company's named CIP gate a German company For seven years now, and I'm mostly involved with the with our infrastructure right now I'm mostly involved with the client infrastructure But I used to be involved with all kinds of servers network stuff and My colleague Cenk. Yeah Morning everybody. Thanks for having us. I'm Cenk and Working since February at CIP gates, and I'm in the team which managers. Yeah systems and network stuff Is anyone of you knows that gate already maybe Okay Just a quick overview We're a company based in Dusseldorf With a in a nice courtyard, so it's kind of hard to find but it looks not look nice We are a telecommunications company. I Just shamelessly stole our website here. I'm sorry. It's a German we do the thing we started with is Voice of IP telephony for private customers. We then added later CIP gate team, which is Which is our Business product, it's a virtual pbx hosted by us. We also added later Mobile we are a mobile virtual network operator so we are able to offer sim cards at least in Germany and We also offer trunking services for your own pbx and we offer various APIs to do real-time call manipulations and we offer both inbound and outbound APIs for all kinds of Fun stuff you can do with phone calls We also have Well, we call it Horizon 2 Products which are new products. One is the satellite app some might have heard of it. It's Basically an app that allows you to have a mobile phone number a German for mobile phone number in an app So it's voice of IP so you can actually Disconnect your Phone number from your sim card so you can actually change your provider user data only sim card or wireless network to use your mobile phone number in Germany or wherever else and This has been out for iPhone for about a year and we just released the Android app two weeks ago and Also Kling is also a new type of product. It's basically a new way of the business telephony. It's a Like the slack for telephony thing You can check out any of those products and ask us questions later on if you like We also have business in England or in the UK, which is basically the same except the mobile stuff Because we're not a network operator in the UK And that's about it with the marketing stuff for today. We have some technical facts for you We run about 700 servers 250 of those are bare metal servers, which is mostly telephony rated because Virtualization is still not good if you do any kind of real-time phone call stuff. At least if you want a decent quality We run databases on hardware, of course virtualization nodes and other infrastructure stuff and it's all debian based Except those few office services which still require Microsoft stuff. It's like it's only a handful Our client systems are all mostly Lenovo with debian or you want to that's up to the person running the notebook to decide or Apple has has been crawling up from behind and taking over many of our systems and We'll do have some Windows test environments, which we mostly run on AWS work spaces Which works quite nicely and it's less a hassle than providing VMs for by ourselves for colleagues and Speaking of colleagues, we're about 180 Employees and we're all based in Dusseldorf. So there's no no home office. No abroad office We also do the UK business from Germany and you may also know us from rather not so famous incidents As be not able to handle ticket systems very well We fixed that I think the seventh twice. It's just the second time But also we're we might know us from various other events like spondering depth of 15 or 17 and last year's mini-deb conf here and this year's depth conf or mini-deb conf and we also Are involved in some other open source product like Camal you which is open source SIP router and there's also other stuff, but that's not my personal domain. So I'm not very involved with that However, we do use lots of open source software, which is Try to name a few which you will find in our infrastructure and Today would like to to give you a short introduction to some of them and by some I mean kind of like those Hope you can read that. Yeah, and But you can also use today and tomorrow to ask us anything about the others if you know something from that slide So first of all, I'm going to do a short introduction to how we use different packaging Which we use a lot internally So this will be a less fancy version of what you just saw earlier Then how we do unattended upgrades How we will kind of had to build our own certificate authority internal certificate authority and why we did have did have did we have to do that and Well, how we do how we use guinity as our virtualization platform, which is present in devian as well I know and How we see the egg stack so we might run off out of time at some point and if that happens We will skip the exit and if there's any interest in that we will Provide this as an as a lighting talk tomorrow But we'll see how far we get So first of all how we deploy applications at Zipgate applications means Our own software stack which is quite a lot of software mostly based on Java and some Pearl which we not talk about and And that's the main part I think So we have an internal GitHub enterprise instance running at the on-premise This pushes a Jenkins instance since the setup is now almost four years old. This is all based on Jenkins one So it's not Jenkins two and we also use a software called Sorry Jenkins devian glue does anyone know that it's been written by Mika Prokop. So you might know him as a fellow devian enthusiasts and This kind of mostly handles all of our needs related to package devian packaging so it actually bundles I think cow builder and P builder and Deep package build package at deep PKG build package and lots of other things and you basically just throw one command in your Jenkins Shell blob and this does the rest so we use Change routes for building or cow builder based change routes So we can build for all kinds of different devian versions at the same time in pristine new environments They will be reset up on each run Well, we then sign and upload packages and This is the aptly logo. So we use aptly as our repository software Well, then we use Ansible to deploy our software So these are all Stacked Jenkins jobs. They just call each other and Push information in between both for example in the very first step during the clone and Build step there will be a version number will be generated and this will be passed along all the jobs and the Ansible playbook in the final Step will install this very package and not something else or the latest one or whatever So we This is like a typical Pipeline view which is this is the view from the it's the build pipeline Plug-in from Jenkins one if you use Jenkins to it has something like that built in So you can actually you can re-execute every step of those this is a pipeline that has been completely run and you can see the The clone step the build step the upload step followed by the deploy step, which is well Ansible playbook being run with the environment pointing to our dev servers and Well doing its thing whatever the playbook does what we're supposed to do It does call some tests and if that works It would be able to be deployed to to the live system, but that is always a manual step So we always need to click on the play button here because we don't do What's what it's called because we do continuous integration, but not in continuous delivery Someone has to check what happened and click on the button We never got around to change that in the past four years, but it works quite well for us Before we had that all of our developers had to package their software themselves and had to Deploy the software themselves that you knew which server to connect to and whatever else is associated with that and that actually reduced our Well time to be able to ship new software to a live system from like the the probation period of six months to probably like the second week of your experience at this subgate and And of course made things a lot more consistent than before Well the the only problem is a problem What do you think how many Jenkins jobs do we have? As I said, it's multiple. I'll see if you've been able to see we have multiple jobs per packets. So it's a lot so it's At some point we removed our staging system because after automating everything there was no difference between the death and the staging system so we just skipped one and that kind of got rid of a lot of tasks, but You don't want to have to manage this amount of Jenkins jobs On your own or by hand. So we actually are using Jenkins job builder, which is a project I think it's somewhere based in the open stack area and it's Basically a set of Yammel files which describe just how a job needs to look like it's just to define some macro And then you can just have a second Yammel file with a long list of things and we do lots of we solve lots of things by by naming conventions so the repository is connected to the package name and to the playbook name and so there's less configuring and more convention going on there and This has made things very very easy for us in the past years and Of course if you need to change something how your jobs behave you only need to change the job Template and all your jobs will be changed in Jenkins. So and actually the Jenkins job configuration job is also a Jenkins job So it's whenever you change something in the repository with the Jenkins configuration It will update itself or destroy itself. Whatever you did Well, we haven't we haven't had that in in the last four years so Okay, okay So this slide was supposed to tell you that we were using rep repro as a repository software earlier but We worse we tried to do well if you if you want to do continuous delivery or You also need to implement rollback. So if you deliver a package that is broken either because Your test didn't cover whatever is broken But if the broken package somehow ends up in the death or life system You want to have a rollback on to the last working version? So we needed to have multiple versions of packages in our repository. It's just something rep repo doesn't support so at that point we switched to aptly because aptly allows for multiple versions of the same package and This is this all works by Jenkins Remembering the version it built and the version that was installed before on the destination system So for example only installs the version that you just built and if that fails it rolls back to the last version To to have all your systems in the consistent state and not just for example one one back end of your five back ends It's just down because it failed and then it just stopped and nobody noticed Until monitoring tells someone else who is not involved with the deployment So That's why we changed to aptly and there were some numbers on here, and I think It said something like we have Roughly 400 or three something packages for Debbie and Jesse and 200 packages for a debut and a stretch and well unique packages and if you take into account the the older versions is like 1500 packages and 1200 for both of those distributions So as you can see we use different packaging quite heavily Which does not mean that everyone knows how to package actually nobody knows how to package at subgate and this is a constant stream of annoyance At least from a point of view from our developers They usually forget how that works and just copy things from existing jobs and well if you start with the with a Nicely and clean debut and folder for a new project at some point you will change things and do something probably nasty that works for this kind of Job and if you copy this one to the next one You will end up with a lot of confusion and at some point someone tells you that Your job builds way too long and then you figure out that he actually builds in the clean step And then in the builds it began and just copied things and didn't know what he did So this is what we use they've been packages, but they sometimes cause friction But for us at least it works well and it had in it and chose a consistent state on the target system So see if the next slide is also broken. There is apparently it is. I found that slide Okay, so yeah, this is these are roughly the numbers that I Told you and we also have a third-party software in our repository, which is for various reasons because for example, it's not available in Debian or it's well probably too old and backpots also too old or whatever and Sometimes we just require extra patches or build flags and we need to rebuild those packages and have them in our repository But we try to not do that Which because it also means we need to take off take care of security patches and keep track of what happens upstream and Usually we try to leave that job with Debian because they do a better job at that end than us and This is oh wow, this works great Okay so Actually, this slide was supposed to show you The activity and our deployment Get repository which contains all of our Ansible stuff. So it's roughly in four years They have been roughly 13,000 commits by 70 people so we have 285 playbooks using 275 roles and Most of them are actively used. We have 25 custom modules, which either means we have written them ourselves or we Were a bit ahead of time in our Ansible installation or actually on the end installation was too old So we just imported a module from your Version and right now our minimum Ansible version is 2.7 which is me which means we're kind of almost up to date and Those playbooks are used for deployment by Jenkins, but also for manual deployment to set up local Development environments on workstations or in vagrant boxes or whatever they used to do maintenance sometimes And onto also to set up servers in general We we also used puppet a while ago, but we stopped doing that in favor of Ansible because well Ansibles were already in place for deployments. So why not use it for setup as well? It works quite well for us And actually at some point we noticed that if you have all your configuration in Ansible Who needs server backups anymore unless you have data on your service and we just Well, I think it's probably we we reduced our backups Backup needs by half because we just doesn't make sense to back up a server which has nothing on it then Stuff that comes out of Ansible. So it's defined somewhere else. That's about it for the Packaging part and now Jenk will tell you a bit about how we do unattended upgrades and not spam your mailing lists anymore Yeah, we are doing unattended upgrades kind of on scale and we're doing it because Yeah, you won't obviously stay up-to-date and be safe against security threats and Yeah, to get the improvements performance improvements of Yeah, of new versions of software and Also, yeah It's better to change incrementally than the tools have smaller changes than have Update from 1.0 to 1.9 or even to a new major version Yeah, we also want to eliminate toll means that we do not want to do manual tasks Repetitive tasks manually and yeah, this mostly is boring and can lead to mistakes and Yeah, before this our automated automated upgrades we used a Mechanism that checked against the security mailing list or and an ISS the feed and checked the packages Against our environment and looked for if we are using this and if yes, you know notified us to manually upgrade this package and Yeah, now mostly we're doing it and upgrades on all of our systems and Yeah, we're doing this With cron jobs. We have configured a cron job on the systems which Has configured to run either on day or at night. We prefer doing this Yeah, not at night because if an incident incident pops up We want to respond as fast as possible and that's mostly possible on day and Yeah, our services are mostly deployed to a minimum of two servers to be redundant and We then just Do 110 upgrades always on one horse of a horse group so that not both Machines will reboot at the end or fail Yeah, and our mechanism is More is a rapper around and then upgrade itself the day being package Yeah, and we doing this with yeah, the cron job starts and checks if they're updates available for the system If yes, it starts with doing pre-upgrade tasks Yeah, and the pre-upgrade tasks is The first checks if there's a lock for this horse group if it's if it's possible to Update them and locks are set mostly when Another upgrade is in progress on the other pair of this service or we are currently deploying to this system So we are not rebooting we're doing anything upgrades on systems which are currently have deployment ongoing Yeah, and if the previous tasks exceeds well exits well, we set a lock for the system and Yeah, sets obviously a downtime on the monitoring for the services and the system itself and Starts doing service specific maintenance tasks. This are mostly tasks like they register from the load balancer and do not accept new requests on the system or Or like Do not accept new phone calls on the system and wait the current phone calls to finish and yeah continue with Reboot Yeah, after the packages get upgraded We get a notification on slack which which tells us which packages will be have been upgraded and After this notification the system starts to reboot and after reboot. Yeah, we're doing post reboot tasks like Yeah, like as well service specific maintenance tasks Like for example register back at the load balancer or start accepting new phone calls and as well as removing the lock or Or Yeah setting the downtime off and yeah, we're doing this since we started with an attend upgrades on every system on October and Till This year May we had around 4060 70 reboots and This means we had around we have around 20 reboots per day We have also the possibility to Yeah, to stop and then upgrades and we built a global kill switch mechanism to not allow reboots or not allow and then upgrades But I think we never used this, but maybe we will need it sometime Yeah, and why we are why we have this Pre-tasks and post tasks and why we don't just reboot or restart Yeah, it's because we when we started with this we here. Yeah, we had we don't have this fancy microservice architecture to just remove the service and Everything works fine We have many legacy software which doesn't allows us to just restart it and We as well not every time a reboot is needed and if an reboot is needed Perfectly Package with tell us if we need to reboot but we experienced that not every packages Not every packages telling us if the reboot is required so to be to be safe we just reboot and Also libraries which get updated to not notify the Services which are using these libraries to restart Yeah, and we are doing all of this without any impact to our users or nothing breaks and Yeah, it just works actually just one one edition When we had to to figure out how to reboot all of our services we just Well the team said well we need to figure out how to reboot which service and what to do before you can reboot a service and Or what pre tasks to execute actually for this very service because the operations team doesn't know each and any service, of course so they actually just wrote lots of sticky notes and put one service per sticky note and just Put them outside on their office or on the on the wall and just We have an internal communication system named Yammer, which is something like an internal Facebook thing I don't recommend it, but it works rather Whatever So that I said well whenever you walk past our office or get the coffee because the coffee machine is around the corner Just look at the the wall now the sticky notes and if you find the system, you know Just put it into into three lanes, which they also prepared and it says a Reboot just reboot or reboot with tasks. Don't reboot at all or Something else was there also I think and within a day they had like I think they had answers to 60 or 70 systems without doing any meetings or consulting lots of people or Fahabteilung as probably other other people would say we don't have that but That was fairly easy and that allowed them to just start with lots of systems because they would have never known how to find out who to ask or Is it safe to reboot or do will things break or with a break later or whatever. So that was actually a quite quite good experience So how to run your own? Certificate authority What do we need to fix certificates for well, for example internal or development websites and services Just plain HTTPS. We use bacula for backup which relies on working CA We use the TLS this log, I hope everybody else does as well Use databases my squirrel, which is rather pain with if you want to do anything with the certificates SSL but we'll still try to use that and Well, you name it whatever else uses certificates so Yeah, well, probably you have heard of let's encrypt at some point in the last years So why not use just that because it's that's exactly what we need The problem is it has those challenges or validation challenges like the HTTP where it connects to open port on your system or DNS where you have to put something in DNS to validate the host name and Unfortunately, all of those don't work for us because well we don't want to open random parts on all of our servers to to their validation system and DNS is one of the last Systems which we haven't automated that well read at all and So this was there wasn't a good option But still again, let's encrypt is open source there things called acne server Whatever it's called it. It is open source Well, why not just use that and if you browse there There the forums you will see lots of people asking Hey, can I use your system because it's open source and I need just that and Well, whatever They saw you on all those questions you find the same answer. Yes, it's open source, but it's not ready for home use There's no real documentation that they have a docker container, which is only for development and from that you can Kind of guesstimate how it works but at some point they just We try to evaluate it and from well you started on day one and on day three They just added an entire Rabbit MQ system somewhere in the middle of that and didn't they just said tell total nobody That is didn't tell or didn't say in there in the read me so we just didn't know why everything broke we tried and At the end actually the the acne server is just a wrapper around CF SSL which I explained on the next slide In a just just adds the whole part about registration email accounts validation and Notification, which is not what we need for internal CA because why So we looked at that CF SSL thing Which is a cloud for SSL and it's just a very small set of SSL related tools and also a simple CA which offers an HTTP API or REST API and But let's just see what would we require from a from a CA or Would we assumed about a CA for example? We want a root CA which lives somewhere offline and runs for a very long time probably a hundred years or 50 or whatever and It's usually Hidden on multiple this media's in various saves somewhere, but it's offline So we have intermediate CAs which are used for the actual signing which are valid for for example one year It's probably a good start And the certificates them themselves are only valid for for example three months or until the intermediate CA expires of course Whatever happens first and The CA only issues certificates for now instead of domains So we don't want to run a CA that just issues random certificates And if someone well knows how to use that he can just issue a certificates for whatever google.de or something That is probably not the best idea Well, and We just assumed or we just said well the crime it is certificates must be renewed automatically along with a service restart or Reload or whatever the service using the certificates It's required to do Spoiler alert my squirrel is not very good with that Well, and also we need or we want the root CA on all of our relevant systems deployed and and all systems need to serve the current intermediate CA to not have that also to be deployed to all systems In our case, this is how the whole thing works we have clients which are just service in our case we use Traffic and console for the low balancing part, but this is not it's not essential to this setup We just do that because we already had that in our infrastructure You can use whatever you like for low balancing So we have multiple back ends running the CS CF SSL API Which is just a simple process and it can either use an intermediate CA store on disk or you can also use like hardware Crypto models that depends on your needs or tasks And we have a shared postgres database, which just swings the state of the CF SSL CA So how does my service receive a certificate and auto renewal? So to do that we are actually As I already said we use Ansible for everything We just provided an Ansible task that it can be included by anyone requiring as if it is the certificate and This does all the magic so it will create It will just install the required Set of scripts that runs on your server and does regular checks if things need to be renewed You also provide the command to actually reload this Service so sometimes it's probably like a service Apache to restart and it can sometimes it's also something more sophisticated Just depends on your service The only thing we had to learn that the more services you have using SSL the more Crappy it gets with the format they expect those certificates sometimes used to This is actually is anyone has anyone been using external at some point that for some reason they require the DH params to be included in the file I don't know but Well, some need the keys and certificates separated. I don't need them in a certain Order and this list just got longer and longer the further we dig into our infrastructure How does the renewal work all the crunch up runs each day Between 11 and 12 am again, we try to do most things to do during work hours to have more people available if something breaks And the this front up is spread between 11 and 12 across all systems A certificate will be replaced ten days ahead of expiry remember if three three month of certificate Validness and Ten days ahead of expiry was just a guesstimate us to think okay Well, if the CA is broken for some reason we have ten days to fix that until whatever sipgate mails down So well after retrieving the certificate the service reloads command will be triggered and that's about it So simple as that security aspects first of all as I already said we only want you to issue certificate certificates certificates for widely said domains not any domain because since the the CA is present of our systems if someone gets a hold of our CA and is able to just request random Certificates all of our clients would trust that which is probably not what you want So well short lift certificates are better than long lift ones from security view if someone is able to to To get a certificate ability is not last that long And we also already think about lowering because it works so well We thought about maybe probably a month is also okay instead of three We do authentication only on a network level because it's simple and good enough for our use case We thought about doing advanced Authentication whatever stuff, but if you for example, let's say you use basic authentication In front of the API you end up storing those credentials on each server and what's the difference if someone takes a hold of your server Which he needs to talk the CA he also probably find the credentials and then that you didn't you don't gain anything from that And if you do something more sophisticated, it will get more complicated and then we'll probably break at some point So we decided to do the first take with just that that's what I just said and This is a look at the database. So with Within the I think that the time frame is actually a year now. That's the point our old CA expired and We have issued 15,000 certificates since then but also one of my colleagues told me that probably two thousand of those are just test runs by him, but It just works and it never broke so far the only thing that Still resisting this my scale. We need to Restart our databases manually because you probably don't want to restart a production database at 11 during the day during daytime Not at night as well at least not without supervision and we still need to work on that But as I far as I know, I think Maria Dubey has implemented hot reloading of certificates So that's probably a way to work around that in the future and Next topic back to Cenk Yeah, how are you playing the visualization game? Yeah, we're using Garnetti. Garnetti is Yeah, our visualization platform which was developed by Google and Google used a long time and I think it's Partly uses it still Yeah, as I said, it's our cluster management for VMs based on Xen KVM or LXE Yeah, and Garnetti is a shared Yeah, shared nothing class. I was job management with job management, which means that we Have one master node which receives all the jobs To be executed like creating deleting a VM or starting or stopping And the master nodes can be swapped between a pre-configured number of Of other nodes so we in case of the master failure We can switch to another node being the master and we have no single point of failure here Yeah Garnetti is mostly key focused and provides some nice command line command line commands and Nice rest RP and we built with this nice rest RP a web front end for us Which you can use its own source Yeah, we have your three Garnetti clusters located on three different locations which consists about 31 nodes on which we have around 400 instances with Yeah, many CPU cores and around three terabytes of memory and Yeah, our When you use the dbd backend for your server server for your instance is what we use which we use Yeah, you can Those of the VMs are the redundant because in case of a node failure the VM will start on the on another hardware nodes With minimal disruption or no disruption if you migrate Then instead of stopping and starting them on the new one Yeah, and Garnetti offers the possibility to set node groups this means that Yeah, you can have some sub clusters in your cluster and we're using this method to Bundle different hardware generations together so that the service runs for example only on the new hardware or Only on the new hardware Yeah, we are using for replication and migration and another network interface or network instead of The for the other ones Yeah, and Garnetti provides a nice tool called H ball This is a cluster balancer and it looks into the to the current state of the cluster with a total free memory disk memory and memory and then places instances In a in a better Distributes the instances a better way to that we have that we all get a note Yeah, better. Yeah, the current state of Garnetti Is mostly stopped school stops developing active developing on the project and Some month ago the last version was released By not by Google, but Yeah Yeah, this is a big issue for for the project as well as we have Problems for example with live migrations with job applications, which Just end up in a broken state if we migrate them live to a new node and in two weeks a Short break in two weeks is the Garnetti con in Umea in Sweden and Maybe they will Yeah We will have a current states of the future Resulted of this can it you can I hope some of you might know Apollo probably who's the current main deviant maintainer of Garnetti and he's actually the the guy who Who we quoted and he was who released the 2.16 version recently and he already has Access to the repository, but it's still we're still waiting for an answer from from Google on How they will proceed and if they will really release the repository to well to the public or at least to the community community. This is Unfortunately has not been settled so far well in December. They said any moment now, so you see how that turns out Yeah, the next yeah as Rudy is said at in the beginning that yeah You maybe have not enough time But we have a little presentation about the exact if anyone wants to see that Come to us. We can present it tomorrow. It's lightning talks as a lightning talk Yeah No spoiler Yeah Can we take one or two questions? Is that fine with the video team? Okay, so if there's some questions, please line up at the microphone I just want to command that postgres does hot reloading of certificates these days Not sure about me riding me. We're not good with postgres unfortunately So you said you have 20 reboots a day is that per server for all the servers and why do you reboot so often? I mean I reboot when I install a new kernel Yeah, yeah, as I said their packages which Requires a reboot like the yeah the kernel when the kernel packages installed updated, but We we we experienced that like when we update the inter microcode package that it does not touch the VAR reward required And we want to be safe with this and just read with every time when a package updates We thought it would be easier to just reboot any time instead of trying to figure out which libraries need service restarts And just do it anytime, but you don't reboot one machine 20 times a day. No, no, no, no, no, it's 20 machines per day Okay, so and how often is each machine rebooted then once a week or yeah mostly all yeah Every time what yeah mostly once a week or sometimes depends on how well the different security team works But usually there's something to update every week and the systems to check on their regular schedule And if there's something they install it and reboot Just a short question You said you won't reboot a machine which has an ongoing call. What do you do if somebody is talking on the phone for like a week? But we do have a maximum It's very it's a very very long period But actually in this case we I think it just fails and notices us that we couldn't reboot the system It just re adds into the the phone Well routing again because we have issues with our phone provider We are the cause automatically stopped like after two hours or something three hours Just yeah, it's just interesting if you have telephone conference and after three hours it goes like dude dude dude and You're disconnected and I was wondering if this is something like they've or we have a security update and we have three boot now I actually thought the phone systems are one of the few systems that don't reboot during daytime because it takes forever to Drain them during daytime, but it works better at night and early night and Actually last year I had to do some manual work on the system and I drained it during the day and after three hours there was only for only one phone phone call left on this system and The destination number was the dutch to come hotline and I I felt sorry for this person and we just skipped the maintenance I thought I would probably not be so nice Thank you Well, if you any if you have any more questions just Fee free to ask us during your day a question. So for your own services Is that Ansible telling the service to? Update or how does it figure out? It needs if you have several microservices doing it This is just because of the lock then only one gets updated or how is it working? Do you have some? How do you call it cannery stuff to update this? How is how is this workflow between Jenkins and unattended upgrade? Is there any workflow between it or is this totally separate? It's separate. That's we use Jenkins to Yeah, deploy packages and for infrastructure stuff. We just running Ansible on our machines. Okay, so that is just instantaneous out of Jenkins update but basically we after implementing the unattended upgrade process we We took a deep dive into the Jenkins job builder and just edited all those jobs at once to check the lock so if the if Jenkins is starting to do deployment and it will figure out before the beforehand that there's an ongoing upgrade and will not deploy and The other way around the same This is what Jenkins described if the unattended upgrade starts and sees there's a lock for the system from Jenkins side It will not do the unattended upgrade to not break the deployment But it will always notice the slack channel that something didn't work for that reason or the other reason Well, there's one command from ISE about the need restart Package or file about the reboot thing. I'm just mentioning it here Okay, so I think we're running a bit out of time. Let's thank the speakers again