 Hey everybody and welcome to the talk infrastructure at arch making so let's go blue and where we talk about the infrastructure at arch and how to make so let's go blue and so anyways our agenda for the day is I'm gonna tell you a little bit about me then I'm gonna introduce the team to you I'm gonna give you an overview about you know our general infrastructure what tools we use and so on and then I'm gonna delve into a past kind of like some history lesson about our our infra where we come from and then we're gonna see where we currently stand and then I'm gonna see well I'm gonna basically give you some kind of roadmap that we have current projects and and you know near future and far future projects and now we're gonna try to tackle those and then I'm gonna show you how to you get you how you can get involved and in infrared arch so a little bit about me I work you know by day time well mostly anyway I work as a freelance DevOps consultant and also software architect depends on the job and I'm involved in quite a few open source projects I also do quite a bit of game dev I do quite a bit of rust and I also like seal it was very much so if you check my GitHub you're gonna see many of those and and I joined our edge in 2020 at 2010 actually as a trusted user and I think I've been using arch since I think 2007 not quite sure but anyway so I eventually became a developer and now obviously DevOps and not quite sure when I became a developer at some point I took over KDE and this is I think when I became officially developer all right enough about me so the DevOps team who are these people so you might see you might have seen some of those faces we're not only DevOps we all have packages so there's who it is Francois who's a very recent joining actually to the DevOps team he's not entirely on board as of this moment currently and we're still working on that there's Grasolini there's a heftig who maintains the market server there's yellow there's untracks and there's me and we all do different like we all have different levels of involvement in this project some have their their their pet parts we all try to kind of be be able to do everybody else's job if need be but mostly I think it's it's fair to say that everybody has their own topics that they care about like like anything else an arches it's pretty much the same and the only thing that's a little bit different I think is that we have this DevOps team which has regular meetings and I think this is not the case and in any of the other arch teams at least I'm aware I think the security people might have something not quite sure and all right let's delve into the overview so let's start with servers everybody likes servers so arches mostly hosted a heads now we have six heads not bare metal servers and 17 heads not cloud servers and total so that's 23 machines and then we also have two off-site backup servers for Borg so we use Borg backup for all our backup needs and on top of that we have five sponsored servers by Cape thanks guys shout out and I'm afraid we have to admit we have yet to make use of them because we have very recently actually got them so still still the process of setting that up as far as services go we have quite a quite a few of services right so we run like if you look at our you might see you know that the front page and the a you are everybody knows of that but there I mean also the wiki of course but you know there's quite a few services public services that we provide and and many of those services that we provide actually you know they might be presented to you guys as like a coherent like one software solution or something like like Archweb but actually there's many moving parts involved in any of these so for instance we we run the archive so you need this in order to use the Arch Linux you know way back machine essentially and it's also it's also does it also plays a role for reproducible builds and we have Seagate which you might know as projects.archinux.org currently trying to get rid of that very recently we've gotten a good lap up and running or well we have we have had it for some time but we're trying to make better use of that as as time goes on and yeah we have the wiki which is a media wiki of course we have a security tracker you guys can give the wolf and security stuff can post some stuff there we do have a patch tracker it's a good old patch work trying to get rid of that as well actually the the packages that that you might see as you know the like the packages an area or like the mirroring infrastructure that's actually just an our sync service which runs somewhere that everybody pushes the packages to and then every like all the tier one mirror sync from that so it's like the the packages thing that we have is essentially like the tier zero mirror and what else yeah we have the brick you might have seen him like it's a it's a little bot that we run to get IOC going to have like IOC moderation to ease IOC moderation actually we still use fly spray as our primary bug tracker I don't very much like that I don't think anybody really likes that but it's what we currently stuck with we are planning immigration at least a partial migration of some of our projects to GitLab but we have yet to work out how our package bugs which is the vast majority of bugs are gonna be migrated so we're still working on that and of course we also have the account service as is all but I'm gonna talk a bit more about that one later in particular and so we have also some staff only services actually we have a build server that we offer our staff so you know it is a pretty beefy machine it's it's kind of last gen but it's still pretty beefy it has 24 cores for 48 logical cores it's also sponsored by Hetzner so thanks guys and so our our staff can use or our packages at least can use that server to build their stuff we also have a Kenboard which we're also trying to actually quite quite soon we're trying to get rid of that one it's essentially like a Kenman board it's a free open source Kenman board trying to get that one into GitLab because GitLab has nice nice features for that and so we're trying to get rid of some of our services which are now super suited by GitLab we obviously have some mail service I say service because we have multiple mail servers we're trying to kind of perhaps condense down almost like a little bit our mail stack has has grown a little bit over the years and trying to get get that in order soon ish we have rest very recently we've gotten a monitoring stack that yellow has been working on so we used to use Zebix and it's Zebix is kind of old you know fashion and so we're trying to switch to Grafana Prometheus alert manager which is working quite well so far we have pretty nice dashboards if we have some time at the end I might be able to show you guys some of that and then we have a matrix that I already said that Teftik maintains and we have a Quasal and I'm not sure who maintains that to be honest anyway so over you about code so we have we we try to follow infrastructure as code where we're possible so that means if you make a change and like if you want to make a change in any of the servers you should may have a corresponding code change somewhere so for instance we use Terraform where we can and Ansible where we must so that means Terraform is a tool that allows you to write declared declaratively specify some infrastructure elements and the nice thing about Terraform as opposed to Ansible is that it actually does take care of cleaning up things after you remove them or that it actually tries to make sure that the the status that you give it is actually the status that exists that's expected in the end and Ansible in some cases has a really hard time doing that so but yeah we still have to use Ansible in many cases so for instance this is what it looks like so we have 8,000 lines of YAML delicious YAML which is pretty much all of it that's Ansible and we have 2,000 lines or lines of HDL so for instance in HDL or Terraform we specify our heads now the cloud servers we service we heads of cloud servers where we specify all of our domains our IPs all of that we specify all of our key clock things yeah all of that some some minor services as well and and Ansible we have like the vast majority of everything so we have 63 Ansible rules that's quite a bit that's that's everything essentially it's like email stuff that's that's all the services that you that you see that I that I talked about earlier except for key clock is in there so everything is in there it's it's quite vast you can take a look at that if you like and I think as time goes on we're gonna see more of those things move to Terraform maybe perhaps depends but we're actually pretty happy with Ansible it's not it's not a bad tool it's just you know YAML gets TVs sometimes and you know when you work with YAML mostly like every day you kind of kind of want to see something else but you know that's what that's what we're stuck with I guess anyway so the past let's let's continue so dark times dark times lay in our past yeah I mentioned like when I talk about the past basically speaking of arch when it was like first made till about four years ago so that that's the past that's the dark times and basically everything was mainly managed so that there wasn't like no centralized server management whatsoever right basically there was just a bunch of servers in some cases quite quite more or less literally in somebody you know else's bedroom and and all the like the devops that wasn't really like the term devops there was not like an arch devops team that there really was nothing like that it's it was more or less just there was some like people that managed some servers they had some scripts they knew how to do it they want to dox we essentially had the best factor of one for every single server so if you're not not familiar with this term basically it means you know if there's one person that knows how to do something and they get hit by a bus now we have zero people that know about how to do something and so that's a bus factor of one and it was really hard for outsiders to collaborate essentially was it was actually impossible to be honest um well because there was no no way right there was no code we couldn't just give SSH access out to outsiders so there really wasn't any way outsiders could possibly collaborate with us and there was no devops team so as I said before there's like only like a loose collection of people different accesses to different servers and there was no centralized issue management so that if somebody like had a problem with mirrors or something and somebody other that managed arch web or some some other service had to be kind of aware of that then there wasn't really any nicely defined way to to share that and there really wasn't any automations that were in place outside of cron so we did have quite a fair bit of cron doing work so automated bash grids essentially but really there wasn't anything that automated things between servers so yeah that was that was a pretty dark time so but then eventually we approached the road to enlightenment and that meant we started using Ansible so the first Ansible commit was made in May 2016 this is essentially also the like how far our infrastructure repo goes back and since then we've had a pretty recent pretty steady stream of migrations towards Ansible and we've also started using Terraform in February 2019 and I think we originally started using that for specifying the service for heads now I think we started documenting things documents documenting stuff is important and we started doing we started the DevOps team which didn't exist before and well and that brings us to the present and well now we do have a DevOps team with consistent access to all services and servers that we have and most things that we have are now in Ansible and Terraform so that's something I'm really happy with there's some stragglers here and there for instance like mail aliases are still managed manually you're not gonna find those in Ansible I'm pretty unhappy about that because you know somebody made a mistake there then that would be pretty horrible for arch emails and also pretty much a security problem but we're gonna get there on baby steps most things that we have are now documented or well most important things anyhow so we also didn't we used to not document any kinds of procedures or processes like onboarding off boarding stuff like that it was kind of just like in the moment like if somebody could remember to do something that they do it and otherwise they wouldn't really have a checklist so we started using checklists quite a bit for that kind of purpose like in the end I would prefer to be automating that but you know in the meantime I think it would be it's okay to just kind of have a checklist it's better certainly than that happening I think at all and we did start a migration I think that's what I said earlier like in February 2019 I think we started the migration from bare metal service to cloud service and so that has a few reasons first of all we like compartmentalization but that means that we want to have one cloud server that it does basically host one service and it allows us to in the end be pretty like pretty cheap about specific servers and services and we also had this effect on bare metal service where we had side effects where basically one service that existed on the server would be influencing another service so this is obviously broken compartmentalization but it was just done that way because it was easy and simple and was it was kind of like a hack and we don't really like hacks anymore so we're we did this migration or we're currently like in the middle of this migration and yeah that's still going on we're still sorting out some servers but in the end I think we're gonna have a cheaper most secure and I hope faster and much better documented um well general info after we finish this migration to cloud servers we're not gonna get rid of them like gonna get rid of bare metal servers entirely because we still need hardware features for some things and obviously like the build box would be much cheaper as a bare metal server but from the vast majority of services I think the cloud servers from hetsna are basically the best alternative that we have and then lastly we also have automations and ci which we used to not have at all right we had no ci and in any other of our patches or merger requests or anything I think we had we're like we had some very like sparsely ci like sparse population of ci files on on github at the time I think we used service for some services um but now with gitlab we we have like we can offer all projects one consistent and and well-known ci that's also trusted so that's also another important thing that we can now do trusted deployments from our own ci because we have our own runners and that's that's a big step forward um this is still kind of like mostly a work in progress but I think it's one of the most important features because arch being mostly volunteer or pretty much all volunteer driven means that any time that we as a team do not spend human resources on any of our well computer tasks like computers are much better doing like automated stuff like building stuff uh deploying things ci stuff making sure that unit is around and stuff like that uh I think all of that time could be much better spent on on actually working on real problems and the distro and it would be much less much less exhausting for everybody that that had to do those tedious tasks and also it will ensure that we just have a much better foundation for for mesh requests and and we make sure that all of those tests are being run and that'll make sure things are more stable for our uh software that we actually develop um so one important thing that I talked about that we didn't have before I think that's really probably like the centerpiece of probably I hope um our our DevOps revolution arts uh GitLab and so we received an ultimately license from um gitlab.com I think um who of our ultimate licenses to all open source projects that can demonstrate that they're open source so now we have GitLab so you and that was a pretty big step so that should be our central repository for code hosting issues collaboration um as I said before we earlier weren't really that easy to collaborate with which is obviously not great for a a project that really requires volunteers to have fun doing what they are doing because there's no other incentive so we're trying to make this a bit less um or making it make it a bit more seamless and and make more fun for everybody and uh yeah I'm used to have to send archpatches for many projects on mailing lists like it's uh 1980 and uh now we actually I think we arrived uh quite confidently in uh the 2010s at least now we have merge requests so all right um and as I said earlier we can also hopefully get rid of patchwork flight spread can but at the least um maybe more I don't know but for now these are these services are slated to be mostly or completely removed in favor of GitLab we have an overview of all our issues and um I can actually show you that so this is an overview that we have of all our currently migrated projects we haven't finished the GitLab migration obviously but um if you look at this this is essentially like the arch group in GitLab and allows you to look at all the issues we have quite a few issues um but keep in mind many are still in flight spray there's no package issues here at all so basically an arch you always like it's not just in distribution that has packages right uh obviously we mostly have packages and we mostly have package bugs but we also have many software projects of our own like arch ISO you might know arch boxes for dvms arch you know stalker um the infra project itself and and many other projects that we have and so these are all listed here um you can actually if you want to you can actually lay um filter by label for instance and see whether the label is um good first issue so some projects tried using or are gonna use are currently using this tag which is a global tag and then you can see which projects currently are by the reporters consider to be good first issues so something how you could potentially get involved um and I said before we have a powerful ci for everything now and that's also actually available to our users we hope it doesn't get abused too bad but if it does we're gonna you know come up with something we said some sensible defaults but guys please don't abuse our ci we only have this one pretty happy that we do have it at all um so um next topic uh that you might be wondering about is arch Linux sso so we have arch Linux single sign-on uh it's just a fancy way of saying that you have one account that's shared among all the services that we have and also one login that you have so that's shared so if you log into one service you should be logged into everything and that allows us to so you know you used to have like accounts for like wiki bucks so fly spray uh a ur possibly security tracker uh maybe if you were a tester you had an arch web account to to sign off packages so there were many accounts and internally obviously we have some more and so this just became really really hard to manage so imagine somebody gets onboarded off-boarded and for users was just plain annoying you know having to manage multiple accounts multiple identities and what for right and we also have the forums and it would be really nice to just have like one persistent set or one persistent account that would just follow your around everywhere and it would also mean that we have one central place to manage accounts and also to to manage access levels so for instance on our SSO currently we force two-factor auth for all users which we think is is really the way forward and to ensure that no accounts are taken away we realize that this is maybe inconvenient for some users but we decided that is probably the best way forward to ensure that uh well users wouldn't you know wouldn't suffer from reuse passwords quite as badly as if they didn't have a two-factor auth and also considering our users are mostly pretty technical uh it's I think it's better to say that this is a reasonable choice to make so we have one account for everything um two-factor everywhere and this is actually realized using the software KeyTlocked by Redhead which is well it's it's a it's a type of uh IAM which is identity uh no identity and access management and it's it's just a piece of Java software that we use which which seems to be the best compromises uh seems to offer the best compromise out of all the various software that does pretty much this thing out of there we also considered LDAP and and you know all of its implementations for a bit but LDAP really is more geared towards like uh systems logins and not so much towards uh web software logins so you can use LDAP with uh web software but um KeyTlocked which uh uses OIDC so OpenID Connect and SAML seems to give us more um let's say access control and and seems to be better geared for the web so this is what we we uh rolled with and also we got to be talked to the KDE people that did the migration to their LDAP back then and they said that we're pretty unhappy with LDAP and they're changing away from that so that's another thing um so yeah what's what's the look at like what's a roadmap like um well I said we had some migrations that we're still doing and we're still in the midst of that and since I just talked about as is all obviously one very hard topic is to get all the accounts merged somehow because we don't really want users to use uh to lose their access to any of their accounts and currently we have as is all enabled only for internal and for like for staff accounts and for users we're gonna have to get creative we're gonna have to make sure that we can really prove access so that the users can really prove access for instance for the like forums account or wiki account or whatever before um merging that into their one SSO account this is gonna be a topic that we're gonna be busy with for quite a long time I believe and and then only then can we actually connect those accounts as those services to to the SSO so currently it's mostly an internal tool but it will certainly at some point become more generally available tool especially for GitLab we're gonna open up GitLab at some point and then or well pretty soon actually I hope we really prepared most of everything for GitLab to be generally available and then hopefully we can give everybody access to that we are also migrating from amendment 2 to amendment 3 which is a bit of an annoying migration but it has to be done because amendment 3 is actually modern and supported and amendment 2 isn't so that if you guys don't know this is our mail bale server list software that we have and we need to automate more things as step before I think anytime that is spent doing things that the computer can do better it's probably time spent that we should be automating it instead and then the computer can do all things like the problem about automation like the only central problem is security because if you give a computer access to do things and you fuck up then you might be accidentally deleting all of our connects this would be quite regrettable I think I hope everyone agrees and so we gotta make sure that or if you know if somebody you know hacks the box or whatever then we wouldn't want to risk that so this is something we have to look into and eventually we get we should get rid of most of our bare metal servers so we want to finish the migration to be a host or vps or whatever and then we can get rid of the hardware service which will also alleviate some of our paints as far as hard drive changes for instance go or upgrading or something like that so that's gonna be much easier so next part is known issues of our future endeavors or things that we know are problematic in our current future up ahead is that for instance our form software fluxvb is essentially unmaintained and that you know it's php software so it's kind of like double bad so you don't want to have unmaintained php software under any circumstances but problem is currently our reality so we're really trying to get something else there instead I don't know we have no ssh audit locks so essentially that means or like if anybody of our dev ops currently look into any server they can essentially do anything they want without being a watch and also anybody can use ansible roles at any time and we would prefer if we had some kind of paper trail for that so this is kind of hard we do trust our dev ops so this is not about trust so much it's more about being able to to know when somebody did something for for general auditing so that you know if something fails you can see who did what and then maybe gather some information about the nature of the failure and as I said before our current may set up really needs innovation and innovation frankly yeah it's it's it's kind of from the very old days hasn't really changed much it's gonna change for sure when we connected to sso to ketlok so yeah that is also still up it's a known issue I don't really have any remedy for that we basically just gotta check it out and see what see what's good we need better gdr compliance automation we currently have we try to our best to be gdr compliant where we can but I think this is done mostly manually and and every team basically manages by themselves so wiki people and forums people have the different ways of dealing with this and it would be really nice to have some more general solution there's also place in there with the sso thing so if somebody has complete control over their account from one central place this would be easier for us to do but it's certainly a hard problem that has no quick fix so for the time being I think we're stuck doing this manually yeah so finally the call for action so get involved arch is 100% volunteer driven and and everything we do is public I might have shown you this before but everything we do is public so this is our info repo if you go to this link you're gonna see this this stuff popping up and and I'm happy to report that actually we've had a recent influx of active collaborators so that's really really something nice especially in the DevOps thing where you know things are sometimes complicated and kind of annoying really happy about having some collaborators join and but we always need more people helping out so make sure that you join Arch Linux DevOps in a free node RSC and just ask to help out just usually somebody around from us if we're not all sleeping at the same time which really happens I grant you that and and just ask for help and or ask for an issue or something or maybe pick an issue right you can join the mailing list it's not super active because we're mostly active on RSC but really important things are posted to the mailing list and yeah get involved we are super friendly bunch breaking servers is great fun especially you know if it's 4am and you really have to get up at like 9am amazing get involved break some servers with us we only have so many arms and legs just you know grab an issue make a PR make sure something breaks we'll deploy it no questions asked and but seriously you don't have to be an super expert to be of any help to us all right you you can basically most part of DevOps is just learning on the job I mean just look at the DevOps culture currently like there's one new silicon valley startup tool coming out like every week every day maybe I don't know some new amazing YAML based technology you can't possibly be an expert in all of these things basically just you know get involved pick something that you it sounds interesting to you and just kind of try to tackle it we're all learning on the job basically just just try your best I suppose it really doesn't matter that much we're all here to help we all make mistakes basically just just try it thanks guys for listening this far if you've made it use some links that I've also shown the presentation and also my my personal page if you if you're interested in that but apart from that thanks for listening get involved and see you on IOC Hey Sven hello let's have a look at some questions nice talk there for a thank this wrap up of all the infrastructure we're running I've split up the questions a bit in servers and services so maybe let's have a look at the servers first one of the prominent questions was basically by a lot of people is the Arch infrastructure actually running on Arch yes I'm happy to report that all of our critical and also non-critical servers are running almost entirely on Arch I say almost because we also use docker sometimes and we sometimes use non-arch docker images and we run some some minor services on other distributions for testing in fact all right cool our server maintenance handled basically updates etc that's a question by Machilano right so I'm gonna take this question as it meaning that how do we get notified about updates and then how do we perform those updates so we get notified by Prometheus yellow build a Prometheus exporter that allows us to see which servers have how many pending updates and then that is like too many then we update and we currently perform the updates manually because you know we gotta make sure that things happen nicely and we don't do any automated updates currently all right very enough um same person Machilano asked if there are any pros or cons of bare metal servers that you're currently seeing or facing basically in this infrastructure sure so the pros are they are a better bang for the buck they have greater performance and cons are they are harder to manage and if you fuck up you have to well basically reboot the same thing and then it's kind of annoying to mess around with that and they're hard into provision because you can't just upload a cloud image at least in the heads and you can't and in vps as you can so and they're also of course kind of hardened to to put into little provisions so this is also why we want to use vps in the future yeah makes sense I guess I guess following up on that there's a question by ebal in regards to vms and their provisioning more or less I can I can read the question for you I'm guessing you're using vms or vps instead of dedicated boxes which we are to some extent so are you using packer to build immutable images based on arch Linux so when you update upgrade these images every week every month or is there like a CICD automation for that right so we don't have any immutable infrastructure currently but we use packer and we use packer to provision the vpses and so the vpses are they are pretty long-lived so we treat them as if they were normal dedicated boxes and so we maintain them and update them just the same but we initially provision them using a packer image that we update a talk once we need a new machine yeah so in the vein of arches install once always update there's a question by arch guests on how do you handle security of the servers in regards to hardening and managing roles and accounts etc sorry can i can you read that again please yeah how do you handle security of the servers this is in regards to hardening and managing roles yeah sure so we use ansible vault for all of that and basically all of our stuff is an ansible vault and it's all in the repository all right cool um do we currently do uh disk encryption on any of the bare metal machines no we are currently trying to set this up john carlo has something in the pipe for that but it's really actually really hard to to really get this right and yeah yeah well we're working on this it's something we're looking into but it's it's really really hard to get right especially because we don't have any machine access directly i have an ansible setup for that if you're interested but i'm not sure if it if it works and yeah it's got the problem though isn't it it works for for my use case currently so i know um i i guess one of the more interesting questions also regards to to hardware and maintenance and so on is what made you choose heads not to host your services rather than another cloud provider that's by dr hashimoto right because well back on the day when we started doing all of this there wasn't that many alternatives really and heads are really was the cheapest by far and we've just kind of grown you know happy with them frankly they provide good services um there's really no reason for us to change yeah cool um i guess the ansible vault question was already answered there was a specific question by peter strudel in in regards to the secrets and so on that's handled by ansible vault um i guess that also leads me to a question that is kind of interesting in general for all of this type of infrastructure question what is the bus factor currently for managing the infrastructure that arge maintains so technically speaking all of the people that were in the list that i had in the other slide i think seven or eight or so they should all have access to all of the service so if something breaks we have at least have those people to be able to manage it now of course there's also the the question of knowledge transfer so not everybody knows how to fix everything and and many services we still are the bus factor of one but it's much better than it was before so still one in many cases um but we're getting there we're getting there that sounds good um let's hop on to the services um there was a question by lurst that's the same person who also asked about the bus factor actually um if we use docker containers for anything uh we do so we use docker containers everywhere on gitlub ci in fact we even built arginalix vm's since very recently we built the vm's inside of docker images using chemo so this is pretty freaky but yeah so we use docker quite a lot um we don't i don't think we use docker in any uh production deployments currently and um probably it's not going to change because we don't really have any reason to change this currently um but yeah it's going to be ci mostly and of course we use docker to build docker images so the official docker images are built inside of docker images yeah docker the docker um i guess something that i also have to look into in the future to fix my stuff with archaisel um there's a question that is probably more tailored towards uh whether dashboards are actually available publicly if i understand this correctly that was asked by kgz uh if there are grafana dashboards in a repo somewhere oh yeah we have a link for that actually so yeah we so all of our infra um or at least you know that's the kind of the idea that everything we do is infrastructure as code and so we also provision 100 of the grafana dashboards that we have as code and so they are available in the repository you can look them up you can set them up yourselves and uh we hope that in the future we are even able to expose some non-critical stats to to others so that is something we're looking into currently hmm yeah good stuff um then we have a question that is more or a set of questions actually that uh tailored around the the entire uh sso um topic uh will there be uh already created forum and accounts and so on get merged uh when we roll out sso well not when we roll out that uh i mean we we've basically rolled it out um there's going to be many steps and i think this whole process is going to take probably years frankly we want to get it right and um it's really hard for us to to make sure that users can really demonstrate that they own an account and we certainly don't want to like or on the on the side of um trusting too much um where it's not like where it's undue so uh we gotta make sure that for all of the services we can find some mechanism where the users can really guarantee that they are that they own this account and only then we will be able to merge so for the like for the time being and for some time to come uh users are gonna have separate accounts i'm sorry for that but at least you know going forward we're gonna have um you know at some point as is all for everything yeah makes sense i guess gotta make sure um are we thinking of switching the forum over to discourse it was a question by uh bittin well so um not discourse in particular we so we put up this to the foremost people and we had them try to kind of decide which forum might carry us until the next decade or something and of course like discourse was one of the choices and for various reasons some people didn't like that and we don't really want to make this like we don't really want to call the shots on that because you know it's it's a community decision afterwards after all and people gotta want to manage and like those forums and use them so we want to don't want to make that shot for them but we have to like i wear my security and maintainability goggles and everybody else like has to also like it so there's no decision currently discourse is one of the choices it's currently not the primary choice um but there's many factors that's coming to this to switch back on a bit of server related topics uh we we have uh mentioned that ansible vault is in use for secret storage and so on um there's a question by night fox or by about uh whether we want to exchange the current secret storage with for instance something like hashy corpse vault um given that we're using terraform sure um we thought about it it's currently not a priority at all maybe i don't know it certainly very much in the backlog okay um also a question by kgz and regards to our primitius setup if we are in need of uh any uh well exporters being written for any of our services still yes someone's offering here yes and we have plenty of projects that we operate and i think pretty much none of them have um like real you know business stats exporters for them so like everything that we have all of the customs over that we need uh have still needs exporters we only have some operation exporters but we really could use some more in-depth software specific exporters yeah nice so kgz if you're hearing this get in touch write some exporters please um we have a few more questions around the encryption of our cloud servers i think you mentioned earlier that none of them are actually at this point right okay so that answers that question as well um and do we apart from the docker containers use any lxc containers for instance uh there's a question by keto any lxc containers lxc yeah or lxc lxc yeah no we don't no okay yeah um we are uh moving to git lab currently uh most of our well internal projects and and so on and uh the question by orhun is if we um if github is used for anything regarding to defops or if it's just uh well our svn to gits um mirror basically for packages and other things right so currently it's only basically the mirror for all of our repositories so we mirror everything or the general ideas to mirror everything from git lab to github and not allow any contributions on github this might change depending on feedback but currently this is the way we use github so basically it's kind of like an outlet for for just giving users maybe a bit more uh kind of like view into the distribution but yeah we currently don't have any kinds of participation plan for using github and i think that is about all the time we have isn't it yeah i think so we got to wrap up i'm sorry if there were uh questions left unanswered you can still hook up hook us up on the isc channel or through the mail if you uh you want to know more yeah i'll certainly be around for any other infrared questions or stuff like that all right