 Thank you everyone who came for this talk. I'm going to talk about a better experience that we had at FreeAPA project with packaging our stuff for multiple distributions, but primarily for Fedora. For those who are curious, those ships are the ships that come every year to Helsinki Harbour for like 300-something years to sell the fish on the fish market once a year in September for a week. And they have this race that was raised from last year basically all in mist. That's why they all graze. Some of these ships they actually have red sails there, which is a bit of an interesting part for me because I have actually two red heads at home, but I'm not wearing it all the time. So I work at Red Hat for the identity and security and do a lot of stuff upstream. So I'm working on FreeAPA as you could have guessed from this talk. And I also work on Samba for the last 15-something years. And I'll contribute to Fedora whenever I can, primarily around the areas that we get affected with for the Fedora packaging, but for example I'm sort of a life support for ZZ-Leap package, which is kind of dead upstream-wise, but it's needed to build documentation for MIT Kerberos. And if we remove it from Fedora, we remove MIT Kerberos from Fedora. So sometimes your path to packaging can be absolutely unpredictable. So if we look at the FreeAPA, it's actually quite a huge thing by itself, but it also a system that packages a lot of or combines a lot of other software together. So I'll start with the framework. It's mostly Python, but it also has a bunch of C plugins and components. So this is kind of code-based statistics that you can see we have around 300,000 lines of Python code. Around 50,000, 60,000 lines of C, if you give or take the comments into account. There's some JavaScript stuff, pretty sizable for the web UI. Some of these things, they actually get calculated wrongly, but that's how all these systems work that take the statistics on it. It's a web application that runs under what WSGA in Apache has tight integration with the Emods.js API, which provides you single sign-on through Kerberos. It has a lot of Python code in the installers and it has a lot of plugins that implement certain things. This is basically to make Kerberos and LDAP working on Linux. And when it started in 2008, 2007, the idea was to bring back this idea of nice manageable services on POSIX compatible or Unix platforms, like Microsoft did when they basically took the Unix technologies and combined them in a nice package and made Active Directory at 10 years before that. So we're still doing this work and people keep us named as Active Directory for Linux, which is not exactly that, we do much more than Active Directory does, but so this is kind of a lot of comparisons on it. We do a lot of stuff around the MIT Kerberos to provide the single sign-on things and we have many features in MIT Kerberos driven by our needs. So for example, the two-factor authentication, the whole process of implementing it from RFC acceptance to driving to support was driven by the needs of free IPA project. And then there's Kerberos proxy implementation which allows you to have indirect access to your KDCs so that you don't put them on the Internet even though it was created to be put into this kind of environment in 1980s. Microsoft invented a protocol that allows you to know this over HTTPS, so there wasn't any implementation until we started working on it. Now there are four or five different implementations including those that are combined with VPN services, so kind of spurred up a lot of effort here and there and got to the nice things. So the other part, pretty important one is the LDAP server which serves as a backend to all these services. We use what used to be called Netscape directory server, then iPlanet server, then when Red Hat bought it, it was called what it was, Fedora directory server, FDS, right? And then it was renamed to 389DS. 389 is the port for LDAP, so kind of this thing. But we have more than dozen plugins on top of it that implement specifics of the behavioral things. This presentation it's uploaded and it has links to other presentations, so it's kind of a web of rabbit holes that you might get into if you get bored. The other part is that we handle the certificates and reissues of the certificates automatically, so we get a lot of stuff and this is another part of Netscape heritage that Red Hat obtained. It's the DOCTAC certificate authority which is upstream for Red Hat Certificates Services product, but also available as a part of free IPA. So we kind of cut down the number of features that we automate for typical users. Our HTS does a bit more. We just do what we want, but we share the same code base. And it's based on Java, so there's a lot of packages. And if you look at the dependency, reverb dependencies is like 350 plus packages altogether. So we are what one of my colleagues says, essentially we are the rail, Red Hat Enterprise Linux. If anything is broken in rail, we most likely will find it because we depend on a lot of that stuff. And then on the client side is the another component of this ecosystem is SSSD, which I think these days available almost everywhere on all distributions, that has a special provider mode that talks to free IPA or IPA and has support for most of its features. Not all of them, but it's kind of growing. It's also growing pain if you don't have those. So we also integrate with DNS, making it easy to dynamically provision hosts, enable, create the zones and records and zones and so on. We do DNS sec automatically and enable kind of a lot of things. It's a bit fragile. Some of the pieces are a bit harder to maintain things because we touch a lot of inside the bind. Bind doesn't have really a good plugable API. So we have to play Frankenstein style operations to get it working. Surprisingly, it works pretty well for most use cases. And this bind in the build up is the backend to look up the data in LDAP. So we have interesting case ones where some guys took the bind in the build up, compiled it with bind on salaries and used that to load the zones that they had. They had something like 30,000 zones with sizable amount of hosts per zones to compare how this would work with like a typical flat file-based thing. And apparently this scaled much better the environment. So the difference was like hundreds of times in the performance. Then we integrate with Samba to get the active directory things and extensively use some Python bindings to provide this integration because we have Python stuff there inside. So from some real-world examples GNOME runs its account system. So account.gnome.org actually runs free IPA web interface for all GNOME participants. Fedora project infrastructure has integration within itself IPA doesn't show you this thing externally, but you're required to use Kerberos authentication to some services and that's exactly where we are. And then Red Hat IT runs a modernization project since like 2015, so 2016 they actually started to deploy it. And last year there was a report on that one. It's linked from here. On the distributions level, so we have more or less full support for both server and client side from Fedora, REL, CentOS and Alt Linux. The full server support is a bit interesting thing and I will go into details just in a few minutes. Because IPA is not a single server, you could run it as a single server, but you actually can scale horizontally by replicating all this data and you can split them in sites, put in different locations and so on. It becomes quite harder to deploy this if you don't have all the infrastructure in place. So in many distributions we actually lack that infrastructure, but in some we do have. So Alt Linux and Fedora and REL based it once. They all base it with NSS and OpenSSL, Interim Peripet, components that basically provision all this crypto, databases for these crypto libraries, so we can do that stuff. In other distributions we are a bit troubled. And on Debian, for example, in derivatives the client support mostly there, so it includes SSSD and components that need to provision the IPA client, but on the server side for a long time it was a bit retaste. On the other distributions it's even more complicated, so Arc Linux doesn't have any packaging for the server, they had packaging for the client that got broke. OpenSusy and Sles they don't have server support anymore because OpenSusy had packaged free APA like four years ago and then they removed it, whatever was the reason. And the client side SSSD is built without IPA ID provider, so you can use it but you cannot use the extended functionality there. So what this kind of mostly complete thing means, we work with a lot of upstream projects and drive changes there as we see bits and pieces. So one of these is SSSD is our sister project, so there's pretty good collaboration on what needs to be added. But if you get a distribution that doesn't have good updates for the released version or they have forced it that you cannot really release a newer version into the updates, you might end up having pretty old versions that don't support the features that you might expect to be there. So in the support channels in free APA there's users mailing this, we get a lot of mails about the situation on Ubuntu with the older versions of Ubuntu where people kept using something that was released like eight to ten years ago and then insist to have features that were released to three years ago. And that happens all the time. Also with CentOS 5, people still force it to use that. I don't know who is forcing them. If you're one of them, please wink. So there's a lot of features were added into SSSD over years. One of the latest ones that I'd like to highlight is the smart card authentication for the local, for the remote users, for users from Active Directory or from IPA and the integration for GNOME environment where you can provision the settings and how desktops looks like, what applications are there through definitions of rules in IPA to GNOME desktop workstations. A pretty neat one, SSSD delivers the part of it. Also there was a talk last year about this with a demo. With Curve us, we have a bit more complex situation because it's an implementation of a protocol that gets a lot of scrutiny in terms of what it was put there and any new changes they basically come through. A committee that defines RFC specifications that might take like 5 years easily like it took for a spake exchange that allows us to finally protect in a single exchange when you obtain the initial ticket, protect your credentials against the men in the middle attacks but also get the two-factor authentication in one go. We got certain APIs and plugins extended in MIT Kerberos because we saw a need for that for enterprise environments. One of them is, for example, certificate mapping. It used to be that if you want to have smart card authentication to get your Kerberos credentials, you have to have strict rules of what is the certificate authority that issued the certificate. The certificate had to have special extensions in it which didn't really work outside of a lab. If you really have an environment where something is supplied by, let's say, your government, like your ID card with the government, you need to trust certain certificate routes and so on and you have to have this sort of definition somewhere and for different groups of users for different machines you have to have these differently defined mappings between what is acceptable and what is not and as a result you get an environment where, let's say, like the French government was talking about 5.5 million users on the matrix yesterday, right? So if you get that kind of environment, suppose that you have to put all these mappings into the kerb5.com for every single machine. It doesn't scale. A lot of things doesn't scale this way. So we are talking with the upstreams for introducing interfaces that we can dynamically load plugins that actually do this sort of decision-making in the code instead of looking into statically defined things. It took several years to get these interfaces, get them rolled out into the upstream releases then in the distribution. So eventually it comes in into Fedora, Rel, Debian and others and you could start using them. But nobody really other than the upstream projects like IPA really tracks that all these things exist. And if you try to package this kind of thing you have to know that you have to get this knowledge somehow. There's automated translation methods, for example, or I didn't write here for the Kerberos stuff location of the KDC according to your locations so that you don't talk to KDC on the other continent because it's just the one that your DNS, SRV record mentions. These kind of things they were needed to be written and created. And this coordination it goes beyond that. So all these projects, they are fairly independent. Like Samba has own needs, MIT Kerberos has own needs. We have good relationship but it doesn't mean that we drive them. We use them, we contribute, but we don't hurt them. It's impossible to hurt all of these fellows who are of strong opinion of what they are doing. You could coordinate but always the coordination might fall. It just fell because somebody forgot to tell somebody else. And especially it's hard on the distribution level where you have, aside from the upstream thing, you have maintainers and maintainers might not give any interest into your tasks. They are interested in their own stuff. So a typical example that Samba often updates its own libraries and these libraries they have own requirements so back ports and all this stuff might break the versions of Samba that are in the back ports under all the release distributions. You could do all this stuff but it's a snowflake that turns into Blizzard quite easily. Well, I'm from Finland so for me Blizzard is kind of a living creature. I escape at one when I came here. It's still there. Yeah, but really dependency chain reaction for these changes limits us from, well, us as upstream plus all maintainers on all distributions from being flexible here. So we rely, as I said, on MIT Kerberos. We do code level dependencies and the thing is while Kerberos is a protocol kind of interoperable you cannot really do this interoperability on the single host. There's simply, it's not designed that you can load compile against MIT Kerberos and load Heimdall implementation of the same library. It just might crash application, might not, typically crashes and depending on how far in the process you are, you might get spectacular crashes. So and also there are features that don't fully match. You get the things like two factor authentication which might differ between implementations. You get support for credential caches that exist in one implementation not the other. So for example, Heimdall doesn't know anything about Kerin, kernel Kerin based credential caches and for a long time MIT Kerberos didn't know about the KCM protocol for the remote credential cache management that Heimdall invented. So it was kind of a game where you really cannot combine things. You could interoperate on the network but not really on all things. On the Debian for example, Samba is packaged with Heimdall. So it's compiled against Heimdall because Debian wants to provide Samba AD DC and we spent seven years Samba upstream to do the MIT Kerberos-based Samba AD build. So it's extremely hard work and we package that in Fedora now so we don't need to have Heimdall in Fedora and we can build everything from one source but Debian still doesn't build MIT version and frankly it's not completed yet. There's some pieces that still missing. We still need to complete them and then switch off the Heimdall support in Samba because it's also quite costly to support two code bases. And yeah, we do have things in IPA that extend Samba so you load this IPA Samba driver that queries LDAP database for all this account information and you can load that but because on Debian Samba is compiled against Heimdall, you get simple scratch and you get structures that basically do not correspond what the code expects. You get quite good crashes and of course a lot of bug reports. Then the other part is Java. So Doctax is all Java. Well, it's all Java plus Python plus C and stuff but it moves slowly because the primary use of that code base is something that is heavily related so if you do Red Hat certificate server deployment you expect that this will be a certificate authority for some big organization which is somewhere in the bunker under the ground and so on so protected with probably less moving parts there so they don't really jump around and implement all the new features. Like JDK 11 was released last year in September or so and so Debian moved on to get this thing and Doctax doesn't support it yet. So we cannot really get Doctax now working in Debian because of certain things are not compatible there on particular on the TLS level because there was a lot of change in JDK 11 around the TLS 1.3 implementation. But the other part is that a lot of operations they assume that you use crypto library called NSS which is part of Mozilla and Firefox things. Firefox people think that this library is within the Firefox. So it's not really a shared libraries but a long time ago Red Hat did standardize on NSS for crypto operations. We thought that we will convert the wall to use NSS and it didn't happen so nowadays whoever uses what they want they use it and we try to get the other side of it which is certification for governmental use and of spread out for NSS at least and the open SSL but Doctax depends on NSS a lot and it expects certain things to work there for PKS 11 modules for hardware crypto modules and so on and there was attempt to get this bridge between mostly open SSL based environment and NSS based environment with a special library that used private functions from NSS that the NSS project didn't want to publicly expose so they forked part of the code and then NSS decided that they killed this code kind of interesting story and NSS PAM finally got to Debian right last year thanks to Tima Althanin who is a lone supporter of this whole infrastructure on Debian distributions and not in his primary employment time and then we got this JDK 11 break and everything else so that's one of the stories that kind of quite sounds quite bad for Debian users and then the other part is that we get a lot of to coordinate system services so you have to run MIT Kerberos you have to run in some environment Samba then LDAPT and all these things together we depend on system D for doing a lot of these things like circuit activation to factor authentication because that's sort of a thing that gets activated automatically from the Kerberos KDC when there is a request coming in and otherwise it's shut down and there is an obstruction layer for doing all these configuration file changes if needed for example for all the rail versions there is auth config and then there was a change for providing auth select in Fedora so we had to adopt the other distribution somebody has to write this code so there was attempt to write recently for Debian so we get a bit of movement for how PAM and NSS configs are NSS which configs are written down there and Arc Linux for example they have a downstream patch that doesn't apply anymore they never submitted it upstream they don't know why we basically would like everyone to work upstream but if it requires changes in the distribution we are not at the position who have to do this work so now I'm coming to the actual thing that happened in Fedora and it's a pretty interesting thing because it's collaboration between multiple distributions while it's benefits in Fedora here and FreeAPA it came originally from Suzy so we get on one side on upstream FreeAPA we have continuous integration system that's put in use for every single pull request it's running VMs in open stack environment so runs them in parallel whatever we have kind of capacity there so that you get pull request, you get some tasks every test kind of unit gets deploying a server deploying replica, deploying clients doing some operations that's specific to the test then tearing this apart then doing all the other things in total there are like 55 test suites for each pull request and you can run maybe five, six pull requests in parallel at the same time at our current capacity and it's roughly two hours of a wall clock so you have to wait but you have to wait anyway in any of those cases then on the other side we have nightly runs so we take whatever is in the Gitmaster or a specific branch we have this for several branches and run this against Fedora version that is released and roughly once a month we update the images so that we include something that was in updates by that time or when we know that there's some problem fixed and in total it's about twice of those test suites so we get around 700, 800 individual tests executed it's pretty large so if you kind of take them sequentially that's not really nightly I tried to find a planet or a dwarf or asteroid that has the same rotational period like this 60 hours or so there's none in this system so then I started looking around what it's exactly and it's parallelized it's around eight hours so that goes into nightly unfortunately we have a team that spawns from Australia to Brasilia the other way around and that means that we have people working almost all the time so for us nightly means that we still waste resources that somebody could use at that point and this is a typical diagram how parallel those tests are the interesting part here is that there are few tests that run two and a half hours long and they are pretty heavy ones so in the middle closer to the lower part there's this long big line that's a DNSSEC test that does a lot of things like enrolling, creating a zone with DNSSEC signing it up then trying to get the renewals for the signatures there going back in time, going forth in time all these things that's why it takes also a lot of time there the shortest line second from bottom is actually built so build is small the real thing is huge the deployment is about the same time as the build so you get like 5-6 minutes to deploy the full single system and then you start doing the actual tests and then we get to Fedora so we do the integration at the update submission time so with Fedora the concept is that you first build packages and then you combine an update request that goes against a particular distribution version and at that point when this update request is submitted you get a bunch of messages actually on Fedora messaging bus that somebody listens to at least I hope somebody I get them on IOC from the Fedora bot but there are other bots that are listening for this and one of the bots is actually running OpenKa set of tests and OpenKa will be described by the next presentation so if you're interested you can look at that and OpenKa was created by OpenSuzi and it's a nice thing that allows you to get a lot of interesting things with the distribution it starts with a bare metal and you can program everything there so we get a lot of things some of them I didn't write this here in detail but the tests we run are typical tests like deploy one instance of IPA server one client try to create some records there are some administrative tasks and then try to use that and we have tasks that actually imitate the full use of it like full desktop experience where you have the server you have a client that's enrolled to that server and then you get a human logging into that system through GDM so the graphical environment supposedly gets in Kerberos ticket at this point so you launch Firefox you do single sign-on into the site into IPA management console and then see what you do there and OpenKa automates checking that things are done right so you do the operations that are actually there and we also do the upgrades so the upgrades is an interesting thing because you can run a server on the previous released Fedora version then upgrade the server then enroll a client which is running on the older version then upgrade the client and verify that it continues to work so it's kind of the work that you really need to do for the release engineering and we do it at every package integration this is a demo so for example right now this shows you a kind of environment where you have a server standing up first then the client standing up then a client connecting to the server enrolling into the IPA deployment and then you try to log in into that client and let me show you it's basically two-minute clips I hope they actually will work this is all recorded automatically by OpenKa it's a bit of sped up of course of what happens you cannot do that manually unless you're a super-typer guy so this is Fedora 28 yes, installed something rebooted installing some other packages that needs to be there this is Havaget to get the randomness in the VM in an easier way and now it's deploying the server Fedora has this roll-keep thing that's scripted and of away any details how the actual deployment happens I think it's around 30-40 seconds in this compressed video so you'll get something it is a local network within the environment OpenKa creates what you don't get there it's a weird IPv6 yes, weird IPv6 because there's no IPv6 in that environment so now script is finished but it waits until the whole stuff settles down because services need to start and I hope we still have 40 seconds to actually complete this particular scenario I think there will be like 10 seconds or so after widget starts moving because something happens behind not visible on the actual console which is recorded here so now it collects all the logs and provides us all the logs including the AS Linux stuff and then you can look up and see it so a lot of uploads updates and final rebooting it oh, this is actually a second one already, right? yes, so this is a second demo now we have a machine we now have a client that tries to enroll to that machine Totem doesn't make it easy without looking at what is happening there so stopping the playlist from execution was my mistake, I didn't do that so it started playing the second video but that's exactly what we need so we have a server I think it's being started again 10 to 20 seconds something happens in the background of what we don't see on the console but after that it should start a graphic session log in into Firefox into cockpit console and through cockpit actually enroll into the system okay, joins the domain as administrator does something, okay now it goes into IPA console and creates users there creates some other things yes, quite fast adds them into the access rules and then tries to verify that you can actually do these operations as that user and you can log in and see your settings change the password and all that stuff, amazing all without human intervention when sometimes it's great, finished we can switch back to the actual presentation so yes, we can catch non-trivial box with this and last time I asked Adam Williamson who is a Fedora QE guy who implemented all these tests in OpenQE how many bugs we caught in I think 18 months that this is in place in Fedora and he sent me a list of 33 bugs we caught some of them are pretty weird you really have to look into details to see why they happen a lot of bugs are as a Linux access denials so we actually find out that you need to extend the policy and so on but there's one thing with two bugs there was a bug in MIT Kerberos that caused it to crash in multiple applications the real cause was actually a bug in 389DS and the LDAP server stepping over several threads they can step over each other's Kerberos credential caches so wipe out tickets for each other and then bedlock in Kerberos library because they kind of use some resources that touches the same credentials cache so while fixing this bug in both of them there was a security fix in MIT Kerberos so they had to do a CVE package update and they fixed it and that was a mistake the guy backed off the patch that fixed the bug for 389DS in Kerberos so we had several times back and forth with the security update that was simply broken for IPA and that was noticed by the open key test that we run and 389DS broke another part of IPA by changing their internal way of how they create LDAP sub-trees so it's now fixed the fix is in Fedora 29 since Friday but the story was like two or three months of interoperability things and it's still visible in the upstream night LCA because we did not update the image that we test there which we should do next week the director server update from Friday actually shows you that all these tests are fine except the first one which has some minor AVCs in the logs that's why it kind of highlighted there so that's the great thing so what's next, what we want to do we want to integrate all this stuff at if we look at the how we would test so we test what code goes into the release of IPA but we also test what code goes into the release of Fedora or Rail because it's the same everywhere and we need to test the other way when things are just added to the packages in Fedora we want to do it at the pull request time but Fedora CI infrastructure the standard one doesn't allow you to run multi-host tests so we want to plug in into that and listen to the messages that are issued on the bus and run this in our PRCI upstream thing to get it there why don't you use the multi-host testing in OpenQA? we do use multi-test testing in OpenQA yeah we do it's just that at that point it's too late we want to test when pull requests are submitted to something like NSS library or something like Curl I've got ideas for that yeah so this is the typical thing it's just something we need to work on to get it done on the other side we look for contributions for other operating systems like I really would love to see Susie actually packaging for APA we're working on it and that's why I want your test yeah and we need to share all these tests because we really share the same stuff there and get your nightly runs on OBS for anything you could do but it's a bit more of work because like I say 300-something packages involved in all this thing and you need to coordinate this is more than herding cats it's like herding asteroids or comets so questions if I have time left five minutes okay sorry the audit part okay the question is IPA stands for identity policy audit when audit will be implemented that's the question we have a project that's called common logging that tries to collect logs from different places and coordinate them together it started with the open shift needs because they have a lot of logs that are not related to the host systems they are related to containers and they need to collect them on the other side so we rely on that when it will be ready for collecting logs again we have a lot of replicas so you have to collect data from the replicas and from the clients as well the other side for regulated industries we have session recording already in Fedora it's the T-LOG project and SSSD has integration with this it's basically replacing shell that your user runs automatically depending on the rules that are defined in IPA in a central way and then that whatever is done in the shell is recorded in journal on the machine we have a lot of metadata in JSON format and integrated with cockpit so that you can browse the changes and see what services started what they produced when somebody did these operations last forced them when we had the identity dev room there was a talk about it okay well if there are no more questions thank you