 So, hello, my name is Alexander Bakov, I am working on different things within the identity management area and on free AP project, on Samba and on Kerberos and other areas, SSSD and so on. So, this is going to be a presentation about the general state of the authentication and identity management and Fedora and it can be seen as a kind of joint report from all of these projects. The idea is to cover what we have done in, let's say, last year or two in reality because it's all spread out in time in many areas. So, but the first thing I wanted to start with is the celebration. So, we actually have 40 years of HOSIX API service for identity because the get PWNAM and get GRNAM were added in version 780 and TUNIX in 1979. So, we have 40 years this year of a celebration of probably one of the longest existing stable APIs in the POSIX environment, which is, yeah, which is an interesting thing in itself because despite all the changes on the protocol level, on the storage everywhere, we still use the same semantics behind all these calls. And the name service switch, the mechanism that actually defines that you can plug in different mechanisms was introduced 26 years ago. So, again, we're just off by one with the quarter of century celebration there. On the other side, the authentication APIs like PAM, the API was created in 1995 and standardize it into a single sign-on spec in 1997. So, again, we're dealing with something that happened before the turn of millennia. And looking at that, the NSS and PAM basically represent the last APIs made in 19th century. So, this is a picture of kids finishing high school in Finland where they celebrated by driving in these kind of trucks and showing the weird kind of posters on it and, yeah, representing themselves in whatever kind of costumes they can have. It happens every year and they throw away candies. So, there are people standing around and watching and gathering those candies for about two hours. So, this is in Helsinki, they have a lot of schools and so this procession is going twice and I think it's around 80 trucks or so. So, quite impressive. And if we look into the situation, the network protocols that we use for identity purposes, they all actually from that century. SMB is already 30 years into it, 35 even, right? Kerberos is 30, LDAP is 26 and Active Directory has just turned 20 last year by combining all of these things together. Of course, there are updates to the protocols but the fundamentals are still there and it's amazing what we achieved as the industry with those fundamentals because like 90% of Kerberos and LDAP and SMB infrastructure is probably what runs the whole world with Active Directory. And on the other side is standard API because it's a lot of work among different groups and we have different implementations of the same stuff. So there's NSS, Pamela Dapp that kind of combines all these things together then 12 years ago we started with SSSD and there are rumors that it actually contains the SSSD, it's the naming after the people who were doing it, right? And Steven was one of them. It was Stilo, Steven, and there are a bunch of Pam modules that come with Linux Pam project but there's a bunch of Pam modules that come from different vendors and different other projects. Notable ones is the Pam windbind for example or RSA secure ID implementations and Kerberos, there are actually two Pamker B5 versions, two separate projects with different code bases but the same naming. And we'll look into the reality of production use of that on the typical environment, you see all these things. So the yellow ones, orange ones, they come kind of loaded into the Glypsy so effectively they load it into every single application. The green ones they load it through the authentication phase in Pam and this looks like a generally good idea to have this extended but it led to a number of real problems over years because first it's a modular stack, yes, but you have to maintain this modular stack in a certain way and then you have to configure it so of course most people don't like to read the documentation and do configuration by hand and 20 years ago OTH config tool was created to kind of help them with it and it became quickly a pasta of multiple choices that contradict each other and create a funny environment where you don't know what applies to what because there's really no way to tell it, you need to know things behind it. But then the other part of complexity is that effectively most of these modules either on the NSS on Pam site they are configured locally, they cannot consult the network for their own configuration because it's a bootstrap and problem and this means that you have to distribute somehow all the details and have them locally because they load it into every single process they need effectively you have access to the system-wide place where these credentials for example and configuration is stored which creates some security problems and more to that we came to the point where there's a bunch of applications that would like to use the same information store for example LDAP to derive and get the access to information which is ultimately non-POSIX like POSIX API has no call and no way to tell that the user has email account as mean you detail that still something that did not come into need for like 40 years but any application that wants to operate like with the user databases most likely wants to use email or some other identifier that it can use to communicate with the user so this information doesn't come from the system database it means there is a duplication of effort to access it and so on so we have new requirements new technically they are 20 plus years old requirements but they are new in a sense that they don't fit into POSIX API here and with all of this thing we have a standard API but we have variety of issues related to it and one set of issues that I described they are really coming from production side but the other is a long-term issues related to maintenance and development of such things so for example because the way how both APIs work for NSS and PAM there is no execution context isolation other than the process itself so whatever you load in NSS will be accessible to any part of a process they can theoretically pick up your credentials if they can snoop on the on the process content itself which is by default they can because that's their own execution process and then as I said it's a local configuration so scaling configuration across thousands of nodes is a real problem and then because these things exist for quite a long time and open source projects not all of them actually persistent or that persistent in the existence the lack of support because original author decides it's not the topic that interests him anymore and so on the lack of support is a real thing so I said about two PAM KIRB5 modules basically both of them died long time ago were redirected last year one of them and one of them was written because the other died five years before so it's kind of confusing thing that that happens there and then the other part is a lack of so so so fancy technology behind it it's not really attractive to new generations of engineers that are coming out of universities and willing to work with fancy app stores and mobile applications where they might get faster breakthrough into top 1% and so on the dreams are not always coming through but they definitely put us away or far away from from the interest of new developers so probably life is easier with rest and we don't have rest we have to maintain this stuff and we use SSSD in Fedora and actually this year is 10 years universe anniversary of bringing SSSD in Fedora the first build 010 some git commit hash happened in 2009 I was even before I think it was Fedora core 11 it was still Fedora core according to the Koji at that point it's Fedora core 11 yeah well the way it works architecture is still stays basically the same and scalable in terms of how it handles things the most important part is that the actual work is separated from the application that loads NSS SSS or Pam SSS so we don't pollute the executable space of that application we just talk to over the Unix domain socket to the responders somewhere and use cash there's some cheating here because this cash has multiple layers and memory cash is actually mapped into the actual process but that should be very low that should create very low overhead to access it and especially for quickly and often access IDs for groups and users this creates quite fast operation even not requiring the context switches in many cases over years this was extended to handle the offline logins extended to store and enforce pseudo rules as a Linux policies some other rules like in case of free APA there are access controls based on the host and service you access and then multi factor authentication things that came a bit later and smart cards of course so the interesting thing is that we tend to think that Fedora is a bit of on the edge with all of this apparently relate when even more extreme than Fedora in its handling of this architecture so Fedora still has all of these pump care be pump because he has 11 modules they mostly rebuilt by the release engineering when mass rebuilds happen but they are still in Fedora repositories so you could configure a system if you need to use them while well decided to stop doing that and deprecate and eventually remove those modules because of the various factors there's as I say some upstream side effectively that there's nothing to contribute to there are better replacements in some cases and just I think this week on Fedora devil there was request let's let's remove an SCD from Glypsy because well it's obsolete apparently it's not fully obsolete but as I said Fedora quickly losing its extreme frontier and of thinking because people are thinking that time has gone for some of those technologies because there's SSSD SSSD handles all of it yes yes yes yes it's just a different module represents access to it and the other part is that you have to manage it you have to somehow configure systems and that became a mess so if you look at the lower part of this slide it's a screenshot from Fedora bugzilla from Fedora 26 actually where just calling off config in the fingerprint package actually makes your system not accessible anymore so you just uninstall a package and your system disable all logins the reason for that is that the off config dash dash update actually requires you to specify all the original options that you specify there if you want to keep them there is no state and it just regenerates configs based on what you provided there and if you don't know what you provided there's no state you lost it so at some point we decided that we need to do something with it the off config maintainer said I'm done with this call there was need also to port it to Python 3 so there was decision made to redo a new tool looking at the actual use cases and covering those that matter of course you can have customization added but for the majority of people this actually went more or less transparent in Fedora so we had this in Fedora for two and a half years before relate came and I think there were few bugs we fixed them but there were mostly no bugs related that you cannot configure PAM and NSS PAM held up or stuff like that while in in rail we got more conservative customers and they are really wanting to have some easy way to generate configs for them on the off select side so there are two two big profiles the knees profile is is there for historical reasons and mostly deprecated because you actually use it in the high performance computing because of the really small overhead they don't need authentication there they just need fast fast access to these ideas because typically clusters are isolated enough in the trusted environment that they really don't need any authentication around it it used to be this way the net group support is actually using LDAP behind the scenes so there's needs API on the host for the net groups but real implementation doesn't use these protocol anymore so you don't need it and the all select itself is configuring is a set of configuration templates that simply configure the the files that define here but it doesn't do create SSSD dot com for example for you or SMB dot com it's not its task for these you have systems that enroll you into a domain IPA has IPA client install Samba has net ADS join but also it has this realm tool that unifies both of them so realm join the text which the main controller you are using and then uses the appropriate method to that so there is a wrapper called off config that implements kind of the basic options that people use by remapping them to the all select options but not all of them and if you want to add something you just add a new profile in all select and maintain it yourself it's extensible then on the other side for the authentication our colleagues from the security team they work it over years to simplify access to tokens to smart cards to all of these devices whether they are virtual software or hardware that implement pkss 11 protocol the pkss 11 protocol itself API itself is there since 1993 but in 2015 there was new RFC that's kind of unified how you specify access as a string access to device and that made possible to generate a common access to everything by parsing this URI somewhere in the P11 kit layer and then deciding where to go so this was plugged in in all crypto libraries that we have in open SSL through the pkss 11 engine into NSS into new TLS and directly in open SSH because they have their unique needs and again there's another tool new PG which has also its unique needs to access resources so there's now interesting unified pkss 11 stack that you can access on a Fedora machine and if you insert a token like a UB key or nitro key you this token suddenly becomes available at any application at the in the top list and actually on there are other applications that use for example new TLS and open SL and all standard mechanisms work there so instead of following complex kind of configurations complex blogs how to set up things with smart cards we are down to really strings like this so if you have a single device you just specify that you actually want to use something from the pkss 11 token and these are just two separate quotes right so this this is in quotes defines the specific token that you want there but this string they can be logged they are mostly human and readable but the P 11 kit tool and pkss 11 tool they can produce them for you so ideally we should have norm tools that allow you to choose which token to use and in Firefox this is pulling automatically so in Firefox you actually have the UI to choose which token to authenticate with automatically and you don't need to do any configuration it's it's working automatically as long as you have access to the hardware which is the other part that you need to have you dev rules that map write access to token that is into it but on the workstation I think it's it's already there for for most of the things for the logged in user while on the Kerberos site we did two big areas of changes over years one of them is on the client side there was a lot of work to simplify how how credentials are stored how they accessed and how we find out users not users our domain controllers over the network and then after all of this security breaches after the problems with hardware and so on there was a lot of pressure to actually stop using weak crypto hashes stop using the algorithms that really lead it through their life and the other the other part that happened is that yes you can recommend that people should not use them but they still use them because they have software that was installed like in 90s and it works and it is a simple thing but new people who come and implement new applications they don't really follow all of these recommendations always so some of the way to apply this is actually force on defaults to not provide access to to the deprecated crypto and we started doing this with the system crypto policies in fedora two two years ago and now we basically got to the point that we removed deaths and triple deaths from it and RC4 hashes on the life support but for example in the current fedora you cannot use them from the command line with the Kerberos so going over the client side first there is a new addition in SSSD that implements a storage for Kerberos credentials using the so-called credential manager protocol that was originally created by Heimdall Kerberos and reused by Apple in in macOS and MIT Kerberos has support for this for what five years or so but the client support only while the server support for MIT didn't exist so SSSD team implemented the server support of KCM storage so you can now use all of these file directory key ring and KCM storages originally we thought that we will use key ring as a secure method because that's the memory in kernel space that you cannot swap out you cannot access if you're not allowed but apparently there are two problems with it one is that it's a limited kernel memory so if you're a sysad mean that has to access with your credentials thousands of nodes you basically generate thousands of small tickets that populate this area and in the kernel side we get only 40 64 kilobytes space there per element and there is a quota on what you can have there and most importantly for its per user so if you're rude you get it used up by multiple applications easily but really if you run parallel SSH you immediately create thousands of tickets that needs to be stored there somewhere so KCM solves this problem by not storing in kernel but storing on disk and hidden by Unix domain socket access and this solves the other problem the other problem is that the key ring is not namespace it so you cannot use it in containers all containers have access to the same key ring that is on the system if they run with the same ID and and that means that you're not secure with it while with the Unix domain socket access you effectively can you're under UID namespacing and you can apply discretional access quite easily so this works well for the containers access and works well for stuff like desktops silver blue and for example the Fedora toolbox already imports automatically your KCM credential which is default in Fedora if you didn't upgrade from the earlier versions the new installations default to KCM and you automatically get your Kerberos credentials working in the toolbox containers yeah so all of these key rings they they had some use cases behind and they still do and another use case with the dear credentials cache is that in MIT Kerberos credentials cache access treated interestingly from the code that resolves to which realm your ticket belongs to and if you have to credential cache collection and you have just as API based application like Firefox then if it sees a ticket from the realm that you want to access servicing it will use that ticket instead of the one that you want so effectively tries to be closer to the final destination but sometimes it's it's not correct behavior sometimes you want to use the cross realm ticket and an access with your own so that that was breaking some assumptions and splitting things required using different types on the client side as well we had the problem with DNS based finding of the domain controllers especially if you had old Kerberos environment and you had access to those resources without fully qualifying service name so you access it all your systems by their first name let's say SSH server one not server one example dot com this doesn't really work well because client needs to find out the mechanism to redirect and construct the full name it supplies to the server and this to the domain controller and what's worse if you get server one as your name there's ambiguity because there might be multiple server one in different DNS so it works well in a single environment single domain environment but really doesn't work over internet so you need to have some tools to discover the right one and you need to find to have some tools to fall back from the canalization or to canalization and these two things were done in Fedora so one of them is to support the DNS URI RFC and it's used by Fedora project itself so you have this Fedora package if you install it installs the snippet of Kerberos configuration that says hey we have Fedora Kerberos domain controllers there and also use the KDC proxy for them to access securely if you know and the client will use DNS URI to discover them and discover the URI to point to the server the canalization part is now a tree state so this was a big problem with two state on and off when people started using when Fedora project deployed free IPA in the environment where it's all fully qualified domain name but some some environments where Fedora contributors work it used the old like non non fully qualified naming of the hosts and it was failing because the library doesn't know really where you want to go before it goes there and tries it so we introduced a fallback it will be in Kerberos 118 and it is back ported already in Fedora and I think in rail as well and it fixes also use the open shift environment where they typically don't have the right principles in place they have principles issued in the name of a container and not the prince the primary principle how it should be so we have also some work in the KDC discoveries so for KDC proxies because SSSD allows to find out the closest site for you to use but the way how KDC locator plug-in in Kerberos library operates together with providers it doesn't really allow you to say hey I have a KDC proxy here use it it just allows to say hey here is an IP address talk to it using standard Kerberos port which is not not necessarily what you want to have we have with the proxy environment on the crypto side so finally we deprecated desk that code removed in Fedora from Kerberos triple desk as well most of arc for features removed except the ones that some bonnets for the active directory operation its market deprecated and there is a nice addition that in in the Kerberos library basically when you do tracing of Kerberos operations you can see encryption types as numbers and you don't know what they mean so now in Fedora those numbers actually prefix it with the short name of the encryption type so you immediately get and it's not only encryption type but also the pre-authentication operations for example offered by the KDC and the client so you can easily see what is there you don't need to go and search for those obscure numbers the other part that is happening is the SPAC pre-authentication this is one thing that supposed to replace encrypted timestamp pre-authentication which allows completely to avoid attacks or password dictionary attacks it is already in Fedora enabled by default for free IPA installs and if you disable encryption timestamp if you don't have old clients then you cannot be attacked anymore with the password dictionary things the next step with SPAC will be to add their multi-factor support so that you can within the SPAC pass all your token values and authenticate easily that is that is one thing that currently SSSD is complimenting by using the host k-tap creating a wrapper and then using that wrapper to pass your token credentials but it doesn't work from command line if you decay in it you need to do a special mangling there to get it working so it will work then the other part that we have is the authentication indicators it's a nice thing that says okay here's a service it's serious enough to grant access to it only to those who in possession of multi-factor tokens or smart cards and in IPA you can say to configuration that only issue Kerberos tickets to that service if the original ticket that the user owns was obtained with a token with a smart card or with something else so we we allow this differentiation and we currently work together with MIT upstream to do this more flexible there's already an interface in 117 MIT Kerberos that allows us to do KDC policy so that KDC driver like free IPA can make a decision whether to issue ticket or not whether to issue ticket with prolonged time based on what was used for example if you use smart card you might get longer ticket if you use just a password you might get ticket that short time and manipulating the authentication indicators and this will also allow us with SPAIC to add to support for FIDON U2F tokens which is very promising thing because it really isolates you from whoever is handling this but there's still need to write an RFC how this integrates with with Kerberos and the question is not in the code it's more or less time in question so who who has time to write this down yes and the one promising thing I'm looking right now into is to map between different approaches so Active Directory with Azure AD integration with Windows Hello for Business with Windows 10 has support for they call it password less so you don't need a password because they can use your biometry or single issue tokens or TPM in your computer to unlock behind the scenes it really is using smart card authentication of PKE in it so certificate based Kerberos operations but it's not visible to users but the users get a Kerberos ticket that has special asserted membership in the special area of that ticket and we can actually map this information to authentication indicator and therefore allow them accessing resources on our site somehow or we can do the other way around we can provision this information in the MSPAC record so that Windows resources or Samba can see that there are these seeds and they can grant access to so it's it's becoming more transparent moving between different approaches that was client sort of site so on identity services and servers and we have plenty of alternatives we have less in rail than than in Fedora on this front right and there are some news on the directory server for example so it has now integrated plugin for cockpit that allows you to configure and manage instance this is my IPA instance that you can see all monitoring details cache statistics and so on so that we don't need to expose this in IPA itself for example it's you can create records just a normal like LDAP modify LDAP add but from this UI not like users but real LDAP entries that you copy there so it's still hardcore but it allows you to configure some parameters that you otherwise need to create LDFs to to manage them there was a lot of work on auto tuning different caches and so on together with Susie and there's some work also on the usability of 389DS console tools together with Susie and a lot of improvements for performance there will be even more soon there's some cache contentions already track it and fixes are coming so on free IPA side it's basically integration right so it integrates most of those fixes and changes as you can see but there's one notable thing is that you can now with free IPA 4800 in Fedora you can run sample file server on free IPA client as official server and there's a single command utility that configures everything for you so all you need to do is to add your shares into the config file it generates it enrolls machine well you have an IPA enrolled machine then you run this tool it doesn't need any credentials it's enough to have the machine credentials itself so you run it on the route it creates all the needed things registers configures SSSD configures the Samba in smb.conf creates all the needed things and it works in my test it also works for users from trusted active directory domains so you can actually have Windows users coming for shares on the Linux side and there's one thing that apparently works in Fedora is that if you take a Windows machine and try to log in into it with IPA user it will work and it will be able to access the shares from IPA but only if it's not a domain controller on the Windows side because there you have to have some additional things which are not in Fedora yet but we are working on them then there's an interesting thing the hidden an advertised replicas which was a request from Red Hat IT which run free IPA on the scale they want to have systems that they remove from customer use but for example run back up on them so now free IPA has a mechanism to mark a replica as hidden if client is enrolled directly into that replica it will continue using it but if it's doing the discovery it will skip and we have added a lot of changes in the certificate management one interesting one is that we have now is ability to supply a snippet for initial dock tag configuration so you can say that I want to have different key length for the CA certificate yeah I want to have it stored in HSM that doesn't work yet but we are working right now on this so hopefully by next Fedora 32 actually release we might even get that and we also have support for IP addresses in the certificate so that should help guys poor guys who run home routers and want to issue certificates for them without using names DNS names in those certificates or open stack that wants to use it and we have health check utility that verifies I think several hundred different conditions that your server might be under and there's a work on going to have recommendations on it how to fix things yes so it will detect that there are conflicts but on a single master you run on all of them and collects all together but it for example identifies broken DNS in trust and that kind of thing that users are not resolvable and this kind yeah it tries to help to point out where the problem is and finally in Fedora we have Ansible free IPA which is a set of roles and tools that provision client master and replica and we started working also on the real management parts like add-in users groups and all this stuff seems to be quite popular now so people report on it and use it so on Samba site Samba 410 it has full Python 3 support so no Python 2 anymore so we got through it took five years yeah yes json-based logging so you get information about users that were logged in into your domain controller and you can then transform it into some other things and do central logging with it there's a lot of improvements on GPO support on Samba AD site and yeah offline yes we have in two things one of them in SSSD if you enroll into Active Directory directly it supports GPOS the other thing is that if you you deploy Samba AD it uses GPOS there so it can provide for Windows clients GPOS and you can manage them then there are some improvements on the LDAP server in Samba like parallel operations and page control added because Samba has its own embedded LDAP server so work in progress very promising is the unification of crypto replacing the homegrown Samba crypto implementations with no TLS that gives us aside from some compliance with FIPS and and so on attempting to get FIPS we also get performance improvements because of optimized crypto and in some cases we get an SMB3 encrypted stream faster than SMB3 checks zoomed stream which you did not encrypt you just checked zoom and the faster means two three ten times so we get really good performance and the same is being added right now to Sieve's client in in the kernel so between the Linux client and Linux server we get boost of performance and the other the other part is that there is a composition in the Linux client in the sieves.co added the operations are kind of combined in a single request that has multiple operations in it and that reduces number of file operations you need to send it so you you actually get something like 30-40% boost over NFS compared to NFS so once we complete the other part the last part the POSIX extensions for SMB3 protocol we will have an ideal replacement for NFS for home chairs for example high performance access so we are very very close to having all this stuff and I'm done thank you