 All right, so my name is Seko Brozek, I work for Red Hat. I mostly work on the SSD. There's my Twitter account. If there's no time for questions, you can just tweet me or meet me somewhere here in the hallways. I also put the little scripts I will be showing in the demos on my GitHub. And there's also a copy of the presentation in PDF, so we can grab it read later. This talk is about SSD, but it's not about the things that most people use when they use SSD, which is normally login as a user from Active Directory, LDAP, whatever. But it's about the parts that are not as well known. They are very often in development, not so stable. There is few topics. I will be going quite fast, not into many details. So if you are interested in any details that I'm not going to cover, feel free to catch me later. In particular, I will be talking about the APIs that SSD has. How can you talk to SSD from a program? It might be a good idea to let SSD handle local users from its e-password and group. Why and how SSD handles smart card logins? Because as we heard, pump-pk-cs-level-1 is dead. And the last topic is how can SSD manage Kerberos I'm not sure even if I'm going to go to the last topic because we'll see how we are with time. Right, first topic is SSD APIs. How can you talk to SSD from your program? The first thing is why you might even want to do something like this. And the primary use case is that there are applications, quite a few of them actually, that implement some sort of an LDAP driver or LDAP connector. Typically, this is because the application wants to let people log in with their LDAP credentials or whatnot and use data about these users. For example, the application might want to know which groups is the user member of to make some policy decisions, or the application might want to, for example, know the email address of the user to display it in the UI. This sounds easy, and it's easy to do it in some sort of trivial way. You just bind to LDAP, read the data, display it. But doing it correctly, there are a lot of details that you might not even know about the problems in the first place, much less to solve them. And some of the problems are stuff like which server should I talk to? We already heard about activator sites. If one LDAP server goes down, what does the application do? It should, in the ideal case, connect to a different server. What if the LDAP server is, what if the network is completely down? In that case, you might want to keep serving some data from your cache. In general, SSSD is a kind of expert in how to talk to LDAP. And it might be a good idea instead of doing your own thing in your application, just let SSD do its thing and provide you with an API that your application uses. Of course, it's already possible to talk to SSSD somehow. If you just have SSD to log in as an LDAP user, you can already run get-and-pass-for-d your username. It will display the user. This is already talking to SSSD, right? But it's not a kind of publicly available API. In the case that I said in the get-and-pass-for-d user, you talk to an interface that is called NSS, Name Service Switch, which inside calls the libc function getPWname, which then talks to SSSD through a function that SSSD provides, the NSS SSS getPWname. This is not trivial to call directly from the application. It's somehow possible you can DL open the library and DL send the functions and then call them. But that's really awkward. And it's not really flexible. So if you call this interface the getPWname, you only get a fixed set of attributes. You only get the username, home, there, shell, and so on. But I had the example with the email address. It's not there. So the way you can talk to SSSD directly already in the versions that are released is currently through a D-Bus-based API. We chose D-Bus for several years because of some advantages we saw that primarily there are many language bindings for D-Bus. I'm going to show some little demo that will use Python. D-Bus is type safe. D-Bus has types. So if you request a number, for example, the UID number, you get back a number. You don't get back a string that probably contains numbers, but maybe not. D-Bus has some advanced features that might be useful like you can get notifications, which are called signals in D-Bus lingo. This can be useful if you have a UI. And you don't want to continuously pull SSSD for if anything changed. You just get a signal when the thing changed. You can redraw your UI. As we found out through bug reports and through feedback, not everybody likes D-Bus. There are disadvantages. The primary one we saw was that it requires a system bus, the message bus running, the broker. That's, for example, not available in containers. Some language bindings were not really great, and we only found out that people started trying to use the D-Bus API. I think the Java bindings were even unmaintained at the time where one project started to use them. But nonetheless, this is used by several projects already. Many of them are head-based because we talk to the people directly, like Manage IQ, Keycloak, which I think they had a presentation earlier today, the Apache module, model-cap identity. Because of the problems we encountered with D-Bus API, we thought maybe it might be appealing to offer some other kind of API, but we never did it. We thought about REST because that's what many people are used to. It's easy to use. And we even talked about offering the LDAP protocol over LDAP. But this is really just discussions. This talk is about, especially, doing more than LDAP client, but I'm not sure if we want to do LDAP server. A little demo time. So I will be showing two scripts. One is written in raw Python LDAP, which is maintaining. And the other is doing exactly the same thing except it's talking to the D-Bus API of SSD. So let's see here. Here. So I'm going to run the first script, which is using the raw Python bindings. And what the script does is that it finds the user who I put on the command line as an argument, and it would print their email and their group membership. That's right. So boom. We know that the FOSDEM 2018 user has the email of FOSDEM 2018 and FOSDEM, and he's a member of three groups. So I'm going to really quickly show the script. It's not ideal on this small screen, but you can see that even in the script I have the user who I authenticate as, and I have the password of the user in the script, which is really not ideal. And there is a lot of parsing and low level LDAP stuff that probably as an application developer you don't want to know. So the script is about 70 lines long. So now I'm going to run the script that does the same thing, except it's using the SSD API. It returns the same data, except I think the groups are in a different order, which is kind of expected with LDAP. So if you look at the D-Bus example, it's about half the size. And more importantly, I think the language is kind of more natural. So requesting the email attribute is calling a method that is called getUserAttribute. Requesting the groups is calling a method that is called getUserGroups. You really don't get down to the raw LDAP level. Right, that's all about the first part about the D-Bus APIs, details. If you want more details, catch me later. The next section is about managing local users and how the SSD relates to users and groups in the HD files. Again, why do you might want to do this at all? One of the reasons is faster access to the users that are stored in the files on the disk. Traditionally, you could use NSCD, which is a name service caching demon that comes from Lipsy. But what we found out through the way of bug reports from users is that people who used NSCD, because they wanted to have faster access to users on the disk, but they also used SSD because of some remote users in LDAP, they were complaining about these two caching mechanisms not really playing well together. NSCD sits kind of in front of SSD and the caching was really unpredictable. I showed the APIs previously. Another reason why we might want to use SSD for local users is that the API that I showed might actually be used also for local users. SSD is serving them in the first place. And then we could have a nice API for user, not management, but reading user information on Linux for all kinds of users, be it remote or local. SSD uses a database to store the user data. Database comes from somebody. It's called LDB. And in the database, it's possible to store additional attributes. So we were thinking about storing stuff like maybe the keyboard layout or your preferred language in the database. And then this information could be through the D-Bus API offered to whoever is interested in them, desktop environment, for example. And the last reason is smart cards for local users. SSD, in order to authenticate some users, it must first know about them. Therefore, SSD must be serving the local users. I'll have a separate section on the smart cards later with the demo. So this is already underway in Fedora. If you're running Fedora, since the Fedora 26 version, SSD is running by default on all installations. And it is mirroring the contents of its e-password and its e-group. We have an iNotify watch that updates the database and drops all caches if anything changes. The way the caching works is that if there is a request coming from application like LSL wants to convert the number to name, the request goes first to SSD. The first request goes to the demon, which is a little slow. But then the demon, as part of the reply, puts the reply into memory mapped cache. So any subsequent request would not even hit the demon. It would just be in the NSS module, which is loaded into the memory space of whoever is calling it. As I said, the reason we're doing this is that this way caching for local users interrupts easily with SSD that might also be running on the same machine and serving users to mold up. Disadvantage, SSD is bigger, fatter, more complex than an SSD. If you just care about local users and nothing else, you might just be happy with an SSD. There's nothing wrong about it. And the other disadvantage is that this actually, in order for this to work, we had to revert the order of the NSSWITCH Conf modules. So traditionally, files was always first in the NSSWITCH Conf database. In Fedora, since 26, we reversed it, so SSD comes first. This does not require SSD to be running at all. It doesn't require a SSD to be installed. If the SSS module finds out that the daemon is not running, it would just gracefully fall back. So if SSD is not running and you want to access root, it would work. So this is very much work in progress. The next thing we want to do is improve the smart card integration. You'll see later why it is currently very user unfriendly. We want to make it possible to extend the local database with more attributes like the keyboard layout or whatnot that I talked about. Currently, it is somewhat possible we have a command line tool that is called SSOverride. But currently, it only allows you to add public certificate for the smart card authentication. But right now, there's no general API or command line tool. There are some people who wanted us to extend the D-Bus API to allow write modifications. This, I think, came from cockpit. So if you had some kind of create user method on the D-Bus API, it would call to lip user and create the user locally. And as I said, we would like to have a unified API for providing information about local and remote users. So what we're thinking about is to merge with this project from Freedestops account service and implement or offer the same API from SSD. Hopefully, this would happen this year. But this kind of project that we've been working on for a long time, we never have the time. That was local users. The next section is smart cards with SSD, primarily for local users. So I'm not a smart card expert by any stretch. The previous Jacob who had a talk right before me is just a link to his talk. He also gave a similar talk on a different conference last week or two weeks ago. The second link is to recording of his talk if you haven't seen it again. I even didn't write this part of SSD. I really literally learned it as I was preparing this talk. But I can illustrate how it works. So traditionally, if you wanted to use smart cards, and login as a local user with smart card, you would use PAM-PK-CLS-11. It has a lot of features. It was super stable, but there's also no development. I think the primary developer abandoned the project and nobody else stepped up. It even doesn't build again recent OpenSL versions. And for that reason, it was dropped for Fedora. So currently, in Fedora, there's no way to authenticate it, I think, as a local user with a smart card. Kind of in parallel to that, SSD was gaining support for smart card users, which wasn't at first meant for local users. It was meant for remote users. The primary use case was free IPA and trust with Active Directory. The way it works on a very, very high level is that the certificate is stored in the directory. If a smart card is inserted, then the SSD runs a search against and tries to find, based on the certificate, tries to match the user. And then either does a PK in it, so Kerberos authentication with a key, or does kind of a local authentication and just on that machine. It's functional for local users. I'm going to show you. I would not call it usable by anyway, because it requires a bit of manual configuration. So first, SSD must be serving the local users, which is already default in Fedora, but I think not elsewhere. The smart card doesn't have to be a real smart card. It can be anything that influence the PK-CLS7 interface. I'm going to use a UbiKey for my demo. You need to load the certificate into the user database with the SSS override tool. I'm not going to show this, because I don't have the time. And most importantly, the authentication, even though it's a local user, must be handled by PUMSS, not PUMUNIX. We haven't really thought through how to change the PUM stack in Fedora to make this possible in kind of generic way. But I'll show a demo. This is something we will be working for Fedora 29 to make it more user-friendly. And everything should work by default, but we are not there yet. Right, demo time. So this is a VM. I wrote a simple program that just runs a PUM conversation and authenticates a user. So first, I'm going to authenticate a user called test. So hopefully, I can write the password down. So the user was authenticated. If we look at the journal, the first line here is from audit. And it says that PUM authentication was successful. And the grantor, the module that allowed the authentication, was PUMUNIX. So this is what you do normally, authentication against its C-shadow. So now I'm going to run second authentication. And as you can see, this time, I was prompted for not for password, but for PIN. And I don't remember the PIN. I have it here in the... This is not secure. Don't do this, except if you are demoing something on FOSDN. Good that you mentioned that. 47231. And I was authenticated. So PUM authenticate return success. And I'm going to show the journal again. And there is user auth. So user was authenticated. And it was authenticated by PUMSS. And the user I was using was called local smart card test. And if we look into at C-shadow, you can see that there's something missing, right? There is no hashed password for this user. So I couldn't have authenticated against at C-shadow with a password, even if I wanted to. If we look at the first user I was using, right? There's a password. OK, how much time do I have? A few more? I haven't been here in my 12 minutes. Good. So I'm going to run the last section, which is why it might make sense to let SSD handle our Kerberos tickets. So I will not go into details on Kerberos. But for the purpose of this, let's just say that if you authenticate against the Kerberos KDC, you get back a ticket. It's a blob of data. And this blob of data must be stored somewhere. That somewhere is called conventional cache in Kerberos speak. There's several options. The easiest one is that you can just write that blob into a file. You can write it into directory that can subsequently contain multiple traditional caches for multiple Kerberos principles. You can put it into Kerberos keyring, which is the default on Fedora and in rel. Or you can use KCM. This is what this part of talk is about. I should mention that KCM is not my idea. It's not our team's idea. It comes from the handle Kerberos distribution. I looked at the Git logs as I was preparing the talk. And the first commits for the KCM part of Orhandal are more than 10 years old. The idea behind the KCM credential cache is that it's not, unlike the files, which you just put a blob right into the file, right? That's it. Here, the credentials are handled by a demon. And the advantage of this is that the demon, because it's kind of active component, it can then do things with the credential cache that, I'll get to the deal later, but there's more functionality you get out of this. If you will read some documentation about KCM, the terminology is that the KCM demon that is managing the credentials is called a KCM server. The tools that are the whoever is talking to the KCM demon, typically it's some command line tools like KNIT or an application that uses Lipker 5 is referred to as a client. Even though there is a server and client, there's no network communication. It's all local communication over a unique socket. At the moment, Heimdall implements the KCM server. MIT doesn't. MIT only implements the client part. So MIT implements the support for this KCM protocol over the unique socket for the tools, but not the server. You can mix and match them. And that's how I was testing our KCM implementation. I was using a KCM server from Heimdall against KNIT from MIT, and it worked. I'm not sure if it worked in some production environment, but it worked for me. So why would you want to do this? As I said, the KCM demon is stateful, so it can do things like renew the Kerberos credentials. Kerberos, the credentials you get back in the C-Cache have a limited time, but sometimes the time is not enough. For example, if you're running some batch job on some cluster, that job might take longer than the lifetime of the credentials. What a demon like KCM can do is renew the credentials for some time. Again, notifications. That's one of the things we want to do. If the credentials are acquired, about to expire, have been renewed, you can send a notification. We would like to use the D-Bus API that I was talking about earlier. And I think GNOME was already interested in using these notifications for this environment. You can also do things like if the user who was using these credentials logs out, you can get a notification from LoginD and wipe the credentials out. The credentials are also not written to the disk, which might be beneficial for sharing the credentials between two processes, because if they are on disk, they need to be in a predictable location, and this might not be super secure. What we also wanted to do is this way make Kerberos usable for containerized environments. The way we thought this would work out is that because the KCM server has a unique socket, then you can run a KCM server in a container. And then if you have other containers that are maybe running some applications, those containers can bind mount the unique socket of the KCM server. And this way you can share the credentials only between selected containers, unlike, for example, the kernel keyring, which is shared between everyone. What we found out is that nobody is really interested in using Kerberos with containers for various reasons. I think mostly. Really? Awesome. We can't talk out. OK. So one person only is interested in using Kerberos with containers. Cool. So the implementation of the KCM server in SSD is called SSD-KCM. It reuses a lot of SSD code, which is one of the reasons it lives in the SSD code base. But it doesn't need the rest of SSD. You don't need to have SSD serve any users, even local or remote. You can just system control enable the KCM socket, and then it works. We're also planning to use some private APIs of SSD. Again, another reason why this is a project inside the SSD umbrella for features like notifications and so on. KCM is the default Kerberos Cache since Fedora 27. There are bugs, though. Some are still open. I didn't really have that much time to look into them. But I will in the next couple of weeks. The current version of what we have in Fedora is just equivalent to the other Cache types. We don't really have implemented the features like the renewals, notifications, and this. But we are going to, it's on my to-do list for Fedora 29. If you want to learn more about KCM, there is a design page on the SSD Wiki. And there is a link to the MIT Kerberos Wiki page which describes much more details about, for example, the protocol that libraries use. That's all. Thank you. Do you have any questions? I would like to ask about the caching of SSD. How often is generated that in case, for instance, the server goes down and the bros management, how long will it be valid? The cache? Yes. As long as SSD stays offline. So if SSD is offline, it never removes anything from the cache. It will just serve cache data. And if you log in with the cached credentials, in that case, you don't get a ticket. So it's just local authentication. And then, for example, if, repeat the question, how long is SSD cache valid? So one interesting case there might be that SSD is offline. At that time, the administrator removes, for example, removes some of your group memberships. The next authentication that would happen online would refresh the cache data. But as long as you can't reach the server, the cache just stays the same. How often is generated? It's configurable. And there are two answers to this one is that for just if you run something like get-in-pass-ready or ID from the command line, there is, I think, 45 minutes or 30 minutes, something like this. But if you log in, as Volcker already said before, log in is the only time where you can get precise group information. At that time, we never use the cache. We always process the reply from the server. I had a question that sort of relates to that. In terms of debugging SSD and knowing where a problem is and knowing when a cache has expired or when SSD probably has issues contacting the server, what are the typical ways of debugging SSD? Enable debug locks. It can look there. So what are the typical ways of debugging problems? In order to learn that SSD, for example, is offline because it cannot contact the remote server. It's just a random example, but it could be like any other problem. So we just talked about this with Fabiano over lunch, that our debugging is very suboptimal. And currently, you need to enable the debug level in all the SSD conf sections and just look there. We have on the SSD, we have a troubleshooting guide, which might help you. I'm not sure if you saw that. And what we did in the last two or three releases, we have a command light tool that is called SSS Control. And that tool has subcommands. One of that subcommands is, I think, domain status or domain info that would show you domain is offline. Another command there is something like user status or user show. I don't remember the commands precisely. And this will show you, for example, when was the last time that the user was refreshed from Alda. So these two maybe can be used. Do SSD support transitive trust relationship? No, only WinBank can do this. Because, repeat the question. Does SSD support transitive trust? No. Somewhere in the backlog, and maybe it is something that Andreas might work on someday, no? But if I understand it correctly, and maybe Falkirk can help me. But I think, so SSD, if it's. We need metal with my integration to SSSC. Yes, yes. We need these phone calls again. Yes. So if SSD joins to an active directory domain, it has credentials only from that domain, a key tab that represents the machine account. And time is up, I will just finish this one answer. And that key tab cannot be used to authenticate or to even to authenticate you against directory servers in another forest. I think WinBank does this by the private, the Windows RPC calls. We don't. We just ask all the mechanisms for authentication. Yes, but we do that over a network. Right. OK. And SSD only uses LAP and Kerberos. And that's not possible. Yeah. There were some phone calls we had maybe one, two years ago where we talked about splitting some parts of WinBind and making them reusable. But nobody did the work. We need to revise that. OK. Use it. Do you want to start with names? Sorry, guys. Time is up. Thank you very much for listening.