 All right, hi everyone, my name is Rick Elrod and I am on the community platform engineering team at Red Hat I work primarily with the Fedora infrastructure and My latest project has been triple a a fan occasion authorization and accounting or auditing Some people use a different final either But yeah, I want to talk about how we do triple a currently and some ideas we have and are currently exploring for Doing things differently going forward. So a couple of statistics that's about the Fedora infrastructure In the database right now for our current account system. We have 60,000 accounts that are labeled active In practice probably about half of those are actively used We have around 500 give or take servers, this is including VMs builders Vercos cloud instances all over the place These are scattered across Multitude of data centers. We have we have space in some donated Racks, we have some cloud instances that we use throughout different regions We have two primary data centers one in Phoenix and then one is on the East Coast We have over 30 web applications and a lot of these high-end to the off system. So we have the wiki we have build front-ends All sorts of various web applications that we use And this is our current account system, right? So if you have interacted with Fedora at all, you've seen a page like this this is what we've been using since around 2007 and This is the system that we're currently trying to move away from and I'll explain why But first off. So what what is fast? What is the Fedora account system? What does it do? The primary thing it does is it allows for self-service user and group management So I guess to elaborate on that a little bit a User who wants to join our community. They can go to the Fedora account system They can sign up they can look at groups. They can request to join a group. They can be sponsored into a group Without without a fast administrator having to kind of handle that process. It's self-service The next thing it does is it's a custom authentication and authorization solution so all of the code in fast is Currently a custom solution It kind of leverages the turbogears one authentication system if you've used turbogears at all that's Turbogears one is quite an old framework, which is one of the reasons we need to get off of it Originally it had a baked-in open ID provider and then later that moved to a separate project called the epsilon Which we use primarily for open ID connect now it is So there's a server component and that is the web application for the users to do self-service tasks It also provides an API so that the client component that I'll talk about can call into it and do various things And it tracks all this information So it allows for auditing when a user joins a group It'll it'll track that when a user changes their information It'll track that just so we can go back and see who did what when it also allows for users to apply and be sponsored Into various groups and then the client component Again, there's an API that the the server component provides the client component will call into that API And the way that we use it in the infrastructure is that we run this on a cron every hour and It will it will call into the API It'll get the user information and group information and it will write out Custom NSS database files with this information. It'll put them into place It will add the users public SSH keys to their home directory. So it kind of does all this manual work right now It's automated, but it's still the process itself is a fairly manual process So why was fast made originally well one of the ideas I wasn't around at the time I joined the community later. One of the ideas that I heard is just the LDAP is scary, right? So there weren't too many people working on this at the time that it was created originally They were throwing around ideas. They were playing with LDAP, but ultimately LDAP was just scary There were some other issues alongside that like again, our infrastructure is distributed. So Being able to do replication and have things stay in sync. It was just really hard to do at that time Things have gotten easier and I'll talk about that later, but LDAP was just scary at the time, right? And I think there might have also been a little bit of non-empted years in it, right? So writing code is fun. People wanted to write code people wanted to play The other thing is there were some custom use cases. There are things that fast does that just weren't possible with other solutions at the time So certificate generation some of the two-factor stuff that we kind of baked in later You know some of the just some of the various information that we store and what we do with it Just weren't possible with other solutions out of the box previously So we we wrote fast to there was a fast one before that again. This was before my time I don't know a whole lot about it, but So we wrote fast to and we've been using it since around 2007 2008 Over the time there's been some problems that have come up with one of them is that the server side component currently only runs on rel6 and Rel6 is that to end of life in November of this year So that's a problem You know authentication is kind of important and we want to run it on a supported platform Again, the current fast is all custom code. So if you know anything about Security encryption authentication the number one rule is you don't write it yourself like you use things that have been tested You use things that you know are known to work And all of all of the authentication code in fast is is custom built Not a good thing Again, it's on turbo gears one which has been an end of life for probably six years now It's really old and It's never had a true full security on it. So again our entire infrastructure is depending on this platform There have been security bugs, which I'll show a couple of them It's never had a full security So as a result there's been a number of security issues a number of big security issues This one was kind of funny so I mentioned that we have a fast client that runs on the the systems and our infrastructure it will set the SSH public key for each user in their home directory. It'll add it to authorized keys So a user on an account could sim link their authorized keys file to any other file on the system The client would write to that sim link and the file that it's linking to is what would get updated because the client is updating NSS databases and you know, it's It needs to write to certain directories. It's running as root as a result Any file on the system could be updated With whatever the user uploaded as their SSH key. That's a problem This was one that I found when I was kind of starting to get involved in this. This was back in 2013 If a if a user had applied to a group that they hadn't been approved yet That particular API endpoint wasn't sanitized So even if they have privacy set on their account all of their information was available publicly through the API if you if you Ask for the JSON response from the API This is one that for those of you who know Patrick. This is one that he found a couple years ago at a flock This is basically saying if you set if you set the X5 verifier to success And you hit and you set X client CN to a username Any validation that you are that user is completely gone out the window you can spoof to be any user That's a problem these all the way by the way all three of these have been fixed so sad and frustrating so Some requirements for a new system going forward, right? What do we need? We need something secure We need something that has a good proven track record something that's been used in production by companies other projects We need something that's well supported well maintained And we still need that self-service to put right we still want users to be able to change their own information We still want people to be able to sponsor people into groups You know we want the community to be able to take control of their data rather than having admins do all this for So choices so our first attempt at fixing this was something called fast 3 this was 2012 2013 era by Xavier Lamian and Well Linus Turbles has quotes as subversion used to say CVS done right and with that slow bit of nowhere You can go there's no way to do CVS, right? Well, there's no way to do fast, right? with the problems that I mentioned before like You can't get away from that. We want to do something completely different. So choices So first thing we looked at was an idea that Patrick originally proposed to me and it was Amazon Cognito and Their slogan if you go to their landing site is simple and secure user sign up sign in and access control And that's well and good, but it literally does that and nothing more That's the one thing that it does it follows the UNIX philosophy. It does one thing and it does it well So it doesn't really lend itself to Community authentication it doesn't lend itself to you know having the custom fields that we need and Storing all this extra metadata that we need to attach to a user It is extensible with AWS Lambda functions but the net result is that we're going to be doing the same thing we're trying to get away from we're going to be writing all this custom code and You know, we're still gonna have to write most of the components ourselves some other things it lacked a couple of API endpoints that we were going to need very heavily and So things like tell me all the users that are attached to a group that was that API endpoint is locked down to admins only Retrieving basic information about a user or a group their their metadata the extra fields they have attached to them And again, these are fixable. You can write AWS Lambda functions and do it But these these API endpoints were not out of the box on cugging it's not set up to provide SSH or system access and Again, there are ways around it. So we could have used more AWS Lambda functions. We could use SSH certificates Where, you know, when a user wants to you SSH into a system They they use a custom SSH client It tells them go to this URL in your browser authenticated against Cognito and A SSH certificate will be generated for you and you'll use that to get into the system So it is possible to work around this in a mostly automated way But it's not set up for that out of the box Also, it's just not open source and that's something that the community is quite vocal about So it's it's worth noting And some people within AWS suggested, you know combining Cognito with another service like free IPA or something else that will store extra metadata about the users that Cognito isn't set out to do out of the box and I Talk to people and we considered this But kind of the consensus was if we're going to be deploying something else anyway Let's just go all in on that solution with the manpower that we have right now And the skill set that we have if we're going to go with something else anyway alongside of Cognito Just go all in on the other solution. So choices. So then we looked at free IPA and Spoiler, this is kind of what we ended up settling So why free IPA? Well Health app in general is less scary than it used to be. There's there's a lot of documentation out there There are really good communities. The free IPA team themselves is very helpful. So it's just over time. It's gotten less scary IPA is actively developed. It's actively maintained is actually audited. It's used in production. It's a red hat product It's well supported. There's companies using it. There's communities using it so Getting help maintaining it getting help with deploying it is not really a problem like there's people using it and Anything that we hit has either already been hit by someone who is using it and they're usually willing to help or There's enough information out there that we can Work with it and figure out what's going on The IPA team has been extremely responsive and supportive. They've really helped us You know, we've had meetings with them internally and you know They've just been extremely helpful and willing to do what it takes to to help us get get this ployed Which is which is awesome and there's an amazing API. So I Don't know if any of you have ever used the the free IPA API But there's endpoints for pretty much anything and everything you could want to do And it's just it's really awesome. It's very well documented if you go into the web portal You know it gives you examples of all the arguments what they take what they do. It's it's really cool And it's entirely false. It's entirely open source It's customizable. So For fields that we need that it doesn't currently have we can write a simple LDAP schema extension to add them And again, this is something the IPA team has helped us with extensively There's built-in customization for access so it has role-based access control out of the box So you can say, you know this user or this group or this set of systems can do this But not these other things like it's very flexible, which is really cool It works out of the box with epsilon and key cloaks So our open ID connect and open ID systems will still work our web apps We'll all work with it pretty much out of the box with a couple of configuration changes Which is another thing that's it's Useful right we use that everywhere and replication works. So I said we have a kind of a distributed infrastructure We have you know six or seven different data centers that we have servers in Replication works. It's well tested nowadays There's again, there's companies and communities using replication You know Yeah, it works There are some downsides right so LDAP schema extensions if you're not familiar with that language can be hard to learn And this is again something the IPA team has helped us with But if you if you threw me at that, I would not have gotten it right the first time So they can be challenging But the other thing is those should be rare, right? So we have a set of fields that we need to add once and then you know adding more is Generally not something we're gonna have to do very often It's not set up for self-service out of the box And I'm gonna talk about the the portal project that we're working on to kind of fill in that gap But free IPA itself It's oftentimes it's deployed in kind of a corporate environment where you have one You know set of global admins that will add users or remove users or disable users as needed As people come into the company or leave the company or need access It's not set up to be a self-service project out of the box And when you're first setting it up it can be you know, there's a lot of new terminology There's a lot of a lot of things to consider when you're first setting it up And if you haven't done that before it can be a little bit complex And again because of that new terminology it can be somewhat challenging to learn so Internally, there's a really good course That that was a red hat can take and learn about it if you're not internal and you're in the community You know, how do we convey? You know, this is this is our technology set This is if you need to do something that you can't do from the portal that we're writing. This is how you do it You know, so that's something that we we need to take into consideration Yeah, it can be challenging to learn so for the self-service component we're working on on a user portal and A couple of goals right so we want it to do most of the things that the fast front end will let you do now So register for an account change your information, etc. Etc. But we want to do those things well and secure So yeah allow users to register modifying their information allow group sponsors to sponsor other people into groups That they maintain Look up information about users and groups, you know, it's my friend in this group. Maybe I want to join that group, too Or show me information about this user I want to see if they're cool and I want to be their friend, whatever reset forgotten and expired passwords And then a couple of non goals right so account deletion This is something that we we handle in infrastructure in a separate process, right? So with GDPR, this is something that we have to be able to do But we have a whole separate process for that and so this isn't a portal goal This is you know, this is something that we have to update in our infrastructure scripts as a separate thing Another non goal is allowing group sponsors to edit group metadata so You know if a if a sponsor of a group If someone is a sponsor of a group all they can do is add or remove users from the group So they can't change the group name. They can't change the group description That's something that happens pretty rarely anyway And it's just not something that we care about being allowed to do from the portal and then administrative actions. So For admins for you know people in the core infrastructure team Who want to you know change information about other users or whatever? This isn't really something that we are going to have in the portal to start if if somebody absolutely needs to do that they can use the CLI tooling from free IPA On the command line and do that It's just not a goal of the self-service portal. So I'm going to talk about the self-service portal the portal that we're developing now It's called secure at us and I'm going to hopefully give a quick demo of it if the Wi-Fi is nice Live demos never fail and nothing could possibly go wrong so So this is all subjects of change. This is kind of very experimental and just something that we're working on right now but So this is secure at us, so There's a registration form a user can obviously register for an account I can log in if I have an account and So right now it's the the image error. It's just pulling from grab a car But all of this operation over here is anal that It's just I'll talk about how the login flow works and we do some interesting stuff there, but all of this Group list that I'm in again all coming from LDAP. I can narrow in on a group and See who all is in that group Hopefully if the Wi-Fi is nice Live demos never fail So I can see the sponsor of the group. I can see the the users that are in the group I can find out more information about those users You can theoretically search for users and groups up here, although if that's finicky sometimes it's something I need to figure out And I can edit my profile and update my information And so this is kind of a proof-of-concept portal that I I developed over the course of a week or two just when we were kind of first having these discussions about looking at using free IPA and You know We knew we were gonna need a self-service portal So this is something that I threw together at least to start a discussion with the team Internally and and you know figure out what all do we need to do and and does this get us close? Is this what we're looking for? so Again total work in progress a lot of it is going to change I Want to talk specifically about the log in flow. So we do some interesting things there So when a user log logs in the portal takes their login information it passes it to the free IPA API Free IPA will validate their credentials and it was it will start a session It'll start a normal session. So it will send back a cookie with the response if if authentication is successful and If it's valid the portal the proof-of-concept that I just showed will take that session Cookie it will encrypt it and then it will store it in the user's browser The reason we do that is so that every request that hits the IPA API Will happen on behalf of the user. So we're not using some, you know glorified admin user to to update people's information or whatnot through the portal Everything is acting on behalf of the user through the portal. So the portal is basically proxying These these requests on the user's behalf as them to free IPA and So we do it this way because you know, we want to use the principle of of least access, right? So we don't want to use an admin user for things that we don't need to use an admin user We only use escalated glorified accounts when absolutely necessary so for things like creating a new account Obviously you need to be an admin to do that Or at least have permissions to do that Resetting passwords like an expired password. You need a permission to do that And so even the escalated accounts are not true global admin accounts, right? So you can use the the role-based access control in free IPA to say, you know These specific permissions are what this escalated account needs to do and another advantage of doing it this way Is all the logging and auditing again? It correctly shows that it's be that these actions are being done by the user not some admin user For for the user they're being done as the user themselves So the technology stack of this portal We're using Python and flask. It's just because what the team working on this is familiar with Kind of over the past couple years. We've in in Fedora infrastructure We've kind of settled on on this stack for most of our custom apps and so this is what we're using for the portal One thing we're trying to do is avoid using external databases if we can We're just trying to sort things in LDAP because we're already using the LDAP for most of the information and so far we've been successful and We're using something called Python free IPA as the kind of proxy layer the the API client that the portal uses This is a third-party module. It's written by the open node group in Estonia. It's an open source library So development is ongoing like I said the proof of concept that I demoed That took a week or two to do it was kind of a free time thing just to Just to get it done and show it to the team and say, you know, hey, is this is something like what we're looking for and now the team is adding features to it and tests and Making it look pretty because I'm not a designer. I don't know if you know that but You know, we have actual designers who are making it look pretty But we're definitely receptive to help right so this is an open source project You know, it's something that everybody in fedora is going to end up using When we switch to it and so I'm definitely receptive, you know, the whole team is receptive to help if anybody's interested Towards the end I have information on how to find out what we're doing and get involved So there's a couple other pieces to consider right so we already are using This custom for our account system fast to So how do we actually do the migration from the old system to the new system and One thing is passwords for a while now have already been being synced over we have an IPA instance in production and We have a little bit of custom fast code that when a user changes their password in fast It will also sync it over to free IPA So the reason we do that is we can't you know, even if we write a script to migrate all of fast over to free IPA Passwords are hashed differently between fast and free IPA. So we can't just sync over all the passwords in a once-and-done transaction So we're doing it as the user changes their passwords. So, you know, it's kind of an incremental shift over to To free IPA having passwords for the users I did write a script to pour over the rest of the user data as well. So things like which groups they're in You know their name other information GPG keys and so on So we do have that We need to announce these changes and prepare people for them because it's gonna be quite a change again A lot of people rely on fast right now. So announcing the changes getting the word out Once we get to that point is something that we will need to consider Updating applications that currently rely on fast. So a lot of applications Currently tie into the API that fast provides and we need to update those applications to tie into the new API and then this one is kind of Something that we're just starting to look at and it's still kind of up in the air But collaboration with the sent us And I don't have a whole lot to announce there yet But there there is some collaboration that we're looking at going on. So for the applications that rely on fast There is in the new system a read-only JSON API That should provide most of the information that those applications currently rely on the fast API for And this is something that the free IPA team Helped us out a lot and and wrote this thing for us and It basically it provides a limited view of some of the public data in in free IPA so that applications can call into it and In and do simple checks like is this men is this group? Sorry, is this user a member of this particular group? Or you know, I know this person's username. Tell me more about them. Tell me their real name There are things still up for discussion. So deployment strategy. How and when do we deploy? Not only initially but going forward as we as we make changes to the system Again sent us collaboration kind of still up in the air Figuring out how fedora specific things should be so We want the greater open source community to be able to set up a similar system like this You know, I know for a fact that fedora is not the only Project that has these kinds of problems that we're trying to solve So I think other communities could could really make use of this So it's a matter of kind of abstracting out the fedora specific stuff like the schema changes and and how much the self-service portal relies on those schema changes Basically figuring out how to extract that so that other communities can kind of use the same toolset and set up something similar And then of course, you know, everybody has features that they want within the community And so figuring out What all the community wants that we didn't think of and how to prioritize those and when to spend time implementing those things a Couple of acknowledgments of course the free IPA team. They've been super super helpful They've added added features to IPA core When I first started looking at this project, there there was no concept of group sponsors in free IPA So you couldn't say, you know, you couldn't say these particular people are sponsors of the group and they can add You know, they they can add people to the group That was only something that core admins of free IPA could do so that that kind of delegation wasn't there originally They've added that for us, which is awesome They wrote the initial custom schema for us so adding fields that we need to to the LDAP schema That weren't there out of the box And again, they wrote the initial read only json api that the applications that we have are going to use Which is awesome super helpful Patrick for those of you who don't know Amazing knowledge of triple a and security processes He's the one that came up with the original idea for how the portal does that kind of handshake process that I talked about to exchange out the the encrypted Cookie and session token He wrote epsilon which we use for open ID connect And again just his his knowledge of triple a and his experience just in general has been super super helpful so Definitely acknowledging him And the original fast contributors like, you know, it has problems, but also it's gotten us this far We've used this thing for like 13 years now. So Yeah, it's had problems, but It got us to where we are The open open node cloud people because we're using their api library So give a hand for these folks, especially if you if you know these people This project would not be possible without without the work they've They've done for us and are helping us to do So, uh, if any of this is interesting if you want to follow along, uh, we have three, uh, three github repositories right now um The last one is not special. It's not supposed to be linked like that or whatever. Uh The portal the self service portal is the first one secure cost Free ipa fast is the schema extension and then fast use on is the uh, the read only api that I mentioned uh, we also have a an irc channel fedora dash triple a and The team of people working on this are are all in there right now But feel free to drop in we have meetings in there regularly But also just feel free to drop in and ask questions or say hey, I want to help. What can I do? Uh, or just idle there and follow along We're more than happy to have you we want to do all of this in the open. It's something that affects the entire fedora community Um, and hopefully other communities if if they end up using our work So we want this to be a very open and transparent thing So we are more than more than happy to have people kind of follow along And ask questions if if something's unclear to somebody who's interested in this then it's unclear to other people as well So ask the obvious questions Speaking of questions. Does anyone have any? Yeah, so the the one that the password sync is happening across right now is that that same instance And We are going to have to upgrade that instance because we need the group sponsorship stuff Which is in a newer ipa than we currently are using Um, but yeah, it will be that same that same deployment Early the biggest challenge He asked what um, what is the earliest and biggest challenge to combining sentos and fedora's authentication There's still a lot up in the air right now. So that that whole idea was only very recently approved by the sentos board And so there's there's a whole whole lot of discussion and and figuring out, you know I would say right now just policy alone is something that's going to take a while to flesh out That's why I didn't want to hit on that too much in the in the talk just because there's there's so much up in the air still Um, yeah, it's it's something we're looking at All right, so the so the question is how are we using key cloak and I guess open idc in general versus IPa on the back end so um So key cloak we don't actually use key cloak right now. We use ip salon, which is another open idc open id connect provider um Basically everything is stored in ipa and will continue to be stored in ipa And ip salon just acts as an open id connect Provider it connects into ipa for doing the authentication But it's like a layer on top of it for web apps to use to authenticate So we're not actually storing information in key clover ip salon We're just kind of using that as a proxy layer to to let web apps authenticate Uh, the question is do we plan to switch to key cloak? That's an open discussion Uh, it's not going to be in the initial phase of the project, but it's something that we have talked about doing later on Thank you. I hope to see you around the irc channel and