 All right. Hey everybody. My name is Dustin Minnick. I work for Red Hat IT and this talk is going to be discussing how we are currently moving systems, Linux systems from using a legacy Kerberos and LDAPs it up to using IDM. If you're not familiar with IDM, it's the downstream of free IPA. If you're not familiar with free IPA, you can kind of lovingly think of that as AD for Linux and it basically has certificate management built-in, Kerberos built-in, LDAP built-in. It also manages Sudur's SC Linux, has a DNS server built-in, so it does a lot of stuff all in one nice package. So to get started I'm going to talk a little bit about Red Hat's IT environment or Red Hat's environment as a whole. I'll let you guys kind of read that slide because there's a lot of words, but the overall, I guess, takeaway that I want you to realize is that Red Hat is a big company and it kind of operates with several set or I guess several groups running kind of like their own startups and that creates challenges for things that you want to change across the entire breadth of the company like this. For instance, I know what IT is doing and I know what systems we manage and I know that, you know, if we give laptops to new employees, I can be reasonably assured what they're going to get on them, things of that nature. But engineering, the people that actually write the software, kind of have their own IT group and that's good for the company because that allows their group to run like bleeding-edge versions of things like OpenStack and whatever they're running. But at the same time, other people in the company need, I would say, more standardized or more red tape style things and more normal offerings that like a normal IT shop would offer, so it's kind of an interesting mixture of where we offer some things that other groups use and then other groups do basically the same things we do. So right now, this is our legacy setup and this is what we're trying to move away from. So we have, you know, a Kerberos master and DC1 that replicates the Kerberos slaves elsewhere. We have DOMTAG, which is RHCS and that is the certificate system and that's multi-master. It's in multiple data centers and writes back and forth. Same for LDAP 389, RHDS writes back and forth in multiple data centers. And for DNS, we run BIND with one single master that writes to slaves in multiple data centers. So, I mean, you can see that this is a lot of software, a lot of stuff to maintain and you can also tell that BIND and Kerberos have at least a little bit of lag time. If something goes down, we'd have to do some work to promote some stuff to master. This goes into how things are configured right now. So we do still have some RHEL 6 boxes hanging around and they're running NSS-PAM LDAP-D, PAM-KRB5, NSLCD, NSCD, took into LDAP and Kerberos. Our RHEL 7 boxes are running SSSD. They're puppet-managed and whatever we give laptops and machines to people, it's either RHEL 7, Windows or Mac OS and those are kind of like blessed images that our desktop team manages. We also support a lot of bring-your-own-devices and self-managed systems. So one thing that's I think also interesting about Red Hat is even for the systems that Red Hat pays for, so if you get a laptop from Red Hat, last time I heard the statistic like 50% of those aren't even running the corporate image. Like people just get it and wipe it and install and do whatever they want, which is actually great for the company because most of those people end up installing Fedora and then they find bugs in Fedora and Fedora gets better and then RHEL gets better. So that's great except for now if I want all of those people that are self-managing their own machines to start using IPA, how do I do that? So Yeah Perfect World Config is moving everything to IPA. IPA does all the things and IPA has replicas and it's all multi-master everywhere. So again, this this is easier to maintain because it's all one big stack and a lot less servers and also we get rid of the single master problem for Kerberos and the single master problem for some of our DNS stuff that we got going on. Ideally everything would be REL7 using SSSD and we hopefully be Ansible managed and as we start to roll out new laptops, let's bake in IDM support to the actual images that you know the salespeople, the finance people, the people that want something that's supported by standard IT processes so that they can use a known system and then let's also give well-documented instructions, RPMs, and other things for everybody else that wants to do their own stuff. So getting there is going to take a long time. So we're currently in about phase two and a half. So we have the existing infrastructure, which was the legacy Kerberos, LDAP, bind, so on and so forth, and we've created an IDM cross realm trust and we've also done a DNS zone delegation. So our legacy Kerberos realms redhat.com, we now have an IPA.redhat.com. They both trust each other through Kerberos bi-directional trust and right now the IPA DNS is only managing IPA.redhat.com and it's subdomains. So that's phase one and I'll pretty soon go into all of the challenges we've had getting past even up Phase two is kind of adoption tooling. So as I mentioned with kind of this open model that we run at Red Hat where people do whatever they want to do, we have to make it easy for people to switch so that they will have incentive to switch and actually want to and and one of that or one of the things that we're trying to encourage folks to do is people that have written Ansible play Ansible roles or puppet modules to manage things like sudoers and se linux and those kind of things that are now that IDM is able to centrally store and manage and also SSH configs and stuff like that because IDM will do HBAC controls we have to get those folks to compare what they're currently doing to what IPA offers and try to show off that IPA can do this better and potentially, you know, coax them into switching. Phase three is going to be cutting off RHCS. So that's going to be the easiest thing for us to kill because basically people are just using it to cut certs, whether that's server certs or client certs and all we have to do is convince folks to start doing those same actions to the IPA CAs. One of the things in phase three also is the RHDS, so that's the LDAP repurpose. One of the things that IPA states in their documentation is that it's really good at storing user and group information and it's really tuned to do that very well and Red Hat, over time, has collected a lot of cruft in our LDAP servers, so people will come to us and say hey, I'd like to store my biography in LDAP or hey, I want my picture in LDAP or things that aren't like your typical Linux and Unix attributes they wanted in LDAP. So we've written schemas and we've extended our old legacy LDAP to support that and we have to push back on that and decide how how dirty we're going to make IDM to support what people want and how how unhappy we can make people to make our lives easier. We've also used RHDS for custom application stuff, so we have mail servers that do MX lookups in LDAP and that's because the mail servers do a lot of work and LDAP just responds really quickly and it's a good purpose fit. But again, we don't want to put that entire structure into IDM, so in all likelihood, we're going to continue to run the LDAP servers that we have now. We're just going to minimize that as much as possible. Phase four is Kerberos cutoff. That's going to be right now. We're scoping probably end of 2020 and the only reason why is again, we don't really have a whole lot of control. We can force people inside of the IT organization to some extent to make moves, but outside of that realistically, the only way we're going to get people off of the old systems is to really persuade them hardly whenever their I guess whenever their stuff is about to time out. So if you're going to be migrating from RH6 to RH7 or if your application is basically going to get rewritten or your device is going to be retired and you're going to get a new laptop, whatever it may be, whenever that brand new thing is rolled out, then it's time for you to move and then once everybody moves, then we'll be able to cut it off. Phase five is Forever Out and that's going to be us examining if OTP and bind can be more widely used. Right now in-house, we have a different OTP solution that seems to work pretty well, so we're not super, I guess we're at this instance, we're not super interested in pursuing that route and bind is also going to be interesting as well because we have some config management and we have lots of DNS servers around and I don't know if people will want to give up the way that they're used to managing that kind of stuff to move to IDM. I'm also like, we haven't done any benchmarking or anything either, but I'm not, I don't know, I'd assume IDM would be able to keep up with the demand, but we have lots of records and we get lots of requests, so I think IDM kind of treats its bind instance a little bit like a black box, like I don't think they want you to go in and mess with it a whole lot and that that leaves me a little concern versus the way that they treat RHDS, RHCS and all that is also kind of black box, but we don't really need special configs there, so I'm okay with it. Let's skip this slide. Last time I gave this talk, I had an hour, so I'm kind of trying to go through this pretty quick. So data synchronization was one of the problems and so IPA officially supports IPA Migrate DS. We use this for the initial data import and it's good for kind of a one and done approach. Unfortunately, to support these gradual migrations that I've been talking about and because we have a very complex account life cycle management that's hooked into things like HR systems and SaaS vendors that track locations of employees and things of that nature, we have to run both systems at the same time and that is terrible. In order to do that, we have written custom code and I would strongly suggest that you don't do this, but basically what's happening here is we have the legacy LDAP system that's still being used for some of the life cycle management of accounts because, again, that's what systems we currently have tied into it and as those things change, we have custom code that is then taking those entries out of legacy LDAP, putting them into IDM and sometimes that's with API calls and other times that's with raw LDAP rights and all the OU structure and everything else changes. It's just not pretty. Additionally, as new apps are onboarded or as new consumers are onboarded, we tell them to use IDM but then every once in a while somebody's using IDM but some of their clients aren't using IDM so then we have to do the reverse as well. Now we have to copy stuff from IDM to the old systems. Anyhow, if you can force a change, force the change because this is not good. Another situation we hit is in RHDs we were using a plug-in called PAMpassthrough and what PAMpassthrough does is it allows you to configure password bookups in different ways using PAM modules and traditionally what we had done with RHDs is delegate to Kerberos PAMKRB and what that meant is your LDAP records did not have a user password attribute and every time you would do a bind against LDAP, LDAP would then talk to Kerberos and validate the password. We did this because sweet, now we only have to store passwords in one place and ideally that's more secure. This did not end well or did not support our migration very well to IDM because IDM supports mass setting passwords and communicating them out to users as a migration strategy. There's no way we could ever do that. Maybe small companies that would work. They also support a web UI migration and an SSSD migration, both of which look for the user password and validate against it and then if Kerberos doesn't have the password, set it. They tackle it from the kind of reverse of what we had. They assume you're going to migrate over an LDAP environment that has user password hashes and you can't migrate over the Kerberos data so then they would set up the Kerberos data. We had the exact opposite. We can't migrate over the Kerberos data and we can't. So we have the ready custom app and the custom app is using a specific account or a special account inside of IDM and what happens is you end up on this web page and it's protected by SAML and it's SAML's tied to our legacy LDAP and Kerberos infrastructure and once you log in it asks you to set your IDM password so you enter your IDM whatever you want your IDM password to be and it goes out via a REST API and using that special account, that special service account and IDM and set your password. It also bypasses the user password reset requirement so normally whenever you set a password and IDM next time the person logs in they immediately have to change it. So this has worked well and this is easy and simple for people to manage so overall this was a win. It just took a little bit of time to figure out a workaround. So I've alluded to this some. One of our biggest problems is we can't lift and shift at all so in addition to me not knowing what the guy across the hall is doing like if he's running a Raspberry Pi under his desk that he's using to you know somehow seed production information. We also don't have really strong team ties to specific DNS subdomains which makes Kerberos migration fun as well. So if you can switch everybody at the same time that's great. If you can't switch everybody at the same time the next best thing you can do or the next best easiest thing you can do is update like the KRB5 configs or top level DNS text records which is what the domain realm stuff there is. So we could say like teamx.redhat.com is still pointing at the legacy redhat.com realm and team y.redhat.com is pointing at the IPA.redhat.com realm but the issue is is team X some of them will be redhat.com some of them will be IPA.redhat.com and because of that we have to end up doing all of the Kerberos information has to be fetched from DNS and as a specific machine is migrated we have to add a text record for that specific machine if they're staying in that same subdomain. So that's working it's fine and this is this is basically the statements of what's happening here when I just went over we have to move one system at a time as they age out and create DNS text records that tell it what basically that point this guy at the IPA realm instead and we're doing that as things age out and we're also again creating thing or we're making things as simple as possible so that when they do age out people will be able to migrate to IDM and hopefully not complain a whole lot. So we're creating puppet classes ansible roles rpms and docs for bring your own devices we're also updating kickstarts and other automations that our desktop team uses to spin up the corporate laptops another problem is people hate change so people are used to how they're getting certs for or from rhcs and people keep asking like well puppet or ansible is doing a really good job of managing ssh allow groups why should we switch to age back inside of IDM and again puppet or ansible is doing a good job with config management for sudoers and se linux why should we change and the last one's again a mention of basically how we're going to have to end up dirtying up the idm schema so people are used to querying LDAP and getting things again like biography or we have like a like a void system or a video conferencing system so we'll store like their video conferencing id and LDAP and things that aren't you know again stock uh stock schema things so to get people to change we're developing and offering even better automation so for the certificate stuff that they're used to we're going ahead and creating better automation using certmonger and other custom code that will spit out or basically take the PEM that certmonger spits out and renews automatically and converts that into pkcs12 and jks and it just really makes it nice so that software developers can just point out a specific file and just know that the cert's always going to be up to date which is a win for them compared to okay well my cert's going to expire and now I have to run an automation script to get a new cert so that's better automation for SSH allow groups we're doing stuff like auto member groups in HBAC so we'll be able to set things like okay you are underneath this manager and because of that you will have access to these systems and that will just happen and we can do auto member groups and auto member host groups and it all just happens automatically inside of IDM whenever it does the refresh stuff so again that's in some ways better than what they're used to with SSH allow groups we're also doing trainings and demo sessions and again pro con list type things for sudoers and se linux to try to get people interested in switching from config management to basically moving that some of that stuff into IDM and here the last bullet point again is talking about us moving some of that non-standard data into IDM there's some links there the my slide that's on the sketch site so you guys can grab that but here's some some of the things that we've done to do that it's not quite as simple as I would like it to be or quite as simple as it was in rhds uh you have to do some stuff and then you have to do some other stuff to get it to actually display the extended schema attributes at the api level and I got five minutes left and ready for questions yeah the question was from a fedora what version last version oh yeah yeah yeah absolutely from a fedora 29 version or laptop can join IDM yes I will say that with one small caveat I know it works with fedora 28 and I have seen a I've got a ticket in my queue and it was somebody that was trying to use fedora 29 and I'm honestly not up to date on the desktop side but I think there was um off configs being replaced by something I'm sure somebody in here knows um and because of that the IPA installer I think was bombing because it was expecting off config six beautiful yes that is true any other questions if you know now would you still migrate the idea with all the stuff from uh yeah the question was if I knew of all the problems that we were going to hit would I still move to idm and the answer is yes um long term this is going to be a game changer for my team because it's going to be less systems to manage and it's also the right thing for red hat to do um we try to dog food whenever we can so the more that we use our own products and you know the more we discover some issues the better off it is yeah sure yeah the the question was how many user entries do we have in the database yeah right now red hat has last time I looked a little over 1200 users uh yeah sorry 12,000 yeah 12,000 missed a zero um now to blow your mind I think we have like 50,000 groups yeah the question was do we keep off-boarded or terminated employees in the user base and the answer is yes uh 12,000 was the active user account I have no idea what the inactive user account is we move them to and legacy I'll have to move them to um a different OU and in idm we just do the inactive thing um yeah anything else cool thanks everybody