 Hey folks before I get started with the slides. I just wanted to thank everybody personally for coming out here There's a lot of great speakers a lot of good stuff going on So I really appreciate that you find this topic interesting enough to come listen to me speak about it The talk is migrating to IDM in a large Linux environment and we're gonna start by Basically going over what the environment currently looks like And just in case there's somebody in the crowd that doesn't know what IDM is it's basically a tightly coupled product that Red Hat is offering that combines Kerberos and some DNS and Pki and also Like a missing some lots of other things as well, but People like to kind of lovingly refer to it as ad for Linux So Red Hat as a whole has different pillars inside that do different things I work for what I'd call the IT pillar and Underneath the known knowns are things that I know about Inside of IT and things that I can help contribute to then there's other pillars. So says the engineering group that Actually writes most of Red Hat software But Red Hat as a company is kind of interesting in that there's lots of different groups that Sometimes consume the services that IT is putting out there. So like we offer email We offer the corporate LDAP people will consume that stuff But the other groups also kind of set up their own things and go off in their own world And I really have no idea what most of those other people are doing a lot of the time And that kind of stuff's listed on the known unknowns under this slide. So first off We know that there's about 3000 IT managed VMs at least 800 cloud instances We're getting close to 12,000 employees now with about a hundred SSO integrations And on the IT side of the house almost everything is built using configuration management whether that's Ansible or Puppet 90% of our server base is rel. We have a very complex account life cycle management process right now. So onboarding You know different managers have to approve different steps There's other systems involved that run specific scripts and create stuff on our back ends and also calls out to SAS vendors and creates things there Same for termination Our current LDAP servers have a tremendous load on them all the time So these are just kind of things that we know about and things we have to be careful with as we start moving to IDM The unknowns that I know are out there is engineering and labs I know that there are some of them are consuming the Kerberos and LDAP stuff that we've set up But I have no idea how they've configured their systems to do that Additionally kind of getting the word out to all of the different pillars at Red Hat is a little bit challenging You know, there's like wide swaths of mailing lists and stuff like that But it's just hard to find all the right people that need to know what you're going to do There's also different pillars that have some shadow IT stuff again We have no idea what those folks are doing people are running containers everywhere and I don't know what that means for them moving to other systems like IDM and We have people running like mission critical applications under their desks So there have been situations in the past where IT's made a big change Something broke. Nobody could figure out why they track it down. Okay. Well this guy that used to work here Left this running here. Nobody knows what it's doing. But okay. Well, it's doing something important So now we have to figure out what broke There's also vendors using our LDAP servers, so we offer Secured TLS endpoints using certificate based off so that specific vendors can query a reduced set of attributes from our employees To provision accounts on their systems and things like that But I'm also relatively certain that there's people inside of Red Hat business owners and other tech like teams that are gathering information from our LDAP servers Because I mean it's easy to do and then feeding that to vendors in ways that we're not aware of whether that's Jumping LDAP to a CSV and SFTP net to a vendor or whatever Getting it out of LDAP and calling API's. I know some of that stuff's happening. I don't know who's doing it and Once we switch to IDM things will just slowly but surely break through those situations So this is going to go over kind of what our current configuration is So we have multiple data centers running a MIT Kerberos master That propagates to the other data centers. We have dog tag, which is RHCS, that's Red Hat's certificate project so you can you know cut TLS certs for servers and clients We also have 389 or RHDS, which is Red Hat's LDAP server implementation Those dog tag and 389 are both Multimaster copying from all the data centers and then we have a DNS master of bind that sends data to slaves everywhere So currently on the IT side of the house The servers that are running RHEL6 are doing NSSS, PAM, LDAPD, PAM Kerberos 5, NSLCD and NSCD Kind of your your legacy setup with Auth config if people are familiar with that and our RHEL7 boxes are using SSSD and Almost all of those servers are puppet managed So my group has already started writing the code that will help people migrate from the old systems to SSSD and to free IPA That we're standing up. There's also IT supported clients. So we have a service desk and a help desk style thing We have a desktop team that puts together corporate standard builds of laptops to do out to our employees So they support RHEL7 Windows and Mac OS X And Then there's also all kinds of people bringing pretty much anything you could imagine to Red Hat and running it and however They feel like it and the perfect world. This is what we would move to So in a couple slides back Each one of these systems are kind of you know, they're all related people use them They're all identity and access management based but each one of those requires a cluster and every data center and Each one of those is not tightly coupled. So basically Our PKI system kind of sits off in its own little corner people access it through APIs People end up cutting certs and then laying them down on their servers in various ways Whether that's through Ansible or puppet or just SCP files over But that that dog tag instance doesn't really have any tie-ins to our directory server and Same similar stuff goes for the Kerberos systems There's automation in place right now that will allow people to get host key tabs and server key tabs and help them copy them To their systems But all of that's kind of like custom bolted on code that we've written in-house Some of the stuff that IDM offers is ways to streamline that and make it a lot easier using commands built into the IPA CLI So instead of having to script getting a key tab I could run a run a command that's something like a IPA get key tab or get cert for getting a TLS cert and It just all magically happens So in addition to reducing our footprint and having less to manage you can see that We're reducing those four or five things into just one cluster. So IDM is Serving all of those purposes DNS Kerberos LDAP PKI So it's really a management win and this is where we're trying to get to because it's a lot easier Handle and it has the built-in abilities as I mentioned before that kind of make automation easier Where we're trying to get the the systems to and the new perfect world config is EL6 and EL7 servers using SSSD and IPA clients again puppet or Ansible managed and then those remote clients That service desk takes care of We want to work with them to make sure that they can release RPMs for the rail seven laptops that they support And then I think they use big fix or something for Windows people and Mac people but basically Make sure that they have all of that stuff in place So once we switch to IDM they can push out changes and the clients that are out there Will start consuming them without you know hundreds of people bombarding service desk one day and saying this stuff doesn't work anymore The other thing I'd like to see us do better is writing instructions for common bring your own devices set up So a lot of people will run for the war on their laptops, and we do have some documentation around that But you know, I've seen people running to lack where I've seen people running a bunch who so Kind of trying to figure out what people use the most and writing instructions so that they can Configure their own systems to use the stuff that we're going to offer through IDM So getting there is going to be a multi-year and multi-phase process The first thing we're going to do is we're going to keep our existing infrastructure, and we're going to add an IDM Cluster as well, then we're going to establish a cross realm Kerberos trust between the two Systems so between MIT Kerberos and IDM then we're going to do a DNS zone delegation to the free IPA free IPA servers Then comes the long and arduous task of getting all the systems and all the users To migrate over to start using Kerberos on IDM and stop using Kerberos on the MIT stuff that we have set up today Once we accomplish that we're going to shut off MIT Kerberos So then phase three is RHS cutoff and reduced so part of this process is taking all of the data that's in LDAP or all the data that we can that's currently in 389 server and moving that to IDM so people can still see their Their user information and LDAP and use it for you know user look user lookups or NSS lookups or whatever They're doing with that Phase four is going to be RHS cutoff So again IPA offers some great features with certmonger and get cert so that it's very easy for people to be able to pull certs from IDM and That's going to be a lot nicer than the RHS stuff that we have now, which is still great But again, it's it's more tightly coupled and easier to use than having something standing off in the corner by itself Which is what RHS is for us right now phase five is kind of exploring the OTP functionality of Free IPA and then also exploring the DNS offerings of free IPA more so this talks mainly on phase one we haven't gotten any further than that but known limitations of things that we've already hit is The IDM migration is still rough around the edges So cross realm Kerberos support is not directly supported it works, but it we have encountered some bugs The out-of-the-box migration scripts are also somewhat limited They're built more to do kind of like a one-and-done approach Like I want to take everything out of LDAP put it into IDM and then the next day Everybody's going to talk to IDM and we're just going to cut off the old systems. Let's talk more about that in a minute It's not really built or we haven't been able to use it in a fashion where we just keep refreshing data on the new systems Some of our internal applications at Red Hat are currently using data in LDAP for their specific application needs So we've developed custom OUs and custom schemas So that a team could for instance look up a product number and an attribute for a product entry things of that nature That meets the applications needs But IDM is really built and optimized for standard users and groups data So whenever we get to this point we have to decide if we want to dirty up what IDM is offering by Including a bunch of custom OUs and schemas there and seeing what kind of performance impact that would have or if it makes sense for us to basically scale down our RHS infrastructure now so that it just offers that Custom work that custom schema and custom OU stuff for the internal applications Our current RHS system uses an HSM for key storage IDM doesn't support that yet, but I know that it's on their radar So if we want to kill off RHS, we either have to get that into the product or we have to be okay with a slightly less secure setup For OTP, I know some of our networking gear uses radius to talk to our current OTP solution And I haven't done any research yet to know if they can talk LDAP to free IPA and if not Then I don't know what we do. We might end up having to keep the current OTP solution There's also our DNS infrastructure right now is very large And it's managed in a way that people are used to so people do get checkouts and get commits and merges and stuff like that And that rolls out across the system So if we want to completely get rid of bind, it's going to be not only do we need to check and make sure that You know, it's going to be able to handle all the records and all the stuff we throw at it and all the changes that we make It's going to be kind of a political battle to get people to change their minds about how they want to update DNS So some of the phase one challenges we've already hit are password migration So in our HDS or in 389 directory servers are really awesome feature called pan pass through and what that's allowed Us to do for several years now is Not keep user password hashes on user entries in LDAP So whenever somebody does a bind to our LDAP servers The LDAP servers Transparenly in the background will reach out to Kerberos and authenticate that person Or we can set it up to do that with any pan module We also have LDAP servers set up to talk radius to our OTP solution So people can again bind with their pen and token and that would off to an OTP set Solution all that's really awesome because that limits how much How much exposure your passwords are, you know, they're stored on one system. They're not stored in multiple places but the IDM really wants you to have a user password hash whenever you're going to go ahead and do migration so they support SSSD and When they support SSSD the user tries or it basically tries against the user password hash and Then if that's there, it will go ahead and update the Kerberos password hash Same with the web UI migration utility that they offer it checks the user password hash and then updates the Kerberos principal hash so Again, we don't have user password in our LDAP. So that's kind of a non those are non starters for us the other thing that they offer or that they suggest doing is mass setting users passwords and That's kind of a management nightmare So, you know, you have to set somebody's password and then okay, how do I contact that person? I don't really want to email passwords. So then we have to do PGP keys and You know, you try to explain to a finance person how to read an email That's PGP encrypted and you don't get very far. So That's not really a good solution either So we've come up with a different solution IPA offers a special Config variable and a special entry where you can list out UIDs that are allowed to set other people's passwords And not only are they allowed to set other people's passwords. They're allowed to bypass password setting restrictions Meaning that this password service account can update a person's password and then the next time that person logs in They're not required to change it immediately So we've set up account and we've set out up a service account like that and then we've written a basic web app that's going to bind to IDM using that credential and People will authenticate to this web app using legacy credentials and SAML and then that web app will use that IPA service account to make a call out to the IPA API and set that person's password So that's that's pretty neat. It's pretty slick. It works So people will log in using their legacy credentials. They'll choose their new password It'll be set immediately in IPA underneath the user password record in LDAP and in the Kerberos hash Other big thing we're hitting is data synchronizations. This is what I hit or what I Hit on a little bit earlier. So IPA migrate DS is Great for kind of a one-and-done approach We used it for our initial data import, but red hat again is a very Widely dispersed company. So and there's a lot of employees We can't just do a one-import and then you know, we do that on Friday Everybody comes in on Monday and all their stuff's working I don't know how to tell people to configure their stuff because I don't know what they're running and On top of that again, you'd would you would run into people that don't understand and we would have to staff like a huge help Desk for weeks that would just receive phone calls and things To help people through that process But with that said if you're a smaller company or if you can do anything to prevent what I'm about to show you The solution is I strongly advise it because what we're doing here is really nasty So we have a data synchronization script and It's running out of cron and it compares entries on a per attribute level Taking into account differences in OU structure So the OU is different and the IPA world than it is in our current RHS system or than our current MIT Kerberos realms Or yeah, and side of the LDAP structure as well And if the attributes don't match with what we've defined as the source of authority for that attribute then the downstream system get up gets updated so what this means is there's business logic and RHS or legacy systems that say This new data anytime this new data is up added here or changed here I want you to replace that information and IPA with that But then there's also the reverse business logic whenever something new gets created an IPA I want you or updated an IPA. I want you to go back and update that stuff and the legacy RHS systems so This business logic will change over time and it's going to be a disaster because It's going to end up with people changing something and not realizing that that's going to impact other people using the legacy systems or the new systems and Yeah, we're going to do the best with what we can with this and then hopefully Get people to migrate quickly so we don't have to keep this kind of hacky solution in very long Other thing we hit was poor bi-directional trust support and apps So some applications are basically Expecting you to use the at old of realm So it's looking for in our case at redhat.com and that's our old MIT Kerberos realm And it does that I'm not sure why but some applications are actually doing that hard in code So one of our own applications RHS. So is looking for at redhat.com because you've configured that as its key tab and all that stuff So it knows what the realm is But it should never be checking for that exact realm because people could have trust Other applications we've hit one with Zimbra. You have to make changes on a per user basis to support principles from different realms and Then we've hit some of the same things where you know bi-directional trust and Those two realms don't necessarily play nice with settings that are out of the box with SSH or a mod-off curb But those were pretty well documented on how to fix those and you can just change those settings globally and those Applications and go about with your day So what we've done to fix that is we filed RFEs for the code issues as people are migrating We're going to work with them to change per user configs And we're putting together documentation on what needs to change in KRB5 comms or and mod-off curb setups so we've kind of fixed or addressed this on the IT side of the house But again with there's different pillars at redhat. I have no idea what those people are doing so the best we can offer is here's documentation on what you might want to do and Kind of distribute that out and let them handle it as they can or You know after the drop dead date and all their stuff breaks, whichever one they feel like doing We hit a couple other issues along the way. These are Minor and more annoying than they are real issues like the other two So one thing we were really looking forward to is auto-membrane and what this allows you to do is Add members to a group in LDAP Based on an attribute that they have so redhat reorgs pretty regularly and with that We have issues where? You know if I used to work on dev team a and then I get moved to dev team b I'm still in all the dev a or dev dev team a's LDAP groups And I still have all the permissions that I had whenever I worked in dev team a So that's something that we would like to solve And auto-membrane is how it is being able to do that But unfortunately it does not currently remove users from groups when you change the attribute I think it does the auto-membrane when things are created, but not necessarily on changes And they're working on that so that will hopefully be fixed in the near future another thing we fit is limited limited look-through limits and That's basically If somebody does an LDAP search and they want to go ahead and pull down all the records from LDAP Eventually there's there's a set limit that says okay You can only get like 10,000 records, and then we're not going to give you any more So we've tried to change that to infinite because that's what people are used to and I know that people are running scripts that Download everything in the LDAP directory and then process it locally and then do stuff with that I'm not super fond of those people doing that, but at the same time like It's a give-and-take because otherwise I've seen people come to me and or people have come to me and ask you Mind if I download the entire LDAP directory like yes, please don't do that But then they say okay. Well, I need to get a recursion tree of all these people So okay, tell me how you're gonna do that Well, I'm gonna look up this user then I'm gonna look up this users group Then I'm gonna like and it goes on and on and then I watch the guy do it in the logs Okay, well you just hit the LDAP server 800 times in a minute, so I don't know You know, maybe it is better for you to suck down all the information The other thing that we've hit is The product supports full server backups, which is fantastic But to run those backups you have to take a node offline not a big deal We were just gonna build an additional node that nobody would access and then take that node offline to do the backups Bring it back online. We could also tinker around with things on it use it kind of like as an admin box for our team Unfortunately, though whenever you stand up a node Or join it as a replica and free IPA currently It's assigned to serve record and a priority and that priority matches all the other systems and Basically what that means is as people start using IPA the backup server will be advertised to them And they will try to connect to it So we've submitted some bugs and RFUs around that because otherwise While that server is doing a backup people's clients will at least suffer a little bit of lag as they have to try another system Or if it's a poorly written application, it would just be broken until you know the net that specific node comes back online Thank you So like I said, we've filed bugs and RFEs for those things and That's basically all I have. Does anybody have any questions or anything? Yeah Management Sure, so the question was Free IPA has kind of a flat OU structure. How am I planning to move? Specific OUs with specific like ACI style stuff to IDM The answer to that is to be determined at this point So yes, we do have in our legacy RHDS system We have several OUs that are specific to apps and we have specific ACI set on that So we'll basically if a group comes to us and say hey We've written this awesome app and we want these people to be able to access it and we want to Authorize them based on group membership and LDAP Okay, cool So we create the group and then we give them a service account so that they can update that group membership So that they can control who's able to access their app in different fashions I'm not sure how we're going to approach that and IDM yet Any other questions? Yeah, so the question was how do I deal with Windows clients so the answer is my team does not But Yeah, that's somebody else's problem, but that's only going to be somebody else's problem for a bit So it's on our radar as we start to take over DNS more and more with free IPA They run AD DNS and other people run other DNS devices There's been talks about us taking over there at least some of the DNS aspects and then probably most likely the active Directory components as well, but we're going to have to explore some before some and figure out exactly what we're going to do with that But yes, we do have a Windows base current legacy RHS has Replication agreements so RHS supports Basically AD replication. So we have it set up. So if you get an account in LDAP it will replicate that to active directory Correct yeah, not vice versa. I believe you can set up both But we haven't because I don't trust what's an AD or the people running it to be honest. So Then there's Yes, exactly Now if you have Windows stuff and I haven't looked into this much at all because it's not a problem We really have here at Red Hat because there's very few Windows systems, but Free IPA has amazing options to integrate with active directory and Windows things I think that's kind of some of their bread and butter that they push a lot of is being able to take this Linux system And now let's make it use AD easily Yeah Yeah, that's a good question All right, so the question was you I have a picture of the migration plan with all the different phasers What about a time frame so phase two? I hope will be done within the next year Phase three and phase four could probably be completed in the next year So I'm guessing 2020 To be done with everything I don't like that timeline, but I just know that it's going to take a while Alternatively We can push people to move faster, but we also need Lots of different manager buy-in and stuff like that So again, I can get that relatively easy on the IT side of the house and move those systems But in all the other situations where I have no idea what people are doing We need to be more cautious Any other questions? Thank you. No, thanks everybody