 Today, we are going to talk about SSS certificate, a way to scale the SSS access. My name is Phulkit Vaishnaw. I'm working as DevOps engineer for three years, and I'm currently working at Moingage. I have worked in different domains like digital marketing, CDN, and e-commerce in the companies like Package Zoom, Moingage, and Iro. I'm DevOps enthusiast and a Linux fan. So the agenda of today's talk is what are the traditional methods to use the SSH, then we'll discuss what are the centralized authentication approaches which are scalable, and what problems they have. Then we'll have an adventurous ride of the SSS certificates, then we'll have a quick demo and we'll discuss about the features of SSS certificate, and at the last we'll discuss limitation and the solutions. So here you can see a hospital, in the hospital there is one ladder, and there is one helicopter. Ladder is a regular way to access the hospital, and to get the platinum access you can come using the helicopter. At Moingage, we were using the regular approach for the SSH access to provide our developers, but we wanted a solution which was scalable, which was easy to set up, and which is secure. So we come to a solution of SS certificates using which we got all the things, it was secure, we have so many nice features like role-based SSH, okay, so before starting about SSS certificate, what SSH is, SSH is a client-server best protocol to access the remote machine on an unsecured network, sorry, remote machine securely over an unsecured network. So we all need to SSH into the machine either that can be for debugging, that can be for setting up anything or to check the performance of the server. Here are some common commands which you would have used to SSH into the server. Now there are some traditional ways, so password authentication to SSH into the servers, we set a password and when we SSH, we entered the password to access the server. The problem with the password base authentication is password can be cracked, can be leaked or passwords are predictable, and nowadays we have a problem like we have so many passwords of different applications, so sometime we forget the password and we lost the access of the server, I recently had this kind of incident. Now the second is public key-based authentication, here we have access key via PM files, like some cloud provider provides you a private file and using those files you can SSH into the server. Problem with this method is if your key is leaked, your that private PM key file is leaked, your system is compromised. The other way is public key authentication method in which we had the public key file of the user to the authorized key file of the server and every time a user adds or delete from the system, we update the authorized key file. Now here on scale like for example you have hundreds of users and thousands of servers, in that case management of this public key on all the servers is a pretty hard problem. So are there any alternatives? Yes, yes there are. We can use some central authentication services like LDAP, Kerberos or Active Directory, but this solutions have some problems like they are hard to install, they are hard to configure and during an incident they are single point of failure. So let's say you have an incident like your centralized server went down or your cloud provider has our outage or the DNS outage, you will lock out yourself from your own system that would be a nightmare for any developer. Now SSH certificate, it comes as a savior like Kung Fu Panda, it's very easy to set up, low maintenance, secure and there are so many cool features like role based access and the certificate validity, we'll discuss this in detail in the further slides. Here are the benefits of SSH certificates, it's cost efficient. So you don't have to set up a new infrastructure, let's say if you use LDAP you have to set up a different service for it, you have to scale your servers, for that if your number of servers increase on your infrastructure you have to scale your centralized authentication service as well. It's simple to set up, you just have to make couple of changes on your server and you are ready to go and it's flexible. It has many features like you can provide role based access, you can add the certificate validity and there are some host based access you can provide using it. Now who uses it, there are companies like Facebook, Uber, Netflix, Lyft and we at Moingage use certificate based authentication methods, there are some open source projects like BLAS, PEM SSH by Uber and we recently open sourced a project, easy SSH. In this talk we are going to discuss about the concept behind the SSH certificates. Now here how SSH certificates work, before that what SSH certificate is, SS certificate is nothing but a certificate based authentication method and it's a certificate authority, it works like a certificate authority and your certificate authority have a public key and a private key. You will populate this public key to all your servers and you will mark this key as your trusted CA user key file. Now when a user request, when a user request that I want access of, I want SSH access what you will do, you will take the public key from the user and you will generate a intermediate certificate using the private key file. The private key file will sign your intermediate certificate from the public key of the user and using that intermediate certificate you will give that intermediate certificate to the user and user then will be able to SSH into all the servers according to the accessibility of that certificate. Now what are the features of intermediate certificate, why do we need intermediate certificate or why should we use intermediate certificate? There are features like certificate identity. So if you use any centralized system like LDAP, it will manage all the users for you. Let's say if you have 100 users in your organization LDAP will manage that on all the servers of the infrastructure but it comes with a cost because you have to run the LDAP client on all the servers but using SS certificate you don't have to do it but you will get the identity who is accessing the server using the certificate identity to every certificate you provide an identity and you can track that identity in the logs of SSH. Then the certificate validity, we see that in any organization people comes and leaves from the organization and if they have access of the infrastructure, if they have access of the SSH key, they can access your entire infrastructure even after leaving the company. So what you can do, you can provide an intermediate certificate let's say for a day. He can access his infrastructure for a day only and after that he has to request for the new certificate so that he can able to access the certificate, sorry, servers. Now the other is principles of role-based access so we can provide the role-based access. There are so many roles which we can create let's say if you have developers in your team, if you have ops in your team, if you have front-end team then you can create different kinds of roles and you can provide access to those servers only with which this person is associated with. The last is instead of providing access to a user, you can also provide access to a host. So this is why you should use the intermediate certificate. Let's go through the demo where we will see how SSH certificate is working. So here we have three servers, CA server, a log server and a bob server. So bob is in the backend team and he wants access of the log server. Now we will generate intermediate certificate for him and provide him the SSH access. Here I intentionally did a mistake. I want you guys to correct me what it is. So let's go through the demo. So here we will generate the certificate authority using SSH key gen hypen t RSA is the encryption type and F is the output file which we want that is the CA here. We generated our SS certificate and here we have two files as CA and CA dot pub. Now we'll copy the CA dot pub and we'll paste it to the all the servers here in our case. It's a log server will go to the log server and will copy our public key on this. So we'll copy this file and past it to the log server here will create one file. I'm creating it in ETC SSH and CA dot pub. I'll paste the public key of the CA from here to this file. I'll save this file and now I need to I need to add this file to to the SSD SSH configuration so that it will market as the trusted CA user. So we'll open the SSH configuration file that is ETC SSH and SSD conf at the end of file will add a line that is trusted user CA keys and the path of the file where we created that is ETC SSH CA dot pub will add this path and will save this file. And then we'll go to the then we'll go to the client here on the Bob's machine. We need to set up something first. We will check whether Bob's have public or private key or not. So here we'll go and check whether Bob so there's not. So generate public private key for him using SSH key gen command and SSH key gen hyphen T RSA we will provide the encryption type that is RSA now add your passwords. If it's recommended to add passwords when you generate the CA you can avoid here. So now we have the public and private key ID RSA is the private key and ID RSA dot pub is the public key of the Bob now we'll copy the public key of Bob and will generate an intermediate certificate for him from the certificate authority which we created. So we'll copy this public key and we'll go to the certificate authority server here will paste this file in the new file Bob dot pub and will generate the certificate. See we created the file now now we need to generate the certificate. So here SSH key gen is the command using which we will generate the certificate minus S hyphen S is for the source file that is our CA which we created now will give him the identity of the file that is hyphen I and Bob is the identity of the file now using hyphen N you can give roles or principles which we were we discussed here you can see there is one developer principle there is one ops principle and there is one given to principle these are all the roles of any system now we'll add the validity of at the validity of this certificate. So in this case I'm giving it for one week so hyphen V and plus one and now hypen Z is for the unique number you give to this certificate identify whose certificate is it and finally the public key of the file that is Bob dot pub here you can see it is signed now we'll check the details of the certificate. So here SSH key gen hyphen LF and the certificate name that is Bob hyphen cert dot pub. So now here you can see it's a certificate user certificate it is created from this public key signed by the CA now here we have key ID that is Bob we the serial number which we gave that is one and it is valid from 20 20 to 27 and here are the principle files developers ops and you want to critical options are there which are some advanced topic which we are not going to do discuss in this talk but here are the extensions which you can change using hyphen option. So let's copy this certificate and paste this certificate to the client machine that is the Bob's machine will will copy this certificate and paste it to the Bob machine we copy and under dot dot SSH folder we will create a file ID RSA that is the name of our private key and will add hyphen cert to that file and will add our certificate. This is the path which by default SSH will use if you want to give your path explicitly you can name it whatever you want but if you want it to implicitly use this you can use this so we'll paste our certificate here and now we will set the known host on on Bob's machine because our client Bob's machine don't know what certificate authority it needs to refer before accessing the SSH so will yeah so here under dot SSH file will create a known host and will add our public key of our CA to this this file so for that will add at the rate cert if cert hyphen authority then we need to specify a host in this case we are just going to access our own machine only so we can add a strict sign here so it will refer this certificate this public file for all the all the SSH access now will copy the public key of the certificate from the CA server and will paste it to the Bob's machine here we paste this file and now will try to SSH into our log server will try to access the log server from the Bob's machine now we try to SSH SSH SSH Ubuntu is the user and and the IP of the logs machine now we are just doing Husky checking and permission deny can you tell me what mistake I did anyone are you feeling sleepy after lunch actually I forget to restart the SSH server I change SSH configuration here but I forget to start the restart the SSH so I have to go to the log server and restart it back so I will go here sudo service SSH restart and after that I will try to SSH again and it worked welcome to the log server here is the log server here is our so you just have to add some couple of couple of things to your servers you just have to this one time setup on your client and you can able to use the SSH certificate so that this is all from the demo what we just did this is the quick recap of that will create a CA certificate will copy CA public key will paste it to all our servers then will set this copied file as the trusted CA file and will restart the SSH this time don't forget to restart now on the client side we need to create a public private key if it is not exist will create one intermediate certificate from the CA will paste that intermediate certificate from CS server and now here will setup the known host will copy the public key of the CA to the known host and then we try to SSH into the our server now here is the quick recap of the command as well what we we used SSH key gen is the open SSH command to which will give the source file that is the CA then will give identity that is Bob then will provide principle whatever you want that could be developer ops or whatever teams you have you can provide the roles you can add the validity in this case one week the unique number and the clients file now to check the details of the certificate SSH key gen hypen is at least for the content of the certificate and F is the file name now here are some features role based access so what do we need to implement the role based access you just need to add a principle file on the server it is a simple text file and you can add all the roles in the new line separated way and after that in the SSH configuration you need to add this this this this line the authorized principle file and the path of the file and you will be able to create the role based access for your system now the second is host based access instead of using providing access to a user you can provide access to your host as well now certificate validity now it is recommended to give as minimal as possible because as we discussed if you give SSH access like certificate validity for like one day or one week or one year to your employee and he left he still have access of your infrastructure because he have that intermediate certificate now the identity of certificate so when you log log all all your SSH access either you you will create new user for for all the users you have in your company or you create you use identity certificate if you so here we we provide the certificate identity so now there are some cons as well so when you rotate your CA when you rotate your certificate authority you have to update the public key on all the servers which you are using the second is if CA is leaked it's a single point of failure everyone can create intermediate certificate and use it to SSH to your infrastructure so it's recommended to add a pass phrase if it is not set and the revocation of intermediate certificate is not possible so it is recommended to give as minimal validity as possible now even SSH certificates are not secure you should be cautious when you are generating intermediate certificate and you should follow some good practices like add small small validity to the certificate use Bayesian host so that instead of users machine you you can manage users from the Bayesian host you can create all the users on Bayesian host and you can manage and provide intermediate certificate on the Bayesian host only you should rotate your CA private key regularly and must must must must add a pass phrase while generating a CA certificate because it's important and if your CA certificate is leaked or compromised it will need password to get access or create a new intermediate certificate now all we have so far all we have done is so so much effort you have to copy the public key to the all the servers then users public key you have to populate and generate an intermediate certificate so it's so much effort so we at moving edge built a open source project easy SSH so what we we have done so far it's it's it's it's it's a three-step process you just have to add your IP of the servers in the Ansible inventory and you just need to run to a script one to set up a CA server and other one is to set up the client so you can use this you can contribute ways so that's all from the talk here are the references any questions so here for the example purse purse I used a CA you just need that certificate file only so you can keep it anywhere you can keep multiple copies of that if you it's a it's a Ansible script I can add it in the roadmap to do that as well yes yeah so the question is role-based users are associated with the users or the system yes yes so you create one file and on the servers you create one file where you have multiple roles let's say there are DevOps there is a backend and there is front-end there are three roles you will add in the new line separated way and while generating the intermediate certificate you will add the strings like backend DevOps or the front-end in the comma separated way security of the public key so if that is compromised it can be easily hacked so it's a public key you might be having everything in Ansible code your public key your CA and whatever CA high yes yes how you are managing that because I have everything I can I can you know collate that information and access the system whatever is required so what you can do is instead of keeping it the private certificate at one place you can replicate it at all the places and the other thing you can do is giving access to this infrastructure the CA infrastructure it is not recommended so you should give this access to the limited persons only to avoid this kind of situation because at the end it's a single single place where which is governing all these things so pulkit will be around here outside maybe you can catch him later so thanks a lot