 In this lecture, I'm going to show you how to create SSH keys. You can probably already create SSH keys using either Puttygen in Windows or SSH keygen in Linux or Mac. But you can actually have Goalign also create your SSH keys for you. And that's what I'm going to show you in this lecture. I'm going to create as a package so that we later on can use this package in other lectures because some cloud APIs like AWS allow you to create a key with their SDK or their API calls but that is not the case with every SDK. So you might want to know how to create those SSH keys in Goalign itself. I already wrote my main function so that we don't have to go over that anymore. What do I do here? I have a private key, public key of a type byte and I'm going to call that generate keys and the generate keys function is going to return a private key and public key. And then I'm just going to write the mykey.pam and the mykey.pup in my current directory so that they have those keys ready to later on do something with it. I'm using this package which is also defined in my Go mod. So whatever I put in my directory here will then be within the package. So it's a very simple layout. I just have my current directory that will contain my Go files. So here I have package SSH generate keys and it will then return this private key and public key. So here I have SSH which is this package which I initialized right here. So I did a Go mod in it of my current directory. So this current directory here is the package and this is called package SSH. I don't have the CMD key gen with it may not go in but this is just as a test to have an executable. If you only wanted a package then you could even remove the CMD. Just have key gen just have this function generate keys and then you could import this from any external package and just use it the way you would like. So generate keys and then I'm going to write this to my local files. To make this work I still need to write this generate keys. So what is an SSH key? This SSH key that we want to create is of format RSA. We actually have an RSA package in Golang that can generate this key for us. So from crypto RSA we have this generate key. We need to supply a random source and we need to say how long it needs to be and then it will reply this private key. So this is fairly easy. I can just say rent reader and then it will import from crypto the rent reader. So make sure that you are using crypto rent and not any other rent package because the crypto rent is actually the safe package. To cryptographically secure generate a random number. You can choose how long it is. I'm going to pick 4096 which should be enough currently to have a decent key size. This returns an RSA private key and error. So our private key error is RSA generate key. If there is an error we can return an error generate key error and then we have this private key. But this private key is of RSA private key. We need a PAM format key to be able to load it using the SSH command or to import it as an open SSH key in Putty. So with Putty you would have to use PuttyGun to import an open SSH format key. Here we will be able to use SSH command because the SSH command supports open SSH format. It supports this PAM format. With Putty you would need an extra step. Private key. So this is just the bind but we still need to convert into PAM format. So there's also a PAM package in Golang. So we can say private key PAM equals and then we have PAM and here we can generate a PAM block and then it will import the encoding PAM package. This block is a struct which has a type headers and bytes. So the type is going to be RSA private key. We use curly bracket because it's not a function, it's a struct type, it's going to be RSA private key and then we have the bytes. The headers are optional so we have the bytes, the decoded bytes of the contents, typically a DER encoded ASN1 structure. So what does that mean? That we still need to convert our private key which is just bytes into a format that PAM would accept. So we have the bytes and to convert this to the format that is acceptable for PAM we can use the X509 package and this X509 package has a Marshall PKS CS1 private key. There's an RSA private key to the PKCS format so this is the ASN1 DER format and this is the kind of key commonly encoded in PAM blocks of type RSA private key. So to know all these things you can read the documentation or you can just find some examples on the internet and read what all these functions do. So we pass the RSA private key here and it will return bytes. So now we have a PAM block that we can already return so this is private key PAM block and if you would do private key PAM dot bytes types or header then it would just only return one of these parts the bytes or the types or the header that is empty. So we need another function to return this to parse this whole PAM block and then return it as bytes. And this function is called the PAM encode to memory and this accepts a PAM block and returns then bytes and we also apply a private key and then we still need to get a public key and the public key needs to be in a format that open SSH can work with. So we're going to use a SSH package for that. The SSH package is not included by default so we need to import it using go get. We'll download it and then we can add it here go along.org the crypto SSH package and the reason we'll do the SSH dot and we need a public function new public key new public key takes an RSA public key which we have in the private key so we have private key and in the private key we have a public key because when you generate a new key it generates a private and a public key. It's a pointer so let's pass it as a reference and then we get back an SSH public key that we can then use public key and error and then it has an error we return the error new public key error and public key is of type SSH public key so again we need to parse this public key into a format that we can use like the authorized keys format if you're familiar with that so what we're going to do is SSH package has a Marshall authorized key and this accept the public key save this so these are the lines to generate a key create this pam block this is the how the pam block looks like but it will have the RSA private key here on in the top and then from this private key that we that also is associated with the public key will do the new public key from the public key which will then give us the authorized key format so let me just show you how that looks like we save this go run cmd key gen may not go going to run the generate keys and then we're gonna write my key pam and my key pup so now I should see those two appearing the my key dot pam is the begin RSA private key then you have the key here and end RSA private key the public key that is associated with it starts with SSH RSA and then is a public key is the public SSH key for this private key so now we can use this format for our authorized keys when we need to work with SSH so this pam format when you have this and your own windows you will need putty gen to convert this to a putty private key I actually have putty gen here so this is putty gen for Mac on windows you have UI but what you can do here is then convert a private key to the putty private key format so putty putty gen and then my key dot pam which is the input the output is a putty private key so it's minus capital O private and then you need to specify the output file my key dot ppk and this would then convert my open SSH format my key dot pam from the open SSH format or the they call it open SSH format in putty I think but it's just the RSA private key the pam format into the putty format and this is the putty user key file that you can then use in putty in the next few lectures I'm going to show you how to use SSH in go so the previous lecture we just create the SSH keys which was fairly simple you just output the private key and the public key now I want to show you how the interaction between the client and the server works to fully comprehend how SSH works it is actually better to first write the server and then the client to see all the complexities SSH itself is quite complex and even with the library there are a lot of things to show you that you have to take into account even when writing the client how does the connection work between our client and our server we are going to do authentication using a server key the my key dot pam and our server will then have the my key dot pop the public key of our private key which is also called the authorized keys if you have a server running with SSHD then you will for sure came across this so this is the authentication part I have the private key and the server has the public key to verify that I have the private key and then we can start communicating using the SSH v2 protocol so it's not going to use those keys anymore it's going to create new keys to do the actual encryption the keys that we are going to use is only for authentication and also on the client side is also going to verify whether the server is really the server that we expect if we have the server dot pop the public key of the server and this is typically the known hosts if you're using putty or the SSH client the first time you connect you will be prompted whether you trust this server which is basically asking where do you want to save this public key locally and from that point on you have this public key and if then the server would change its private certificate or it would be another server then you would know so if you are going to write our client in SSH in go then we can already supply this server dot pop because we know this upfront which is also the most secure way to verify whether the server is really the server it claims so this is the authentication in nutshell I'm first going to start to build this SSH server and then we'll have a look at the client so in the meantime that we don't have the go client you will have to use an SSH client I'm just going to use SSH in macOS which is open SSH which you can also download on Windows if you don't have it so try it out and see if you have the SSH command if not you can download open SSH for Windows as well you can also use putty but for putty as I explained in the previous lecture you would have to convert the private key to be able to log in to the server this demo of this SSH server brings together lots of different concepts like how to do networking and how to work with asynchronous calls so it's actually quite an interesting demo that brings together a lot of things I already wrote my main function the server may not go and my main function will read the mykey.pop which is necessary to allow the client certificate and the server.pam which is a server certificate that we still need to create so let me start by creating these certificates so if I do go run cmd keygen keygen.go this will generate the mykey.pam and then mykey.pop so this one I don't need anymore and this I'm going to rename into the server.pam and then always you need to rename the public key as well because the public key and the private key need to match they go together and that's going to be our server.pam and our server.pop and I'm just going to run this keygen again and then we're going to have our mykey.pam and pop which is going to be used by the client eventually and then I'm going to include this package the ssh package in the ssh package we have the start server and the start server is defined in the server.go going to pass this private key and the authorized keys so the authorized keys is basically one or more public keys that we want to give access to in our ssh server so how do we start writing this ssh server well again there's an ssh package so let's have a look at this ssh package so this is the ssh package and there are a lot of functions if you think about on a high level what we need to do is we need to parse our public key or our public keys we need to parse our private key and then we need to build a config and then we need to start a server so here you also have the client implementation but we are really looking for the server implementation so the best way to figure out I find is to have a look at the examples here you have a lot of examples we have the client example dial example which is also going to be for the client and then we have that new server connection so this is going to be accepting new connections so this is what we're going to need and here we already have quite an evolved example with different examples you have a password callback if you want to do password authentication and then you have a public key callback and then then it opens a port and then you have this new server connection this ssh new server connection and then you have the code to handle your request so let's try to write this from scratch and we start by parsing our alt-right keys so let's take this bit first because this example we can already use we need to import our alt-right keys and this code block will make sure that we can do that so we have the alt-right keys which is a byte and then alt-right keys map so we're going to store it in a map and these are our alt-right keys so we're just going to replace this alt-right keys well we're not going to do a fatal so we're going to change this into a return of an error parse alt-right keys error and then the way this works is let me try to save this first and i just need to add the error here so we are going to declare a map and the key is going to be our alt-right key and the bool just that we have it so we're going to put everything to true that way we also don't have duplicates so as long as there's data as long as there's bytes in alt-right keys then we're going to parse the alt-right key using these alt-right keys and this is going to be our public key that we extract and in the rest there might be more public keys we only pass one but we could pass more so this example actually already takes into account that you could have more and as long because this is a for loop as long that you have more keys in there then we're going to parse more keys and then put these keys in our alt-right map so you see here this is our public key this is the rest what is left over we put the string of this public key we put in our map we assign it true to it and then we say what is left over we put in our alt-right keys so if you're only using one key it will do one iteration then rest will be empty we assign empty to our first keys and then we are finished then we can already start building our ssh config so this is the ssh config the ssh server config that we are going to use and let's copy paste this and let's have a look what we need to change here so we don't want password authentication we rather just want to use keys so we can remove this and it actually says in the comments remove this to disable password auth and we're just going to use public key authentication so the public key callback is a function and this is the function that's going to validate whether our public key is correct and we have this alt-right keys map that contains all our public keys so you are going to return this ssh permission and then we can add this extension if you want so record public key used for authentication so then later on you could perhaps use it so then we return this ssh permission if our authentication is granted because if this key exists in this map then it's granted and otherwise we are going to say it's an unknown public key so this is our authentication part we have this ssh package taking care of all these things now we would like to start our server and to start our server the server itself to create the networking socket is not implemented in this package which is actually quite smart because then they don't have to take care of that it actually uses the implementation of the networking in go lang itself so it's going to use a net package and that you see here so here we have this net listen and this will open port 2022 or you can give it another port and then we can accept connections we still need to parse our private key as well we don't need to read it anymore so we don't need this code but we need to parse private key so parse private key we're going to add this private key to our config we're going to start the server and then if there's a new connection then we're going to accept it and then once it is accepted we still need to do something with this connection we need to make sure that the ssh package is going to send the correct data and respond to the correct data using the ssh protocol so that's why we need this new server connection and this new server connection uses this listener that accepts connections and also needs this config and then it gives us a lot of variables that we can then work with to communicate with our clients so let's copy all this paste this and then we need to maybe change a few log fatal f's let's start from the top parse private key private key so now we have a ssh signer type that we need then we add the host key which accepts the ssh signer now we load the server dot pam the private key in our config and then once the server config has been configured the connection can be accepted net needs to be imported i'm going to save this now net is imported you can put it on a different port if you like and then we'll say parse private key error listen error and then we'll also say accept error listener listener accept error and then this is the most important function this new server connection new server connection starts a new ssh server with c as an underlying transport so it does the handshake and if the handshake is unsuccessful it will close and return error the request and new channels must be serviced or the connection will hang so that will do afterwards let's just see if he would be able to log in so i'm just gonna remove these variables for now and then we'll see if he can start a server failed to handshake or i'll just call it new server con error and then we're gonna print something if it's successful okay go run server dot go allow so now our server is running and then like i said you can download open ssh for windows if you don't have the ssh command or you can use putty localhost and then you need to pass the p flag before after localhost to indicate what port you want to connect to and then you might actually get this error remote host identification has changed so if you already have four localhost on port 2022 you already have a public key in your system then you might want to remove that if you ever connected to localhost on this port it will already have a private key and then it will say okay you already have a public key in your system and this is going to be in the known host files because it tries to connect and it wants to check the public key of the server and the public key of a service different what we already have in a known host and if it's different it's just going to say that there's a problem so you want to either remove this or change the port so i'm going to remove this so i now removed the last line of this file and then i'm going to try again connection refused because what happened our server stopped working because we are not really in a loop here we are just only accepting one single connection and if that connection fails or that connection is in use we are not doing anything anymore so we will need to add a for loop here to make to make sure it keeps on running so maybe let's do that first so we listen to this port this only needs to happen once but then we want to keep on accepting we want to keep on accepting clients so we're just going to make a for loop of it and then we also need to make sure that we don't exit just like that so our returns might actually just need to be print f print f here if you have an error our server will just print it instead of exiting and then the server can just run forever basically until we hit control c or something and then we don't need this return anymore so this will run forever until the process gets killed or we hit control c or something let's try it again go run and what is happening now is that it's not iterating of this loop constantly this is actually blocking so as long as we don't have anyone connecting to our server we are just stuck at this line just waiting waiting until someone connects if someone connects then we will do the next lines so now we cannot establish the authenticity of this host so we're going to say yes we would like to do that and then we will not get this warning anymore because then the public key will be saved within within our known hosts okay what happened here we didn't supply a private key yet so our server crashed somewhere oh yeah it crashed here because we don't have these extensions pub key because we are not logged in because these extensions is only set when we are logged in so let's try to check for this if con permissions is not equal to nil then we can output this i guess let's try it out let's try to connect again that didn't work either because our connection is probably nil so if connection is not equal to nil and connection permissions is not equal to nil then we are going to output it okay that's much better new server connection ssh no auth passed yet unknown public key for Edward Viana which is my login here so this ssh client uses Edward Viana by default as my login unknown public key for Edward Viana let's try to pass this key the mykey.pam which is our private key and then we are logged in logged in with key sha 256 and then this fingerprint so now we are logged in but we cannot do anything it is even blocked even control c doesn't work anymore so you would have to probably click this trash to get rid of it if you would like to close it now so it's completely blocked because we are not doing anything yet we need to handle this connection we need to do still with something with this connection we would like to probably give them the user a terminal to then type something such close this the server and now the client is also closed and let's try to see if we can do something within the connection there's still some code that we can use from the example so we need to define here the channel the requests and then we can use those variables the incoming requests channel must be serviced so that was already something that this new server connection was saying that once a client is connected they need to service this connection and that we do with go which means it will run in the background ssh discard request rex so let's add that and then service the incoming channel and this is where we get into the protocol details channels have a type depending on the application level protocol intended in the case of a shell the type is session and the server shell may be used to present a simple terminal interface so we are indeed going to present a simple terminal interface to our ssh client because that's what our ssh client will ask for so let's copy paste this and let's start with that so i copy pasted this and i need an extra curly bracket here at the end need to chen chen chen and rex needs to be defined again so let's try to identify what is really happening here so the new server connection replies us these new channel and new requests and they are in goal line of type channel and when it is defined like this channel with a less than and a dash then it's a read only channel so this is a read only channel that we are getting back and this is not a read only channel so we cannot write to this channel but we can listen to this channel so what's going to happen here the incoming request channel must be serviced so that is the package that asks us that whenever we do a new service connection then we need it needs to be serviced so this servicing we do with go ssh discard requests and discard requests in the background will consume and reject all requests from the past in channel i don't know myself exactly all the details why this is necessary but it's in the package documentation so we're just going to use what they tell us to use and this discard requests will listen to what is happening on this request channel and then it's going to do what it needs to do to make the protocol work and it will do that in the background what are we going to do we are going to listen on the incoming channel so in ssh you have concept channel and in goal line you have a concept channel so that's why it's a bit confusing this channel is of new channel so for every new channel in ssh you're going to listen to this go channel so this go channel basically means that we are going to wait here we are going to listen to this channel in go and if there's a new variable on it the variable of type ssh new channel we're going to go into this loop and then we're going to have this new channel variable so for every new channel that is created in this ssh connection we're going to have an iteration in this loop then channels have a type and if the channel type is not a session then we are just going to reject it because we are only going to implement these session types and when we connect with the rsh client we only need this session channel type to respond to because the session channel type that's what's going to open a terminal for us or we're going to implement a terminal that is going to open when the client is asking for this new channel of type channel type so if it's of type channel we're going to accept it here we rejected it if it's not not session then we reject it otherwise you're going to accept it and once we accept it then we have the channel the requests and errors so here again i'm going to just print print this if you cannot accept the channel and what is then going to happen we then can respond to requests so within a session channel we can then respond to request so you can have a shell request or you can have a pty request or you can have an n request and then in the example it's only going to handle shell requests and we are going to change it a little bit so this runs in the background because this function has a keyword go so it's going to run the background and then it's going to go again to the for loop and then here it's going to wait for if there's a different channel so if there's only one channel we're only going to iterate at once and then it will just be kind of stuck here but if it's stuck here then we will never be able to accept another connection while we are dealing with our current connection so ideally we would run this also in the background so we can make another asynchronous function go handle connection and this handle connection will need the channel but probably also the connection so this chance is of neutral so i'm going to pass this and then i'm also going to pass the connection because we might need that later oh but this needs to be just chance and then and then i just need this in the function signature of handle connection so i'm going to create handle connection handle connection connection is of ssh server con and then we have this chance chance and this is of new channel so i'm going to put this here it's going to be my new function that runs asynchronous and then all this stuff here this for loop this is going to be be in my function and here it will go all the way back to this for loop and then it will accept new connections so this is how this asynchronous works so now we have this in the function i'm going to save this so then this runs in the background for every connection that we have and it will iterate over the multiple channels so if there's a new channel it will just iterate over the next one this we are not using yet so i'm just gonna put an underscore here and then i want to know what requests that this ssh client is going to make so let's do the following for requests i'm just going to say requests made by client and then the request type request type made by client and then we can see what we have to implement let's see if this works wow we are logged in and then when we are making our requests then there's a request made by client pti requests and requests and a shell request and only the shell request was accepted because here we do a request reply so here we have this request ssh request we do a reply and if it's of type shell then we're going to say true so we're going to reply through here this is a boolean reply send a response to a request to a request so if it is shell then we're going to say okay you can get a shell but then we get this error pti allocation request failed on channel zero so ideally we also want to reply yes if we have a request for a pty rec so again this is like ssh protocol so you see you make a connection it asks three things so you would have to implement those three so what did i maybe not explain yet so this function here so we have to go this runs asynchronous so that we can go to the next channel if there is one this has a function and the function input is the ssh request so this is the request request is of type channel so this is another read-only channel this function will be executed straight away because this anonymous function runs in the background will be requests will be passed so here we have request is then called in and then when there's a new request when there's a new request a new ssh request that is being put on the channel then we are going to run this for loop so this for loop will run indefinitely within this function in the background for every new request that that is coming in so here we have three requests so we run it three times this loop for every request that was being put on the channel the next step would be that if we get this request of pty request or shell that we provide our client here with a terminal and a terminal is basically comes from a long time ago in unix where you would have terminals but this is still implemented within ssh and other programs where you have a terminal and in a terminal you open a shell for example these are also two terminals here and they support things like resizing and your cursor can basically move so to implement this ourselves would be quite some work so in goline we also have a terminal package that we can use let's try first with a switch right here on rack type if rack type oh and you can see there was a request made for a window change so you see depending on what we do there might be some more requests being made you can implement all of them if you would like but that's not really the goal of this demo the goal is to have just something working so that we can experiment with sending commands because what do you do typically between servers you often just want to connect to a server execute some commands see what is happening and then exit and this is typically what you need within automation jobs you just need to send a command or you might even have to run a server like an ssh server that can receive commands so that's what really we want to build here using go and within our function where we here get the request to open a shell we kind of want to give a shell and then maybe implement some commands so first we need to open this terminal so we don't use this terminal package for it if we receive shell then there will be certain actions if there is pti request there will be certain actions and then there will be a default where we don't really implement that and then we can just say we don't implement it so we say we reply false and here we reply through so here in pti request i'm going to open my terminal and then even n we might capture and say we implement this there's also payload by the way so even we could capture this payload which is bytes so in case of n then there might be some environment variables being passed let's keep it simple for now let's try to implement this terminal and for this start terminal we need a new package so i'm gonna do go get go lang xterm and we're going to use this x term package term provides support functions for dealing with terminals as commonly found on unix systems let's try to implement that and we're gonna put in a new function create terminal and we're going to pass our connection and our channel channel was the one that we got returned here so channel channel connection it just depends what you need but we need for to create a terminal we need this connection and channel so let's say connection channel what do we go to return maybe not yet anything maybe an hour later connection is of ssh server connection and channel is of type ssh channel term dot new terminal and where are we going to open this new terminal so this channel is actually a read or writer so this channel this ssh channel influence read write and close so we can use this interface read writer that can read and write to this variable channel so this is a ssh channel so this is not a go lang channel anymore there's just an ssh channel that we can write to send data and receive data so we are going to say open me a channel using this term package and we're gonna have a prompt this is going to be a prompt then we have a terminal instance that is being returned to term terminal then this channel also needs a defer channel close so when the function is finished we are going to close our terminal and then we want to do something with this terminal this terminal instance also has a function read line so read line whenever our user using our assist client is going to type something then we want to read this line read line and error so read line is going to be blocking so we want to do that in a separate go function and we want to keep on doing this so we're going to make a loop so we're going to have this loop here and then as long as our channel is open we're going to wait for new lines if there's an error what should we do when there's an error maybe escape the function or maybe just print it let's print it read line error just so that we have all the error logs and let's then break the function to break the for loop and then the functional clause then we have line let's use another switch and what should we check for case i'll just try to come up with something who am i which is not the same as the who am i command in linux but we can just use it as an example command so who am i then it returns a login here we can say who am i and then we can send a response default we can say that we didn't understand this and then we can send it back to the client so how do you return something let's have a look in this term instance and there's term instance dot write and we can just write a buffer and this will turn the amount of bytes written and an error but yeah let's just keep it simple for now we could capture all the output and all the errors but let's just try to fire and forget to see if there's anything working you are and then that's why i also send this connection you are and you need to do an sprint maybe so that it's nicely formatted you are s what are you connection dot connection dot user and if it default we will say we will say command not found and if we just press enter then the line is going to be empty we don't do anything we just keep it empty so we have a switch close our for loop and then we have a go function and our go function needs to be executed as well but there's no argument so i'm just gonna put these brackets here and then i'll go function automatically be executed immediately if error is not equal to null this looks pretty good so let's try to see what's happening so we are in a channel new channel has been created of type session we accept the new channel then we get a SSH channel returned and this request so this request we can then use so if there's a new request then we'll go in here then we'll be waiting here for those new requests this in is an SSH request if there's a new request then this will fire and we'll say request type made by client and if the request is pti rec then we create a terminal and actually because this is in the background it will also just continue here all the way to the beginning and then it will wait until there's a new channel to be opened so everything is pretty asynchronous and we could even have more code here if you want to execute it just when the client connects and it didn't make any requests so then it will create a terminal term instance is our terminal so this will start a terminal and then it will give us like like here we have the dollar sign it will give us a greater than sign we'll make sure the channel is closed at some point read line who am i what is this a sprint f it is and then if you don't understand the command we'll say command not found so then we have a little shell that we can try out so this is not really a shell comparable with bash like we have here so the user cannot really do anything it just has this little shell that we interpret the commands and can then execute something so this can be nice when you have a server that users need to log in do certain actions server is running oh and connection is immediately closed and i get a read line error because it's end of oh and i see what i did i put the defer channel close here and then it's executed when this function is finished because this create terminal function will finish but then this go function here is separate and it will finish later so it is this defer i want to have it in my go function which will keep on running forever so only when we have an error for example a read line error we will break and then when we come here at the end then the channel closed that's what we want let's try that again allow yeah this seems to be working better we made this request three times we say okay we can make request but only once we executed something and then we have a shell and what does that mean to have a shell i can type something i can go left i can go right i can remove my letters i can say ls okay command not found so all these commands are not found because i am just checking here with my read line what commands i'm sending if i press enter then it's a case of empty let's now try who am i you are at the word piano because that's the user that i connected with how do i exit here control c and then you see the other read line error and then you go out or what i can also do is i can also make another case for quit and then i can say goodbye on the terminal goodbye and then we're gonna close it and how do we close it channel close we're gonna close this channel let's try this again allow nice and we try who am i and quit quit says goodbye connection closed so that also works that's pretty nice so we now have created with go with just two packages that are from golang.org which means that there are from golang but they just don't follow the stability problems that they have because these are very stable here the apis could change a little bit but those packages are very reliable still we only use two packages that's sage package and the term package one is for the ssh client server and one is for the terminal support so you see using the default packages and just some easy to use concepts we have written a complete ssh server that opens a terminal where you can enter some commands interpret those commands and then execute some code you could also execute commands in the background based on what you are getting here or you could even pipe everything to a shell but that's not the point of this server we just want to have a shell where you can just type some commands so this is nice to have to then test our golang client but when we are going to use our golang client we just want to send one single command we don't really want to have our full shell and this is also supported within the ssh client because you can also add commands here you can also add commands and those commands will be executed but the way that the ssh protocol is implemented this ssh client will not ask for a pti and nsh shell it will just ask for exec so nth is still there but it will ask to execute something and then it says exec request failed on channel zero so in the next lecture what i want to show you before we're going to start with the client is how do we execute single commands over ssh which then uses the request type exec and then i think we are ready to start implementation of a client let's pick up where we left so we do the ssh command and instead of opening a shell we want to send a command so then we did an exec request so what did we really hit here that we have this exec request failed we have here the default which captured the exec but we need to have an exec for this request type and what do we then need to do we need to execute our who am i if what we supplied is who am i so let's just see what happens now and now that we have this exec here we reply through install files well we reply through but what now nothing happened nothing really happened so let's have a look what a payload is payload and then rack dot payload contains a payload rack type is a type rack payload is a payload and this payload is what we will have to analyze payload is who am i so that's good that we know that the payload is who am i so that we can now use to send something back let's make a function of this execute something and we're gonna pass a payload the payload and then what we can do is we can make a function execute something payload is bind and then let's give maybe a string back and then we can reuse this up and we can say exec something payload but payload is just gonna be who am i because here we already know what it is and it should be bind exec something and then we return this but when do we return this only if the payload is only if the payload contains who am i and then we have a default what could be the default just not found maybe command not found i'm not sure if this n needs to be there probably yes just to make it nice okay is who am i okay let's convert this to a string command not found and then let's also output the command and what is the command to payloads oh we need the connection and still connection is of type ssh server connection and then we also need to pass it twice here connection and the connection here and then so this returns a string so then how do we reply this back let's have a look what's in rack not much what else do we have here channel channel right we can write something back to a channel channel right exec something and then we just need to convert it back to bytes because channel right expect bytes and then we have the replies we say reply true we can reply that we can handle this type of exec and then we also write the response let's try that out sounds logical command not found who am i and i was expecting this because when i was preparing this lecture i figured out something interesting that is not easy to debug immediately because it has command not found who am i and i'm just checking here for who am i but how does this who am i really looks like if we translate these two bytes so if we have exec and then we will say event the print f let's say v and v and the first one will be bite who am i and the second one will be the payload so i want to see in bytes what who am i is in bytes and what the payload is in bytes and then i'm going to get a difference and the first one mine is just a string so 119 104 and here i have just in the payload some data before this where am i i'm not 100% sure what this prefix is here it's something in the s h protocol that i'm unaware of that i haven't read about yet and i'm just going to strip it because i don't need it but if i would need it or if i'm doing something wrong i would have to correct it later on in my github repository unfortunately but for me it works and for the goal client it also works so hopefully it will work for all the clients so how do you strip this i can say payload bytes trim trim left so that's also i can trim my bytes from the left to remove this zero zero zero six so this zero is actually null bytes so this doesn't really mean anything but i'd still need to get this six away as well bytes trim left of the rec payload which is all bytes and then the string trim left and these are stream left bytes maybe let's have a look trim okay trim prefix accepts bytes that's probably gonna be easier to deal with so i'm gonna say zero zero zero six i want to remove that from the beginning of my bytes and then i'm gonna send this payload save this i can remove let's try to run it again and let's try to run my command again okay who am i you are adward viana but what's happening here it is keeping the command open it's not closing it so there's still a reason why it is waiting for input from my channel i can just close the channel after i get the exec i guess because because if it sends the exec maybe you don't need the channel anymore and this is just how i want to do the implementation basically i'm not 100% sure if it's all rfc compatible but if i'm writing my own client and it's only my own client connecting i don't really need to be fully rfc compatible in my own script if i just want to do it quick and dirty if you would want to write a real ssh server and for people to download it you probably want to be rfc compatible so that every client can connect but there are already a lot of ssh servers out there i don't think you're gonna release a new ssh server these ssh servers that i am typically building are really for internal use to give a shell or be able to send some some commands over encrypted using ssh keys so that we still have skewer connection or you might even use sshd and only write the client so i'm going to save this go run exit again and then here i closed my channel after i send a reply and then i get the correct output now there is something to add to make sure that you're compatible with multiple clients because the go client is going to want to see an exit code so an exit code is if you aren't the exit code of the previous program then exit code was 255 which is an error code if you just do who am i and i ask for the code then i get zero which means it is successfully exited and this is something that clients will look for so so we can actually send another reply to our client and just like we have request here we can also send requests to our client we can say channel send the request send a channel request and it'll wait for reply if we want to reply send the request we can say exit status do we want to reply no and what is our payload is just going to say our payload is four times zero which will then translate in an exit code zero and our goal and client will look for that so if you don't have this line then our goal and client in the next lecture might complain so we just want to make sure that we have it here as well so that's it let me just test this another time to make sure that it is still working and it is still working and also if i now check for the exit code i see zero instead of 255 so that's probably better that we send this along also something that we are not doing is this nth we could also have an case nth and use this payload to set the environment variables or have a look what it is sending you are free to improve this ssh server i was even thinking of maybe making a separate directory in my github repository so that i have the server that i covered during lecture and maybe a separate one for everyone who wants to do prs to have some improvements because there can definitely be some more improvements and then in the next lecture we can work on a goal and client that we could use with this server but maybe also with other goaling servers let's start now with our client so i created the main goal in cfd client and i just have a few lines that already wrote here just to make it easy to read the mykey.pam and the server.pub this mykey.pam is my private key that i would need to log into the ssh server otherwise i wouldn't get in this is the same that i pass here the mykey.pam and the server.pub is the public key of a server this is the public key that is now stored in known hosts if i use ssh because the first time i connected it asked do you want to save this host and then it saves a fingerprint of this public key now here what i do is i already pass the server.pub so that i'm sure that i'm connecting to the right server then we need to parse these to the private key we parse with the ssh parse private key and we pass them the bytes of the private key and then we have the public key parsed which is being parsed by the ssh parse archrised key so this archrised key is the public key and this archrised key gives back the public key the comments potential options any other keys that would be in this public key but this is only one so i'm just ignoring this variable and a potential error and then how would i connect to this ssh server i already showed it to you in the previous demo there's a dial function so let's have a look at the documentation there's the dial function so again the crypto ssh documentation and here we have the dial function either you go to the dial function or you go to the examples there's also an example so let's have a look we have the dial client dial and here's an example dial starts a client connection to the given ssh server it's a convenience function that connects to the given network address i mean the network address which is a string initiates the ssh handshake and setups the client for access to incoming channels and requests use the net dial with new client connection instead so this is a higher level dial function that we can use if we don't want to deal with channels and requests like in the server so in the server we have to define it here in the client we can just use this higher level function that we don't really need to deal with channels and requests and we can use some higher level functions that will then initiate the request that we need search like a server we need a client config we need the public key so let's just copy paste this client config then we're going to change this because the authentication method we are going to use is not password but is key based and then we're going to dial protocols tcp this is going to be localhost 2022 and then the config so i'm just going to copy paste this config the username can be username you're not really checking on it and the authentication method is going to be public keys so public keys has the correct return method which is alt method we are looking for alt method so we can actually have multiple alt methods if you want it but we are only going to use the keys ssh signer the signer is this one private key parsed this is the private key that we supply and then the host key callback is the public key that we need so it's a fixed host key it's a public key parsed because this is the ssh public key so these are ssh config localhost 2022 i think it was and let's see go run seem declined go i think it connected did it connect maybe not as clear as just to be just to be sure oh it logged in with key so it connected but it didn't do anything there's no requests going so how do we make sure that there are requests going so that we can execute something or you could also start a shell if you wanted to but starting a shell you could as well use the ssh command so what i'm going to do is i'm just going to execute who am i just to see how that works because typically when you write a go client that does something with ssh you're gonna just execute a command show the output and an exit client new session client new session opens a new session for this client so let's try this new session is session and error session error new session and then we need to do something with this session let's just see what this function does so go run what does session do nothing yet okay well let's continue session now that we have this session keyword we just need to add a colon here now that we have this session keyword we can do something with it so we have run shell wait or we have output output runs a cmd on the remote host and returns it returns its tenant output that's kind of what we are looking for we just want to send a command and show the output so this session we also need to close when we are finished so we're going to say the first session close and then the output is a byte and an error so out out byte and then out error equal session output what are we going to execute who am i and like three if i just define it like this then i don't need this output variable okay if there's an error then session output error session output error and if there is a new client error new session error if there is a new session error then new session error and then we can just output out i guess output is out so let's see what this is going to do it's not doing any requests right now but i expect that this output is then going to do an exec request with the payload who am i and the output will then be outputted here on screen let's have a look go run client.go and we see that the only request that is made is the exec request because the session output uses the exec request and will then collect this output and you are username and why is this not Edward Vianna because the username that we supplied this time is username and not Edward Vianna you have to supply a username but we are not really using the username in our server we are just checking on this key if you wouldn't supply this key then we wouldn't be able to to log in so if i would not supply this key what would happen then handshake failed no authentication passed what would happen if i would pass a password it would be the same it would also give us a handshake failed because we are not supporting any password method if you just want to use password you could change the server and then implement the password method check on login and password maybe even with a third party database and then allow users that way so this is how we can implement a client so this client is pretty straightforward thanks to this high level function that we have dial we don't really need to deal with these sessions and requests because our session output is doing that for us we just pause the command it sends the command using the exec just like we did with our ssh command that was being passed here and then it just outputs the outputs or we have the output in this variable that we can then use so this is actually quite interesting because we could have like scicd pipeline where we have this client for example that would make a connection to an ssh server and execute something show it on screen and exit and this all you could then potentially integrate with other go packages to integrate this with an api or with a product in alias or with something in kubernetes so just being able to programmatically make a secure connection to a server using the keys that you probably already have is quite powerful in this section i'll be talking about identity providers i'll mainly be talking about open id connect or oidc let's start with what is an identity provider simply put an identity provider abbreviated idp manages and maintains identity data for users it's often used in conjunction with single sign-on sso it gives a user a single login and password an optional multi-factor authentication capability it can be used for multiple applications and websites and while very convenient for the end user it's also more secure so just think about this as being able to log in to multiple applications and websites with one single login and password and potentially multi-factor authentication two often used implementations for authentication within an identity provider setup are oidc so this open id connect and seml oidc open id connect and seml security assertion market language are authentication mechanisms they don't store login password information themselves so it's an authentication mechanism they don't store the actual login and password to validate your login and password when you're logging in you would still need to validate the login and password and potentially the mfa token with a separate mechanism users can be in a database in ldap in max of the active directory or other technologies where user credentials can be stored seml 2.0 was released in 2005 while open id connect oidc was launched in 2015 you can see that open id connect is much more recent than seml seml is still used a lot but open id connect is more lightweight and much easier to implement why implement oidc in this section i'm going to implement oidc and you can follow along the lectures will be built in a way that you can code some of the endpoints first before checking out the lectures so why implement oidc it's a great learning experience there's a flow to follow there's jlot involved there is api calls that need to be done using rest so implementing this yourself will be a great learning experience to learn about authentication flows and to actually do the implementation yourself you will get the exposure to a lot of technologies amongst others it will be rest wow jlot and jwk you're often exposed to an identity provider and it's worth understanding the inner workings so authentication is something that happens a lot it's implemented in a lot of projects so getting a good understanding of how authentication works how oidc works is definitely great to have you can build your own identity provider authorization server a client or application so depending how you're going to the implementation you might be the client connecting to an application that is using oidc or you might be the application that needs to implement oidc or you might be the server itself i'm going to implement all of them but in the end you might not run your own authorization server you might use a third party for that but then you would still maybe need to implement the application or a client so it's still definitely worth to understand the full flow and to have a look at how all these components work together understanding how the flow works will help you when you need to build one of these components what is oidc oidc is short for open id connect it's a simple identity layer on top of the o out to 2.0 protocol oidc can verify the identity of a user using an authorization server oidc uses rest endpoints so it's easily implemented so rest endpoints are just the normal endpoints that we have been using in this lecture already get imposed using json oidc uses this instead of for example xml that some users so it's pretty easy to implement oidc uses json and json web tokens jwt json web tokens are used in a lot of applications we have used it already before in one of our demos oidc also uses jlts so we are going to make use of those in the oidc flow there are different oidc flows that you can implement you have the authorization code flow which is the one we are going to implement in our demo it's for web applications that can store a client secret so if you have a back end application a back end application can store a client secret without a client knowing because the code will be parsed in the back end you don't see it on the front end so it can store a client secret to make the connection to the authorization server this is the flow we are going to implement and then you have other flows that you can implement for example when you only have a front end so there's an implicit flow for front ends and mobile apps that cannot store a client secret so if you cannot store a client secret because you only have front end code then you could go for the implicit flow and then you also have a hybrid flow and the hybrid flow combines the authorization code flow and the implicit flow it has immediate access to an id token so where the authorization flow you will see there's multiple steps with the hybrid flow you don't need to have multiple steps so let's talk a little bit about the flow itself this authorization code flow that we are going to implement on the left you have the user which can be the web browser then we have the web application which can be a goal and back end and then we have the authorization server and the authorization server has an authorized endpoint and a token endpoint at least those endpoints to connect to to be able to authenticate our user so what's going to happen our user is first going to connect to our web application it's going to access protected resources or maybe there's a login button that needs to press to login to the web application then the web application will send a redirect to the user so the redirect is an HTTP redirect to the authorization server so the application will say i'm not responsible for login go to the authorization server so it's the user that then will go to the authorization server to handle the authentication part and this is done using this authorized endpoint the user the browser will go to this authorized endpoint to ask for authorization to do an authorization code request it will ask for a code of this authorization server and this code can be requested using this authorized endpoint then the authorization server might send another redirect to a login prompt so this is the login and password box that you see on websites this can come from the authorization server if you're not logged in yet that the authorization service asking the user to login so then the user will send its login and password to the authorization server if the login and password is correct then the authorization server will send again a redirect to the user and it will redirect the user back to the web application with a code query parameter the first time that the user hit the authorization server was to the authorized endpoint for this code request now that the user is logged in it will actually get this code as a query parameter and this code it can send back to the application and because it needs to go back to application is the authorization server that will redirect the user back to the application these redirects are widely set so the actual redirect will come from the application but they are widely stolen authorization server so the user cannot be just redirected anywhere it needs to be a specific redirect to a specific URL so then the user will follow the redirect and it will go to the web application send this code query parameter to the web application so that the web application can read this code and then the web application can exchange a code for a token so using the token endpoint the web application can go there to the token endpoint and ask for a token which will be a JLT token so here you see that the user gets a code and not a token so why would the user not be able to just get to the token endpoint and get the token itself it's because of this secret that is being stored by the web application so our web application has a secret that it will pass to this token endpoint and authorization server will only give a token if you have this secret so the user doesn't have access to this token it only has access to a code and it cannot retrieve this token itself it is the web application and only the web application because it has a secret that can retrieve this token and then the token endpoint will send to the application the access NID tokens and potentially a refresh token to refresh those tokens when they expire so that's the complete flow a logical overview of how to authenticate and this is the flow that we are going to implement in go we're going to build this authorization server the web application and then we're going to use a browser to log in to this web application which will redirect us to the authorization server we log in to the authorization server and then the web application will see that we are logged in and then we'll be able to retrieve some of our user information because in this ID token there is actually user information like a login and it proves that we are logged in and that we then can access the web application on this diagram we only have one web application but the idea of ORFC is that we can have multiple applications and only one authorization server that takes care of the authentication of our users distributes the codes to the users and the tokens to the applications and then the applications have user information can make API calls using the access token and extract some user information from the authorization server for every user or make external API calls for this user so it proves that the user is logged in but we can also make API calls because once we have this JWT another application could also verify whether this token was issued by the authorization server the implementation details will become more clear once we start implementing this flow so how does the flow work on an HTTP level here on the left we have the user here on the right we have the authorization server if we going to the authorize endpoint then we're going to do a request to the authorization endpoint the exact name of the authorization endpoint can differ it depends on the implementation of the authorization server here I called it slash authorization the user is going to supply a client ID the redirect URI and this redirect URI is then the application server so our application server here is running on port 8081 and we do a redirect to the slash callback page so this slash callback page will then get a code basically we define a scope the scope is open ID we can add other things to a scope if it is supported by authorization server like a profile if you want to receive profile information or things like offline access if you want to have refresh token these things all need to be implemented we are just going to start with a simple implementation with scope open ID the response type is code so we expect the authorization server to give us a code and we also need to pass the state so the state is a random string that we supply that then the authorization server will then get back to us then the authorization server is going to say you are not logged in so go to the slash login page and the slash login page will show a login and password box that the authorization server will then validate if we are successfully logged in then the authorization server will send us a redirect back to the application server so the application server is still running on 8081 and it will redirect us to this callback URL so this callback URL is typically whitelisted in the configuration of the authorization server so that we cannot just pass any redirect URI that we want it needs to be whitelisted it will also specify the code and the random string so the code is the code that comes from the authorization server and that we send to the web application so here below we see that we do the redirect to the web application now the web application has access to this code and it can request from the slash token endpoint a jail t token based on the code that we are giving the application so the application is going to a post on slash token grant type is the authorization code it needs to specify the client ID and the client secret and only because it has the client secret it can receive this token it cannot receive any token just based on this client secret it also needs to supply this code that the user has been given to the web application because it entered the correct credentials on this authorization server it also passes the redirect URI that was specified just so the authorization server can validate it still the combination of this client secret and the code makes the authorization server give this jail t token so then the application will then get the jail t tokens that can then be used as proof that the user is logged in now that the web application has these jail t tokens it can still validate whether those tokens were issued by the authorization server there's a jwks.json where the public keys are available so the web application can download these public keys from the authorization server and using those public keys it can validate whether the jail t was indeed issued by the authorization server those jail t's are encrypted by a private key of the authorization server so using the private public key algorithm when we have the public key we can validate whether this jail t was indeed signed by a private key and that is what the application can do once we have the public key from the jwks URL we can validate whether the signature of this jail t is indeed signed by the private key by using our public key so the last step when we receive the jail t's we can still validate them this validation can happen at any point as long as we have the public key available so other application servers could also validate a jail t as long as we have the public key available the exact implementation details you will see in the demos and once you start doing the implementation i'm not going to go over all the details because they will become clear once we restart writing code this section is also a challenge you can write the code of these components yourself first so i will provide instructions first so that you can test it out yourself you can write the code as an exercise and then once you have written one function or one call you can validate whether you did it correctly using my demo lectures so if you would like you can write the orc implementation yourself first i will include instructions to step by step write implementation the start project contains already the function signatures and i have written unit tests for those functions to be able to test the validity of your code so you can write one function by one function and i have unit tests written so you can just execute a unit test for a specific function to see whether you wrote the correct code the start project is located in github under oidc start so this is the github URL the goal line for devops course so in there there is a directory called oidc start and that's where you will find the function signatures and the tests the actual implementation has not been written yet in that code and the solution code is in the folder oidc demo so i will write also in the next lectures the correct code starting from oidc start and we'll end up eventually with the solution code that is located in the folder oidc demo before every demo i will supply you with some information so you can try to write the code yourself first in those functions and then i will do the implementation myself and explain all the details that you need to know to write the oidc implementation to get started with the oidc demo i have created this oidc start that already contains the function signatures it doesn't really contain any logic just some function signatures and some boilerplate code in this lecture i want to go over the files and diaries and the go files that we have in this oidc start project and this is where you should get started so you can just download or copy this directory somewhere and get started yourself so the first file that i created here is the go mod file so the go mod file defines that this is a module if the oidc start module uses go 1.18 and has two dependencies one dependency is the jlt dependency so i'm using from the golang jlt project the jlt dependency i'm not going to encode and decode jlt's myself i'm going to use a library for that you could actually do that yourself if you want to maybe you could start with the library and then if you want to get rid of the library you could try to encode and decode jlt yourself it's not super difficult but it would definitely make this project a little bit longer so i opted to use this golang jlt it's used a lot this library so that's why i don't mind using this library i also used in my other projects and our second dependency is the ssh demo and we just use that to create our rsa keys because we did it in the ssh demo you could also just copy out this code from the ssh demo if you don't want to use that as a dependency we're going to have an app server and the authorization server in the app server i have the main.go and every time we need to start a web server so what's going to happen here is we have the listen and serve which is going to create a server and it's going to listen on port 8081 we're going to have an index and a callback because the application needs to be able to parse this code at a slash callback so that's part of the flow and at the index we will just have a login button these are our function signatures two calls that we can make and i have a separate file here jlt.go and in jlt.go we have a function signature for the get token from code so once we have the code we can retrieve a token and the reason why i put this signature here this function signature here is so that i have a test so here we have test get token from code so when you run test it will actually try to run this test and because we just return nil nil nil it will say claims is nil but if you write the actual code in here to get token from code you can use this jlt test to validate whether your code is correct and then here in your callback you will need to use that get token from code so it's an app server so here you just have two endpoints the authorization server will be a little bit more complex we have also a main.go and in this main.go we have a little bit more boilerplate code to get our server started so what do we have here we're going to read a configuration file because we need definitely a configuration file to store values like the client id the secret id the callback that the callbacks are wire listed so we definitely need to keep some config so what are we going to do here we're going to see does this config exist here we will have to put the config file the name of the config file i'm going to use a yaml file but you can choose whatever you want you can also use a json or tumble file those are all valid and i'm going to read the file here and then we have in config just the bytes and what else are we going to do we're going to check whether we have this encryption key because we need an encryption key in the rsa format which is the same as our ssh format that we were using so we are testing does this encryption key exist if not we are going to generate a key and if the key didn't exist we generate it and we also write it out so that the so that the second time we execute it we are not generating a new key anymore we are just going to read this private key so we need a config and a private key in the rsa format well here it just bytes because there will be a pam format format but then eventually we'll have to convert it to the rsa private key format and then we'll start our server we'll start our server passing the htp server struct from the htp package just so that we can easily pass the port if you want to change the port you can change it in the main instead of going to dig into this package and we pass the private key and the config but for the config we are already going to pass this byte into a server config struct and this server config struct i already have defined so i already have some boilerplate just so that we can agree on the attributes in our variables so that when you write a code it's not going to be completely different than my code so this starts uses the server package and this server package is defined in here in pkg so if i have a look at this server package you see we'll have all the files most of these files are completely empty right now just with the function signatures in let's have a look at this config maybe first and then at the start server so this config where is this defined right click go to definition so here we have the read config which is currently empty so you will have to pass this config which is just bytes because it's a file that we pass a file content and we have to return this config how this config look so in server we have types and in the server types we just have this config defined so right now we have an apps which is a map the key is a string and the value is the config so we can have multiple applications our authorization server can support multiple applications and for every application we're going to keep a client ID, secret, issuer and the redirect URIs we're also going to find on a root level the URL and we also have load error that we can use internally if there was a error while loading the config we can put it in here the URL will then be just the URL of the server so we don't have to define it for every application so this is the types and then in htp.go we'll have this start server so here we have this start that was called from the main so the start requires this htp server so that we can pass this server address and to also help us with the testing the private key and the config then we do new server which is defined right here new server is struct where we keep this information so that we can access it at all our endpoints new server we pass the private key and the config and then we can use the functions that we're going to define for every endpoint we have an authorization endpoint token endpoint login endpoint jwks.json so these are explained already in the overview this as well so this is the authorization this is the first step this token is to exchange a code into a token the login will show our login page and we will also be able to do the actual login there jwks.json is going to show our public keys so we can validate our tokens this well-known open id configuration is just going to be a discovery endpoint so that we don't have to hardcode all our URLs these discovery endpoints will then just show us a json with all the token and the authorization and the jwks URLs so we don't have to hardcode it and the user info is an endpoint that we can call with a token it will then give us a json of the user info the user that you pass a token of and we do the listen and serve let's have a look at the tests we have an htp test here that's going to start a server and then it's going to test whether we have all these endpoints this test should already work yeah it passes so it starts a server and then it checks whether we have all these endpoints and then we are just checking whether the code falls in a certain range so if then something goes wrong then this test will then stop working and then you might have to make changes to your endpoints so we have the main goal the server and then we have all these files what are all these files in authorization we just have the authorization endpoint in config we read the config in discovery we have the discovery endpoint which is also just a signature so right now it just returns nothing we have the htp file which are even over jwks also just an empty endpoint the login the token the types and the user info they are all empty and we will have to one by one write a code for every endpoint to start working so let's say that we write the authorization code here then we can run the authorization test we can run the test authorization code and now it will fail because we are expecting this location header but there is no location header set because because we haven't written any code but then once you write the code this test should pass so that's why we have all these tests so every underscore test has actually test files to help you to see whether the code that you have written actually works so we'll do this step by step we also are going to focus on the oidc flow itself we also will have a user package for the authentication so once that a user enters a login and password then we're going to validate that login and password and that's why we have the user's package for but we are not going to do any complex authentication any complex login and password checking you could if you have a database or LDAP or active directory somewhere that you you would like to use you can change this code and make a connection to your directory server here to validate a login and password so here you can change the login and password into your name for example so it's easy to login and this is only going to check our authentication and we'll then return a user so this is the user that has been statically configured in this package so you have the Edward user that i'll be using to login to the authorization server we'll also have this oidc package and in this oidc package we'll have a few helper functions so for example here we have the parse discovery which takes url and then just returns the discovery utility so this discovery this discovery endpoint is basically just going to be returning a JSON in the structure of this so you have an issuer authorization endpoint token endpoint user info endpoint so this is standardized and you should have this endpoint which is going to make it a lot easier for your application servers to be configured because you will only have to pass this discovery endpoint so this parse discovery will just go to this url do a get request on this url and try to unmartial this json into a struct and return a struct so i left it here because it's just such a simple api call so we're not going to spend too much time on this it just goes to this discovery endpoint and gives us a discovery struct then we have this jlt.co but i actually moved this to here to the jlt.co so this one we don't need then we have a get random string in the rant.co which will just generate a random string based on the amount of characters we want to return here so we have n which is how many characters you want to return and we are going to randomly generate something randomly generate a string base 64 encode this with url encoding so this url encoding is important because we want to pass that as a craze string attribute so this is going to be base 64 just random so every time we need a random string we are going to call this function so no need to write this yourself it's pretty simple it just uses the crypto rant generates generates a random string we're going to base 64 encode it and whenever you have an equal sign we're just also going to remove it so now we just have a random string with no characters that could conflict in our url and then we have the types so i already discussed this discovery type but we also have a token type and a jwks type that is just an array of jwks keys so here we have our public key information so this is also standardized so this is standardized this is standardized this is our token endpoint what is being returned from our token endpoint and this is also standardized because this is all standardized i created this oidc package and these types are going to be all the same for everyone who is going to write an oidc implementation so that's why i already put them right here just so that we have some boilerplate code so i could also write my test because in my test i'm also loading these structs and it will make our life easier once we start writing the code for all our endpoints so oidc has these standardized types and just a few functions then the server is going to be a server implementation and the users is going to be our users implementation that you might want to change if you would like to be able to login with a real login and password so that's it you can download this oidc start package and then we can start working on the implementation of the first endpoints in this demo i will parse the config file so right now if we would start the server it would show us an error the config file doesn't exist to solve that we would first need to have a config file so we need to define this constant here config file because that's what we are checking on here and then we're going to read the config file and we are going to parse the config file so we have this server read config so this read config will have to write using codes that will parse the config file you can choose the format of the config file yourself i'll be using yaml so first i will need to define the config file constant i will call my config file config dot yaml and my config dot yaml is going to be right here config dot yaml in the main directory because that's where i'm going to start the server let's have a look at our types again so in types you have the config consists of apps URL load error and then within the apps we have a map with client that declined secret issuer and redirect URLs let's start with URL and we are going to host it on localhost 8080 currently we can change that over time so right now it's localhost 8080 but we might change it over time because if you use a service like ngrok if you want to have it accessible on the internet or we deploy it somewhere we might change this and then the apps and the first app is going to be app one app one is our map key our map key and then we are going to define the app config client id i'll just say one two three four client secret it's going to be our secret issuer is going to be the same as our URL because that's the server that is going to issue the jdle at the and then we have the redirect your eyes which is an array and the localhost 8081 is the app server so that's what i'm going to use as a redirect server name but then the path is going to be the callback so we are going to allow to redirect back to our application server on the callback URL i'm going to save this and we then need to parse this in this config config.go this config.go will do the config parsing using this config type but this config type doesn't have any metadata yet in the struct so let me add that first if we are going to use yaml we can add yaml metadata and that can then be read by a library so there is a library for yaml available the go yaml slash yaml yaml support for the go language the yaml package enables go programs to comfortably encode and decode yaml values it was developed within canonical as part of the juju project so we can use this one it's been based on the lib yaml c library but it's purely written in go there are more libraries available so it doesn't really matter exactly which one you are going to use this one is a good one you can use this one or another one to install it run this command go get the yaml v3 and now we'll be able to use it in the config.go let's now make sure that we put the correct metadata in place yaml apps and this then need to match exactly how we wrote our config.yaml so if you have a look at this config.yaml apps right here all lowercase and you see here we are using camel case first a lowercase and then uppercase if we start a new word so we just need to make sure that matches url load error we're not using client id client secret issuer redirect your eyes save this and then we can do the parsing works very similar as how we are parsing our json instead of json unmartial and marshal we do yaml dot unmartial the binds and then the interface which in our case is the config variable this returns an error if the error is not equal to nil we are actually not going to stop the program we're going to say config dot load error equals the error and then we are going to return the config just so that we can later capture the error and then do something with it rather than already returning an error so if we have a way to start without a config we might as well do that so we are just waiting to return an error here we're just going to return the config but we are encapsulating the error in our config variable save this and save this let's try to run our server and I think our server is now running localhost 8080 page not found so that seems to be working so our server started we don't even have an endpoint for slash so that's actually normal to get a 404 I could try token okay and this token gives a 200 but no output let's maybe just see if we have our config output and then that's going to be end of this part of doing the config work and then we can start with these endpoints so this plus V is going to show us also the keys so we have the config apps is all applications is the map we have the key which is app one client ID we have a secret issuer to redirect your eyes the URL and we have no load error so it's all great I'm going to remove this line again save this and we can continue with the next part of our server going back to our OIDC slide that shows the flow with the htp request so this authorized endpoint on says authorization is the first endpoint we need to write for the flow to start working it takes a client ID the redirect URI so the client ID needs to match the client ID that we put in our config file the redirect URI also need to match the redirect URI that we put in the config for that specific client ID so these things we will need to check we will need to check whether the scope is definitely not empty we are only going to support open ID but we can as well ignore that or just save it for later use and then check it later we are not really specifying in our config file what scopes we can receive open ID is for OIDC but you can also have an offline token and other scopes that gives you more functionality response type is code we need to check for that there's only valid response type that can be passed so it just says that we are going to then reply with a code and then we also need to check for the state the state is a random string and that state we later need to give back so you need to also save all this information somewhere on the authorization server we can validate the information right now save it somewhere and then we need to redirect the user so we need to send a redirect URL to the login page once it goes to the login page we also need to track the user because it will be entering its login and password information so when we do the redirect we want to give some random ID like a session ID with it so that we later on can track what user was actually logging in so we have a match between the client ID and all the information that it supplies to us and the actual request so we cannot need to track the request as well when we are going to redirect this user to other endpoints so we don't lose track of the user itself so let's try to implement this so we are going to implement this in the authorization function let's declare all the variables that we need to have and then we can start checking on them and getting them from the request so we have the client ID, reject your eyes, go, response type and the state that we need to receive so let's store the correct information in all of them and just check whether they are all not empty if client ID equals to and we have the request in R and then the URL in the URL we have the query and we can then do get get gets the first value associated with the given key the key is going to be client ID so if client ID equals to client ID and even the client ID is empty we want to draw an error how we want to draw error we probably want to do return error and then write a function to return error and then say what the error is client ID is empty and then we want to return to stop our function so this return error we don't have yet so let's try to write this I'll just write an HTTP I'll just have a function here funk return error w is HTTP response writer and error this is not a pointer like if something goes wrong there's a header HTTP error code 400 that we could return back it's called bad request bad request is an int of 400 and we can say write bite and then we just return error and we also going to print on screen so we know that we drew an error that's easier for debugging just like that and then we can always use this return error when we want to say there's a bad request because something is missing and we just need to show the error to the end user so now we can do this for all these variables let's just copy paste this five times so now we copy paste this five times and we are checking whether client ID is not empty redirect URL URI is not empty scope is not empty still need to change this and this looks all good except that the response type we can say if it's not equal to code because we're only accepting code here now we are sure that the client ID is not empty let me just check this one more time client ID client ID redirect URI scope response type state all looks good now we have for sure the client ID it's not empty so let's try to find the client ID in our config and remember our config is in this server variable so we can loop our config apps for apps in as config apps for app if app client ID equals our client ID then we can say app config is an app config we can now assign this app to our app config if we didn't find anything then app config will be empty if app config client ID is empty we're going to return error client ID not found so it's not empty but now it's not found I also need to pass our w and this needs to be rf and return the client ID is empty but we still need to validate the redirect URI redirect URI is an array in YAML so we will need to iterate over that found is false I will say and then for redirect URI config range app config which is now our app config that matches with our client ID redirect URIs if the redirect URI config equals to the redirect URI that was supplied found is true so if you don't find this redirect URI that was supplied in the request in our config then we also need to we also need to draw an error return error redirect URI not found in config or not widely set what's that the next step I was talking about tracking our user with a session ID because we want to keep this information somewhere if you lose this information we don't have this information later on in the later stage so we'll need to store it somewhere and we can store it in our server struct so we'll need to make a new type that we then put in our server struct so we can keep these variables so I'll just copy paste those those are the ones we're going to initially keep and I will make a new type login request with those variables if I give them capitals and they're also accessible outside a package which might be useful later on maybe not in this demo but later on if you need access to this login request at some point outside this specific package I'm going back to authorization and here I need to store them in a server the server is declared in htp.go htp.go and then here I'm going to make a new variable how am I going to store this request information per user I'm going to create a new unique ID I will call it a session ID and that's how we are going to track this information on a per user basis so login request is going to be a map and then this string key is going to be a unique session ID if I make a map I also need to use the make function to initialize the map make right here save this and then now I'm going to say session ID is going to be OIDC get random string and I'm going to make it pretty long so it's definitely unique if there's an error then return error and we're going to say get random string error and we return the error and then we also need to stop from this function and here as well we need to stop the function so once we have a session ID we can say a login request within the server so this s comes from the server it's a pointer so changes that we make here are actually saved the next request as well so if later on we hit another endpoint this login request will contain the same information login request is a map session ID is going to be my key and then I'm going to have a new login request struct with client ID redirect URI scope response type although I probably never will need this just for completeness and the state this will be saved until the server stops so if the server stops it will be gone so if we stop a server then the sessions will be unknown to the server because we are not persisting this to disk but for our use case that is fine if we stop a server the sessions can be as well done if you would like to run this in the production environment and you want to keep the sessions you would need a session store like a ready server where you will store the sessions for example that if you have multiple authorization service running and then you deploy a new version that you still keep check of the sessions so now we have our login requests what are we going to do we are going now to send the user a redirect with the session ID so that we can keep track of the session ID a redirect is done at the header level so if you do header add location I'm going to redirect our user to slash login and that's where we will have our login page where the user can enter his login and password and to keep track of this user now that is on a different page we're going to pass this session ID and we can do fmt asprint f to make it nice and the session ID is going to be session ID and then as a header as the htp status header we are going to say htp found status found which is 302 so we found the location we found the page and we are redirecting the user to a different location I'm gonna save this and like I said I will have tests available to see if our implementation is working correctly it doesn't test on everything and I will probably add tests over time so just have a look here and see how many tests there are right now there's one there might be more in the future you do run test and what will this one do it will build this endpoint based on app one and it's going to do new requests so it's using the htp test package doing a new request and then checking whether we have the htp status found and whether we have location so redirect uri is empty location ahead of not set so we might have done something wrong and yes we have done something wrong we are checking for redirect uri but we should be checking for redirect uri with an underscore I'll do the other ones look correct run again and now we passed our test we got the location location is slash login and this is a session ID so this test is enough for now if you wrote something wrong we will see it in one of the next lectures because then we are going to build this login page there might always be a mistake in your code that will come out later once you test the whole flow and that's why I'm probably want to add a little bit more test cases over time this one is already quite decent actually because you see we are passing all this information and actually this is wrong now that I'm looking at it we don't pass the client secret let me just correct this run the test again and the test still works because we are not using this client secret so there's no reason to pass it in our test so client ID is a client ID redirect uri is the first redirect uri response type is code and the state is a random string so it looks all good we can continue with the slash login implementation in this demo I'll be handling the login flow so now that we went to the outrise endpoint we got a redirect to the login endpoint so we have initialized our flow and now the authorization server should get us a login and password page where the end user can then login click submit or login and then we are going to do a post request it makes most sense to do it on the same endpoint so if you are going to log in we do a get request we get a login box we do a post request on the login endpoint with our login and password and our session ID and then the authorization server should check whether our login and password is correct if it is correct then the authorization server should reject us to the redirect uri that has been specified when we went to the outrise endpoint so let's try that out let's see if you can put it in code this is the login.go the login endpoint and what are we going to do on this login endpoint first if it is a get request we are going to show a login page and i already have a template for that we are using this embedding so these lines here they make sure that the templates folder is actually embedded in the binary of go so that we don't have to supply it separately so the template directory is right here in the template directory we have a login.html so we're basically bundling this with the binary and to bundle this with the binary we can use the slash slash which is a comment but it will be interpreted by the go compiler go colon embed the template directory and the templates directory will now then be accessible using this template fs variable so here i put a comment to access the login template use the template fs.open templates login.html which will then load this login.html file the contents of this file into the template file and then we can use this template file so let's try this out if there was an error with loading this file we can do a return error again and we can say template fs.open error error and then we can output the template file. What is a template file? fs file so we still need to read the contents of that file io read all can do this io read all expect an io reader and template file is of fs file which is also implementing an io reader so if i do io read all template file i will have all the template file bytes and i will get an error if something went wrong this needs another return and this i want to copy paste read all error and template file let's have a look at this login.html first so this is what we want to show the user it's just an hml file it's a login example you are free to changes into your own login form so it's a login form so we have the login input and the password input that if we then going to click this submit button here this login button then we'll do a post on slash login we also have room here for a session id so this is type of type hidden so that means that we will not see it on the screen but it's still being passed as a variable to the slash login endpoint when we are doing a post on it so this session id we can actually then replace with our session id because when we are getting this redirect the redirect is happening on slash login but we are also passing this session id this unique code we have generated so let's try to extract this session id replace this in this login.html and then not just output that to the user and then users should be able to log in so we should have done that first probably how do we read this again our URL our URL query get session id make sure that it is exactly the same session id is a string if session id is then empty let's return an error to the client so now you have a session id template file string equals we are going to do a replace of the session id strings replace we can use for that strings replace and let's copy this just to make sure we don't make any typos strings replace session id into the session id oh and we need to also pass this template for our bytes so replace is first a string then the old string and a new string so the string where we want to do replacement needs to also be of string now it's on bind so i'm just going to put string around it so now it will convert from bytes to string then we replace this string and we look for the string session id and we replace it into session id how many times minus one is no limit so minus one this is a new variable so a colon and then i think we can just output it right template file string and then we indeed we need to convert it to byte that's what visual studio did for me and did we miss something i don't think so let's just test this to be sure go run and allow i will just use curl but you can also just use your browser local host 8080 login session id is empty okay abc and now i get the html and the session id here is filled out to replace also works so now if i open this in a browser or if i go to a browser to the authorization endpoint then i get redirected then it will show this html i click submit and then there will be a post request happening so before we are going to test the flow in the application server let's just finish this login endpoint by splitting out get and post and if there is a post we need to try to log in then redirect to the redirect to your eye with the code and i think if you have that we can start working on the application server because we can then already test part of the flow if our method is post then we are going to do something else we are going to check whether use is authenticated and otherwise we are going to just return his html when we do a post request we can use the post form variable post form contains the part form data from patch post or put body parameters so we first get a login we press the submit button then we hit again the login with the login password and session id and then we need to be able to parse these three elements and to be able to parse these three elements we can use the parse form so they say this field is only available after parse form is called so we first need to do a parse form our parse form and it doesn't take any parameters but it can return error so if you say error equals parse form what if there's an error then we're gonna return the error return error parse form error and then return out of the function and if that doesn't give any error then we should be able to use post form get and we should be able to use our session id so it makes sense to first retrieve our session id because before we go into test our login we want to make sure that we have information on this user how do we know whether we have information on this user well if we have the login request session id set login request okay equals login request session id if okay which is a boolean if okay is set to true then we have this key in this map the login request map and then the login request will contain the login request if not okay then we can return an error login request login session session not found so now we have the login request that has been created at the authorization endpoint as login request session id so now we are just checking whether we can find this and now that we have this we have the redirect URI we have the scope we have the state and we can use that next we can do the authentication flow so we have this users auth that we can use users auth now users is imported and we can pause login passwords MFA if we have MFA but we don't really have MFA implemented here so I'm going to use an empty string and then it will be returning three variables whether we are authenticated the user and whether we have an error what do we pass our post form post form get login and this needs to match the input boxes so the input boxes from the html form were also login and password if we have an error authentication error now we can say if the authentication was successful we do something if the authentication was not successful we probably need to return an error error we can reject the user back to the login page with an error or we can just output something to keep it simple I will just output something w write header htp stages unauthorized w write byte and then authentication failed the authentication was failed if we are authenticated though then we need to do some extra work we need to do another redirect so we're going to do another location we need to do another redirect to the request urign that is in the login request fmt sprint f login request redirect urign so we redirect to this redirect urign if the authentication is successful but we also need to supply the code and the state so we have a code that we still need to generate and the state is going to be login request state so we're just going to return back the state so that the app server could potentially check this the code is going to be a unique string again so we can just generate a unique string we just need to make sure that if we generate a new string that we can still find it somewhere on our server and also we need to do a w write header htp status font otherwise it's not going to work so let's generate a new code oidc get random string how long doesn't really matter 64 sounds about right there's also an error that comes back get random string error but how can we then later on find this user we'll have to create something new in our server struct otherwise we cannot find this we also need to save this user object this user object looks like this we want to save this this is some user information that we need to know in the server if the app server asks for some user information we need to be able to return that so why do we need to be able to keep the code it's because there will be another request on slash token and then the app server will supply us this code with the client secret and we'll then want to have a jlt token but we can only return this jlt token if we know what code we issued so we're going to store this code somewhere let's go back to htp and let's create another map and let's just call this codes we do initialize it again codes make map of string and we can reuse this login request so now that we have the code we can say as codes we use code as a key and we say it equals to login request but now we also want to add some more information because we still have this user variable that we also want to store and we should also keep track of when those codes are issued so that we can have a timeout on them so let's go to the types and let's just add a few fields here code issued add time time and user user's user so let's now add these extra fields login request code issued add time now and they should be code issued add code issued add login requests user is equal to the user and then we assign this to the s codes that is a map with all the codes in there with the login request and then it has a code and it maps to a login request now that we have logged in a user we could as well delete the session id we have the s login request session id that we want to remove so how do we do that delete we can use for that so delete the map the map is the s login request and the key that we want to delete is the session id so then we also remove the session id from the login request so that we cannot log in twice is there anything else that we need it looks good let's have a look at these tests we have a test login get which passed here we are also passing the client secret which is not necessary it's because i copy pasted this i will have to remove it everywhere because it first does authorization flow and then is going to do the login flow so we are doing this flow and then i will have to remove this client secret everywhere so here we are doing the authorization flow the login get flow and then the login post flow and we are expecting a location and a code so let's try again this test login get that works test login post and this also passed so we have the redirect ui the code that has been generated and then the state a state we are just passing a random string so that's fine if you change your login values in the users package then you might want to change them here as well so here we are doing the login get flow then the login post flow and then we are checking where you have the location so this all looks pretty good so i think we are now ready to start working on our application server we'll still need one more small thing before we can do that but then we can already test our authorization and login flow to see if we get a code back and then afterwards if we get a code then we can start working on the token exchange once we have a code the application server can actually go to the authorization server with the secret that we don't know here the client doesn't know it the browser is not gonna know it the application server is the only one that can go to the authorization server to get these tokens it's time now to start working on the app server but before we can do that we need to make the discovery endpoint and the reason why we need the discovery endpoint is that we don't really want to pass to our application server all the endpoints that are possible like the outrise endpoint the token endpoint the endpoint where we will find all the public keys there are a lot of endpoints in oidc so there is actually a mechanism where you specify just one URL the discovery URL and that discovery URL will reply all the URLs in the JSON and then within the app once we want to use oidc once we need to know a URL like the authorization endpoint we can go to that endpoint do a get request get a JSON translate the JSON into a ghost struct and then we can use the individual URLs if you have a look at the types here is this discovery struct so this discovery struct is standardized so I already put it in place in the oidc package so the JSON is going to contain the issuer the authorization endpoint the token endpoint user info endpoint so we just need to make sure that we generate that in the server the server has this endpoint defined it's also standardized dot well known slash open id configuration so if you're going to pass an oidc URL to an application that application will know that it needs to go to the well known open id configuration to get the discovery information we just need to make sure that the discovery function right here actually gives the correct information let's try to build this endpoint discovery is a new discovery struct from oidc discovery and then we just are going one by one define all these variables the issuer is going to be the sconfig URL the authorization endpoint sconfig URL and now it depends whether we added a slash at the end for the URL if you didn't add a slash here then you need to add a slash in the discovery you can still do a string replace on double slashes or actually parse every URL that's also possible authorization is our authorization endpoint token endpoint is slash token user info endpoint is user info j wks uri is going to be j wks dot json so we only did the authorization endpoint for now we still have to do all these other endpoints scope supported what are the scopes that we support currently oidc response time supported what are we going to respond a code token endpoint authentication method supported so let's say you go actually to the token what is the authentication method that is supported we don't have any authentication method supported now we need to reply json so json marshal json marshal this discovery if there's an error which is very unlikely but we still need to capture it call on here and then we can write the outputs go run main.go curl loclosed 8080 i'll just copy paste it to be sure and then we have it the issuer authorization endpoint token endpoint and if i pass this to jq then i can actually have it much nicer authorization token user info jdliks that looks all good scope support or idc so if we are going to support more we can add more to that as well offline tokens if you won't be able to refresh tokens but then we will have to add more code to a support all these so for now we are just implementing one flow so our oidc is actually good enough we're actually not going to do anything with these scope supported anyways our application server is only going to use this authorization endpoint and then maybe the user info endpoint is only going to use these endpoints here to use these scope supported is just going to ignore so that looks all good and the next step is then going to be to build the application server so in the application server we can now pass this one URL our authorization server and our application can then retrieve the URLs that it needs okay let's get started with the application server so what do we need to do we are going to create a login button and the login button is then going to redirect to our authorization endpoint and that will then handle the login process what's going to be our redirect URL this one localhost 8081 callback this will have to supply to the authorization endpoint how do we know the authorization endpoint well we'll first have to retrieve this well-known open id configuration and pass this let's start with the oidc endpoint the oidc endpoint we are going to pass as an environment variable now we have the oidc endpoint and we have this helper function here in discovery.go in oidc discovery.go will do a get request and pass this into the discovery struct so very simple function let's use this instead of writing this ourselves it's just some very simple htp get function oidc pass discovery what do we need to pass the endpoint so we have the oidc endpoint and then we're going to add this well-known open id configuration discovery equals discovery error what if we have an error let's do the same as we did with the server let's have a return error function but if something goes wrong on the application server it's most of the time going to be an internal server error which is an error 500 that's probably a better error code to use on the application server if error is not equal to nil then return error pass discovery error and then return let's start to build this authorization URL authorization URL is alias sprintf first we have the URL this comes from the discovery endpoint and then we need to supply some parameters we have to pass client id which will be supplied also as an environment variable client id we'll have the scope scope is open id we will have well let's have a look on our slide just to be sure client id we need to redirect ui the scope and the response type and the state response type code state is the state and what else did we need the redirect ui so this redirect ui is going to be a constant localhost 8081 callback redirect ui the constant and then what else do we have just the state and the state is going to be a unique string again so we can just say state error equals oidc get random string how long 64 should be fine and then we need to capture the error and we can pass the state i think we have everything now we just need to return something to the browser how do we return something well just some html we're going to return let's try backticks yeah that looks better body and then we're just going to have one simple button with a link and where are we going to redirect our user when they click the link to the authorization URL and we have the button login i think this should work it's not very pretty let's maybe just make sure that it's wide enough so we can pass a width that should do it let's try to get this started what do we need to pass a client id client id should be 1234 client id client id 1234 and oidc URL oidc endpoint and that is localhost 8080 and then go run let's make this a bit bigger maybe go run go run cmd app server main.go and let's now just open our browser localhost 8081 here we have our login button and this is going to redirect us to localhost localhost 8080 authorization client and d1234 redirect uri there's a callback scope is operandd response type is code and then the state is this unique string and i made a typo i see localhost it's here in the config and then i will need to restart my server let's try this again i will need to refresh this page otherwise it will still drag me to the old url now login and i get the form because now you're hitting slash login it's a get request we got the redirect so we basically first went to authorization then the authorization made the redirect to slash login and now we have the session id as well login and password so the login and password is going to be in the users package so that's why we do the authentication you can change the login there into your own name for example my login is adwart and the password is just password login and then we have localhost 8081 callback with a code code equals a random string again from the server and we are hitting the application server with this code the next step would then be for our application server to then use its client at the client secret and code to get a token we get a white page because we are not doing anything yet in slash code so maybe let me just put something there hit refresh to see if we are hitting this slash callback endpoint so we wrote the index and here you have the callback callback code is our URL query get code restart application server refresh this callback code is and this is our code this code is visible for the end user but this code doesn't mean anything because I could just change the code here in my URL bar we still need to go to the authorization server with our client secret to see whether we will get a token and only the token the it token will be the proof that we are authenticated writing this slash token endpoint is going to be the next step so now that we have implemented the authorization endpoint the login page both get and post now we are able to do the redirect back to the callback the application server gets the code and now wants to exchange the code using this slash token endpoint with the slash token endpoint we can exchange our code into tokens we do a post request with grant type is authorization code we need to supply the client ID the client secret the redirect uri again and the code and then the token endpoint will give us a token if everything goes okay so let's try to write the necessary code that we need to put in this slash token endpoint to make it work so slash token goes to s token and then we have here our function signature that is still empty what is the first thing that we need to check is a post request if it's not a post request then return an error not a post request then we need to parse again these post entries and for that again we need this parse form so if there's an error with parse form then we can need to return an error let's not forget this return here in case something goes wrong and if parse form goes wrong parse from error and now we can use the our post form get and we have the code but what else do we have this grant type so this needs to be equal to authorization code if it is not equal to authorization code then we again can return an error we can say invalid grant type we can always return what the grant type was so that it's a bit easier for debugging so what is the code now the code is our post form get code we need to do a lookup of that code so we can say login request okay equals s code and then the key of this map is the code code codes I think it is codes okay that's it if not okay we will say that we couldn't find a code invalid code if we have the code we still need to check whether the code is still valid login request had a code issued add if the time now is after I will save after code issued add and we're going to add 10 minutes if the time now so now we are a specific time if it's after the code that is issued at a specific time plus 10 minutes then we are also going to return that the code is expired so you have 10 minutes to complete this flow you can change these values in a bit less time or more time doesn't really matter what else do we need to be checking we pass the client ID in the client secret we also need to check that is the client ID equal to post form get client ID if it's not equal we can say invalid client ID client ID mismatch and we do the same for login request client secret which we are not saving in the login request so let's change maybe a little bit our authorization flow and a login request let's add the app config app config is the app config the app config i extracted right here somewhere see we have here the app config so that way we have the config information and login request app config client secret needs to be equal to the post form client secret otherwise it's also not good and then we can say invalid client secret if it's not the same next we also need to validate whether the redirect ui is still the same if login request redirect ui is not equal to our post form get redirect ui then we are going to say that the redirect ui is invalid let's have a look on our slide again we do a post we check for post grand type authorization code we have that client ID we checked client secret we checked redirect ui we checked and the code we also checked because if it's not in the map we had an invalid code so now we can issue these jlts how do we issue jlts well we can use the jlts library the colon jlts jlts let's have a look at the reference documentation and we have here examples parsing building and signing building and signing a token is what we need and here we have an example of jlts new with claims and the signing method this signing method is hmak which is a method that shares a secret we are not going to share a secret we are going to use a private and a public key so we will need another signing method we will use a signing method for rsa keys then we're going to have the jlts map claims these are key value pairs that will be in the jlts token that we supply these are quite random here foo and bar but we will use some standardized ones and then the next step will be to sign the token using our rsa private key rather than here this hmak sample secret first step is maybe to get the private key we have this s private key which is still in byte format it's in pump format but luckily this jlts has a private key function to convert this but we might have to import it first github.com golang jlts jlts version 4 i'll need to go get maybe i already did it earlier but it doesn't hurt jlts and then we have some parse functions here parse rsa private key from pamm parse rsa private key from pamm and then we just need to input a byte so that's what we need and then we get a private key and an error if the error is not equal to nil then let's return our error private key parsing error let's create a new jlts jlts new with claims is going to return as a token that's fine new jlts with claims the method jlts signing method so it's going to be rs for rsa and rsa 256 signing methods rs 256 should be okay the other signing methods like 512 just uses another sha algorithm but we are going to use 256 for our purposes then the claims which should be of jlts claims let's maybe do that separately jlts map claims we should be able to use map claims is a map string interface and that should work with the jlts claims because claims is an interface so when we do a new with claims it will generate a new token there'll be a header claims and a method type will be jlts algorithm will be then rsa algorithm and the claims will be here so these are key value pairs and these key value pairs we can define right here what are the key value pairs that we have to define these claims are very standardized so we have the claims standard claims and these are the standard claims structured version of the jlts claims set as referenced at here at rfc so there's an rfc where they explain what your key names would be they even say here they do not follow the specification exactly the use of this is this courage that's why we are not going to use it we are just going to use map claims and define our own claims which ones are the standard ones so we need the audience expires at issued at the issuer and definitely also the subject not before at we could use so let's start with the subject who is a subject we have our login request here login request user and the user has a subject so it is something that typically comes from the user itself it needs to be like a unique login then we also have the issuer the issuer is going to come from the config config URL is the issuer the audience so we're actually going to issue two tokens an id token and an access token the id token is for the application server to know whether we authenticated the access token is for the application server if it wants to make api calls so the audience for this id token is going to be the client id login request client id then we have the expiration which is in unix time when it's going to expire we can say one hour to keep it simple it depends on your use case how quickly you want it to expire if it expires a new one needs to be issued if we would work with refresh tokens then within one hour the token needs to be refreshed with a refresh token if you don't work with refresh token you have to go through the whole login flow again this is unix time not before should be time now and issued at is also time now so then we have this token but we still need to sign the token and i want to make one more change to this token so we have the claims but we also need something specifically set in the header for our jwks to work so the way this public key endpoint is going to work this jwks.json is that we need to specify a key id and because we only have one key we are just going to specify a static number 0001 is our key because we only have one private key and this is the key id so the key id stands for key id and this is something that we need to put in the header now we can sign this jlt with our private key token signed string so we're going to retrieve the signed string from this token and we need to get the private key for that this returns the signed token signed id token an error if there's something going wrong if something went wrong then we need to capture this signed string error error here how are we going to return this we have this oidc token for that so in types we have this token so this is access token token type reverse token expires in an id token that we typically should return at the token endpoint so we're going to use this one oidc token so this is also standardized token output is oidc token and token has the access token the token type the refresh token expires in an id token so we will actually need to create two tokens because we just created the id token which can be used to validate whether we are authenticated but we also need an access token let's start with id token is signed id token and then we're going to just copy paste this and then this will be our access token signed access token and then access token is the signed access token but the access token needs to be looking a little bit different because it has different audience and the audience is actually a string slice of the endpoint it can access so what can it access it can access any API that we would give access to as in once we have a second application server and we would enter that in the audience here then then the second application server could actually parse the jlt have a look in the audience have a look whether it has access to a specific endpoint right now we are just going to give it access to our user info endpoint so we have our URL and with this token the application server can go to our user info endpoint and we will then return some user info so this is the difference between access token and our id token token type it's going to be bearer it's going to be a bearer token so then we can use authorization bearer and then the token the jlt token in htp access calls what else do we have refresh token so if you want to supply a refresh token for example when the scope is offline access then you just need to generate a random string for here and to make another endpoint that if this refresh token is supplied that you issue a new jlt currently not going to do that we are going to focus on this first flow that we are trying to implement we have like sparse in as well expires in is an integer how many minutes will it expire in so we have one hour token output then we still need to parse it token output json marshal our token output out an error and then we can check for errors token marshal error just in case that we have an error and we need to do one more thing so we are going to just write this first to the app server where it's connecting which we also need to make sure that we remove this code so we have this code s codes where we keep all the codes now that we issued a token we need to get rid of this so s codes and we have the code the code is our post form get code so you also need to remove this code from the map so then when the app server would execute it again then it will fail because we don't have this code anymore you just have to go through the whole flow again if you want to have a new token you cannot reuse your codes that looks okay i think could be that there's still a mistake how do we check for that well that's why i wrote these tests token tests what is this is going to do first going to hit authorization endpoint and here i will also remove this secret because it's not necessary for our first endpoint i just going to paste it all this code multiple times so that's why this the same mistake is in all these tests so first hit authorization flow we get the code or the redirect we get the redirect we go to the login flow we get the code and we need to then exchange a code into a token and then we need the secret actually so this all and i didn't specify this last time this also uses this test config test config is defined right here in hp test and whenever we run our test we are going to do a setup of this test config which generates an rsa key so we don't need an rsa key just to run the test the test will actually generate its own key and we'll have here a map of some default configuration well let's hope it works no it didn't work and standard claims he should add needs to be of in 64 he should add added time now needs to be unix and that's an in 64 and this also needs to be unix so that's why we have these tests easy to then find whether your code is working or not still not working get on marshal string into ghost truck because we have two times we have it two times once for id token once for access token and this works got valid token from a token endpoint so i'm gonna leave it at that and then in the next lecture we can try to see whether our application server can now get this token with the core that was supplied at the callback endpoint in our application server now that we have written the slash token endpoint let's try to add this code to our web application to hit this token endpoint with our code and then retrieve our token that will be step one to retrieve a token step two will be to validate the token so i will first try to get a token and then we need to validate the token to validate the token we need the public key so we need another endpoint at the authorization server for that the jwks endpoint which will output the public keys we will need to parse that and then the public key we can use to validate our jwt that was returned or our jwts that were returned so let's first start with a post request on this slash token endpoint we have still the app server and then here in the callback we have the code this code we need to validate we have a jwt and jwt test so if you use this jwt signature get token from code then it will be easier for us to validate our code because we have a test in jld underscore test here we have a test to test to get token from code so let's try to use this code get token from code what do we need for that the token URL the jwks URL the redirect ui client ID client secret so we can use this we'll just copy paste it for now this discovery endpoint token URL is discovery token endpoint jwks is discovery jwks ui redirect ui is static client ID is client ID as an environment variable then we have the client secret which will also pass us an environment variable and then we have the code which is coming as a query parameter what does this return a token claims an error and we can say token received claims subject i'm not 100% sure whether i will use these standard claims because they are discouraged maybe just for the simple get token from from code but really map claims is much more flexible so we probably should be using that token received if token or we also have the claims here if you want token claims maybe you shouldn't even return the claims on token dot let's keep it like this for now save this and let's start working on this get token from code we'll have to do an HTTP post request on the token URL and with the post we have a URL content type and a body what is the content type going to be it's a specific content type for form encoding and because this is actually a special type there is post form post form issue as opposed to specified URL and then the content type header is set to application x www for your line code so that we don't have to set it manually token URL and then the data the URL values so URL values is of type URL values it's a map string and then a slice of strings we have some add functions that we can use what do we all need you need to pass the client ID client ID client secret redirect to your eye the code and this grant type authorization code then we have this post form responds an error if we have an error then turn nil nil error defer res body close and then we can read the body res body body error is IO read all because this is an IO reader if res status code is not 200 let's also return something we will then say status code was not 200 and I'll just output the body for now to the screen here to see something happened not sure what's going to be this maybe we should just return not yet fully implemented and then here in the main we can just say get token from code error as long as we didn't write the complete flow then we will get this error which makes sense I think so this get token from error there's one thing we didn't do and that is we didn't validate our state so we generated here our state we generated here our state in the index get random string but we didn't store it anywhere so let's try to store it somewhere we are not going to store anything in this state or match any user with this state so I think it's enough to just store the state and then see if we issued this random state string in the callback if not then we just reject the request if you are going to store something in the state or match a user or something else with it then I would recommend you to encrypt the state so that nobody can change or read it what's really inside and you can even pass it in a cookie with the client so that at the callback you could compare it with the client so I'm going to keep it simple and create a state map string boolean make map string boolean here for the state and when I create this state I'm going to add this the state called states states so that way I for sure know that the request originated from this index page and then here in the callback we are going to check if okay a states and then the state our URL cray get state if it's not okay then we will just return an error then we will just return error state mismatch error and then we can remove this state delete a states the state so that also means that if you're going to refresh this callback page it will not work anymore so we'll have to go back through the flow to get a new state should we try this out I cannot test my code yet because I'm just checking whether I'm getting a token we would still need to validate the token let's go and run the server client secret is the secret which is conferred in the config.tml or dc endpoint go run and then we also need to specify the jail at t.co let's open the browser localhost 8081 login we can supply our login and password information get token from code not yet fully implemented but did we get the token output on our screen yes we did so here we have the body the access token and the id token will also be swear there refresh token expires in id token now we just need to validate this token so this is jlt you can actually input it in jlt.io which is a simple json web token tool and what do we see algorithm rs256 the key id the type and here we have the audience expiration date issued at issuer not valid before and the subject if you would paste our public key here then it would also be able to validate our signature so it can decode it but it cannot validate it because we still need to validate it to validate it we need to get the public key and to get the public key we need to make this endpoint the jwks endpoint and that's what we'll be doing next now that our web application received our tokens we still need to parse and validate those tokens and to validate those tokens we are going to do a get request to the jwks endpoint and that endpoint will supply us the public keys the public keys that can be used to validate the jlt signatures we'll have to write the jwks endpoint where we will have to return a json with the public keys we only have one so we'll just have to output one public key in a standardized format and then our web application will get that json parsed that and then see if we can validate the jlt with that public key that we received from the authorization server if everything is valid then the token is valid and we can be sure that the user is actually successfully logged in we can trust now this token and the contents within it it is now sure that the authorization server created that jlt and signed it with its private key so this is where we left the body is the token that has been returned from the slash token endpoint so now we need to validate this jlt so what i'm going to do is i'm first going to parse this body of our token is of type oidc token and we're going to json unmartial this body that is the json into this token if there is an error then we're going to return an unmartial error unmartial token error so we know where it comes from and now we can start validating the jlt we're going to return the claims and i was saying that we are going to use the standard claims the standard claims because that's our there's needs to be a pointer our standard claims because that's our function signature but then if you have a look at these standard claims it says that they only support integer-based date fields and singular audiences so what do they mean with that you see here not before expired at expires at only supports integer-based fields which we are only returning so that would actually work for us but singular audiences let's have another look at our token endpoint and here we can see we have these id token and this has a single audience but if you look at our access token here we have a slice so we have multiple audiences and this standard claims can only deal with a single audience so what do they say here in documentation use registered claims instead for a forward compatible way to access registered claims so yeah i can actually just change this into registered claims and we just need to change this function signature and it is forward compatible so our test will still work because here we also still have the subject the subject is still a string but if you need to use the audience it's not just a string anymore and if you have expires at it's also not an integer anymore so this should be a better way of using claims what is then the next step we need to parse these token we have two tokens but let's start with id token parse with claims and now our id token should be in the token variable because we un-martialed it we're going to parse the claims so the parse with claims will have look at the claims that are within the token and put it in this claims variable and then we have the key function and the key function is going to be a function where we have to validate this signature of this jlt and here we'll actually need this public key to be able to validate it how does key function look like jlt key function key function is a function token and interface if we just copy this jlt token and then we need a return save this and then we don't have a compile error but we are not validating it yet what does it return a token and an error I don't think we need the token we just need the error because if there's an error then our token is invalid return nil nil fmt error token parsing failed and the error why it has failed so if if the validation failed then it will throw an error so what do we return here still I think we actually need the token yeah the jlt token so I will call this parsed token parse token claims and then we don't need to return an error just nil and it should be it we then need to start working on this validation code but we didn't implement our jlks URL yet so we'll have to do that first we'll first have to go back to the authorization server implement this jwks URL and then we can write a get to get the public key in this lecture I will show you how to do the jwks parsing so it's standardized so we need to output a standardized json and we will have to do that for every key we only have one key you have the private key in our server struct so as private key is a byte slice it's a byte slice because it is just the contents of this file encryption key.pam and this is a pampformer.file so we'll have to first parse this private key into an rsa private key object and I think we already did that earlier when we generated the token so let's have a look how we did that previously so here we have the private key and we use the jlt parse rsa private key from pamp let's copy that so now we have the private key and the private key also has a public key and this is the rsa public key so we can say public key equals the rsa private key public key now we need to output this jwks URL and because it's standardized I already have it in oidc declared so we have the jwks struct that contains the keys and for every key that is in this array you have an e algorithm use kid and kty key type is just going to be key type so what does that mean what does that mean if you have look at this public key go to definition public key has this and an e and an e here so basically this information is already known for us it's the modulus and the public exponent it's part of the rsa key it's properties of the rsa key so if we can output these in a readable format because this is just a big int we would need to convert this if we can convert this put this in a string make sure that we fill out the fields algorithm and so on then we actually already have what we need we just need to be able to output the public key in an agreed format the format that has been agreed on is to do it in base 64 so we'll have to base 64 convert it so jwks is equal to oidc jwks and here we have the keys which is of type oidc jwks key and here we have one element and this key we need to give an identifier so this identifier is going to be the kid and i'm going to use 0001 as kid because when i generated the jlt in token i specified this header 0001 as kid so this is the id of the key i used to encrypt it it can just be any string you just need to be an identifier what else do you have use algorithm key type algorithm rs256 key type rsa use signature these three are just some informative fields and then now we need the actual public key information n and e re is always going to be the same so this exponent that we are going to use is always going to be the same we can just define it as a static string so this is already convert from the exponent so if you have look at public key this exponent will always be like this so we don't really have to convert it maybe in the future if you have different exponents then you might need another one but for our purpose this will work for n we're going to use a public key n dot bytes and these binds we are going to convert because you cannot just send the binds into this json attribute we are going to convert them with base 64 so we're going to have base 64 base 64 standard encoding encoder string and that should be it now we can convert it json marshal this jwks out an error jlks marshal error if there's an error and if there's no error w write out let's do a go run of the server curl localhost 880 jwks.json uh that looks pretty good with jq so we have the keys the and the e the algorithm to use the k id and the key type so the next step is to go back to our application server do a get on this url parse this back into the jlks struct and then use this to validate our jl t now that we have the jwks endpoint available let's go back to our application server so how does this parse with claims work parse with claims will be executed and then this function will be invoked and we then need to return the public key here it accepts an interface so anything can be returned we should return an rsa public key and when we return an rsa public key then the public key will be then the token validation will happen with the public key so the signature will be compared this is all done in this jl t package we just need to supply this rsa public key to do that we need to first extract this k id so we have access to this token so this token is parsed but not validated yet so if you have a look at token header k id that will be the k id if you don't have the k id then we need to return an error k id not found if we have the k id then we should return the public key let's write a separate function for that so we have the public key that we need the error that it can be returned and we can say get public key from jl ts jl ts URL we need to supply and this k id we need to supply k id is of type interface but it's really a string so we need to still convert this to a string if we then have an error let's just then return the error if you don't have an error let's return this public key function signature get public key from jl ts jl ts URL is a string and the k id is a string what do we return an rsa public key and this will come as a pointer so i'm just going to return a pointer and the error so what do we need to do here to get this public key we need to do a get request on the jl ts URL htp get on the jl ts URL which returns a response and an error if there is an error then we'll return this error and otherwise we want to read the body so we're going to defer the rest body close body error is equal to io read all rest body if there's an error then we return an error and now we can parse the jl ts contents which is a json json unmartial data is in the body and we are going to parse it as jl ts oidc jl ts jl ts again if we have an error then we return an error if the rest status code is not 200 we're also going to return an error because then the json will probably not be valid and we should put that probably in front of the unmartialing we should go right here parse jl ts and now we need to do the opposite of what we did we first need to loop jl ts entry key entry hjl ts.keys so now we have the keys and we need to see if there is a key with the kid range here with the kid that is equal to mykid mykid here that's applied if it is equal then we can extract if we didn't find the kid then we can say no public key fonts with kid kid so how do we extract this again with base 64 standard encoding decode string jl ts key entry and then the n what does this return n bytes error if error is not equals to nil return then an error decode string error but the n is still in bytes and it is actually a big int remember when we were doing this jl qs then we had a public key and the n is in a big int format so we still need to convert the bytes back into a big int n is big from the big package because this is also big big int so we're going to do a big int new int new int accepts an int 64 and returns a pointer to a big integer we're just going to say zero because we actually want to do an n set bytes because we have bytes sets z to that value and then returns z but i don't think we need to return because it's just going to set n to the bytes and then we can return an rsa public key we don't need to pass an error because we don't have an error and n is n we don't really have to pass this e because this e is going to be defaulting to this so not really necessary here we still need to return error let's have a look it actually looks like it's finished now because we did the get request we passed the jl qs we checked whether there was a key id we recorded this string create a new public key with the modulus so we get the public key right here return the public key so that the jl t parsed with claims can validate the id token if we have an error then token parsing failed otherwise we'll just return the r token and the claims as i look at the main function so if you don't get an error here then it is valid let's run the server and let's run the application server and actually just before i'm going to run the application server i'm going to run this test because i have a test to test whether these actually works and it failed verification error token parsing failed rsa verification error so we still might have something wrong i have this jl qs endpoint here that is actually an hp test new server so i have this mock server that i created with a slash token endpoint and a slash jl qs dot json endpoint and here i just create a new key and we create a new token return that solely just a part of the server that is being mocked here and we get an error from the get token from code crypto rsa verification error and actually i had a look offline for a second and it's very logical where i made a mistake because i said earlier in this lecture that we can just use e because it will default to the value that we need but that's actually not true because you see here e is zero and the value that we need if we have another look at our jl qs it's this value and it's definitely not not zero it's not zero but it's actually constant still so if we just go back here add this constant here which you can find on the internet this exponent is a constant or you can probably also decode it from this but if it's a constant anyways we can also put the exponent as a constant and then let's run the test again and the test seemed to work so let's now run the app server and let's get back to our app server login and now we get a token received and also the validation should have happened so if we now go back to our main.go you see the claims now of type registered claims has this subject the subclaim and it didn't throw an error so now we can be sure that the user has been logged in the token is valid and we have the subject what is next let's have a look at our HTTP endpoint so we did the authorization endpoint the token the login the jwks the well known and the last one that we have here is the user info right here we have these claims from the registered claims but it doesn't really contain a lot of information and you also want to keep your jlt quite lightweight in general so what you can do is when you make the initial call which we do here we say scope open id you can also add more scopes you can add a profile or you can add email to have these included into your jlt and then you can pass them in your claims that's one way or another way is that we have also an access token and this access token and this access token can go back to the authorization endpoint and hit this user info this user info will then return us some more information about the user in json format in this lecture i will show you how to build a user info endpoint so this user info endpoint works a little bit different because now we actually have a token the user info endpoint expects that token in the authorization header and that is what you see here below the authorization error needs to contain the token and with the keyword bearer the id token is for the application to validate whether the user is logged in the access token is a token that we can use to make api calls so to make an api call to the user info endpoint we need the access token and the access token also has the audience claim that states the URL of the user info endpoint so we could actually also check on that to make sure that the application doesn't use the id token to get to the user info endpoint once the token is validated then the authorization server can reply some user info in a standardized json you can have a look at the oidc specification document if you want to see the specification of the json it's already something that we have in our users package so i'll be just returning the user here we have our user info endpoint what is the first thing we need to do we need to extract the token from the authorization header authorization header equals our header get authorization the authorization header has a prefix bearer so we can remove that string replace bearer in this authorization header do nothing we can also check whether the authorization header was supplied so if the authorization header was empty then we return an error authorization header empty jwt parse with claims we can use again token string is going to be the authorization header the claims we can use again registered claims you can also use map claims map claims is just a map and then you can parse yourself let's see how far we can get with these registered claims authorization header claims expect a pointer and then again this function signature jlt keyfunk you see i cannot remember any signature i just need always to have a look in the documentation ended and returns a token and an error do we need the token probably not because we can continue with the claims so let's see what we can return return private key are we still to parse a private key private key equals jlt parse rsa private key from pem s private key and this can return error if error is not equal to nil then return the error parse private key error error return private key public key nil no error that should work and then we get the error back from the parse with claims what if there's an error then we have an invalid token parse token error and return so if you come at this point we are sure that our token is valid but does our token have access to this endpoint so we can have a look where it has access for audience in claims audience but what is audience actually audience is a claim string claim string is basically just a slice of strings this type is necessary since art claim can either be a single string or an array so if there's a single string they will put it still in an array when they do the json-martial and that's when I do this they have this martial json and unmartial json which are the custom functions that are being invoked when they martial or unmartial json so actually I just need the range keyword that should work and now I have an audience found is false if audience is equal to the config URL was the user info then found is true and if found is false then we didn't find this and then we can say return error token has incorrect audience and we can even output the audience just in case that we get an error that we can easily debug it so what's audience of this token we can join this claims audience together with a comma so now we know for sure that this token has been issued using our private key because we have this validate function with our public key we are sure that the token has access to this endpoint because we checked the audience now we need to return the user the user is the subject let's have a look at our users package we did the authentication we return the user these are subjects so we can compare our subject from our token to the subject here and then return the user the user has the json sub name given name family name preferred username email and picture which is our standardized response that we need for users in for user in range users get all users if user sub equals to claims subject and let's also check if claims subject is not empty just to be sure it's empty then return error if something goes wrong in our in our authorization server and we would issue a token with an empty subject we don't really start comparing this so subject is empty just as a safety measure if we find it what are we going to do do json marshal this user and if there's an error then we're going to return error and otherwise we're going to write the output and if you didn't find the user that if you didn't find the user then we do a return error where we say user not found return we have a user info test that we can use to test our code that we just have written test user info and we get an error you get status code 400 power stock and error keys of invalid type let's have a look keys of invalid type and i guess this should be a pointer to the rsa public key because otherwise it probably doesn't check for both the rsa public key type and the pointer type and now it is working code user info json the subject name the given name family name preferred username email and pictures empty so this is the user info endpoint you can have a look at this test to see what is actually necessary to go to the whole flow so this is using the htp test package to do requests on the authorization to do the login flow and so on all the way until we hit this user info endpoint we are now going to do the same but in the application server in the next lecture so now we will need to hit this user info endpoint and see if we can display this to our end user so this is really going to be our last step for our oidc flow our first simple flow that we have in our oidc server our authorization server we're just going to hit this user info endpoint with our application server to test whether everything works and then we should be able to use this authorization server as an authorization server also for other applications that support the oidc flow let's have a look at this main so in this main we have this callback and then we have access to our token maybe let's return also the access token or let's just return access token and not the id token so let's copy this access token claims access token parsed access token access token also has the kid so we should be able to verify that and then let's just return this parsed access token and together also the parsed token or let's not do that because we have to change too much i'm just going to only return the access token because you are not even using the identity token we will just verifying it parsed access token save access token so now we need to do a call and we need to change the request headers so our call will look a little bit different new requests it's a get request the url is going to be on discovery user info endpoint and then the body it's going to be nil this is going to be request if error is not equal to nil return an error new request error request header add authorization in the authorization we have the bear and then the access token access token raw is a string so this should be the access token now we have the request we still need an htp client htp client and we can do an action using a request and this gives us back the response just like we would do htp get and post do request error defer res body close and let's get the body read all res body read all error and i just print out the body as senate token received user info and then the body i'm gonna make this fmt sprint f comma here res body like this we still have an error somewhere we are not using the claims save this run the server run the app server this is from previously state mismatch error so if we try to refresh it it doesn't work so our authorization server is doing this correctly we cannot use our previous state and our previous code to get a new token let's authenticate again and token received token validated we went to the user info endpoint and the user info endpoint gives us back this subject the name given name family name and so on so this is now the whole flow that we completed the authorization the callback we have created every endpoint that we need so if you want to plug in now another ydc compatible application that should work we might not have implemented the complete protocol so here and there you might have to add something for example if the implementation expects that it would receive some more claims if it adds the profile to the scope then you would also need to implement that this is a basic or dc server but it's a great starting point to extend it if you need any of those features because our default implementation is actually done at this point the application server is also very basic of course because we are not really saving this token or anything right here so what you could do is you could change the application server to store this token somewhere locally so that per user you can then access their tokens you can then use the tokens for authentication validation is user authenticated or for authorization what else does exist in this token maybe the user is in some group that has then special access on the application server and also you have the access token that you can use to make api calls to the user info but maybe to another application server that is running so if you then need to make a call to another application server you can then use the access token those tokens will expire so within an hour this token will not work so you will either need to authenticate again or implement the offline access that means that you need to have code where you can then refresh the token so if you have a valid JLT you can then get a new JLT with another 60 minutes as long as you have a refresh token we haven't implemented that but that shouldn't be too difficult to implement at this point also some authorization servers they don't really look at the offline access scope they just give you automatically a refresh token it all depends you have the reference documentation of ORDC everyone can do the implementation some implementations are not completely following the reference documentation but that is often okay because the standard also leaves some room to do your own implementation it kind of gives you a minimum that you should implement but then for example the user info endpoint could also return us a few more fields that is not really specified in this standard so that is it for our minimal authorization server if you would find mistakes in there or you want to do some improvements all PRs on my GitHub are welcome if they are adding functionality I might not immediately merge them with the solution that we have but I might for example create another directory in my GitHub repository with a v2 where we can have some improvements that were out of scope for these lectures now that we have an ORDC compatible authorization server we can start adding applications that support ORDC there are a lot of applications and SaaS applications that support ORDC often next to SAML there's also companies that can act as an ORDC provider itself like Google, Apple, Facebook those social media logins have ORDC capabilities you could either use the authorization server and trust their token or write an integration to validate a successful social login and issue your own token with your own server often plugins are available to existing tools and software to implement ORDC in the next lectures I'll show you the ORDC integration with Jenkins which is a popular CICD tool and AWS I am Federation with ORDC in AWS will make AWS trust our ID token to issue access keys to our users so within AWS we can do federation and we can say that if a user can log into a specific app then that user can be granted AWS access keys to be able to use the AWS services using the AWS API in this demo I want to show you that our authorization server actually works with real world applications I'm going to set up a Jenkins server and I'm going to change the authentication part into OpenID Connect or IDC and I'm going to use our authorization server for that so Jenkins will be another application for our authorization server let's first start to run the server and I opened already the config here because we will need another redirect URI here so we could actually create another app if you want or we could just reuse this app the smartest thing would actually be to create a Jenkins application client ID one two three five secret is secret the issue is the same and the callback will have to change but I don't know yet what the callback is we'll change it then we need to run a Jenkins server and we can do that using Docker so if you don't have Docker installed it's best to install Docker for Windows or Docker for Mac and then you can use Docker commands there's an image Jenkins slash Jenkins and it has the LTS tag and this will launch a Jenkins image I'm just going to add a few flags I'm going to add a volume the volume is called Jenkins I'm going to mount it in var Jenkins a home and I'm going to expose a port 80 90 and I'm going to map it to 80 80 so 80 80 on Jenkins is going to map to 80 90 on my machine minus it so they can hit control c when necessary and then maybe I can also add minus minus RM so if I quit my container then it will be automatically deleted it will not delete the volume so we can then still launch Jenkins later on you can also give the name name Jenkins if you want to find it easier in the process list and then actually I need just docker run because we are running Docker run Jenkins Jenkins with all these flags and then we can start it so you have to wait a few seconds and it says Jenkins initial setup is required an admin user has been created and a password generated so please use the following password to proceed so now we can go to localhost 1890 and then we should see this screen unlock Jenkins can put this password you can still find it here if you didn't see it in the output and then we can install the suggested plugins or select plugins to install I don't really need any plugins I will just install plugins later on and I can create a user so let's do that Jenkins URL and we can start so we have one user my adword user but now we are going to configure oidc so I first need to install the oidc plugin under available plugins open id there's two plugins we can use but I checked and this one has more installs so most likely this one is better install and then we can go back to the dashboard manage Jenkins configure global security so security realm Jenkins own user database or login with open id connect client id we then need to enter one two three five client secret secret configuration mode automatic localhost 8080 well known open id configuration and this will actually not work because now we are in our container so we would have to reach our whole system which we cannot do but there's an easy solution for that and we'll also need it in the next demos you can install something called ngrok ngrok serve web commands with one command you can sign up for free but you can also just download it and you don't really need to sign up macOS brew install ngrok for windows you can download the zip file or do it with chocolaty chocolaty is a package manager to easily install packages or even with docker you could do that it looks like i'm going to use the macOS version so i'm just going to open another window here i have ngrok already installed and i'm going to do ngrok htp 8080 and that will give me an endpoint this endpoint and this is a public endpoint that will map to my localhost server that i should be able to use here and it's also htps so it's encrypted this works and now what i should actually do is to also change a configuration file so i'm going to put this in place and then stop the server start the server again and then in advanced you can also have some more advanced options so for example the full name field for me is going to be name so Jenkins will also hit our user info endpoint to get this extra information we also have an email field i think that was email i could have a look if we have look at users we have here email i could enter it there as well so here we have name sub email so let me just add email here and you could also add a groups field name so if you add a groups field name that means that it's going to expect an area of strings and then you can have groups and the benefit of having groups is that you can then download another plugin for authorization matrix authorization then you could specify groups and then you can match groups to specific privileges so now we logged in users can do anything but if you want more granular access you could add groups and then use the authorization plugin in Jenkins to give more granular access to those groups so i'm going to save this then i'm going to log out so here we are the adward user now i'm going to log out i'm going to go back to localhost 8090 you might get this warning just because we are doing a redirect then you just have to press this button redirect uri not white listed because i didn't know the exact uri so i'm just going to copy paste it from the url and then in our Jenkins application i'm going to add this as redirect uri refresh this oh i still need to restart our server refresh this and then we have our login form i'm going to log in and then i'm logged in welcome to Jenkins who am i now here's my full name and i'm the Jenkins user id 9999 because that's my user id so if you have a look at the people here we now have the users this is the user from open id connect and this is the user that we added at the installation and then you click on my user configure then we can see the full name comes from the authorization server and also the email address comes from the authorization server so that was nicely filled out so we can even have more mappings to fill out more information if you want that's the benefit of having an authorization server it can distribute user information to your application servers so that's it for Jenkins there is always a possibility that you try out an application and that our authorization service is not immediately working with it then you just need to have a look at the flow that it is using you have to make sure that all the OIDC capabilities that the application needs are also implemented because we might not have implemented every single step or every single capability when preparing the ALS OIDC lecture where we will get ALS tokens based on our token that we get from our application i realized that our discovery endpoint is missing a few attributes that we should return otherwise this is not going to work because ALS IM identity and access management is actually checking for a few more attributes that we don't have so I have those added in the types it's id token signing algorithm value supported claim supported and subject type supported so we need to add those otherwise the next demo is not going to work so we're going to have id token signing algorithm supported which is rs256 then claim supported also a string slice and these are the claims that we are returning so we have sub and we can just have a look at our token endpoint we have issuer sub audience expiration issued at and not before and then lastly we also have subject types supported and there is a mechanism to hide some privacy sensitive information so the only thing we need to do is say that we only support public we just return the subject the other thing we need is when we are going to use the app server we are only replying one token and we are going to need both so I'm just gonna make an array of this and then here we have the parse access token right here we have the parse id token and I'm just going to jwt token going to add both of them first id token and then the access token parse the id token and I need another column here save this go to the main so here we have both tokens and tokens the second one is the access token and I'm just going to print on the console the id token because the id token is the one we need the next demo tokens raw so that's it we should be able to run the server again and also maybe one last thing to mention from this discovery endpoint if you want to know more about this discovery endpoint you can have a look online and just type in open id configuration endpoint and you will find some information for example I found this held up wiki and this held up wiki explains every single attribute that you can have so here is an example and here at the bottom you actually have what attributes are required so the issue is required the authorization endpoint is required which one is optional and which one is recommended so we have the username for endpoint which is recommended but not required but then we had some other required missing which didn't work very nicely with alias so this information is actually very useful when creating this open id configuration endpoint in this demo I will show you how we can use our OIDC authorization server to get alias access tokens we can configure within alias im identity and access management our OIDC server that way every user logged in into our authorization server could potentially get alias credentials from alias im let's have a look how this would work first we need to again set up our ngrok tunnel so I'm just going to run the server first and then open another window this is the same as I did in the previous lecture if you didn't follow the Jenkins lecture you just have to download ngrok from the ngrok website or with brew any type ngrok hp8080 and that will tunnel port 8080 from your local host to a public available hostname this is our hostname our server is running we just have to make sure that we update our config file as every time you use ngrok this hostname will change we then need to restart our server and then we should be able to use this hostname in alias so this is my alias account it's a sandbox account it's empty I just created a new one and we need to go first to IAM manage access to alias resources and here we have zero users so we only have our root user and we are not going to create a user we are going to use our identity provider so we click on identity providers and here we can add a new provider use an identity provider IDP to manage your user identities outside of alias but grant the user identity permissions to use alias resources in your account so you first add a provider and here you can choose between SAML and open id connect so in the first lecture I was explaining you SAML and OIDC where SAML is the older standard and open id is a newer one and you can see that indeed SAML is still used in a lot of places in the past even only SAML was used in alias and then only later they supported open id because also when alias came out the open id standard didn't exist yet we are going to choose for open id connect and then we can provide this URL I'm going to provide my ngrok URL and then you can click on get thumbprint this thumbprint is from the CA certificate so the CA validity is until 2024 so ngrok is configured with HTTPS there's a certificate in use the certificate I actually checked is from lads encrypt it just to make sure that alias validates our certificate it's not going to do it on the service certificate because a service certificate typically is only valid for a couple of months or for a year so it does it on the CA and the CA is typically valid for a couple of more years so we are now 2022 so it's valid for two more years now we can configure the audience and audience in our id token is the client id so that's why we have to also supply the client id to be able to log into alias we cannot use the access id it's going to check the audience and the audience needs to be equal to the client id so I'm going to use the client id from our first application server 1234 and then I'm going to add this provider so the provider is now created and now we can link a role to this provider so we can click here and assign a role to it I'm going to create a new role and you could say that whoever has access to our application server our first application server has access to the complete alias account so admin permissions or you could say whoever has access to this account has a specific subset of permissions I'm going to add as three permissions just as we are working with some kind of development server we just created the server so if something goes wrong then at least we don't have admin privileges so this is our identity provider audience is 1234 and alias calls is the audience because the audience is equal to the client id but we always call this a client id and not really the audience so in the id token audience is equal to the client id and that's why they call it audience here permissions so here we could give administrator access or I'm just going to do s3 full access tags if you want to create tags review oidc demo s3 role I'm going to call it create a role and now we can assume this role using oidc using our oidc token I need this errand because that's what I want to assume and I will have to use the alias command line utility so if you don't have installed yet you can install it from the alias website or again with brew or with chocolatey so that you have the alias command available then we will also need the client token so we need to log in to our application and we have to make sure the client id matches so the client that we entered matches with our app one so I'm going to log into app one and that is going to give me the id token so here we have app one I'm just going to start our application server so this oidc endpoint can still be local host it was just going to verify this token it doesn't need to have access to our application server and now what's going to happen here I'm going to see the id token in the output because I added this fmt printf here id token token zero zero should be the parse id token okay let's try that out local host 8081 login my login and password token received and let's just make sure that I copy again this arn and then we can try to log into alias and get the tokens let me maybe also create an s3 bucket so that we can do an s3ls to list the buckets and when that works I'll be able to see whether whether the tokens actually work my test bucket I'll just add some random string here because we know it might already be in use because it needs to be unique within this alias region so just have to make sure that it's unique enough because my test bucket will probably already exist so I have a bucket so I'm going to try to assume this role here and here you can also see that there is this trust relationship where we have this audience needs to be equal to one two three four so only the tokens from this app server will work on our oidc provider that is what alias going to check is going to check this token see if the signature matches with the public key and if that matches it will accept the audience the audience is equal to our client id if this audience is one two three four then I am granted access here is the id token so let me just open another window so our role role eran is this one don't want to lose it and our token is this one so I'm just going to say token equals and then here I can do echo token echo role eran if you're if your windows you can use set I think or you can just copy paste it in one string that's also possible because it's just easier for me to work but you could also just enter everything in one string so you need to have the alias account but you don't need to be logged in alias stls unable to locate credentials so I have no credentials and now I'm going to try to obtain credentials and I'm going to do this with alias sts assume role with web identity so I think if I just add help to it then I will get the help function and if you scroll a little lower it will show us the arguments the role eran check we have it role session name is just an identifier so could be our user for example it's just a value it's not being checked on and then the web identity token so this is what we need to be able to get a token so I'm going to do alias sts assume role with web identity role eran is our role eran web identity token is our token and role session name is just going to be my name at the word that should be it really let's try it out if it works or before we try it out let's just verify our token jwt.io I'm just going to paste this token here so that if you have troubles yourself that you can have a look at my token so I have the algorithm kid and the audience is important one two three four this is the client id because this is our id token the kid is going to be used by alias to get this public key from our jwks endpoint and to validate our signature right here so once the signature is validated it can be sure that our authorization server issued this jlt so let's try and we get the credential now you can actually see using ngrok the calls that happened so these calls just happened here it went to the open id configuration and it also went to the jwks json to download the public key and this is when I logged in I did the login and then here are some other requests that either me or alias did so let's go back so we now have these access keys and these access keys we could configure in our dot alias slash credentials or we can just configure them as environment variables and see if we can do s3ls so there's also an exploration so what time is it now here it's 1108 and this is udc so this must be valid for an hour let's try to configure them so if I do alias access key id equals to this secret alias secret access key equals to this part and then the alias session token equals to this part and then I don't really need to blur any of this because this will be expiring in an hour anyways alias s3ls and I get my test bucket so you can see that with our id token we can get alias credentials that are valid temporarily assume this role right here and then use it to access alias and we didn't even have to create a user on alias this is also called federation in alias so if you would like we could potentially write an application a go application that lets you do the authentication with a user a password maybe even with mfa and then does this sts assume role with web identity api call to get yourself credentials admin credentials normal credentials so that you can get access to the alias api and this is actually the same mechanism that alias uses for pods in kubernetes so if you use the kubernetes service on alias then also every pod will get a jld and then there's also an environment variable that is exposed to the path where this token is and the alias sdk will automatically read this token and then will get automatically the credentials let me just have a look what the name is of this environment variable if you're interested in it so here we have the configuring the idles sdk for go v2 and within kubernetes or if you would be writing something where you would also use jlds then if you use the load default config there's always a path that it follows so it actually says here when you initialize an alias config instance using config load default config there's a default credential chain environment variables the one we used right now so if you configure those then it will automatically find your credentials or also the web identity token so if you have configured the alias web identity token file and your token is in that file then you can also just configure this environment variable and then the alias sdk will automatically go to the file read the token and get those variables if those two are not found then it will also have a look in the alias folder in credentials and also in the alias config so there's a certain path that it follows so if you want to make sure that your credentials are properly configured just have a look at this documentation and make sure that you either supply one of these as paths for alias to find your credentials so this is it if you have any problems with rdc and alias have a look whether your discovery endpoint is okay have a look whether your jlds endpoint is there whether the flow works with the normal application whether your id token is valid and then it should work if it still doesn't work for you feel free to reach out to me directly and i can always help you in this section we are going to talk about tls tls stands for transport layer security and is used for data encryption web encryption typically uses tls to encrypt communication between client and server tls is a successor of ssl secure socket layer the default port for unencrypted htp traffic is 80 and a default port for encrypted htps traffic is 443 tls itself is not an encryption algorithm with a protocol to negotiate and agree on a common set of encryption and hashing algorithms called the cipher suite with tls enabled the htp server offers the client an x509 certificate which can be validated by the client to ensure the server can be trusted the hostname of the server will be included in the server certificate the server certificate will be signed by a certificate authority abbreviated ca if the client can validate the server certificate we can trust the server and this is what exactly happens when we go to a secured website using htps in the browser to be able to validate the certificates we'll need to always have the certificates of the certificate authorities that can sign the certificates they're also called the root certificates browsers typically have this list built in and within go it'll also look for those files in hardcore the system paths to be able to validate certificates so these certificates are provided by the operating system so on the linux system they will be different than a window system or a macOS system and with macOS or linux for example it will be the operating system itself keeping them up to date so go will just look into a few directories to see if it can find those root certificates once it finds one directory with the root certificates in it it will stop looking and it will use that directory so with the browser this is all pretty straightforward you're using this every day where you go to an encrypted website the validation happens because your browser has those root certificates and you can automatically trust the server if a certificate will be expired or the hostname would not match or the signature cannot be validated you would typically get an error which sometimes happens on websites that don't maintain their certificates what i just talked about this is our client server communication where the htp server offers the client a certificate that can be validated this is called one-way TLS you can also set up two-way TLS or mutable TLS abbreviated mTLS in this scenario communication can only be established when the client also has an x5 or 9 certificate so this you will typically not see with your browser because then your browser would need a certificate to be able to access a website this two-way TLS is used often in server to server communication for example to secure communication between microservices in the following lectures i'm going to add TLS support to a simple gohp server there are multiple strategies to implement TLS using a self sign certificate we will issue the certificate authority certificate ourselves so only someone who has a specific ca certificate will be able to validate our server certificate using a real certificate issued by a company that can sign with a root certificate so companies like the dessert geotrust rsa global sign they own these root certificates and they can issue a certificate for you that is signed by one of their root certificates and those will then be valid if you would use a browser and visit your website with that certificate installed because your browser will have these root certificates built in and is able to validate the signature of this server certificate and also using let's encrypt a non-profit certificate authority so this is kind of a third option because the second option the certificate issued by a company you most likely will have to pay this company because they need to verify that you are the real owner of this host name of this domain name and with let's encrypt this is actually free it's a non-profit so you can get a free TLS certificate with let's encrypt and they will have an automated procedure to get you verified in the next lecture i will explain these strategies in a little bit more detail and then we will start implementing a self sign certificate all these approaches can be used for one way TLS for two way TLS a self sign CA is common but other approaches would also work so for example within Kubernetes you have a lot of two-way TLS because you have a lot of services talking to each other and you need to be able to trust each other within microservices you have two way TLS with VPNs you have two way TLS one way TLS is what you see every day in your browser and two way TLS is probably something you have less exposure to when you have two different services talking to each other so every approach can be used for two way TLS as well although self signed CA with two way TLS is a bit more common than using real certificates although it is still possible to do two way TLS technically with a real certificates or even with let's encrypt let's go over the TLS strategies we will look at the self sign strategy signed by a root CA and let's encrypt first the self signed CA this will also be the next goal demo that I will show you it's using a self signed CA in that case we're going to have a server a client and only the server is going to have certificates not declined because it is one way TLS and this server certificates need to be signed by a CA a certificate authority so that the client can validate whether the certificate is real because it is self signed we are also going to be the certificate authority so we're gonna have to create a certificate authority certificate and a certificate authority key with this key we can actually sign one or more certificates so here we only have one certificate but we could be signing a lot of certificates we only need one CA to be able to sign multiple certificates once we have this setup we can use the certificate authority key to sign the server certificate the client can then connect to the server using TLS the server certificate will be offered the server key will not be offered because the server key is secret to the server and the client can then also download the certificate authority certificate to validate the server certificate because this is not a root certificate the client is not going to have this bundled in the browser or bundled in the operating system so you'll have to provide this CA certificate file to the client to be able to validate the server certificate if we maintain the client for example the client's also a go program or it is curl or it is our own browser then this setup would work because we can have this CA certificate installed where it wouldn't work is if you would have a production server a public production server and we would offer any client that will connect to our server any browser this certificate where they don't have the CA then they would get an error because they don't have this CA certificate and that's why all the servers that we connect to in a real world scenario are actually using one of the root certificates so root signed CA is next where we still have the server the client the server certificate and the server key but the client in this case has the root certificates available to be able to check the server certificate it just needs to be signed now by a certificate authority company this certificate authority company we have the certificates of this certificate authority company but not the key so we cannot sign any server certificate ourselves we don't have access to the CA key or else everyone could sign any certificate and this whole validation process would just not work we all agree to trust these certificate authority companies to only sign server certificates after they've validated the host name in this server certificate so they will go and send an email for example to the email attached to the domain name and ask are you the real owner can you verify that you are the real owner and only then they will sign your server certificate so how does it work well you need to first give an unsigned server certificate to this certificate authority company this unsigned server certificate is in the form of a certificate signing request or abbreviated csr so you basically give your certificate that you generated using your server key and the certificate authority company is going to sign this for you and give it back to you so now you have a signed server certificate and then the client can connect using tls to the server will receive this server certificate and will be able to validate this server certificate because it has the root certificates without the certificate authority companies in there to be able to validate this server certificate so you can imagine if you have a very old operating system or very old browser that is 10 years old and hasn't been updated then actually those certificate authority certificates or root certificates they expire and when you then visit a server you would get an error so if you for example install a very old browser or operating system that is 10 years old you will see that those root certificates are not valid anymore and you will get an error when you just browse to some common websites and then we have let's encrypt and let's encrypt is very similar to the root certificates because let's encrypt is also a company that is included in the root certificates the difference with let's encrypt is that it's an organization it's a non-profit and they don't ask money for a tls certificate we still have the server and the client the server certificate and the server key the root certificates that has let's encrypt in it and let's encrypt itself the company we still don't have access to the c a key because that is managed by let's encrypt what let's encrypt is going to do is it needs to be able to validate whether we really own this domain name let's encrypt is validating our website using this automatic certificate management environment let's encrypt will connect to the server that needs to be publicly available and will go to a specific path that you can see here it's dot well known slash ace me challenge slash token and this token is something that let's encrypt will give you if you start the procedure it will see whether you really own this domain by putting something unique on your domain name that can be verified by let's encrypt so let's encrypt will give you a random string you put that random string on your server if you are able to do that then let's encrypt trust that you own this server that you own this domain name and then it will then sign your server certificate here again you need to give an unsigned server certificate a csr certificate signing request to let's encrypt so that it can sign your certificate based on this certificate signing request you then have a service certificate that is signed by let's encrypt and the procedure now is the same the client connects with tls the server certificate that is signed is offered and the client can verify it using the root certificates that includes also let's encrypt let's encrypt is actually the newer strategy compared to the first two because the first two already around for a very long time and let's encrypt is a new approach that came into existence so you don't have to pay these companies always for your tls certificates let's encrypt is very popular so there's a lot of websites that use let's encrypt so they don't have the cost of paying for every tls certificate it might be just a bit harder to get a certificate because you have to use this automated process with the other companies that sign your tls certificates there are more options to do the validation process but if you can use this automated validation process it will end save you some money and it will also save you time to renew the certificates because you can automate your renewal as well one strategy that i didn't cover yet is two-way tls so the mutual tls so with the mutual tls the client also has a certificate so how does it look then we have a service certificate and server key and the client certificate and the client key we have a certificate authority certificate and a certificate authority key so in this case we have self-signed certificates because we will also be the certificate authority with the certificate authority key we can sign both the client certificate and the service certificate and then we can distribute the certificate authority certificate to validate both the client certificate and the service certificate so the client will be able to use the ca certificate to validate the service certificate and the server will be able to use a ca certificate to validate the client certificate. So the client makes a TLS connection. The server will be able to validate whether the client has a certificate and whether the certificate is valid. And the client can do then the same thing. So the benefit of Mutal TLS here is that you can have a two-way trust. In one way TLS, the server doesn't know anything about the client. In two-way TLS, the client is also identified by client certificate that is also signed by the certificate authority. So the server also knows that the client has obtained a certificate that has been signed by the same certificate authority. Like I said, you also have variations on this, where you have two certificate authorities, or you can also use a root certificate. In the next lecture, I will show you how to do a one-way TLS setup with an HTTPS server in Go. So it will be a very simple server using the self-signed strategy. This is the TLS start project from my Github repository. And this TLS start project, I'm going to use to show you how to create a command line utility. A command line utility to create a certificate, a CA certificate first with the key and a server certificate. Because in this TLS start, we have a test server. And this test server is going to launch an HTTPS server. So if we have a look here, so we can run this by Go, run a CMD test server main.go. We have the main function. And we only going to handle slash the index page. And instead of listen and serve, we are going to use listen and serve TLS. TLS needs the address, which can be 443 or 8443 if your port 443 is blocked by maybe not a program already. We need a cert file and a key file. And it's going to be the server certificate and the server key. So if we are going to run this now, go run, CMD test server main.go, it will complain because we don't have this server certificate and server key. This is a server certificate and server key that we still need to create. And how would you create those in the past? Well, you have the open SSL command line utility, but we don't want to use it open SSL utility, we want to write our own command line utility. And our own command line utility will generate this CA certificate and key, and we'll sign the server certificate and key, just as I explained in this previous diagram. Just as a refresher, this is the setup we are going to build. So the server here on the right with the server certificate and server key, that's what you are going to pass to this function to start our server. But first, we need to create the CA key and certificate to sign our certificate. We can then have a client that loads the CA certificate, then connects to the server and can then validate the server certificate using the CA certificate. Our client can be a browser can be another goaling program, or can be something like curl or postman, I'll be using curl. So this is our server. And we just want to write a command line utility. So we are going to have a TLS command. So TLS command, go run TLS main dot go. And nothing will happen because we don't have any code in there. But that's how we're going to run our TLS executable. You can also build it obviously, and then execute like that. So once it's ready, you could actually make the binary and use it even yourself to create certificates, or even extend on it, because our functionality will be pretty basic, but you could still extend on it. Because the open SL command line utility is quite extensive, you can do a lot of things with it, we will just do a subset of that, we will be able to create CA certificates and sign them. But compared to the open SL utility, it's not a lot. So this main goal will load the PKG CMD package, PKG CMD. So that's where we're going to write our command line utility code. So we have the root dot go and root dot go, we have the execute function. We are going to use a library for that. It's a popular command line utility library. It's called Cobra. So we'll be using Cobra for the command line utility code. So we don't have to write everything ourselves. Cobra is used by popular tools like Qubectl. So it will look very familiar if you already used Kubernetes at some point. So this in CMD, we'll have our command line utility codes. And in the third package, we'll have our certificate codes. So if we are going to write codes for these x five nine certificates, we'll do it here. We need to create a CA certificate and create a server certificate. So the server certificate is not very special. It's only the CA that is special. So we can have a function for the CA certificate, and then a function for a regular certificate. And then later on when you need to create a client certificate for two way TLS, you could reuse this function. We have the PEM dot go and the PEM dot go just has one function, PEM two x five one nine, it just reads input in binds, which is a PEM format file, and then we can decode it and parse it into an x five nine interest to help a function right here that you don't have to write yourself. In times, I have the times, the CA cert, the cert and the cert subject. Why do I have already those because we are going to use a conflict file. And this conflict file is something I will just supply to you. It's called till as a demo here. And here we are going to define the information that we need to create our CA certificate and our service certificate. So we have a CA certificate. Every certificate needs a serial. How long is it valid valid for 10 years? And this is a subject, every certificate, the CA certificate and the service certificate, they have a subject. They have a country, organizational organizational unit locality, and a common name. There's even some more information that you can give that you will see in these types of go. So you see here, the subject is a little bit longer, there's some more attributes that you can add here. The common name in a certificate is either just a name like CA certificate to show someone looking into the certificate that this is a CA certificate, you could even add the name of your company. But then, for example, if you are going to use certificates, server certificates that we are going to use on websites, then the common name is going to be the host name. So here we have common name is go demo local test dot me, and local test dot me, and all the sub domains of local test dot me, they actually resolve to the IP address 127001. So we can use this address, because it will resolve back to our own machine. And that gives us a DNS name that we can then use to connect to in our browser or call. Our service certificate also has these other subject lines. It has a sale valid for years. And you will see that the CA is typically valid for longer than the certificates itself, the service certificates. Its best practice to rotate them once a year, for example, could be earlier with let's say crypt is going to be earlier with root sign certificates you can often choose. We are choosing here for one year, but it could be as well two or five years or something that you typically want your CA to last longer than your certificate. So if you pick 10 years here or 15 years, you will also have to increase this one. And then the last one is that if you want to have more DNS names, you can also use DNS names. And then you will have your common name, plus more DNS names. For example, if you are using your own domain name, and your own domain name is called local test of me, then for example, you could have local test of me, and then the DNS name could just be the root of the domain so that you cover both of them. We are just going to use code demo, but feel free to use any name that you like. So this is the last demo, we'll still need to parse this. We just have the types available, so we don't have to create them. Here's the YAML annotation, so that we can un-martial this with the YAML. We'll do that in the root package in the next lecture. It's one of the first things that we'll do after I explain you the basics of Cobra. So next lecture, we will start with Cobra. This will go in the CMD package, then the certificate code will go in the search package. And then we also have the key package. So every certificate needs an RSA key. And because I already went over RSA keys in our previous section of SSH, because SSH is also using RSA, I already have this code prepared. So the code is very similar to the SSH code. So if you want to know the details of RSA keys, I would suggest to have a look at the SSH lectures. We are just going to use these functions. It's actually very simple. We have to create RSA private key with a search on length, which will use generate key and reply a private key. With this RSA private key, we are not that much, we need to transform this private key to PEM format. So this is what the RSA private key to PEM does. It takes a private key in the private key format, and returns a PEM block with RSA private key. So then we can save this in a file. And then we just have a save function create RSA private key and save, which will create the RSA key with permission so that only the owner can read it. It will encode it using PEM and then save and close this file. And then we have the function to read it back. So if you have a file in PEM format, we can use a private key PEM to RSA, which will then PEM decode the RSA private key and put it in the RSA private key format. So for that, we use the parse pkcs1 private key function to convert it from a PEM format to an RSA private key format. If we ever need the public key, the public key is accessible within this private key as well. So these are the functions, the RSA functions that we have to create a new key and convert between PEM and RSA format. So that is it. I also added this git ignore to make sure that we don't commit any PEM CRT or key file to GitHub. And then we also have the go mod file, where I define the module name, and I also replace the mock version from 143 to 144, which you probably already have seen in one of my earlier lectures. There's just something wrong with this 143 version. So it's just saying if you come across this 143 version, just replace it with 144. So this is it for the TLS start program. In the next lecture, I will start working on this program starting with Cobra. If you need any of the solutions, then you can have a look at TLS demo where you'll find all the code that has been written. In this demo of the TLS command line utility tool, I'm going to use Cobra. Cobra is a project that you can find on GitHub. So it's github.com SPF 13 Cobra currently Cobra is a library for creating powerful modern CLI applications. Cobra is used in many go projects such as Kubernetes, Hugo, the GitHub CLI to name a few. So it is actually quite popular because Kubernetes is a quite popular project, and it is using Cobra. So what's so good about it? They have an easy way to create subcommands. For example, you can do app server, app fetch. So it's pretty easy to use. You can also nest your subcommands, we can have in our case, TLS create, TLS create key, TLS create CA TLS create cert. So those are all subcommands that we can create in a nested create command. So let's have a look how we install this. Go get GitHub.com SPF 13 Cobra at latest. So I'm going to copy this already. And then we have the user guide here. So if you want to know a little bit more about Cobra, I would recommend you to read their user guide. It's not very long. So what we already did is we created this CMD directory in our PKG directory. And this is where you typically put your goal files for Cobra. Our main.go looks exactly like this. We have a main with CMD execute. So the main is very bare, it serves only one purpose to initialize Cobra. So we're going to initialize Cobra in the execute function within our CMD package. They have Cobra code generator if you want to use it, but we are not going to use it. We already have some code ready to go. The first goaling program that you would make is for the root CMD. Ideally, you place in root.go. So that's what we are going to do first. That's where we have our execute function. So let's try to copy this code and see what happens. We'll replace this you go by our TLS explanations. So first I do the go get. And this is going to download Cobra 150. It's updating the go mode file. And now I can start writing in root.go, the first lines that are going to use this Cobra package. So I'm going to copy this and just replace this, save this so the imports are correct. So we imported Cobra. This is the root CMD. And when we do execute, it will execute the root CMD.execute. If something goes wrong, it will show an error on SCDR. So I just want to replace this, the use, the short, the long, which is the command and the explanation TLS. TLS is a command line tool. Or it could be utility or interface for TLS. And then we have a long explanation. TLS is a command line tool for TLS, mainly used for generation of certificates, but can be extended. I think that's fine. Let's just have a look what this does. So let's run our go run CMD TLS main.go and it doesn't do anything yet. Because we are just hitting this do stuff here. But because we already initialized Cobra, we should be able to use the help function. And this is what is nice about Cobra. Now that we initialized our root CMD, it already has a help function. TLS is a command line tool for TLS. And then you see can put some nice text here usage TLS flags, there's a help flag, and now we can start adding commands. Every command can go in a different go file, and we just need to add more of these Cobra commands. Once we tie these Cobra commands together, they will show up in the usage, we'll also be able to add flags and then they will show up in the flags. So this is a very nice package that will make it for us very easy to write a command line utility. So one last thing that I want to mention here is that we have this execute function, and this execute function happens on the root CMD. But the root CMD is actually declared outside our function. So the scope of this variable root CMD is not only within this function, but within the whole package. So once we create more go files, we'll also be able to access this root CMD. That way we'll be able to tie all these commands together to the root CMD. And that's how then the root CMD execute will tie everything together. The root CMD execute will use this root CMD, everything will be tied to this root CMD. And that's how this Cobra package will be able to find the link between all these go files, and then we'll be able to show the commands that you can execute and the flags that you can use. So how to add flags and commands will all become more clear in the next lecture. This was just a simple introduction to the Cobra package. In the next lectures, we'll be adding the config file and more commands to be able to create our keys, CA and certificate. Now that we initialized our Cobra package, let's try to parse this config file. So this config file describes a certificate, our authority certificate. That's a serial valid for years in a subject. And then it has certificates that are going to be signed by this CA certificate. So search is a map, because we have here a key, and we can have more certificates that are identified by key, so that we can have one or more certificates, or even zero. That way we can have now one certificate, but if you later want to add more certificates, we can add more certificates. So the first thing is going to be to add a flag in our root.go to allow the user to supply a config file, just the path to a config file. And maybe we can even put tls.yaml as a default, so that if you don't supply anything, it will automatically look for tls.yaml. So how do we do that? Let's maybe have a look at the documentation of Cobra. So create root cmd. That's what you already did. We have function execute. You will additionally define flags and handle configuration in your init function. So remember that when we have an init function, that function will be automatically executed. So we can write this init function and add some more flags. So here we have the init function and then we typically do Cobra on initialize init config. So this will run this init config. So when Cobra initializes this function will also run where we can then initialize the config. So I'm going to do it very similar as the user guide. We then are going to add to the root cmd a flag. When we use persistent flags, then for every subcommand, this flag will also be available. So for our subcommands, we are going to use local flags that are only available on the subcommand itself. And on our root command, we're going to use a persistent flag that is going to be available for every command because we're going to have this config found. So I'm just going to copy this and add this right here. We will also then need an init config, funk init config and the config file. So this config file is going to be global as well for config file config. And let's then define a config type, which is going to be a struct with a CA cert of type cert CA cert. I'll make it a pointer so that it can also be nil and then YAML CA cert. And this YAML should of course match with this CA cert. Looks like it's matching. We need to do the same for the certificate. And this will be a map, map string cert dot cert. And then the YAML is going to be certs, certs here, the key. And this is going to be the cert object going to save this. So this is a config. So config files of type config init will automatically run, it will say on initialize, run this init config. And this is not going to be a string var. So actually, this is the config file path that we need. So we need another variable config file path is string. And then we will need just a variable config. And we need to parse the contents of this config file path into this config config file path is going to be this one config file path. This is the parameter config, the default value, we can leave it empty and then fill it out ourselves if you want in the init config config file default is and then we can say default is still as a demo. We can also choose that there is a shorter version of config. So if you say string var P, then we will have an extra parameter. We have the name, which is config, but also the shorthand and shorthand can then be C. So we can use either dash dash config or C. Then we need to parse this config file, which is YAML. So if config file path is empty, then we're going to say config path path is T less of YAML. So we can either check for the config file here, or we can input T less of YAML here. So it's the same way you put it. If you're going to check here in the init config, you know that it was not supplied. So you could take other actions like not reading the config file, for example, if it does not supply it. So then the next step is going to be to read the contents of this config file path config file binds is IO read file IO util read file config file path. If there is an error, then we get output that has an error while reading config file. And actually this needs to be print F because we are not returning an error. We're just printing something on screen and then return it. And then we can un-martial into into this config file. So un-martial takes the binds config file binds, and we can marshal this into our config error handling. While parsing the config file, I have one error. And sometimes what happens is that visuals to the code decides to import some random library. It's best to check what library it is importing. This is the one I typically use go get. And now everything is green. So this should pass our config file. And what we can do is we can output it just to see if it worked. Confid called past will then later on remove this. It just to see whether our config file was parsed. Go run. So this is the help flag doesn't seem to be parsing it with the help flag. Let's remove the help flag config file parsed. And this seemed to have worked. Cannot see all the data because it's a pointer. But this would seem to have worked. And this shouldn't happen every time when we execute a command using the command line utility. So whether you want to parse a config every time when you execute it depends on how you want to write this command line utility. If it is empty, you could also say I'm not parsing it. Or you could say I'm trying to parse it. And I will not give an error if there was no config file supplied. And I couldn't read the file. You can also check whether the file exists, for example. It's kind of all up to you how you want to write this command line utility. It should be as natural as possible for the end user without doing crazy things that the user wouldn't expect. I think this is fine. Let's say that you always need a config file. So that when you want to use this tool, you just need to have the last demo in your directory, or you need to supply it with config. So if you say config, my other config demo, then I get an error because the file doesn't exist. So that's how it should work. I can even try with a shorthand flag, the shorthand flag minus C. So that also works. And if I use help, it also shows us the flag. So in the next lecture, what I'm going to do is I'm going to start writing our sub commands. So now I'm going to create the commands and the sub commands. I want to have a create command and some sub commands under this create command. Later, you could then extend this tool with some other commands. Create is what we need for this test server, but maybe we'll also want to read certificates. And you would need an info command. We are just going to focus on this create command. So I am going to create a new file here, create dot go package CMD. And I'm just going to write an init function here. And now I have access to all these global variables here. So I have access to the config, the config file path, but mainly to this root CMD. So if I create a new command, I want to say root CMD, add command. And then I will add a create command. And this create command can be of the same type of Cobra command. So this is going to be my create CMD. And I'm going to import Cobra. And here I can have the same information. Use short and long. And actually, I will delete this run function here, because we don't need to have a run function. When we just execute the root command without anything, use create, create CA certs or keys, commands to create resources, CA certs, keys, save this. And let's see what the behavior now is. Go run main dot go. So now that we don't have a run function, it actually shows the help. So now you have usage, available commands, completion, help, flags. And then we have additional help topics, T last create. But this is going to be another available command, but we haven't written any sub commands yet. So we can already see that it recognizes that we have this create command because create CA certs or keys is actually here in the help. So if you do help, we get the help. If you do create, we just get this create help. Same. So we would need more sub commands. So let's create another sub command for keys, key dot go, package CMD, another init function. Instead of root CMD, we're now going to use create CMD, create CMD add command. And we're going to have a key create CMD key create CMD is going to look similar key create CMD with a cobra command key key commands, commands to create keys. And then we're going to have this run function because now we actually want to do something run, which is a function. And here we can put our code. So let's see if we now use create. Now we get a help create create key key commands. So if you do create key, nothing happens, because this function is still empty. And now if you enter go run without any argument, we see the available commands. Now we have the create command here as well, because now this create command is actually doing something. So it's a real command. We have the create command. We have the config flag, which is global. I'm missing here a bracket. This key command could use a config, but I don't think our key command would need the config. And now we can even add commands at the key level. So what we want to do now, we want to add a key, where do we want to write the key? So we could use a destination key, a destination key as a flag. So we could create a flag within this key. So a flag we can only use in in our create key sub command. So now it is not going to be a persistent flag, it's going to be a normal flag. Key create cmd, flex, string var p, where are we going to store it? Var key out is a string key out and then a name key out, shorthand. So now that we have c already for config, it needs to be something unique. So maybe just k, the value is going to be key dot PEM and the help destination path for key. So now we can write a key code. So how to create a key? Well, we already have this function written create RSA key, and then convert it to a PEM create RSA private key and save. So we could use this search dot create RSA private key and save off these key out. We could even add a flag for the bytes. We could add a flag for the bytes. So the byte length key length called L is cannot be a string. It's an integer, I think an integer key length integer 1496 is a default can choose a default. We are going to make it a default is going to be int var key length, key length and help. And this is going to be cert, cert would be still imported. Looks like it is not being recognized. So go mod copy this pkg cert. And now it is recognized. Not declared cert. It's not going to be cert. It's going to be key make sense. And what does it return an error? If there is an error, then we need to say something, create key error and an error, and then we can return if you want. So let's try that out. This will be our first function. So we have the create. And actually, if I miss type create, that's what is also nice. Did you mean this create? So this is also nice that it has these little things that we don't have to write ourselves. So now we have the last create available commands key, create key. And the length is going to be key length 1496 by default. So we don't really need to supply anything. Create key and nothing has been returned. That's also a little bit awkward. So let's say key created with length. And I need to say key out key length. Save this for the return. And key created key dot BAM with length 1496. Okay, and now you have the key dot BAM. So this is our first sub command key dot go. And now we're going to create another sub command or two other sub commands, CA and cert. And the difference there is going to be that here we already had a function create RSA private key and save. We still need to write a code for the certificates, the CA certificate and the cert. So the Cobra code will be very similar. But then we're actually going to dig into the package cert to be able to generate a CA certificate and then a server certificate. Now that we have finished the key dot go, let's copy paste this key dot go. And then let's create a new file CA dot go. And the CA is going to output two files, the certificate and the key. So we're not going to really use this key dot go anymore. We are not going to separately create a key and then input the key. When we create a CA, we're just going to create a new key. So we're going to have CA key and CA cert. Copy this because they are both going to be a string CA key, CA cert, key out, cert out, CA dot key, and CA cert, destination path for CA key, destination path for CA cert, add CA create CMD to the create command, still a Cobra command, CA, CA commands, commands to create the CA. And we are going to use cert, create CA cert. Oh, this is the string. And what does it need? The config, which should be initialized. So we should have our config in this variable because we did this. So create CA cert config, CA cert. So that comes straight from our config. There's going to be this one here. And then we have the key file path and the CA cert file path, which is the CA key and the CA, CA cert, create CA error, CA created with CA created key cert, and then CA key cert, save this, looks like it's going to work. Clear this. And we have an error. We have the add flag error, because look, I'm still doing key create CMD within the CA. So this is not good. So if you do something like that, you will get an error. And let's try again, available commands CA and key now. And you see the global flags is a config. So we can supply the config when doing the CA command. And that's useful because we have here the config that we parsed. So this is it for the Cobra implementation. Now I will need to do the actual implementation of the creation of the certificate for the CA. So how do we even start with this? So we know that there is an X 509 package X 509. And we are going to create a certificate. So we have X 509 the package. And what does X 509 have create certificates. So now it imported crypto X 509 create certificates, create certificate creates a new X 509 v3 certificate based on a template. The following members of template are currently used. And then there's all the parameters of a template of an X 509 certificate. So what do we need for this? We need an IO reader and we need a template. So why don't we start with creating a template template equals X 509 certificate. And this needs to be a pointer so I'm just gonna make a reference to this. And then what do we need? Serial number Serial number CA Serial because we supply the serial number right here. What else we have valid for years subject subject we also have pkixname is a type. So we will need this one. And this has all these parameters. So we can just supply all these parameters. We just need to make sure because this is a string array that if they were not supplied in the config file they will be empty. So we don't want to return an empty string in a string slice. So let's maybe first make a function. Remove empty string input is a string slice and output is a string slice. And what we are going to do is if length of the input is one because we only have one item in this string. And the first element is empty. Let's then just empty. Let them just return an empty slice. Otherwise, let's return input. Remove empty string or yeah. And then if we then do country, remove empty string. So if our CA subject country would be empty, then it's going to be an empty string. But then this function will remove this empty string by returning an empty slice. So otherwise, it will not show nicely in our certificate. And now basically for every type that we have, because I don't know if I have all the types for every type that we have, we can define these values. So that should work. What else is there? Not before. Not before is of time time. Not before time now, not after time now add and add date. And then we can add some years. Now the only thing that you might want to check is that whether this valid years is not zero or empty. It's an integer. So it will be zero. If this is zero, you probably might want to send to one or you might want to return an error. Not before, not after. What else does it have? Key usage, we should also define extensions, XY extensions is CA also need to say is CA is true. And then I'm going to add this extension key usage is of type extension key usage. And then we need to add some key usage. So this key usage for the CA needs to be five or nine key usage digital center, and actually also the key usage third sign. If you're interested in what exactly these flags mean, besides that they put some constraints, you can have a look in the standards documentation. These bits are actually explained and what exactly that they do. So we are just going to assume these are the ones that we need. And what I'm now going to do is that this create CA cert and this other function create cert can use the same template. So I'm just going to copy this over right here. And the key usage needs to be different here because our server certificate doesn't need this one. But I can now change the CA right here, insert, we'll still need to change this template a little bit. Don't need a CA. But we can now create a common function for these create CA cert and create cert. We just need to make sure that we have all the parameters here as well. I need just one more basic constraints valid indicates where the CA are valid. So if you are using is CA, then we also need this basic constraints valid to true. Let's now try to create a function that can be used by both this create cert function and the create CA cert function. In the next lecture, where we'll do the certification, where we'll finish the create cert, we'll probably make some changes here still. But this is good enough to get an idea. So what would we need to create a function that we can both use for the CA and the cert? So we are going to take this template as an argument. So I call this create cert with a lowercase C. So outside the package, nobody can exit it x five or nine certificates. So I can say template needs to be a pointer x five or nine certificates. What do I need to create the certificates? I will create the private key. But I will for sure also need the CA certificate in case we are going to create a normal certificate and it needs to be signed. So I'm going to pass CA key. That's going to be our say private key and a CA cert. And that's going to be an x five or nine certificate. And because these are pointers, what I can do is if I'm going to create the CA certificate, I don't have the CA key and the CA certificate yet. So I'm just going to pass nil. And what am I going to return? Most likely bytes for the key for the certificate so that we can write it out or use it and an error. So we still have errors because we are not using this template. So let's try to complete our certificate functions. Key binds. Cert binds error is create certificate template. And we are not going to pass a key or a certificate because that's for the create cert. If error is not equal to nil, we're going to return the error. And otherwise we are going to write to a final key bytes and assert bytes. So let's leave it like this for a second so that we can focus on this create cert. Let's see, we'll just leave it like this. And then in the next lecture, we can focus on this. So create cert. What do we need to do? We need to create an RSA key first. We have this function in the key RSA go key key create RSA private key and the bytes and are we passing them? We are not really passing how many bytes so I will just keep it on 1496. What does this return the private key and error? If the error is not equal to nil, we return the error. Now if our template is a CA, we are going to do something different than whether our template is not a CA. So is CA is true? Then we are going to do something else. We are going to do something else. Our certificate is a CA. So we can do X519 create certificate. Rent reader template. The parent. The parent because it's a CA is also the template. The public key. So private key public is a public key. Public key. And the private key and public key also needs to be a pointer. So I'm going to add this ampersand so that it's being sent as a pointer. If you don't do it, then it will just say that it doesn't recognize public key because it's only checking for pointer public keys. What does this return a byte error? We probably want to do a create certificate here as well. So I'm going to create a variable here. Got there bytes. That's a file type that we get back. And error. If error is not equal to nil, return the error. And now we need to just convert it into PAM format because we also want to return it always in PAM format so we can then save it. PAM.go. Could we use this? No, we couldn't use this because this already expected to be in PAM format, but we can use PAM encode instead of decode. So PAM encode. We need to write it somewhere. So I'm going to have a sort out and this needs to be an IO writer. So I'm going to have bytes buffer and bytes buffer implements the read and write methods for bytes. So PAM encode, sort out and the PAM block. PAM block that is matching PAM block. What do we need to add here to type optional headers and the bytes. So bytes we have a type is going to be certificate. And the bytes is going to be the bytes. Sort out does not implement IO writer. And that's because bytes buffer implements it as a pointer. So I need to reference it here. And there's an error that returns. So if this error is not equal to no, then we're going to return error. If there's no error, then it will have taken this PAM block with the certificate type with the bytes. And it will have written it to sort out. We can do the same for the RSA private key. So if we make a key out bytes buffer changes into key out PAM block. And I think we have a function here RSA private key to PAM. And this is returning this PAM block type RSA private key. And then it's going to marshal this private key into bytes. So we can use this instead of the PAM block key. And then the private key. And this should also work. So we have a certificate private key. And insert out and key out we now have the bytes PAM encoded. So we can just return it now. Set out bytes. What does bytes return bytes key out. Maybe I'll put a key first. I think I always put the key first. So if you put a key first, we should always put the key first. No errors. But we only did the CA obviously. And now we can have key out here key bytes. And we can write this to a file. Key bytes and sort bytes. IO utl write file file name, which was supplied key file path, the data key bytes, and the permissions for the key is going to be 0 600. What does it return an error? If error is not equal to nil, going to return error. And we're going to do the same for the cert bytes. And the cert bytes can actually be 644. Because RCA certificate can be public. So it doesn't need to have these extra permissions. It's just for your local machine just to make sure that nobody else can read it accidentally. It's kind of best practice to have your key always written with no group or other reads permissions, cert file path, CA cert file path, and cert bytes. Save this. And let's hope that it all works. Only one way to know go run. That's where see whether it compiles. Okay, available command is the CA. So we do help not sure that they're gonna work minus H unable to redefine C. Okay, because we already have the config. See there is already a C here for the config. So I'm just gonna use oh, okay, and oh, okay, the last create CA flags, the O flag, the key flag, the H4 help, and the global flags. So if you want to supply a config file, we don't need to really supply anything, then it will take the default CA key CA cert. And that created everything. I just need another return here. How can we validate this? Well, we can actually use the open SSL utility, open SSL in CA cert, text no out, I think that it is. And if you don't have the open SSL utility, you can either download it, or you can also validate online, there's tools online, if you type open SSL, read certificate, there are websites where you can paste your certificate and it will show the output for you. You can do this with the certificate, but don't do this with a key, because the key you should keep secret. This was not a command. Open SSL X 509. It is because it's of X 509 type. And this shows that in text, everything serial number one. So that was good. Not before and not after there's 10 years in between. So that's also good. The issuer is the same as the subject. Because it's our self signed CA country US locality New York organization, go like demo organization, organizational unit certificate management and common name CA certificate. Then we also should have these extensions. The CA is true. Digital signature certificate sign, web client application that all looks good. So I think we now have our CA created. And now we can use this CA, the CA key to sign our service certificate. So in next lecture, I will create a service certificate and have it signed by our CA. So let's go back to our X 509.go. And here we have this create server that we didn't finish template, serial number. We have the country, the common name, the common name. But we also need to add DNS name. Not before, not after. Let's see if you just have DNS name DNS names. It's also a string. So if it's empty, then we return empty string. If it's not empty, then it should just be the DNS names. Third subject or third. Oh, I think we are not parsing this. Where is our types? Yes, we can add it here. DNS names. It's a string DNS names. And then here DNS names. What else do we need to do? We need to parse the CA key and CA cert and then pass it to this create cert function key and private key PEM2RSA. Yeah, that's the one CA key. And it's going to return as the private key because we need the private key to be able to sign. This returns an error. So we have the CA private key and the error. If there's an error, return error. And then we have the CA certificates. And for that we have in PEM.go, PEM2X599 CA cert, parsed, CA key, parsed. So let's now just copy this, the template, the X5NN certificate, and then the CA key and the CA cert, CA key parsed, CA cert parsed. And then we're going to write the key file path and the cert file path. So we're still going to write the server key and the server cert based on what we get back. Save this. And then we just need to make sure that if it's not a CA, that we also have some code, create certificates. The parent in this case is going to be the CA cert. And the private key is then going to be the CA private key. Because to sign a certificate, we need the CA private key. Save this. This is going to create the certificate. And then we just need to make sure that we have the code in our CMD for the cert. I can copy paste most of the stuff from the CA.go. So that's what I'm going to use. Cert key and cert, cert key, cert. Oh, cert is already declared. So I'm just going to call this path at the end. Cert path. So this shouldn't really give us an issue because those are local flags. But now we also need the CA key and certificate. So let me just change this, create CMD cert, CMD. And then this needs to be cert, CMD, cert, cert commands, commands to create the certificates. And then we're going to have a create cert. And we also need to supply the CA key and the CA cert. So this is going to be cert key path and cert path. But then in between we have these two still to add. Create cert error, cert created, cert key path. Okay, I think we have everything, but we need to ask the user for a few things. We need to know the CA certificate and the CA key, but also the certificate name. So cert name, that's going to be another flag. Cert name, or just name with N, what is going to be the default? That's not going to be a default. Name of the certificate in the config file. And then we also need, let me just maybe copy paste it from here, and cert, CA key and CA cert. And maybe let's not give them shorthand flags, might be confusing a bit. Defaults, CA key and CA cert. Yes, why not? Is the CA key to sign certificate, CA cert. Which ones are mandatory now? Well, those three are not mandatory. So we can say cert create CMD, mark flag required, name, mark flag required, CA key is required, CA cert is required, and the name is required. And then I have to pass the CA key, but it needs to be in bytes. So I need to pass it first. CA key binds error is IO util read file of this CA key. If the error is not equal to nil, CA key error, and we'll have to stop. And we are going to do the same for CA cert, read the CA cert, CA cert error, read error and read error, we'll call it. And we can add the CA key binds and the CA cert binds. Save this. We still get an error, no new variables on left side. Config CA cert that's now going to be something else. So we still have to say cert config. Okay, is config cert, which is the map. And we have the name cert name. So you're asking for the name, we checking in this map whether this key exists. If this key exists, okay, it'll be true. If okay is not true, then we're gonna say, could not find certificate name in config, and we'll stop. And this should be done the cert. And that is a cert. All right, will this work? Hopefully. Maybe I made a mistake somewhere, but then we'll find it out very quickly. If this doesn't work, go run create cert, because it's called cert error required flags CA cert CA key and name not set. So that works pretty nice. And then we have this flag that we can set. So let's add these flags. The name is this key here, go demo local test me CA key is CA dot key and CA cert is CA cert could not find certificate name in config, cert name, cert name here, string var, probably did something wrong. Sort path, which would be sort name. Because that's a name, cert created key is cert key and sort is cert cert. Let's do the same with this open SL utility on our cert cert. And actually, I didn't want to call it cert cert. I wanted to call it server dot cert and server dot key. So let me just run it again, because that's how we put it in our test server, I think server dot CRT. Okay, server dot CRT is created. Open SSL utility on server dot CRT seems to work serial number one issuer is our CA certificate. And the subject is this code demo. One year valid. You have these extensions, and we have these alternative names. Right, right. Let's try to run our test server. And it's going to run on 443. Let's try first maybe to use the browser. Port 80. This side can be reached. I'm going to add HPS. And then it says your connection is not private. Because our certificate authority invalid, it doesn't know about our certificate authority, the CA certificate that is being issued. And that is because we self signed our certificate, because I was explaining that our browser knows about our root certificate, but not about our issuer CA certificate that we created ourselves. So we need to add this CA certificate to every client that we are going to use. Some clients can do it easily. Some clients cannot. But you see that there is a reason why you should use a root certificate for a public website, or let's encrypt and that a self signed certificate will not work. If you are not a client, it still works for development purposes, because I can do advanced. And I can do proceed. And then I get it's working, but I will still get this not secure because the CA cannot be validated. So let's try to do it with curl. And let's see if we can have the CA validated. And if we can have the CA validated, then we should not get an error curl, unable to get a local issuer. Now let's try curl with CA certificate CA CRT. So we are now supplying the CA certificate to our client curl. And then it will be able to use the CA certificate to see if our service certificate is valid. And actually, I get another error, no alternative certificate subject names matches. And that's because I should probably have not only the common name here, so either you use not DNS names, and then it will just work on the common name, or I need to make sure that I define everything in DNS names. So I'm going to save this. And then I will have to reissue my certificate. Let's run this open as well tool again. And then I see subject alternative names. Now I have both of them. And let's try now. Again, connection refused because I didn't start it. Go run testing the and now it does it's working. So only when you supply the CA certificate, only when you supply this argument CA cert, then it will be able to validate it. Because now if you do curl without it, it only has the root certificates from somewhere a directory on our operating system, where we have all these root certificates, but we are not using a root certificate, we are using a self signed certificate, which we then need to supply with CA cert. So in your browser, you could also make it work by importing this CA certificate. It's not really recommended on your browser to use every day to start importing certificates. So in general, if you are going to work with a server, then you should use let's encrypt or root certificates. It's for testing, or it can still be in production when you can supply your own CA certificate, or in companies that manage every browser on every machine, they also sometimes inject their own CA. It's also useful for 280 less, as I explained earlier with 280 less, you can have your CA bundled with your client and your server, and then you can also verify it. So this is the end of the demo. I hope that this whole certificate CA has been useful. It teaches you not only how to do it in Go, but also shows you in much detail how this signing and validating works, and what really the process is when connecting to a secure website. Now that we have our test server running with our self signed certificates, let's now investigate how we would run our test server with a let's encrypt certificate. Let's encrypt is different because let's encrypt can sign using a root certificate, a root certificate that is recognized by your browser or by curl. So we don't need to import the CA certificate to be able to validate the server certificate. Even though let's encrypt signs with the root certificate, it was still useful to explore self signed certificates, because it's also useful for two way TLS. And like I explained in the previous lecture, there are different scenarios where self signed certificates are preferred. Let's try to create another server or let's encrypt server. And in this let's encrypt server, let's make a new main dot go package main. And let's copy paste from our previous lecture. And what we are going to do different now is we're going to use auto cert. Auto cert is a package provided by go. So we need to do a go get first go lang.org x so it's extra packages crypto acme this protocol that is being used by let's encrypt and auto cert. And this package contains helper functions to integrate within our HP package, the automatic certification. So we will not have to do much we will just have to add this auto cert to our TLS server. And then it will create a renew packages for us. So let's have another look at this process. So this is the schematic. What needs to happen is that we need to support this acme. So our auto cert package will implement this acme which is in a separate package. And it will create this well known URL with this token to be able to verify our server. Let's encrypt will then know that we own this domain and will issue a server certificate that we can use. So that is what all happens in the background. The only thing that we need to do is to import this auto cert, copy our code. And we're then going to change the listener serve into just serve. And serve will then ask for a listener and net listener. And this net listener is implemented by this auto cert. So this auto cert has a new listener. And we can just apply domains which needs to be a real domain. So if you want to test out this demo yourself, you will have to register a domain or use a domain that you already have. My name is registered by Namecheap.com. And Namecheap gives you a DNS panel where you can add records to point a DNS name to a server. So it's an in a DNS record. And I'm running a VM on digital ocean. But you can really run a VM anywhere. And I'll have to then copy this binary, this goal binary that I will create here to my digital ocean VM. So the domain name is go demo test new tech academy. So if I try to resolve this, host go demo test, this goes to my VM. So you need to have this setup first. And then what I'm going to do is I'm going to copy this binary to this VM, run this on this VM, and see if I will get the certificate issued by let's encrypt. So once this code gets executed, it will serve on for for three. So I just have to curl to for three. This new listener of auto search will then go to let's encrypt and ask to issue the certificates. So new listener is convenience function for common configuration. You can also do more complex configurations by using the auto search manager type. Also important certificates are cashed in the golden auto search directory. So it's not going to issue a new search every time it will create a demo directory and you will find your key and insert in there. So this should be it really. Let's try to build it. Go build. Let's encrypt server. And my server is a Linux server. So I will need to also specify that I want to build this for Linux. So I will specify Go operating system Linux, Go architecture AMD 64. And I will compile CMD let's encrypt server main.go. Then I will copy this over to my test server, SCP. I will need to specify my SSH key root code demo test. And the source file is let's encrypt server. Now I will have to start this binary on the VM. Now I'm logged into the VM. And I can start it using let's encrypt server. And let me now create a new window. And I'm just going to go to this website. Go demo test new tech academy. And if everything is working, I should get that it's working. It's working. So no CA certificate warnings. A new certificate has been issued, which is now a let's encrypt TLS certificate. So if I try this in a browser, HTTPS. Go demo test. The HP will not work because we didn't open anything on port 80, but you could actually add something on port 80 that just will forward to HTTPS. And if we now click here, connection is secure. Certificate is valid. And it is a let's encrypt certificate that has been issued. And this is valid. So if you are doing one way TLS, and you need your server to be accessible to everyone on the internet, then the easiest way to do that without having to buy a TLS certificate from one of these known companies is to use let's encrypt. And you see that it's very easy to integrate it within your go program. What happened in the background is that go did this ACME validation for you in this auto set package that's code to that for you. And if you then go back to the VM, I will see these errors as well. It just other people like bots trying to make connections, but we only whitelist this domain name. So we'll only ask a certificate for this hostname, not for any hostname that people are trying to access. So if we now have a look here in the dot cash directory, there's a golden auto cert directory. And here we have our certificate. Go demo tests, new tech Academy is our certificate. And this is our key. So here's our certificate that has been issued. So this is it for this demo. It's just a very small change to have in go a let's encrypt certificate issued and use it with a TLS listener. I have shown you how to start a server using a self signed certificate. I've shown you how to use a let's encrypt certificate. Now let's try to create a server that is two way TLS. So any client connecting to our server will also need a client certificate. This is what I explained in one of the previous lectures, the mutual TLS diagram. So what I'm going to do is I'm going to create a new certificate, a client certificate. And the client is then going to connect using TLS to the server. The server will need to be reconfigured a little bit. The server is only allowed to accept clients if they have a client certificate. So the server will also have to validate the certificate authority certificate. So we also need to supply this CA cert to the server. Again, this is not very different than our test server. So let's start again with our test server. Let's create directory empty less server and then we'll have an empty less client main.go. So what is going to be different? The listen and serve will need to supply our own configuration. We'll have to change these TLS configuration flags. So instead of using HTTP listen and serve, we're going to define a server HTTP server. And then we're going to change HP into server. And server now accepts only the sort file and key file. So we're going to remove the port. And then the port is then defined here in address. So address is going to be port 443. And this is not any different than our test server. The difference is going to be the way we are going to configure this server. So in server, we have TLS config optionally provides a TLS configuration for use by serve TLS and listen serve TLS. So we're going to use TLS config TLS config is a type. And what are we going to specify client authentication determines a service policy for TLS client authentication. The default is no client cert. So the default is that we don't need to client to provide a certificate. We are going to change it into require and verify client certificates. So we will require a client certificate. And we will also need to verify it. If we are going to verify it, we also need to supply the CA client CA is defined as a set of root certificate authorities that service use if required to verify client certificates, client CA's. And this is of type X 519 sort pool. We need to find those CA is X 519 sort pool. This need to be of sort pool type. But we have in this X 519 package, new sort pool as a function. And then we have CA append sort from PEM. And then we can just load these PEM sorts as bytes. To do that, we still need to read the CA cert, CA bytes, error is IO utl read file CA cert. If there's an error, then log fatal error. And then we can supply the CA bytes right here. This reply is okay. So if it's not okay, then we also can supply an error, CA cert not valid, for example. And otherwise in the CA, we have the correct CA now. If you want, you can also supply a minimum version TLS version TLS 13, for example. So 1.3 is minimum, then now that we are defining error, we don't need to use this column here anymore. Gonna save this. And everything is green. So we loaded the CA's, still listening on port 443. And we specify require and verify client cert. So let's try to test this out. Go run, send the empty server main.go. And let's curl to local host. Let's curl to this correct host name. Go demo local destiny, unable to get local issuer, CA cert, CA cert. Okay, so when we verify the server certificate, then we get an error back from the server because we don't have a client certificate. And this is only because we have this line. So let me try to remove that line. And then curl again. If we curl again, it says it's working because we are not verified a client certificate. So to make it working, we actually need a client certificate. So let's try to create a client certificate. And let's try with curl to connect to the server. And then in the next lecture, we can create our own client in goal. Let's create this client certificate. So this is the command that we used to create the server certificate. This is not what we want. We want to output now a client certificate with the same CA. So I'm going to do minus H. And it says, third out and key out, third out client, third and key out client key. So now we have a new certificate signed with the same CA client certificate. Let's try to curl again. So we get the error. Oh, it's still working. So I'm just going to make sure this is saved, run this, allow to make sure that when I do a curl, I get the bad certificate error because the required verify is enabled. So let's now try to use these legs key and cert. So I'm going to use a client key. And we are going to use the client dot cert. And how it is working. So when we are using a key and a client certificate, where the client certificate is signed by a CA that is in the list of client CA's, it doesn't necessarily need to be the same CA as a server certificate, it just needs to be in this list, then it will accept a connection. And this is Mutal TLS. Mutal TLS is where we can make a connection over TLS, but where the server will also verify the client certificate. And the client certificates CA needs to be within this client CA's. So these client CA's needs to contain the CA that our client certificate is signed with. So if you have microservices, for example, then you can have a lot of different server certificates. Then you can give every microservice its own certificate and its own key have them all signed by a CA trust the CA. And then you can have communication between them using TLS and where every client is also verified, giving you a very secure way to communicate between services, not only giving you encryption, but also every microservice will have a verifiable identity. It will have an identity because you can actually create a separate certificate for every client. So here are reused actually our go demo local test of me, but I could have created a new certificate called go demo client. And then I could have really defined a server certificate in the TLS AML and a client certificate, which would actually have been a better way to do it. I didn't really think it through when I was making this demo, but we should have done go demo client local test of me. And this should have been the client signed. Maybe without any DNS names. So every server can basically have its own identity using the common name. And you're going to have a different common name for every certificate. And they can have communication between your microservices that the setup with client key worked is logical because you can also use a server certificate and a server key if you want to connect to your server, because they are still signed with the same CA. So if you use a server key, it is still working. So in this simple example that I did here, it is all working with either the client certificate or the server certificate. Typically, you're not going to use a service certificate for the client, but it is actually something that could work. So make sure that you always keep your server key private to your server. Otherwise, if you have the server certificate and a server key, you could also connect to the server. And maybe that is not intended. So in the next demo, I will create a mutual TLS client to do the same that I did in Curl within Go. And I will create a separate Go demo client certificate that will have a different common name. So that we really have a server certificate and a client certificate that is not just a regenerated server certificate. In the previous lecture, I showed you how to use Curl with this server certificate and this client certificate. I'm now going to create a Go demo client. So I'm going now to create this Go demo client certificate. And I'm also going to write a client in Go that will connect to the server. So this client certificate that I created with the same common name as the server, that's not really how you do it. It was just good for a test. But I'm now going to recreate my certificate. So I'm going to remove the client certificate and client key and recreate it. So we have this TLS command create cert CA cert CA key. And then the cert name should be Go demo client local test me. And cert out is going to be client dot cert and key out is client key. And then we'll have our own common name for our client. So it will have its own certificate using its own common name, its name and not certain name name and the search is created. Let's now create a new command cmd empty last client. So this is going to be our client. So I have a main dot go. And then I'm just going to write the client that is going to connect to the server package main, funk main. So how did we do it previously? HTTP get off this host name. Go demo local test me HTTPS HTTP gets returns response and error. If there's an error, we're going to do a log fatal of the error defer response body close. And then we're going to read the body body error is equal to IO read all of this response body. And then we can say FMT print F body status status code HP status code and the body is a string. Let me have rasp status codes and the body. So let's try that out. So I'm going to run the server. Let's close all these shells. And then I'm going to run this client. Go run CMD empty less client main dot go. It just a test because I didn't add the certificates yet. Certificate is not trusted. I get so I will have to add the CA. And because we have this require and verify client certificate, we also will have to add the client key and the client certificate. How do we do that? So when I was doing this empty less server that we started the server is HP server, then we have this parameters that we can add and we can do server listen and serve TLS. It's very similar with the client. So we can say client equals HTTP client. And then we can do client gets and clients as then variables that we can configure transport specifies the mechanism by which individual HP requests are made. If nil default transport is used transport is HP transport. And what else do we have? We also have a timeout. So if you want, we can also have a timeout of 60 seconds, for example, then in HP transport, we can add a lot of transport configuration, TLS client config, TLS handshake timeout, TLS client config specifies the last configuration to use with the last client. This is the one we need TLS config, what is in TLS config certificates. And we will also have something in there called root CAs. And the root CAs are the root certificate authorities that the client uses when verifying service certificates. If root CAs nil, TLS uses the host root CA sets. So you're going to create again a sort pool and then add this CA, just like we did other server. We are going to copy this. It's exactly the same. We're going to read the CA, the CA cert, create a new cert pool. And then we should be able to validate the server certificate. Let's try to run again our client. Now we can verify our server certificate because we have the CA certificate. But now the remote which is a server says TLS bad certificate, because we are not supplying this client certificate. So if we have a look now we get client didn't provide a certificate. So we need to provide a certificate that can be validated by the server. And the server is only going to accept client certificates that are signed by our CA, because the only CA cert that we have here as client CAs is our CA dot CRT. So what do we have in the TLS conflict? We have the certificates certificates contain one or more certificates chain to present to the other side of the connection. So this is what we are going to need certificates, TLS certificates, we just need to now read our certificates. We can do that using TLS load x 519 key pair. And this accepts a cert file and a key file. So client CRT and client key, and it will then read these two files and then store it in our certificate. And our certificate can then be used here. And this is actually not a CA, this is the error. And if there's an error, then we're gonna do a log fatal. So now we have loaded our client certificate, our client key, we will make a connection to the server. And it will provide our certificate, our certificate. And it also needs the key to make the connection. So without a key, you cannot make the connection. And then we should get the body and the status code if all goes well. So let's try that out. Body status 200, it's working. So now we have made a connection using go to the server. So we didn't get an error. Server just replied with it's working. So now we have an empty less connection between our client and server. And I just want to show you one more thing. As I said, we should use a different common name for every client and every server. And the reason for that is that we are giving our client an identity. Our identity is go demo client local test me. And we can use that we can use it in our server to see what identity our client has. So if you have two clients with two different certificates, we could distinguish between them. So if you go to empty less server, we can add a function show common name, response writer, HP request. And we can then extract our common name. And then we'll add another handle funk. Where is our common name? It's in the request. So common name is going to be the common name, you just need to extract this common name. Our common name is a string. And request TLS has this information. REC TLS verified chains and verify chains has the X 519 certificate. But it's a chain. So it contains also the root CA, the client CA. So it basically has all the certificates in there. But the certificate that we need is the first one. So what we can do is if REC TLS is not nil, and the length of REC TLS verified chain is more than zero. And if the zeroed element is also greater than zero, then our common name will be REC TLS verified chains, the first element, and then subject common name. And we know that the first element of this slice is basically our client certificate. So let's see if we can get this working. Common name is show common name, save this, run this. And then go run. That's the body. And then if I look for common name, your common name is go demo client local test me. So this is the client certificate that has been signed by the CA. I'm able to make a connection. It's encrypted, but I also have an identity. The server can verify who is this client exactly, what is the common name of its certificate, and then base actions on that, it can, for example, give me more access to an API, or maybe deny access to a certain API endpoint. So this is a quite powerful tool to have within, for example, microservices, where every microservice can have its own identity. And then when we communicate to other microservices, we are aware what service is contacting us. The only prerequisite for this is that you create a certificate, a unique certificate with a unique common name for every client that is going to connect to a server, or if you have multiple servers, just for every server that you're going to run. So that's it about MTLS. In this section, I'll be talking about DNS, writing a DNS server and resolver in Go. First of all, what is DNS? Simply said, DNS provided translation between host names and IP addresses. For example, if you type in your browser, www.google.com, which is a host name, a DNS server will translate that in the IP address. So the IP address now, when I tested it on my machine is 142.250.114.99. But if you test this on your machine, it could be already another IP address. How do you test this on your machine? You can use tools like NSLockup on Windows or host or dig on macOS or Linux to do the translation of a host name to an IP address. Typically, the laptop or desktop you're working on has one or more IP addresses from your internet provider to run DNS queries. So the server that you can call to do these DNS queries is typically provided by your internet provider. In Linux and macOS, this you're going to find in etcresolve.conf. And on Windows, it's going to be in the DNS server settings. You can check this with IP config or you can go to the network settings on a macOS machine. So even if you're not using any tooling yourself to translate the host name to an IP address, whenever you are using your browser, you type in www.google.com, the browser will ultimately translate the host name to an IP address using a DNS server. So DNS is something very common that you do in the background on a daily basis. So how does a DNS request look like? So you have your browser on one side, your browser is going to do a lookup for, for example, Google.com and it's going to do this lookup using a DNS resolver. So your internet service provider DNS resolver will be used or other public DNS resolvers like 8.8.8.8 or 4.4.4.4, which is provided by Google. So it's not because we are doing Google.com. It's just something that Google provides. Google provides a public DNS server that you can set up and use instead of your internet service provider, DNS resolver. Or you can also use 1.1.1.1.1 and that one is maintained by Cloudflare, which is not a company. You can also set up this IP address as your DNS resolver and then you'll be able to resolve host names using that DNS resolver that is running on that IP address. So every time your browser is going to go to Google.com and you wouldn't know the IP address yet because once you know it, you can cache it. But if you don't know it yet, then it's going to make a call to this DNS server to obtain the IP address for Google.com. This DNS server doesn't really know where to find the IP address of Google.com immediately. So it will have to go to something that's called the root servers. All the DNS resolvers will have these root servers configured in their DNS resolver settings. So they'll have a conflict file somewhere where they have the IP addresses of these root servers. These are the IP addresses that need to be known to be able to start DNS resolving. So this DNS resolver will then go to one of these root servers and it will ask, what do you know about Google.com? It can ask what you know about Google.com or it can ask what you know about .com because the root servers will not know where to find Google.com, but they will only know the top level domain, the TLD. So as a DNS resolver, you can actually choose to send Google.com to the root servers or just .com. So we're going to ask, do you know about Google.com? And they will say, no, I don't know anything about Google.com. I am not authoritative for that. Try .com at IP addresses. So the root server will say, Google.com, I don't know about, but I can tell you where to go to get information about .com domain names. So the root servers only know where to find the servers of the top level domains. And it will reply the host names and IP addresses of those servers. So then our DNS resolver knows that it can go to another server, specific servers for .com domain names. And that is then what the DNS resolver will do. The DNS resolver will then go to these top level domain servers and will ask, do you know something about Google.com? I want to know what the IP address is. And the TLD servers for .com will then answer, I am not authoritative for Google.com. But the Google.com name servers, the Google.com DNS servers, you'll be able to find at this IP address. It can also say you can find them at these host names. And then you would still need to do more lookups to know the IP address of those host names. But in the most simple example, it just going to give us the IP address. And then we can just go to this IP address where the name servers for Google.com are. So these are the Google.com name servers. And then we're going to ask again, what is IP address for Google.com. And the Google name service, I'm going to say, yes, I have the IP address of Google.com. I'm authoritative for this domain name, so I can give you this IP address. And then the DNS resolver will then be able to send to the browser. The IP address is this IP address that I found for Google.com. The DNS resolver will then typically cache all these IP addresses. How long depends on another field that is being passed. There's a time to live TTL that is also passed. So that TTL will say how long these records can stay in cash. And the DNS resolver will then keep them in cash. It doesn't always have to go to those servers because Google.com, for example, or just to come till the our credit quite a lot. So it doesn't really need to go to the root servers every time it can often skip some servers because they already resolved domain names earlier. But this is the most typical flow where the DNS resolver first goes to the root server, figures out where the top level domain is, then goes to one of the top level domain servers. And then they will give you a hostname or a hostname and IP address. And then you will be able to go to the name servers for that domain name. And they will give you an authoritative answer that then can be sent back to the client. So what are those root servers? So those root servers are machines hosted by operators, companies that maintain these servers. These IP addresses, they don't change that often or at all. So they are all configured within these DNS resolvers. There is a website where you can also go to to get the latest list. So that lists you download as a config file in your DNS server. And these IP addresses you will have to steadily configure to be able to know where the root servers are so that you can use them as a starting point. So it's not that every machine, every laptop or PC has this list. They typically have these DNS resolver configured. And the DNS resolver will then reach out to those root servers. It's not typically the browser itself that will do that. The browser will just use a resolver that then can handle any DNS resolving for the browser or the user on the machine. Also maybe interesting to note is that even if you see a single IP address here, one IP address is not equal to one server. There are actually multiple servers scattered around the globe that are representing this IP address. So there's also routing in effect that routes you to the closest root server in your region. So there will be multiple servers responsible to be able to handle requests for a single IP address that you see here. How does a DNS packet look like? So before we are going to start writing this DNS resolver in Go, let's have a look how a DNS packet really looks like so that we know a little bit more in detail how the protocol works so we can then easily implement it. Once we will start writing our Go program, we are not going to parse the DNS packet ourselves. We are going to use a library that is available within Golang for that. So don't worry if you don't fully grasp all the concepts that I'm going to explain because we will still be able to use this package that comes with Go to be able to pack and unpack these DNS packets. So a DNS packet can be sent to the resolver using UDP or TCP. So if you are a browser then you can send a DNS packet to the resolver and the packet can be UDP or TCP. So the difference between UDP and TCP is that TCP needs a handshake so it takes a bit longer to get an answer. The handshake within TCP gives you an extra bit of latency whereas UDP is stateless in just one single packet so there is much less latency to get an answer. A DNS UDP packet cannot be larger than 512 bytes. This to avoid the UDP packet being fragmented over multiple packets. Everything larger than that need to be sent over TCP. Let's focus solely on UDP packets to be able to write a simple DNS resolver server. So when we are going to write our DNS server, our resolver, we are only going to do the UDP implementation. So we will not be able to take any packets larger than 512 bytes because that's the maximum. You could use the same mechanism to then also implement TCP later on but I'm just going to explain the concept and we're going to keep it simple so we're going to just do UDP. So what is a DNS packet? How does it look like? So our browser will do a lookup and it will use resolver. It will send a DNS packet using mostly UDP and this DNS packet consists of headers. There will be headers and then actual data so the actual data which is going to be the questions and answers I'm going to go over later let's start just with the headers. This DNS packet headers is going to be the same whether it's a question or an answer. So if the browser sends a lookup to a resolver it's going to have these headers here and if the resolver sends something back is also going to have these headers that I'm showing. So it's not that they have different headers for requests and responses it's going to be the same format headers they're just going to have different values. So the headers start with a query ID which is 16 bits long this is just a unique ID that is used to send a request. When the resolver then answers it will also use the same query ID so that the browser that did the lookup will know that this is an answer for this specific query ID. On the next row you will then have flags. You can find the explanation of these flags online if you want to know where every flag stands for. We are going to use this AA code here so authoritative answer if a server sends an authoritative answer then this flag will be set and this is something we will need to check on to know whether we reached the server that can give us the correct answer or whether we have to reach out to other servers. Then the QR for example is question or response so this one will be set to one if it's a response or a zero if it's a question. Then we have our code which is result code which can contain an error code if there's an error and there's also TC for truncation if this use this flag needs to be set to one and then you have recursion desired and recursion available. Recursion means that if the service is not authoritative that it will itself then go reach out to the authoritative server to get the request so a resolver is actually recursive because the resolver will go to the different servers to find you the answer. If it wouldn't be recursive then if you would ask for google.com it would just say I'm not authoritative for this domain name so here's an error because it is doing recursive lookups it will actually go and find the correct server for you and find you the answer. This Z here is also reserved so we are not going to use this. There are some other flags as well for example the R code here the result code if there is an error code then this flag will be set for example. So these are the flags and then you also have more information in this header you have the question count so if the browser is going to look up one host name then the question count will be one the answer counts if it's an answer it can be higher than zero if it's a question only then the answer will be zero. The name server count so this is going to be necessary if you cannot get the answer from the server but the server knows other name servers where we can get the answer from then this will be set and this is just a count so this is a field that just counts how many occurrences we will have in the data part of our packet and then we have the additional count the number of additional resource records so this counter is showing how many additional records that we have for example this can be IP addresses if we have the name servers available then the name service will only be host names if we also have the IP addresses then it'll be available in the additional records and here we just have the count of those records so this is just the header now we can have glued together within the same packet also the questions and the answers so this can be our header for the lookup and then we're going to have also questions so then attached to the same packet we're going to have this question part and this part is going to have a question name the DNS name to resolve so in this case it's going to be google.com names are split into labels and start with the length of the label first and end with a zero a zero octet so it's going to be double zero if you are reading it in hex decimal for example if you have www.google.com you will see three which is the first part the first label that's the length of three www is length of three then six because google has six letters so the length of google is six and then three more because then there is come and then it ends with a zero in the next decimal this will be zero zero then we're going to have a question type which in our case is going to be an a record but it can also be an mx record mx record is for emails so if you want to know the email server where you send the emails to then you will ask for the mx record otherwise you will ask for an a record a record is for ipv4 if you have four a's you're going to ask for an ipv6 record and then you're going to have the class the question class and that's going to be typically internet address so there's a code for internet address and that is the code that we're going to use so this question packet together with a header is going to be sent to the resolver we also going to have an answer part of the packet and that's going to be attached to the question but it's going to be empty because we don't really have an answer when we are only asking the question what is the ip address of google.com so let's have a look at an example packet let's say that we want to do a lookup for www google.com then we're going to have the header the question and the answer within our packet so the header is going to have a unique id 59e2 is our id in hexadecimal format then we're going to have our flags which will include that recursion is desired and that we have one question within our question we'll then find this length three which represents three times w then length six for google length three for cum and then it also going to pass the type and the class and type a for the a record is going to be one so in hexadecimal it's going to be one and the class is also going to be one so this is how our question is going to look like so this three six three for the label is just to make it easy to parse so programmatically this need to be parsed and because this field doesn't have set lengths like the header we also need some help to know where the field starts or ends and these lengths they help us with that we will not have to do these ourselves so like I was already saying we have a package in golang that will parse this for us even though it's going to be parsed for us it is actually very beneficial to have this schema in mind so that when we are going to be digging into this package that you have an id what the information is that is available in the header what is available in the question and what's going to be available in the answer so let's now have a look at the answer packet so the answer packet is going to come back from the resolver to the browser so the answer packet is going to have the name as well which is the same format as the question name in the question packet so if you ask for google.com then it will also send back google.com also the answer packet will have the same headers so this packet that comes back I'm only showing the answer part of this packet it will still also have the headers and the questions in the same packet so we have the name the type the type is going to be a record the type can be a record mx record or something else the class the time to live so if anyone is caching this record how long can we catch this record that's what the time to live is going to tell us in seconds the response data length so how long is the response going to be and then the response data so response data for a record is an IP address so first we are going to say how long this IP address is and then we're going to actually put straight after that the response data so what does the packet look like we are going to have the same id in the header we're going to have again the header the question and the answer in the header we're going to have the same id but then our header our flags will look a bit different you will see here the hexadecimal representation of the flags which are not the same ones as when we were sending the question because here we are setting for example that we have one answer available we then have the original question in our question packet so google.com is still here and we then have the answer for google.com the IP address so the answer from your resolver can come back like this the first cell here c00c is actually a pointer that is saying the name can be found at the position 00c and that's going to refer exactly to google.com in our question data so that way we don't have to repeat the name in the answer so there's a mechanism to make the packets smaller then we have the type so it's an a record so it's going to be one the class is also going to be one the time to live so this is the representation of time to live if you would convert this hexadecimal number to just a number then you would get the amount of seconds that you can cash this entry then how long is the response going to be the response going to be four bytes so 16 bits so eight hexadecimal letters and if you would take those eight hexadecimal letters and convert this again then you would see the IP address that starts with 142 so this is an example answer packet that could come back from the google server that then the resolver sends back to the browser so if you would like to know exactly what is in this header then you would need to expand these hexadecimal numbers because you would need to see on a bit level whether a certain flag is set to zero or one to see what flags have been set so lastly go packages luckily go has packages available to parse those dns packets so we don't have to do it ourselves golang.org X for extra net dns dns message is the package that can be used to unpack which is decode and pack and code dns packets so when a dns packet comes in over udp those bytes can then be parsed by this packet and that way we don't have to manually go and try to parse this binary data we have this package available to us to do that for us and then we just can use this dns message to extract the information that we need to build our resolver so then we still need the net package that can be used to listen for udp packets and send udp packets to other dns servers so those are the main packages that we would need in our program to then write a dns server that can resolve host names if you would send a query to it and that's what i will show you in the next lecture in the next lecture i will show you how to build a dns resolver in go before we start writing our dns server our resolver in go let's go over the logical flow diagram so that we get an idea of what we are going to create how is our resolver going to look like well we are going to have to listen to udp so we're going to create a udp socket on port 53 port 53 is the dns port we'll then have a client that is going to connect to our udp port on our machine and the client is going to send a packet so we need to handle this packet whenever a new packet comes from a client we need to handle it and we can then start parsing it so we can use this dns message package from golang to make it easy to parse a dns packet but we still need to figure out what fields we need to extract and have look at so the question part of the packet looks like this it will have a question name for example www.google.com question type what do we need to know for this name for this hostname we need for example a record the question class and for us it's always going to be an internet address let's go in a little bit more detail so we're going to parse our question and then we're going to create a for loop we are going to do a few actions in a loop until a certain condition is met we are first going to query the server the first dns server that we're going to query is going to be the root server or one of the root servers when the dns package comes back from the root server we are going to ask a question is the answer authoritative yes or no if it is yes the first iteration of the loop it will never be because we are asking the root server and the root server is always going to retract us to another server but this is part of our loop if it is yes then we can send the answer back to the client if the answer is no and if we are sending a query to the root server it's going to be no then we need to parse the authorities section in the dns packet that comes back from the root server to get the new servers so those servers are going to be the name servers for example.com so once we have these new servers we'll have to again query the server but instead of querying the root servers we are going to query this new server this new authority that our root server gave us in this dns packet that we now know and that we now can query so let me quickly show you with the dns tool how this actually works so I have a tool called dick and I can use dick trace google.com and it's going to show me the trace of all the queries that you would need to do so the first query is going to be to the root servers so these are the root servers a root server.net you saw these names on the slide so we will query these these root servers and we'll ask where can I find .com or google.com typically you send less data to a dns server so it's going to be .com and the query would then go to one of the root servers and the root server would then say here look these are the name servers and it's going to be in this authority section so here we have a dot gtld server.net and these are going to be then the servers we can query then we can go to these servers and ask where is google.com and then they say google.com you can find those at ns1 google.com ns2.google.com and so on and then we can go to one of these name servers that belong to google and then we can get our a record so these would then be a few iterations in our for loop to get eventually to the IPRS there's just one detail missing here and when we say is the answer authoritative then we say no then we're going to pass the authorities to get the new servers any server that we're going to query might actually not give us the IP addresses so the authorities only contain host names we might get the IP addresses but we also might not get the IP addresses so let's have another look in this for loop in more detail so we have the query server is the answer authoritative no then we're going to pass the authorities to get the new servers these are typically the host names and then we're going to pass the additional resources and in those additional resources which is part of the DNS packet we are going to find the IP addresses if the server the DNS server is going to give them to us in this DNS packet so here we have to ask ourselves again a question is the IP address included if yes then we have IP address and we can query the next server and do another iteration of our loop but if it's not included then we have to start the whole loop function again to query a record of the name server for example if the name service ns1 google.com then we have to start the whole loop again so typically when we are going to query google.com and google.com has a name server within the same domain name they will supply the IP addresses but if google is using another name server of another domain it might as well not supply the IP addresses if we are querying google.com and the name server is ns1 google.com then typically the IP addresses will be supplied because otherwise there's no way of knowing the IP address because we would just end up in an indefinite loop it's often that when we could query google.com and the name server would not be in the google.com domain name it would be something completely different for example it would run another company for some reason then the IP address might not be known and then we would have to do a lookup so then we have to start a new loop we kind of have to go again in the same function to do a lookup of this domain name here with google.com the IP address is actually supplied so we could immediately create a new server but with other domain names it's not always the case so although main name xyz might have an amazon name server and then we have to look up the amazon.com domain name get the IP address and then only we can continue so once we have this IP address of this ns1 google.com or some other host name then we can actually continue because then the IP address we have and then we can go to yes and create a new server with that IP address so this will become more clear once we start writing the code you just have to remember that we are going to parse the authorities to get these servers we are going to parse additional resources to get the IP addresses if the IP address is not included we are going to have to do more queries if the IP address is included we can just continue our loop and create a new server that we have found let's now have a look how to write this dns server which will be our resolver in go i opened my dns start project that comes from my github.com repository go lang for devops course and in this dns start project i have a cmd dns resolver so to start my project i will enter something like go run cmd dns resolver main.go which just says starting the dns server there's no code yet so here i will have to write my code to start the server to listen to udp packets then i will have a package the resolver.go and this resolver.go it has the root server defined so that you don't have to copy paste it in there and i will keep it up to date if there's a ip that would ever change then i will update it here and this resolver.go has the function handle packet so we can start a server main.go but then we can use the function handle packet in this dns package to handle an incoming packet the benefit of doing it like this is that i also wrote a test file here i have this test file to test this test handle packet where i will try to resolve two host names and if it returns an error then it will stop so if i execute this now then serve error not implemented yet because i have the error not implemented yet returned here within this handle packet we will also have to contact other dns servers and for that we have the outgoing dns query and for this outgoing dns query which will make a dns query to a server defined here in the parameters for that i also have the test outgoing dns query test where we test where we can reach the root servers and we're going to try to try to resolve the com which we know that we can resolve that we're going to ask for the name server type so we're going to ask the root server what is the name server for dot com just as a test and if that works then our test will succeed so right now doesn't work we have no header found because the header is nil because we are just returning nil so these tests they test these two functions so if you use this start package with these functions it's easy to actually test whether your code works i also have this mock packet con because that's what i'm passing here i'm asking for a packet connection so this is an interface for a packet connection because we will have to use this variable here this pc to send something back to the client so we can send a udp packet back and that's why we have this so i'm just mocking this so that we can test this function without having to send a package back well we will trigger sending a package back in this function but nothing will really happen if you mock this function so i'll first start with this function should be relatively easy outgoing dns query where we're just going to contact over udp another dns server and next i will then start the server and then start working on the handle packet code and then when everything is finished the test should also succeed let's start with this outgoing dns query let's have a look at our signature so we have servers the service is a slice of net ip so net ip represents an ip address it's in type bytes so we'll have to convert a string ip address into bytes if you want to use this but that is actually quite easy if we have a look here we have the function netpars that can take a string and it returns a net ip address so we have our servers can be the root server can be the google name server can be the amazon name server can be the name server for the dot com tlds for example and then we have the question and the question is of type dns message question so i was saying that we are going to use this dns message package so a dns message question is a question struct that contains a name type and class just like i showed you what a question really contains in a packet so it contains a name the type and the class and that's what we need to then ask a question to an external dns server we are going to return the dns message parser which will then give us access to the packet that is being returned from the server and we'll also return the header just like i showed you in the slides the header contains an id response which is a flag operation code whether it's authoritative or not whether it's truncated recursion desired recursion available authentic data checking disabled and the r code so this is the header of the packet this is the parser where we can then get more information from the packet so if we're going to trigger this this outgoing dns query then we can trigger this with a question here's our question dns message question it has a name type ns and a class and then it's going to call this outgoing dns query using the server which is the first of our root server because we just kind of split them based on the comma here and here it is comma separated ip addresses and we're going to take the first ip address parse this and then parse this net ip address to outgoing dns query to get rid of question so what are the next steps that we need to do to then actually reach out to the server and ask this question we have to start by sending a udp packet to the server and what are we going to put in this packet well we're going to put a dns packet in there and to put a dns packet in there we need to craft our message and package it and then we'll be able to send it so let's start with making our message so our message is going to be of type dns message message which is a struct and in this struct we can put our header and our question so this is going to be our dns packet so we have the header we have a look at this message we have the header questions answer authorities and additionals to send a question to a server we only need those two the header and the questions questions is slides and i already have my questions available here so the header is of type header what do i need i need an id for sure and i need to say that it's going to be a question so response is going to be false we don't need to enter anything so what i might do is just for clarity add a few of these headers like is the response no it's false but then it's clear that this is a question let's start with the id because it also comes here first and the id is an unsigned int 16 so you want to return a random number between 0 and the maximum unsigned int 16 what is this maximum we can actually use the variable max and assign this u int 16 0 with this flag in front of it which will return the maximum possible unsigned int 16 so this max will have the maximum unsigned int 16 in this max variable and then the next step would be to use the crypto rent library to generate a number between zero and this max unsigned int 16 how do we do that we have the rent library and then this one shows math rent but we need crypto rent we need an integer this needs a random reader and then we need to supply a big integer and it then returns a big integer which we then have to cast back to unsigned integer and an error if there's an error so rent reader and then big has a new int new int which accepts an int 64 so we can say in 64 of this max we change this type from an unsigned into an int 64 now we have the maximum here defined and we can say random number error is run int if there's an error return the error if there's no error you would still need to convert the random number to an unsigned integer so unsigned 16 of this random number but this random number is of big int so random number has actually a function to return an int 64 and then once we have an int 64 we can convert to an unsigned int so it's all of steps just to get the right format but then we have exactly this id in the unsigned int 16 because that's a type that it asks what's next let's have a look in this header again opcode opcode and opcode is a DNS operation code so this opcode if you're going to have a look in this package then we see that opcode is of type opcode and it's also unsigned in 16 an opcode is a DNS operation code and we typically just use a static value for this which is just going to be zero so we can use DNS matches opcode as a function to change the type of this zero to opcode so now we are changing this zero here to type opcode so we're using this type of code to match the correct type and this is just going to be zero what else we can leave it like that I'm not even sure if we really need those last two fields because probably the defaults will be zero and false anyways a just to make it clear that we need these set like this so we have the message we have the header then we have the questions so here we have one question so we can say DNS message question and we have only one question so this is a slice of the type question and we only supply one question now this message is of DNS message message and this DNS message message is struct and has the pack function which returns the bytes so now we can just say buffer error equals message dot pack and now we can use this buffer to send to our DNS server if there's an error we're going to return it so how do we make a connection to our server we can even have multiple servers so we should do this in a loop for server in range servers we can use net dial so net dial dial connects to the address on the name network no networks are TCP and also UDP UDP so we can say we are going to make a UDP connection to a certain address the address is the server and this is of type net IP and this needs to be of string so we're just going to use a string function and we're going to do this on port 53 because that's a DNS port so dial here are some examples so here's an example for DNS this is an IPv6 but we are going to use IPv4 so we are doing dial and what does dial return a connection and an error our connection we probably want outside our for loop so I'm going to say var con is net con con and error equals net dial if there's an error we are going to continue so if there is no error then we can break and if there's an error then we will just continue until we hit a server that actually works so we then just need to check our if so we either then need to check our error or our connection so if connection is still nil there's no connection made then we can return an error failed to make connection to server servers I will just say what is the error return nil nil and then the error so now that we know that we have a connection we can write something to our connection so we can use con write access bytes here's our buffer and then it will return the bytes that it has written and the error so if there's an error then we're gonna return the error if there's no error we're going to read the bytes from our connection as I've looked at this connection so read and write and close so we can use a reader a goal and reader a stream reader to read this out I can use buff IO new reader of this connection and then I can read this into a variable reads data into p so I just need to define a variable where our answer will go in I can make a new byte and it's not going to be more than 5 and 12 bytes because it's a maximum and I can read my answer or read will read data into p p is my answer and it returns the amount of bytes that it has read and the error so if there's an error I will return it and now I can try to parse this again with our DNS message because this what we get back this answer should be a DNS message before I'm going to do that I can actually close the connection just to make sure that we closed already because we don't need it anymore we just want packets so we can close it and then I need this DNS message and this DNS message has a parser parser is a struct so now this is all going to be empty and our parser has a function to start parser allows incrementally parsing a DNS message when parsing is started the header is parsed next each question that can be either parsed or skipped alternatively all questions can be skipped at once so basically because we are doing it incrementally we can say first start then we get a header and then we get the questions if we start with the questions if you don't need questions we do skip questions let me just hear skip questions or skip all questions so that we can either go to the next questions if there's more or skip all the questions if you want to go to the answers all answers authorities and additionals can be either parsed or skipped in the same way and each type of resource must be fully parsed or skipped before proceeding to the next type of resource so this is a funny way to use this package but this parser is very much optimized so it's not going to read the data that you might not need so first you read the headers because the headers come first then the questions if you don't need questions you can skip the questions and then you can start reading the answers and you can do the same then afterwards with the authorities and the additionals know that there is no requirement to fully skip or parse the message so if you just need the headers you don't need to skip anything or if you only want to read the questions you don't need to skip the questions you can just read the questions so this is the parser so we're gonna say p is our parser p has start and accepts our bytes our bytes are an answer so we're gonna say answer but are we going to send all the bytes to our parser because we only actually need the bytes that we have read so this is 5 and 12 big but our message might only be 100 bytes so we're gonna stop at n because that's what we have read so if you read only 100 bytes we get the first 100 this gives us the headers and the error so headers error if there was an error we just return fmdrf so it's clear if there's an error parser start error the error and then we could use the headers now what we could do is we can check the header whether our headers contains what we want and we can also check the question whether our question contains the question information that we were looking for and if that is all okay we would then return the parser so that we can later use it so we're going to return the parser and we're also going to return the header so we can also use the header later on so those are the headers or is the header it's actually is it a header or headers it's a header so that way we can then use the header and the parser later on if after calling this function what else do we need we might want to do some checks here whether we have a success response code or not so let's maybe do that first then test it and then write some more checks just so that we can do some basic checks for example whether we have the same question as we asked for so before i'm going to return the parser i already want to check the questions because there might be something wrong in the questions and we want to do these checks first then skip the questions and then when we return this function we can immediately continue with the answers so the questions are p questions or p all questions and now if i want i can check whether we have the same question so we can i can say lan questions is equal to my original message questions so here i can also check the length so that my original message is here on top so do i have the same amount of of questions returned that i started with if not i can return an error because then something went wrong and then i can say okay now i did my question checks now i'm going to skip the questions skip all questions and this only returns an error so here just between all questions and skip all questions i can do my question checks if i want to check these things for example i can check now whether question name is equal to the original name and then i skip them and now next i'll be able to return the parser and the parser will be able to process the answers so processing the answers i will then do in the handle packet so i could test this now because if you have a look here i'm going to have a header i'm going to have an answer my header could check the response code then i can skip all the answers and then i can check the authorities so if i want to check the authorities i need to skip the answers even then i want to check on the additional resources i need to do skip authorities so that's how it goes you go down the packet by getting the questions then the answers then the authorities then the additional records so let's test it by running the test if something goes wrong you can always do a debug test to see new outgoing DNS query for com on the servers and my test is passed if your test wouldn't complete and you have exactly the same code also check whether you actually are allowed to contact the root server directly maybe your internet provider would forbid that by blocking port 53 for example to anything else than their own DNS servers that is a possibility a remote possibility but most of the internet providers just allow DNS because you also have Google's and Cloudflash public DNS servers so that should in general that shouldn't be really a problem if you get an error or panic have a look at the code whether you did something wrong different and you can of course use the debug test so it's not much happening here we are just doing a test and here we are checking whether we are getting the answers and we're getting some answers so i could actually output the past authorities just to see if it actually is working past authorities and then run this and then i have the past authorities so we have the header name type class time to live and a length so we have the name com and the ns records so here we have all these ns records for the com domain name so that we know what the name servers are for the dot com top level domain name you could also change this into google.com figure out what the name service for google.com and then use the server name server google.com here and then you could do a query on google as well so then that could work as well if you want to test so test outgoing DNS query just makes with net dial a connection to the server or to the list of servers and then writes our message is our message with our questions and then parses the DNS message using the parser we parse the questions already in our function which we don't have to do i'm just doing it here you can also just return the parser without doing this and then you have to also parse the questions every time you are calling this outgoing DNS query so it's up to you whether you want to do that or not and then to test you can run the test and if your test succeeds if it passes and you can print the authorities for example and you get something then we have made a call to another dns server so this will be a building block for when we are going to handle an incoming packet in our dns server when we are going to run as a server because when we have requests for for example google.com we will have to first contact the root server the root server will tell us where the dot com top level domain server is then we can ask this server where google.com is then we can do a dns query to the google.com dns server and every time we are making dns query we are going to use this outgoing dns query function so this function is quite important to have this one working first and the next step is going to be to write our server let's now start writing our resolver so here i am again in visual studio code we wrote our outgoing dns query now we have to start a server and then write this handle packet so that we can run another test our test for test handle packet where we try to resolve google.com and amazon.com we are going to put these in a dns message it's going to be the header it's going to be the questions so name type an a record we're going to ask we are going to pack this into a byte slice and then we're going to send this to a handle packet so we use a mock packet connection so because we are not really initiating a real connection our ip address is going to be localhost and then we're just going to send our buffer which is this dns question if this all goes well then our test works we might want to write some other test though because this test is just going to test a handle packet you might also want to manually test a few things how do you test whether our dns server works on mac or linux you can use dick dick with and then the name server that you want to use and then for example google.com nothing is running yet so it will not work with ns lookup on windows you can use ns lookup and you can for example say on resolve google.com on the name server 127 0 0 1 and the same on linux as well if you want to resolve google.com you can also do that on 127 0 0 1 so there are tools on your machine that you can use that are installed by default or that you can install for example dick you can install both on windows linux and on mac ns lookup should be default on windows ns lookup also has an interactive shell where you can just type requests it is something like ns lookup and then you just don't enter the hostname and then the name service 127 0 0 1 and then you're going to end up in a shell in windows there's also ns lookup on linux and on mac but i never really use that maybe it is even a bit compatible i often use dick or host so this is how you test in a shell but if you don't have any tools you can also use test handle packet which test this handle packet you could even write your own test here that is going to do an outgoing dns query on your own server that could also be a possibility so you could actually also write everything in go so these are our tests we don't have to really look at it straight away we can first start writing a little bit of code and then we can enter debugging for example to see how it goes or just what i also do is a lot of print lines to see what actual content is when we are then using the command line tools to do manual dns queries so let's get started with the server starting dns server and now we need to start it for real i'm going to use that net package listen packet the listen packet function listen packet announces on the local network address and then we can process the packets the network is udp and the address is just 53 so we're going to listen to all the ip addresses that we locally have and this is going to give us a packet connection and an error packet connection error what if there is an error if there's an error i'll just panic for now it's plain simple so that we also get the error message we want to defer packet connection close so at the end of the function we want to close the socket that we opened and then we're going to have a loop a for loop so we're going to read the bytes if there are bytes coming and if there are bytes coming we're going to put them in a buffer and then we're going to trigger a handle function this handle function this handle packet is local so we're going to create another one for and then let's make a new buffer of type bytes packets are never going to be bigger than 512 so we can put just 512 here to make it ourselves easy packet connection read read from reads a packet from the connection copying the payload into p our payload is going to be the buffer and then it returns the number of bytes read the network address and an error so bytes read the address and an error well if there's an error we can just print something read error from the address and this is the error address string this is going to return as address in the string and the error and then we just want to continue because then we can just continue the for loop if there's no error then what we can do is we can use this handle packet but if you would use this handle packet then we are not going to go back to our for loop because then we will be then we would be executing that function so what we want to do is we want to use a go routine here so that we return back into our loop and then we will be blocking here we will be blocked here until there's not a client that has set us some data and if there's not a client that's not some data then you will again execute this go routine in the background so we are going to create a go routine for every client that is connected and that sends us a package so for that we need another function handle packet and it is also in a package the DNS package so I need a capital H to make it accessible and then go routine is not going to return anything so handle packet go handle packet what are the parameters my packet connection the address and the bytes packet connection the address and the bytes the bytes are in the buffer how many bytes am I going to pass all 512 no I only want to pass the bytes that I have read so I can say from the beginning to the bytes read go DNS handle packet it's gonna be and then it's going to be importing my DNS start slash PKG DNS now this should be all the code I need in my main.go and now I can go to my resolver.go and then here I can say if handle packet which returns an error same parameters if the error is not equal to nil then I'm going to just print something so I have something on screen handle packet error maybe I will pass also the address so that I know and that's the going to be the error the address string and the error save this maybe I want here as well a return at the end handle packet handle packet so now this is the only code that we have to write outside our test all the other code can go straight into this handle packet let's have a look in this handle packet what do I receive the DNS packet how do I parse this I'm going to use the DNS message to parse this packet and then I can say P start of whatever is in my buffer and it's going to give us a header and an error and if you have an error I will just return it now that we have started parsing this DNS message I can extract one or more questions P has all questions which is a DNS message question slice or just question for simplicity now I will just assume that there's only going to be one question but we can also do all questions and then we can have another loop but I will keep it simple for now so we're going to say P question so the question is P question and also returns an error I think question an error question an error if there's an error I will return it there and now we know that here we have a question now we can see what is in this question so it's going to be a host name a type and a class and that is our question that we have to solve what is the type the a record of a certain host so we're going to use a separate function for that we're going to do a DNS query and we're going to query a specific server with our question so we're going to have a question and we're going to have a server or servers what servers are we going to use well the first time we're going to run this is going to be the root servers but we still need to parse this so I will also make a separate function for that get root servers so we're going to do a DNS query function we still have to write and we're going to pass the server to you on the query and the question that we have once this is done we'll have a response maybe an error as well and that response we then can send back to the client so how do we send it back to the client if we have a response well let's first fix the error if there's an error we're going to return it let's say we already have our answer our response what's going to be the format of this it's probably going to be a dns message and then maybe just a whole message that's what we want to construct in our DNS query and then we can easily send it back to our client so what is special about this is that we have to use the same ID we have here this header with an ID in that is randomly generated by a client so what we want to do is when we send a response we want to make sure that this ID is the same so dns query let's try to write our function for this dns query get root servers we can reuse this it's going to be always an IP address we're going to have a question so this is the same we have a question that we're going to want to pass the question here is of type dns message question what are we going to return we want to return a full message dns message message so it's going to be a full message that we want to return so we can send it back to our client and then we're going to have also maybe an error so I'll just return nil nil so we don't have any compile errors then we have the response which is dns message so if there's an error we return the error get root servers maybe you should pass this first so this needs to be a net IP address so let me write this first get root servers maybe all the way at the bottom get root servers and we're going to apply a net IP slice root servers is strings split well split these root servers it's comma separated and then we can loop those root servers and then we just need to create a new IP address for every root server the net package has parse IP which takes a string root server is a string and then we need to put this in root servers I know I will call these root servers are net IP and then I will just move this here so our root servers are net IP slice and then we just need to append it so now we convert our strings that we split into a net IP return root servers so now we have a slice of net IPs with IP addresses of our root servers so get root servers and then we have the response what are we going to do with this first we need to make sure that our response header ID is the same as our header ID because the client sent us a DNS packet with a specific header ID a DNS header ID and we need to when we send the response have the same ID and now we could send this to the client our response is still in the DNS message format so we need to do response pack and that will return the bytes so if you say these are new buffer do I need to make a new buffer for that response buffer error if there's an error we return and then we have this net packet connection that we can use PC write to we can write the response buffer where are we going to write it to to the address that we also passed here this is a client address and then we have the bytes returned I don't think we need these bytes returned we just need to make sure that if we have an error that we pass error and then return nil save this so then we just have to write our DNS query function which is going to be very long and complex so to try to see if the code that we have written works let's try to return a simple DNS message an error for example so our DNS message is going to contain a header and remember this header also has in this DNS message header field should I put returns probably yes we have this rcode rcode is response code and what if we just send a failure rcode server failure so we have a server failure then at least we are responding something back to the client so that when we have a packet that comes in we can see if it gets parsed if we get the questions and then we can do the DNS query now that we also have the questions let's just also print out the questions that we have to verify where we can actually see something question because we only parsed one question so now we can actually test this easily we could actually write a unit test of this handle packet with a little bit more simpler names we could do a test handle packet and see whether we get the right code back we could actually use this test I think because we are not really checking on errors we're just checking whether whether this handle packet doesn't return any errors and will it return errors no it will not return errors I think because we will just send something back to the client a server failure so let's maybe do the unit test first and then let's do it using the command line okay so this worked we have printed the question google.com type a amazon.com type a and it just passes because we are not really checking whether the packet that comes back contains the answer that we are looking for we are just simply checking whether we are not getting any error so that works now let's try to run our server starting the DNS server I'm going to open a new shell and now we can use dick or nest lookup or host dick I'm going to use dick first so dick add one two seven zero zero one and then I'm going to try google.com and what did I get status serve fail so I did actually get a serve fail so my server is running I got a response because let's try actually first what happens if there's no server running you see nothing happens I cannot connect it will time out then I can run the server and then you will see immediately the question coming up because dick keeps on retrying dick keeps on retrying which can also be annoying at some point so you can also put it off there is something like tries tries so if you do dick plus tries is one there will only be one try tries one I'm carrying my local host google.com or xyz.com and then here we'll get google.com and xyz.com so our server works we are able to handle packets we are launching go routine for every packet we are passing our question that works we are starting the DNS query function with our root service with our question we are printing the question but we don't have any other code so we are sending a server failure and then it goes back here we fix the header ID we package our response into binds and we send the bytes back to the client and then our client is actually getting a serve fail so how does it actually look with other clients so if you would use host host google.com one two seven zero zero one you also get the serve fail so google not found serve fail let's try with ns lookup how does it work I haven't actually used it before yeah it's the same so ns lookup and then ns lookup you can use on windows I wonder where it is actually the same behavior so yeah if you use ns lookup dash one two seven zero zero one then you can just type your host name that you want and then make sure that the server that you're querying is then one two seven zero zero one so that all works you can use dick host ns lookup connect to our server or you can use the unit test here benefit here is that you can do debugging so I can say I want to know what this ID is so I put a breakpoint I run debug test and then I have my debugging console so what is the ID the ID is this hexadecimal number how does my response look like it's a message with headers these are the headers there's no and there's no answer there's only this serve fail where is this serve fail the rcode our code serve failure and then I can actually continue to see what happens and then it does the same for amazon because we have a loop here so I can actually continue here and then we are doing the same for amazon and then it stops and here's my output it passes so multiple ways to test this which is going to be necessary because you want to test this all the time every time you write some code you want to see if it is working so next we are going to write this the nest query this will be the actual meat of our resolver this is where we're going to do our resolving for real before we continue let's have a look where we are so we did the listen to udp socket on port 53 we tested whether a client can connect we have a new packet that we then send to the handle packet function we parsed the question we didn't send it to the root server yet that's going to be the next step we actually just returned an answer after the parse question back to the client to test whether that works so what are we going to do now is we're going to enter our for loop and our for loop is going to query the server first the root server we're going to check whether the answer is authoritative if yes then we can send the answer back if not then we're going to have to parse the authorities check whether there's an IP address yes or no and then create new server and continue this loop to make sure that if I would write a mistake somewhere that we don't end up in a loop querying one of these root servers I'm just going to also limit the for loop to let's say three or five loops so if something goes wrong we write the wrong code then at least we are not hitting one of these servers with DNS packets as fast as we can so we are going to limit our for loop and that should be it so we just have to write what's in the for loop here and that's going to be that's going to be this code because the previous slide actually doesn't show everything here we have then whether our IP address is included so we're going to parse these additional resources where we can find potentially an IP address so it's a little bit more complex but should be doable it is surprisingly not a lot of lines of code because we can use this DNS message package we don't really have to worry about the DNS packet itself so let's get started let's clear our screen a little bit we'll not need this for now DNS query this should be a for loop you can leave the print for some time then afterwards you can remove it for i is zero as long as it's less than three let's increase it and then we are in our for loop and the first thing we have to do is we have to do a DNS query we have this outgoing DNS query function that we have written so we're going to do outgoing DNS query our servers our question and it's going to return the parser parser an error what is the parser it is actually the DNS anthraq we'll call it outgoing DNS query and there's also the header so the outgoing DNS query you have scrolled down here we do a skip all questions so now I can actually parse immediately the answers so we can say first let's handle this error DNS answer dot get all all answers these are all the answers these are the parsed answers we'll also have an error there's an error in the parsing and let's maybe start with the end of our loop which will not happen at this point but the most easy part is if the header says that it is authoritative then we can actually just send back a message the DNS message we have a header DNS message header here we put the ID so we only need to put that we now have a response then the answers what is the answer well we have the parsed answers here we can just say answers the parsed answers is an error no there's no error and actually my return error here is also wrong so here return nil error and here we return the error message so if our answer is that we get from the DNS server is authoritative then we can just return the message with answers and then we just make sure that the ID is correct here and that should work but this is only going to work at the end it's not going to work when we are just creating the root servers what are the root servers going to do the root servers are going to tell me if it is not authoritative it's going to have in this DNS packet the authoritative name servers so we should parse those and those would also be in this arser dns answer authorities all authorities and this returns a resource authorities error if there's an error we exit so this is of type resource this is of type resource what is the resource resource has a header and a body so we will need to parse this body as well resource header the name a type class and then a body so name is going to be the actual name so let's output what is actually in there or let's use debugging to see what was actually in there so we know how to parse this so these are authorities and it's a slice so we will have to for sure loop them and actually what we can also do is just output them so we can use debugging or just output it what is in there maybe i will just output it authority but what if there are no authorities so just in the case that there are none let's write a check so if there are no authorities then it will be zero if it is zero then we can send a message back server failure what else we have refused not implemented format error name error it's probably not really a name error we will need to have a look at the dns specification what exactly we would have to return here but let's keep this for now you can always look it up later let's keep this for now so that if our authorities would be empty for some reason because we wrote something wrong then at least we know that we see this name error here let's try to run this go run cmd main.go and let's use dig tries one of google.com and just to make sure we are going to query three times so let me just break here so we don't create three times unconditionally terminated just for now and then we return the server failure code so we don't query three times so we get the serve fail but what we get here is more interesting so what do we get new outgoing dns query for google.com these are the root servers and what is in this authority header name com name server ttl and then the body length 20 but that's here this length is because the in the dns packets it doesn't want to repeat always the same name so there's some optimization going on so let's have a look what is in these bodies i think we can do it like that and we have the must new name and then we have the names of these name servers so agtld servers or not and so on bcdef so this we want to save in a variable so we're gonna have name name servers make what is it going to be a string the size lan authorities i don't really like this go string here it doesn't really show exactly the name it shows how it looks in go so it's an ns resource so we just need to check exactly what the type is so if authority header what is in the header we have a type and the type is of dns message type if this is equal to type s then we have an ns type and then we can for sure know that the authority body is of type dns message ns resource and when we know it's the ns resource then we have ns and then we can do a string this will give us the ns string which will be this this we can add to the name servers so now that we have sized it correct we can do it like this we don't need to append because it starts from zero and then we already right sized it right here so now we have all the name servers could check again if that's exactly what we have name servers yeah the other name servers but we don't have IP addresses yet so we would have to now check whether we have these IP addresses and these are going to be in another place within the same dns packet as they're going to be in dns answer the additionals the additionals is going to give us these additionals an error if there's an error and these additionals we can loop again so just like with it here additionals additional it's a resource what is the type of this additional we don't know yet let's comment it out let's have a look what is in there so you can also just use debugging i'm just gonna print it what is in these additionals i can remove this do another query right it says handle the name and then we have type a this is ipv6 is ipv4 you're gonna keep it simple so let's use ipv4 and then the body is going to be the IP address so the additionals are going to have the IP addresses so if the type is what's the type here type a what can i do with this i can put this in a variable resolver servers which is going to be an IP address so this type a and i just need to use append now because i don't know how many they're going to be but i probably shouldn't shouldn't just add all these IP addresses because who knows what can be in there so let's compare them with the authority and only when the name matches with an authority then we're gonna add it authorities if authority authorities no name servers we can loop over the name servers because it's already a string so only when the additional header name i have it from here had a name string is equal to the name server then let's append to these resolver servers the IP address and how do we know the IP address so we know that this is type a already so we have the additional additional has a body the body only has this go string so here we use ns resource and here it will be the type dns message a resource a resource because the type is a you can also just print the go print output and it will also tell you that a resource and then it has an a record and then it can return the bytes so here we have the bytes and that we should be able to just add to the IP because IP is also bytes now that we have all these ifs we also want to make sure that we know whether we have found IP addresses we can say new resolver servers found it's false and here we can say true so that at least we found one and then if you didn't find any then we can do something but if we found IP addresses then we can actually do a query by iterating our loop again and then we just need to change the servers question can stay the same we can just change the servers now the service is the service from here which is the root servers now we can actually say the servers is the new resolver servers and maybe I should just have called these servers servers then I don't need this line because then we are just changing this servers servers we make it empty and then this needs to be servers and this needs to be servers so if we do the iteration then we say servers is going to be these new servers if you don't have any new servers then we're going to break for now and then we gonna end up here because we haven't implemented this yet if we have found new servers then we'll just go to the next iteration of so let's try to test that so let's click clear our screens maybe remove any prints yeah there's no prints anymore let's see if that works go run and what do we have starting DNS server the first question is for google.com google.com there's a new outgoing DNS query to the root servers the root servers returned these new NS records we assign this to the servers we do a new iteration the new outgoing query is still for google.com but now we have different servers and we keep on iterating on this until we have found an authoritative server and then once we find it we return the message and then actually we got the message there's no error we got google.com and the IP addresses so maybe it's a bit unclear so let's run the resolver test in debug mode so we can have a better look at how this works so the first outgoing query we have an answer so this is going to be the answer of the root servers and this is the message let's have a look at this answer it's not authoritative so we are getting all the authorities and these are these are these names so let's continue a bit so now you are figuring out what the names are are a gtld so we're gonna fill this name servers and then we're gonna do the same for the IP addresses so let's continue here to the next loop oh actually I see what I did wrong I should put my breaking point here so let's restart actually let's just maybe remove amazon.com just to make sure that I'm not accidentally getting in that one so we'll stop debug so how many questions do we have before we get the answer one two three and these are the servers root servers gtld servers and then the google name servers these are the google name servers and then at this last iteration we actually get the authoritative answer so that's the third one and then here we pass the answer and here we are authoritative so the answers are now coming from these servers which are the name servers so first the root servers gtld servers so the com top level main servers and then the google servers and the google servers are authoritative and then we are responding so this works but what is not going to work if we don't have any resolver server found if you say resolver server not found and as uncomment amazon.com and amazon currently doesn't have any IP addresses for their name server because why don't they have it they have another domain name you see so it's not the amazon.com domain name and when you ask the com.com servers they will only give you these host names so if you run our test again resolver server not found for amazon so we don't get a reply we get these host names right here but we don't get the IP addresses so we are here we get these additional there's nothing in them the new resolver found is still false so we are now here so for google.com we actually have it working but for amazon.com it doesn't work because we didn't complete this flow so remember the slides i have two flows one that we check whether we have the IP address and one that we don't check whether we have the IP address google is one server because we have here google.com and it's one google.com and then we cannot look that up so we have the IP addresses in the additionals for amazon.com we can actually do a look up of these host names to figure out what the a record is and then reply the a records and then use the a record as a server but that requires a little bit more code right here so let's try to add the necessary cultures right here you can actually write this code in two ways you can say i have these name server records here and i'm going to loop over them for name server range name servers and i'm going to say i'm going to find all the IP addresses of the name server this might be a little bit inefficient because we only need one working name server so if you have one IP address we might have enough now we might have an IP address it doesn't work so then we would be in trouble so if you have one IP address and just hope that IP address works and return that IP address back in our loop here as a server or we can get all IP addresses so it's up to you whether you want to write it efficient or not if you want to write it in a way that you get all IP addresses it is actually possible that if you would write a caching resolver that these IP addresses are cached anyways in your resolver so that the latency that you get by resolving all these name servers is not completely crazy but yeah you can already see one two three four five six are we going to look up six host names to figure out all these IP addresses whereas maybe one name server would be enough it's up to you i might just actually look up one if i find an a record of one i will just return that so what i'm going to say is i'm going to use again this if here if new resolver is not found then i'm going to continue in this loop and once i find one i will just put this to true and what i'm going to do now is that i actually need to restart this whole loop again again i need to run this dns query just like i run it here this dns query because if i just run one query that's not going to be enough i need to run multiple outgoing queries because we have to start again with the root servers the root server is going to say ns servers for dot net are going to be here then you're going to have the names of this dynamic dot net or this ultra dns dot net and then you're going to have the a record so i'm going to have to recursively go into this function again to find the IP address of this hostname here so dns query gives us a message respond an error and i need to get the root servers again because i need to query the root servers and then i can pass a question so dns message question the name is going to be the name server but actually the name is of type dns message message so there is a must new name that can convert a string into a name type so must new name name server the type is going to be the a type type a because i'm looking for an a record and the class is going to be the enet class if there's an error if there's an error i'm just going to print something a warning look up of name server failed and then the error and if there's no error then i'm going to do something else so that means that if there's an error we will just continue to loop until we find a name server that works if we find names that works and we have an a record we can put this to true the response will then have the answers and i will have to loop those because it's a slice so for answer range answers so this is a message actually so dns query it's a message we don't have a parser so we don't need to parse these things because it is actually the message that is being returned from hopefully here if we are authoritative after we went through the loop a couple of times we're gonna have these answers here so for answer what do we need from the answer this is going to be our IP address could be multiple IP addresses if answer header type is type a is type a i get a warning here i didn't add error if it is type a what do we do then we can say servers which is now empty append servers and our answer body should contain a type a so it's a resource just like here an a resource and this should be the IP address if it doesn't work we will see but normally we are testing dns message type a so this is for sure going to be an a resource we return ip we append these to servers and then we can do the loop again i don't think i am missing anything there's still this break here do we still need this break no we don't need this break anymore although before we are going to do all these crazy things maybe we should just say ip is found to see whether it's really working ip is found servers and then we do a break what about it i can live with that so let's try to do that so google.com that works because we didn't really go in that loop amazon.com ip is found okay new outgoing dns query for pdns1 ultra dns so we found i hope the correct ip address for ultra dns yes that seems to match let's remove this breaking point and let's hope for the best and we get an answer amazon.com and the a record is right here yeah that seems to match they just in a different order so this was our last step if there is no a record provided then we need to resolve it by running our dns query again and then after coming back here using the new servers we have an authoritative answer and then return the correct message so this seems to work let's to be sure also run our unit tests and that also seems to work so how many lines of code do we have less than 200 okay between also have the main function but our library less than 200 lines and we have written a dns resolver that is actually working for simple queries it definitely could use some improvements some more checks some more tests we can for example also test whether we can run mx records so host t mx google.com is handled by smtp google.com and now if we ask our own server using domain server the localhost one then we also get mail is handled by smtp google.com so that seems to be working you can do the same with dig you just have to do dig minus type i think type equals mx so within 200 lines you can write yourself a dns server in go using only this xr package that is still from golang.org so it's actually maintained as well it doesn't have the same promises as these built-in packages but these are pretty stable and pretty high quality because you can see that they are very efficient in parsing these dns packets so you can play a little bit with it to see if it works for all cases or that it works for a domain name that you own but for simple queries it seems to be working so our resolver can answer simple queries it doesn't catch anything yet so you see it does for every query a lot of dns queries itself so that means that it wouldn't be very useful to use this as a real resolver you would still have to write a caching layer that takes into account the ttl as well to cache the host names i will make these files available on my github so that you can have a look at the solutions and hopefully also try to build it yourself and improve on it