 In this section I'll be talking about identity providers. I'll mainly be talking about OpenID Connect or OIDC. Let's start with what is an identity provider. Simply put, an identity provider abbreviated IDP manages and maintains identity data for users. It's often used in conjunction with single sign-on SSO. It gives a user a single login and password and optional multi-factor authentication capability. It can be used for multiple applications and websites and while very convenient for the end user it's also more secure. So just think about this as being able to log in to multiple applications and websites with one single login and password and potentially multi-factor authentication. Two often used implementations for authentication within an identity provider setup are OIDC, so this OpenID Connect and SAML. OIDC, OpenID Connect and SAML, Security Assertation Markup Language are authentication mechanisms they don't store login password information themselves. So it's an authentication mechanism they don't store the actual login and password to validate your login and password when you're logging in. You would still need to validate a login and password and potentially the MFA token with a separate mechanism. Users can be in a database, in LDAP, in Microsoft Active Directory or other technologies where user credentials can be stored. SAML 2.0 was released in 2005 while OpenID Connect, OIDC was launched in 2015. You can see that OpenID Connect is much more recent than SAML. SAML is still used a lot but OpenID Connect is more lightweight and much easier to implement. Why implement OIDC? In this section I'm going to implement OIDC and you can follow along. The lectures will be built in a way that you can code some of the endpoints first before checking out the lectures. So why implement OIDC? It's a great learning experience, there's a flow to follow, there's JLOTs involved, there is API calls that need to be done using REST. So implementing this yourself will be a great learning experience to learn about authentication flows and to actually do the implementation yourself. You will get the exposure to a lot of technologies, amongst others it will be REST, OAuth, JLOT and JWK. You're often exposed to an identity provider and it's worth understanding the inner workings. So authentication is something that happens a lot, it's implemented in a lot of projects, so getting a good understanding of how authentication works, how OIDC works is definitely great to have. You can build your own identity provider authorization server, a client or application. So depending how you're going to do the implementation you might be the client, connecting to an application that is using OIDC or you might be the application that needs to implement OIDC or you might be the server itself. I'm going to implement all of them but in the end you might not run your own authorization server, you might use a third party for that but then you would still maybe need to implement the application or the client, so it's still definitely worth to understand the full flow and to have a look at how all these components work together. Understanding how the flow works will help you when you need to build one of these components. What is OIDC? OIDC is short for OpenID Connect. It's a simple identity layer on top of the OAuth 2.0 protocol. OIDC can verify the identity of a user using an authorization server. OIDC uses REST endpoints so it's easily implemented. So REST endpoints are just the normal endpoints that we have been using in this lecture already. Get imposed using JSON, OIDC uses this instead of for example XML that SAML uses so it's pretty easy to implement. OIDC uses JSON and JSON web tokens, JWT. JSON web tokens are used in a lot of applications. We have used it already before in one of our demos. OIDC also uses JLT so we are going to make use of those in the OIDC flow. There are different OIDC flows that you can implement. You have the authorization code flow which is the one we are going to implement in our demo. It's for web applications that can store a client secret. So if you have a backend application, a backend application can store a client secret without a client knowing because the code will be parsed in the backend. You don't see it on the front end so it can store a client secret to make the connection to the authorization server. This is the flow we are going to implement and then you have other flows that you can implement for example when you only have a front end. So there's an implicit flow for front end and mobile apps that cannot store a client secret so if you cannot store a client secret because you only have front end code then you could go for the implicit flow and then you also have a hybrid flow and the hybrid flow combines the authorization code flow and the implicit flow. It has immediate access to an ID token so where the authorization flow you will see there's multiple steps with the hybrid flow you don't need to have multiple steps. So let's talk a little bit about the flow itself this authorization code flow that we are going to implement. On the left you have the user which can be the web browser then we have the web application which can be a goal and backend and then we have the authorization server and the authorization server has an authorized endpoint and a token endpoint at least those endpoints to connect to to be able to authenticate our user. So what's going to happen our user is first going to connect to our web application. It's going to access protected resources or maybe there's a login button that needs to press to login to the web application. Then the web application will send a redirect to the user so the redirect is an HTTP redirect to the authorization server. So the web application will say I'm not responsible for login go to the authorization server so it's the user that then will go to the authorization server to handle the authentication part and this is done using this authorized endpoint. The user the browser will go to this authorized endpoint to ask for authorization to do an authorization code request it will ask for a code of this authorization server and this code can be requested using this authorized endpoint. Then the authorization server might send another redirect to a login prompt so this is the login and password box that you see on websites. This can come from the authorization server if you're not logged in yet that the authorization server is asking the user to login so then the user will send its login and password to the authorization server. If the login and password is correct then the authorization server will send again a redirect to the user and it will redirect the user back to the web application with a code query parameter. The first time that the user hit the authorization server was to the authorized endpoint for this code request now that the user is logged in it will actually get this code as a query parameter and this code it can send back to the application and because it needs to go back to application it's the authorization server that will redirect the user back to the application. These redirects are whitelisted so the actual redirect will come from the application but they are whitelisted on the authorization server so the user cannot be just redirected anywhere it needs to be a specific redirect to a specific URL. So then the user will follow the redirect and it will go to the web application send this code query parameter to the web application so that the web application can read this code and then the web application can exchange a code for a token so using the token endpoint the web application can go there to the token endpoint and ask for a token which will be a JLT token. So here you see that the user gets a code and not a token so why would the user not be able to just get to the token endpoint and get the token itself it's because of this secret that is being stored by the web application. So our web application has a secret that it will pass to this token endpoint and the authorization server will only give a token if you have this secret so the user doesn't have access to this token it only has access to a code and it cannot retrieve this token itself it is the web application and only the web application because it has a secret that can retrieve this token and then the token endpoint will send to the application the access NID tokens and potentially a refresh token to refresh those tokens when they expire. So that's the complete flow a logical overview of how to authenticate and this is the flow that we are going to implement in go. We're going to build this authorization server the web application and then we're going to use a browser to log into this web application which will redirect us to the authorization server we log in to the authorization server and then the web application will see that we are logged in and they'll be able to retrieve some of our user information because in this ID token there is actually user information like a login and it proves that we are logged in and that we then can access the web application. On this diagram we only have one web application but the idea of OIDC is that we can have multiple applications and only one authorization server that takes care of the authentication of our users, distributes the codes to the users and the tokens to the applications and then the applications have user information can make API calls using the access token and extract some user information from the authorization server for every user or make external API calls for this user. So it proves that the user is logged in but we can also make API calls because once we have this JWT another application could also verify whether this token was issued by the authorization server. The implementation details will become more clear once we start implementing this flow. So how does the flow work on an HTTP level here on the left we have the user here on the right we have the authorization server if we going to the authorize endpoint then we're going to do a request to the authorization endpoint the exact name of the authorization endpoint can differ it depends on the implementation of the authorization server here I call it slash authorization the user is going to supply a client ID the redirect URI and this redirect URI is then the application server so our application server here is running on port 8081 and we do redirect to the slash call back page so this slash call back page will then get a code basically we define a scope the scope is open ID we can add other things to the scope if it is supported by authorization server like a profile if you want to receive profile information or things like offline access if you want to have refresh token these things all need to be implemented we are just going to start with a simple implementation with scope open ID the response type is code so we expect the authorization server to give us a code and we also need to pass the state so the state is a random string that we supply that then the authorization server will then get back to us then the authorization service going to say you are not logged in so go to the slash login page and the slash login page will show a login and password box that the authorization server will then validate if we are successfully logged in then the authorization server will send us a redirect back to the application server so the application server is still running on 8081 and it will redirect us to this callback URL so this callback URL is typically wire listed in a configuration of the authorization server so that we cannot just pass any redirect array that we want it needs to be wire listed it will also specify the code and the random string so the code is the code that comes from the authorization server and that we send to the web application so here below we see that we do the redirect to the web application now the web application has access to this code and it can request from this slash token endpoint a JVT token based on the code that we are giving the application so the application is going to a post on slash token grant type is the authorization code it needs to specify the client ID and the client secret and only because it has the client secret it can receive this token it cannot receive any token just based on this client secret it also needs to supply this code that the user has been given to the web application because it entered the correct credentials on this authorization server it also passes the redirect to your eye that was specified just so the authorization server can validate it still the combination of this client secret and the code makes the authorization server give this JVT token so then the application will then get the JVT tokens that can then be used as proof that the user is logged in now that the web application has these JVT tokens it can still validate whether those tokens were issued by the authorization server there's a JWKS.JSON where the public keys are available so the web application can download these public keys from the authorization server and using those public keys it can validate whether the JVT was indeed issued by the authorization server those JVTs are encrypted by a private key of the authorization server so using the private public key algorithm when we have the public key we can validate whether this JVT was indeed signed by a private key and that is what the web application can do once we have the public key from the JWKS URL we can validate whether the signature of this JVT is indeed signed by the private key by using our public key so the last step when we receive the JVTs we can still validate them this validation can happen at any point as long as we have the public key available so other application service could also validate a JVT as long as we have the public key available the exact implementation details you will see in the demos and once you start doing the implementation I'm not going to go over all the details because they will become clear once we restart writing code this section is also a challenge you can write the code of these components yourself first so I will provide instructions first so that you can test it out yourself you can write the code as an exercise and then once you have written one function or one call you can validate whether you did it correctly using my demo lectures so if you would like you can write the ORDC implementation yourself first I will include instructions to step-by-step write the implementation the start project contains already the function signatures and I have written unit tests for those functions to be able to test the validity of your code so you can write one function by one function and I have unit tests written so you can just execute a unit test for a specific function to see whether you wrote the correct code the start project is located in github under ORDC start so this is the github URL the goal link for DevOps course so in there there is a directory called ORDC start and that's where you will find the function signatures and the tests the actual implementation has not been written yet in that code and the solution code is in the folder ORDC demo so I will write also in the next lectures the correct code starting from ORDC start and will end up eventually with the solution code that is located in the folder ORDC demo before every demo I will supply you with some information so that you can try to write the code yourself first in those functions and then I will do the implementation myself and explain all the details that you need to know to write the ORDC implementation to get started with the ORDC demo I have created this ORDC start that already contains the function signatures it doesn't really contain any logic just some function signatures and some boiler plate code in this lecture I want to go over the files and directories and the go files that we have in this ORDC start project and this is where you should get started so you can just download or copy this directory somewhere and get started yourself so the first file that I created here is the go mod file so the go mod file defines that this is a module is the ORDC start module uses go 1.18 and has two dependencies one dependency is the JLT dependency so I'm using from the golang JLT project the JLT dependency I'm not going to encode and decode JLT's myself I'm going to use a library for that you could actually do that yourself if you want to maybe you could start with the library and then if you want to get rid of the library you could try to encode and decode JLT's yourself it's not super difficult but it would definitely make this project a little bit longer so I opted to use this golang JLT it's used a lot this library so that's why I don't mind using this library I also used it in my other projects and our second dependency is the ssh demo and we just use that to create our RSA keys because we did it in the ssh demo you could also just copy out this code from the ssh demo if you don't want to use that as a dependency we're going to have an app server and the authorization server in the app server I have the main.go and every time we need to start a web server so what's going to happen here is we have the listen serve which is going to create a server and it's going to listen on port 8081 we're going to have an index and a callback because the application needs to be able to parse this code at a slash callback so that's part of the flow and at the index we'll just have a login button these are our function signatures two calls that we can make and I have a separate file here JLT.go and in JLT.go we have a function signature for the get token from code so once we have the code we can retrieve a token and the reason why I put this signature here this function signature here is so that I have a test so here we have test get token from code so when you run test it will actually try to run this test and because we just return nil nil nil it will say claims is nil but if you write the actual code in here the get token from code you can use this JLT test to validate whether your code is correct and then here in your callback you will need to use that get token from code so it's an app server so here you just have two endpoints the authorization server will be a little bit more complex we have also a main.go and in this main.go we have a little bit more boilerplate code to get our server started so what do we have here we're going to read a configuration file because we need definitely a configuration file to store values like the client ID the secret ID the callback that the callbacks that are wire listed so we definitely need to keep some config so what are we going to do here we're going to see does this config exist here we'll have to put a config file the name of the config file i'm going to use a yaml file but you can choose whatever you want you can also use a json or tumble file those are all valid and i'm going to read the file here and then we have in config just the bytes and what else are we going to do we're going to check whether we have this encryption key because we need an encryption key in the rsa format which is the same as our ssh format that we were using so we are testing does this encryption key exist if not we are going to generate a key and if the key didn't exist we generate it and we also write it out so that this so that the second time we execute it we are not generating a new key anymore we are just going to read this private key so we need a config and a private key in the rsa format well here it just binds because there will be in a pam format format but then eventually we'll have to convert it to the rsa private key format and then we'll start our server we'll start our server passing the htp server struct from the htp package just so that we can easily pass the port if you want to change the port you can change it in the main instead of going to dig into this package and we pass the private key and the config but for the config we are already going to pass this byte into a server config struct and this server config struct i already have defined so i already have some boilerplate just so that we can agree on the attribute in our variables so that when you write a code it's not going to be completely different than my code so this starts uses a server package and this server package is defined in here in pkg so if i have a look at this server package you see we'll have all the files most of these files are completely empty right now just with the function signatures in let's have a look at this config maybe first and then at the start server so this config where is this defined right click go to definition so here we have the read config which is currently empty so we will have to parse this config which is just bytes what is the file that we pass a file content and we have to return this config how does config look so in server we have types and in the server types we just have this config defined so right now we have an apps which is a map the key is a string and the value is the app config so we can have multiple applications our authorization server can support multiple applications and for every application going to keep a client ID secret issuer and the redirect uris we also going to find on a root level the url and we also have load error that we can use internally if there was a error while loading the config we can put it in here the url will then be just the url of the server so we don't have to define it for every application so this is the types and then in htp.go we'll have this start server so here we have this start that was called from the main so the start requires this htp server so we can pass this server address and to also help us with the testing the private key and the config then we do new server which is defined right here new server is a struct where we keep this information so that we can access it at all our endpoints new server we pass the private key and the config and then we can use the functions that we're going to define for every endpoint we have an authorization endpoint token endpoint login endpoint jwks.json so these are explained already in our view this as well so this is the authorization this is the first step this token is to exchange a code into a token the login will show our login page and we will also be able to do the actual login there jwks.json is going to show our public keys so we can validate our tokens this well-known open id configuration is just going to be a discovery endpoint so that we don't have to hard code all our URLs these discovery endpoints will then just show us a json with all the token and the authorization and the jwks URLs so we don't have to hard code it and the user info is an endpoint that we can call with a token it will then give us a json of the user info the user that you pass a token of and then we do the listen and serve so let's have a look at the tests we have an htp test here that's going to start a server and then it's going to test whether we have all these endpoints this test should already work yeah it passes so it starts a server and then it checks whether we have all these endpoints and then we are just checking whether the code falls in a certain range so if then something goes wrong then this test will then stop working and then you might have to make changes to your endpoints so we have the main goal the server and then we have all these files what are all these files in authorization we just have the authorization endpoint in config we read the config in discovery we have the discovery endpoint which is also just a signature so right now it just returns nothing we have the htp file which are even over jwks also just an empty endpoint the login the token the types and the user info they are all empty and we will have to one by one write a code for every endpoint to start working so let's say that we write the authorization code here then we can run the authorization tests you can run the test authorization code and now it will fail because we are expecting this location header but there is no location header set because because we haven't written any code but then once you write the code this test should pass so that's why we have all these tests so every underscore test has actually test files to help you to see whether the code that you have written actually works so we'll do this step by step we also are going to focus on the oidc flow itself we also will have a user package for the authentication so ones that a user enters a login and password then we're going to validate that login and password and that's why we have the user's package for but we are not going to do any complex authentication any complex login and password checking you could if you have a database or LDAP or active directory somewhere that you would like to use you can change this code and make a connection to your directory server here to validate a login and password so here you can change the login and password into your name for example so it's easy to login and this is only going to check our authentication and will then return a user so this is the user that has been statically configured in this package so you have the Edward user that i'll be using to login to the authorization server we'll also have this oidc package and in this oidc package we will have a few helper functions so for example here we have the parse discovery which takes url and then just returns the discovery utility so this discovery this discovery endpoint is basically just going to be returning a json in the structure of this so we have an issuer authorization endpoint token endpoint user info endpoint so this is standardized and you should have this endpoint which is going to make it a lot easier for your application servers to be configured because you will only have to pass this discovery endpoint so this parse discovery will just go to this url do a get request on this url and try to un-martial this json into a struct and return a struct so i left it here because it's just such a simple api call so you're not going to spend too much time on this it just goes to this discovery endpoint and gives us a discovery struct then we have this jdot.go but i actually moved this to here to the jdot.go so this one we don't need then we have a get random string in the rant.go which will just generate a random string based on the amount of characters we want to return here so we have n which is how many characters you want to return and we are going to randomly generate something randomly generate a string base 64 encode this with url encoding so this url encoding is important because we want to pass that as a craze string attribute so this is going to be base 64 just random so every time we need a random string we are going to call this function so no need to write this yourself it's pretty simple it just uses the crypto rant generates generates a random string we're going to base 64 encode it and whenever you have an equal sign we're just also going to remove it so now we just have a random string with no characters that could conflict in our url and then we have the types so i already discussed this discovery type but we also have a token type and a jwks type that is just an array of jwks keys so here we have our public key information so this is also standardized so this is standardized this is standardized this is our token endpoint what is being returned from our token endpoint and this is also standardized because this is all standardized i created this oidc package and these types are going to be all the same for everyone who is going to write an oidc implementation so that's why i already put them right here just so that we have some boilerplate code so i could also write my test because in my test i'm also loading these structs and it will make our life easier once we start writing the code for all our endpoints so oidc has these standardized types and just a few functions then the survey is going to be a server implementation and the users is going to be our users implementation that you might want to change if you would like to be able to log in with a real login and password so that's it you can download this oidc start package and then we can start working on the implementation of the first endpoints in this demo i will parse the config file so right now if we would start the server it would show us an error the config file doesn't exist to solve that we would first need to have a config file so we need to define this constant here config file because that's what we are checking on here and then we're going to read the config file and we are going to parse the config file so we have this server read config so this read config will have to write using codes that will parse the config file you can choose the format of the config file yourself i'll be using yaml so first i will need to define the config file constant i will call my config file config dot yaml and my config dot yaml is going to be right here config dot yaml in the main directory because that's where i'm going to start the server let's have a look at our types again so in types we have the config consists of apps URL load error and then within the apps we have a map with client that declined secret issuer and redirect URLs let's start with the URL and we are going to host it on localhost 8080 currently we can change that over time so right now it's localhost 8080 but we might change it over time because if you use a service like ngrok if you want to have it accessible on the internet or we deploy it somewhere we might change this and then the apps and the first app is going to be app one app one is our map key our map key and then we are going to define the app config client id i'll just say one two three four client secret it's going to be our secret issuer is going to be the same as our URL because that's the server that is going to issue the jdle t and then we have the redirect your eyes which is an array and localhost 8080 one is the app server so that's what i'm going to use as a redirect server name but then the path is going to be the callback so we are going to allow to redirect back to our application server on the callback URL i'm going to save this and we then need to parse this in this config config.go this config.go will do the config parsing using this config type but this config type doesn't have any metadata yet in the struct so let me add that first if we are going to use yaml we can add yaml metadata and that can then be read by a library so there is a library for yaml available the go yaml slash yaml yaml support for the go language the yaml package enables go programs to comfortably encode and decode yaml values it was developed within canonical as part of the juju project so we can use this one it's being based on the lib yaml c library but it's purely written in go there are more libraries available so it doesn't really matter exactly which one you are going to use this one is a good one you can use this one or another one to install it run this command go get the yaml v3 and now we'll be able to use it in the config.go let's now make sure that we put the correct metadata in place yaml apps and this then need to match exactly how we wrote our config.yaml so if you have a look at this config.yaml apps right here our lowercase and you see here we are using camel case first a lowercase and then uppercase if we start a new word so we just need to make sure that matches url load error we're not using client id client secret issuer redirect your eyes save this and then we can do the parsing works very similar as how we are parsing our json instead of json unmartial and marshal we do yaml.unmartial the binds and then the interface which in our case is the config variable this returns an error if the error is not equal to nil we are actually not going to stop the program we're going to say config.loaderror equals the error and then we are going to return the config just so that we can later capture the error and then do something with it rather than already returning an error so if we have a way to start without a config we might as well do that so we are just waiting to return an error here we're just going to return the config but we are encapsulating the error in our config variable save this and save this let's try to run our server and i think our server is now running localhost 8080 page not found so that seems to be working so our server started we don't even have an endpoint for slash so let's actually know what we get a for for i could try token okay and this token gives a 200 but no output let's maybe just see if we have our config output and then that's going to be end of this part of doing the config work and then we can start with these endpoints so this plus v is going to show us also the keys so we have the config apps is all applications is the map we have the key which is app one client id we have a secret issuer the redirect u rise the url and we have no load error so it's all great i'm going to remove this line again save this and we can continue with the next part of our server going back to our or dc slide that shows the flow with the htp request so this authorized endpoint on the session authorization is the first endpoint we need to write for the flow to start working it takes a client id the redirect u rise so the client id needs to match the client id that we put in our config file the redirect u rise also need to match the redirect u rise that we put in the config for that specific client id so these things we will need to check we will need to check whether the scope is definitely not empty we are only going to support open id but we can as well ignore that or just save it for later use and then check it later we are not really specifying in our config file what scopes we can receive open id is for rdc but you can also have an offline token and other scopes that gives you more functionality response type is code we need to check for that there's only valid response type that can be passed so it just says that we are going to then reply with a code and then we also need to check for the state the state is a random string and that state we later need to give back so you need to also save all this information somewhere on the authorization server we can validate the information right now save it somewhere and then we need to redirect the user so we need to send a redirect URL to the login page once it goes to the login page we also need to track the user because it will be entering its login and password information so when we do the redirect we want to give some random id like a session id with it so that we later on can track what user was actually logging in so we have a match between the client id and all the information that it supplies to us and the actual request so we kind of need to track the request as well when we are going to redirect this user to other endpoints so we don't lose track of the user itself so let's try to implement this so we are going to implement this in the authorization function let's declare all the variables that we need to have and then we can start checking on them and getting them from the request so we have the client id, redirect to your rise, code, response type and the state that we need to receive so let's store the correct information in all of them and just check whether they are all not empty if client id equals to and we have the request in r and then the URL in the URL we have the query and we can then do get get gets the first value associated with the given key the key is going to be client id so if client id equals to client id and even if the client id is empty we will not draw an error how we want to draw error we probably want to do return error and then write a function to return an error and then say what the error is client id is empty and then we want to return the stop our function so this return error we don't have yet so let's try to write this I will just write an HTTP just have a function here funk return error w is HTTP response writer and error this is not a pointer like if something goes wrong there is a header HTTP error code 400 that we could return back it's called bad request bad request is an int of 400 and we can say write byte and then we just return error and we also going to print on screen so we know that we drew an error that's easier for debugging just like that and then we can always use this return error when we want to say there's a bad request because something is missing and we just need to show the error to the end user so now we can do this for all these variables let's just copy paste this five times so now we copy paste this five times and we are checking whether client id is not empty redirect URL URI is not empty scope is not empty still need to change this and this looks all good except that the response type we can say if it's not equal to code because we're only accepting code here now we are sure that the client ID is not empty let me just check this one more time client ID client ID client ID redirect URI scope response type state all looks good now we have for sure the client ID it's not empty so let's try to find the client ID in our config and remember our config is in this server variable so we can loop our config apps for apps in as config apps for app if app client ID equals our client ID then we can say app config is an app config we can now assign this app to our app config if we didn't find anything then app config will be empty if app config client ID is empty we're gonna return error client ID not found so it's not empty but now it's not found I also need to pass our w and this needs to be rf and return the client ID is empty but we still need to validate the redirect URI the redirect URI is an array in YAML so we will need to iterate over that found is false I will say and then for redirect URI config range app config which is now our app config that matches with our client ID redirect URIs if the redirect URI config equals to the redirect URI that was supplied found is true so if you don't find this redirect URI that was supplied in the request in our config then we also need to we also need to throw an error return error redirect URI not found in config or not widely set what's that the next step I was talking about tracking our user with accession ID because we want to keep this information somewhere if you lose this information we don't have this information later on in the later stage so we'll need to store it somewhere and we can store it in our server struct so we'll need to make a new type that we then put in our server struct so we can keep these variables so I'll just copy paste those those are the ones we're going to initially keep and I will make a new type login request with those variables if I give them capitals and they are also accessible outside a package which might be useful later on maybe not in this demo but later on if you need access to this login request at some point outside this specific package I'm going back to authorization and here I need to store them in the server the server is declared in htp.go htp.go and then here I'm going to make a new variable how am I going to store this request information per user I'm going to create a new unique ID and I will call it a session ID and that's how we are going to track this information on a per user basis so login request is going to be a map and then this string key is going to be a unique session ID if I make a map I also need to use the make function to initialize the map make right here save this and then now I'm going to say session ID is going to be OIDC get random string and I'm going to make it pretty long so it's definitely unique if there's an error then return error and we're going to say get random string error and we return the error and then we also need to stop from this function and here as well we need to stop the function so once we have a session ID we can say login request within the server so this s comes from the server it's a pointer so changes that we make here are actually saved the next request as well so if later on we hit another endpoint this login request will contain the same information login request is a map session ID is going to be my key and then I'm going to have a new login request struct with client ID redirect URI scope response type although I probably never will need this just for completeness and the state this will be saved until the server stops so if the server stops it will be gone so if we stop a server then the sessions will be unknown to the server because we are not persisting this to disk but for our use case that is fine if we stop a server the sessions can be as well done if you would like to run this in the production environment and you want to keep the sessions you would need a session store like a ready server where you would store these sessions for example that if you have multiple authorization service service running and then you deploy a new version that you still keep track of the sessions so now we have our login requests what are we going to do we are going now to send the user a redirect with the session ID so that we can keep track of the session ID a redirect is done at the header level so if you do a header add location and we're going to redirect our user to slash login and that's where we will have our login page where the user can enter his login and password and to keep track of this user now that is on a different page we're going to pass this session ID and we can do fmt sprint f to make it nice and the session ID is going to be session ID and then as a header as the htp status header we are going to say htp found status found which is 302 so we found the location we found the page and we are redirecting the user to a different location gonna save this and like i said i will have tests available to see if our implementation is working correctly it doesn't test on everything and i will probably add tests over time so just have a look here and see how many tests there are right now there's one there might be more in the future you do run test and what will this one do it will build this endpoint based on app one and it's going to do a new request so it's using the htp test package doing a new request and then it checking whether we have the htp status found and whether we have location so redirect URI is empty location header not set so we might have done something wrong and yes we have done something wrong we are checking for redirect URI but we should be checking for redirect URI with an underscore all the other ones look correct run again and now we passed our test we got the location location slash login and this is a session ID so this test is enough for now if you wrote something wrong we will see it in one of the next lectures because then we are going to build this login page there might always be a mistake in your code that come up will come out later once you test the whole flow and that's why i'm probably want to add a little bit more test cases over time this one is already quite decent actually because you see we are passing all this information and actually this is wrong now that i'm looking at it we don't pass the client secret let me just correct this run the test again and the test still works because we are not using this client secret so there's no reason to pass in our test so client ID is a client ID redirect URI is the first redirect URI response type is code and a state is a random string so it looks all good we can continue with the slash login implementation in this demo i'll be handling the login flow so now that we went to the entrise endpoint we got a redirect to the login endpoint so we have initialized our flow and now the authorization server should get us a login and password page where the end user can then login click submit or login and then we are going to do a post request it makes most sense to do it on the same endpoint so if you are going to login we do a get request we get a login box we do a post request on the login endpoint with our login and password and our session ID and then the authorization server should check whether our login and password is correct if it is correct then the authorization server should redirect us to the redirect URI that has been specified when we went to the entrise endpoint so let's try that out let's see if you can put it in code this is the login.go the login endpoint and what are we going to do on this login endpoint first if it is a get request we are going to show a login page and i already have a template for that we are using this embedding so these lines here they make sure that the templates folder is actually embedded in the binary of go so that we don't have to supply it separately so the template directory is right here in the template directory we have a login.html so we're basically bundling this with the binary and to bundle this with the binary we can use the slash slash which is a command but it will be interpreted by the go compiler go colon embed the template directory and the template directory will now then be accessible using this template fs variable so here i put a comment to access the login template use the template fs.open template login.html which will then load this login.html file the contents of this file into the template file and then we can use this template file so let's try this out if there was an error with loading this file we can do a return error again and we can say template fs.open error error and then we can output a template file. What is a template file? fs file so we still need to read the contents of that file io read all can do this io read all expect an io reader and template file is of fs file which is also implementing an io reader so if i do io read all template file i will have all the template file bytes and i will get an error if something went wrong this needs another return and this i want to copy paste read all error and template file let's have a look at this login.html first so this is what we want to show the user it's just an hml file it's a login example you are free to changes into your own login form so it's a login form so we have the login input and the password input that if we then going to click this submit button here this login button then we'll do a post on slash login we also have room here for a session id so this is type of type hidden so that means that we will not see it on the screen but it's still being passed as a variable to the slash login endpoint when we are doing a post on it so this session id we can actually then replace with our session id because when we are getting this redirect the redirect is happening on slash login but we are also passing this session id this unique code we have generated so let's try to extract this session id replace this in this login.html and then just output that to the user and then users should be able to log in so we should have done that first probably how do we read this again? R URL R URL query get session id make sure that it is exactly the same session id is a string if session id is then empty let's return an error to the client so now you have a session id template file string equals we are going to do a place of the session id strings replace we can use for that strings replace and let's copy this just to make sure we don't make any typos strings replace session id into the session id oh and we need to also pass this template file binds so replace is first the string then the old string and the new string so the string where we want to do replacement needs to also be of string now it's on bind so I'm just going to put string around it so now it will convert from bytes to string then we replace this string and we look for the string session id and we replace it into session id how many times minus one is no limit so minus one this is a new variable so a colon and then I think we can just output it right template file string and I mean indeed we need to convert it to byte that's what Visual Studio did for me and did we miss something? I don't think so let's just test this to be sure go run and allow I will just use curl but you can also just use your browser localhost 8080 login session id is empty okay abc and now I get the html and the session id here is filled out so replace also works so now if I open this browser or if I go to a browser to the authorization endpoint then I get redirected then it will show this html I click submit and then there will be a post request happening so before we are going to test the flow in the application server let's just finish this login endpoint by splitting out get and post and if there is a post we need to try to log in then redirect to the redirect to your eye with the code and I think if you have that you can start working on the application server because we can then already test part of the flow if our method is post then we are going to do something else we are going to check whether use is authenticated and otherwise we are going to just return his html when we do a post request we can use the post form variable post form contains the parsed form data from patch post or put body parameters so we first get a login we press the submit button then we hit again the slash login with the login password and session id and then we need to be able to parse these three elements and to be able to parse these three elements we can use the parse form so they say this field is only available after parse form is called so we first need to do a parse form our parse form and it doesn't take any parameters but it can return error so if you say error equals parse form what if there's an error then we're going to return the error return error parse form error and then return out of the function and if that doesn't give any error then we should be able to use post form get and we should be able to use our session id so it makes sense to first retrieve our session id because before we go into test our login we want to make sure that we have information on this user how do we know what we have information on this user well if we have the login request session id set login request okay equals login request session id if okay which is a boolean if okay is set to true then we have this key in this map the login request map and then the login request will contain the login request if not okay then we can return an error login request login session session not found so now we have the login request that has been created at the authorization endpoint as login request session id so now we are just checking whether we can find this and now that we have this we have the redirect ui we have the scope we have the state and we can use that next we can do the authentication flow so we have this users out that we can use users out now users is imported and we can pause login passwords mfa if we have mfa but we don't really have mfa implement here so going to use an empty string and then it will be returning three variables whether we are authenticated the user and whether we have an error what do we pass our post form post form get login and this needs to match the input boxes so the input boxes from the html form were also login and password if we have an error authentication error now we can say if the authentication was successful we do something if the authentication was not successful we probably need to return an error we can redirect the user back to the login page with an error or we can just output something to keep it simple i will just output something w write header htp stages unartrised w write byte and then authentication failed the authentication was failed if we are authenticated though then we need to do some extra work we need to do another redirect so we're going to do another location we need to do another redirect to the request urign that is in the login request fmt sprint f login request redirect urign so we redirect to this redirect urign if the authentication is successful but we also need to supply the code and the state so we have a code that we still need to generate and the state is going to be login request state so we're just going to return back the state so that the app server could potentially check this the code is going to be a unique string again so we can just generate a unique string we just need to make sure that if we generate a new string that we can still find it somewhere on our server and also we need to do a w write header htp status font otherwise it's not going to work so let's generate a new code ydc get random string how long doesn't really matter 64 sounds about right there's also an error that comes back get random string error but how can we then later on find this user we'll have to create something new in our server struct otherwise we cannot find this we also need to save this user object this user object looks like this we want to save this this is some user information that we need to know in the server if the app server asks for some user information we need to be able to return that so why do we need to be able to keep the code it's because there will be another request on slash token and then the app server will supply us this code with the client secret and will then want to have a jlt token but we can only return this jlt token if we know what code we issued so we're going to store this code somewhere let's go back to htp and let's create another map and let's just call this codes we do initialize it again codes make map of string and we can reuse this login request so now that we have the code we can say s code we use code as a key and we say it equals to login request but now we also want to add some more information because we still have this user variable that we also want to store and we should also keep track of when those codes are issued so that we can have a timeout on them so let's go to the types and let's just add a few fields here code issued at time time and user users user so let's now add these extra fields login requests code issued at time now this should be code issued at code issued at login requests user is equal to the user and then we assign this to the s codes that is a map with all the codes in there with the login requests and then it has a code and it maps to a login request now that we have logged in a user we could as well delete the session id we have the s login request session id that we want to remove so how do we do that delete we can use for that so delete the map the map is the s login requests and the key that we want to delete is the session id so then we also remove the session id from the login request so that we cannot log in twice is there anything else that we need it looks good let's have a look at these tests we have a test login get which passed here we are also passing the client secret which is not necessary it's because i copy pasted this i will have to remove it everywhere because it first does authorization flow and then is going to do the login flow so we are doing this flow and then i will have to remove this client secret everywhere so here we are doing the authorization flow the login get flow and then the login post flow and we are expecting a location and a code so let's try again this test login get that works test login post and this also passed so we have the redirect ui the code that has been generated and then the state as state we are just passing a random string so that's fine if you change your login values in the users package then you might want to change them here as well so here we are doing the login get flow then the login post flow and then we are checking where you have the location so this all looks pretty good so i think we are now ready to start working on our application server we'll still need one more small thing before we can do that but then we can already test our authorization and login flow to see if you get a code back and then afterwards if you get a code then we can start working on the token exchange once we have a code the application server can actually go to the authorization server with the secret that we don't know here the client doesn't know it the browser is not gonna know it the application server is the only one that can go to the authorization server to get these tokens it's time now to start working on the app server but before we can do that we need to make the discovery endpoint and the reason why we need to discover endpoint is that we don't really want to pass to our application server all the endpoints that are possible like the address endpoint the token endpoint the endpoint where we will find all the public keys there are a lot of endpoints in oidc so there is actually a mechanism where you specify just one URL the discovery URL and that discovery URL will reply all the URLs in a JSON and then within the app once we want to use oidc once we need to know a URL like the authorization endpoint we can go to that endpoint do a get request get a JSON translate the JSON into a ghost struct and then we can use the individual URLs if you have a look at the types here is this discovery struct so this discovery struct is standardized so I already put it in place in the oidc package so the JSON is going to contain the issuer the authorization endpoint the token endpoint the user info endpoint so we just need to make sure that we generate that in the server the server has this endpoint defined it's also standardized dot well known slash open id configuration so if you're going to pass an oidc URL to an application that application will know that it needs to go to the well known open id configuration to get the discovery information we just need to make sure that the discovery function right here actually gives the correct information let's try to build this endpoint discovery is a new discovery struct from oidc discovery and then we just are going one by one define all these variables the issuer is going to be the sconfig URL the authorization endpoint sconfig URL and now it depends whether we added a slash at the end for the URL if you didn't add a slash here then you need to add a slash in the discovery you can still do a string replace on double slashes or actually parse every URL that's also possible authorization is our authorization endpoint token endpoint is slash token user info endpoint is user info j wks ui is going to be j wks dot json so we only did the authorization endpoint for now we still have to do all these other endpoints scope supported what are the scopes that we support currently oidc response time supported what are we going to respond a code token endpoint authentication method supported so let's say you go actually to the token what is the authentication method that is supported we don't have any authentication method supported now we need to reply json so json marshal json marshal this discovery if there is an error which is very unlikely but we still need to capture it call on here and then we can write the output go run main.go curl localhost 8080 i'll just copy paste it to be sure and then we have it the issuer authorization endpoint token endpoint and if i pass this through jq then i can actually have it much nicer authorization token user info jdliks looks all good scope support oidc so if we are going to support more we can add more to that as well offline tokens if you won't be able to refresh tokens but then we will have to add more code to support all these so for now we are just implementing one flow so oidc is actually good enough we're actually not going to do anything with this scope supported anyways our application server is only going to use this authorization endpoint and then maybe the user info endpoint is only going to use these endpoints here to use these scope support is just going to ignore so that looks all good and the next step is then going to be to build the application server so in the application server we can now pass this one URL our authorization server and our application can then retrieve the URLs that it needs okay let's get started with the application server so what do we need to do we are going to create a login button and the login button is then going to redirect to our authorization endpoint and that will then handle the login process what's going to be our redirect URL this one localhost 81 callback this will have to supply to the authorization endpoint how do we know the authorization endpoint well we'll first have to retrieve this well known open id configuration and parse this let's start with the oidc endpoint the oidc endpoint we are going to pass as an environment variable now we have the oidc endpoint and we have this helper function here in discovery.go in oidc discovery.go will do a get request and parse this into the discovery struct so very simple function let's use this instead of writing this ourselves it's just some very simple htp get function oidc parse discovery what do we need to pass the endpoint so we have the oidc endpoint and then we're going to add this well known open id configuration discovery equals discovery and error what if we have an error let's do the same as we did with the server let's have a return error function but if something goes wrong on the application server it's most of the time going to be an internal server error which is an error 500 that's probably a better error code to use on the application server if error is not equal to nil then return error parse discovery error and then return let's start to build this authorization URL authorization URL is alias sprintf first we have the URL this comes from discovery endpoint and then we need to supply some parameters we have to parse client id which will be supplied also as an environment variable client id will have the scope scope is open id we will have well let's have a look on our slide just to be sure client id we need to redirect uri the scope and the response type and state response type code state is the state and what else did we need the redirect uri so this redirect uri is going to be a constant localhost 8081 callback redirect uri the constant and then what else do we have just the state and the state is going to be a unique string again so we can just say state error equals oidc get random string how long 64 should be fine and then we need to capture the error and we can parse the state i think we have everything now we just need to return something to the browser how do we return something well just some html we're going to return let's try backticks yeah that looks better body and then we're just gonna have one simple button with a link and where are we going to redirect our user when they click the link to the authorization URL we have the button login i think this should work it's not very pretty let's maybe just make sure that it's wide enough so we can parse the width that should do it let's try to get this started what do we need to parse the client id client id should be 1234 client id client id 1234 and oidc URL oidc endpoint and that is localhost 8080 and then go run let's make this a bit bigger maybe go run go run cmd app server main.go and let's now just open our browser localhost 8081 here we have our login button and this is going to redirect us to localhost localhost 8080 authorization client and d1234 redirect URI there's a callback scope is open and the response type is code and then the state is this unique string and i made a typo i see localhost it's here in the config and then i will need to restart my server let's try this again i will need to refresh this page otherwise it will still drag me to the old URL now login and i get the form because now you're hitting slash login it's a get request we got the redirect so we basically first went to authorization then the authorization made the redirect to slash login and now we have the session id as well login and password so the login and password is going to be in the users package so that's why we do the authentication you can change the login there into your own name for example my login is adward and the password is just password login and then we have localhost 8081 callback with a code code equals a random string again from the server and we are hitting the application server with this code the next step would then be for our application server to then use its client ID client secret and code to get a token we get a white page because we are not doing anything yet in slash code so maybe let me just put something there hit refresh to see if we are hitting this slash callback endpoint so we wrote the index and here you have the callback callback code is our url query get code restart application server refresh this callback code is and this is our code this code is visible for the end user but this code doesn't mean anything because i could just change the code here in my url bar we still need to go to the authorization server with our client secret to see whether we will get a token and only the token the it token will be the proof that we are authenticated writing this slash token endpoint is going to be the next step so now that we have implemented the authorization endpoint the login page both get and post now we are able to do the redirect back to the callback the application server gets the code and now wants to exchange the code using this slash token endpoint with the slash token endpoint we can exchange our code into tokens we do a post request with grant type is authorization code we need to supply the client ID the client secret the redirect ui again and the code and then the token endpoint will give us a token if everything goes okay so let's try to write the necessary code that we need to put in this slash token endpoint to make it work so slash token goes to s token and then we have here our function signature that is still empty what is the first thing that we need to check is a post request if it's not a post request then return an error not a post request then we need to pass again these post entries and for that again we need this pass form so if there's an error with pass form then we can need to return an error let's not forget this return here in case something goes wrong and if pass form goes wrong pass from error and now we can use the our post form get and we have the code but what else do we have this grant type so this needs to be equal to authorization code if it is not equal to authorization code then we again can return an error we can say invalid grant type we can always return what the grant type was so that it's a bit easier for debugging so what is the code now the code is our post form get code we need to do a lookup of that code so we can say login request okay equals s code and then the key of this map is the code code I think it is code okay that's it if not okay we will say that we couldn't find the code invalid code if we have the code we still need to check whether the code is still valid login request had a code issued at if the time now is after I will save after code issued add and we will add 10 minutes if the time now so now we are a specific time if it's after the code that is issued at a specific time plus 10 minutes then we are also going to return that the code is expired so you have 10 minutes to complete this flow you can change these values in a bit less time or more time it doesn't really matter what else do we need to be checking we pass the client ID in the client secret we also need to check that is the client ID equal to post form get client ID if it's not equal we can say invalid client ID client ID mismatch and we do the same for login request client secret which we are not saving in the login request so let's change maybe a little bit our authorization flow and a login request let's add the app config app config is the app config the app config I extracted right here somewhere see we have here the app config so that way we have the config information and login request app config client secret needs to be equal to the post form client secret otherwise it's also not good and then we can say invalid client secret if it's not the same next we also need to validate whether the redirect ui is still the same if login request redirect ui is not equal to our post form get redirect ui then we are going to say that the redirect ui is invalid let's have a look on our slide again we do a post we check for post grant type authorization code we have that client ID we checked client secret we checked redirect ui we checked and the code we also checked because if it's not in the map we had an invalid code so now we can issue these jlotys how do we issue jlotys well we can use the jlotys library the colon jlotys jlotys let's have a look at the reference documentation and we have here examples parsing building and signing building and signing a token is what we need and here we have an example of jlt new with claims and the signing method this signing method is hmac which is a method that shares a secret we are not going to share a secret we are going to use a private and a public key so we will need another signing method we will use a signing method for rsa keys then we're going to have the jlt map claims these are key value pairs that will be in the jlt token that we supply these are quite random here foo and bar but we will use some standardized ones and then the next step will be to sign the token using our rsa private key rather than here this hmac sample secret first step is maybe to get the private key we have this s private key which is still in byte format it's in pump format but luckily this jlt has a private key function to convert this but we might have to import it first github.com golang jlt jlt version 4 i will need to go get maybe i already did it earlier but it doesn't hurt jlt and then we have some parse functions here parse rsa private key from pem parse rsa private key from pem and then we just need to input the byte so that's what we need and then we get a private key and an error if the error is not equal to nil then let's return our error private key parsing error error let's create a new jlt jlt new with claims is going to return as a token that's fine new jlt with claims the method jlt signing method so it's going to be rs for rsa and rsa 256 signing method rs 256 should be okay the other signing methods like 512 just uses another sha algorithm but we are going to use 256 for our purposes then the claims it should be of jlt claims let's maybe do that separately jlt map claims we should be able to use map claims is a map string interface and that should work with the jlt claims because claims is an interface so when we do a new with claims it will generate a new token there'll be a header claims and a method type will be jlt algorithm will be then our rsa algorithm and the claims will be here so these are our key value pairs and these key value pairs we can define right here what are the key value pairs that we have to define these claims are very standardized so we have the claims standard claims and these are the standard claims structured version of the jlt claims set as referenced at the here at rfc so there's an rfc where they explain what your key names would be they even say here they do not follow the specification exactly the use of this is this courage that's why we are not going to use it we are just going to use map claims and define our own claims which ones are the standard ones so we need the audience expires at issued at the issuer and definitely also the subject not before at we could use so let's start with a subject who is a subject we have our login request here login request user and user has a subject so it is something that typically comes from the user itself it needs to be like a unique login now we also have the issuer the issuer is going to come from the config config URL is the issuer the audience so we are actually going to issue two tokens an id token and an access token the id token is for the application server to know whether we authenticate it the access token is for the application server if it wants to make api calls so the audience for this id token is going to be the client id login request client id then we have the expiration which is in unix time when it's going to expire we can say one hour to keep it simple it depends on your use case how quickly you want it to expire if it expires a new one needs to be issued if we would work with refresh tokens then within one hour the token needs to be refreshed with a refresh token if you don't work with refresh token you have to go through the whole login flow again this is unix time not before should be time now and issued at is also time now so then we have this token but we still need to sign the token and I want to make one more change to this token so we have the claims but we also need something specifically set in the header for our jwks to work so the way this public key endpoint is going to work this jwks.json is that we need to specify a key id and because we only have one key we are just going to specify a static number 000001 is our key because we only have one private key and this is the key id so the key id stands for key id and this is something that we need to put in the header now we can sign this jwt with our private key token sign string so we are going to retrieve the sign string from this token and we need to get a private key for that this returns the signed token signed id token an error if there's something going wrong if something went wrong then we need to capture this signed string error error here how are we going to return this we have this oidc token for that so in types we have this token so this is access token token type reverse token x bars in an id token that we typically should return at the token endpoint so we're going to use this one oidc token so this is also standardized token output is oidc token and token has the access token the token type the refresh token x bars in an id token so we will actually need to create two tokens because we just created the id token which can be used to validate whether we are authenticated but we also need an access token let's start with id token is signed id token and then we going to just copy paste this and then this will be our access token signed access token and then access token is the signed access token but the access token needs to be looking a little bit different because it has different audience and the audience is actually a string slice of the endpoint it can access so what can it access it can access any API that we would give access to as in once we have a second application server and we would enter that in the audience here then then the second application server could actually parse the jlt have a look in the audience have a look whether it has access to a specific endpoint right now we are just going to give it access to our user info endpoint so we have our URL and with this token the application server can go to our user info endpoint and we will then return some user info so this is the difference between access token and our id token token type it's going to be bear it's going to be a bear token so then we can use authorization bearer and then the token the jlt token in htp access calls what else do we have refresh token so if you want to supply a refresh token for example when the scope is offline access then you just need to generate a random string for here and to make another endpoint that if this refresh token is supplied that you issue a new jlt currently not going to do that we are going to focus on this first flow that we are trying to implement we have like sparse in as well expires in is an integer how many minutes will it expire in so we have one hour token output then we still need to parse it token output json marshal our token output out an error and then we can check for errors token marshal error just in case that we have an error and we need to do one more thing so we are going to just write this first to the app server where it's connecting which we also need to make sure that we remove this code so we have this code s codes where we keep all the codes now that we issued a token we need to get rid of this so s codes and we have the code the code is our post form get code so we also need to remove this code from the map so then when the app server would execute it again then it will fail because we don't have this code anymore you just have to go through the whole flow again if you want to have a new token you cannot reuse your codes that looks okay i think could be that there's still a mistake how do we check for that well that's why i wrote these tests token tests what is this going to do first going to hit authorization endpoint and here i will also remove this secret because it's not necessary for our first endpoint i just could be pasted all this code multiple times so that's why this the same mistake is in all these tests so first hit authorization flow we get the code or the redirect we get the redirect we go to the login flow we get the code and we need to then exchange a code into a token and then we need the secret actually so this all and i didn't specify this last time this also uses this test config test config is defined right here in hp test and whenever we run our test we are going to do a setup of this test config which generates an rsa key so we don't need an rsa key just to run a test the test will actually generate its own key and we'll have here a map of some default configuration well let's hope it works no it didn't work and standard claims he should add needs to be of in64 he should add added time now needs to be unix and that's an in64 and this also needs to be unix so that's why we have these tests easy to then find whether your code is working or not still not working get on marshal string into ghost truck ah because we have two times we have it two times once for our id token once for our access token and this works got valid token from a token endpoint so i'm gonna leave it at that and then in the next lecture we can try to see whether our application server can now get this token with the code that was supplied at the callback endpoint in our application server now that we have written the slash token endpoint let's try to add this code to our web application to hit this token endpoint with our code and then retrieve our token that will be step one to retrieve the token step two will be to validate the token so i will first try to get a token and then we need to validate the token to validate the token we need the public key so we need another endpoint at the authorization server for that the jwks endpoint which will output the public keys we will need to parse that and then the public key we can use to validate our jwt that was returned or our jwts that were returned so let's first start with a post request on this slash token endpoint we have still the app server and then here in the callback we have the code this code we need to validate we have a jwt and jwt test so if you use this jwt signature get token from code then it will be easier for us to validate our code because we have a test in jld underscore test here we have a test to test to get token from code so let's try to use this code get token from code what do we need for that the token URL the jwks URL the redirect ui client id client secret so we can use this we'll just copy paste it for now this discovery endpoint token URL is discovery token endpoint jwks is discovery jwks ui redirect ui is static client id is client id as environment variable then we have the client secret which will also parse as environment variable and then we have the code which is coming as a query parameter what does this return a token claims an error and we can say token received claims subject i'm not 100% sure whether i will use these standard claims because they are discouraged maybe just for the simple get token from from code but really map claims is much more flexible so we probably should be using that token received if token or we also have the claims here if you want token claims maybe you shouldn't even return the claims on token dot let's keep it like this for now save this and let's start working on this get token from code we'll have to do an HTTP post request on the token URL and with a post we have a URL content type and a body what is the content type going to be it's a specific content type for form encoding and because this is actually a special type there is post form post form issues a post to specify the URL and then the content type header is set to application x www form your line court so then we don't have to set it manually token URL and then the data the URL values so URL values is of type URL values it's a map string and then a slice of strings we have some add functions that we can use what do we all need we need to pass the client ID client ID client secret redirect URI the code and this grant type authorization code then we have this post form responds an error if we have an error then turn nil nil error defer res body close and then we can read a body res body body error is IO read all because this is an IO reader if res status code is not 200 let's also return something you will then say status code was not 200 and i'll just output the body for now to the screen here to see something happened not sure what's going to be this maybe we should just return not yet fully implemented and then here in the main we can just say get token from code error as long as we didn't write the complete flow then we will get this error which makes sense i think so this get token from error there's one thing we didn't do and that is we didn't validate our state so we generated here our state we generated here our state in the index get random string but we didn't store it anywhere so let's try to store it somewhere we are not going to store anything in this state or match any user with this state so i think it's enough to just store the state and then see if we issued this random state string in the callback if not then we just reject the request if you are going to store something in the state or match a user or something else with it then i would recommend you to encrypt the state so that nobody can change or read it what's really inside and you can even pass it in a cookie with the client so that at the callback you could compare it with the client so i'm going to keep it simple and create a state map string boolean make map string boolean here for the state and when i create this state i'm going to add this the state i'll call it states states so that way i for sure know that the request originated from this index page and then here in the callback we are going to check if okay a states and then the state our URL cray get state if it's not okay then we'll just return an error then we'll just return error state mismatch error and then we can remove this state delete a states the state so that also means that if you're going to refresh this callback page it will not work anymore so we'll have to go back through the flow to get a new state should we try this out i cannot test my code yet because i'm just checking whether i'm getting a token we would still need to validate the token let's go and run the server client secret is the secret which is covered in the config.tml odc endpoint go run and then we also need to specify the jlot.go let's open the browser localhost 8081 login we can supply our login and password information get token from code not yet fully implemented but did we get the token output on our screen yes we did so here we have the body the access token and the id token will also be swear there refresh token expires in id token now we just need to validate this token so this is jlot you can actually input it in jlot.io which is a simple json web token tool and what do we see algorithm rs256 the key id the type and here we have the audience expiration date issued add issuer not valid before and the subject if you would paste our public key here then it would also be able to validate our signature so it can decode it but it cannot validate it because we still need to validate it to validate it we need to get the public key and to get the public key we need to make this endpoint the jwks endpoint and that's what we'll be doing next now that our web application received our tokens we still need to parse and validate those tokens and to validate those tokens we are going to do get request to the jwks endpoint and that endpoint will supply us the public keys the public keys that can be used to validate the jlot signatures we'll have to write the jlotks endpoint where we will have to return a json with the public keys we only have one so we will just have to output one public key in a standardized format and then our web application will get that json parse that and then see if we can validate the jlot with that public key that we received from the authorization server if everything is valid then the token is valid and we can be sure that the user is actually successfully logged in we can trust now this token and the contents within it it is now sure that the authorization server created that jlot and signed it with its private key so this is where we left the body is the token that has been returned from the slash token endpoint so now we need to validate this jlot so what i'm going to do is i'm first going to parse this body of our token is of type oidc token and we're going to json unmartial this body that is the json into this token if there is an error then we're going to return an unmartial error unmartial token error so we know where it comes from and now we can start validating the jlot we're going to return the claims and i was saying that we are going to use the standard claims the standard claims because that's our there's needs to be a pointer our standard claims because that's our function signature but then if you have a look at these standard claims it says that they only support integer based date fields and singular audiences so what do they mean with that you see here not before expired at expires at only supports integer based fields which we are only returning so that would actually work for us but singular audiences let's have another look at our token endpoint and here we can see we have this id token and this has a single audience but if you look at our access token here we have a slice so we have multiple audiences and this standard claims can only deal with a single audience so what do they say here in documentation use registered claims instead for a forward compatible way to access registered claims so yeah i can actually just changes into registered claims and we just need to change this function signature and it is forward compatible so our test will still work because here we also still have the subject the subject is still a string but if you need to use the audience it's not just a string anymore and if you have expired at it's also not an integer anymore so this should be a better way of using claims what is then the next step we need to parse these token we have two tokens but let's start with id token parse with claims and now our id token should be in the token variable because we unmarshalled it we're going to parse the claims so the parse with claims will have a look at the claims that are within the token and put it in this claims variable and then we have the key function and the key function is going to be a function where we have to validate this signature of this jlt and here we'll actually need this public key to be able to validate it how does key function look like jlt key function key function is a function token and interface if we just copy this jlt token and then we need to return save this and then we don't have a compile error but we are not validating it yet what does it return a token and an error I don't think we need the token we just need the error because if there's an error then our token is invalid return nil nil fmt error token parsing failed and the error why it has failed so if if the validation failed and it will throw an error so what do we return here still I think we actually need the token yeah the jlt token so I will call this parsed token parse token claims and then we don't need to return an error just nil and it should be it we then need to start working on this validation code but we didn't implement our jlks URL yet so we'll have to do that first we'll first have to go back to the authorization server implement this jwks URL and then we can write a get to get the public key in this lecture I will show you how to do the jwks parsing so it's standardized so we need to output a standardized json and we will have to do that for every key we only have one key we have the private key in our server struct so as private key is a byte slice it's a byte slice because it is just the contents of this file encryption key.pam and this is a pam formatted file so we'll have to first parse this this private key into an rsa private key object and I think we already did that earlier when we generated the token so let's have a look how we did that previously so here we have the private key and we use the jlt parse rsa private key from pam let's copy that so now we have the private key and the private key also has a public key and is the rsa public key so we can say public key equals the rsa private key public key now we need to output this jlks URL and because it's standardized I already have it in oidc declared so we have the jwks struct that contains the keys and for every key that is in this array you have an e algorithm use kid and kty key type it's just going to be key type so what does that mean what does that mean if you have a look at this public key go to definition public key has this n and e and an e here so basically this information is already known for us it's the modulus and the public exponent it's part of the rsa key it's properties of the rsa key so if we can output these in a readable format because this is just a big int we would need to convert this if we can convert this put this in a string make sure that we fill out the fields algorithm and so on then we actually already have what we need we just need to be able to output the public key in an agreed format the format that has been agreed on is to do it in base 64 so we'll have to base 64 convert it so j wks is equal to already see j wks and here we have the keys which is of type already see j wks key and here we have one element and this key we need to give an identifier so this identifier is going to be the kid and i'm going to use 0001 as kid because when i generated the jlt in token i specified this header 0001 as kid so this is the id of the key i used to encrypt it it can just be any string you just need to be an identifier what else do you have use algorithm key type algorithm r s 256 key type rsa use signature these three are just some informative fields and then now we need the actual public key information n and e re is always going to be the same so this exponent that we are going to use is always going to be the same we can just define it as a static string so this is already convert from the exponent so if you have a look at public key this exponent will always be like this so we don't really have to convert it maybe in the future if you have different exponents then you might need another one but for our purpose this will work for n we're going to use a public key n dot bytes and these bytes we are going to convert because you cannot just send the bytes into this json attribute we are going to convert them with base 64 so we're going to have base 64 base 64 standard encoding encoder string and that should be it now we can convert it json marshal these jwks out an error jlks marshal error if there's an error and if there's no error w write out let's do a go run of the server curl localhost 8080 jwks.json that looks pretty good with jq so we have the keys the end e the algorithm to use the k id and the key type so the next step is to go back to our application server do a get on this url parse this back into the jwks struct and then use this to validate our jwt now that we have the jwks endpoint available let's go back to our application server so how does this parse with claims work parse with claims will be executed and then this function will be invoked and we then need to return the public key here it accepts an interface so anything can be returned we should return an rsa public key and when we return an rsa public key then the public key will be then the token validation will happen with the public key so the signature will be compared this is all done in this jlt package we just need to supply this rsa public key to do that we need to first extract this k id so we have access to this token so this token is parsed but not validated yet so if you have a look at token header k id that will be the k id if you don't have the k id then we need to return an error k id not found if you have the k id then we should return the public key let's write a separate function for that so we have the public key that we need the error that it can be returned and we can say get public key from jlks jlks URL we need to supply and this k id we need to supply k id is of type interface but it's really a string so we need to still convert this to a string if you then have an error pledges then return the error if you don't have an error let's return this public key function signature get public key from jlks jlks URL is a string and the k id is a string what do we return an rsa public key and this will come as a pointer so i'm just going to return a pointer and the error so what do we need to do here to get this public key we need to do a get request on the jlks URL htp get on the jlks URL which returns a response and an error if there is an error then we'll return this error and otherwise we want to read the body so we're going to defer the rest body clause body error is equal to io read all rest body if there's an error then we return an error and now we can parse the jlks contents which is a json json unmartial data is in the body and we are going to parse it as jlks oidc jlks jlks again if we have an error then we return an error if the rest status code is not 200 we're also going to return an error because then the json will probably not be valid and we should put that probably in front of the unmartialing go right here parse jlks and now we need to do the opposite of what we did we first need to loop jlks entry key entry is jlks.keys so now we have the keys and we need to see if there is a key with the kid range here with the kid that is equal to mykid mykid here that's supplied if it is equal then we can extract if we didn't find a kid then we can say no public key fonts with kid kid so how do we extract this again with pay 64 standard encoding decode decode string jlks key entry and then the n what does this return n bytes error if error is not equals to 0 return then an error decode string error but the n is still in bytes and it is actually a big int remember when we were doing this jlks then we had a public key and the n is in a big int format so we still need to convert the bytes back into a big int n is big from the big package because this is also big big int so we're gonna do a big int new int new int accepts an int 64 and returns a pointer to a big integer we're just gonna say 0 because we actually want to do an n set bytes because we have bytes sets z to that value and then returns z but i don't think we need to return because it's just going to set n to the bytes and then we can return an rsa public key we don't need to pass an error because we don't have an error and n is n we don't really have to pass this e because this e is going to be defaulting to this so not really necessary here we still need to return error let's have a look it actually looks like it's finished now because we did the get request we passed the jlks we checked whether there was a kid we recorded this string create a new public key with the modulus so we get the public key right here return the public key so that the jlt parsed with claims can validate the id token if we have an error then token parsing failed otherwise we'll just return the rstoken and the claims as i look at the main function so if you don't get an error here then it is valid let's run the server and let's run the application server and actually just before i'm going to run the application server i'm going to run this test because i have a test to test whether these actually works and it failed verification error token parsing failed rsa verification error so we still might have something wrong i have this jlks endpoint here that is actually an hp test new server so i have this mock server that i created with a slash token endpoint and a slash jlks.json endpoint and here i just create a new key and we create a new token return that solely just a part of the server that is being mocked here and we get an error from the get token from code crypto rsa verification error and actually i had look offline for a second and it's very logical where i made a mistake because i said earlier in this lecture that we can just use e because it will default to the value that we need but that's actually not true because you see here e is zero and the value that we need if we have another look at our jlks it's this value and it's definitely not not zero it's not zero but it's actually constant still so if we just go back here add this constant here which you can find on the internet this exponent is a constant or you can probably also decode it from this but if it's a constant anyways we can also put the exponent as a constant and then let's run the test again and the test seemed to work so let's now run the app server and let's get back to our app server login and now we get a token received and also the validation should have happened so if we now go back to our main.go you see the claims now of type registered claims has this subject the sub claim and it didn't throw an error so now we can be sure that the user has been logged in the token is valid and we have the subject what is next let's have a look at our htp endpoint so we did the authorization endpoint the token the login the jwks the well known and the last one that we have here is the user info right here we have these claims from the registered claims but it doesn't really contain a lot of information and you also want to keep your jlt quite lightweight in general so what you can do is when you make the initial call which we do here we say scope open ID you can also add more scopes you can add a profile or you can add email to have these included into your jlt and then you can pass them in your claims that's one way or another way is that we have also an access token and this access token can and this access token can go back to the authorization endpoint and hit this user info this user info will then return us some more information about the user in json format in this lecture i will show you how to build a user info endpoint so this user info endpoint works a little bit different because now we actually have a token the user info endpoint expects that token in the authorization header and that is what you see here below the authorization error needs to contain the token and with a keyword bearer the id token is for the application to validate whether the user is logged in the access token is a token that we can use to make api calls so to make an api call to the user info endpoint we need the access token and the access token also has the audience claim that states the URL of the user info endpoint so we could actually also check on that to make sure that the application doesn't use the id token to get to the user info endpoint once the token is validated then the authorization server can reply some user info in a standardized json you can have a look at the oidc specification document if you want to see the specification of the json it's already something that we have in our users package so i'll be just returning the user here we have our user info endpoint what is the first thing we need to do we need to extract the token from the authorization header authorization header equals our header get authorization the authorization header has a prefix bearer so we can remove that strings replace bearer in this authorization header to nothing we can also check whether the authorization header was supplied so if the authorization header was empty then we return an error authorization header empty jwt parse with claims we can use again token string is going to be the authorization header the claims we can use again registered claims you can also use map claims map claims is just a map and then you can parse yourself let's see how far we can get with these registered claims authorization header claims expect a pointer and then again this function signature jlt keyfunk you see i cannot remember any signature i just need always to have a look in the documentation ended and returns a token and an error do we need the token probably not because we can continue with the claims so let's see what we can return return private key are we still to parse a private key private key equals jlt parse rsa private key from pem s private key and this can return an error if error is not equal to nil then return the error parse private key error error return private key public key nil no error that should work and then we get the error back from the parse with claims what if there's an error then we have an invalid token parse token error and return so if you come at this point we are sure that our token is valid but does our token have access to this endpoint so we can have a look where it has access for audience in claims audience but what is audience actually audience is a claim string claim string is basically just a slice of strings this type is necessary since art claim can either be a single string or an array so if there's a single string they will put it still in an array when they do the json marshal and that's when i do this they have this marshal json and unmartial json which are the custom functions that are being invoked when they marshal unmartial json so actually i just need the range keyword that should work and now i have an audience found is false if audience is equal to the config URL was the user info then found is true and if found is false then we didn't find this and then we can say return error token has incorrect audience and we can even output the audience just in case that we get an error that we can easily debug it so what's audience of this token we can join these claims audience together with a comma so now we know for sure that this token has been issued using our private key because we have this validate function with our public key we are sure that the token has access to this endpoint because we checked the audience now we need to return the user the user is the subject let's have a look at our users package we did the authentication we return the user these are subjects so we can compare our subjects from our token to the subject here and then return the user the user has the json sub name given name family name preferred username email and picture which is our standardized response that we need for users in for user in range users get all users if user sub equals to claims subject and let's also check if claims subject is not empty just to be sure if it's empty then return error if something goes wrong in our in our authorization server and we would issue a token with an empty subject we don't really start comparing this so subject is empty just as a safety measure if we find it what are we going to do json marshal this user and if there's an error then we are going to return error and otherwise we're going to write the output and if you didn't find the user then if you didn't find the user then we do a return error where we say user not found return we have a user info test that we can use to test our code that we just have written test user info and we get an error get status code 400 bar stock and error keys of invalid type let's have a look keys of invalid type and I guess this should be a pointer to the rsa public key because otherwise it probably doesn't check for both the rsa public key type and the pointer type and now it is working code user info json the subject name the given name family name preferred username email and pictures empty so this is the so this is the user info endpoint you can have a look at this test to see what is actually necessary to go to the whole flow so this is using the htp test package to do requests on the authorization to do the login flow and so on all the way until we hit this user info endpoint we are now going to do the same but in the application server in the next lecture so now you will need to hit this user info endpoint and see if we can display this to our end user so this is really going to be our last step for our oidc flow our first simple flow that we have in our oidc server our authorization server we're just going to hit this user info endpoint with our application server to test whether everything works and then we should be able to use this authorization server as an authorization server also for other applications that support the oidc flow let's have a look at this main so in this main we have this callback and then we have access to our token maybe let's return also the access token or let's just return access token and not the id token so let's copy this access token claims access token parsed access token access token also has the kid so we should be able to verify that and then allow us to return this parsed access token and together also the parsed token or let's not do that because we have to change too much i'm just going to only return the access token because you are not even using the identity token we were just verifying it parsed access token save access token so we so now we need to do a call and we need to change the request headers so our call will look a little bit different new requests it's a get request the url is going to be on discovery user info endpoint and then the body it's going to be nil it's going to be request if error is not equal to nil return an error new request error request header add authorization in the authorization we have the bear and then the access token access token raw is a string so this should be the access token now we have the request we still need an htp client htp client and we can do an action using a request and this gives us back the response just like we would do htp get and post do request error defer res body close and let's get the body read all res body read all error and i just print out the body as i send it token received user info and then the body i'm gonna make a fmt sprint f comma here res body like this we still have an error somewhere we are not using the claims save this run the server run the app server this is from previously state mismatch error so if we try to refresh it it doesn't work so our authorization server is doing this correctly we cannot use our previous state and our previous code to get a new token let's authenticate again and token received token validated we went to the user info endpoint and the user info endpoint gives us back this subject the name given name family name and so on so this is now the whole flow that we completed the authorization the callback we have created every endpoint that we need so if you want to plug in now another oidc compatible application that should work we might not have implemented the complete protocol so here and there you might have to add something for example if the implementation expects that it would receive some more claims if it adds the profile to the scope then you would also need to implement that this is a basic oidc server but it's a great starting point to extend it if you need any of those features because i'll default implementation is actually done at this point the application server is also very basic of course because we are not really saving this token or anything right here so what you could do is you could change application server to store this token somewhere locally so that per user you can then access their tokens you can then use the tokens for authentication validation is user authenticated or for authorization what else does exist in this token maybe the user is in some group that has then special access on the application server and also you have the access token that you can use to make API calls to the user info but maybe to another application server that is running so if you then need to make a call to another application server you can then use access token those tokens will expire so within an hour this token will not work so you will either need to authenticate again or implement the offline access that means that you have need to have code where you can then refresh the token so if you have a valid jlt you can then get a new jlt with another 60 minutes as long as you have a refresh token we haven't implemented that but that shouldn't be too difficult to implement at this point also some authorization servers they don't really look at the offline access scope they just give you automatically a refresh token it all depends you have the reference documentation of or etc everyone can do the implementation some implementations are not completely following the reference documentation but that is often okay because the standard also leaves some room to do your own implementation it kind of gives you a minimum that you should implement but then for example the user info endpoint could also return us a few more fields that is not really specified in this standard so that is it for our minimal authorization server if you would find mistakes in there or you want to do some improvements all prs on my github are welcome if they are adding functionality i might not immediately merge them with the solution that we have but i might for example create another directory in my github repository with a v2 where we can have some improvements that were out of scope for these lectures now that we have an oidc compatible authorization server we can start adding applications that support oidc there are a lot of applications and SaaS applications that support oidc often next to SAML there's also companies that can act as an oidc provider itself like google apple facebook those social media logins have oidc capabilities you could either use the authorization server and trust their token or write an integration to validate a successful social login and issue your own token with your own server often plugins are available to existing tools and software to implement oidc in the next lectures i will show you the oidc integration with jankins which is a popular ccd tool and adls i am federation with oidc in adls we'll make adls trust our id token to issue access keys to our users so within adls we can do federation and we can say that if a user can log into a specific app then that user can be granted adls access keys to be able to use the adls services using the adls api in this demo i want to show you that our authorization server actually works with real world applications i'm going to set up a janking server and i'm going to change the authentication part into open id connect oidc and i'm going to use our authorization server for that so jengas will be another application for our authorization server let's first start to run the server and i opened already the config here because we will need another redirect uri here so we could actually create another app if you want or we could just reuse this app the smartest thing would actually be to create a jankins application client id client id 1235 secret is secret the issue is the same and the callback will have to change but i don't know yet what the callback is we'll change it then we need to run a janking server and we can do that using docker so if you don't have docker installed it's best to install docker docker for windows or docker for mac and then you can use docker commands there is an image jankins slices jankins and it has the lts tag and this will launch a jankins image i'm just going to add a few flags i'm going to add a volume the volume is called jankins i'm going to mount it in var jankins a home and i'm going to expose a port 8090 and i'm going to map it to 8080 so 8080 on jankins is going to map to 8090 on my machine minus it so they can hit ctrl c when necessary and then maybe i can also add minus minus rm so if i quit my container then it will be automatically deleted it will not delete the volume so we can then still launch jankins later on you can also give the name name jankins if you want to find it easier in the process list and then actually i need just docker run because we are running docker run jankins jankins with all these flags and then we can start it so you have to wait a few seconds and it says jankins initial setup is required and admin user has been created and a password generated so please use the following password to proceed so now we can go to local host 8090 and then we should see this screen unlock jankins you can put this password you can still find it here if you didn't see it in the output and then we can install the suggested plugins or select plugins to install i don't really need any plugins i will just install plugins later on and i can create a user so let's do that jankins url and we can start so we have one user my adword user but now we are going to configure oidc so i first need to install the oidc plugin under available plugins open id there's two plugins we can use but i checked and this one has more installs so most likely this one is better install and then we can go back to the dashboard manage jankins configure global security so security realm jankins own user database or login with open id connect client id we then need to enter one two three five client secret secret configuration mode automatic local host 8080 well known open id configuration and this will actually not work because now we are in our container so we would have to reach our whole system which we cannot do but there's an easy solution for that and we'll also need it in the next demos you can install something called ngrok ngrok serve web commands with one command you can sign up for free but you can also just download it and you don't really need to sign up macOS brew install ngrok for windows you can download the zip file or do it with chocolatey chocolatey is a package manager to easily install packages or even with docker you could do that it looks like but i'm going to use the macOS version so i'm just going to open another window here i have ngrok already installed and i'm going to do ngrok htp 8080 and that will give me an endpoint this endpoint and this is a public endpoint that will map to my local host server that i should be able to use here and it's also htps so it's encrypted this works and now what i should actually do is to also change the configuration file so i'm going to put this in place and then stop the server start the server again and then in advanced you can also have some more advanced options so for example the full name field for me is going to be name so Jenkins will also hit our user info endpoint to get this extra information we also have an email field i think that was email i could have a look if we have a look at users we have here email i could enter it there as well so here we have name sub email so let me just add email here and you could also add a groups field name so if you add a groups field name that means that is going to expect an area of strings and then you can have groups and the benefit of having groups is that you can then download another plugin for authorization matrix authorization then you could specify groups and then you can match groups to specific privileges so now we logged in users can do anything but if you want more granular access you could add groups and then use the authorization plugin in Jenkins to give more granular access to those groups so i'm going to save this then i'm going to log out so here we are the adword user now i'm going to log out i'm going to go back to localhost 8090 you might get this warning just because we are doing a redirect then you just have to press this button redirect uri not whitelist it because i didn't know the exact uri so i'm just going to copy paste it from the URL and then in our Jenkins application i'm going to add this as a redirect uri refresh this oh i still need to restart our server refresh this and then we have our login form i'm going to login and then i'm logged in welcome to Jenkins who am i now here's my full name and i'm the Jenkins user id 9999 because that's my user id so if you have a look at the people here we now have the users this is the user from open id connect and this is the user that we added at the installation and then you click on my user configure then we can see the full name comes from the authorization server and also the email address comes from the authorization server so that was nicely filled out so we can even have more mappings to fill out more information if you want that's the benefit of having an authorization server it can distribute user information to your application servers so that's it for Jenkins there is always a possibility that you try out an application and that our authorization service not immediately working with it then you just need to have a look at the flow that it is using you have to make sure that all the oidc capabilities that the application needs are also implemented because we might not have implemented every single step or every single capability when preparing the alos oidc lecture where we will get alos tokens based on our token that we get from our application i realized that our discovery endpoint is missing a few attributes that we should return otherwise this is not going to work because alos im identity and exit management is actually checking for a few more attributes that we don't have so i have those added in the types it's id token signing algorithm value supported claim supported and subject type supported so we need to add those otherwise the next demo is not going to work so we're going to have id token signing algorithm supported which is rs256 then claim supported also a string slice and these are the claims that we are returning so we have sub and we can just have a look at our token endpoint we have issuer sub audience expiration issued at and not before and then lastly we also have subject types supported and there's a mechanism to hide some privacy sensitive information so the only thing we need to do is say that we only support public we just return the subject the other thing we need is when we are going to use the app server we are only replying one token and we are going to need both so i'm just gonna make an array of this and then here we have the parse access token right here we have the parse id token and i'm just going to jwt token going to add both of them first id token and then the access token parse id token and i need another column here save this go to the main so here we have both tokens and tokens the second one is the access token and i'm just going to print on the console the id token because the id token is the one we need the next demo tokens raw so that's it we should be able to run the server again and also maybe one last thing to mention from this discovery endpoint if you want to know more about this discovery endpoint you can have a look online and just type in open id configuration endpoint and you will find some information for example i found this held up wiki and this held up wiki explains every single attribute that you can have so here is an example and here at the bottom you actually have what attributes are required so the issue is required the authorization endpoint is required which one is optional and which one is recommended so we have the username for endpoint which is recommended but not required but then we had some other required missing which didn't work very nicely with alias so this information is actually very useful when creating this open id configuration endpoint in this demo i will show you how we can use our oidc authorization server to get alias access tokens we can configure within alias im identity and access management our oidc server that way every user logged in into our authorization server could potentially get alias credentials from alias im let's have a look how this would work first we need to again set up our ngrok tunnel so i'm just going to run the server first and then open another window this is the same as i did in the previous lecture if you didn't follow the Jenkins lecture you just have to download ngrok from the ngrok website or with brew any type ngrok hp8080 and that will tunnel port 8080 from your local host to a public available hostname this is our hostname our server is running we just have to make sure that we update our config file as every time you use ngrok this hostname will change we then need to restart our server and then we should be able to use this hostname in alias so this is my alias account it's a sandbox account it's empty i just created a new one and we need to go first to iam manage access to alias resources and here we have zero users so we only have our root user and we are not going to create a user we are going to use our identity provider so we click on identity providers and here we can add a new provider use an identity provider idp to manage your user identities outside of alias but grant the user identity's permissions to use alias resources in your account so you first add a provider and here you can choose between saml and open id connect so in the first lecture i was explaining saml and oidc where saml is the older standard and open id is a newer one and you can see that indeed saml is still used in a lot of places in the past even only saml was used in alias and then only later they supported open id because also when alias came out the open id standard didn't exist yet we are going to choose for open id connect and then we can provide this URL i'm going to provide my ngrok URL and then you can click on get thumbprint this thumbprint is from the ca certificate so the ca validity is until 2024 so ngrok is configured with htps there's a certificate in use the certificate i actually checked is from lads encrypt it just to make sure that alias validates our certificate it's not going to do it on the server certificate because a server certificate typically is only valid for a couple of months or for a year so it does it on the ca and the ca is typically valid for a couple of more years so we are now 2022 so it's valid for two more years then we can configure the audience and audience in our id token is the client id so that's why we have to also supply the client id to be able to log into alias we cannot use the access id it's going to check the audience and the audience needs to be equal to the client id so i'm going to use a client id from our first application server 1234 and then i'm going to add this provider so the provider is now created and now we can link a role to this provider so we can click here and assign a role to it i'm going to create a new role and you could say that whoever has access to our application server our first application server has access to the complete alias account so admin permissions or you could say whoever has access to this account has a specific subset of permissions i'm going to add s3 permissions just as we are working with some kind of development server we just created the server so if something goes wrong then at least we don't have admin privileges so this is our identity provider audience is 1234 and alias calls is the audience because the audience is equal to the client id but we always call this a client id and not really the audience so in id token audience is equal to the client id and that's why they call it audience here permissions so here we could give administrator access or i'm just going to do s3 full access text if you want to create text review oidc demo s3 role i'm going to call it create a role and now we can assume this role using oidc using our oidc token i need this errand because that's what i want to assume and i will have to use the alias command line utility so if you don't have the installed yet you can install it from the alias website or again with brew or with chocolate tea so that you have the alias command available then we will also need the client token so we need to log in to our application and we have to make sure the client id matches so the client that we entered matches with our app one so i'm going to log into app one and that is going to give me the id token so here we have app one i'm just going to start our application server so this oidc endpoint can still be local host it was just going to verify this token it doesn't need to have access to our application server and now what's going to happen here i'm going to see the id token in the output because i added this fmt printf here id token token zero zero should be the parsed id token okay let's try that out local host 8081 login my login and password token received and let's just make sure that i copy again this arn and then we can try to log into alias and get the tokens let me maybe also create an s3 bucket so that we can do an s3ls to list the buckets and when that works i'll be able to see whether whether the tokens actually work my test bucket i'll just add some random string here because we know it might already be in use because it needs to be unique within this alias region so just have to make sure that it's unique enough because my test bucket will probably already exist so i have a bucket so i'm going to try to assume this role here and here you can also see that there is this trust relationship where we have this audience needs to be equal to one two three four so only the tokens from this app server will work on our orc provider that is what alias is going to check is going to check this token see if the signature matches with the public key and if that matches it will accept the audience the audience is equal to our client id if this audience is one two three four then i'm granted access here is it token so let me just open another window so our role role erran is this one don't want to lose it and our token is this one so i'm just going to say token equals and then here i can do echo token echo role erran if you're if you're on windows you can use set i think or you can just copy paste it in one string that's also possible because it's just easier for me to work but you could also just enter everything in one string so you need to have the alias account but you don't need to be logged in alias sgls and able to locate credentials so i have no credentials and now i'm going to try to obtain credentials and i'm going to do this with alias sts assume role with web identity so i think if i just add help to it then i will get the help function and if you scroll a little bit lower it will show us the arguments the role erran check we have it role session name is just an identifier so could be our user for example it's just a value it's not being checked on and then the web identity token so this is what we need to be able to get a token so i'm going to do alias sts assume role with web identity role erran is our role erran web identity token is our token and role session name is just going to be my name at the word that should be it really let's try it out if it works or before we try it out let's just verify our token jdt.io i'm just going to paste this token here so that if you have troubles yourself that you can have a look at my token so i have the algorithm kid and the audience is important one two three four this is the client id because this is our id token the kid is going to be used by alias to get this public key from our jwks endpoint and to validate our signature right here so once the signature is validated it can be sure that our authorization server issued this jdt so let's try and we get the credential now you can actually see using ngrok the calls that happened so these calls just happened here it went to the open id configuration and it also went to the jwks json to download the public key and this is when i logged in i did a login and then here are some other requests that either me or alias did so let's go back so we now have these access keys and these access keys we could configure in our dot alias slash credentials or we can just configure them as environment variables and see if we can do s3ls so there's also an expiration so what time is it now here it's 1108 and this is udc so this must be valid for an hour let's try to configure them so if i do alias access key id equals to this secret alias secret access key equals to this part and then the alias session token equals to this part and then i don't really need to blur any of this because this will be expiring in an hour anyways alias s3ls and i get my test bucket so you can see that with our id token we can get alias credentials that are valid temporarily assume this role right here and then use it to access alias and we didn't even have to create a user on alias this is also called federation in alias so if you would like we could potentially write an application a go application that lets you do the authentication with a user a password maybe even with mfa and then does this sts assume role with web identity api call to get yourself credentials admin credentials normal credentials so that you can get access to the alias api and this is actually the same mechanism that alias uses for pods in kubernetes so if you use the kubernetes service on alias then also every pod will get a jail d and then there is also an environment variable that is exposed to the path where this token is and the alias sdk will automatically read this token and then we'll get automatically the credentials let me just have a look what the name is of this environment variable if you're interested in it so here we have the configuring the itlas sdk for go v2 and within kubernetes or if you would be writing something where you would also use jail d's then if you use the load default config there's always a path that it follows so it actually says here when you initialize an alias config instance using config load default config there's a default credential chain environment variables the one we used right now so if you configure those then it will automatically find your credentials or also the web identity token so if you have configured the alias web identity token file and your token is in that file then you can also just configure this environment variable and then the alias sdk will automatically go to the file read the token and get those variables if those two are not found then it will also have look in the alias folder in credentials and also in the alias config so there's a certain path that it follows so if you want to make sure that your credentials are properly configured just have a look at this documentation and make sure that you either supply one of these as path for alias to find your credentials so this is it if you have any problems with or dc and alias have a look whether your discovery endpoint is okay have a look whether your jdleqs endpoint is there whether the flow works with the with the normal application whether your id token is valid and then it should work if it still doesn't work for you feel free to reach out to me directly and i can always help you in this section we are going to talk about tls tls stands for transport layer security and is used for data encryption web encryption typically uses tls to encrypt communication between client and server tls is a successor of ssl secure socket layer the default port for unencrypted htp traffic is 80 and a default port for encrypted htps traffic is 443 tls itself is not an encryption algorithm but a protocol to negotiate and agree on a common set of encryption and hashing algorithms called the cipher suite with tls enabled the htp server offers the client an x509 certificate which can be validated by the client to ensure the server can be trusted the hostname of the server will be included in the server certificate the server certificate will be signed by a certificate authority abbreviated ca if the client can validate the server certificate we can trust the server and this is what exactly happens when we go to a secured website using htps in the browser to be able to validate the certificates we'll need to always have the certificates of the certificate authorities that can sign the certificates they're also called the root certificates browsers typically have this list built in and we didn't go it'll also look for those files in hard-coded system paths to be able to validate certificates so these certificates are provided by the operating system so on the linux system they will be different than a wino system or a macOS system and with macOS or linux for example it will be the operating system itself keeping them up to date so go we'll just look into a few directories to see if it can find those root certificates once it finds one directory with the root certificates in it it will stop looking and it will use that directory so with the browser this is all pretty straightforward you're using this every day where you go to an encrypted website the validation happens because your browser has those root certificates and you can automatically trust the server if a certificate will be expired or the host name would not match or the signature cannot be validated you would typically get an error which sometimes happens on websites that don't maintain their certificates what i just talked about this is our client server communication where the htpserver offers the client a certificate that can be validated this is called one-way TLS you can also set up two-way TLS or mutable TLS abbreviated mTLS in this scenario communication can only be established when the client also has an x509 certificate so this you will typically not see with your browser because then your browser would need a certificate to be able to access a website this two-way TLS is used often in server to server communication for example to secure communication between microservices in the following lectures i'm going to add TLS support to a simple go hp server there are multiple strategies to implement TLS using a self sign certificate we will issue the certificate authority certificate ourselves so only someone who has a specific ca certificate will be able to validate our server certificate using a real certificate issued by a company that can sign with a root certificate so companies like dgcert geotrust rsa global sign they own these root certificates and they can issue a certificate for you that is signed by one of their root certificates and those will then be valid if you would use a browser and visit your website with that certificate installed because your browser will have these root certificates built in and is able to validate the signature of this server certificate and also using a let's encrypt a non-profit certificate authority so this is kind of a third option because the second option the certificate issued by a company you most likely will have to pay this company because they need to verify that you are the real owner of this hostname of this domain name and with let's encrypt this is actually free it's a non-profit so you can get a free tls certificate with let's encrypt and they will have an automated procedure to get you verified in the next lecture i'll explain these strategies in a little bit more detail and then we will start implementing a self sign certificate all these approaches can be used for one-way tls for two-way tls a self sign c a is common but other approaches would also work so for example within kubernetes you have a lot of two-way tls because you have a lot of services talking to each other and you need to be able to trust each other within microservices you have two-way tls with vpns you have two-way tls one-way tls is what you see every day in your browser and two-way tls is probably something you have less exposure to when you have two different services talking to each other so every approach can be used for two-way tls as well although self sign c a with two-way tls is a bit more common than using real certificates although it is still possible to do two-way tls technically with a real certificates or even with let's encrypt let's go over the tls strategies we will look at the self sign strategy signed by a root c a and let's encrypt first the self signed c a this will also be the next goal demo that i will show you it's using a self signed c a in that case we're going to have a server a client and only the server is going to have certificates not declined because it is one-way tls and this server certificates need to be signed by a c a a certificate authority so that the client can validate whether the certificate is real because it is self signed we are also going to be the certificate authority so we're gonna have to create a certificate authority certificate and a certificate authority key with this key we can actually sign one or more certificates so here we only have one certificate but we could be signing a lot of certificates we only need one c a to be able to sign multiple certificates once we have this setup we can use a certificate authority key to sign the server certificate the client can then connect to the server using tls the server certificate will be offered the server key will not be offered because the server key is secret to the server and the client can then also download the certificate authority certificate to validate the server certificate because this is not a root certificate the client is not going to have this bundled in the browser or bundled in the operating system so we'll have to provide this c a certificate file to the client to be able to validate the server certificate if we maintain the client for example if the client also go program or it is curl or it is our own browser then this setup would work because we can have this c a certificate installed where it wouldn't work is if you would have a production server a public production server and we would offer any client that will connect to our server any browser the certificate where they don't have the c a then they would get an error because they don't have the c a certificate and that's why all the servers that we connect to in a real world scenario are actually using one of the root certificates so roots signed c a is next where we still have the server the client server certificate and the server key but the client in this case has the root certificates available to be able to check the server certificate it just needs to be signed now by a certificate authority company this certificate authority company we have the certificates of this certificate authority company but not the key so we cannot sign any server certificate ourselves we don't have access to the c a key or else everyone could sign any certificate and this whole validation process would just not work we all agree to trust these certificate authority companies to only sign server certificates after they validated the host name in this server certificate so they will go and send an email for example to the email attached to the domain name and ask are you the real owner can you verify that you are the real owner and only then they will sign your server certificate so how does it work well you need to first give an unsigned server certificate to this certificate authority company this unsigned server certificate is in the form of a certificate signing request or abbreviated csr so you basically give your certificate that you generate it using your server key and the certificate authority company is going to sign this for you and give it back to you so now you have a signed server certificate and then the client can connect using tls to the server will receive the service certificate and will be able to validate this server certificate because it has the root certificates with all the certificate authority companies in there to be able to validate this server certificate so you can imagine if you have a very old operating system or very old browser that is 10 years old and hasn't been updated then actually those certificate authority certificates or root certificates they expire and when you then visit a server you would get an error so if you for example install a very old browser or operating system that is 10 years old you will see that those root certificates are not valid anymore and you will get an error when you just browse to some common websites and then we have let's encrypt and let's encrypt is very similar to the root certificates because let's encrypt is also a company that is included in the root certificates the difference with let's encrypt is that it's an organization it's a nonprofit and they don't ask money for a tl certificate you still have the server and the client the server certificate and the server key the root certificates that has let's encrypt in it and let's encrypt itself the company we still don't have access to the ca key because that is managed by let's encrypt what let's encrypt is going to do is it needs to be able to validate whether we really own this domain name let's encrypt is validating our website using this automatic certificate management environment let's encrypt will connect to the server that needs to be publicly available and we'll go to a specific path that you can see here it's dot well known slash ace me challenge slash token and this token is something that let's encrypt will give you if you start the procedure it will see whether you really own this domain by putting something unique on your domain name that can be verified by let's encrypt so let's encrypt will give you random string you put that random string on your server if you are able to do that then let's encrypt trust that you own this server that you own this domain name and then it will then sign your server certificate here again you need to give an unsigned server certificate a csr certificate signing request to let's encrypt so that you can sign your certificate based on this certificate signing request you then have a service certificate that is signed by let's encrypt and the procedure now is the same the client connects with tls the service certificate that is signed is offered and the client can verify it using the root certificates that includes also let's encrypt let's encrypt is actually the newer strategy compared to the first two because the first two are already around for a very long time and let's encrypt is a new approach that came into existence so you don't have to pay these companies always for your tls certificates let's encrypt is very popular so there's a lot of web science that use let's encrypt so they don't have the cost of paying for every tls certificate it might be just a bit harder to get a certificate because you have to use this automated process with the other companies that sign your tls certificates there are more options to do the validation process but if you can use this automated validation process it will end save you some money and it will also save you time to renew the certificates because you can automate your renewal as well one strategy that i didn't cover yet is two-way tls so the mutual tls so with mutual tls the client also has a certificate so how does that look then we have a service certificate and server key and the client certificate and the client key we have a certificate authority certificate and a certificate authority key so in this case we have self-signed certificates because we will also be the certificate authority with the certificate authority key we can sign both the client certificate and the service certificate and then we can distribute the certificate authority certificate to validate both the client certificate and a server certificate. So the client will be able to use the CA certificate to validate the server certificate and the server will be able to use a CA certificate to validate the client certificate. So the client makes a TLS connection, the server will be able to validate whether the client has a certificate and whether the certificate is valid and the client can do then the same thing. So, the benefit of MutalTLS here is that you can have a two-way trust. In one-way TLS, the server doesn't know anything about the client. In two-way TLS, the client is also identified by client certificate that is also signed by the certificate authority, so the server also knows that the client has obtained a certificate that has been signed by the same certificate authority. Like I said, you also have variations on this where you have two certificate authorities or you can also use a root certificate. In the next lecture, I will show you how to do a one-way TLS setup with an HTTPS server in Go, so it will be a very simple server using the self-signed strategy. This is the TLS start project from my Github repository and this TLS start project I'm going to use to show you how to create a command line utility. A command line utility to create a certificate, a CA certificate first, with the key and a server certificate because in this TLS start we have a test server and this test server is going to launch an HTTPS server, so if we have a look here, so we can run this by Go, run, cmd, test server, main.go, we have the main function and we are only going to handle slash the index page and instead of listen and serve we are going to use listen and serve TLS. TLS needs the address which is going to be 443 or 8443 if your port 443 is blocked by maybe not a program already, we need a cert file and a key file and it's going to be the server certificate and the server key. So if we are going to run this now, go run, cmd, test server, main.go, it will complain because we don't have this server certificate and server key. This is a server certificate and server key that we still need to create and how would you create those in the past? Well you have the open SSL command line utility but we don't want to use it, open SSL utility, we want to write our own command line utility and our own command line utility will generate this CA certificate and key and we'll sign the server certificate and key just as I explained in this previous diagram. Just as a refresher, this is the setup we are going to build. So this server here on the right with the server certificate and server key, that's what you are going to pass to this function to start our server but first we need to create the CA key and certificate to sign our certificate. We can then have a client that loads the CA certificate, then connects to the server and can then validate the server certificate using the CA certificate. Our client can be a browser, can be another GoLang program or can be something like curl or postman, I'll be using curl. So this is our server and we just want to write a command line utility so we are going to have a TLS command. So TLS command, go run TLS main.go and nothing will happen because we don't have any code in there but that's how we are going to run our TLS executable. You can also build it obviously and then execute it like that. So once it's ready you could actually make the binary and use it even yourself to create certificates or even extend on it because our functionality will be pretty basic but you could still extend on it. Because the OpenSSL command line utility is quite extensive, you can do a lot of things with it, we will just do a subset of that, we will be able to create CA certificates and sign them but compared to the OpenSSL utility it's not a lot. So this main Go will load the PKG CMD package, PKG CMD so that's where we are going to write our command line utility code. We have the root.go and in root.go we have the execute function. We are going to use a library for that, it's a popular command line utility library, it's called Cobra, so we will be using Cobra for the command line utility code so we don't have to write everything ourselves. Cobra is used by popular tools like Cube CTL so it will look very familiar if you already used Kubernetes at some point. So this in CMD will have our command line utility code and in the third package will have our certificate codes. So if we are going to write codes for these X59 certificates we will do it here, we need to create a CA certificate and create a server certificate. So the server certificate is not very special, it's only the CA that is special so we can have a function for the CA certificate and then a function for a regular certificate. And then later on when you need to create a client certificate for two way TLS you could reuse this function. We have the pem.go and the pem.go just has one function, pem2x519, it just reads input in binds which is a pem formatted file and then we can decode it and parse it into an X519. It is just a helper function right here that you don't have to write yourself. In types I have the types, the CA cert, the cert and the cert subject. Why do I have already those? Because we are going to use a config file and this config file is something I will just supply to you. It's called TLS.yaml here and here we are going to define the information that we need to create our CA certificate and our server certificate. So we have a CA certificate, every certificate needs a serial, how long is it valid, valid for 10 years and this is a subject. Every certificate, the CA certificate and the server certificate they have a subject. They have a country, organizational, organizational unit, locality and a common name. There's even some more information that you can give that you will see in these types that go. So you see here the subject is a little bit longer, there are some more attributes that you can add here. The common name in a certificate is either just a name like CA certificate to show someone looking into the certificate that this is a CA certificate, you could even add the name of your company but then for example if you are going to use certificates, server certificates that we are going to use on websites then the common name is going to be the host name. So here we have common name is Godemolocaltest.me and localtest.me and all the subdomains of localtest.me they actually resolve to the IP address 127001 so we can use this address because it will resolve back to our own machine and that gives us a DNS name that we can then use to connect to in our browser or with Chrome. Our service certificate also has these other subject lines. It has a sale valid for years and you will see that the CA is typically valid for longer than the certificates itself, the service certificates. Its best practice to rotate them once a year for example could be earlier with let's encrypt is going to be earlier with root sign certificates you can often choose we are choosing here for one year but it could be as well two or five years or something that you typically want your CA to last longer than your certificate. So if you pick 10 years here or 15 years you will also have to increase this one. And then the last one is that if you want to have more DNS names you can also use DNS names and then you will have your common name plus more DNS names. For example if you are using your own domain name and your own domain name is called local test of me then for example you could have www local test of me and then the DNS name could just be the root of the domain so that you cover both of them. We are just going to use code demo but feel free to use any name that you like. So this is still as a demo we'll still need to parse this we just have the types available so we don't have to create them here's the YAML annotation so that we can un-martial this with YAML we'll do that in the root package in the next lecture it's one of the first things that we'll do after I explain you the basics of Cobra. So next lecture we will start with Cobra this will go in the CMD package then the certificate code will go in the search package and then we also have the key package. So every certificate needs an RSA key and because already went over RSA keys in our previous section of SSH because SSH is also using RSA I already have this code prepared so the code is very similar to the SSH code so if you want to know the details of RSA keys I would suggest to have a look at the SSH lectures. We are just going to use these functions it's actually very simple we have the create RSA private key with a search on length which will use generate key and reply a private key with this RSA private key we are not that much we need to transform this private key to PAM format so this is what the RSA private key to PAM does it takes a private key in the private key format and returns a PAM block with RSA private key so then we can save this in a file and then we just have a save function create RSA private key and save which will create the RSA key with permission so that only the owner can read it it will encode it using PAM and then save and close this file and then we have the function to read it back so if you have a file in PAM format we can use a private key PAM to RSA which will then PAM decode the RSA private key and put it in the RSA private key format so for that we use the parse pkcs one private key function to convert it from a PAM format to an RSA private key format if we ever need the public key the public key is accessible within this private key as well so these are the functions the RSA functions that we have to create a new key and convert between PAM and RSA format so that is it I also added this git ignore to make sure that we don't commit any PAM CRT or key file to github and then we also have the go-mode file where I define the module name and I also replace the mock version from 143 to 144 which you probably already have seen in one of my earlier lectures there's just something wrong with this 143 version so it's just saying if you come across this 143 version just replace it with 144 so this is it for the TLS start program in the next lecture I will start working on this program starting with Cobra if you need any of the solutions then you can have a look at TLS demo where you'll find all the codes that has been written in this demo of the TLS command line utility tool I'm going to use Cobra Cobra is a project that you can find on github so it's github.com SPF 13 Cobra currently Cobra is a library for creating powerful modern CLI applications Cobra is used in many Go projects such as Kubernetes Hugo the GitHub CLI to name a few so it is actually quite popular because Kubernetes is a quite popular project and it is using Cobra so what's so good about it they have an easy way to create subcommands for example you can do app server app fetch so it's pretty easy to use you can also nest your subcommands we can have in our case TLS create TLS create key TLS create CA TLS create cert so those are all subcommands that we can create in a nested create command so let's have a look how we install this go get github.com SPF 13 Cobra at latest so I'm going to copy this already and then we have the user guide here so if you want to know a little bit more about Cobra I would recommend you to read their user guide it's not very long so what we already did is we created this CMD directory in our PKG directory and this is where you typically put your Go files for Cobra our main.go looks exactly like this we have a main with CMD execute so the main is very bare it serves only one purpose to initialize Cobra so we're going to initialize Cobra in the execute function within our CMD package they have Cobra code generator if you want to use it but we are not going to use it we already have some code ready to go the first goaling program that you would make is for the root CMD ideally you place in root.go so that's what we are going to do first that's where we have our execute function so let's try to copy this code and see what happens we'll replace this you go by our TLS explanations so first I do the go get and this is going to download Cobra 150 it's updating the go mod file and now I can start writing in root.go the first lines that are going to use this Cobra package so I'm going to copy this and just replace this save this so the imports are correct so we imported Cobra this is the root CMD and when we do execute it will execute the root CMD.execute if something goes wrong it will show an error on SCDR so I just want to replace this the use the short the long which is the command and the explanation TLS TLS is a command line tool or it could be utility or interface for TLS and then we have a long explanation TLS is a command line tool for TLS mainly used for generation of certificates but can be extended I think that's fine let's just have a look what this does so let's run our go run CMD TLS main.go and it doesn't do anything yet because we are just hitting this do stuff here but because we already initialized Cobra we should be able to use the help function and this is what is nice about Cobra now that we initialized our root CMD it already has a help function TLS is a command line tool for TLS and then you see can put some nice text here usage TLS flags there's a help flag and now we can start adding commands every command can go in a different go file and we just need to add more of these Cobra commands once we tie these Cobra commands together they will show up in the usage we'll also be able to add flags and then they will show up in the flags so this is a very nice package that will make it for us very easy to write a command line utility so one last thing that I want to mention here is that we have this execute function and this execute function happens on the root CMD but the root CMD is actually declared outside our function so the scope of this variable root CMD is not only within this function but within the whole package so once we create more go files we'll also be able to access this root CMD that way we'll be able to tie all these commands together to the root CMD and that's how then the root CMD execute will tie everything together the root CMD execute will use this root CMD everything will be tied to this root CMD and that's how this Cobra package will be able to find the link between all these go files and then we'll be able to show the commands that you can execute and the flags you can use so how to add flags and commands will all become more clear in the next lecture this was just a simple introduction to the Cobra package in the next lectures we'll be adding the config file and more commands to be able to create our keys CA and certificates now that we initialized our Cobra package let's try to parse this config file so this config file describes a certificate our 30 certificates there's a serial valid for years and a subject and then it has certificates that are going to be signed by this CA certificate so search is a map because we have here a key and we can have more certificates that are identified by key so we can have one or more certificates or even zero that way we can have now one certificate but if you later want to add more certificates we can add more certificates so the first thing is going to be to add a flag in our root.go to allow the user to supply a config file just the path to a config file and maybe we can even put tls.yaml as a default so that if you don't supply anything it will automatically look for tls.yaml so how do we do that let's maybe have another look at the documentation of Cobra so create root simd that's what you already did we have function execute you will additionally define flags and handle configuration in your init function so remember that when we have an init function that function will be automatically executed so we can write this init function and add some more flags so here we have the init function and then we typically do Cobra on initialize init config so this will run this init config so when Cobra initializes this function will also run where we can then initialize the config so I'm going to do it very similar as the user guide we then are going to add to the root simd a flag when we use persistent flags then for every subcommand this flag will also be available so for our subcommands we are going to use local flags that are only available on the subcommand itself and on our root command we're going to use a persistent flag that is going to be available for every command because we're going to have this config file so I'm just going to copy this and add this right here we will also then need an init config func init config and the config file so this config file is going to be global as well for config file config and let's then define a config type which is going to be a struct with a ca cert of type cert ca cert I'll make it a pointer so it can also be nil and then yaml ca cert and this yaml should of course match with this ca cert looks like it's matching need to do the same for the certificate there's going to be a map map string cert dot cert and then the yaml is going to be certs certs here the key and this is going to be the cert object going to save this so this is a config so config files of type config init will automatically run it will say on initialize run this init config and this is not going to be a string var so actually this is the config file path that we need so we need another variable config file path is string and then we will need just a variable config and we need to parse the contents of this config file path into this config config file path is going to be this one config file path this is the parameter config the default value we can leave it empty and then fill it out ourselves if you want in the init config config file default is and then we can say default is tless of yaml we can also choose that there is a shorter version of config so if you say string var p then we will have an extra parameter we have the name which is config but also the shorthand and shorthand can then be c so we can use either dash dash config or c then we need to parse this config file which is yaml so if config file path is empty then we're gonna say config path path is tless of yaml so we can either check for the config file here or we can input tless of yaml here so it's the same where you put it if you're going to check here in init config you know that it was not supplied so you could take other actions like not reading the config file for example if it was not supplied so then the next step is going to be to read the contents of this config file path config file binds is IO read file IO util read file config file path if there is an error then we get output that has an error error while reading config file and actually this needs to be print f because we are not returning an error we're just printing something on screen and then return it and then we can unmartial into into this config file so unmartial takes the binds config file binds then we can marshal this into our config error handling while parsing the config file I have one error and sometimes what happens is that visuals to the code decides to import some random library it's best to check what library it is importing this is the one I typically use go get and now everything is green so this should pass our config file and what we can do is we can output it just to see if it worked well done later on remove this it just to see whether our config file is parsed go run so this is the help flag doesn't seem to be parsing it with a help flag let's remove the help flag config file parsed and this seemed to have worked cannot see all the data because it's a pointer but this would seem to have worked and this shouldn't happen every time when we execute a command using the command line utility so whether you want to parse the config every time when you execute it depends on how you want to write this command line utility if it is empty you could also say I'm not parsing it or you could say I'm trying to parse it and I will not give an error if there was no config file supplied and I couldn't read a file you can also check whether the file exists for example it's kind of all up to you how do you want to write this command line utility it should be as natural as possible for the end user without doing crazy things that the user wouldn't expect I think this is fine let's say that you always need a config file so that when you want to use this tool you just need to have tless.yml in your directory or you need to supply it with config so if you say config my other config.yml then I get an error because the file doesn't exist so that's how it should work I can even try with the shorthand flag the shorthand flag minus C so that also works and if I use help it also shows us the flag so in the next lecture what I'm going to do is I'm going to start writing our subcommands so now I'm going to create the commands and the subcommands I want to have a create command and some subcommands under this create command later you could then extend this tool with some other commands create is what we need for this test server but maybe you will also want to read certificates and you would need an info command we are just going to focus on this create command so I am going to create a new file here create.go package cmd and I'm just going to write an init function here and now I have access to all these global variables here so I have access to the config the config file path but mainly to this root cmd so if I create a new command I want to say root cmd add command and then I will add a create command and this create command can be of the same type of Cobra command so this is gonna be my create cmd and I'm gonna import Cobra and here I can have the same information use short and long and actually I will delete this run function here because we don't need to have a run function when we just execute the root command without anything use create create CA certs or keys commands to create resources CA certs keys save this and let's see what the behavior now is go run main.go so now that we don't have a run function it actually shows the help so now you have usage available commands completion help flags and then we have additional help topics the last create but this is going to be another available command but we haven't written any subcommands yet so we can already see that it recognizes that we have this create command because create CA certs or keys is actually here in the help so if you do help we get the help if you do create we just get this create help same so we would need more subcommands so let's create another subcommand for keys key.go package cmd another init function instead of root cmd we now I'm going to use create cmd create cmd add command and we're going to have a key create cmd key create cmd is going to look similar key create cmd the cobra command key key commands commands to create keys and then we're going to have this run function because now we actually want to do something run which is a function and here we can put our code so I'll see if we now use create now we get a help create create key key commands so if you do create key nothing happens because this function is still empty and and now if we enter go run without any argument we see the available commands now we have the create command here as well because now this create command is actually doing something so it's a real command we have the create command we have the config flag which is global I'm missing here a bracket this key command could use a config but I don't think our key command would need the config and now we can even add commands at the key level so what we want to do now we want to add a key where do we want to write the key so we could use a destination key a destination key as a flag so we could create a flag within this key so a flag we can only use in in our create key sub command so now it is not going to be a persistent flag it's going to be a normal flag key create cmd flags string var p where are we going to store it var key out is a string key out and then a name key out shorthand so now that we have c already for config it needs to be something unique so maybe just k the value is going to be key dot pam and the help destination path for key so now we can write a key code so how to create a key well we already have this function written create RSA key and then convert it to a pam create RSA private key and save so we could use this search dot create RSA private key and save off these key out we could even add a flag for the bytes we could add a flag for the bytes so the byte length key length called L it cannot be a string it's an integer I think an integer key length integer 4096 is default can choose a default we are going to make it a default is going to be int var key length key length and help and this is going to be cert cert would be still the imported looks like it is not being recognized so go mod copy this PKG cert and now it is recognized not declared cert it's not going to be cert it's going to be key make sense and what does it return an error if there is an error that we need to say something create key error and an error and then we can return if you want so let's try that out this will be our first function so we have the create and actually if I mistype create that's what is also nice did you mean this great so this is also nice that it has these little things that we don't have to write ourselves so now we have the last create available commands key create key and the length is going to be key length 1496 by default so we don't really need to supply anything create key and nothing has been returned that's also a little bit awkward so let's say key created with length and then we just say key out key length save this for the return and key created key dot PAM with length 4096 okay and now you have the key dot PAM so this is our first sub command key dot go and now we're going to create another sub command or two other sub commands CA and cert and the difference there is going to be that here we already had a function create RSA private key and save we still need to write a code for the certificates the CA certificate and the cert so the Cobra code will be very similar but then we actually go into dig into the package cert to be able to generate a CA certificate and then a server certificate now that we have finished the key dot go let's copy paste this key dot go and then let's create a new file CA dot go and the CA is going to output two files the certificate and the key so we're not going to really use this key dot go anymore we are not going to separately create a key and then input a key when we create a CA we're just going to create a new key so we're gonna have CA key and CA cert copy this because they are both gonna be a string CA key CA cert key out cert out CA dot key and CA cert destination path for CA key destination path for CA cert add CA create CMD to the create command still a cobra command CA CA commands commands to create the CA and we are going to use cert create CA cert this is the string and what does it need the config which should be initialized so we should have our config in this variable because we did this so create CA cert config CA cert so that comes straight from our config there's gonna be this one here and then we have the key file path and the CA cert file path which is a CA key and the CA CA cert create CA error CA created with CA created key cert and then CA key cert save this looks like it's going to work clear this and we have an error we have the ad flag error because look I'm still doing key create CMD within the CA so this is not good so if you do something like that you will get an error and let's try again available commands CA and key now and you see the global flags is a config so we can supply the config when doing the CA command and that's useful because we have here the config that we parsed so this is it for the cobra implementation now I will need to do the actual implementation of the creation of the certificate for the CA so how do we even start with this so we know that there is an X509 package X509 and we are going to create a certificate so we have X509 the package and what does X509 have create certificates so now it imported crypto X509 create certificates create certificate creates a new X509 V3 certificate based on a template the following members of template are currently used and then there's all the parameters of a template of an X509 certificate so what do we need for this we need an IO reader and we need a template so why don't we start with creating a template template equals X59 certificate and this needs to be a pointer so I'm just gonna make a reference to this and then what do we need serial number serial number CA serial because we supply the serial number right here what else we have valid for years subject subject we also have PKIX name is a type we will need this one and this has all these parameters so we can just supply all these parameters we just need to make sure because this is a string array that if they were not supplied in the config file they'll be empty so we don't want to return an empty string in a string slice so let's maybe first make a function remove empty string input is a string slice and R plus a string slice and what we are going to do is if length of input is one because we only have one item in this string and in the first element is empty let's then just is empty let and just return an empty slice and otherwise let's return input remove empty string or yeah and then if we then do country remove empty string so if our CA subject country would be empty then it's going to be an empty string but then this function will remove this empty string by returning an empty slice so otherwise it will show nicely in our certificates and now basically for every type that we have because I don't know if I have all the types for every type that we have we can define these values so that should work what else is there not before not before is of time time not before time now not after time now add and date and then we can add some years now the only thing that you might want to check is that whether this valid years is not zero or empty it's an integer so it will be zero if this zero you probably might want to send to one or you might want to return an error not before not after what else does it have key usage we should also define extensions XY extensions is CA also need to say is CA is true and then I'm gonna add this extension key usage is of type extension key usage and then we need to add some key usage so this key usage for the CA needs to be five or nine key usage digital center and actually also the key usage third sign if you're interested in what exactly these flags mean besides that they put some constraints you can have look in the standards documentation these bits are actually explained and what exactly that they do so we are just going to assume that these are the ones we need and what I now want to do is that this create CA cert and this other function create sort can use the same template so I'm just gonna copy this over right here and the key usage needs to be different here because our server certificate doesn't need this one but I can now change the CA right here insert we'll still need to change this template a little bit don't need the CA but we can now create a common function for these create CA cert and create cert you just need to make sure that we have all the parameters here as well I need just one more basic constraints valid indicates whether ECA are valid so if you are using ECA then we also need these basic constraints valid to true let's now try to create a function that can be used by both this create cert function and the create CA cert function in the next lecture where we'll do the certification where we'll finish the create cert we'll probably make some changes here still but this is good enough to get an idea so what would we need to create a function that we can both use for the CA and the cert so we are going to take this template as an argument so I'll call this create cert with a lower case C so outside the package nobody can exit it X519 certificate so I can say template needs to be a pointer X519 certificate what do I need to create this certificate I will create the private key but I will for sure also need the CA certificate in case we are going to create a normal certificate and it needs to be signed so I'm going to pass CA key that's gonna be a RSA private key and a CA cert and that's going to be an X519 certificate and because these are pointers what I can do is if I'm going to create the CA certificate I don't have the CA key and the CA certificate yet so I'm just going to pass nil and what am I going to return most likely bytes for the key and for the certificate so that we can write it out or use it and an error so we still have errors because we are not using this template so let's try to complete our certificate functions key binds cert binds error is create certificate template and we are not going to pass a key or a certificate because that's for the create cert if error is not equal to nil we're going to return the error and otherwise we are going to write to a file the key binds and the cert binds so let's leave it like this for a second so that we can focus on this create cert see we'll just leave it like this and then in the next lecture we can focus on this so create cert what do we need to do we need to create an RSA key first we have this function in the key RSA go key key create RSA private key and the bytes are we passing them we are not really passing how many bytes so I will just keep it on 1496 what does this return the private key and error if the error is not equal to nil we return the error now if our template is a CA we are going to do something different than whether our template is not a CA so each CA is true then we are going to do something else we are going to do something else our certificate is a CA so we can do X519 create certificate rent reader template the parent the parent because it's a CA it's also the template the public key so private key public is a public key public key and the private key and public key also needs to be a pointer so I'm going to add this ampersand so that it's being sent as a pointer if you don't do it then it will just say that he doesn't recognize public key because it's only checking for pointer public keys what does this return a byte error we probably want to do a create certificate here as well so I'm going to create a variable here got there bytes there's a file type that we get back and error if error is not equal to nil return error and now we need to just convert it into PAM format because we also want to return it always in PAM format so we can then save it PAM.co could we use this no we couldn't use this because this already expected to be in PAM format but we can use PAM encode instead of decode so PAM encode we need to write it somewhere so I'm gonna have a search out and this needs to be an IO writer so I'm gonna have bytes buffer and bytes buffer implements the read and write methods for bytes so PAM encode search out and the PAM block PAM block that is matching PAM block what do we need to add here type optional headers and the bytes so bytes we have a type is going to be certificate and the bytes is going to be that their bytes search out does not implement IO writer and that's because bytes buffer implements it as a pointer so I need to reference it here and there's an error later turns so if this error is not equal to nil then we're gonna return error if there is no error then it will have taken this PAM block with a certificate type with the bytes and it will have written it to search out we can do the same for the RSA private key so if we make a key out bytes buffer changes into key out PAM block and I think we have a function here RSA private key to PAM and this is returning this PAM block type RSA private key and then it's going to marshal this private key into bytes so we can use this instead of the PAM block key and then the private key and they should also work so we have a certificate private key and insert out and key out we now have the bytes PAM encoded so we can just return it now set out bytes what does bytes return bytes key out maybe I'll put the key first I think I always put the key first so if you put a key first we should always put the key first no errors but we only did the CA obviously and now we can have key out here key bytes and we can write this to a file key bytes and search bytes IO utl write file file name which was supplied key file path the data key bytes and the permissions for the key it's going to be 0 600 what does it return an error if error is not equal to nil going to return error and we're going to do the same for the third bytes and the third bias can actually be 644 because our CA certificate can be public so it doesn't need to have these extra permissions it's just for your local machine just to make sure that nobody else can read it accidentally it's kind of best practice to have your key always written with no group or other reads permissions search file path CA search file path and third bytes save this and let's hope that it all works only one way to know go run let's first see whether it compiles okay available command is the CA should we do help not sure that I'm gonna work minus H unable to redefine C okay because we already have the config see there is already C here for the config so I'm just gonna use oh okay and oh okay the last create CA flags the O flag the key flag the H for help and the global flags so if you want to supply a config file we don't need to really supply anything then it will take the defaults CA key CA cert and that created everything I just need another return here how can we validate this well we can actually use the open SSL utility open SSL in CA cert text no out I think that it is and if you don't have the open SSL utility you can either download it or you can also validate online there's tools online if you type open SSL read certificate there are websites where you can paste your certificate and it will show the output for you you can do this with a certificate but don't do this with a key because the key you should keep secret this was not a command open SSL X519 it is because it's of X519 type and this shows that in text everything serial number one so that was good not before and not after there's ten years in between so that's also good the issuer is the same as the subject because it's our self-signed CA country US locality in New York organization goal and demo organization organizational unit certificate management and common name CA certificate then we also should have these extensions the CA is true digital signature certificate sign web client application that all looks good so I think we now have our CA created and now we can use this CA the CA key to sign our service certificate so in next lecture I will create a service certificate and have it signed by our CA so let's go back to our X519 not go and here we have this crate so that we didn't finish template serial number we have the country the common name the common name but we also need to add DNS name not before not after let's see if you just have DNS name DNS names it's also a string so if it's empty then we return empty string if it's not empty then it should just be the DNS names search subjects or search oh I think we are not parsing this is our types yes we can add it here DNS names it's a string DNS names and then here DNS names what else do we need to do we need to parse the CA key and CA cert and then pass it to this create cert function key and private key PEM to RSA yeah that's the one CA key and is going to return as the private key because we need a private key to be able to sign this returns an error so we have the CA private key and the error if there's an error return error and then we have the CA certificates and for that we have in PEM.go PEM to X519 CA cert parsed CA key so let's now just copy this the template the X519 certificate and then the CA key and the CA cert CA key parsed CA cert parsed and then we're going to write the key file path and the cert file path so we're still to write the server key and the server cert based on what we get back save this and then we just need to make sure that if it's not a CA that we also have some code create certificates the parent in this case is going to be the CA cert and the private key is then going to be the CA private key because to sign a certificate we need the CA private key save this is going to create the certificate and then we just need to make sure that we have the code in our CMD for the cert I can copy paste most of the stuff from the CA.go so that's what I'm going to use cert key and cert cert key cert oh cert is already declared so I'm just going to call this path at the end cert path so this shouldn't really give us an issue because those are local flags but now we also need the CA key and certificate so let me just change this create CMD cert CMD and then this needs to be cert CMD cert cert commands commands to create the certificates and then we're going to have a create cert and we also need to supply the CA key and the CA cert so it's going to be cert key path and cert path but then in between we have these two still to add create cert error cert created cert key path okay I think we have everything but we need to ask the user for a few things we need to know the CA certificate and the CA key but also the certificate name so cert name that's going to be another flag cert name or just name with n what is going to be the default that's not going to be a default name of the certificate in the config file and then we also need let me just maybe could we paste it from here key and cert CA key and CA cert and maybe let's not give them shorthand flags might be confusing a bit defaults CA key and CA cert yes why not is the CA key to sign certificates and CA cert which ones are mandatory now well those three are now mandatory so we can say cert create CMD mark flag required name mark flag required CA key is required CA cert is required and the name is required and then I have to pass the CA key but it needs to be in bytes so I need to parse it first CA key bytes error is IO util read file of this CA key if the error is not equal to nil CA key error and we'll have to stop and we are going to the same for CA cert read the CA cert CA cert error read error and read error we'll call it and we can add the CA key binds and the CA cert binds save this we still get error no new variables on left side config CA cert that's now going to be something else so we still have to say cert config okay is config cert which is the map and we have the name cert name so you're asking for the name we checking in this map whether this key exists if this key exists okay will be true if okay is not true then we're gonna say could not find certificate name in config and we'll stop and this should be then the cert and that is a cert all right will this work hopefully maybe I made a mistake somewhere but then we'll find it out very quickly if this doesn't work go run create cert because it's called cert error required flags CA cert CA key and name not set so that works pretty nice and then we have these flags that we can set so let's add these flags the name is this key here go demo local test me CA key is CA dot key and CA cert is CA cert could not find certificate name in config cert name cert name here string var probably did something wrong sort path which would be sort name because that's a name cert created key is cert key and sort is cert cert let's do the same with this open SL utility on our cert cert and actually I didn't want to call it cert cert I wanted to call it server dot cert and server dot key so let me just just run it again because that's how we put it in our test server I think server dot crt okay server dot crt is created open SSL utility on server dot crt seems to work serial number one issuer is our CA certificate and the subject is this code demo one year valid we have these extensions and we have these alternative names right right let's try to run our test server and it's going to run on 443 let's try first maybe to use the browser port 80 this side cannot be reached I'm going to add hps and then it says your connection is not private because our certificate authority invalid it doesn't know about our certificate authority this CA certificate that is being issued and that is because we self signed our certificate because I was explaining that our browser knows about our root certificate but not about our issuer CA certificate that we created ourselves so we need to add this CA certificate to every client that we are going to use some clients can do it easily some clients cannot but you see that there is a reason why you should use a root certificate for a public website or let's encrypt and that a self signed certificate will not work if you are not a client it still works for development purposes because I can do advanced and I can do proceed and then I get it's working but I will still get this not secure because the CA cannot be validated so let's try to do it with curl and let's see if we can have the CA validated and if we can have the CA validated then we should not get an error curl unable to get a local issuer now let's try curl with CA certificate CA CRT so we are now supplying the CA certificate to our client curl and then it will be able to use this CA certificate to see if our service certificate is valid and actually I get another error no alternative certificate subject names matches and that's because I should probably have not only the common name here so either you use not DNS names and then it will just work on the common name or I need to make sure that I define everything in DNS names so I'm going to save this and then I will have to reissue my certificate let's run this open as well tool again and then I see subject alternative names now I have both of them and let's try now again connection refused because I didn't start it go run testing the and now it says it's working so only when you supply the CA certificate only when you supply this argument CA cert then it will be able to validate it because now if you do curl without it it only has the root certificates from somewhere a directory on our operating system where we have all these root certificates but we are not using a root certificate we are using a self-signed certificate which we then need to supply with CA cert so in your browser you could also make it work by importing this CA certificate it's not really recommended on your browser to use every day to start importing certificates so in general if you are going to work with a server then you should use let's encrypt or root certificates it's for testing or it can still be in production when you can supply your own CA certificate or in companies that manage every browser on every machine they also sometimes inject their own CA it's also useful for 280 less as I explained earlier with 280 less you can have your CA bundled with your client and your server and then you can also verify it so this is the end of the demo I hope that this whole certificate CA has been useful it teaches you not only how to do it in go but also shows you in much detail how this signing and validating works and what really the process is when connecting to a secure website now that we have our test server running with our self-signed certificates let's now investigate how we would run our test server with a let's encrypt certificate let's encrypt is different because let's encrypt can sign using a root certificate a root certificate that is recognized by your browser or my curl so we don't need to import the CA certificate to be able to validate the server certificate even though let's encrypt signs with the root certificate it was still useful to explore self-signed certificates because it's also useful for two-way TLS and like I explained in the previous lecture there are different scenarios where self-signed certificates are preferred let's try to create another server our let's encrypt server and in this let's encrypt server let's make a new main.go package main and let's copy paste from our previous lecture and what we are going to do different now is we're going to use auto cert auto cert is a package provided by go so we need to do a go get first golang.org x so it's extra packages crypto acme this protocol that is being used by let's encrypt and an auto cert and this package contains helper functions to integrate within our htp package the automatic certification so we will not have to do much we will just have to add this auto cert to our TLS server and then it will create a renew packages for us so let's have another look at this process so this is the schematic what needs to happen is that we need to support this acme so our auto cert package will implement this acme which is in a separate package and it will create this well-known URL with this token to be able to verify our server let's encrypt will then know that we own this domain and will issue a server certificate that we can then use so that is what all happens in the background the only thing that we need to do is to import this auto cert copy our code and we then going to change this listen and serve into just serve and serve will then ask for a listener a net listener and this net listener is implemented by this auto cert so this auto cert has a new listener and we can just apply the domains which needs to be a real domain so if you want to test out this demo yourself you will have to register a domain or use a domain that you already have my name is registered by namecheap.com and namecheap gives you a DNS panel where you can add records to point a DNS name to a server so it's an in a DNS record and I'm running a VM on digital ocean but you can really run a VM anywhere and I'll have to then copy this binary this go binary that I will create here to my digital ocean VM so the domain name is go demo test new tech academy so if I try to resolve this post go demo test this goes to my VM so you need to have this setup first and then what I'm going to do is I'm going to copy this binary to this VM run this on this VM and see if I will get the certificate issued by let's encrypt so once this code gets executed it will serve on for for three so I just have to curl to for three this new listener of auto search will then go to let's encrypt and ask to issue the certificates so new listener is convenience function for common configuration you can also do more complex configurations by using the auto search manager type also important certificates are cached in the goal and auto search directory so it's not going to issue a new cert every time it will create a demo directory and you will find your key and insert in there so this should be it really let's try to build it go build let's encrypt server and my server is a linux server so I will need to also specify that I want to build this for linux so I will specify go operating system linux go architecture amd64 and I will compile cmd let's encrypt server main.go then I will copy this over to my test server scp I will need to specify my ssh key root go demo test and the source file is let's encrypt server now I will have to start this binary on the VM now I'm logged into the VM and I can start it using let's encrypt server and let me now create a new window and I'm just going to go to this website go demo test new tech academy and if everything is working I should get that it's working it's working so no ca certificate warnings a new certificate has been issued which is now a let's encrypt tls certificate so if I try this in the browser htps go demo test the htp will not work because we didn't open anything on port 80 but you could actually add something on port 80 that just will forward to htps and if we now click here connection is secure certificate is valid and it is a let's encrypt certificate that has been issued and this is valid so if you are doing one way tls and you need your server to be accessible to everyone on the internet then the easiest way to do that without having to buy a tls certificate from one of these known companies is to use let's encrypt and you see that it's very easy to integrate it within your go program what happened in the background is that go did this acme validation for you in this autocert package there's code to that for you and if you then go back to the VM I will see these errors as well it just other people like bots trying to make connections but we only whitelist this domain name so we'll only ask a certificate for this host name not for any host name that people are trying to access so if we now have a look here in the dot cache directory there's a golang autocert directory and here we have our certificate go demo tests new tech academy these are certificates and this is our key so here's our certificate that has been issued so this is it for this demo it's just a very small change to have in go a let's encrypt certificate issued and use it with a tls listener I have shown you how to start a server using a self signed certificate I've shown you how to use a let's encrypt certificate now let's try to create a server that is two-way tls so any client connecting to our server will also need a client certificate this is what I explained in one of the previous lectures the mutual tls diagram so what I'm going to do is I'm going to create a new certificate a client certificate and the client is then going to connect using tls to the server the server will need to be reconfigured a little bit the server is only allowed to accept clients if they have a client certificate so the server will also have to validate the certificate authority certificate so we also need to supply this ca cert to the server again this is not very different than our test server so let's start again with our test server let's create directory mtls server and then we'll have an mtls client main.go so what is going to be different the listen and serve we will need to supply our own configuration we will have to change these tls configuration flags so instead of using htp listen and serve we're going to define a server htp server and then we're going to change htp into server and server now accepts only the cert file and key file so we're going to remove the port and then the port is then defined here in address so address is going to be port 443 and this is not any different than our test server the difference is going to be the way we are going to configure this server so in server we have tls config optionally provides a tls configuration for use by serve tls and listen serve tls so we're going to use tls config tls config is a type and what are we going to specify client authentication the term is a service policy for tls client authentication the default is no client cert so the default is that we don't need the client to provide a certificate we are going to change it into require and verify client certificate so we will require a client certificate and we will also need to verify it if we are going to verify it we also need to supply the ca client ca defines a set of root certificate authorities that service use if required to verify client certificate client ca's and this is of type x519 sort pool so we need to find those ca is x519 sort pool this need to be of sort pool type but we have in this x519 package new sort pool as a function and then we have ca append sort from pam and then we can just load these pam sorts as bytes to do that we still need to read the ca cert ca binds error is io utl read file ca cert if there's an error then log fatal error and then we can supply the ca bytes right here this reply is okay so if it's not okay then we also can supply an error ca cert not valid for example and otherwise in the ca we have the correct ca now if you want you can also supply a minimum version tls version tls 13 for example so 1.3 is minimum then now that we are defining error we don't need to use this column here anymore gonna save this and everything is green so we loaded these ca's still listening on port 443 and we specify require and verify client cert so let's try to test this out go run simd until a server main.go and let's curl to local host let's curl to this correct host name go demo local test me unable to get local issuer ca cert ca cert okay so when we can verify the server certificate then we get an error back from the server because we don't have a client certificate and this is only because we have this line so let me try to remove that line and then curl again if we curl again it says it's working because we are not verified a client certificate so to make it working we actually need a client certificate so let's try to create a client certificate and let's try with curl to connect to the server and then in the next lecture we can create our own client in go let's create this client certificate so this is the command that we used to create the server certificate this is not what we want we want to output now a client certificate with the same ca so i'm going to do minus h and it says cert out and key out cert out client cert and key out client dot key so now we have a new certificate signed with the same ca client certificate let's try to curl again so we get the error oh it's still working so i'm just going to make sure this is saved run this allow to make sure that when i do a curl i get the bad certificate error because the required verify is enabled so let's now try to use these flags key and cert so i'm going to use a client key and we are going to use the client dot cert and now it is working so when we are using a key and a client certificate where the client certificate is signed by a ca that is in the list of client ca's it doesn't necessarily need to be the same ca as a server certificate it just needs to be in this list then it will accept a connection and this is mutual tls mutual tls is where we can make a connection over tls but where the server will also verify the client certificate and the client certificates ca needs to be within this client ca's so these client ca's needs to contain the ca that our client certificate is signed with so if you have microservices for example then you can have a lot of different server certificates then you can give every microservice its own certificate and its own key have them all signed by a ca trust the ca and then you can have communication between them using tls and where every client is also verified giving you a very secure way to communicate between services not only giving you encryption but also every microservice will have a reliable identity it will have an identity because you can actually create a separate certificate for every client so here are reused actually our go demo local test of me but i could have created a new certificate called go demo client and then i could have really defined a server certificate in the stls email and a client certificate which would actually have been a better way to to do it i didn't really think it through when i was making this demo but we should have done go demo client local test of me and this should have been the the client maybe without any dns names so every server can basically have its own identity using the common name and you're going to have a different common name for every certificate and then you can have communication between your microservices that the setup with client key worked is logical because you can also use a server certificate and a server key if you want to connect to your server because they are still signed with the same ca so if you use a server key it is still working so in this simple example that i did here it is all working with either the client certificate or the server certificate typically you're not going to use a server certificate for the client but it is actually something that could work so make sure that you always keep your server key private to your server otherwise if you have the server certificate and a server key you could also connect to the server and maybe that is not intended so in the next demo i will create a mutual dls client to do the same that i did in curl within go and i will create a separate go demo client certificate that will have a different common name so that we really have a server certificate and a client certificate that is not just a regenerated server certificate in the previous lecture i showed you how to use curl with this server certificate and this client certificate i am now going to create a go demo client so i'm going now to create this go demo client certificate and i'm also going to write a client in go that will connect to the server so this client certificate that i created with the same common name as the server that's not really how you do it it was just good for a test but i'm now going to recreate my certificate so i'm going to remove the client certificate and client key and recreate it so we have this tless command create cert ca cert ca key and then cert name should be go demo client local test me and cert out is going to be client dot cert and key out is client dot key and then we will have our own common name for our client so it will have its own certificate using its own common name its name and not cert name name and the cert is created let's now create a new command cmd empty last client so this is going to be our client so i have a main dot go and then i'm just gonna write the client that is going to connect to the server package main funk main so how did we do it previously htp get of this hostname go demo local test me htps htp get returns responds an error if there's an error we're going to do a log fatal of the error defer response body clause and then we're going to read the body body error is equal to IO read all of this response body and then we can say FMT print F body status status code HP status code and the body is a string then we have rasp status code and the body so let's try that out so i'm going to run the server let's close all these shells and then i'm going to run this client go run cmd empty last client main dot go it just a test because i didn't add the certificates yet certificate is not trusted i get so i will have to add the ca and because we have this require and verify client certificate we also will have to add the client key and the client certificate how do we do that so when i was doing this empty server that we started the server is HP server then we have this parameter that we can add and then we can do server listen and serve till us it's very similar with the client so we can say client equals htp client and then we can do client get and clients as then variables that we can configure transport specifies the mechanism by which individual htp requests are made if nil default transport is used transport is htp transport and what else do we have we also have a timeout so if you want we can also have a timeout of 60 seconds for example then in htp transport we can add a lot of transport configuration till as client config till as hand shake timeout till as client config specifies the last configuration to use with the last client this is the one we need till as config what is in till as config certificates and we will also have something in there called root ca's and the root ca's are the root certificate authorities that the client uses when verifying service certificates if root ca is nil till as use the host root ca set so we are going to create again a sort pool and then add this ca just like we did at the server so we are going to copy this it's exactly the same we're going to read the ca the ca cert create a new cert pool and then we should be able to validate the server certificate let's try to run again our client now we can verify our server certificate because we have the ca certificate but now the remote which is a server says till as bad certificate because we are not supplying this client certificate so if we have a look now we get client didn't provide a certificate so we need to provide a certificate that can be validated by the server and the server is only going to accept client certificates that are signed by our ca because the only ca cert that we have here as client ca's is our ca dot crt so what do we have in the tls conflict we have the certificates certificates contain one or more certificates chain to present to the other side of the connection so this is what we are going to need certificates tls certificates we just need to now read our certificates we can do that using tls load x519 key pair and this accepts a third file and a key file so client crt and client dot key and it will then read these two files and then store it in our certificate and our certificate can then be used here and this is actually not a ca this is the error and if there's an error then we're gonna do a log fatal so now we have loaded our client certificate our client key we will make the connection to the server and it will provide our certificate our certificate and it also needs the key to make the connection so without the key you cannot make the connection and then we should get the body and the status code if all goes well so let's try that out body status 200 it's working so now we have made a connection using go to the server so we didn't get an error server just replied with it's working so now we have an empty less connection between our client and server I just want to show you one more thing as I said we should use a different common name for every client and every server and the reason for that is that we are giving our client an identity our identity is go demo client local test me and we can use that we can use it in our server to see what identity our client has so if you have two clients with two different certificates we could distinguish between them so if we go to empty less server we can add a function show common name response writer hp request and we can then extract our common name and then we'll add another handle func where is our common name it in the request so common name is going to be the common name you just need to extract this common name our common name is a string and request tls has this information wreck tls verified chains and verified chains has the x519 certificate but it's a chain so it contains also the root CA the client CA so it basically has all the certificates in there but the certificate that we need is the first one so what we can do is if wreck tls is not nil and the length of wreck tls verified chain is more than zero and if the zeroed element is also greater than zero then our common name will be wreck tls verified chains the first element and then subject common name and we know that the first element of this slice is basically our client certificate so let's see if we can get this working common name is show common name save this run this and then go run that's the body and then if I look for common name your common name is go demo client local test me so this is the client certificate that has been signed by the CA I'm able to make a connection it's encrypted but I also have an identity the server can verify who is this client exactly what is the common name of its certificate and then base actions on that it can for example give me more access to an API or maybe deny access to a certain API endpoint so this is a quite powerful tool to have within for example microservices where every microservice can have its own identity and then when we communicate to other microservices we are aware what service is contacting us the only prerequisite for this is that you create a certificate a unique certificate with a unique common name for every client that is going to connect to the server or if you have multiple servers just for every server that you're going to run so that's it about mtls in this section I'll be talking about DNS writing a DNS server and resolver in code first of all what is DNS simply said DNS provides a translation between host names and IP addresses for example if you type in your browser www.google.com which is a host name a DNS server will translate that in the IP address so the IP address now when I tested it on my machine is 142.250.114.99 but if you test this on your machine it could be already another IP address how do you test this on your machine you can use tools like nslookup on windows or host or dig on macOS or linux to do the translation of a host name to an IP address typically the laptop or desktop you are working on has one or more IP addresses from your internet provider to run dns queries so the server that you can call to do these dns queries is typically provided by your internet provider in linux and macOS this you're going to find in etcresolve.conf and on Windows it's going to be in the dns server settings you can check this with ipconfig or you can go to the network settings on a macOS windows machine so even if you are not using any tooling yourself to translate the host name to an IP address whenever you are using your browser you type in google.com the browser will ultimately translate the host name to an IP address using a dns server so dns is something very common that you do in the background on a daily basis so how does a dns request look like so you have your browser on one side your browser is going to look up for for example google.com and it's going to do this lookup using a dns resolver so your internet service provider dns resolver will be used or other public dns resolvers like 8.8.8.8 or 4.4.4.4 which is provided by google so it's not because we are doing google.com it's just something that google provides google provides a public dns server that you can set up and use instead of your internet service provider dns resolver or you can also use 1.1.1.1 and that one is maintained by cloudflare which is another company you can also set up this IP address as your dns resolver and then you'll be able to resolve host names using that dns resolver that is running on that IP address so every time your browser is going to go to google.com and we wouldn't know the IP address yet because once you know it you can cache it but if you don't know it yet then it's going to make a call to this dns server to obtain the IP address for google.com this dns server doesn't really know where to find the IP address of google.com immediately so it will have to go to something that's called the root servers all the dns resolvers will have these root servers configured in their dns resolver settings so they'll have a config file somewhere where they have the IP addresses of these root servers these are the IP addresses that need to be known to be able to start dns resolving so this dns resolver will then go to one of these root servers and it will ask what do you know about google.com it can ask what you know about google.com or it can ask what do you know about dot com because the root servers will not know where to find google.com but they will only know the top level domain the tld so as a dns resolver you can actually choose to send google.com to the root servers or just dot com so we're going to ask do you know about google.com and they will say no i don't know anything about google.com i am not authoritative for that trying dot com at ip addresses so the root server will say google.com i don't know about but i can tell you where to go to get information about dot com domain names so the root servers only know where to find the servers of the top level domains and it will reply the host names and ip addresses of those servers so then our dns resolver knows that it can go to another server specific servers for dot com domain names and that is then what the dns resolver will do the dns resolver will then go to these top level domain servers and it will ask do you know something about google.com i want to know what the ip addresses and the tld servers for dot com will then answer i am not authoritative for google.com but the google.com name servers the google.com dns servers you'll be able to find at this ip address it can also say you can find them at these host names and then you would still need to do more lookups to know the ip address of those host names but in the most simple example it's just going to give us the ip address and then we can just go to this ip address where the name servers for google.com are so these are the google.com name servers and then we're going to ask again what is the ip address for google.com and the google name service i'm going to say yes i have the ip address of google.com i'm authoritative for this domain name so i can give you this ip address and then the dns resolver will then be able to send to the browser the ip address is this ip address that i found for google.com the dns resolver will then typically cache all these ip addresses how long depends on another field that is being passed there's a time to live ttl that is also passed so that ttl will say how long these records can stay in cache and the dns resolver will then keep them in cache so it doesn't always have to go to those servers because google.com for example or just to come till the our credit quite a lot so it doesn't really need to go to the root servers every time it can often skip some servers because they already resolved domain names earlier but this is the most typical flow where the dns resolver first goes to the root server figures out where the top level domain is then goes to one of the top level domain servers and then they will give you a host name or a host name and ip address and then you'll be able to go to the name servers for that domain name and they will give you an authoritative answer that then can be sent back to the client so what are those root servers so those root servers are machines hosted by operators companies that maintain these servers these ip addresses they don't change that often or at all so they are all configured within these dns resolvers there is a website where you can also go to to get a latest list so that list you download as a config file in your dns server and these ip addresses you will have to steadily configure to be able to know where the root servers are so that you can use them as a starting point so it's not that every machine every laptop or pc has this list they typically have these dns resolver configured and the dns resolver will then reach out to those root servers it's not typically the browser itself that will do that the browser will just use a resolver that then can handle any dns resolving for the browser or the user on the machine also maybe interesting to note is that even if you see a single ip address here one ip address is not equal to one server there are actually multiple servers scattered around the globe that are representing this ip address so there's also routing in effect that routes you to the closest root server in your region so there will be multiple servers responsible to be able to handle requests for a single ip address that you see here how does a dns packet look like so before we are going to start writing this dns resolver in go let's have a look how a dns packet really looks like so that we know a little bit more in detail how the protocol works so we can then easily implement it once we will start writing our go program we are not going to parse the dns packet ourselves we are going to use a library that is available within go lang for that so don't worry if you don't fully grasp all the concepts and i'm going to explain because we will still be able to use this package that comes with go to be able to pack and unpack these dns packets so a dns packet can be sent to the resolver using udp or tcp so if you are a browser then you can send a dns packet to the resolver and the packet can be udp or tcp so the difference between udp and tcp is that tcp needs a handshake so it takes a bit longer to get an answer the handshake within tcp gives you an extra bit of latency whereas udp is stateless it's just one single packet so there is much less latency to get an answer a dns udp packet cannot be larger than 512 bytes this to avoid the udp packet being fragmented over multiple packets everything larger than that need to be sent over tcp let's focus solely on udp packets to be able to write a simple dns resolver server so when we are going to write our dns server our resolver we are only going to do the udp implementation so we will not be able to take any packets larger than 512 bytes because that's the maximum you could use the same mechanism to then also implement tcp later on but i'm just going to explain the concept and we're going to keep it simple so we're going to just do udp so what is a dns packet how does it look like so our browser will do a lookup and it will use resolver it will send a dns packet using mostly udp and this dns packet consists of headers there will be headers and then actual data so the actual data which is going to be the questions and answers i'm going to go over later let's start just with the headers this dns packet headers is going to be the same whether it's a question or an answer so if the browser sends a lookup to a resolver it's going to have these headers here and if the resolver sends something back is also going to have these headers that i'm showing so it's not that they have different headers for requests and responses it's going to be the same format headers they're just going to have different values so the headers start with a query id which is 16 bits long this is just a unique id that is used to send a request when the resolver then answers it will also use the same query id so that the browser that did the lookup will know that this is an answer for this specific query id on the next row you will then have flags you can find the explanation of these flags online if you want to know where every flag stands for we are going to use this AA code here so authoritative answer if a server sends an authoritative answer then this flag will be set and this is something we will need to check on to know whether we reached the server that can give us the correct answer or whether we have to reach out to other servers then the qr for example is question or response so this one will be set to one if it's a response or a zero if it's a question then we have our code which is result code which can contain an error code if there's an error and there's also tc for truncation if this is used this flag needs to be set to one and then you have recursion desired and recursion available recursion means that if the service is not authoritative that it will itself then go reach out to the authoritative server to get the request so a resolver is actually recursive because the resolver will go to the different servers to find you the answer if it wouldn't be recursive then if you would ask for google.com it would just say i'm not authoritative for this domain name so here's an error because it is doing recursive lookups it will actually go and find the correct server for you and find you the answer this z here is also reserved so we are not going to use this there are some other flags as well for example the r code here the result code if there's an error code then this flag will be set for example so these are the flags and then you also have more information in this header you have the question count so if the browse is going to look up one host name then the question count will be one the answer counts if it's an answer it can be higher than zero if it's a question only then the answer will be zero the name server count so this is going to be necessary if you cannot get the answer from the server but the server knows other name servers where we can get the answer from then this will be set and this is just a count so this is a field that just counts how many occurrences we will have in the data part of our packet and then we have the additional count the number of additional resource records so this counter is showing how many additional records that we have for example this can be IP addresses if we have the name servers available then the name servers will only be host names if we also have the IP addresses then it'll be available in the additional records and here we just have the count of those records so this is just the header now we can have glued together within the same packet also the questions and the answers so this can be our header for the lookup and then we're going to have also questions so then attached to the same packet we're going to have this question part and this part is going to have a question name the DNS name to resolve so in in this case it's going to be google.com names are split into labels and start with the length of the label first and end with a zero a zero octet so it's going to be double zero if you are reading it in hexadecimal for example if you have www.google.com you will see three which is the first part the first label that's the length of three www is length of three then six because google has six letters so the length of google is six and then three more because then there is come and then it ends with a zero and in hexadecimal this will be zero zero then we're going to have a question type which in our case is going to be an a record but it can also be an mx record mx record is for emails so if you want to know the email server where to send the emails to then you will ask for the mx record otherwise you will ask for an a record a record is for ipv4 if you have four a's you're going to ask for an ipv6 record and then you're going to have the class the question class and that's going to be typically internet address so there's a code for internet address and that is the code that we're going to use so this question packet together with a header is going to be sent to the resolver we are also going to have an answer part of the packet and that's going to be attached to the question but it's going to be empty because we don't really have an answer when we are only asking the question what is the ip address of google.com so let's have a look at an example packet let's say that we want to do a lookup for www google.com then we're going to have the header the question and the answer within our packet so the header is going to have a unique id 59e2 is our id in hexadecimal format then we're going to have our flags which will include that recursion is desired and that we have one question within our question we'll then find this length 3 which represents 3 times w then length 6 for google length 3 for cum and then it also going to pass the type and the class and type a for the a record is going to be one so in hexadecimal it's going to be one and the class is also going to be one so this is how our question is going to look like so this 363 for the label is just to make it easy to parse so programmatically this need to be parsed and because this field doesn't have set lengths like the header we also need some help to know where the field starts or ends and these lengths they help us with that we will not have to do these ourselves so like i was already saying we have a package in golang that will parse this for us even though it's going to be parsed for us it is actually very beneficial to have this schema in mind so that when we are going to be digging into this package that you have an idea what the information is that is available in the header what is available in the question and what's going to be available in the answer so let's now have a look at the answer packet so the answer packet is going to come back from the resolver to the browser so the answer packet is going to have the name as well which is the same format as the question name in the question packet so if you answer google.com then it will also send back google.com also the answer packet will have the same headers so this packet that comes back i'm only showing the answer part of this packet it will still also have the headers and the questions in the same packet so we have the name the type the type is going to be a record the type can be a record mx record or something else the class the time to live so if anyone is catching this record how long can we catch this record that's what the time to live is going to tell us in seconds the response data length so how long is the response going to be and then the response data so the response data for an a record is an ip address so first we are going to say how long this ip address is and then we're going to actually put straight after that the response data so what does the packet look like we are going to have the same id in the header we're going to have again the header the question and the answer in the header we're going to have the same id but then our header our flags will look a bit different you will see here the hexadecimal representation of the flags which are not the same ones as when we were sending the question because here we are setting for example that we have one answer available we then have the original question in our question packet so google.com is still here and we then have the answer for google.com the ip address so the answer from your resolver can come back like this the first cell here c00c is actually a pointer that is saying the name can be found at the position 00c and that's going to refer exactly to google.com in our question data so that way we don't have to repeat the name in the answer so there's a mechanism to make the packets smaller then we have the type so it's an a record so it's going to be one the class is also going to be one the time to live so this is the representation of time to live if you would convert this hexadecimal number to just a number then you would get the amount of seconds that you can cash this entry then how long is the response going to be the response going to be four bytes so 16 bits so eight hexadecimal letters and if you would take those eight hexadecimal letters and convert this again then you would see the ip address that starts with 142 so this is an example answer packet that could come back from the google server that then the resolver sends back to the browser so if you would like to know exactly what is in this header then you would need to expand these hexadecimal numbers because you would need to see on a bit level whether a certain flag is set to zero or one to see what flags have been set so lastly go packages luckily go has packages available to parse those dns packets so we don't have to do it ourselves golang.org acts for extra net dns dns message is the package that can be used to unpack which is decode and pack and code dns packets so when a dns packet comes in over udp those bytes can then be parsed by this packet and that way we don't have to manually go and try to parse this binary data we have this package available to us to do that for us and then we just can use this dns message to extract the information that we need to build our resolver so then we still need the net package that can be used to listen for udp packets and send udp packets to other dns servers so those are the main packages that we would need in our program to then write a dns server that can resolve host names if you would send a query to it and that's what i will show you in the next lecture in the next lecture i will show you how to build a dns resolver in go before we start writing our dns server our resolver in go let's go over the logical flow diagram so that we get an idea of what we are going to create how is our resolver going to look like well we are going to have to listen to udp so we're going to create udp socket on port 53 port 53 is the dns port we'll then have a client that is going to connect to our udp port on our machine and the client is going to send a packet so we need to handle this packet whenever a new packet comes from a client we need to handle it and we can then start parsing it so we can use this dns message package from golang to make it easy to parse a dns packet but we still need to figure out what fields we need to extract and have looked at so the question part of the packet looks like this it will have a question name for example www.google.com question type what do we need to know for this name for this host name we need for example a record the question class and for us it's always going to be an internet address let's go in a little bit more detail so we're going to parse our question and then we're going to create a for loop we are going to do a few actions in a loop until a certain condition is met we are first going to query the server the first dns server that we're going to query is going to be the root server or one of the root servers when the dns package comes back from the root server we are going to ask a question is the answer authoritative yes or no if it is yes the first iteration of the loop it will never be because we are asking the root server and the root server is always going to retract us to another server but this is part of our loop if it is yes then we can send the answer back to the client if the answer is no and if we are sending a query to the root server it's going to be no then we need to parse the authorities section in the dns packet that comes back from the root server to get the new servers so those servers are going to be the name servers for for example .com so once we have these new servers we'll have to again query the server but instead of querying the root servers we are going to query this new server this new authority that our root server gave us in this dns packet that we now know and that we now can query so let me quickly show you with the dns tool how this actually works so i have a tool called dick and i can use dick trace google.com and it's going to show me the trace of all the queries that you would need to do so the first query is going to be to the root servers so these are the root servers a root server.net you saw these names on the slide so we will query these these root servers and we'll ask where can i find .com or google.com typically you send less data to a dns server so it's going to be .com and the query would then go to one of the root servers and the root server would then say here look these are the name servers and it's going to be in this authority section so here we have a.gtldservers.net and these are going to be then the servers we can query and then we can go to these servers and ask where is google.com and then they say google.com you can find those at ns1 google.com ns2.google.com and so on and then we can go to one of these name servers that belong to google and then we can get our a record so these would then be a few iterations in our for loop to get eventually to the IPRS there's just one detail missing here and when we say is the answer authoritative then we say no then we're going to pass the authorities to get the new servers any server that we're going to query might actually not give us the IP addresses so the authorities only contain host names we might get the IP addresses but we also might not get the IP addresses so let's have another look in this for loop in more detail so we have the query server is the answer authoritative no then we're going to pass the authorities to get the new servers these are typically the host names and then we're going to pass the additional resources and in those additional resources which is part of a DNS packet we are going to find the IP addresses if the server the DNS server is going to give them to us in this DNS packet so here we have to ask ourselves again a question is the IP address included if yes then we have the IP address and we can create the next server and do another iteration of our loop but if it's not included then we have to start the whole loop function again to query the a record of the name server for example if the name service ns1 google.com then we have to start the whole loop again so typically when we are going to query google.com and google.com has a name server within the same domain name they will supply the IP addresses but if google is using another name server of another domain it might as well not supply the IP addresses if we are querying google.com and the name server is ns1 google.com then typically the IP addresses will be supplied because otherwise there's no way of knowing the IP address because we would just end up in an indefinite loop it's often that when we could query google.com and the name server would not be in the google.com domain name it would be something completely different for example it would run another company for some reason then the IP address might not be known and then we would have to do a lookup so then we have to start a new loop we kind of have to go again in the same function to do a lookup of this domain name here with google.com the IP address is actually supplied so we could immediately create a new server but with other domain names it's not always the case so a domain name xyz might have an amazon name server and then we have to look up the amazon.com domain name get the IP address and then only we can continue so once we have this IP address of this ns1 google.com or some other host name then we can actually continue because then the IP address we have and then we can go to yes and query the new server with that IP address. So this will become more clear once we start writing the code you just have to remember that we are going to parse the authorities to get these servers we're going to parse additional resources to get the IP addresses if the IP address is not included we are going to have to do more queries if the IP address is included we can just continue our loop and query the new server that we have found. Let's now have a look how to write this dns server which will be our resolver in go. I opened my dns start project that comes from my github.com repository go lang for devops course and in this dns start project I have a cmd dns resolver so to start my project I will enter something like go run cmd dns resolver main.go which just says starting the dns server there's no code yet so here I will have to write my code to start the server to listen to udp packets then I will have a package the resolver.go and this resolver.go it has the root server defined so that you don't have to copy paste it in there and I will keep it up to date if there's an IP that would ever change then I will update it here and this resolver.go has the function handle packet so we can start the server main.go but then we can use the function handle packet in this dns package to handle an incoming packet the benefit of doing it like this is that I also wrote a test file here I have this test file to test this test handle packet where I will try to resolve two host names and if it returns an error then it will stop so if I execute this now then serve error not implemented yet because I have the error not implemented yet returned here within this handle packet we will also have to contact other dns servers and for that we have the outgoing dns query and for this outgoing dns query which will make a dns query to a server defined here in the parameters for that I also have the test outgoing dns query test where we test where we can reach the root servers and we're going to try to resolve the com which we know that we can resolve that we're going to ask for the name server type so we're going to ask the root server what is the name server for the com just as a test and if that works then our test will succeed so right now doesn't work we have no header found because the header is nil because we are just returning nil so these tests they test these two functions so if you use this start package with these functions it's easier to actually test whether your code works I also have this mock packet con because that's what I'm passing here I'm asking for a packet connection so this is an interface for a packet connection because we will have to use this variable here this pc to send something back to the client so we can send a udp packet back and that's why we have this so I'm just mocking this so that we can test this function without having to send a package back well we will trigger sending a package back in this function but nothing will really happen if we mock this function so I'll first start with this function should be relatively easy outgoing DNS query where we're just going to contact over udp another DNS server and next I will then start the server and then start working on the handle packet code and then when everything is finished the test should also succeed let's start with this outgoing DNS query let's have a look at our signature so we have servers service is a slice of net IP so net IP represents an IP address it's in type bytes so we'll have to convert a string IP address into bytes if you want to use this but that is actually quite easy if we have a look here we have the function net pass that can take a string and it returns a net IP address so we have our servers can be the root server can be the google name server can be the amazon name server can be the name server for the dot com tlds for example and then we have the question and the question is of type dns message question so I was saying that we are going to use this dns message package so a dns message question is a question struct that contains a name type and class just like I showed you what a question really contains in a packet so it contains the name the type and the class and that's what we need to then ask a question to an external dns server we are going to return the dns message parser which will then give us access to the packet that is being returned from the server and we'll also return the header just like I showed you in the slides the header contains an id a response which is a flag operation code whether it's authoritative or not whether it's truncated recursion desired recursion available authentic data checking disabled and the r code so this is the header of the packet this is the parser where we can then get more information from the packet so if we're going to trigger this this outgoing dns query then we can trigger this with a question here's our question dns message question it has a name type ns and a class and then it's going to call this outgoing dns query using the server which is the first of our root server because we just kind of split them based on the comma here and here is comma separated IP addresses and we're going to take the first IP address parse this and then parse this net IP address to outgoing dns query to get rid of question so what are the next steps that we need to do to then actually reach out to the server and ask this question we have to start by sending the udp packet to the server and what are we going to put in this packet well we're going to put a dns packet in there and to put a dns packet in there we need to craft our message and package it and then we'll be able to send it so let's start with making our message so our message is going to be of type dns message message which is a struct and in this struct we can put our header and our question so this is going to be our dns packet so we have the header we have a look at this message we have the header questions answer authorities and additionals to send a question to a server we only need those to the header and the questions questions is slice and i already have my questions available here so the header is of type header what do i need i need an id for sure and i need to say that it's going to be a question so our response is going to be false we don't need to enter anything so what i might do is just for clarity add a few of these headers like is the response no it's false but then it's clear that this is a question let's start with the id because it also comes here first and the id is an unsigned int 16 so you want to return a random number between zero and the maximum unsigned int 16 what is this maximum we can actually use the variable max and assign this u int 16 zero with this flag in front of it which will return the maximum possible unsigned int 16 so this max will have the maximum unsigned int 16 in this max variable and then the next step would be to use the crypto rent library to generate a number between zero and this max unsigned int 16 how do we do that we have the rent library and then this one shows month rent but we need crypto rent we need an integer this needs a random reader and then we need to supply a big integer and it then returns a big integer which we then have to cast back to an unsigned integer and an error if there's an error so rent reader and then big has a new int new int which accepts an int 64 so we can say in 64 of this max we change this type from an unsigned into an int 64 now we have the maximum here defined and we can say random number error is rent int if there's an error return the error if there's no error you would still need to convert the random number to an unsigned integer so unsigned 16 of this random number but this random number is of big int so random number has actually a function to return an int 64 and then once we have an int 64 we can convert it to an unsigned int so it's all of steps just to get the right format but then we have exactly this id in the unsigned int 16 because that's a type that it asks what's next let's have a look in his header again opcode opcode and opcode is a DNS operation code so this opcode if you're going to have a look in this package then we see that opcode is of type opcode and it's also unsigned in 16 an opcode is a dns operation code and we typically just use a static value for this which is just going to be zero so we can use dns matches opcode as a function to change the type of this zero to opcode so now we are changing this zero here to type opcode so we're using this type of code to match the correct type and this is just going to be zero what else we can leave it like that I'm not even sure if we really need those last two fields because probably the defaults will be zero and false anyways a just to make it clear that we need these set like this so we have the message we have the header then we have the questions so here we have one question so we can say dns message question and we have only one question so this is a slice of the type question and we only supply one question now this message is of dns message message and this dns message message is struct and has the pack function which returns the bytes so now we can just say buffer error equals message dot pack and now we can use this buffer to send to our dns server if there's an error we're going to return it so how do we make a connection to our server we can even have multiple servers so we should do this in a loop for server in range servers we can use net dial so net dial dial connects to the address on the name network no networks are tcp and also udp udp so we can say we are going to make a udp connection to a certain address the address is the server and this is of type net ip and this needs to be of string so we're just going to use a string function and we're going to do this on port 53 because that's a dns port so dial here are some examples so here's an example for dns this is an ipv6 but we are going to use ipv4 so we are doing dial and what does dial return a connection and an error our connection we probably want outside our for loop so i'm going to say var con is net con con and error equals net dial if there's an error we are going to continue so if there is no error then we can break and if there's an error then we will just continue until we hit a server that actually works so we then just need to check our if so we either then need to check our error or our connection so if connection is still nil there's no connection made then we can return an error failed to make connection to server servers i will just say what is the error return nil nil and then the error so now that we know that we have a connection we can write something to our connection so we can use con write access bytes here's our buffer and then it will return the bytes that it has written and the error so if there's an error then we're going to return the error if there's no error we're going to read the bytes from our connection as i've looked at this connection so read and write and close so we can use a reader a go line reader a stream reader to read this out i can use buff i o new reader of this connection and then i can read this into a variable reads data into p so i just need to define a variable where our answer will go in i can make a new byte and it's not going to be more than five than 12 bytes because it's a maximum and i can read my answer or read will read data into p p is my answer and it returns the amount of bytes that it has read and the error so if there's an error i will return it and now i can try to parse this again with our dns message because this what we get back this answer should be a dns message before i'm going to do that i can actually close the connection just to make sure that we closed already because we don't need it anymore we just want packets so we can close it and then i need this dns message and this dns message has a parser parser is a struct so now this is all going to be empty and our parser has a function to start parser allows incrementally parsing a dns message when parsing is started the header is parsed next each question that can be either parsed or skipped alternatively all questions can be skipped at once so basically because we are doing it incrementally we can say first start then we get a header and then we get the questions if we start with the questions if you don't need questions we do skip questions let me just hear skip questions or skip all questions so that we can either go to the next questions if there's more or skip all the questions if you want to go to the answers all answers authorities and additionals can be either parsed or skipped in the same way and each type of resource must be fully parsed or skipped before proceeding to the next type of resource so this is a funny way to use this package but this parser is very much optimized so it's not going to read the data that you might not need so first you read the headers because the headers come first then the questions if you don't need questions you can skip the questions and then you can start reading the answers and you can do the same then afterwards with the authorities and the additionals know that there is no requirement to fully skip or parse the message so if you just need the headers you don't need to skip anything or if you only want to read the questions you don't need to skip the questions you can just read the questions so this is the parser so we're gonna say p is our parser p has start and accepts our bytes our bytes are an answer so we're gonna say answer but are we going to send all the bytes to our parser because we only actually need the bytes that we have read so this is 512 big but our message might only be 100 bytes so we're gonna stop at n because that's what we have read so if you read only 100 bytes we get the first 100 this gives us the headers and the error so headers error if there was an error we just return fmdrf so it's clear if there's an error parser start error the error and then we could use the headers now what we could do is we can check the header whether our headers contains what we want and we can also check the question whether our question contains the question information that we were looking for and if that is all okay we would then return the parser so that we can later use it so we're going to return the parser and we're also going to return the header so we can also use the header later on so those are the headers or is the header it's actually is it the header or headers it's a header so that way we can then use the header and the parser later on if after calling this function what else do we need we might want to do some checks here whether we have a success response code or not so let's maybe do that first then test it and then write some more checks just so that we can do some basic checks for example whether we have the same question as we asked for so before I'm going to return the parser I already want to check the questions because there might be something wrong in the questions and we want to do these checks first then skip the questions and then when we return this function we can immediately continue with the answers so the questions are be questions or be all questions be all questions and now if I want I can check whether we have the same question so we can I can say lan questions is equal to my original message questions so here I can also check the length so my original message is here on top so do I have the same amount of questions returned that I started with if not I can return an error because then something went wrong and then I can say okay now I did my question checks now I'm going to skip the questions skip all questions and this only returns an error so here just between all questions and skip all questions I can do my question checks if I want to check these things for example I can check now whether question name is equal to the original name and then I skip them and now next I'll be able to return the parser and the parser will be able to process the answers so processing the answers I will then do in the handle packet so I could test this now because if you have a look here I'm going to have a header I'm going to have an answer my header could check the response code then I can skip all answers and then I can check the authorities so if I want to check the authorities I need to skip the answers even then I want to check on the additional resources I need to do skip authorities so that's how it goes you go down the packet by getting the questions then the answers then the authorities then the additional records so let's test it by running the test if something goes wrong you can always do a debug test to see new outgoing DNS query for com on the servers and my test is passed if your test wouldn't complete and you have exactly the same code answer or check whether you actually are allowed to contact the root server directly maybe your internet provider would forbid that by blocking port 53 for example to anything else than their own DNS servers that is a possibility a remote possibility but most of the internet providers just allow DNS because you also have Google's and Cloudflash public DNS servers so that should in general that shouldn't be really a problem if you get an error or panic have a look at the code whether you did something wrong different and you can of course use the debug test so it's not much happening here we are just doing a test and here we are checking where we are getting the answers and we're getting some answers so I could actually output the past authorities just to see if it actually is working past authorities and then run this and then I have the past authorities so we have the header name type class time to live and a length so we have the name com and the ns records so here we have all these ns records for the com domain name so that we know what the name servers are for the com top level domain name you could also change this into google.com figure out what the name service for google.com and then use the server name server google.com here and then you could do it on google as well so then that could work as well if you want to test so test outgoing DNS query just makes with net dial a connection to the server or to the list of servers and then writes our message this is our message with our questions and then parses the DNS message using the parser we parse the questions already in our function which we don't have to do I'm just doing it here you can also just return the parser without doing this and then you have to also parse the questions every time you are calling this outgoing DNS query so it's up to you whether you want to do that or not and then to test you can run the test and if your test succeeds if it passes and you can print the authorities for example and you get something then we have made a call to another DNS server so this will be a building block for when we are going to handle an incoming packet in our DNS server when we are going to run as a server because when we have requests for for example google.com we will have to first contact the root server the root server will tell us where the dot com top level domain server is then we can ask this server where google.com is then we can do a DNS query to the google.com DNS server and every time we are making DNS query we are going to use this outgoing DNS query function so this function is quite important to have this one working first and the next step is going to be to write our server let's now start writing our resolver so here I'm again in visual studio code we wrote our outgoing DNS query now we have to start a server and then write this handle packet so that we can run another test our test for test handle packet where we try to resolve google.com and amazon.com we are going to put these in a DNS message it's going to be the header it's going to be the questions so name type an a record we're going to ask we are going to pack this into a byte slice and then we're going to send this to a handle packet so we use a mock packet connection so because we are not really initiating a real connection our IP is going to be localhost and then we're just going to send our buffer which is this DNS question if this all goes well then our test works we might want to write some other tests though because this test is just going to test a handle packet you might also want to manually test a few things how do you test whether our DNS server works on mac or linux you can use dick dick with and then the name server that you want to use and then for example google.com nothing is running yet so it will not work with ns lookup on windows you can use ns lookup and you can for example say on resolve google.com on the name server one two seven zero zero one and the same on linux as well if you want to resolve google.com you can also do that on one two seven zero zero one so there are tools on your machine that you can use that are installed by default or that you can install for example dick you can install both on windows and on linux and on mac ns lookup should be default on windows ns lookup also has an interactive shell where you can just type requests it is something like ns lookup and then you just don't enter the host name and then the name service one two seven zero zero one and then you're gonna end up in a shell in windows there's also ns lookup on linux and on mac but i never really use it maybe it is even a bit compatible i often use dick or host so this is how you test in a shell but if you don't have any tools you can also use test handle packet which test this handle packet you could even write your own test here that is going to do an outgoing dns query on your own server that could also be a possibility so you could actually also write everything in go so these are our tests we don't have to really look at it straight away we can first start writing a little bit of code and then we can enter debugging for example to see how it goes or just what i also do is a lot of print lines to see what actual content is when we are then using the command line tools to do manual dns queries so let's get started with the servers starting the ns server and now we need to start it for real i'm going to use the net package listen packet the listen packet function listen packet announces on the local network address and then we can process the packets the network is udp and the address is just 53 so we're going to listen to all the IP addresses that we locally have and this is going to give us a packet connection and an error packet connection error what if there is an error if there's an error i'll just panic for now it's plain simple so that we also get the error message we want to defer packet connection close so at the end of the function we want to close the socket that we opened and then we're going to have a loop a for loop so we're going to read the bytes if there are bytes coming and if there are bytes coming we're going to put them in a buffer and then we're going to trigger a handle function this handle function this handle packet is local so we're going to create another one for and then let's make a new buffer of time bytes packets are never going to be bigger than 512 so we can put just 512 here to make it ourselves easy packet connection read read from read the packet from the connection copying the payload into p our payload is going to be the buffer and then it returns the number of bytes read the network address and an error so bytes read the address and an error what if there is an error we can just print something read error from the address and is the error address string this is going to return as address in the string and the error and then we just want to continue because then we can just continue to for loop if there is no error then what we can do is we can use this handle packet but if you would use this handle packet then we are not going to go back to our for loop because then we will be then we would be actually in that function so what we want to do is we want to use a go routine here so that we return back into our loop and then we will be blocking here we will be blocked here until there's not a client that has set us some data and if there's not a client that does some data then you will again execute this go routine in the background so we are going to create a go routine for every client that is connected and that sends us a package so for that we need another function handle packet and it is also in a package the DNS package so I need a capital H to make it accessible and a go routine is not going to return anything so handle packet go handle packet what are the parameters my packet connection the address and the bytes packet connection the address and the bytes the bytes are in the buffer how many bytes am I going to pass all five and 12 no I only want to pass the bytes that have read so I can say from the beginning to the bytes read go DNS handle packet it's going to be and then it's going to be importing my DNS starts less pkg DNS now this should be all the code I need in my main.go and now I can go to my resolver.go and then here I can say if handle packet which returns an error same parameters if the error is not equal to nil then I'm going to just print something so I have something on screen handle packet error maybe I will pass also the address so that I know and that's the going to be the error the address string and the error save this maybe I want here as well a return at the end handle packet handle packet so now this is the only code that we have to write outside our test all the other code can go straight into this handle packet let's have a look in this handle packet what do I receive the DNS packet how do I parse this I'm going to use the DNS message to parse this packet and then I can say p start of whatever is in my buffer and it's going to give us a header and an error and if you have an error I will just return it now that we have started parsing this DNS message I can extract one or more questions p has all questions which is a DNS message question slice or just question for simplicity now I will just assume that there's only going to be one question but we can also do all questions and then we can have another loop but I will keep it simple for now so we're going to say p question so the question is p question and also returns an error I think question an error if there's an error I will return it there and now we know that here we have a question now we can see what is in this question so it's going to be a host name a type and a class and that is our question that we have to solve what is the type the a record of a certain host so we're going to use a separate function for that we're going to do a DNS query and we're going to query a specific server with our question so we're going to have a question and we're going to have a server or servers what servers are we going to use well the first time we're going to run this is going to be the root servers but we still need to parse this so I will also make a separate function for that get root servers so we're going to do a DNS query function we still have to write and we're going to parse the server to be on the query and the question that we have once this is done we'll have a response maybe an error as well and that response we then can send back to the client so how do we send it back to the client if you have a response well let's first fix the error if there's an error we're going to return it let's say we already have our answer our response what's going to be the format of this it's probably going to be a dns message and then maybe just a whole message that's what we want to construct in our dns query and then we can easily send it back to our client so what is special about this is that we have to use the same id we have here this header with an id in that is randomly generated by a client so what we want to do is when we send the response we want to make sure that this id is the same so dns query let's try to write our function for this dns query get root servers we can reuse this it's going to be always an ip address we're going to have a question so this is the same we have a question that we're going to want to parse the question here is of type dns message question what are we going to return we want to return a full message dns message message so it's going to be a full message that we want to return so we can send it back to our client and then we're going to have also maybe an error so i'll just return nil nil so we don't have any compile errors then we have the response which is dns message so if there's an error we return the error get root servers maybe you should parse this first so this needs to be a net ip address so let me write this first get root servers maybe all the way at the bottom get root servers and we're going to apply a net ip slice root servers is strings split we'll split these root servers it's comma separated and then we can loop those root servers and then we just need to create a new ip address for every root server the net package has parse ip which takes a string root server is a string and then we need to put this in root servers you know i will call these root servers our net ip and then i will just move this here so our root servers are net ip slice and then we just need to append it so now we converted our strings that we split into a net ip return root servers so now we have a slice of net ip with ip addresses of our root servers so get root servers and then we have the response what are we going to do with this first we need to make sure that our response header id is the same as our header id because the client sent us a dns packet with a specific header id a dns header id and we need to when we send the response have the same id and now we could send this to the client our response is still in the dns message format so we need to do response pack and it will return the bytes so if you say these are new buffer do i need to make a new buffer for that response buffer error if there's an error we return and then we have this net packet connection that we can use pc write to we can write the response buffer where are we going to write it to to the address that we also passed here this is the client address and then we have the bytes returned i don't think we need these bytes returned we just need to make sure that if we have an error that we pass error and then return nil save this so then we just have to write our dns query function which is going to be very long and complex so to try to see if the code that we have written works let's try to return a simple dns message an error for example so our dns message is going to contain a header and remember this header also has in this dns message header field should i put returns probably yes we have this rcode our code is response code and what if we just send a failure our code server failure so we have a server failure then at least we are responding something back to the client so that when we have a packet that comes in we can see if it gets parsed if we get the questions and then we can do the dns query now that we also have the questions let's just also print out the questions that we have to verify whether we can actually see something question because we only pass in one question so now we can actually test this easily we could actually write a unit test of this handle packet with a little bit more simpler names we could do a test handle packet and see whether we get the right code back we could actually use this test i think because we are not really checking on errors we're just checking whether whether this handle packet doesn't return any errors and will it return errors no it will not return errors i think because we will just send something back to the client a server failure so let's maybe do the unit test first and then let's do it using the command line okay so this worked we have printed the question google.com type a amazon.com type a and it just passes because we are not really checking whether the packet that comes back contains the answer that we are looking for we are just simply checking whether we are not getting any error so that works now let's try to run our server starting the dns server i'm going to open a new shell and now we can use dick or nest lookup or host dick i'm going to use dick first so dick add one two seven zero zero one and then i'm going to try google.com and what did i get status serve fail so i did actually get a serve fail so my server is running i got a response because let's try actually first what happens if there's no server running you see nothing happens i cannot connect it will time out then i can run the server and then you will see immediately the question coming up because dick keeps on retrying dick keeps on retrying which can also be annoying at some point so you can also put it off there is something like tries tries so if you do dick plus tries is one there will only be one try tries one i'm carrying my local host google.com or xyz.com and then here we'll get google.com and xyz.com so our server works we are able to handle packets we are launching go routine for every packet we are parsing our question that works we are starting the dns query function with our root service with our question we are printing the question but we don't have any other code so we are sending a server failure and then it goes back here we fix the header id we package our response into binds and we send the bytes back to the client and then our client is actually getting a serve fail so how does it actually look with other clients so we would use host host google.com 127 001 you also get the serve fail so google not found serve fail let's try with ns lookup how does it work i haven't actually used it before yeah it's the same so ns lookup and then ns lookup you can use on windows i wonder where it is actually the same behavior so yeah if you use ns lookup dash 127 001 then you can just type uh your host name that you want and then make sure that the server that you're querying is then 127 001 so that all works you can use dick host ns lookup connect to our server or you can use the unit test here benefit here is that you can do debugging so i can say i want to know what this id is so i put a breakpoint i run debug test and then i have my debugging console so what is the id the id is this hexadecimal number how does my response look like it's a message with headers these are the headers there's no and there's no answer there's only this serve fail where is this serve fail the r code our code serve failure and then i can actually continue to see what happens and then it does the same for amazon because we have a loop here so we can actually continue here and then we are doing the same for amazon and then it stops and is my output and passes so multiple ways to test this which is going to be necessary because you want to test this all the time every time you write some code you want to see if it is working so next we are going to write this the ns query this will be the actual meet of our resolver this is where we are going to do our resolving for real before we continue let's have a look where we are so we did the listen to udp socket on port 53 we tested whether a client can connect we have a new packet that we then sent to the handle packet function we parsed the question we didn't send it to the root server yet that's going to be the next step we actually just returned an answer after the parsed question back to the client to test whether that works so what are we going to do now is we going to enter our for loop and our for loop is going to query the server first the root server we're going to check whether the answer is authoritative if yes then we can send the answer back if not then we're going to have to parse the authorities check whether there's an ip address he doesn't know and then create a new server and continue this loop to make sure that if i would write a mistake somewhere that we don't end up in a loop querying one of these root servers i'm just going to also limit the for loop to let's say three or five loops so if something goes wrong and we write the wrong code then at least we are not hitting one of these servers with dns packets as fast as we can so we are going to limit our for loop and that should be it so we just have to write what's in the for loop here and that's going to be that's going to be this code because the previous slide actually doesn't show everything here we have then whether our ip address is included so we're going to parse these additional resources where we can find potentially an ip address so it's a little bit more complex but should be doable it is surprisingly not a lot of lines of code because we can use this dns message package we don't really have to worry about the dns packet itself so let's get started let's clear our screen a little bit we will not need this for now dns query this should be a for loop can leave the print for some time then afterwards we can remove it for i is zero as long as it's less than three let's increase it and then we are in our for loop and the first thing we have to do is we have to do a dns query we have this outgoing dns query function that we have written so we're going to do outgoing dns query our servers our question and it's going to return the parser parser an error what is the parser it is actually the dns answer i will call it outgoing dns query and there's also the header so the outgoing dns query if i scroll down here we do a skip all questions so now i can actually parse immediately the answers so we can say first let's handle this error dns answer dot get all all answers these are all the answers these are the parsed answers we'll also have an error if there's an error in the parsing and let's maybe start with the end of our loop which will not happen at this point but the most easy part is if the header says that it is authoritative then we can actually just send back a message the dns message we have a header dns message header here we put the id so we only need to put that we now have a response then the answers what is the answer well we have the parsed answers here we can just say answers is a parsed answers is there an error no there's no error and actually my return error here is also wrong so here return nil error and here we return the error message so if our answer is that we get from the dns server is authoritative then we can just return the message with the answers and then we just make sure that id is correct here and that should work but this is only going to work at the end it's not going to work when we are just creating the root servers what are the root servers going to do the root servers are going to tell me if it is not authoritative it's going to have in this dns packet the authoritative name servers so we should parse those and those would also be in this arser dns answer authorities all authorities and this returns a resource authorities error if there's an error we exit so this is of type resource this is of type resource what is the resource resource has a header and a body so we will need to parse this body as well resource header the name a type class and then a body so name is going to be the actual name so let's output what is actually in there or let's use debugging to see what was actually in there so we know how to parse this so these are authorities and it's a slice so we will have to for sure loop them and actually what we can also do is just output them so we can use debugging or just output it what is in there maybe i will just output it authority but what if there are no authorities so just in the case that there are none let's write a check so if there are no authorities then it will be zero if it is zero then we can send a message back server failure what else we have refused not implemented format error name error it's probably not really a name error we will need to have a look at the dns specification what exactly we would have to return here but let's keep this for now you can always look it up later let's keep this for now so that if our authorities would be empty for some reason because we wrote something wrong then at least we know that we see this name error here let's try to run this go run cmd main.go and let's use dig tries one of google.com and just to make sure we are going to query three times so let me just break here so we don't create three times unconditionally terminated just for now and then we return the server failure code so we don't query three times so we get the serve fail but what we get here is more interesting so what do we get new outgoing dns query for google.com these are the root servers and what is in this authority header name com name server ttl and then the body length 20 but that's here these length is because in the dns packets it doesn't want to repeat always the same name so there's some optimization going on let's have a look what is in these bodies I think we can do it like that and we have the must new name and then we have the names of these name servers so agtld servers or not and so on bcdef so this we want to save in a variable so we have name name servers make what is it going to be a string the size land authorities I don't really like this go string here it doesn't really show exactly the name it shows how it looks in go so it's an ns resource so we just need to check exactly what the type is so if authority header what's in the header we have a type and the type is of dns message type if this is equal to type ns then we have an ns type and then we can for sure know that the authority body is of type dns message ns resource and when we know it's the ns resource then we have ns and then we can do a string this will give us the ns string which will be this this we can add to the name servers so now that we have sized it correct we can do it like this we don't need to append because it starts from zero and then we already right sized it right here so now we have all the name servers could check again if that's exactly what we have name servers yeah the other name servers but we don't have the IP addresses yet so we would have to now check whether we have these IP addresses and these are going to be in another place within the same dns packet as they're going to be in dns answer the additionals the additionals is going to give us these additionals an error if there's an error and these additionals we can loop again so just like with it here additionals additional it's a resource what is the type of this additional we don't know yet let's comment it out let's have a look what is in there so you can also just use debugging i'm just gonna print it what is in these additionals i can remove this do another query all right it says header the name and then we have type a this is ipv6 this ipv4 you're gonna keep it simple so let's use ipv4 and then the body is going to be the IP address so the additionals are going to have the IP addresses so if the type is what's the type here type a what can i do with this i can put this in a variable resolver servers which is going to be an IP address so this is type a and i just need to use append now because i don't know how many they're going to be but i probably shouldn't shouldn't just add all these IP addresses because who knows what can be in there so let's compare them with the authority and only when the name matches with an authority then we're gonna add it authorities if authority authorities no name servers we can loop over the name servers because that's already a string so only when the additional header name i have it from here header name string is equal to the name server then let's append to these resolver servers the IP address and how do we know the IP address so we know that this is type a already so we have the additional additional has a body the body only has this go string so here we use ns resource and here it will be the type dns message a resource a resource because the type is a you can also just print the go print output and it will also tell you that a resource and then it has an a record and then it can return the bytes so here we have the bytes and that we should be able to just add to the IP because IP is also bytes now that we have all these ifs we also want to make sure that we know whether we have found IP addresses we can say new resolver servers found it's false and here we can say true so that at least we found one and then if you didn't find any then we can do something but if we found IP addresses then we can actually do a query by iterating our loop again and then we just need to change the servers question can stay the same we can just change the servers now the service is the service from here which is the root servers now we can actually say the servers is the new resolver servers and maybe I should just have called these servers servers then I don't need this line because then we are just changing this servers so servers we make it empty and then this needs to be servers and this needs to be servers so if we do the iteration then we say servers is going to be these new servers if you don't have any new servers then we're gonna break for now and then we're gonna end up here because we haven't implemented this yet if we have found new servers then we'll just go to the next iteration of the loop so let's try to test that so let's click clear our screens maybe remove any prints yeah there's no prints anymore let's see if that works go run and what do we have starting DNS server the first question is for google.com google.com there's a new outgoing DNS query to the root servers the root servers returned these new DNS records we assign this to the servers we do a new iteration the new outgoing query is still for google.com but now we have different servers and we keep on iterating on this until we have found an authoritative server and then once we find it we return the message and then actually we got the message there's no error we got google.com and the ip addresses so maybe it is a bit unclear so let's run the resolver test in debug mode so we can have a better look at how this works so the first outgoing query we have an answer so this is going to be the answer of the root servers and this is the message let's have a look at this answer it's not authoritative so we are getting all the authorities and these are these are these names so let's continue a bit so now we are figuring out what the namesores are gtld so we're gonna fill this name servers and then we're gonna do the same for ip addresses so let's continue here to the next loop oh actually i see what i did wrong i should put my breaking point here so let's restart actually let's just maybe remove amazon.com just to make sure that i'm not accidentally getting in that one so we'll stop debug so how many questions do we have before we get the answer one two three and these are the servers root servers gtld servers and then the google name servers these are the google name servers and then at this last iteration we actually get the authoritative answer so that's the third one and then here we pass the answer and here we are authoritative so the answers are now coming from these servers which are the name servers so first the root servers gtld servers so the com top level main servers and then the google servers and the google servers are authoritative and then we are responding so this works but what is not going to work if we don't have any resolver server found if you say resolver server not found and let's uncomment amazon.com and amazon currently doesn't have any ip addresses for their name server because why don't they have that they have another domain name you see so it's not amazon.com the main name and when you ask the com.com servers they will only give you these host names so if you run our test again resolver server not found for amazon so we don't get a reply we get these host names right here but we don't get the ip addresses so we are here we get these additionals there's nothing in them the new resolver found is still false so we are now here so for google.com we actually have it working but for amazon.com it doesn't work because we didn't complete this flow so remember the slides i have two flows one that we check whether we have the ip address and one that we don't check whether we have the ip address google is one server because we have here google.com and it's one google.com and we cannot look that up so we have the ip addresses in the additionals for amazon.com we can actually do a lookup of these host names to figure out what the a record is and then reply the a records and then use the a record as a server but that requires a little bit more code right here so let's try to add the necessary code right here you can actually write this code in two ways you can say i have these name server records here and i'm going to loop over them for name server range name servers and i'm going to say i'm going to find all the ip addresses of the name server this might be a little bit inefficient because we only need one working name server so if you have one ip address we might have enough now we might have an ip address it doesn't work so then we would be in trouble so if you have one ip address and just hope that ip address works and return that ip address back in our loop here as a server or we can get all ip addresses so it's up to you whether you want to write it efficient or not if you want to write it in a way that you get all ip addresses it is actually possible that if you would write a caching resolver that these ip addresses are cached anyways in your resolver so that the latency that you get by resolving all these name servers is not completely crazy but yeah you can already see one two three four five six are we going to look up six host names to figure out all these ip addresses whereas maybe one name server would be enough it's up to you i might just actually look up one if i find an a record of one i will just return that so what i'm going to say is i'm going to use again this if here if new resolver is not found then i'm going to continue in this loop and once i find one i will just put this to true and what i'm going to do now is that i actually need to restart this whole loop again i need to run this dns query just like i run it here this dns query because if i just run one query that's not going to be enough i need to run multiple outgoing queries because we have to start again with the root servers the root server is going to say a net servers for net are going to be here then you're going to have a name source of this direct net or this ultra dns net and then you're going to have the a record so i'm going to have to recursively go into this function again to find the ip address of this hostname here so dns query gives us a message respond an error and i need to get the root servers again because i need to query the root servers and then i can pass a question so dns message question the name is going to be the name server but actually the name is of type dns message message so there is a must new name that can convert a string into a name type so must new name name server the type is going to be the a type type a because i'm looking for an a record and the class is going to be the innet class if there's an error if there's an error i'm just going to print something a warning lookup of name server failed and an error and if there's no error then i'm going to do something else so that means that if there's an error we will just continue the loop until we find a name server that works if we find a name server that works and we have an a record we can put this to true the response will then have the answers and i will have to loop those because it's a slice so for answer range answers so this is a message actually so dns query it's a message we don't have the parser so we don't need to parse these things because it is actually the message that is being returned from hopefully here if we are authoritative after we went through the loop a couple of times you're gonna have these answers here so for answer what do we need from the answer this is going to be our IP address could be multiple IP addresses if answer header type is type a is type a i get a warning here i didn't add error if it is type a what do we do then we can say servers which is now empty and servers and our answer body should contain a type a so it's an a resource just like here an a resource and this should be the IP address if this doesn't work we will see but normally we are testing dns message type a so this is for sure going to be an a resource we return ip we append this to servers and then we can do the loop again i don't think i am missing anything there's still this break here do we still need this break no we don't need this break anymore although before we are going to do all these crazy things maybe we should just say ip's found to see whether it's really working ip's found servers and then we do a break what about it i can live with that so let's try to do that so google.com that works because we didn't really go in that loop amazon.com ip's found okay new outgoing dns query for pdns one ultra dns so we found i hope the correct ip address for ultra dns yes that seems to match let's remove this breaking point and let's hope for the best and we get an answer amazon.com and the a record is right here yeah that seems to match they're just in a different order so this was our last step if there is no a record provided then we need to resolve it by running our lina square again and then after coming back here using the new servers we have an authoritative answer and then we return the correct message so this seems to work let's to be sure also run our unit test and that also seems to work so how many lines of code do we have less than 200 okay between also have the main function but our library less than 200 lines and we have written a dns resolver that is actually working for simple queries it definitely could use some improvements so more checks so more tests we can for example also test whether we can run mx records so host t mx google.com is handled by smtp google.com and now if we ask our own server using domain server the local host one then we also get mail is handled by smtp google.com so that seems to be working you can do the same with dig you just have to do dig minus type i think type equals mx so within 200 lines you can write yourself a dns server in go using only this xr package that is still from goline.org so it's actually maintained as well it doesn't have the same promises as these built-in packages but these are pretty stable and pretty high quality because you can see that they are very efficient in parsing these dns packets so you can play a little bit with it to see if it works for all cases or that it works for a domain name that you own but for simple queries it seems to be working so our resolver can answer simple queries it doesn't cache anything yet so you see it does for every query a lot of dns queries itself so that means that it wouldn't be very useful to use this as a real resolver you would still have to write a caching layer that takes into account the TTL as well to cache the host names i will make these files available on my github so that you can have a look at the solutions and hopefully also try to build it yourself and improve on it