 All right. Hello, everybody. We're here to talk to you today about our journey through getting the open tofu registry open running It's been a long process and hopefully we'll take you through all of the steps But just to get things started first My name is James Humphreys. I'm a software engineer at SpaceLift and I'm an open tofu core engineer I've been around in the open tofu project since day number one and And I'm Errol. I've also been an open tofu since day number one and I'm a principal engineer named zero So let's get started. We have a lot to go through First of all, I'll give you a bit of background on the start of open tofu We start off as a fork of terraform some pre 1.6.0 version of it and we had a bunch of stuff we had to get going We had to stabilize the work in progress features like the testing feature That was new at the time we had to fix some bugs and crashes that were in the testing feature mainly during that time and remove a lot of Potential trade trademark violations of terraform for obvious reasons Well, that should have been very straightforward At least that's what we thought because later we found out that terraform registry has changed its terms of service Now it can all be used with or in support of HashiCorp terraform and we are not HashiCorp terraform So we're in a bit of a pickle. So we had to create our own registry Now most of you here probably know what a terraform or open tofu registry is if you're using terraform or open tofu But I'll give you a quick round-dome anyway. It's basically a catalog of providers and modules providers and modules are the main building blocks of terraform and TF configuration in general and Terraform and open tofu can't function without that They need providers and modules for every TF configuration you have you're probably if you're using AWS Whatever you're using the registry. So that's a very important part of terraform and open tofu When you run terraform in it or open tofu in it That's when you contact and communicate with the registry and you download the providers and modules that you need And if you've ever had issues with your providers or modules You probably know that the implementations are in public GitHub repositories and for every new version. There's a release That's how they manage versions for providers and modules Now for how the registry works behind the scenes There's an API called the V1 API that it implements it has four endpoints two for listing versions for modules and providers and Two for getting download information For a specific version of a provider or module. So when terraform CLI or open tofu CLI Decides that it needs a specific provider or module it first of all asks what are the available versions and gets back the available versions and When it decides, okay, this is the specific version that I need of the provider or module It asks for download information of that provider or module and gets back a download URL from the registry So that's basically a quick round off how it would work. So now we'll talk about the alpha registry yep, so We call it the alpha registry because it was what we wanted to really get started with and we knew we were going to throw part of this away so We realized really quickly that we had a couple of requirements for this alpha registry to get things going so we first off We wanted to get started immediately. We realized we had to move quickly We wanted to be making sure that we can get a release of tofu hour and into the hands of people because people really wanted to be Moving over and testing it so to get us going. We decided to take a two-pronged approach We started off with getting the alpha registry set up first and this was a quick and hacky solution Which I'm going to talk you through and you'll realize how quick and hacky it was And then alongside that we wanted to take all of our learnings from the alpha registry Take that work with the community and build our stable registry, which is what we're using today So we had a couple of requirements a drop-in replacement is what we were aiming for and what we kept saying throughout open tofu and Hopefully you'll see that we achieved this But we wanted people to come along and just run tofu and it on their existing projects We didn't want people to have to go and make code changes We didn't want people to have to worry about how do I migrate people should be able to come along and use tofu out of the box with no problem Our next Requirement sorry was they had to handle the load of the open tofu usage We know that for every single provider every single run that's going on There's a lot of HTTP requests and we have to handle all of those So we want to make sure that our alpha registry isn't going to fall over Because if we go and do a fork and release and no one can use it. It's no good But we wanted to make sure that it was there and ready for everyone to use So to go into detail about the alpha registry a little bit We dove in deep with AWS because it's what we know and what we what we use And we try to keep it as simple as possible because we our focus was to move fast So we set up one AWS Lambda which handled all of our API requests These are the four requests that arrow was talking about earlier and Then we threw an API gateway in front of it We preemptively tried to cash as much as we could because we realized a lot of people are requesting the same data A lot of people love the AWS provider But not many people are using it to manage their Spotify playlist or anything like this So this was working really well. We were handling a lot of requests very cheap and very quickly and Once we got through all of this We hit an issue We were talking over to GitHub every single time someone made one of these API requests and we were quickly getting Relimited 5,000 requests is what GitHub gives us and that was not enough for the load that we were doing We were quickly falling over and we was taken down the registry a lot So if everything was working until it didn't and so we panicked and we added in caching We threw in an API gateway between ourselves our lambda and the GitHub repository and what this allowed us to do was cash every request that we're going to get hope We're doing a lot of the same requests and we wanted to make sure that we're not wasting our API Rate limit because that's a very precious resource And that was working really well And this is a common theme in this section of the Alpha registry where we'd fix one thing and another thing would pop up And we're just moving bottlenecks all over the place We were hitting API lambda timeouts people were requesting the API provider We were trying to talk to GitHub and fetch information about every one of its 232 releases and GitHub doesn't like having a fast API. So what did we do? We added in more caching We implemented a document DB store where we would every time somebody makes a request We try and serve that information as quickly as possible and store it away in a document DB We realized that we can dump this data and serve it back up for another hour We've implemented our own version of caching and we're simplifying things and we also Introduced another lambda here, which may seem a little bit confusing, but we were asynchronously Doing updates at any time a user came in and requested a version of our provider We'd also tell our other lambda. Hey, go and update document DB if we need to and this meant sure that made sure That what we were doing was great But as you've all probably preempted caching is difficult and caching validation is one of the biggest problems in programming as always So we were hitting problems where github dpi would fail for one of our few requests And all of a sudden a version of the AWS provider wasn't available a version of the cloud posse Deep merge module wasn't available and all of a sudden people couldn't use tofu a hundred percent of the time It was down for five seconds every hour and I wasn't very happy with that. We weren't very happy with that so It was good enough it got us going it got us working with an alpha release I believe once we got through and we were releasing to alpha We got to a point where we were hitting maybe one issue a day for 30 seconds Not the time that I would like but it was good enough and we got it working really quickly But we took away some of these learnings. We made sure that we knew we had to keep it simple We don't want to keep adding lambdas. We don't want to keep adding caching in different places We didn't want to keep adding to the situation. We wanted to try and simplify our process And then we also realized that we had to make it make it work along with how open tofu work We've inherited this code. We've inherited how it will talks to the registry and we want to make sure that we're doing things in the right kind of way Low maintenance Aral and I and other open to tofu core members were spending all of our times looking in AWS x-ray looking in cloud Cloud watch trying to figure out why a certain request was failing and we didn't want to do this anymore It's not that we don't love debugging and firefighting we do But it's something that we can't keep doing and especially when we're an open-source community We are full-time engineers working on the open tofu project, but our time is really valuable We've got a website to run. We've got a CLI to add features to and we've got to prioritize our time So this is where we came into picking a stable design Alongside all of this work on the alpha registry that we were doing We were trying to make sure that we had a stable design ready for our implementation. This was our v2 We wanted to take that alpha and throw it away So we took everything that we learned we went out to the community and started asking for RFCs request for comments and This was a great process. We went out to our slack community Which you should all join off the link at the end And we went out to everyone in general saying hey, we would like to talk to the community about how to best do this design We're all smart people But we're aware that there's thousands of smart people in the community as well And we will be silly to ignore any solutions that they come to us with So At the end of this process we decided that we should let the technical steering committee decide Now the technical steering committee is something you might hear thrown around with open tofu It's a board of some of the founding members of the open tofu project So from each of the people who are submitting a full-time engineers a lot of the projects who are involved and we have board of members who make decisions so We put it out we put out requests myself and one of my co-workers Cuba ended up putting in two different requests here We ended up getting an external contribution from a existing registry called terror reg And we were also talking to another member of the CNCF called Artifact hub to try and figure out if we could piggyback on some of their existing Infrastructure and perm providers and for this they offer a helm charts so When we got to the end of the day We ended up deciding on the technical steering committee ended up deciding as well on working with a homebrew like registry design And this was king and just to talk you through a little bit of background on homebrew and what it is I'll pass over to our Yeah, so I'll talk a bit first of all about the inspiration I had for this issue this RC Homebrew first of all is anyone here using macOS? Do you know homebrew? Yeah, okay, not enough hands, but good enough Yeah, so I'll give a quick rundown of what homebrew is It's basically a package manager for macOS. It's a very popular one You can use it to download a pretty much any package or binary you would like for so for example If you'd like to use open tofu, you can just use brew install open tofu and it will download and install open tofu for you Yeah, that's it. That's homebrew now How does it work behind the scenes? Well homebrew pretty much has its own registry kind of it has the homebrew core public repository That has all of the formulas there You can see the formula folder there if we click on that we will see a bunch of folders Mostly single letters which are the first letters of the binary you would like to use for in our case It's oh and you'll find open tofu RB there now one point regarding these folders We talked to the homebrew project lady actually commented on the issue for the RFC itself and told us why they did that It's mainly for better get performance in case of projects with a huge get rid of history So that was very helpful when they talked to us about that. We actually did the same thing and you'll see that in a minute so Let's take a look at what's inside the open tofu RB file It's mainly just metadata that you need in order to download the open through binary It doesn't have it contained the binary itself. It just redirects There's a URL for the artifact in the github releases. So that's all there is there Now homebrew only manages the latest version of Binary so what happens when there's a new version coming in well, you can bump it up You run a brew bump formula PR command with you provide the tag and the binary and it creates a github PR for you Changing the Ruby file according to the new information you check sums and everything that needs to change Yeah, so that's it regarding homebrew what I really liked about this is the clarity and open source nature of it Everyone can see when a formula was added and when it was changed and why VFRs, which is what the entire community knows How to look at so we really like that and adding a new A new formula would be via PR as well people would be it would be pretty easy for the community to add new formulas We did want to streamline that a bit more. We will talk about that in a second and Something we could not do ourselves will have this brew bump formula PR Version for providers or modules. There are a ton of versions all the time We have a bunch of providers and modules We couldn't expect the community or people to run these commands whenever there's a new version coming up So we couldn't do that So now let's talk about the actual stable registry design that we ended up with It had four parts first of all it had Repository folder structure pretty similar to what we had in homebrew we had Jason files containing metadata for the provider and module and versions and Each version you have metadata that I'll talk about in a second and We had to run a one-off hydration To have all of the existing providers and modules there We use github search api and google big query for that. You can take a look at the PRs we had with the hydration to see to have more info on how we did that and Okay, now that we have the jason files. We had to get new versions. So we had to periodically update The existing jason files with new versions After that the v1 api we ended up not having a v1 api we ended up mimicking it with hosting static files You'll see that again in one minute And for adding a new provider or module all you had to do Was create a PR but we decided to simplify that a bit more So for the folder structure, you can see it's pretty similar to homebrew We have the modules and providers folders here if you'd like to go to integration slash github provider For example, you have to go to providers and then I Integrations and then you find github.json there and the github.json file has all of the metadata You would need it contains all of the versions and for each version All supported os's and architectures and the checksums for each release for each of those So again like homebrew it doesn't contain the binaries themselves It contains download URLs and all of the necessary data To get those binaries. They are mainly hosted on github Release artifacts Now for version bumping We decided to go with scheduled github actions What we liked about that is this is very visible to the community the community could take a look and see Okay, the new version is not here yet. It's going to be there in 10 minutes, for example Or they can actually see if a version bump process failed and notify us about that So that's something we really liked about that the open source nature of it We use some heuristics to not hit the github api rate limit that you saw earlier in the graphs And I'll talk about that soon and that's what that's what helped us scrape around 30 000 repositories every hour Which we do right now every hour Yeah, so that's how we did that for modules instead of relangle releases We could just rely on the tags so we ran git commands specifically git ls remote And we get all of the tags for all of the modules and if there's a new one We rebuild the json file with the new versions now for providers We did have to rely on releases so We went to GitHub releases rss feed and tried to check for each provider if there's a new version and only if there was one Then we called github api to get the release And rebuild the json file. So that's how we went from A couple thousands requests each hour to just a few dozen Yeah, so that's how we periodically build the json files So on the back of that one, please don't tell github that we're circumventing their api request the re-limits We have to try and We had to go and implement those four different api v1 calls that we had Described at the start of the registry So we're fetching the module and the provider versions and then fetching a download link for each of those individual ones so We wanted to make sure that all of the data we're storing going back to what we were talking about as a requirement It was simple. So we realized that we can serve all of this data statically If we can take all of the metadata that we're storing in github and turn it into these json files that are all describing We can serve everything statically 100 from cloudflare r2 that cloudflare r2. We're really thankful for There are a business sponsor of ours and so they've given us everything but you can completely take our registry We're working with it right now and it sits 100 in the free tier And we're currently handling like hundreds of gigabytes of data coming through this So we're really thankful for cloudflare r2, but we made sure that everything was static. We have no compute going on here We're really simplifying everything down to just serving files So for example, if you want to go and fetch the integrations github that our all mentioned earlier You can go and request this and we're not quite sure if this is a url or a folder path It's because they both match What we're doing is we're we're creating a directory structure that matches the api that we're doing We're mimicking an actual api, but what we're really doing is we're just serving up static data We have no compute no complexity. We just leave everything up to cloudflare to serve a bucket So for each of these modules or providers we'll create a set of files We'll create one versions file and this maintains like a list of all of the versions operating systems and architectures that we support for that provider or module And then for each of those versions we'll create individual download links And what this means is we've got a lot of files being generated from the metadata that we're going through We we've got what did you say almost 30,000 providers and modules? So for each of those they can have hundreds of releases. We're creating a lot of files here Um, and then we hosted all in r2 r2 is again free egress like go and use them. They're great Don't have to pay as much as aws want to charge you Um, so as you can see here, we have our two github actions at the top One is to go and generate all of that data. This takes about 10 minutes to go and we're running this hourly Uh, and that generates all of these json files and stores them And then we take another 10 minutes to go and sync it all up to r2. We're talking Thousands and thousands and thousands of files So we have to make sure that we're specifically only uploading those that change because otherwise I think our initial hydration was like two hours to go and get it all up there It costs a bit of money to go and upload it and we don't want to be hitting that every time So we're trying to be smart about what we're uploading Um, if you want to go and add a new module or provider we hydrated our registry initially with 30,000 different items roughly Uh, but there are new modules and providers being created all the time And we can't keep scanning github because they're gonna hate us Uh, so uh, if you want to go and add your own module or provider to our registry It's really simple. You just need to go and create a pull request to go and add your uh to the registry So to do this you can go to our registry. We've automated everything for you Go to our registry repository and you can create a new issue There'll be one of two templates there a provider or a module There's also some other templates if you've got issues or anything you want to raise But these are the main two that people are going to be using And we ask you to fill in a super complicated form of two fields We just want to look at your github repository and where it is If you want to add a provider to the repository that you're not an author of that's perfectly fine If it's public and on github, we're gonna go bring it into our registry so you can use it Um, what this means is there's a low barrier of entry coming along and adding your stuff to open tofu is a much much simplified process If you want to uh, once once you've added that and you created that issue We have a bunch of automation that happens under the hood We'll go and check that this provider has a bunch of releases. It is an actual terraform provider repository Um, it's got releases that are valid. It's not Trying to be malicious by swapping some gpg keys around and things like this We're trying to do our best that we can to validate these providers before they come into our registry But this is all automated takes about five seconds to run anytime. So one creates the issue The only thing you're going to be gated by is that right now is a human merging those prs We for security reasons, we don't want people misusing the registry They still requires a human to hit that merge button at the end Uh, but we're getting dozens a day dozens a week. Sorry and we can handle this load perfectly fine for now So where are we today? We have been really great in uh, everything that we've designed and it's really paid off So we're serving hundreds of gigabytes of data And we're hitting uh cloudflare's cash it at like 99.9 percent We are handling all of the load perfectly fine. We've had people load testers and Get blocked by cloudflare and if cloudflare is going to block you before we do then i'm really happy with that We've had dozens of contributions We've had people come in and add their module to the registry We've had people go to their providers and say, hey, can you add to the open TOEFL registry? And it's there in five minutes ten minutes because it's super quick to do We we're successfully a drop-in replacement We are super happy today to say that you can come along take your existing terraform code and just run TOEFL in it And it works perfectly fine. You are not going to be hitting the hashicorp registry You are not going to be uh abusing their terms of service and they can't chase someone down I'm not a lawyer. Please don't come after me. I'll talk to you about this. Um, but the main thing is this is a drop-in replacement Uh, and then uh, yeah, so migration is super easy Just come along take your existing code. You know code changes needed just run TOEFL in it So takeaways from all this stable registry and where we're at Community driven design. This was super important to go and get all the communities input As I all mentioned earlier, we have the homebrew Call contributor Get in touch with us and give us some advice about how to host things in github We had multiple people from across the existing ecosystem of providers provider Registries that are already existing for private and they all came to us and started talking about how we can do things And we've ended up with a great design. We really can't go wrong Maximum visibility if you want to go and say, hey, where's my provider? You can go and check if it's there in github You can go and see if the file's been added when it was added when the versions were updated When it was synced up to our registry there's full visibility and this helps people debug all of their issues So this day with we're very rarely having issues I think we're a total of three and the biggest impact was that something was an hour late So I'm really happy with that Um maintenance maintenance cost has been zero We have had this running now for a few months and we have had zero issues Like we say just one issue where someone's provider was an hour late We've put no development time into this and this is really important because as we're a small team of an open source community We have to make sure that our developer time is put into things like adding features to open tofu that you guys want state encryption and stuff is coming soon We've got a website to run We've got a lot of pull requests and issues to come in and we've got a triage them and we've got to say that our time is really valuable And simplicity is key. This is the fun part all of this design all of our alpha registry learnings This meant that we could go and make this registry in a week We wrote some simple code is I think it's under a thousand lines now We wrote it in a week. It's perfectly visible and it just works and we're really happy So what's next for the registry? We are looking at adding a user interface for our registry right now We don't have one in hashi club does and we don't like that So we want to make sure that people can go and read the provider documentation Their module documentation through a user interface and we're looking to go through that same design process It was successful before and we're hoping it's going to be successful again So we're looking forward to the community giving us some contributions about how we can do a registry user interface We've already had a couple community members who are working on like proof of concepts Uh, and we're really happy to go and see all this unfold So, uh, if you do have anything you want to go and see you can go and look up our uh repository open tofu slash registry racism issues have a poke around or submit an rsc for how you want the design to be So, uh The cncf is asked us to add a qr code here if you want to go give us feedback You can scan this uh, and we'd be amiss if we didn't mention that you can join our open tofu slack Over here with this qr code as well. We've got over 1500 members. It's a very active community and we'd love to have you all there Uh, thank you Thank you Do we have time for questions or Yes Any questions anybody? Yeah Okay, so, um, does it support right now module signing verification of modules the registry or is there any plan to support that? so Right now there isn't a mechanism to go and check the signatures of modules because modules under the hood Work as just cloning a git repo. And so what we're doing is we're trusting the git repository, uh, to work under the hood I believe there's an issue open. If not, I'll go and open one right now and we can go and discuss it Uh, but we are currently supporting Signing for providers. So if you are already signing your providers with a gpg key Uh, in the same process that I mentioned about adding a provider and a module There's also a form for uploading a gp a public gpg key Uh, and it's all automated and works perfectly fine If your module provider is either missing from the registry or the gpg key is missing and someone runs tofu in it There's a nice little help message that says you can go and submit this here and uh, hopefully that pushes people to go and do this Because we're looking forward to the the world where everyone's uploaded all of their keys and everything's super secure There's no man in the middle of tax or anything like this. Okay, great. Thank you. Thank you Any other questions, I think there was another one Good morning guys, thank you for all the work by the way I'd really plans to support anything other than github for example github Are they the enemy? Yeah, not right now. I didn't see any issues regarding that but For now, no, I think you can raise an issue for that if you'd like on the registry But I think it's pretty standard right now at least 30,000 Uh repositories are on github right now. Um Yeah, I don't know we we can take a look at that if there's a need for that We've discussed it and we can do it It's just how she called decided that everything had to be on github github is definitely not the enemy at all Yeah, we've been talking to a couple of github guys are great Is the implementation compatible with other github providers as long as they support the htp? Yeah, it is we just need to uh Work have different api requests that that's it the registry itself still will be on github But we can request any data from wherever we want. So yeah, it should work fine. Thank you guys We have time for one or two more questions One of the hills first, I think yeah right at the end Thank you. So first of all, thanks for explaining how it works underneath also knowledge Uh one question is regarding the how the Open tofu binary and this registry is coupled because if I understand it correctly It is the same like in the terraform. So it is tightly coupled and there's no way to Host or just create your own registry and point open tofu to your own registry. Are there any plans or Decisions regarding that can go ahead. Yeah, so this is a really fun one Because I think other people want to be running the registry I think pepe who spoke earlier raised an issue about running the alpha registry We want people to be able to run mirrors of our registry We want people to be able to take this data and host it themselves somewhere if they want to The fun part behind that is that we've inherited a large code base and that large code pace was Maintained by hashi korp who hardcoded registry terraform.io everywhere into their code base We have an issue open about making a dynamic registry So we're hoping that you may be able to set either a piece of configuration or an environment variable Where you can say this is where my registry is the default registry and it fetches all of the providers and modules from there And that would be great for going forward There's a lot of people who like to run providers provider registries internally or privately and we're hoping to target that Just one point about that that was just for a default registry right now If for every provider you define you just write your own registry path You can use a private registry if you'd like and we aim for you to have you could have our implementation For that or terra rag or anything else. So It is possible right now Exactly what I wanted to know. Thank you All right one more question Now I think most of it is based on the terraform registry api like are you planning to Diverge a lot in any near future Uh, it's a really well documented api And what we would like to say is that we we want to try and keep compatibility with terraform as much as possible as well Um Whilst the v1 api is great for downloading modules and providers. We are going to have to diverge. So hashi corp has a v2 api which powers their documentation engine their front end And we're going to be looking at implementing our own version of that when we're looking at the registry ui I think so We're trying our best not to diverge. We don't I'll provide a community is really important to us and Making it easy to move from terraform to hashi corp is really important I'm making sure that people can use private registries on top of this really well documented v1 api is really important So we're trying not to diverge where possible. We're trying to make sure everything's a drop-in replacement as I've said a few times Uh, hopefully people get the very liminal message Uh Yeah, we're trying to keep it as close as possible Where we can there is there has been one small change with the registry and how we handle modules But where possible we're trying to keep it the same Yeah, and then as the false said at atlantic said it's really hard to know who's using what in the community. So Uh, we have no idea if there are third-party tools using that api So we we're trying to be a bit cautious about that before we make any big breaking changes Mm-hmm And uh If anyone wants any if anyone wants any open tofu swag, uh, you can go find some Space lift h24 is on the the things in front of you or I think m0 and scaler also have swag around Go get open tofu t-shirt stickers and treat yourself. Thank you. Thank you very much