 All right, hello, everyone. Welcome to our talk on Fernet tokens. Right now, the only thing standing between you and lunch today, so it should be a good talk. We want to do this talk because Fernet tokens have been around for a while, and people have been successfully ignoring them, especially operators. But we're here to tell you the day of Fernet tokens is coming. It's going to become default. And so when you watch this talk today, you need to plan for how you're going to switch to them or how new clouds you build are going to use them. Just a brief intro and what we're going to cover today, starting with the basics. What is a token? Then we'll dive into why was Fernet created, why do we need to do format. We'll cover the importance of non-persistence in your Keystone tokens. Then we'll cover rotation, key rotation in two scenarios, just on a single system and also in cluster. And then we'll dive into some operational bits about Fernet tokens, starting with just a brief overview of OpenSec at Time Warner Cable, the prerequisites we needed when we switched to Fernet tokens, the migration path to Fernet tokens, and then just some things we picked up while operating for Net Tokens. With me here today, who might be able to introduce earlier, is Lance Bragstad from Rackspace and Dolph Matthews from Rackspace. And Dolph's going to start by diving into some details on Fernet tokens. Awesome. So what does a token start there? So if you interact with OpenStack in any means whatsoever, if you use Horizon, if you use OpenStack Client, you use Keystone Client, Nova Client, anything, you're always going to start by authenticating with Keystone. Keystone's going to give you back a token. And then the client or yourself or Curl or whatever, every interaction with OpenStack beyond that point is going to involve that token. And it kind of looks like this. This would be an example request to Nova. So why do we need another token format? Let's start with some history. We can go back through kind of how we got to Fernet. And hopefully that'll help explain. So we started with UUID Tokens way back in Diablo. UUID Tokens are basically just big random numbers. We represent them as strings. And Keystone writes them to a token table generally. They have to be persisted somewhere. Basically the way they work is when you hit Nova with your request, Nova calls back to Keystone over HTTP and says, hey, Keystone, is this a valid token or not? Keystone goes, looks up in the database, is this a valid token? And if so, Keystone remembers the identity and authorization context that belonged to that token, returns that to Nova, and your call proceeds. So PKI Tokens. We perceived that online validation process had to not be scalable. And so we implemented PKI Tokens. PKI Tokens are gigantic signed responses that Keystone would otherwise be returning to Nova. But instead, we are basically signing it using a PKI cert, base 64 encoding the whole thing, and passing them around over HTTP and headers. That turned out to be a problem. This is about as small as PKI Tokens get. This one has a very minimal service catalog in it. And we found that as soon as you start adding a lot of services to your OpenSec deployment, these get too large for HTTP implementations. So you start having to recompile Apache and stuff as these blow past 8K. They get really, really big very, very quickly. So we tried to address that. We implemented PKI Z Tokens. As you can tell, this one's way smaller. So they're almost exactly the same as PKI Tokens, except we introduced a layer of compression in the pipeline. And so they actually are a little bit smaller if you have a very large catalog. Catalogs are pretty compressible, and so it kind of works out. But these are still far too big to be practical. You can't copy and paste them around in your CLI because it'll flip past your screen, things like that. And so that brings us to Frenet. We knew we wanted something small. PKI was simply too big. But we're also pursuing non-persistence. Non-persistence is a super valuable feature of Frenet that I'll get into in a second. We also had another problem with PKI that's just kind of a hurdle to making it the default token format in Keystone. And it requires a little bit of setup out of the box. And really, the setup that Keystone offers is not really production grade. And so we kind of fixed that with Frenet. Frenet does require some setup, which we will also get into. But you can use it in production. It's ready to go. And like UUID tokens and unlike PKI tokens, Frenet tokens must be validated online. So we kind of gave up on that front. This is a Frenet token. It's also base 64 encoded. But unlike PKI tokens, there's not a gigantic JSON blob in there. So we'll dive into the details of that in a minute. So what does non-persistence matter? Again, this is really important. So if you've operated Keystone at scale before, or even small DevStack cluster for a long period of time, you've probably learned very quickly that your token table gets very, very large. And Keystone grows slower and slower and slower. And so for a long time in Keystone's history, it was just sort of like a documented thing that, hey, you need to go purge your token table periodically. Eventually, we implemented a command called token flush to actually help you do that. And you can run it on a cron job or whatever. So if you have 24 hours worth of tokens in your token table, it's still big, still kind of slow. And if you want a longer expiration window than your token table, it just gets bigger and bigger and bigger. So if you're actually using Frenet, then your token table is empty. And we're not just hiding Frenet tokens in memcache or persisting them somewhere else. They actually do not need to be persisted at all. So when you authenticate with Keystone, there's actually no write operations anywhere. We just generate a token and send it back to the user. And so the impact of that, when you authenticate, again, there is no write to a back end. So we can authenticate very quickly. It's just a few read operations, and the token goes back out and you're done. As a result of not doing all those writes, and if you're using UID or PKI, most of the write operations to your database are going to be token creates. And so if you're doing large multi-region deployments where you want all of your tokens to work on all of your other data centers, that's a lot of data to be shuffling around with Glare or whatever. So also because we're not persisting them, there's actually no replication lag. So one of the use cases we're trying to tackle was we have data centers in, say, Sydney and Washington DC. And we actually have race conditions where you can authenticate and say Sydney and get a token back and immediately try to go use it in Washington DC. We're actually like racing our replication process, and you might beat it. In which case you get a four one, and you're like, well, I just authenticated. Wait a couple seconds or a minute or whatever till that replication catches up and then your token works. And that's not a great user experience. Again, because your token database is empty, you don't have to run that cron job anymore, or you can, but it's essentially a no up. Fernet has some downsides though, and that's that token validation is suddenly kind of difficult. We have to do a lot more work to validate Fernet tokens. With PKI tokens, all we had to do was check the signature, decode it, return it, you're done. With EUID tokens, all we had to do was a single read from the database, good to go, return the response. With Fernet, we have to do a whole bunch of reads and rebuild that entire authentication and authorization context on every token validation. And token revocation is also hard. With EUID tokens, we can just delete them from the database and your tokens revoked, there's no record of it anymore. With PKI, that was a little bit more of a challenge, but it's essentially the same process. We deleted or market is deleted. Your entire cluster gets made aware of that and kind of you're done. With Fernet, we had a long process of implementing token revocation events, and we won't get into that today, I don't think, but it's a more interesting problem than we've ever had before. So what's inside of Fernet token? It's kind of one of the first questions we get asked when we talk about, you know, they're not persistent. Here's all the advantages they have. So what makes them tick? So again, every token represents a user's identity and a scope of authorization. And so that generally comes in the form of a user ID and a project ID. We tack on an expiration and we message back the whole thing together for Fernet. Message pack is a lot like JSON, but it's basically a binary format. So we call that our payload, and then we actually AES encrypt the payload, append another timestamp, the Fernet version, take an HMAC of the whole thing to sign it, and that's our decoded token. We just base 64 and code it and we're done. That's our final token. So if you actually want to dive more into the details of that process, we didn't actually write the spec for this. This is actually a spec originally implemented and proposed by Heroku, and we just actually took advantage of a library and defined our own payloads for it and kind of wedged it into OpenStack, made it work. But the HMAC and the AES bits of that are interesting because they both require keys. So what is a Fernet key? Fernet keys are a combination of a 128-bit key that we use for SHA-256 HMACs and a 128-bit key that we use for a symmetric key for AES encryption, and so that string on the bottom is actually a Fernet key, and they're actually just base 64 and code it as well. So we store those in files on disk, and because we're actually using the Fernet keys to generate ciphertext that we're then passing back to users constantly and you can request as many tokens as you want, they're eventually vulnerable to being broken, so you have to rotate your keys periodically, and I will let Lance explain how key rotation works. So now that we understand a little bit more about what's inside of a Fernet token, let's talk about key rotation, we're gonna break it into three different parts. First, we're gonna talk about how to configure keystones so that you can issue your Fernet tokens, but also make sure that our config allows us to do successful key rotation. Second, we're gonna go through some commonly asked questions that we get around key rotation, and then third, we're gonna go through a few different examples. So starting with config, the only thing that you really need to set in order to issue Fernet tokens is your token provider, which is pretty straightforward, but there's two other things that you need to know and you need to be aware of before you start issuing Fernet tokens, and that's your key repository and your max active keys count. So your key repository is just a location on disk, but it changes depending on where you deploy keystone and how you deploy it. Your max active keys count is the total number of keys that you can have in your repository at a given time, and this is gonna be very dependent on your token expiration, as well as how often you plan on rotating your keys, which brings us to our next biggest question, and that's how often should we be rotating our keys? This is gonna be dependent on your deployment security model, but you can start by asking yourself a question, that's how fast would you expect somebody to crack a 128-bit AES encryption key, as well as a 128-bit SHA-256 HMAC signing key? Your goal as an operator is to make sure that you rotate more frequently than whatever that is. So once we know what that timeframe looks like for us, the next thing we need to know is our token lifespan, and then using those two things, we can actually step through and figure out the number of keys that we need. So these are the three things that you need to know before you can actually start implementing a key rotation strategy. So here's an example. Let's say we want to do rotations hourly, and we're going to have a token expiration time of one day. We need to have a max active key count of 25 keys. The reason why is if we go below 25, we're gonna be removing keys that have been used to encrypt tokens that are still valid, which is poor user experience. You can set that value higher than 25, but you're just gonna have unused keys in your repository, which is fine, it's more or less a cleanup operation later. Here's another example using token expiration times of one week, and doing daily rotations, you're gonna have to have a max active key count of at least eight. And another example using token expiration times of 90 days, and doing monthly rotation, you're gonna have to have a max active key count of at least five. So some other questions that we get around key rotation. How many max active keys should you have? We set the default to three, and we suggest three or more, because if you go below three, you're prone to removing a key that's been used to encrypt a token that's still valid. Another thing that we get all the time is when I deploy a cluster of keystone nodes do I have to initiate that rotation simultaneously across my cluster? And the way that the key repository is set up and how the keys work together, it allows you to do a rotation on a single keystone node, inspect that everything works, make sure that tokens issued from that node are still validable and usable across the rest of your cluster, before syncing that repository out to the other nodes in the deployment. So as an operator, that's kind of nice, right? Because you can do your rotation, make sure everything still works, and then commit that across the rest of the nodes in your deployment, which kind of brings us to the third question, and that is, why not just put the keys in the database? Because then technically, if you have a replication solution already set up, you can leverage that to do your cluster sync, right? But then we'll start looking at the problem and we'd be putting keys in that are not encrypted, so we have to encrypt the keys that we use for encryption and so on and so forth. So keystone provides a couple of different tools to help you manage your key repository, and these are packaged with our keystone managed command. The first one is Frenet setup, and this is going to bootstrap your key repository location that you specified in your config with a couple of different keys so that you can actually start using Frenet tokens. This is a one-time operation and you have to do it before you can actually start issuing your Frenet tokens. The second is Frenet rotate. Frenet rotate is the actual thing that goes in and will give you a new key to start encrypting with. This is something that you should be doing consistently over the life of your deployment, and we have to keep in mind that Frenet kind of gets its strength from your rotation policy, so the more often you rotate or having like a dedicated rotation schedule is very important. So let's start looking at an example, and this is going to be examining a key repository on a single keystone node. So at this point, we're going to be using a max active key count of four, and we have initiated our Frenet setup command. So when we have done this, we're going to be left with two keys in our repository, and please note the colors on the slides, this is actually going to denote the actual contents of the key, and then the state or the type of the key is to the right of that, and every key in your repository can be one of three different types. There's a staged key, and the staged key is always going to be named zero, it's always going to be the next primary key in the repository, and there's only ever one of them. The other kind of key you can have is a primary key. Primary keys are used for token generation as well as token validation, so they can be used to encrypt and decrypt. They're also the key with the highest index. There's only ever one in the repository at a given time. The third type of key that you're not seeing yet is a secondary key. Secondary keys are used to only decrypt. They've been the primary for some given amount of time previously, and there can be multiple of them in the repository at once. So at this point, if we're to ask Keystone for a Frenet token, we'd be getting back one that's been encrypted with the green key. So we're going to move on to our rotation. We're going to take our maroon key, we're going to promote it to be the new primary key, and then we're going to introduce a new staged key. So at this point, if we were to take the token that we had gotten on the previous slide, pass it back to Keystone and ask Keystone to validate it, Keystone still has access to the key that was used to encrypt it, so that validation can still happen. But if we were to re-authenticate against Keystone, we would be getting back a token that's been encrypted with the maroon key, which is now the new primary. So this same cycle goes through on our next rotation as well, and we're going to take our yellow key, promote it to be primary. We've demoted our maroon key, and we introduce a new staged key. But at this point, we're also at our max active key count. So on our next rotation, we're actually going to do another step, and that's going to be removing the oldest secondary key from the repository, which is going to look like that. So if we were to take the token that we had gotten on the very first slide of this example and pass it back to Keystone, Keystone cannot validate it because it no longer has the key that was used to encrypt it. So this is where setting your max active key value is very important with your rotation schedule because you have the ability to remove keys that could be encrypting tokens that are still valid. And then your max active key count keeps those keys in check as you keep doing subsequent rotations. So that's what rotation looks like on a single Keystone node if we were just to inspect a single repository. But what happens when you need to do rotations across the cluster, right? You have a pile of Keystone nodes and they all need to be able to validate each other's tokens. So we were solving for two different things in this case. The first was that as a client, I want to be able to validate my token against two different Keystone nodes without any sort of replication race. The second was that rotations are something that should happen frequently. So they shouldn't be causing any sort of downtime. For this example, we're going to use the same kind of color scheme that we did previously. And then we're also gonna change our max active keys value to three. But we're also gonna be working with three different nodes. The nodes on the right side of the slide actually depict the actual step that we're in. The nodes on the bottom of the slide highlight some facts about the repositories as we go through each rotation. So at this point, we're not ready to start issuing for net tokens yet because nodes two and node three don't actually have a repository to start encrypting keys with. So we've initiated our for net setup on node one and our next step is going to be syncing that rotation to the other nodes in the deployment. So we go ahead and push that to node two and to node three. And at this point, everybody has the same primary and secondary keys. So any token that you get can be validated across any other node in the deployment. And we're gonna go initiate our next rotation that's gonna be initiated from node one. So we're gonna take our maroon key, we're gonna promote it to be the new primary, and then we're gonna introduce a new staged key. So at this point, we've done a rotation. Our cluster is out of sync though too. And that's fine, technically everything still works because node one is gonna be issuing tokens that have been encrypted with a maroon key. Node two and node three have that key and that is the staged key. So they can technically validate tokens issued from node one. On the other side of the coin, we also have node two and node three that are still issuing tokens with the green key. And node one still has the green key around as a secondary key, which is only used for decryption. So that validation can still happen. So everything is still working. We can validate it, nothing's on fire, nothing's broken. And then we can go ahead and we can sync that out to the rest of the nodes in the deployment. So we're gonna do another rotation, but we're gonna initiate it from node one. But we're also at our max active key count. So we now remove the green key. So at this point, regardless of where we get the token or where we decide to validate a token, any tokens that have been encrypted with a green key should hopefully be invalid due to expiration. But the same pattern repeats. So at this point everything still works. We can still validate node one tokens against node two and node three. And we can validate node two tokens and node three tokens against node one. So we're gonna go ahead and sync our cluster out to the other nodes in the deployment. And everybody's back up to the same state. So your rotations are gonna go through that cycle where you initiate a rotation on a single node and then you go ahead and you push that out to the other nodes in the deployment. So this is actually what Keystone does. This is our implementation. And you can see that our colors have now been used with integers. So in Keystone we actually use integers for names. So if you remember the stage key is always a zero key and the primary key is always the key with the highest index. And then we go ahead and remove the old keys that are secondary. So what happens when you over rotate? And what we mean by over rotating is that you've taken and you've done a rotation twice without doing some sort of sync. So how does that affect your cluster and how does that affect token validation? So we're gonna move back to the same example. We're gonna initiate our rotation from node one and we're going to take our maroon key promoted to be primary and then we're gonna introduce a new stage key. So everything works at this point like we had seen in the previous example. So our next step should be doing a sync to the other nodes. But let's say we go ahead and initiate another rotation from node one. So we've taken our yellow key, it's new to primary. We have a new stage key, we're gonna remove the green key. At this point, depending on where we get a token and where we validate a token, that is going to be a problem because node two and node three can no longer actually validate tokens that have been issued by node one. Likewise, node one can no longer validate tokens issued from node two and node three because it's already cleaned up the green key. So we got kind of a mess here. But this is just going to show what happens if you break that cycle of rotation and syncing. And this is how we kind of suggest to do key rotation upstream and just some facts about it. And we used all of this with the Furnet setup and the Furnet rotate commands. You don't have to use that. It's just tooling that we provide with Keystone out of the box. We do have operators that use other things and Matt can actually talk to that because he's kind of broken that mold a little bit. All right, thank you. Okay, I'm gonna just give a brief overview of OpenStack, a Time Warner cable and then talk about how the Furnet switch went for us and how we're doing things like key rotation. So first, we run OpenStack in multiple regions. But for Keystone specifically, we have a region spanning MySQL Galera cluster that way users have the same credentials in both regions. Versions is an interesting story for us. Right now, we're running stable Liberty Keystone in a Docker container. We run mixed versions of other services. For example, Horizons off of Master. Most everything else is Liberty. But we don't tend to keep everything in sync because we're in using containers so we have a little more freedom there. And we've been using Furnet tokens in production since July of 2015, so about nine or so months. So before we started, what were the prerequisites for Furnet tokens? And some of these we knew, they were documented and some of these we kind of discovered along the way. The first one, very clear, is you need Keystone, Kilo or Neewer. That's when they were introduced. However, for stability and performance reasons, we recommend you use at least Liberty in probably Metaka now. Also, between Kilo and Liberty, Furnet tokens underwent a slight format change where some padding was removed and that change is disruptive. So if you're going to do Furnet, just move to Liberty first and save yourself the hassle. Okay, what about Horizon? Horizon needs Django OpenStack off of at least Kilo or Neewer. What about everything else? So all the other services use Keystone Middleware to talk to Keystone to validate tokens passed into them. They also are holding their own service token which is how they talk to Keystone. And as it turns out, if you have an old copy of Keystone Middleware pre-Kilo, the services don't realize that the token they're holding is no longer valid and let us keep trying to use it until it expires. So this can be disruptive. So recommend you upgrade all your services to at least Kilo before you do this or you have to restart services. I mean, other services like Neutron and Cinder when you switch token formats. Cashing, this is not technically a prerequisite. You don't have to do it but Keystone does a lot more work to validate a Furnet token than it did for UUID token. And so the very first thing we did after we released out with Furnet is we turned caching on. On this, this saves Keystone from having to do a ton of work. Okay, so what was our migration path here? The first question people would ask me is like, why do this in July of 2015? Dolph's already alluded to most of these problems. We had some weird token replication issues between data centers. It wasn't a super typical use case but we did have a bunch of API check scripts. They would get a token in the West region, try to use it in the East region, it wouldn't work. It would fail like a certain percentage of the time. So the solution was to go to all our check scripts and put sleep calls in them and hopefully those were long enough to get the token replicated and it was just kind of a horrible hack. Also, we had a bunch of cron jobs that were written to clean up the token table but because you have a six node cluster, you don't want all six nodes trying to lock the table to delete tokens at the same time so we had this crazy logic like East node two can delete tokens on odd hours and I don't know, it was just messy. Also, we tried to kind of like to be running the latest greatest thing. That was kind of the latest greatest thing back in 2015 and kind of gets us ahead of the game for when Furnet becomes default, hopefully in Newton or Alcada. Okay, how do we orchestrate the upgrade? We don't do anything in our cloud without automation and for this work, we chose Ansible to drive Puppet and Puppet was gonna do the things it's good at which is changing config options in Keystone and restarting services. Ansible is gonna do the stuff it's good at which is handling the inner node dependencies. Also, I should say, we knew this was gonna cause a brief outage and so we wanted to sort of minimize the number of outages and also we just like doing things the hard way so we did Furnet token switch, upgrade Keystone version and switch from a vent led to Apache all at once. And so this set of playbooks I have a link to here for is designed with that in mind, all three of those things. And so it includes things like database backups shutting down part of the cluster, anticipation of database migrations. If you're just switching token formats, the playbook is a good start but it will be totally overkill. You'll be deleting stuff. Okay, what was our outage? So in production, our API outage was 16 seconds. However, almost all of that was due to things like Puppet having to upgrade packages, run database migrations, get Apache configured, get Apache restarted and then at the end, like I said, we had to bounce all the other services so that they would get a valid token. If you are just switching token formats and not doing anything else and you're on Liberty, it should be very brief, just a couple seconds probably. Token rotation, let's dive into this because we don't do it the way that upstream does. Okay, step one, we don't use Keystone Manage for net setup, we don't use Keystone Manage for net rotate. The reason why is because if you watched his slide, he's kind of implying, Lance is kind of implying that one node there is like a master node and all the rotations happen on the master node and then all the keys move over and that kind of violates how we view our cloud where all Keystone nodes are just throw away, we don't have any special code. We don't like to run any special code on one node versus another. If Keystone node one died and we didn't rebuild it for a month, we want to not have to handle that situation with key rotation. So what do we do instead? We actually store our keys in encrypted EAML and have Puppet drop them on the box just into the Etsy for net keys folder and that happens whenever we do a deploy. So this one's interesting. So we actually wrote tooling since we're not running net rotate, we wrote a Python script that would basically take that encrypted EAML, check it out, do a key rotation, re-encrypt it and then post it up to Garrett. And so you might ask yourself as my team often does why am I reviewing an encrypted version of a Furnet key? I can't read these to make no sense. The only point of this for us is to prevent over rotation. We thought about having this just run automatically every week but there's certain weeks we don't do deploys like this week where most of my team is here and if we have something rotating every week and checking in and we miss a deploy, we've now done two rotations and it's gonna break all the outstanding keys. Lance went into this but to reiterate, the rotations do not have to be in sync. When we do deploys, we do one region first and then the other and within those regions we actually serialize Keystone nodes so anywhere along the way when we've done a rotation our keys are about to sync. In addition, we'll stop the deploy after one region and do a bunch of validation and if we find that the region is kinda screwed up we won't do the other region and in fact we had a deploy where we waited two days between doing East and West. So for two days, East and West had different sets of Furnet keys and doesn't matter because they're all using keys that are already on the box, the staged and the old primary. How often do we rotate? So all the factors were explained before but a big primary factor for us is how often we do deploys and that's weekly and so therefore we rotate keys every three to four weeks generally. Okay, so since July, what have we learned here by operating the Furnet tokens? First, already said but empty token table which was great. Our Keystone database is super small now. There's hardly anything in Keystone database without tokens, it's just users, some assignments and some projects and it's pretty simple. Also, one of my most fun parts of this was deleting all the horrible cron jobs and all the horrible hacks we had to deal with UUID token replication and cleanup so that was great. Finally, without having so much database traffic especially and not having to replicate tokens across regions, the replication lag vanished. Just in general the load of the Keystone replication load decreased greatly we're only syncing users and projects and stuff and there's just not that much going on. Okay, performance, this is probably a big one. So Furnet tokens are slower to validate than UUID tokens. They're faster to get but they're slower to validate and token validation is important for your cloud performance. Every API call you make to OpenStack requires a token to be validated and I just scanned through some logs a few weeks ago and it looks like for every token we create we validate about 10 tokens. So you're doing a lot of token validations. So what we highly recommend is you just turn on Memcache. There's config options for it in Keystone. You're basically Keystone seeing the same tokens over and over and over again. Get those cached in Memcache, save Keystone all the work of having to re-validate it. Also if this is important to you we've been benchmarking our cloud in terms of token create and validate performance and kind of watching that, seeing what tweaks we could make. So we recommend you do that if it's a concern. And the other good news is LanceDolph and the rest of the Keystone community is gonna be spending some time on token performance and Keystone performance in general in the Newton cycle and hopefully most of those patches should be back portables, especially if they're for Furnet tokens at least back to Mataka. So things should get faster. Okay. I wanna just leave this slide up as we do the closing remarks. This is basically just a list of where you can get more info. It covers key rotation. It covers like what's in a key and just some of my experiences and some of their experiences with running Furnet tokens. But hopefully you guys will learn something today about Furnet tokens and how they're created and you've least thought about some of the things you're gonna have to consider when you switch your cloud over to using them. And we have some time for questions. I have two questions. The first is in all your examples, you have max keys set to essentially n plus one of your rotation schedules. There a reason you couldn't do n plus two to work around the over rotation problem? Yeah, absolutely. That's an option. You can always set that higher. You don't have to set it to the absolute minimum in case you do accidentally do an over rotation. It kind of gives you a little bit of a buffer there. So setting it to n plus two, that's totally acceptable. The other question was, you mentioned key synchronization a lot, but not the mechanism for doing so. Could you describe a couple of ways that you all do that? So the OpenStack Ansible project has an interesting way of doing that that kind of touches on the point that Matt made about not having a single keystone node be the master and that's just done through or syncing between the different nodes and syncing that repository. So it's really kind of whatever tooling makes sense for your deployment. If you already have something that maybe syncs keys between, or like SSH keys or whatever, between different nodes, you can build on top of that and leverage that same kind of tooling for syncing your cluster with your front keys. But there's a few different ways of doing it. It just kind of depends on what you use. Maybe back in. I was just saying there's no official upstream way to do it. So that's why we just kind of put it into our existing deployment process that used to Puppet and EAML. But everyone's cloud may be different there. All right, thanks. How is Keystone protecting the keys that are stored on disk? We totally leave that up to the deployer. So disk is a fairly safe place to put them. Put them in a RAM disk or something if you're really worried about things being on disk. But we punt, totally. So first, thanks for the talk, it was really useful. I didn't forget, you mentioned that to determine the frequency of the rotation, you need to analyze what you expect, like what time you expect for people to break the keys. But since it's like fixed size, I expect that upstream can have a recommendation for that. Do you have a recommendation for the frequency? All of your examples were like every half and every week and every month. And so it's totally different. So you're asking what the recommended rotation frequency is? Yeah. Totally up to you. Well, that's broken, I think. There should be an upstream recommendation about that because people don't know how to calculate that. I mean, I can tell you, most people do, I think, weekly. But it'll be hard to get upstream Keystone, I think, to make a recommendation on that. Because it's just gonna depend on how many tokens you're putting out and your security. I mean, I understand, but my point is that if Furnit is default, then that must be a recommendation because people don't know how to choose that. So. Oh, sure. So we are working to make Furnit the default. It's been a long road to make that happen. But based on the other variables we have in Keystone, we currently have a default expiration. We currently have a default max active keys, which just kind of exercises the full rotation cycle. And so from there, you have a minimum value of how frequently you need to rotate, or sorry, maximum rotation frequency. So, could start with that number, I guess, but it's just a recommendation. Yeah, I mean, I'm just saying there should be an upstream recommendation because most people will not know how to come to a conclusion for that. I believe there is a Keystone work session either today or tomorrow, where Furnit defaults getting covered. So it would probably be good feedback to come and bring to that. Sounds good. I think so. Again, thanks for all the talk, it was pretty good. That's it. Thanks, everybody. I think we're done. Oh, one more. Informally, I keep hearing statements that Furnit is not quite ready. There's some missing elements to it for production use. Can you speak to that? Well, I think Matt would disagree with you. The road to getting it default is probably the biggest blocker. Tempest makes some assumptions about, like, what it can do with tokens. And so it's been a matter of, like, changing Tempest's assumptions or increasing the timestamp precision and Furnit to avoid race conditions just in a very aggressive testing suite. So that's about it. Yeah, I just said we haven't had any major issues with it. We noticed this a little bit, this performance change and validation when we implemented it, but getting rid of all the database replication stuff was a win for us. We're also using it in Rackspace, public and private cloud, essentially, moving forward. Definitely production ready for us. One of the things in the release notes was that performance verification is indeed very slow, probably a couple of times than issuing. So how or where would you try to mitigate that? Or when would you see so many revocations? Because usually you'd expect them just to expire. Revocating is not something that happens all that often. You're asking about token validation performance? No, revocation. Replication of keys? Yeah, so keys or tokens? Tokens. So token revocation is something I'm concerned about because the performance in Keystone as that table grows gets really horrible and in fact we like to just kind of keep an eye on it. The only time I know that we really revoke tokens at our company is when my testers here are testing whether or not it works and when people sign out of Horizon. So we typically only have like 5 or 10 in the table unless the test suite is just run. So I haven't been super worried about it performance-wise, but it definitely hurts. And I have a blog post on it that you could probably go find off my website. Thanks. I think we're out of time, but we can talk to someone. We can just talk to you up here if you want.