 All right, I think we're going to go ahead and get started. Go ahead and get your laptops out. Start pulling these Docker images. The repo is, again, DCOS-Labs slash secure-mesos-workshop. They're large Docker images. You want to get them early before everybody else steals your Wi-Fi. And yeah, I'll hand it over to Vishnu. My name is Adam Bortalon. I work at Mesosphere, Mesos Committer, DCOS Committer, all kinds of things. It's my fourth Mesos con. So I've been doing this a while. Yeah, that's kind of like me. I am me at Apache.org, and Jurg is our track lead. He'll be around to help you with the workshop during the exercise time as well. And the node will be joining us shortly. One of the original Mesos committers and contributors worked at Twitter there for a while, is now managing the Mesos core team at Mesosphere. And we've got Vishnu. I work in sales. All right, so actually, this is actually a continuing conversation from anybody that was at Mesos con 2016 where Adam actually put together a Mesos security best practices talk. So there's been a couple of improvements, obviously, over the course of a year in Mesos security. We're trying to address most of the topics that are still relevant, but also introduce a couple of new topics. So for those of you just triggered in after we got started or are just arriving, all of the slides, all of the content, read me, and all the link to the Docker images that you need to download are available at this URL. The slide should also be up on the Schedule site. I think Yorga uploaded it, so if you got your schedule open, it should be there too. All right, great. As mentioned earlier, it's a continuing discussion. And I'm constantly surprised by my interactions on Slack, the community mailing lists, so on and so forth. When people come to us and ask us for advice on how they should be running their Mesos cluster, by saying, I suddenly find myself running a Bitcoin miner on my cluster, what do I do? So this has happened more often, and some of you may laugh at this, but there's plenty of people out there that are getting owned, and especially given what's happened at Equifax and how it's affected almost everybody. I don't think it's not affected. There's nobody that's unaffected by it. We want to make sure that you try to the best of our knowledge to set up a cluster that is somewhat relatively tolerant to most known attack vectors. And again, I personally work with a lot of customers. We have customers in financial services, health care, government, you name it. And they're all handling very, very sensitive data and very important data that affects our lives. And a lot of this stuff is running on Mesos as of late. And we're trying to make sure that they understand the implications. This talk is still very introductory at the end of it. And even at the end of the labs, it's only scratching the surface of what we really need to put together for Mesos and all of the components around it. In fact, there's a lot of things that are out of scope, and we won't be able to get to them. But if you have any questions on any pointed questions that are not covered by this talk between Adam and the rest of the crew here, I'm pretty sure we can answer most of that. So the other things that are sort of interesting is over the past year, at least, we've been able to a lot of primitives that enable better multi-tenancy. I know it's an overloaded term, but we'll try and qualify what we mean by multi-tenancy, at least in this realm. But in general, it's like, let's try and prevent all of the easy mistakes from happening, is really what this talk is about. Stuff that's out of scope. I mean, container security, again, is a very, very broad topic. A lot of the very specific things about the container riser security is covered in this Mesos ticket 4936. So I'm not going to talk about mandatory access controls at the Linux kernel level, or capabilities, or SEPCOMP, or user namespaces. They all have their role to play, and they all need to work really closely together. And there's still gaps in what we can support today and do support today, as it relates to other container risers, like Docker and so on. So just quickly, Mesos does have capability support. SEPCOMP is a work in progress. User namespaces is notoriously difficult. So getting this design right is something that's going to take us and everybody else a lot more time. And again, we're going to talk about hardware security primitives. I mean, if you're really using something like a TPM, Trusted Platform Module, or a hardware security module, like an HSM for key management, or other sort of credentials, or secret management, I'm going to talk about that. OK, so where do we start? So let's make sure we fence off the outer boundary of the Mesos cluster. That's what's securing the perimeter is all about. Make sure everything is, you shouldn't be able to access, let's say, port 5050 on your Mesos master. You shouldn't be able to access 8080, or 8443, on Marathon, for example, by default, at least from the outside world, and the outside world being some sort of threat vector. So what are we talking about? So we have a whole bunch of Mesos components. We have potential actors like a framework developer or user. We have these service users. They don't really necessarily care that stuff is running on DCS. And finally, we have the actual operators or admins of the DCS cluster, right? How do we stitch together all of these sort of components and building blocks together, and arrive at something that's out of the box a little more secure than the default? We all know that the masters listen on port 5050, as it's shown. There's also 2181 on there, which is the zookeeper, and securing zookeeper is very, very difficult, if not impossible. It doesn't do TLS yet, and the best technology that we have for zookeeper is ACLs, which is also why, at the start of the keynote we mentioned, getting rid of zookeeper is actually a pretty big priority. It's very hard to secure. It solved a problem for a lot of people, including us, for a very long time. But we're quickly running up against its limitations pretty badly, and hopefully, we can get rid of it, and replace it with something that's actually secure as well. The agents listen on port 5051, and then you have, I wanted to introduce this concept of a public agent, which is a special class of agents, which is usually going to be used for ingress traffic. So we don't want anything directly being able to communicate with your agents, ideally, directly. Instead, we're going to proxy all these communications through a limited subset of nodes that are going to be sitting in a DMZ environment, and that's this notion of a public agent. So that's why I put 80 and 443 there. You're typically going to run some sort of proxy, HA proxy, NGINX, something like that, and then have that talk to your actual service is running on the private agents, and it's all filtered through a firewall. So let's start with just the Mesa security. I know this is kind of interesting to watch. So effectively, what this is trying to say that almost all communications travels through a firewall. If the master wants to talk to a private agent, technically, it should go through a firewall. The public agent talking back to a private agent, so on and so forth, they should technically go through a firewall. But what this also means is that each of these things are in different network domains or network zones, right? You have a network zone for your public agents, you have a network zone for your private agents, and your masters, which is the control plane, should also be in a different zone. And then the firewall sits between the zone and you're routing traffic back and forth and only permitting the ports that actually matter for this communication to work, right? But in practice, it turns out that, you know, it's more of a free for all. You really want a lot more permissive access between the masters and the agents and between the public agent and the private agent, more specifically from the public agent to the private agent, yeah, because your applications are actually running on your private agents. So in practice, it's actually harder to secure this communication, which is why this entire thing is in a specific boundary. And now if you introduce the administrators, they will go through a firewall, so they will only access port 8, port 50, 50, for example, on the masters. But that, again, we aren't talking about authentication yet, but just the port security at this point. And this itself will help you a lot, because at this point, only the master is accessible. You typically don't want your agents directly accessible, right? You want to go through something else, I'll get to that later. So now throw in the developers or the framework users, the people that use Marathon, welcome, Vinod. And now these are the people that are actually using frameworks, or developing frameworks, or writing frameworks, or using Marathon. So even they need to go through a separate set of firewalls to make sure that they are only accessing the services that they should technically be accessing on the ports that they should be accessing. So I've also introduced the load balancer here, and I didn't mention this in the previous slide, but this is in any typical highly available environment. You've got to have multiple masters, multiple agents. And in this case, even the frameworks are going to be at least two or three, depending on your level of availability. So there's a load balancer involved as well. And finally, we have this third class of users, the people are just trying to connect to the service that's being hosted by your Mesos cluster. And they don't necessarily talk to any of the administrative infrastructure. They shouldn't even know about the control plane. Ideally, they shouldn't even know that's running on Mesos, right? They only want to talk to whatever it is, that Tomcat application, or whatever. And that goes through a separate set of firewalls and load balancers, and they only talk to the public agents. So you're restricting the attack vector at this point. Does it make sense so far? Okay, very simple stuff. But this is the kind of things that will prevent you getting owned by a Bitcoin or Ether miner. Don't wake up one morning and find that running on your cluster. There's another important thing that we have this in DCS, but it's a nice pattern to also include. We call this a service gateway proxy, whatever you want to call it. But effectively, it's something like an Nginx that sits in front of all of this that provides user-friendly routes, effectively. So when I want to access my Mesos masters, I would just say go to this host name, whatever, gateway.mycluster.com slash Mesos, and it'll take me to the Mesos UI. If I go to slash agent, that's also a proxy by which I can actually get to the agent UI, more specifically the sandbox logs, right? You don't want the sandbox logs directly exposed over an open network, so you can attack the agents that way. So this is still a nice thing to introduce in between, which will help complete the picture both from a usability perspective, and also it does improve the security of the platform because you're forcing everything to now enter, just like we did for the HAProxies on the public agents, a finite set of elements that you tightly control and lock down. So that's really what the service gateway is for. Okay, obvious things that everybody should be using by now. It is 2017. It should be TLS, not SSL. Please use TLS and modern versions of the protocol. So some people still use the words interchangeably, but yeah, SSL is dead. We still use the environmental variables that they called SSL, but as you can see, there's TLS involved. So very specifically, by default, Mesos does not build with TLS support, so if you're going to build your own Mesos or roll your own Mesos, make sure you compile it with those flags. You need to use libEvent, and basically this is wrapping open SSL libEvent, libEvent, open SSL, one of those libraries. So yeah, this is effectively required for you to even enable TLS support for Mesos by default. It's not built that way unless you're using one of the Mesos official binaries, which now, by default, include TLS. Yeah, there's a whole bunch of variables. I mean, but the important ones are, yes, you want it to be enabled, right? So this downgrade thing is also very dangerous. I mean, you typically don't want to have, once you've enabled a TLS connection, to go back to insecure connections. Is that correct? Yeah, just for upgrades. Just for upgrades. So yeah, so we always want to make sure that if you are trying to connect over an insecure and non TLS connection, we want to upgrade you up to TLS and say, no, no, I want you to connect with TLS. Obviously for any TLS communication, you need a key file, a cert file, and you want to verify your certificates. We're not doing mutual authentication of the master and the agent or whatever is communicating over TLS at the moment, but it does help you verify the chain of trust if you actually say require cert. So this is a nice feature to have. So you, and this actually effectively makes it more strict. So you need verify and require. Require implies verify. So that's what's happening there. Other things, I really like using the certificate authority. I mean, so in the labs, what we're ending up going to do is we're going to spin up like self-sign search, snake oil search. This is not what you should be doing in production. You really want to be using a proper CA, but if you want to get started, at least the communication is over TLS and not in plain text. So that's really what this is about. Use proper ciphers. I know people care about the version. So please try to be on at least one dot two. One one is still around. One should not be used. It is deprecated. So even an SSL V3, if anybody enables that, you should fire them. So don't do that. And then yeah, you can also pick the kind of curves that you want to use if you're really particular, if you have like very finite security requirements, you can also do that. Okay, so this is a CA structure. And you could set it up any which way. Most organizations that I run into have a root CA which is completely offline. There is no way to get to it. It's usually an HSM. It's very expensive stuff. And then what they instead do is have online intermediary CAs to which there's usually an API attached if you're actually a savvy customer and you don't want to kill somebody. Because generating search with the CSR, getting it signed, getting it back, there really should be an API for it. There's toolkits out there like CFSSL from Cloudflare, which is what we use inside DCS for example, but really it should be an API driven thing. It's got another problem that I won't cover yet, but which is, if somebody submits a CSR to this API, why should I sign it? Why should I give it back? We're not gonna talk about that problem specifically. But usually the intermediaries can be tiered. So you can actually have at the very top, you have this intermediate CA for, let's say the servers, you have an intermediate CA for all of the application certs. And then you could also distribute and have another intermediate, second tier intermediary CA for region specific certs. It's rare to see more than an intermediary, or at least usually the first levels of what I see with the offline route at the top. But I've also seen the pattern depending on how sophisticated the customers are. And finally you issue certs for each of these things and then you have TLS, so. And that's the open-ended item, which is how do I actually get this key insert? How do I generate the key insert? And then get this certificate signed, so. And then finally we also have framework schedulers that also need to talk to the mesos masters over TLS. How do we get searched for that? How do we secure that communication? How do we give the scheduler the private key and the certificate as well to bootstrap itself? Those are all problems that we're gonna hand wave at this point. Come talk to us about some strategies that we've employed. There's different kind of strategies for each of these things. Very simply, how would we do this? I really like the CA APIs, but the only one thing I wanna make sure is most people have APIs, actually CFSSL has this, where you can ask it to generate the private key on the server itself. Usually that's a bad strategy. You really wanna keep the private key completely only on the entity that's generating or requesting a cert. That's really what I wanna show here. So either if you're a master or an agent or a scheduler, make sure the private key doesn't unnecessarily move across the network. That's really what that's about. So you generate a private key pair, the public key, private key, generate a CSR, submit the CSR to the certificate authority for signing. If it's valid and it's an open-ended question, why should I give you a certificate back for your CSR? Zooming, we're taking the short route here. We'll get a valid sign and certificate and we're off to the races. Otherwise we get an error. So I'm gonna leave that open for now. Any questions so far? Makes sense? Yeah, TLS is important. I'll hand it over to Adam for modules. All right, so modules themselves are not specifically a security feature. But they enable us to plug in dynamic functionality via shared libraries, which gives us a few different benefits. One is you can isolate external dependencies if there's some complex sassel or something that you don't want to bundle into Mesos itself. You can put that in a module and then have that loaded dynamically. You also don't have to recompile all of these things separately. So the modules can be compiled separately from the core Mesos. And this has enabled people to experiment. Grad students can go take Mesos and inject their own functionality for allocators or whatever else. It also has enabled companies like Mesosphere to build our own proprietary implementations of authenticators, authorizers, that we can plug into our distribution of Mesos. And we, as Apache Mesos committers and the community, we've built out module injection points and hooks into various aspects of the master API and lifecycle, the agent API, various different points in the container launch and destroy lifecycle. There are, yeah, many different module types. So the allocator was one of the earliest ones. We long talked about how it was pluggable, but once we introduced Mesos modules, we actually made it pluggable. So you can build your own implementation, get rid of DRF if you want, experiment with something new. There are companies that have worked with that for their own particular frameworks and workloads. There's authentication, which we'll talk quite a bit more about today. And we won't actually show you how to build your own authenticator module or authenticate the authenticator module today, but it's not too hard. Reach out to us and we can walk you through the process offline. Authorizer as well, which we like to separate authentication, which is just validating the identity from authorization of what is this identity allowed to do. And so you can swap out either of those independently or both of them. Secrets is another thing that, we previously used some generic modules and hooks for the Mesos containerizer and the Docker containerizer, but recently have added secret generator and resolver modules so that secrets are becoming first class in Apache Mesos. I'll talk a little bit more about isolator, which was the very first module that we introduced. And it's a bit of a misnomer now. Originally it was the idea that, okay, you can have custom behavior, you have some resource that you want to isolate. And before a container launches, you do some setup, wallets running, you do some monitoring when it dies, you clean up. And then it turns out that that's just generically useful to have these hooks into the container lifecycle. So we've used that for many different things, injecting secrets before you start the container, modifying environment variables, setting up new file systems. You can do all kinds of things with an isolator. So if a lot of times when partners and customers say, oh, I want to inject some custom functionality in Mesos, we say it's probably going to be an isolator module. We also have modularized the master contender and detector, which is what's putting us on the path towards getting rid of ZooKeeper. So rather than using ZooKeeper for a leader election, you could swap this out for at CD console, text files, I don't know, whatever you want to do. We have modules for container loggers and QoS controllers and resource estimators, which is used for oversubscription. Any questions on modules? All right. You want to cover authentication? I think everybody knows what authentication means. So effectively verifying the identity of something, that you're authenticating to be who you are. So in Mesos, we want to make sure that frameworks can be authenticated. We don't want a rogue spark framework logging in and consuming all of your resources, for example. So we want to make sure those frameworks are authenticated. So you have a principle which identifies this framework that presents some sort of credential and says, hey, I am who I am. I know something about, or you also know something about me that makes me authenticated and how I can launch your tasks. So that's really what framework authentication is. We also want to make sure that we don't have rogue agents joining the Mesos cluster and trying to subvert resources or tasks away to it. So you can actually have a task that launches on a rogue agent and now you can perhaps slurp a secret away. So that's one other measure that we have. And finally, even with operating or interacting with all of the various HTTP endpoints, all the API endpoints that we have and the operator endpoints that we have, we also want to make sure that's authenticated. We don't want somebody saying go and tear down a node or tear down a framework without the requisite privileges. So that's the three types of authentication that we have. Mesos by default does ship with an authentication module that uses a cram MD5 authentication. We'll get into some of these details in the lab. We also can use GSS API for interacting with Kerberos because all of this is based on the Cyrus Sassel library, simple authentication and security library. I think that's what it's called. Okay, so the next thing is, yeah, so what is it that allows this thing to work? We have to present a credential, like I mentioned before, which consists of a principle. Principle is this term in Mesos, which is kind of used for identifying what your subject is, like what is your framework has a principle, right? That's what identifying that I'm Spark principle foo or whatever. And then this is going to present a secret which in combination becomes the credential. And then if the credential validates, I am allowed to register the framework, register as a framework and launch tasks. That's effectively what's going on. Authorization is the twin of authentication. So once I've proven who I am, what am I allowed to do, right? So it's very, usually most people are familiar with ACLs. ACLs typically operate with a subject and action and an object, right? So there is something trying to perform some action on some object and that that triple effectively becomes an ACL, right? So we'll see examples of an ACL in action and we'll put together both the authentication and the authorization to allow certain kinds of tasks to launch or not launch based on the ACLs. There's a whole bunch of stuff that's, the reason I put this up here is actually because when I first started in Mesosphere two years ago, we maybe had four or five actions and over the course of two years, a whole bunch of other authorizable actions have been added. So you can see that I think one of the first ones was register frameworks and teardown frameworks and then a whole bunch of other stuff got added. And this is what we were talking about earlier, right? In order to access the sandbox of your task, for example, we want to make sure that only people who are authorized to access the sandbox are allowed to access the sandbox and so forth. This is a diversion. It's not directly related to Mesos, but it's a very useful concept, namespaces. You might have seen this in other, I guess, orchestrators or security systems in general. It's kind of like a nice way of organizing your mental model of how you should manage permissions. It's in Cloud Foundry. It's not really completely fleshed out in Kubernetes, but if you use Cloud Foundry or something like that, namespaces exist there. Usually this is an easier way of looking at how we can do things because it's a mental map directly to how organizations end up being. So at the very first year, I like to segregate them out by environment. So you have a developer environment, a test environment, a production environment. And then your business units and project structure starts to overlay on top of that. And the reason for this is to reduce the ACL explosion, really. So if you have, for example, 10 tasks under some hierarchy, instead of giving 10 ACLs, you can basically say at the sales level, imagine sales runs 10 different kinds of tasks, you can basically assign an ACL at that higher level and avoid completely having to write very, very specific ACLs for everything. So it kind of helps with the ACL reduction. And this is where you would apply permissions, right? So you basically say a developer working in sales cannot have any permission to tasks running under the edge org, for example. So that's one way of looking at this. And I wanted to mention this because one of the ways we do secrets authorization is directly mapped onto the namespace. So it's actually a one-to-one map. So if I have a secret in that namespace, for example, dev secret, we'll see how that actually allows you to access a particular secret. That's a great... The question is, is there any mapping of these namespaces onto the hierarchical roles that Ben H mentioned this morning? The answer is that right now, namespaces are purely a DCOS concept. Masos doesn't know about them, but in DCOS we are planning to tie hierarchical roles to namespaces that we use for authorization. The hierarchical roles are specifically about namespacing the resources and which frameworks are subscribed to particular roles in the hierarchy to be offered those resources. These namespaces are used for... Once you've launched a task, the task then exists in a namespace and you can control... Have ACLs that control who can access which tasks based on a namespace definition that so you can specify you have access to everything under sales without having to specify each individual one. And then secrets are also tied into that. So you can say this secret is available to any tasks under sales, which I think you have a slide for that. It's a little complicated to visualize it here, but you can think of it as in DCOS at least that a secret has a path and the secret is shared with any tasks under the namespace that the secret lives in. So a secret that's at the top level is essentially shared with everything. So it's not really secret. A secret that is under dev sales secret is accessible to dev sales app or dev sales foo slash bar slash whatever slash app, but is not available to dev slash app. Fino, do you want to talk more about the first classing of secrets in Apache Mesos? Okay, it is a bit hard to see. Can we just zoom in the top part? That should be good. Okay, so what we did with secrets in the latest release of Mesos 1.3 was trying to make it more understandable to Mesos core. Previously, most of the efforts for trying to deal with secrets were done using modules and autoband hooks and autoband knowledge, but we wanted to have a first class concept in Mesos that properly understands secrets and make sure it's properly propagated to all the endpoints that are needed. So what that means is we introduced two different interfaces in Mesos for handling secrets. The first interface is the secrets volume isolator and the environment isolator. So we added two new isolators, first class isolators into Mesos, and both of these isolators are responsible for taking a secrets object, which is also a new primitive in Mesos that we added. It's a new protobuf message. So given that secret message, both these isolators are going to set up either an environment that uses the secret that wants to access the secret value or for apps that want to access the secret value as a file. So these isolators, you could use them to actually inject the secrets into your container, basically, as an environment or as a file. And we also added a new interface called secrets resolver. That's right up there. So what this does is it's essentially an interface that allows you to hook into your secret backend. For example, if you have something like a vault and you're storing all your secrets data in vault and you want to be able to access that data from vault and give them access to your environment or file with secrets, you could actually implement this resolver interface to talk to whatever backend your organization uses to store the secrets. So this is a resolver that gets called whenever these isolators realize that there is a secret in play, they're going to talk to the resolver to resolve a secret. So there's a secret name that you give in the secret proto and the way to resolve that name to a value is done by the resolver. And you could write modules for whatever backends you have for the secrets. In DCOS, Enterprise, we wrote a backend, we wrote a resolver to deal with the vault backend. But you could definitely write it for whatever other backends you have. That's pretty much it. Do you have anything else to say? Just want to comment that in Apache Mesos, we have support for secrets, not just to inject them into your container in a file or an environment, but also image pull secrets. So if you needed to pull an image from a private Docker registry, you can configure your task with an image pull secret that would have the credentials to access that private registry and pull particular images. And then that secret is not even injected anywhere into the container. So the container doesn't have access to it, but Mesos is given that to retrieve the image. I think in the future, we're probably also going to leverage this to fetch URIs that are behind authenticated endpoints, like your S3s or HDFSs. Right now, we do not support any authentication for them or there needs to be some hacks outside the Mesos system to give them the credentials. But ideally, you would like to use the secrets and primitives to basically get fetch artifacts in remote locations. Yeah, and I just want to point out, this is an example where, you know, as Mesosphere, we took advantage of modules to experiment with secrets functionality and play around with what would solve the problems for our customers. And then once we'd come up with a good system, we generalized it and pushed it back into Apache Mesos to make it available to the rest of the community. Rather than us throwing hacks into Apache Mesos and ripping it out and changing the APIs over and over again, we have room to experiment and then provide a stable API to the community. So next, we're going to be doing the lab. So if you brought a laptop, we're going to ask you to go to the GitHub repo. It is dcos-labs slash secure Mesos workshop. And if you did not pull down the Docker images at the beginning of the talk, this is a good time to get started. So it could take you a little while. Wi-Fi is really slow. Wi-Fi is pretty slow, but hopefully most of you have already downloaded the images and you'll be able to get started really quickly. So, all right, good. Most of you have downloaded the images. So that means the Wi-Fi should be lightning fast for the rest of you. So, just a general overview of what this exercise is. We have Docker images provided by Mesosphere for Mesos Master and Mesos Agent. We're using RC5 of the latest Mesos 1.4. So, going to be voted in any day now. And so you get to play with the latest and greatest. What we're going to do is have you start a master in an agent and then we'll use the Mesos Execute scheduler that comes along with Mesos. So, you can, we'll start out with a completely insecure default master in agent. They connect no SSL, TLS, no authentication or anything like that. And you'll run the Mesos Execute command which registers a scheduler and launches a task and then exits. And then there are different steps to the exercise where we'll enable encryption on the master and then the agent and then the scheduler and then the executor. We'll enable authentication between the agent and the master and authentication between the Mesos Execute scheduler and the master. You can also add ACLs for authorization. You can add HTTP authentication. Those are more left as a bonus points exercise where we have not given you the solution ahead of time. In this repo, there you'll find scripts. One that is just the default insecure which starts a master and an agent and then an insecure run command that will run the Mesos Execute. We recommend you start with that, either make a copy or edit it directly and follow along with the tutorial to add the flags and environment variables and play around with the Mesos UI and the logs and everything you can to see how you're securing it piece by piece. Yeah, and we also put in a secure cluster start and secure run command that have all of this done already. If you just wanna run that and then cheat and look at all the flags. I know not all of you are excited about, well, I hope that all of you are excited about walking through each of these exercises step by step. If not, you can cheat and go home early. Any questions? We'll be running around helping you out, just raise your hand and we'll come and answer all of your questions. And, yeah, I guess we've got a lot of other talks and the events that we wanna recommend. All our friends and family here at MesosCon. And along with the slides, there's also links to papers that are references and we've got business cards, we've got stickers at the Meso spear booth. Come talk to us, ask us anything you need to know. And if you're interested in learning more about MesosCon, you can go to MesosCon.com. You can find the link in the description below. Thanks so much. Thanks for coming. Thanks.