 Okay, no. Maybe it's the other one. Should I try the other one? Yes, yes, yes. Okay, no. Still not. Yeah, okay. Okay, this one too. Okay, yeah. 15 minutes, 15 minutes, 15 minutes, 15 minutes. So what I have to do? 6, 10. Okay, great. Okay, good afternoon. Can you hear me okay? Yes. So, I promised you a Kubernetes tutorial. So first, Mia Culver, I suggested I would give you hands-on access and so on, and I just realised when practicing with people in a two-hour situation that this wouldn't translate to a one-hour session. So I really do it in mode tutorial of doing things and showing you what I'm doing. But if you want to follow along yourself, either you've got your own Kubernetes cluster already or you do it after the event. Okay, I hope that's okay for everyone. Okay, so I'm Michael Bright. I'm a developer advocate at Containers. Containers are the creators of traffic. Some of you may know it. It's a reverse proxy load balancer and can be used as a Kubernetes ingress controller. That's something I'll be showing as a demo. Just out of interest, what's people's experience here with Kubernetes? Are you all fairly new to Kubernetes or are some people... So who is totally new or pretty well to Kubernetes? Okay, good. You're the people I'm aiming at, that's fine. And just out of interest, is anyone here using traffic? Do you use traffic already? No. Do people know about traffic? No. Okay. Good. Thank you. I just say this tutorial is open source. The initial version, which was swarm, Kubernetes and mazars, we've run about three, four times. And now this is the third or fourth time after I've run the purely Kubernetes version. And it's all on GitHub. And you're welcome to open any issues on it or just send me feedback directly. I'm keen to have this evolve. Okay, so this is what we'll be covering. Basic concepts of Kubernetes, the basics, the command line client. Then we look at running pubs. So if you know about containers, and you know that... The basic unit of execution within Kubernetes is a pod, which is one or more containers. So we look at that concept and running pods and what we call deployments. We look at how to perform and rolling upgrade across our cluster. And then how to expose our applications as services. There are several ways of doing that. And I will show just one way, which is the ingress controller. And this time, which we'll very quickly just talk about Helm, which is quite an interesting tool for Kubernetes. If you want to do this afterwards, then there is a page that talks about a set of options for just simply running your own cluster. The simplest way, of course, is to run Minicube. Minicube being... It used to be just a VM for running Kubernetes as a single node cluster within a VM. There's also a non-VM option as well, so you can just run it with Docker to simulate a Kubernetes cluster. Just one thing, so you'll hear me talk about traffic and so on. I mentioned that I work for containers, for the creators of traffic, and we're hiring, by the way, for remote workers too. Okay, let's get started. Okay. Don't be put off by the fact that whatever we're showing you when I execute stuff, mainly I'll be doing it within the Jupyter environment. You don't have to worry about what that is. It's basically a notebook where you can mix markdown and executable code, and so it's quite a nice way of documenting things as you go along. So I'll start with some slides and then go back into the notebook. Kubernetes, first of all, what is it? It's basically a cluster manager managing a cluster of containers. It could be a whole data center that you're going to manage with it. The principle, of course, is that as we move towards microservices, we've got more and more containers running in our data center, and it's becoming impossible for operators to take care of all the little details of deployment and you really need some sort of deployment platform which is going to allow you to say declaratively, okay, I want these services running. Kubernetes, go away and take care of where they actually run. And you want that platform as well to tell you, okay, well, no, I don't even want it to tell you. When there is a problem, I imagine maybe a node of a cluster that goes down, that would be a physical node or a VM, or a pod, a container, goes down, you don't even want to be informed. You just want Kubernetes to relaunch neural resources in the place. So if you know about Cloud Native and the discussion about pets and cattle in the old world, all software and VMs tended to be pets. There was a problem with the VM and you would go in and try and fix it and do your best to care for it like you would for a pet, whereas here we're much more in a scenario of cattle where a container dies and you just launch another one in its place. Okay, very different philosophy where we tend to declare what we want to be running on our cluster. So historically, Kubernetes, it's an open source project that was created by Google. It was based on their experience with deploying containers within their infrastructure, so notably the internal projects of Borg and Omega. And they really wanted to bring those same principles into an open source project. They donated this to the CNCF, Cloud Native Computing Foundation, in 2015. And a lot of companies bought into that as a container orchestration platform. And it's currently at the 1.9 release, surely 1.10, obviously. Okay, this is just a Google Trends just showing the tendency of the different container orchestration engines. There are more, but I just compared Docker Swarm, Apache Mesos and Kubernetes. We can see quite easily that the blue line, which is Kubernetes, has just taken off completely. And in 2017, we've really seen the effect of that with some surprising or not announcements from Amazon. Everyone was expecting them to launch Kubernetes services and they did that with their ETS and also Fargate services. To me, more of a surprise was Docker, when they announced at DockerCon that they would support Kubernetes as an alternative container orchestration to Docker Swarm. We've seen new managed Kubernetes services. So this is where you will pay to use basically a Kubernetes control plane, a dashboard to deploy a cluster in some other infrastructure. It could be out on Amazon EC2, for example. It could be on-premises, actual physical nodes on your site. So we saw new services launched from Microsoft and HPE and no matter how many others, they managed Kubernetes space. There are many other distributions. One of the early adopters of Kubernetes was Retat. Their OpenShift pass is based implicitly on Kubernetes. Others have been, well, CoreOS with Tectonic, who have just been bought by Retat. Canonical also have their distribution. And we see new tools. There are many, many tools appearing around Kubernetes. There's one which is looking particularly interesting today, which is Helm. She essentially is a sort of, allows you to do, create whole stacks of applications, a bit like a stack with Docker swarm today. But also with a catalog of applications that you can pull down, or full stacks that you can pull down. Okay, so if you look at the Kubernetes architecture, this is a bit small, this diagram, but it's really just to show three things. Basically we have master and worker nodes. Normally, in production at least, it's on the worker nodes that you actually run pods with containers in them. And the master node is the master nodes, are the control plane nodes that will monitor the workers and distribute tasks to the workers. There's also some sort of distributed key value store, so ETCD, which is used to manage the configure cluster, and in particular to say which of those masters, there's only one here in this diagram, which of them will be the leader. And so this is something that, I don't know what size Kubernetes clusters scale today, but essentially, as we're talking about, you might have a whole data center, or you might have several clusters in a data center. So if you look just a bit closer at the master nodes, there are three main components within a master node, apart from the ETCD demon itself. The API server, so most things you will do with Kubernetes will be either using command line tool like QCTL, or the dashboard, or some other application built from the API. That is the way that an operator will interact with the cluster. There's a scheduler and a controller. So as I said, Kubernetes is very much based on the idea of declaring what you want to be running on your cluster. It's very much a declarative approach. And in the same way, the controller is actually, think about 50 software control loops, each of which are checking some particular aspect. And they notice a discrepancy. Let's say you declare that you want 10 replicas of the Apache web service. For some reason, there are only nine. A part has died or a node has gone down and so on. The controller will detect that and it will tell the scheduler, okay, we need one more Apache web service. The scheduler then, based on constraints, if there are any, will then decide where that new Apache web server should be deployed, on which worker node. Are there any questions about that? So if we look at the worker nodes, a few more elements in there. So the main contact point is the Kubelet. So this manages the API that the master node will use to talk to the worker nodes. Obviously, there's a container engine. So typically today, it's Docker, but Rocket is already supported. We're seeing the creation of new container engines. So there is a specification, the CRI, container runtime interface, which is also being used by Docker already. And Red Hat have created one instantiation of that CRIO, which then provides a new container and time. So who will see choice in that area? The Kube Proxy, so... I'll open up the web pods first. The pods are one or more containers in each pod. They are on a separate network. So to get access to the actual applications running in those pods, the Kube Proxy is the one that will provide the routing onto those internal networks. There are various add-ons like Kube DNS to allow service discovery and so on. And there's the dashboard, which itself is an add-on. And in fact, all of those elements themselves can run in containers on a Kube system, as we will see. So the pods. I said one or more containers. So the philosophy, just as like with containers, the philosophy is really to have one or more containers in a pod. In a pod... Sorry, I meant one or more process in a container. In the same way, a pod will have one or more containers in it. The idea is that you should have, you know, one principal functionality provided by this pod. And if there are other containers, then they should be providing some supporting function to that main container. So for example, I don't know, could be a Apache web server again. Maybe another container, which is maybe Fluent D, maybe doing some distributed logging to elsewhere. Or another good example for a web server. You might have a container, which is doing a Git repo sync. So it will be pulling from a Git repo any updates, which are actually the static content of the website that you're serving up. So those are the two quite distinct functionalities, but the Git repo is supporting the main web server functionality. And those containers, OK, share namespaces. So these will have different process IDs within them. They will have the same IP address. They're sharing the same IP address, and they mustn't use the same ports and so on. So it's an abstraction, which is a little bit like a machine, conceptually. Oh, an important point is, as I say, a pod will be one main functionality and then some supporting functionality, but really tightly linked. And the idea is that as you scale, this is the minimum functionality that you want to scale. I want to scale from 10 to 20 web servers. What I need, let's say, the Git repo associated with each of those. So we'll see in a moment the command line client. I've heard about having it working from the command line all the time, but I should try and come back to the dashboard as much as possible because it's a pretty nice dashboard. So let me, rather than show you slides here, let me actually launch that dashboard. So, OK, Kubernetes dashboard. First thing to notice is within our cluster, we have three namespaces. When we run, we can create new namespaces, of course. When we run new pods by default, they will obviously go into the default namespace, so which is empty at the moment. I've not created anything. Other namespaces are Kube Public and Kube System, but if we can change into Kube System, and there we see already that there are some things deployed Kube DNS, Kube dashboard, so the dashboard is showing itself in the system there. We can interrogate the number of, well, replica sets, which haven't presented as a concept, and pods. So in fact, all that is a set of pods that represent our Kubernetes cluster itself. And in this example, I'm running an A3 node system, one master and two worker nodes, default. I'll go back there. Looks like I didn't do cleanup. OK, so as I mentioned, for this part of the lab, I'll be working with this three node cluster. It's actually a complicated environment, but later when doing ingress control, I will switch to another mini-cube-based environment, but I wanted to show a bit of working in multi-node. OK, so let me just do some cleanup. I see I did still have some things running. The advantage of the notebook is that it keeps it historic of everything I've done, so it allows me to archive with the whole tutorial, but to run it, it'd be better than I likely have stuff. So I mentioned the command cube-ctl. Normally I'd spend, or cube-cut all, and normally I'd spend time just showing a bit of the option so that, but I really want to try and keep them to one hour. So I do a cube-ctl get all. So basically with get, we can interrogate services, deployments, replica sets, and so on, which we'll be seeing in a moment. If we do all, it will give us those types of resources from the start. So if we look at the most basic way of starting pods in a Kubernetes cluster, the quick and dirty is cube-ctl run, and what this does behind the scenes is it creates something called a deployment, which itself creates a replica set, and then will create pods that will actually implement our application. So what are these concepts? A deployment represents essentially a deployment, so a deployment of a particular version of an application, and we see that when we do a rolling upgrade, this is the resource that will take care of upgrading from one version of an application to another. It creates a replica set, which is responsible for obviously creating the number of replicas that we've asked for. And in this case, I specified replicas equal to, so we have two pods. Of course, at a lower level, then each pod will have one or more containers. Okay, so I'm going to do a live-satellite hold. I'm going to do a cube run of this image, of this image, K8S demo, colon1, so colon1 is standard Docker image notation, saying it's version 1 of this image. This is basically going to create a deployment called the same thing, K8S demo. I'm going to apply labels. I haven't talked about the concept of labels. It's something that's very important in Kubernetes. Essentially, Kubernetes already applies labels to various resources, and you can explicitly apply labels, and it's a way of marking things. For example, if you're running on the same cluster, you're running development pods, test pods, staging pods, and production pods. I don't recommend it, but if that's what you're doing, then you could label, okay, I'm launching this just to test, and I would say, you know, I don't know, status equals test or something, and it would be a way of, later, if I just wanted to clean up, I could delete just the test pods, for example, not touching my production environment. Generally, labels are really useful. Another thing is, imagine the nodes in your cluster. You might have nodes with particular physical properties. You might have nodes that have a GPU graphics card, particularly good for the high-intensive compute operations. You might have nodes that have SSDs, others that have hard disk drives, and they would be useful for different types of applications. So you could use labelling to then select and say, when you run a pod, say, okay, I want to run this, only on nodes that have a GPU. Anyway, just to explain labels, as before, I will say, we want two replicas, and I want the pod to expose port 8080. I'm going to do a kubectl get-all immediately behind that, just to grab it in an intermediate state. Okay. So that first line, deployment K test, demo created, that's the output of the run command. And then the get-all, curiously, lists the deployments twice, and the replica sets twice, but I'll start looking from here. So it's launched deployment. It says here, desired two. It's saying that basically we want, I think that's pods rather than containers at this level. We've declared that we want two replicas of this pod. There are currently two. It's already launched them. They're up to date, but zero are available. Similarly, at the level of the replica set, there are two of them, desired and present, but zero are ready. If you look at the pod, we can see why. It's in the status container creating. So typically it's actually pulling the Docker image off the Docker hub. Probably not the case in this example, so I'd already done that, but still there's a little time of container creation. And here, the ready, we have zero of one. It's actually telling us that's zero containers out of one, desired. A particular pod specifies only one container. Okay, oops. Okay, I'm going to run that command again, which will be basically the same, except this. Now, we see two available, two ready, and of course, both pods are running with one container out of one. Okay. Just to show an example of the sort of input we get with kubectl. We can also explicitly interrogate the pod's deployments and so on. And with get pods, taking the option wide, we get just a little more information we had to hear before. This is the IP address of the pod that's created, and this is obviously the node on which the pod was assigned. So normally you run pods only on the worker nodes, though if this was a single node cluster we only have a master. So for development purposes, you can taint a node, allowing that to run pods as well. But if you were to scale to 10 pods here, we would see that they would all be distributed across the worker nodes, node one and node two. Interesting as well. I can see the nodes we have here. What I'm going to do, so there's a command called describe, which gives much more detailed information, a lot of information. If I look at those nodes, we can see the actual IP address of the nodes themselves, and we can see that, well, I know that these are slash 16 subnets. Knowing that, we can see that these are a different subnet than the pods. And in fact, I think that every deployment we create creates a new subnet specifically to those pods. Okay. So how can we access our applications? I'm going to use that information about the IP address of the pods. Again, this is not how you do it, but I'm working through to the different ways of accessing your application. Given that we have the IP address of the pods with a bit of bash magic, we can pull out those IP addresses. Okay. And then we can do a curl on the port ATHG that we were exposing. Okay. This is what we get. And the other one can't see both, but basically that's a different container name from the other one. Okay. What happens if a pod dies? I say, well, you know, we will automatically launch another one. So I just get to get the ID of the first one of those pods. And I'm going to delete that pod. I do quickly get pods behind it just to show, okay, we're in this intermediate state. So that first one, and then in S5, is already in the terminating state. It's not quite finished yet. And we can see that obviously the controller has detected that that pod is going down. Well, I actually would even know from the schedule. And it's creating another container in place. And a lot of the operations that would happen within Kubernetes will be on this basis of you want to make a change or you actually just, you know, kill the existing pods and launch others in the place. And we'll see that with rolling upgrades. Yeah. Okay. So now we just have two nodes again. But one of them, the new one, has only been running for 49 seconds. Okay. So we've seen how to launch pods via a deployment and replica set. Actually, cutting down the tutorial to one hour, just one point that I didn't mention is. So I've been creating those from just a kubectl run command. That is the quick and dirty method. The way you would create resources and we'll see it a bit later, is we tend to put those in a YAML file to actually, again, declare the overall set of resources that you want with your system. And we would actually do a kubectl create or apply of that YAML file. Which is quite nice because you can also, and when you want to stop things, you can do a kubectl delete. Or if you want to modify parameters, you do a kubectl apply. Okay. So when we start doing stuff here, I'm going to use a different cluster. It's no longer this multi node cluster. It's just a mini cube, a single node mini cube. And I'm going to show other ways of accessing our application. Obviously, what I just showed, is I'm going to use a particular pod directly. That's no good. I had to know the IP address of a particular pod to access it. And also I was accessing directly an internal network. So that's a no-no. So let's look at how we can use services to expose those pods. So the basic principle is you will have a service which is going to expose one or more IP addresses, so that some external user can get to those pods. Okay. And there are different ways of doing that. And it's important as well, not just because users shouldn't have access to a pod which is internal, but as well, because of this whole nature of when we change things, pods will come and go. We can't keep track of those pod IP addresses. Okay. So there are many ways, actually. At least a couple I haven't listed here. These are maybe three most interesting ways of creating a service to access our application. The first one is node port. Wow. I'll skip to the following slide. So the idea of node port is that you will expose the IP address of each worker node at a particular port that will be used to access one application. Okay. So, well, yeah, but that means you don't have to know the addresses of the pods anymore. You don't have to access that internal network, but you still have to know the address of the worker node, which might go away and be replaced by another one. And it's also no doubt an internal network as well. So that's really something that can be useful, maybe, for internal testing or something, but probably not something you'd want to use in production. As it says here, then, yeah, you'd also be using one port for each service to expose. It's also in this 30,000 upwards range. There are some disadvantages, but it can be useful for testing. Another way is load balancer. So typically, if you create a cluster on GKE, the Google Kubernetes engine, you could use this method, and an external load balancer will be created for you automatically, and you can use that. That's sort of like the disadvantage of this. It's great when you're running in some cloud provider and they will provide the external load balancer for you. And this will provide you an IP address, a known IP address, and this will be on some safe subnet that you do accept to expose. And then it will load balance between the worker node or between the pods, I should say. So this is good for deployment. But there's a third way. That's the one I want to present to you, which is to have a Kubernetes ingress controller. And this one provides more functionalities. So it will provide public addresses you can use to get in to the services running in the pods. The thing is that you will no doubt be having a large number of services in your cluster, and it becomes complicated to route between them. The concept of ingress controller allows us to do different types of routing to get to those services, like path-based recruiting. So it's an example where, based on the host name that we attack, it will be directed to a different service, and it's the ingress controller that will provide us that functionality. So traffic is an example of a reverse proxy load balancer that can be used as an ingress controller. That's how I'll use that in a demo. Traffic allows quite extensive combinations of host name, path prefix, sort of a URL path, and port-based routing to be able to direct to different services within your cluster. It also allows to do hot reloads of configuration. You don't have to restart traffic if you reconfigure it to add another service or to change an existing service. And it also has less encrypted support, so it can actually do automated encryption. So I'm going to go away from this previous dashboard. I'm going to close that. Okay, so this is Kubernetes dashboard of another Kubernetes cluster. So I said this one is based on Minikube, so we should see here it has just a single node. I think I have nothing running. Let me check that. Okay, so I'm going to show how to deploy traffic itself as an ingress controller. For the moment, I didn't show you explicitly, again, if I cleaned up properly. No, I didn't clean up properly, apparently. Okay, let me... Okay, so I just got a scripted demo. I'll talk you through the different steps. So I'm going to install traffic itself. And we see that's quite simple to do with the appropriate YAML file. I'm going to deploy some pods of an application that shows Jesus, as you have seen. Traffic is really into Jesus. And I will deploy a service for those different applications. And then I will set up different paths to those services. Okay, so the first step, I won't show the actual YAML files there. I've actually applied the configuration to install traffic. And now I'm going to create my applications, my services. Okay, so we have... I'm looking for the image. We have an application called cheese. I'm going to deploy that. So I've deployed three services, actually all based on the same image, called Stilton, Cheddar and Wensleydale. And we can see that those three services are running now. Okay, I actually still have... Okay, something wasn't cleaned up properly. So I have an Ingress running, but that's not going to create this problem. So I'll get an error here, I think, because the Ingress is already created. Okay, so it actually might work better than before. So let me come clean. I have a working version of this demo. And it was the first time I installed it myself. So this may or may not work. If it doesn't work, I'll switch to the working version that I already had. But it would be really cool if this works. Well, now let me change, because that's not... That's the working version. And this is the maybe working version. Okay, so just to say the address up there, which is tokyo and readall, is trafficui.miniCube3. And I have an ETC host sentry with the IP address of that particular node. Okay, so this is a traffic dashboard. I didn't mention that traffic actually can work with many different back-ends, so, obviously, Kubernetes. Docker Swarm, Apache Mesos, Nomad Rancher, and also some other back-end services like Consul, ZooKeeper, Eureka. There are a lot of back-end services. What we're seeing here is, okay, in terms of providers, we have one configured, which is Kubernetes. So when we configured it as an ingress controller, basically enabled this tab. What we're seeing is a number of front-ends. So these are actually the ingresses for the services I created. Now, this first one actually is an error. This was me trying to set up the system earlier. And then these following ones should be good. Okay, actually, I'm a bit surprised by the address, but okay. And these front-ends are going to basically root through to the back-end services. Okay, yep, HTTP, it always helps. That's fine. I don't normally use that address. Service and available. Okay, never mind. So I'll go to my working version then. So I'm cheating. Whoops. So I'll start with the dashboard of the other system that I have. Okay. This is more, to be honest, this is more than I was expecting of cheddar.mini-cube, Stilton.mini-cube, and the last one, Wensleydale.mini-cube. So these are the host names. Again, it's the same IP address. I have entries for these in my et cetera hosts. So now, if I try to access cheddar.mini-cube, I will be directed to the cheddar service, Wensleydale. Okay, and I think you get the idea. But I fancy that it's Stilton anyway. Okay. So that's, I just wanted to show you the idea behind an ingress controller where basically you can configure different routes. In this case, basically if the host is cheddar.mini-cube, then we're going to route through to the back end cheddar.mini-cube, and this is going to direct through to the pods. What traffic is doing here is, there are different ways it can be configured, but it's actually interrogating the Kubernetes API. So we don't have to configure traffic to say, okay, these are the services. No, it's traffic which has detected that those services are running. If I normally delete stuff, delete the pods, then those back ends will disappear. That's one of the advantages, at least of traffic, that it has this dynamic configuration reloading. And generally this is sort of functionality that reverse proxies ingress controllers will provide, but maybe not with such dynamic reloading. So in this case, it's very simple filtering. We're basically saying it comes in on your cheddar.mini-cube hostname with path slash, then go to this service. We could do things more extravagant. We could have a regex in there and this sort of thing. So there's a lot of possibility for quite flexible routing of different services. Are there any questions on that before I go on? Yeah. Traffic behind another traffic. Behind another traffic. So you have one load of reverse proxies and then you need to email in their own reverse proxies for their service. I guess you could. I'm not sure to understand the use case, but yes, a lot of the use cases we have are treating where a head is already been modified because they passed through different systems. I guess you could have a chain of reverse proxies. Does that answer your question? Yeah. The use case was more if there was a multi-tenant nice cluster, and each team controls the routing of the service. But then there's still a global. So, yeah, I think that would be the fifth possible. I tried to remember it as an idea for a demo. It would be pretty cool. Okay, finally. Go back. Let's say this is really the shortened version. This is the cheap one-hour version. Speaking of which, this is the first time I present traffic. The second time would be this evening. And maybe think earlier of, say it never happened to you. I got in a plane once and the pilot proudly announces I'm trying to put people at ease. Don't worry. This is this plane's first flight. So everyone's happy, except the engineers. I think, what? Engineering, reliability curves. It's the same. My first time presenting traffic. So a few hicks. Hopefully you'll go a bit better for the second time. See if I can debug in the MRT later. Okay. So I mentioned, I mean, there are tons and tons of tools coming out for Kubernetes. I even bumped into a guy this morning, the speaker room, who presented three new tools. Something that he's working on. Pretty cool. But there's one in particular, Helm, which is particularly interesting. So it was a company called DICE who started working on this and some of the tools. And DICE got bought by Microsoft, Plastium. And that continues to operate as an open source project. But there's real momentum. I mean, I guess DICE must have created the project two or three years ago. And already, in January, I think, there was a Helm summit. Wow. Okay, so what is Helm? It basically allows you to specify in a YAML file a whole application stack. And then those YAML files themselves are available on the hub. That's pretty nice. That means you can actually browse. I go, do I have it over now? Okay, let me go there. This is the Helm website. Down here, there's the option to get charts. I go over there. Cube apps. And explore apps. So if I want, I don't know what you're with this, I could even, I could even install traffic using Helm. But, you know, but okay, you can see a whole load of application stacks. There's a command line tool as well. It allows you to do quite simply Helm search. Helm fetch, I think it is. Then Helm install. Maybe that's not quite the plan. But it's really nice to be able to install complete applications across your Kubernetes cluster. Okay. So I'm going to stop there. So in the end that was a 55 minute version, not the hour version. So, any questions from people? Yeah. So let me see. So jithub.com, container orchestration, labs. Actually then it's Kubernetes, orchestration Kubernetes. And it's actually on a branch across Asia. And yeah, I'm interested in any feedback, any issues you want to open against the tutorial. This is a one hour version. I mean, could actually run for anywhere between one and three, before hours next time. Okay. Thank you for your time.