 Good day, ladies and gentlemen. My name is Antti, and it's really good to be here today. I'm here to talk to you a little bit about the evolution of our OpenShift architecture here at ELISA. Well, it was nice to hear the Asiakastieta mentioning that the company is over 100 years old. Well, ELISA is even older than that. It was founded in 1882, so talking about legacy is still... Well, a couple of words about myself first. So I work as a service manager at ELISA, like I said. That means that I lead the DevOps team. We are responsible for the cloud offering that we offer to our developers, the infrastructure behind that, and we also do maintain some tools and technologies and participate in architecture discussions with the devs. So in general it would be fair to say that we try to make the lives of the developers easier. I also get to input strategy, take part in partnerships, negotiations, do budgeting, and various other duties that managers generally get to do. But that's only my day job, so during night I turn into Linux kernel hacker. So I really do actually enjoy turning the bugs or oopsies that you can see on the right side of the screen to upstream commits that looks something like the left side of the screen. But that's just the way I'm weird. But yeah, coming from a kernel background and being this simple-minded low-level guy, I'm not ashamed to admit that I feel the containers are sometimes a bit overwhelming. The technology is rather new and it's evolving rather rapidly, so it may feel like it's hard to keep up. So in the next part of the presentation I will actually go through a little bit about the history, how the containers stack that we currently run at ELISA became what it is today. Well, I'm not going to go into the early, early beginning. I'm just going to start by mentioning that it all basically started when we tried to introduce Ansible to do software deployments. It worked rather well, so we had our existing virtualization environment and we did offer Ansible as the tool for the devs to deploy their stuff into production. They were really happy about it and eventually we got good adoption. But with good adoption came increasing amount of requests to have the ability to control the infrastructure as well with the Ansible. Well, at the time we couldn't really do that, so we took up and created another set of technologies that could really comply. And this one was and still actually is based on OpenStack, so you get API availability and you don't need to do tickets and so on and so forth. The thought was actually to run heat templates which describe the infrastructure and then run Ansible on top of that to really provision the software. So you could control it entirely. Well, this became immensely popular. The devs loved it and we were happy about that. But also during that time the container revolution happened. And after a little while we started to investigate what were the people actually deploying into production with this beautiful stack. And it turns out they were deploying Kubernetes. They were using that to deploy Kubernetes and then run their stuff on that. And actually this was fine in our opinion, so we liked it. We even endorsed it to some extent, so we created some installation scripts to help devs install Kubernetes because we saw that it would probably become the mainstream container orchestrator at some time in the future. Well, boy did they install it then. So eventually we started sprouting up multiple Kubernetes clusters with different configurations, with different sizes, with different features and with different operations. And if you, like us, are in an organization where there is separate team or department that is responsible for the operations and 24-7 and being on called UT, well, good luck introducing this architecture to them. So you will instead most definitely get this drake reaction from the guys where they will tell you that they would much rather operate only single instance of Kubernetes. Well, this turned out to be more difficult than we would have hoped, so at the time Kubernetes didn't do multi-tenancy basically at all. I don't know if it still does, but at the time at least it was really, really hard. So we set our sights to another container orchestrator solution that would be based on Kubernetes and would be compatible with the multi-tenancy requirements of our multiple teams, so we didn't want teams sharing the environment. Well, it's pretty easy to tell that the openshift fit the pill quite nicely, so it does multi-tenancy and all the requirements, and it's based on Kubernetes that the devs already loved. So eventually the evolution of the software stack that we offer looks like this. In the next part of this presentation, I'm going to talk to you a little bit about how the openshift installation itself has evolved during the years. So our first attempt at install it looked a little bit like this. Well, we were smart enough to do multi-master installation from the beginning, so we added a bunch of masters, and then we added a bunch of nodes, the sizes of which we did not have any clue about, but we made an educated guess. Then we also added a router node, which was used to direct the traffic into the cluster. Well, it's pretty easy afterwards, and looking at the picture, it's pretty easy to spot the single point of failure. So we pretty soon discovered that you can't take the router down for maintenance without bringing the whole cluster down. So another version of the cluster was set up where we added the backup router, which would be then set up using the keep alive D to switch over if something happened to the main one. And this is actually the version that we first offered to our developers. But unfortunately we didn't get much adoption, so nobody wanted to use it. And when starting to look into the reasons why, it became evident that the devs were actually educated well enough by our information security department. So they were really worried that if they deploy their stuff here, it would be entirely exposed to the evil internet. And they didn't want to do that for the working progress stuff. Well, looking into this some more, we discovered that hey, we can install another pair of routers that is connected to the intranet, basically the office Wi-Fi, so that if you install your stuff and if you do certain type of configuration, you will end up exposing your service only to the internal network of the company and not to the whole wide internet. At the same time, we also changed the default so that if you didn't specify where do you want to go, you ended up using the intranet. So for information security reasons, that kind of eased the pain a little. And this is actually the first version that saw some limited production use from some smaller teams. But still the adoption rate wasn't at the level that we were really hoping. And again, looking into and asking the developers what's going on or why aren't they using this, it became clear that most of the teams actually have some external dependency that they would like to access. External database or external API or external whatever. And they also stated that they would not want to share the access to the external service with the other users of the cluster. And at the time OpenShift couldn't really separate the external traffic between the projects. But we were in luck because OpenShift 3.7 came along and we did a tech preview feature called static egress IP networking. And this allows you to basically allocate a static IP for each project that can access the external network. So that way you can set up access control lists on your databases or external services that allow only certain project inside the cluster to use it. And this one was actually the setup that worked well enough that we started to see adoption rates in the development department that we were hoping to see. In general, by the way, I don't recommend utilizing the tech preview features in production because there were some pains in setting it up. But luckily with upstream collaboration and collaboration with Red Hat support we were able to make the feature work stable enough for our users. Actually stable enough so that with the usage increasing we ended up sharding the routers once more to really have the bandwidth available for the increasing traffic. And this is pretty much the architecture of our OpenShift where it stands today in our data centers. But when I say data centers I really do mean it. So we ended up discovering that it's rather nice to have multiple ones set up for HA reasons. So if you want to take or do upgrades in one of them, it's pretty nice to have the data center number two available for operations at the same time. So we ended up setting up another cluster into a data centers. We also did it like so that the developers have the chance to choose which one they would like to use. So if they wanted to use data center one, they could or if they wanted to use the number two, they could. We obviously gave strong recommendations to use both at the same time. And that was kind of what we wanted them to really do because that would have allowed us to do the maintenance things rather nicely. But even evidently they did not. And when I asked that why don't you use both of them at the same time, it turned out that it's really hard for them to do load balancing between the clusters so that you could expose your service from both of them at the same time and have traffic flowing even if the other one was taking down for maintenance. So DNS round robin didn't really work for that. So most of the browsers for example don't really cope with that sort of mechanism for balancing the load. So we went to the drawing board with this issue and came up with a handy little piece of software that is an integration between OpenShift and our existing load balancing infrastructure. The piece of software actually listens to OpenShift API events related to road creation or updates and can provision external load balancer when needed. So if you set up route in data center one, this one will create it to the load balancer and start listening events or traffic from there. If you then add the same route to data center number two, it will be added to the load balancer pool as another destination of the traffic. This piece of software was actually announced to be open sourced in the last OpenShift common scattering in San Francisco last May. But if you still want to take a look, there is the GitHub URL where you can find it. Okay, well this setup is pretty much the one we nowadays run, but going even further we discovered that the workloads between, or there are different kinds of workloads that people want to run. Basically the difference that we ended up discovering is that there is production workload which you expect to be rather highly available and highly responsive and it's generally quite predictable. But then there is the development workload that people do compilations or load tests or unit tests or AI training cycles that is generally hard to predict, but luckily can usually wait a little while so you don't need to immediately service the request of doing some load test for example. So we set up two more clusters for the development use and they changed the policies so that if you want to do some development you could only request or create a project in OpenShift dev cluster and nobody would be stopping you. Instead for the production clusters we ended up creating policy that you need to request the project to be created for you so we could kind of track the resource usage a little better. And this is finally the architecture of our all data centers as it stands today. Well, you may now wonder why do we go through this all so why go through the trouble of setting all this up and why such a massive operation is even needed and the answer is rather simple so we have made it our strategy to increase the cloud technology utilization. We hope that it will be able to do automation with smaller iteration which in turn enables faster learning for the dev teams. That's quite a valuable thing to get. We also aim to shift the responsibility of end user experience to the teams themselves with the cloud technologies. A good example of a team that has been following these guidelines is actually the Elisa Viihde Entertainment Services video on demand store or vuokraamo in Finnish which at the moment runs on top of OpenShift. And being a sizable operation as it is, it already merits for several data centers to be used to get the kind of store up and running all the time. But that's not in my opinion the most interesting thing that we do on these clusters so I'm gonna talk a little bit about the coolest stuff that we currently are working on which is the self optimizing network. Being a telecom operator, the complexity of radio networks is kind of mind boggling. We have a single base station which may have multiple cells, let's say 2G, 3G, 4G, 5G is coming up. And each of these cells may have multiple antennas directed to different directions. And each of these may have somewhere ranging from hundreds to thousands parameters that you can fine tune to kind of create the optimal coverage for your cell phones. Now imagine if you would for let's say cost saving or performance reasons which to load balance your users between the cells or towers. You adjust a small parameter in one of the antennas to get more traffic directed to it but usually it happens so that they come from the neighboring towers or neighboring antennas. Now these ones need to be adjusted because they have different usage because of the one parameter that you changed. And when you adjust these, the next ones need to be adjusted so you end up with this cascading ripple effect that covers your entire network from a small parameter change in one of the base stations. And we figured that this no longer is a task for humans to do so we created neural networks and AI learning machinery that can actually do the job for us. So they try adjusting the parameters every day as often as we can run them in production. Well we are in luck as the algorithms themselves are pretty much nicely parallel level. So you can run multiple of them at the same time and for that reason they are quite handy workload for well container orchestrators. So if you are actually interested in learning some more about this, well in either order. So if you want to learn more about the product that we are trying to build around the self optimizing network stuff, visit elisaautomate.com. And if you are truly interested in becoming an expert on the field and working with the technologies over there, go check out elisa.fi's last jobs. We have several openings available there. But with that I think it's time to say thank you all and very nice that you had me here.