 So, I know we're finished, but we're not finished and that's a bad joke. What I was hoping and most of you, how many of you know Reddit and AMA? Ask me anything, panels, all right. So we've got three folks up here who are, one who is a pseudo gamer, game refinery on his off time doing work at game refinery, but also a Red Hatter. So there's a little expertise here beyond just production use as well. But you, this is your opportunity to ask them anything about their current production, their future and all of that. And I know, so I have a couple of questions that I wanted to ask after listening to their talks today. And the first one for auntie, auntie like auntie, can you tell us a little bit about what's in the application stack under SON, so what are you, you want to recruit people? So you know, what cool technology are you trying to run there to do all that AI and machine learning? The AI stuff is actually mostly Python, so it's rather simple stack. We tried to keep, or we have discovered that the simpler your containers are, the more kind of scalable they are. And it's Python and TensorFlow, so for the machine learning stuff. So have you looked at the Kubeflow images? We have been investigating and I was really happy to hear that they are becoming part of OpenShift possibly in the future. That was I think in OpenShift, or Red Hat Summit in San Francisco was some talk about that. We've made a number of, if those of you who don't know Kubeflow, the TensorFlow community along with the Kubernetes community have created another open source upstream project to make TensorFlow and other, not just TensorFlow, but other things run very nicely for the ML stacks. There is, as I always say, an OpenShift machine learning special interest group and David Arenschek from Google and Matt Farley from Red Hat co-chair that, and you'll be happy to know that Kubeflow is first class now and you can run it on OpenShift. Can I have a question? Go ask a question. How many developers do you have? In total? Total. It's something around 120, I guess. Oh, we jealous. Yeah. I don't know. Diane is so interested about Python because her Twitter handle is PythonDJ. PythonDJ. Yeah. Okay, I have a bias. So yeah, I'm going to, you know. We became best friends. Best friends just now here. But that also for Arrow, one of the things that we were talking about after yours was one of your biggest challenges is finding resources. How big is the team that you're running with? Right. So, is this on? Yeah. Cool. So, like I mentioned, our IT is a total of 30 people just about and the guy is doing OpenShift by a significant effort. It's me and Jenny over there. She's the ops guy. I'm the dev guy. So. Two. Yeah, basically two guys. So, since Antti did his shameless blog, I'm going to do mine. If you want to do some cool stuff, then pick up the phone. This is what happens at community events. Shameless recruitment, all right, which is quite okay. But I think one of the things that you've just made this acquisition or merge with UC, and I heard the word mainframe used. There's going to be a lot of educational challenges and bringing people up to speed on what containers are. What are your plans for trying to educate that whole crew of new folks and convert them to cloud native container enthusiasts? Yeah. Well, that's the question, isn't it? Well, let's put it this way. They're pretty tired of the whole mainframe thing themselves. So they're looking for alternatives. And I think having the best solution is the first part. And then it's about, I guess, selling it to them as well. We have to get our own stuff together properly first, and then we can roll out. But I can definitely say they've been interested, because there's all these things I've been able to promise already. So time will show, I guess. Yeah. So I'll put a shameless plug in for learning.openshift.com and the stuff that we're doing at Catacoda. Just please take advantage of that, and people can learn, if there's any of these new topics, pretty much there's a learning session with online access to deploying and testing with learn.openshift.com, which has been really good. And also, mini-shift, if you haven't heard of that, as well, is a great way to deploy OpenShift locally and get started and have some hands-on experience. We're going to be hearing in the afternoon from Jeremy Brown about the Open Innovation Lab and some of the cool techniques they have for teaching as well. So that'll be good. Tara. Again, is there any questions in there? You mentioned that you used a hosted OpenShift from Biente and Ombrem. What were the main differences between those two OpenShifts? Well, I mean, I don't think there was a huge difference. I mean, it was pretty easy, because we actually migrated the testing environment for the GDPR service, and it was pretty painless, actually, to get it running. So I mean, I guess you have to be a little more careful with the ambiention one, because if you make a route to your program, then it's in the internet. With us, it's still beyond one extra firewall. Cool. So I had one question for you, Tara. You keep asking questions, but you talk a lot about externalize everything, or externalize as much as possible. What else is left to externalize? What else would you like to be able to externalize for your game refinery platform? One note first. We're also hiring. Okay. Actually, we don't need to externalize anything, but we can basically run whatever we want on OpenShift nowadays. Maybe not functions yet, but the serverless functions will be in multidimensional mode in OpenStone line soon. But if we see that something is better run elsewhere, then we move it like the MongoDB. We saw that we get the better performance. It's easy to manage. We get, let's say, like a new integration from there, we get a better authentication there. Then it's just for the business, it's better run elsewhere. It's not that there is a rule that you cannot run this on OpenShift, it's just, what's the best way to do it, we just do it. You're advising to externalize everything, but both of these guys have lots of legacy 1800s, 1902, or 1905. Are you allowed to do this sort of externalizing at Alisa or at ... Can I make a correction? I mean, externalizing, if there is a service that you can use, use it, don't build yourself. That's what I mean, externalizing. Yeah. Alisa is actually pretty good at building themselves. Thank you, Tero. So are you able to take advantage of externalized services yet? To some extent, but obviously a telecom operator is quite interested in the data and it's actually regulated as well. So not all stuff can be externalized in the field. Both of you are in highly regulated industries too. So that really has some repercussions for what you're allowed to do. Right. And for the Swedish credit information, for example, I can foresee it might be a problem to move their stuff in Finnish servers. But I'm not really familiar with the legislation. Don't worry, they'll get you up to speed fast. Yeah. It's just going to screw it up once and I'll know. Yeah, you'll know. Right away. All right. So are there any questions in the audience here? One way over here. My God. Thank you. Yes. If he doesn't have a Finnish accent, I'm going to have a ... We need to have a svak for best questions. Hi. I'm Juolena. I have a question for Eero. How do you prioritize picking the applications that are re-done into cloud-native environment? Do you pick the low-hanging fruits from the point of view that is easy to migrate into cloud-native? Or do you want to have operations load taken up from the operators for them to focus on other things? Yeah. So I think we're going with generating more revenue first. I think that's the arrowhead. What sort of new business can OpenShift give us rather than going for the cost savings? That was the question, right? So doing 24.7 for some services will bring that in and being able to respond to heavy loads very quickly is probably something. We've lost one or two, like a stockman who looked by that type of thing. There would be a lot of credit checks coming in within one or two days, but currently we can't promise we can handle that load. So maybe once we can scale into an external cloud in real time, that will be generating a lot of income for us. There are any other questions? There we go. Oh, God. We're getting them warmed up here. It's the get-up battle. So I'd like to ask, what kind of a task it was to set up the on-premise OpenStack environment from the scratch? What kind of hardware you have and any known pitfalls that you would like to tell about at this stage? How much time we have? Really? Oh, we got 15. Yeah, go ahead. Yeah, okay. That's a pretty wide question, actually. Well, the task was rather difficult, yes. It's not, well, maybe I should say that at first we even didn't run any Red Hat version. We installed the community version. And, well, that took a lot of kind of understanding of what the components do and what they can use for and how they should be configured and lots of trial and error. Well, luckily, since we have our own data centers, we have quite a good amount of hardware readily available, so it's not a big issue to set up some sort of cluster to do test installations or some such. So we ended up kind of, well, guessing a good set up of hardware that we could try on. Well, we got it up and running, but pretty soon it was kind of clear that maybe the supported version would work better for our environment because there would be a need for operations, capability, and, yeah, well, the community version isn't well documented, so it's really hard to know what to do to maintain it and keep it running properly. So I don't have an estimate here of how long or how much time or how much effort it took, but I could say that it was significant. But actually, it was good to kind of do the exercise runs with the community edition, in my opinion. So now we have a couple of guys that are really skilled at, well, maintaining the cluster. So in that experience with OpenStack and using the community edition, you mentioned that you had contributed to the upstream in your talk about that. And I'm supposing that your experiences with OpenStack allowed you to contribute back to the OpenStack community feedback, probably on their lack of documentation or other things. But can you talk a little bit about, because you've also open sourced your load balancer integration with OpenShift? Thank you very much. What it took for the cultural shift for Alisa to start working in the open source communities? Well, in my opinion, that's still a work in progress. So not all kind of are working with the open source model. We are trying towards it. And I guess the DevOps team is leading the way in that area because we work a lot with the open source technologies. So that way it became natural for us to kind of run or lead the way. I should also, by the way, mention that we have contributed to Kubernetes and OpenShift as well. So we have collaborated with the upstream directly too. Yeah. And I think that's one of the points that I'm trying to make as well, is that it's not just OpenShift that we want you to do. You've contributed to OpenStack and to Kubernetes and lots of other places. And all of that gets fed into, you know, we all benefit from that. But I'm curious culturally, like how you worked with management to get them to allow you to open source, say, your load, the steps that you had to take, maybe legal and convincing management that this was a good thing to release this code into the wild. Was that an issue for you, Alisa? Well, not really. I am the management. And as I said, I work with kernel development already. So I'm pretty familiar with open source development myself. So there wasn't really too much convincing to do on my direction. So I get to work with all different market sectors and not all of them are Linux kernel developers. And so in highly regulated industries like finance or banking and stuff like that, there's a lot of issues in terms of them being allowed to disclose what they're working on and pieces. Have you been contributing back, Ero? No, we haven't. We haven't. Is there a cultural issue or a legal issue of why or is there any reason, any barriers for you to contribute other than lack of opportunity? No, I think we've been busy getting things running and didn't have the time to focus on, you know, the other stuff yet. OK, we'll get you there. We'll get you there. Actually, the Finnish culture is, like you said here, it's difficult to get questions. So people may be a bit shy to do their contributions also. So that may require some encouragement. Yeah. And this is me encouraging you. But I'm also going to say that contributions come in lots of forms. Your questions or contributions, your stories or contributions, documentation support, changing, you know, grammar, checking, all kinds of ways you can contribute back to communities. It's not just code contributions or open sourcing projects. So I really encourage you to think about when you're worried about whether you're making a code contribution or you want to be a committer or something like that. Make smaller steps. Your feedback is really one of the biggest contributions you can give to any community, whether it's OpenShift, Kubernetes, OpenStack, or any of the myriad of other tools that you probably are using. So is there another question in the audience? Oh, my goodness, you guys. All right, there you go. I know we're going to make everybody run around here a bit. So question actually 20 about the Liza clusters. So how much are you using the normal workload compared to workload which needs to be optimized for the hardware? So like high-packet network transmission or some hardware accelerators or some protected from noisy neighbors. And overall, how much of your clusters are on top of OpenStack and how much are bare metal? OK, thank you for the question. Well, let's start with the one that is easier for me to answer, so all of the clusters on top of OpenStack. And that also sets some boundaries at the hardware accelerations that technologies that we can use. So we generally don't do those as much. We have been looking into GPU processing for the AI training cycles, but that's not done in production yet, but it's been planned. But that's pretty much it. We do have a separate cloud environment for the telco stuff that's more like networking heavy or needs certain latency requirements. And that's probably, I think it's using DPDK for the acceleration, but I'm not too familiar with the technology that it's running on currently. So I feel like I'm adding all the bookmarks here. So if you're looking for information about OpenShift on OpenStack on bare metal, there's a great number of blog posts and videos and briefings by a gentleman from Red Hat called Jeremy Eater, E-D-E-R. He's done some great work on performance. And Ramon Alvarez from the OpenStack team at Red Hat has a great briefing on the OpenShift Commons briefings, which is again on YouTube at RH OpenShift. If you Google either of their names, you'll find lots of information on the bare metal question that you've asked. So I'm not the expert on it, but anyway. So Taro, you've got a day job. It's at Red Hat. We know he's a Red Hatter. How do you manage switching gears between your day job and the game refinery in the evening? And where do you find time to do all of that? I travel a lot because of I'm a Red Hatter. So during the hotel, during the evenings, from basically from 9 to 5, I don't touch the game refinery. I might get some slack misses. But then after I spend an evening at the hotel, then I concentrate on that. But how big is the team that's actually behind game refinery? There is two front-end developers, one full-time again developer plus me, not doing that much. And then some management and then each analyzer who actually play the mobile games and analyze the games. So about 15. 15 people. So it's interesting because what we have up here is we have an open stack open shifter. We have an online open shifter. And we have an on-prem open shifter. And none of you are using dedicated yet. So we've pretty much got one of each of the variations that you could think about deploying OpenShift on. And I'm wondering, from your perspective, because you're such a smaller thing, the benefits of running on OpenShift online and having that operations team behind you, how that's played out for you. I would say that it makes things a lot easier. I know that because of my day job, I know what it needs to manage an OpenShift. So we don't have time for that. And since we started from day one, we started from OpenShift. So we don't have any requirements that we need our own OpenShift. We need cluster admin rights. We are happy with the Sandworks that we have. So we are not even thinking about having our own cluster, because we don't basically have ops. We just deploy containers around the application. So it's a no-brainer for us to run online. So are any of you doing hybrid deployments? So running some on-prem and some, you talked a little bit about externalizing at your company. Yeah, right. We're looking at that at the moment. But so far, it's on-premise. We have this philosophy of doing stuff on-premise. Perhaps it's due to some legislation or something like that. But yeah, we've definitely spoken about moving workloads to an external cloud as well. All right. All right. Any other questions from the audience? All right. One in the way back there. All right. It's always on the opposite side from where the microphone person is. Yeah, I bet they're coordinating. Yes. So if you could just pause for one second before you ask your question. A woman just walked in the back of the room who we have to say thank you to before she takes off and goes and organizes another one. Tanya Repo here has been the main organizer behind this whole day for the past three months while you are all on vacation. Let me just tell you. Finns take vacations. But Tanya, really, thank you very much before you take off. Thank you. All right. So please, your question. Question to Antti and Elisa. Are you considering running this hybrid cloud like OpenStack and OpenShift together? Have any idea of that? Together. Well, our OpenShift is installed on top of OpenStack. And some workloads do actually run on OpenStack alone. So we don't run them in OpenShift. That's basically because we started with the OpenStack-only version. And OpenShift came along later. So some workloads were set up in OpenStack. And we haven't migrated or don't see the need to migrate everything onto OpenShift. OK, thanks. All right. Are there any more questions? There should be one in the front, because we're going to make him run that. Apparently, same back. All right. Well, I want to thank our three, oh, one more. Oh my god, they're waking up. The fins. This is wonderful. You're breaking a cultural stereotype today. Well, yes, you showed the last one to ask. I wonder when you started this use of OpenShift and you discussed with your people on how they will work in the future. And of course, it makes some change into how the developers have to behave and how maybe business have to go on. Did you encounter some kind of new issues, like, oh, we have been doing this before. Why should we now suddenly change? Or was that already clear for everybody what's going to happen when you take this new system into use? Just generally, some thoughts on that? Oh, we're going to make you talk first. OK. Yeah, obviously, people have been used to working in a certain way. And with the changing cloud environment and cloud-based kind of production model, there have been some kind of issues that people think the cloud works in a certain way and then doesn't. And they feel that this is wrong and not easy and all that. That's why, actually, we do have the DevOps team at hand. So we try our best to help with these pitfalls before they happen. My key philosophy is that I try to teach people to embrace that things might be broken at some point. So please do engineer your solutions so that they can withstand certain amount of brokenness. OK, my turn. I told them to do so, so they do it. No. Actually, we don't, the developers do development on their own machines. They don't use opens if they don't, or even containers. They don't build containers. They don't do Dockerfiles. They run Jbos, sorry, Wildfly, in local machine, they run their HTTP server on local machines. Once it goes to Git, then it's container. So developers can do whatever they want as long as it compiles during the build. So it's easy. So there is no cultural mismatch that now we are developing for OpenShift. Now we are just writing code. We are not developing on OpenShift. For us, I think there's definitely two things here. First of all, I was talking about how we're going to make it incremental and easy to get into the OpenShift. I think that's one critical part. And the second is what Tero was talking about in his presentation about being the champions, having the champions. I think, I guess, I'm the dev champion of our company. And I think our organization is sufficiently small to the point where I can just go and talk to the people. I know all 15 of us, and I can go and show them why it's better than what we currently have. And so I think it's a bit easier than with, say, 120 people. It's going to be a challenge with the new acquisition, though, to scale yourself to do that for the next 300. Agreed. You might be training hiring some champions. Yes, training champions. And hiring. You'll be doing. So that's all the time we have for AMA. Thank you for your questions. Much appreciated. We're going to now thank our panelists for sharing their stories with us and their production use cases and ask them to keep coming back. And as they change and grow and add new things into their projects, we'll have them at meetups and upcoming events. And we hope to hear from all of you as well in the future and so that you'll share your stories too. So thank you and thank you.