 So this session is going to be called members talk back the great thing about the open shift commons is It's it's not it's not sort of a traditional sort of a foundation as we're seeing a lot today where it's mostly vendors This has been made up sort of from the beginning of Obviously a bunch of technology vendors because we're we're trying to get towards common standards and common best practices and forth but it's also made up of a ton of of Working groups and meeting groups of sort of end-user customers as well as a bunch of Systems integrators and other folks who are helping us scale this so what we brought today was a Set of people from those different backgrounds And so we hope we can give you guys a good sense of how the project is working from end-users Perspectives from folks who are trying to do systems integration along vertical accesses Telcos service providers and also the technology vendors as well And again same as the other panels We're gonna throw out some topics for these folks to talk about if you have questions raise your hand We'll get you a microphone and we'll kind of go from there. So I'm gonna do a quick introduction I'm gonna kind of introduce all of you and then I'll let you guys do a quick introduction Andrew Randall CEO from Tigera Jart take the arson from the and I'm gonna get this wrong So let me just let me just give you the basic from the tax group of Norway You're in charge of all the taxes of Norway That's right, and I would pronounce that so badly. So I'm gonna skip that Darren Ratcliffe from the cloud group at Atos and Josh mayor from Accenture from the financial services group of Accenture Andrew, why don't you start just quick introduction and what your role is and as far as working with the open OpenShift Commons community. Sure. Yeah, so we set up Tigera because We we saw a lot of issues with how virtual networking had worked in the in the VM world and open stack and we really wanted to kind of start over with cloud native and do it right and So we do with this with an open source project called Calico The initial integration obviously in terms of cloud native things was was With kubernetes and we've been working in that community for a long time and more recently been Been looking at well, how do we actually make this a really seamless smooth experience for OpenShift users as well. So we've actually just in fact, I just heard today. We've got got that merged into the OpenShift project so hopefully be there in 1.5 and shift origin and You know, so it's pretty early days for us in terms of the engagement on the open shift side But we've been in the kubernetes side for a long time, right? So That's that's that's our role cool so I work in the government in over Norway or in the IT partner for the tax like the tax authority in Norway so we We are currently developing our new platform to run and develop application. It's based on OpenShift container Platform the latest release. We've been doing this for 16 months or something since the 103 origin release or something So it's We kind of just creating we're using OpenShift some glue on top to integrate with our Existing infrastructure and our existing demands because we got lots of them. We're a government so we have lots of QA processes and QA teams and Lots of different environments for all the applications like the guy from Volvo said they have like They have test and QA in production. We have like seven steps for something that we need to go through so We have lots of things we need to do but we we try to automate as much as we can Hi, I'm Darren from ATOS. I work in our technical strategy part of the business within our business and platform services We're working jointly with Red Hat on our customer base where we're helping them to help deploy OpenShift platforms Particularly in the area of operating those looking at how they scale those how they integrate them into the ecosystem so Working with the guys that weave as an example on how you integrate in Visualization and operational tooling that helps them as part of their CICD process Also within ATOS. We've got a large contingent of developers and we're helping to help them understand How you start to use these sorts of platforms to build Applications in a more cloud native microservice sort of way so that we can support our customers Not just only in the delivery of OpenShift platforms and Kubernetes But also in the actual life cycle of the applications as they build those to address their new business value They're looking to create Josh Meyer and so I lead a group within Accenture that's focused on cloud enablement within the financial service industry So that's a a small part of a much bigger group within our overall businesses focused on cloud but Through that collaboration we've done a lot with both with Red Hat around OpenShift but also directly around the Kubernetes so we've deployed some fairly large Kubernetes clusters and We're involved in the moment on my patch with a number of our financial services clients around Significant transformations of their applications into OpenShift And and I particularly look at quite a broader set as well I've been working a lot with Red Hat in the broader sense also around the cloud orchestration topic I think a lot of our financial services clients are Still in you know a transformation transition phase and so there's still a lot of Workloads that are not going to live in OpenShift and how do you orchestrate all that together So that seamlessly you can start transforming and playing with apps that are in OpenShift as well as outside That that's yeah, we're doing a lot of that stuff. Very cool So Before we jump into questions one of the things that I get all the time is I get a chance to get out and talk to the marketplace as people go Kubernetes cool It's it's been out there for 18 months two years and it's an open-source project and it moves really fast and is anybody using this and One of the things about working for a public company is you can't usually disclose all your customers But there is these fantastic services that talk about people that are hiring for jobs that are more than public So I like to use some of that data talk about sort of who's using Kubernetes so that people have a sense of like are there other people Putting their jobs on the line are there important services that are being delivered and I kind of frame it up like this The financial services industry trust this planes trains and automobiles trust this the the media industry trust this the travel industry Trust it big manufacturing trust it and on and on and on and It's kind of a it's kind of a nice thing. It gives you a sense of I'm not alone in this Which obviously we're talking about here at opened OpenShift Commons But it also gives you a sense that as you're deploying Kubernetes you're deploying OpenShift there are people who are solving similar type of scale problems Solving similar types of how fast can I deploy problems? Mobile problems big data problems and so forth. So it's nice to have a sense of you know, how big this community is starting to get I thought what I would talk about with these gentlemen is Really kind of these four things we've been talking about all the different stuff We wanted to hit on kind of wanted to talk about these four topics and guys I don't want to put any pressure on you But we're sort of like right at that perfect lunch coma the cycle. So I need some good answers I need some good discussions and again questions from the audience So we're gonna look talk a little bit about what their experience has been with OpenShift Some of their experiences with decision-making, you know what the people in process and squishy, you know Meatware part of these types of challenges. How do I move to these new things? We're gonna talk a little bit about fast-moving projects the joys of upgrading every three months If that's the cycle you want to keep up with and then we're gonna talk a little bit about You know in an OpenShift context, when should I use my own project i.e. Prometheus versus what's embedded in the stack and when is it a good decision and why did that decision get made and How should you think about that? So lots of stuff we could talk about we're gonna kind of hit on these four things Guys, I'm gonna start sort of to begin with give me a sense Give me give me an experience from from each of your perspectives What what you've seen in terms of working with OpenShift whether it's you know, how your customers have adopted it How the technology has rolled out both in terms of you know positives and some challenges But you know any I'm gonna sort of throw these out to you guys as a thing So jump in anywhere you want to rather than doing the up and down the aisle First so actually why don't you start as the sort of end customer you've got the most scars in this Why don't you give us a sense of what your experience has been so far? Yeah, well Our experience is mostly positive about this We have some part of organization is more skeptical than other parts of it that natural I think it's healthy to be a little bit skeptic about new technology, but we We we have a good experience running this We're very happy with the with the way like the building blocks of OpenShift like the scheduler and the API server and the API is very flexible. It's easy for us to integrate our external services in We use the same like the same technology that use internally with labels and listening to labels and creating controllers for integrating our own stuff like we have mostly we We we kind of we are forced to use Oracle for storage because that's what we do Currently we want to kind of Do more but as of this moment Oracle is what we do So we integrated that into into our projects So you just say I want an Oracle database with some information about it and we created for you and we inject it in So with via a secret and a config map. So So the API is very flexible. It's very easy to integrate your existing stuff into it And that's really important for us. So it works really well I'd add maybe if I look back at the probably last will five six years where a lot has been happened in the past space The difference between maybe a lot of the IAZ experiences versus the containerization past spaces I think there's been a lot more positive experiences very quickly Whereas a lot of people had very big frustrations early on with complexity of open stack or other big deployments and And a lot of lost money and investment in projects that maybe didn't quite deliver the value They wanted whereas Paz has often really delivered value particularly directly to the business much much quicker And certainly developers have taken on to this, you know very very quickly even some of the earlier versions of OpenShift Before it was using Kubernetes and some of the customers that are actually using chemis But I think the struggle has been and where the past base has really matured a lot is the operator side Where I'm starting to see a lot of the operators now really get much more value that they feel like yes This is making my life easier as well Whereas the developers have kind of been convinced for some years already In fact, I've seen a couple where developers have started out You know the story was always sass was being bought on a credit card by the business I've seen developers buying pass on their credit card to bypass their infrastructure folks, right? And so the interesting thing is I think developers were convinced already for a while And now we get in the operator train really involved and they're starting to see the value in some of this But that's also bringing the broader complexity of some of these certainly our customers very very large organizations That deal with security regulation other things that need to still adhere to yeah And you think part of that is because Containers are now sort of that common language that that the dev team and the ops team can both sort of talk about as opposed to Ops being sort of a black box in previous paths Yeah, by the way, I'm not good. I don't know what the others agree But I'm not sure we're quite there yet All right There's still a lot of organizations that are struggling culturally with that, you know developer operator mix and what's the if you want to call it the kind of common rule book In the New World I'd say there's still a lot of work there and that's cultural change and for me I often tell my customers that The technology is the easy part of this right is the cultural things. That's that's really difficult to get it, right? But I don't yeah, yeah, I was just gonna I was just gonna agree I think that My experience of working with customers with regards to implementing kubernetes and open shift is that the Technology isn't the main challenge right the the introduction into an organization The necessity to change the processors. We've just heard from Volvo The whole culture shift, you know reshaping your development organization Those sorts of things are the more challenging spaces so to be able to go in and help a customer holistically Helps with the adoption of the technology I would say that the the speed of maturity with regards to kubernetes with the way that how quickly the the gaps have been closing You know so the challenges and the questions that we get back from customers with what about this? What about that you heard earlier from from Clayton, you know our bike and And all those things that are coming into the into the core kubernetes and then also heard from Clayton how The the baking period between a kubernetes release and an open shift release particularly in the enterprise Release of open shift gives it that extra curation time And then the commercial support backed off to red hats or to take care of any issues, you know I think we're all helping with regards to Putting a mindset in place for the wider adoption of containers moving forward Yeah, yeah, so I'd echo all of all of those things I think another dimension that's being I found really interesting over the last year or so has been how fast the topic of hybrid cloud has really come in and Not perhaps not in the sense of people originally envisaged it of, you know application spanning multiple clouds but but with the the desire to be be deploying applications that can move from one cloud to another so that They don't have a lock into the to the cloud provider and and I've certainly seen for instance, you know large financial customer who Probably, you know two years ago. They would never have thought of moving or core application code into public cloud and I've seen them doing that with this this technology and You know we we're providing the security piece that enables them to really authenticate that you know what they think is Talking to them from the public cloud is coming back into their the core database, which is still on premise You know is is trusted But you know the ability to To build that in a kind of container orchestrated container Environment was was really key to moving it into in fact in that case that application is is movable across Two different public cloud providers, so I think I think that that's that for me I've seen a real kind of minds mindset change and That customer as well another kind of anecdote here. I guess is you know, we first helped them with a proof of concept probably January February last year and You know that was a small initial upfront engagement just a kind of a bit of consultancy really to help them get that Get that going and then they came back you know within I would say within seven or eight months You know which is pretty fast right from a from a POC they came back and said okay This is now our strategic platform for all our applications going forward And so you guys have better work out how you're gonna help us support a production environment come and train You know more than like the one engineer who knows about this because our whole team is going to need to understand this And so it really you know that that speak you know for a large tier one financial to go from POC through to this being the key strategic platform in less than nine months. I think speaks to How you know how fast this is really getting adopted and and how easy it is to adopt as well Yeah, and I think and I think we hear we hear that from red hat perspective We hear that and probably 60 70% of the the customers we engage with it's how am I going to get this to also work on Azure on a Ws on GCP on you know somewhere other than my own data center for for a bunch of different reasons So we see that pretty consistently I wanted to talk a little about decision-making and Josh. I'm gonna start with you I thought it was sort of interesting you get into decision-making and people want to talk about things like Conway's law I'm gonna give Robert a little bit of a hard time. He talks at a lot of our events You know there used to be this thing with Volvo where the joke about sort of the Volvo thing was their boxy But they're safe and I was watching his deployment thing his sort of before and it dawned on me Their old CI deployment thing was it was boxy But it was safe and the nice thing. It's kind of cool is walking with that his after picture is now We use open shift we go faster and that very much starts to align with where you see Volvo starting to evolve even in How they build their cars. They're no longer sort of square and boxy. They're much more stylish You know safety is no longer the thing they're looking at so that Conway's law thing does sort of come into play from time to time Use the analogy for the crash test dummy So Josh I want to talk about you so you mentioned that that you know Relatively technology is is the easier part decision-making changing the culture. Give us a sense You you work quite a bit in financial services. How do you convince people who for years were worried about compliance still are But are now worried about I can't move this thing fast enough to give you enough features to be a Competitive, you know financial services Institute. Yeah, and and I think the interesting thing is actually there is there there's probably There's a there's a lot of guys in the financial services that are absolutely they already right and so it's kind of there There's almost an internal battle sometimes with different parts of the organization you mentioned the compliance guys I think The the compliance officer is often the the biggest blocker right and and not necessarily because there's there Certainly not a technological reason right but but often because they have a certain risk framework that they have built out over years and Although there may be a right thing to change it actually that's gonna be a lot of work for this guy and his team right and so Some of the elements of a constraint have nothing to do with technology the other aspect is just you're in a lot of cases Certainly in financial services many of the companies have spent many years Actually split and apart their organizations right so in many cases they even have separate legal entities for their Application development side and their infrastructure house Well, suddenly you're asking these guys to work extremely close together and there's been a lot of turf wars around well You know I want to build this and so I'm gonna build it in my my entity and then the other entity is saying no No, I'm gonna build this and I'm gonna build it over here so they've built the same thing and Suddenly you're trying to make a decision on well, which one of these two am I picking or so so some of this has nothing to do with Because those could be valid options both of them right, but it's it becomes almost more of a political thing And we know given the way the political landscape of the world is at the moment politics always difficult and tricky right you know Navigating that and I think that's what we're really dealing with it's some of this has you know, it's easy For those of us that get to play a lot in the kind of tech space I love getting back on my on my tech with my guys, right? But but sometimes I have to deal in that almost diplomacy world and it's that's tough That's just you know, there's no civil bullet for that stuff. You got to go through and educate people around well What are we trying to do here? Actually, how do we still have the security and resilience stuff that embedded in I? Spend a ton of my time with CISO is actually saying hey guys The cloud and everything we're doing here is more secure It's actually you should be embracing this whole scale You should be the one champion for this all this because it's going to make your life easy Yeah, but that's hard so it's almost like we you know, we talk a lot about value mapping when building applications You almost have to do sort of their business model mapping or their org chart mapping to figure out like We're gonna be my bottlenecks to sort of make this evolve make it so that app team and dev team can talk together Whether it's gonna be a legal problem. It's gonna be a culture problem and and so forth. I would thank Yeah, I also think that the the drives coming from a different direction these days as well I mean everybody talks about the disruption in business from startups and And all those sorts of things and you see a lot of the the thrust or desire for a new applications coming direct from the business Waking up and recognizing that somebody else is eating their lunch who six months ago didn't exist as a business and and to a lot of these More mature organizations who have all these mature processes in place that that's a really difficult step change So so the introduction of these types of technologies becomes an absolute necessity and often they stand them up in Parallel to their existing business. Nobody's gonna flick a switch overnight and move their entire trading business into You know into containers But they will start to build the new types of services to counteract the disruption Using these services because they have to because the speed that they need to introduce them The speed that they need to keep the change flowing in those sorts of things necessitates the need for these sorts of platforms Yeah, now you guys are all across Europe In back in the States we tend to people want to emulate what happens in Silicon Valley Do you see Europe has a lot of little pockets? You know the UK has financial services the the Nordics have a ton of software background from from days past Eastern Europe's got that Manufacturing in Germany. Do you see certain pockets where people are trying to emulate a region? Or do they take advantage of the the resources in their region to help get over some of these things? Absolutely, but but I would actually argue. I think certainly my in my patch in financial services. I'd argue Our European clients are leading so I have a global role So I also travel a lot with our US clients and our European clients are leading. I have European banks who are Building rebuilding their core banking into microservices on top of paths. I don't see any of my US clients doing that Right, so the reality is actually I'm not sure there's this whole lead out of Silicon Valley going on here I think Europe is certainly financial services have really taken a charge in the last 12 18 months my clients have gone absolutely wild in this topic and you know For various different reasons, but I actually think some of the regulatory pressure happening in Europe certainly financial I'm talking about financial services I know I'll let the other guys talk about other industries But the regulatory stuff is pushing these guys over the edge and they're having to GDPR is coming in And you're gonna have to have a better system than the old stuff and actually it's easier to maintain if you are on something Like OpenShift You're in the software you're in the software to find world but on the network thing which yeah So we're based in San Francisco. So that's right. So, you know, we do a lot of work with the Silicon Valley You know web staff, you know large-scale web companies and You know, I think a lot of them are leading in many in many ways But I see just as much innovation out here and and I think we see you know broadly, you know The activity in Europe is is just as strong in North America, but you know, I definitely see the organizational challenges and What what what I've seen is where customers being most successful is where they've literally broken apart the silos created a cloud team and You know that cloud team is tasked with me, you know making this new technology work and you know that that Where that where that hasn't been the case at the projects in very big of the rails, right? So you've got to have an ability to to do that locally in the culture. Yeah. Yeah, you had a question Yeah, yeah, we'll repeat it To On the one hand, if you standardize your continuous integration, including security scans and the deployment, through this standardization, you can create security baselines, which makes it easier than for a variety of products. All providers have a better infrastructure than most companies. On the other hand, we have in the last years here in Europe a lot of deep dives into practical securities from our regulations. That means nowadays a regulator will come, buff in or the EZB, European Central Bank, and visit you, and they bring in your experts into your house. And then you see also that the old good systems are not so shiny at all. So this affects the whole bandwidth of installations you have. So now you have to see what is the best path through all these diplomacy. I perfectly agree, but on one hand, which is really blocking, is a legal aspect, which is often underestimated. At the end, a regulator wants to have the potential possibility to seizure you or your data. And this is especially very difficult in the cloud, especially for banking services. Trading is a little bit different, but for health data, for clearing, which is at the end a large bank, it's very, very difficult. And there we are looking for solutions for sure, but this is a legal aspect. Yeah, Diane, I think he just volunteered to be the head of the regulatory SIG. I really spent a lot of time. I'm physicist, so I see it as a multi-dimensional problem, and you have to find an optimal path through it. So diplomacy is one part, and the other part is really where you have to work together with the regulation offices here. It's definitely the case that... And a lot of work, in any case. Yeah, a lot of the compliance standards need updating to take into account these new technologies. We have a lot of conversations with the security teams that come in and validate designs that we'll work on with the cloud architecture teams, and they'll say, okay, great, so where's the VLAN separation? And we try and explain, no, we do much finer grain policy-based traffic isolation, and this is why it's so much better, and where's the VLAN? Section 16.something needs... I need to be able to identify the VLAN. Yeah, so... And I think the frameworks are in place for these standards all to be extended, but that work has still to be done. It's interesting, and to that point around the regulators, it's interesting that if I go back a few years, the regulators really wouldn't have been bringing a consulting or SI player like myself into their offices on a regular basis, and I'm getting a ton of calls to fly out to Latin America and all over Europe to speak with the regulators because they're recognizing now actually they need a lot of education, a lot of advice to understand things like open shift, because suddenly they're realizing, well, actually I'm trying to regulate all these financial services firms, and I don't understand this technology, and the technology has fundamentally shifted the environment I'm trying to regulate. And so I actually find that a really interesting day. Absolutely, I think the regulators are there, and a number of them, I mean, some of them have gone public, like the Dutch regulator, I would argue is probably the one ahead of everybody in the world, and they're really open and kind of embracing a lot of this stuff, and I know I can quote them because they quoted themselves publicly, so I'm not getting in trouble there. But I really think this space has moved so quick in the last few years that actually that is why the pace of adoption around things like open shift is just going crazy now at the moment, right? Because a lot of these barriers are being broken down. What I see interesting as well is that there's the increased collaboration and the speed even that Red Hat is rolling open shift compared to the old days of REL, right? The speed at which you guys are bringing stuff directly from Kubernetes into open shift, that shows as well just the pace of the stuff that's going on. Now I think the security space is the really next one where we're going to see a lot more stuff being built into open shift. I know you guys have done a great job already working so that's not, but to me I still think there's some additional stuff to come there particularly as we look forward to more regulation, not less, right? So I'm going to skip ahead to the next sort of topic here. We all sort of come into these events and we pat ourselves in the back because there's twice as many people at this event as there was in Seattle and as many as there was in San Francisco and in Austin. The expectation is it's going to be twice as many. We also all went through the days when VMware would release once a year and you struggled to keep up with and then the open stack community was releasing every six months and people were struggling to keep up with that and we're now releasing every three months and Docker is every couple of months and people sometimes want to beat up Red Hat because we're going to announce version 3.5 and 1.6 Kubernetes is going to get announced and they go, why are you two or three months behind? What is wrong with you? I'm curious as you guys take, how are you keeping up with all these projects that on one hand like Clayton talked about, we want to make them stable and boring and on the other hand we want to keep up with them in terms of new features. Give us a sense of how do you deal with three month upgrades? I think some of the challenges are addressed because Kubernetes and certainly OpenShift are abstracted a little bit further up. You touched on VMware and keeping up to date with VMware's releases and so on and so forth. That was so intrinsically baked into the physical infrastructure itself and the lack of things like software defined networking via Calico or anybody else. That made it an incredibly difficult job and a highly disruptive job to just keep flowing that infrastructure moving forward. I think with the tooling and all of the processes that are coming from Kubernetes itself and from Red Hat around OpenShift, the ability to be able to roll forward infrastructure as opposed to upgrade infrastructure is a key attribute so that you've got this whole traceability to roll back in the event of an issue. The whole fact that you can do rolling upgrades as well. These things were traditionally difficult in an old traditional virtualized world. The challenge as a service provider is how you feed that confidence level into the consumer as well. I think with things like OpenShift sat on top of Kubernetes and it being predominantly a developer experience, the developers are going to start to gain a level of confidence with how they're deploying their application, how they're distributing their applications. The knowledge that they'll gain from using that platform will help them understand that the platform itself rolled in an upgrade underneath it is not necessarily hugely disruptive to the application, assuming that they've distributed their application in a correct and thoughtful manner. I think all of those things help now with moving together at the pace of change of the technology. I think we'll always, just like Clayton said, there's a soaking period between a Kubernetes major release and an OpenShift major release from a service provider perspective and a consumer perspective. You can expect a similar sort of pause while they choose to roll that forward. There has to be sufficient technical change in the product as well to warrant wanting to move, not necessarily move to 3.5, maybe they wait to 3.6 because they're waiting for some specific component. It doesn't always necessarily need to follow the same release cadence, so long as you keep the whole roll forward model going. My perspective, we've been using like OCPs since 3.3 and rolling upgrading an hour from 3.2 to 3.3 and 3.4, and 3.4 finished last week. Of course, sometimes there are issues, there will always be issues, but then we file a takeover of that, we fix it, we use like the standard playbooks, a little bit configured for our needs and what we need, but when we find an issue, we just take it up and it's fixed and hopefully we contribute so that others can do it. I'm a big fan of if something hurts, you should do it often. If you upgrade often, then you know how to do it, you know how to handle that something can fail. So we kind of started this kind of thing. We want to follow the releases that come, but of course we first deploy them in one of our testing clusters that we don't run in production, and then we test things there, and we of course investigated everything will work as expected. Sometimes we need to hold back to fix some things, because sometimes there are changes, but it works pretty well I guess. Yeah, on the aspect as well of choice, I don't know whether that's particularly in project or on the kind of major release type stuff. Am I skipping to the next question? No, that's okay. But I think that fast pace of change, these two really relate, which is there's a number of customers that are making that decision and saying, well actually I've got some stuff further upstream that I prefer, and then I'm swapping that out of the stack. The key for OpenShift is modularity, I think, because there are certain customers that want some of that stuff that maybe isn't the one that the main part of the community wants. I think that's just got to be the ability to, that's got to continue to be there, and the trick is of picking some of those, that's always got to be down to the individual use cases. But I agree in terms of actually the fast pace of change, personally I think it's a good thing, because as you say, if you actually do it often, you get better at it, and you manage all that in a much easier way. Certainly I'd say OpenShift upgrades are actually easy, and then as we go in the back, and some of the things you're also saying, right? Yeah, and I'd love to get folks feedback. I know we have active discussions about there are things we've done in OpenShift for a while, so templates did certain things about how you described applications, how this sort of evolved is the thing within CNCF that people are using more and more. We're having open discussions about, do we make that to default, do we make that an add-on similar to what we're doing around CNI, where you can plug in Calico, you can plug in OpenVSwitch, so I think we're also having those active conversations. It's around monitoring, it's about charts, it's about a bunch of different areas of what should we make the default, what should we look to CNCF to be the project, even if it's an early incubating project, and I think we've got to figure that out. I think we would love to get feedback from people as to how early do you want those pushed back into them, when would you like to see us wait for them to become stable and sort of align with the stability of OpenShift. So any feedback you've got, feel free to give it to us sort of after this. We'd love to hear some of that. Yeah, I think one thing we have to always be aware of when you're talking about new technologies is, technology adoption curve, people we work with so far, early adopters and the customers want to be at the leading edge. That's why they're getting involved early. But I think we're kind of reaching that inflection point or the chasm where we're getting to the point where it's much more mainstream adopters and they just expect, this is maybe a good segue into the next area, they're just expecting it to work out of the box and they don't necessarily want the latest release all the time. But I think one of the things that this community has done really well is actually set a pretty high quality bar for each release. I don't hear a lot of moaning and whining about major issues with each new Kubernetes or OpenShift release. Mostly the mainstream things work. I saw some comments recently about how the Kubernetes community talks about alpha and beta feature labelling. The Kubernetes alpha is what everyone else calls a beta, the Kubernetes beta is what everyone else calls production ready. I think we have been pretty good about keeping that quality bar up and there are other projects out there that release fast and break stuff and don't necessarily go back and fix what they're broken and that does create some bad blood. But I feel really good about this community that we're in here. I'll throw out one thing, there'll be an announcement. Is it going out today about what the work that... We did a bunch of work, Red Hat did a bunch of work. Last KubeCon, we just finished some work this KubeCon around scalability so people ask all the time. I know it's got Google DNA and Google Scales to WebScale but give me some details. We just published another 1,000, 1,500 node. 2048 node scalability OpenShift Kubernetes paper that will be out on the CNCF page so if you're curious about some of that, details about it, all that's got published today. Guys, I'm going to wrap up. I'm going to give you all the last word, I'll just go down the line. The last bit of feedback for these folks about either working with Kubernetes, working with OpenShift, taking on a new project, just any guidance for these folks in the last minute or so that we've each got. Go ahead and start. Josh, you can start, or Andrew, you can start. Either way. Okay, I'll start. So I think one... Certainly in terms of experience with my customers, it's a lot quicker to actually get involved with this and adopt so I think increasingly less talk and more action will actually prove to really get you going quicker. I think A, don't underestimate that cultural element of this that is really difficult. In terms of the mindset change. But I think the reality to the point maybe on that scalability, one of the things I'm really excited is we've got some massive scale on a couple of the particularly Kubernetes and now increasingly on some of our OpenShift deployments that we've had with customers and that point around the fast-paced versus the kind of resilience or scalability elements. I think Kubernetes at the moment has it really right and seen large scale without too much difficulty. We're used to kind of being brought in because it's complex and difficult and have to clean up a bunch of mess and actually there's been a lot less mess to clean up with some of these deployments. I don't know. Personally, that's what I would bring from... I think the community is moving along at a good pace where we actually got something that is relatively stable. Darren? Just adding to that, because I agree what was said. I think the other thing is that the adoptions are easier and easier and easier of Kubernetes and OpenShift. The work that the community has put in around how to adopt these tools is incredible. The quality of the documentation, the actual automation tooling around bringing these services up is so quick and ultimately what you get is an incredibly powerful tool. The other thing is that the ecosystem that wraps around Kubernetes and OpenShift is getting stronger day by day by day. We've talked about Calico, I mentioned Weave. There are tons and tons of organizations that are building value-added services that when you plug that in to the core of a system all of a sudden you finish up with this solution that you feel confident with that you can give to your developers and then your developers can start to build true value at a pace that they quite literally have never been able to do before. Yeah, I can just echo what Josh said earlier about the culture change. We see that as well a lot, but the technical aspect, there are issues there as well. The culture changes what hits us most, really. For instance, our ops guides are used to patching. They have patching strategies to show patch. So often, now we come here when Docker is here, it's not the same Java. We are in a Java shop. You can't patch Java in a Docker container. It's very mutable. And they say, what? Then how do we patch? So we need to figure out how to do that. Many people have solved it. We have solved it, but it's kind of something you need to think about. There are new issues as well. But I would just recommend everybody to go play with it. Use Minishift. Use OZ Cluster App or anything and just go play with the same API because it's really powerful. Yeah, so maybe a slightly different way of looking at things. We talk about the community. We talk about open source and all the great things it brings. I caution people not to think that open source equals free. It's going to take a lot of resources. And I see people thinking, I'll just download these projects, put them together and try it out. And hitting a wall fairly quickly. So write a check to our friends at Red Hat to help you out. Write a check to us, whatever. But come and work with the vendors. And I think don't struggle on your own. And I think there's an expectation that an open source community will give a level of support that it's not really geared up to. And that's where folks like Red Hat come in. Gentlemen, thank you for your time and your insight and your expertise. And we are on a break, I believe. Yes.