 It's quite impressive to be here today, and I'm actually quite honored like Yeah, two and a half years ago when the open shift commons was introduced You know, I was one of the first commons calls that Diane always so nicely Moderates and they were like, I don't know 20 people roughly. Yeah, and ten of them were redhead guys. So We were joining that and in 2014 we were doing a lot of stuff with quote-unquote the old version of open shift and Yeah, we we went through quite some challenges with that version to be honest and in 2015 you know, we were basically saying hey We make the right decision to go with open shift because all of our large customers with two systems and those of you Who don't know T systems? We are the enterprise arm of Deutsche telecom They were they were making the decision to go with Cloud Foundry, right? So the diners the The Bosch's of the world they were saying okay Cloud Foundry and then in in 2015 You know Chris Morgan from the redhead team and Ashish they came around and said hey We're going to do here something with the Kubernetes. We completely re-architecture it, right? Is it okay that that looks promising and thank God we we stuck with the with the version now things are really Turning into the positive and we see a lot of customers coming in now. Let me introduce please next slide So let me introduce myself real real quick, you know carrying a decree of Informatics how we call it computer science and also an MBA And like everything that's that's fast and that has two wheels So we probably need to put two wheels on an open shift so that I even do that in in my off-time two kids that keep me busy in the off-time and one little baby That keeps you busy. That's called a bad job. That's our product that we wrapped around the open shift and That we're working on for like now three years almost Next slide, please You know what I have Chris right already mentioned a lot of the points, please. It's animated You know what makes the life of a developer real miserable? You know the provision of environments still these days in the enterprise take weeks You know if a physical hardware is involved and especially around a big data Environments, you know, it still takes 12 to 13 weeks till a developer can log on to something and Install his his stuff on it in 12 to 13 weeks. I mean, you know developer has gone through. I don't know five six sprints already so that shows how Things need to change here the data center folks Chris also mentioned the data center folks They think they are the real IT guys, right and the developers think no, we are the real IT guys we write the we write the the stuff out there, but the the data center folks they say we get up at night at Two o'clock, you know and and makes things work again when when things break and Next point is big data Seems to break the bank, right if you think about, you know terabytes and petabytes and These days we talk exabyte of data these things get, you know expensive, you know from the get-go and The licenses for commercial products are, you know, you know, they're they're breaking the bank, too So and and I have a slide or a couple of slide on the on cloud performance when it comes to big data You know, and I you know if you know those couple of slides will in the end undermine or under Right mind and build the foundation of that cloud performance really sucks for productive big data workloads And Obviously in the enterprise data needs to be protected, you know, we all we live here in Germany We we have this data protection data privacy legislation, which you know kind of Prohibits the cloud adoption But things are moving and we need to take care that People who are using the cloud are, you know, feeling comfortable with what we can offer to them And then when when things are developed, you know, the fun the whole fun starts over and over again So next slide, please so what we what we came up with and A year ago my Yeah, my boss asked me hey Thomas you have been so successful now with With open shift and with platform as a service. Don't you want to take over also the big data? team as well and I was also scratching my head and said Let me think about for one weekend and then I remembered what happens to me in the early days We had a couple of smaller customers that approached us and and they had in mind already, you know, big data Use cases they want to develop mobile apps that are attracting a lot of data So they were asking us Thomas tell us about The throughput the performance of your of your underlying cloud So we gave them the numbers and they said now we have to walk away with this. This will not suffice our needs ultimately so that was really that was really, you know You know make my heart bleed at the time because if you have to go have to let go away a customer that really Sucks So when I said, okay, I'll put things together Pass platform as a service and big data because then we have both things together and I'll have under An underlying infrastructure here that support both, you know the cloud on the left-hand side for smaller volumes when we talk about You know gigabyte sizes, so you know provision that fast so the developer can can start using it using containerized technology Using technology from cloud era from Hortonworks map are and They can try or they can try it out And and not break the bank because if it's if it's not working They have not spend a lot of money if it's working great They can stick on the same platform and and scale out For for big volumes, you know, we say this is for for multiple terabytes We were using the same platform Ultimately and for huge volumes multi petabyte We then say we we cut the we cut the platform and move you out to a Real bare mathal platform. What does that mean? I give you an example car manufacturers are these days all You know thinking about autonomous driving, right? They they have that in mind and for test driving From a legal perspective, they need to document that they have, you know, done their diligence So a lot of video sequences are coming in a high resolution So that they can document what what happens with the car in a specific situation So that needs to be stored somewhere And with a car manufacturer here in the southern part of Germany He's talking to us and we're running a proof of concept with him Of you're storing exabyte of data. That's something that really, you know, you can't do that on the cloud anymore So you need to have an hardware on-demand model Which we have and I will talk to you a little bit about that next slide, please So as I said, you're the underlying infrastructure, you know, we started with With the cloud from VMware. Why did we do that? Because it was right there, right at the time two and a half three years ago It was just there. We had that as a t-systems cloud Today, we are actually offering the same services on Azure and OTC OTC stands for open telecom cloud, which is a an open stack based cloud solution And on the right-hand side the bare mathal cloud Really using on-demand Hardware where we have the agreements with our hardware vendors that are bringing in the hardware as we do as we need them And on top you see a lot of the things that come with with OpenShift anyway, but we have enriched that with Things like the cloud era With cloud era or talent for ingestion or sauce those guys are really now coming to us and say, okay We want to be part of the game here and they they provide their Applications in a containerized version so that the developer here on the left-hand side can already use their technology and Make use of it and then see how they can how can leverage that technology, please next slide Now We run some yeah, some some proof of concepts and some some tests with it, especially around the Big data Hadoop ecosystem what we've done we we used for this proof of concept. We used hdfs obviously for From Hadoop for the file system method used to spark in Various standard flavors Scala Python are hyphen tests And we used for the metadata storage. We used MySQL part and you in an extra part and While you deploy it you can say how many how many slaves you want for the Hadoop cluster And you can even You know say you want to have it persistent or you want to have it ephemeral And that that system that we've tried out is really use is really designed for small data users For development users next slide, please so you see that Up there you see the TSI AF which actually stands for in the former time We were called at fabric we had to do away that name because that's a that's a Microsoft server name product So we had to call it ab agile and you're all familiar with this kind of setting and So we created the Hadoop quick starts on it if you go to the next slide So this is where you see how spark is being started on the ab agile Hadoop master and You can see still where we are on the on the on the on the line here And please go to the next slide So this is basically one of the use cases where we Implemented the whole the whole Hadoop platform Created Some some databases and uploaded publicly available data to it and did some research So this is a nice showcase to see and tell the developers what they could do Next slide please When I asked my my architect, could you in preparation of this of this meeting here? Could you please send me some nice fancy architectural diagrams? He sent me this thing. I said is this it? He said, yeah, it doesn't it doesn't get more complicated. So Basically, you know the Hadoop master Part and then my sequel part for the metadata and then the slaves you scale it out So pretty simple, but it took us, you know quite some quite some time until it really worked It was like I don't know two three months until until my guys really figured it out And they had it run on a on that environment next slide, please Now This thing is really some you know some sort of a break in the presentation You see this abadge all logo here, you know where open shift is really 80% of it And you have Azure on the right-hand side now these two arrows between those two mean that Number one we run on Azure now Number two there is this concept of a data trustee with With Microsoft where Deutsche telecom runs the Azure platform in Germany, right as their data trustee so that means The the data center is a Deutsche telecom data center where Azure in Germany runs The people are T systems people who are running it who are running it so no one else then People who are bound to the data Legislation that is valid in Europe and Germany are actually accessing this platform, right and then I Think at the beginning of this year red hat was announcing the partnership with Microsoft implementing the Open shift also on on Azure So that now we have we have all forces in in one boat, right? Microsoft Azure us is the operator an open shift as the de facto You know development engine So next slide, please. So what we what we were trying now is to Really underline what we what we said that Clouds for productive big data workloads really, you know kind of sucks. So we were using a Hortonworks Deployment that was offered on Azure, you know as it is as a standard HTTP 2.2.5. I'm not mistaken The you know we have laid out four core engines With 14 gigabyte of memory each each server for the for the workers 200 gigabytes of disk So next slide, please The deployment of the whole thing took 15 to 20 minutes pretty fast for a for a whole Hortonworks Big data environment. So that's fine. We access the Emboury We access the environment through the Emboury shell or from the SSH from the internet to the master, right? we took the Ubuntu SNOS and as I said HTTP 2.5 and choose a replication factor of three Those of you who know a do fs. That's that's a standard, right? So we had like 266 gigabyte of usable storage next slide, please now What we always do is we do we use a Terra sort to test the performance Usually you go there with a terabyte of storage And and you sort it in you know check how long it takes to to sort all that data Now we don't have enough space So we have only 266 gigabyte of Storage So we said okay a 50 gigabyte of Terra sort should use like 2.8 percent of the time And you know did all the all the math around that so in theory This Terra sort on a small scale should take roughly two minutes of runtime until it's done, right? and You know one one side note down at the bottom here You know we had only one local disk which usually is not very optimal So we had some some bottlenecks now here next slide, please now this is really the the outcome right we had like 80 percent weight IO on that on that environment and during the during the map phase And during the reduce phase On that machine really no reduce could could run right so basically the outcome was and next slide Yeah, this busy obviously if you have to wait IO this busy is another You know another sign of waiting next slide please So actually it took like 33 and a half minutes, but we have expected like two and a half minutes to be so that means Okay, you would have to either add more hardware more spindles to it to run the same workload or You just wait longer right if you want to have if you want to don't want to spend enough money or more money Just wait longer, which is not an option really these days So next slide now My you know my presentation is entitled to you know be called okay, we have Big data and pass as being you know siblings tied at the hip but also we think it's you know, it's also an enabler for DevOps in the enterprise and Over the last two two and a half years if we really were struggling with with dev ops the thought of dev ops In the enterprise, I don't know who of you you know is working in a company with the 500 employees and more Hands up. So you must know what I'm talking about right? so the the ops guys still are Out there following the the itel world, right? They have to make changes The changes need to be approved Having elite times That really you know prolongs you know Getting things into production Now next slide please What we came up with this list is this little Propeller where you have on top the customer Customer want to be fast right he has the developers inside Waiting till the till the environments come along and You have the engineering guys, you know, they're agile their service oriented And you have the operations. They're following the itel itel processes and in the beginning, you know, you have People working together. That's nice, but at some point when the first release is out You know things get difficult They need to be operated at night as I said and You know the the ops guys still you know go back to their Used behavior from the past and they say if you want to if you want to come out with a new release you can but You know six weeks from now is the right time now what we Implemented is a concept of a we call it a deaf ops engineer for lack of a better term which builds actually the the layer between Really the developers and the ops guys so those kind of guys speak the language of the ops guys But they also know how developers think so if the developers come along and say yeah, we want to Deploy something that needs to have some Changes on the platform those deaf ops engineers are talking to the ops guys. So that concept we have deployed at Tall collect maybe you guys know that the road charging system here in Germany is going through a large re-architecturing and They are actually having like four people on that deaf ops engineering level and after they have that implemented the whole system got very much calmed down and They they tell the guys how to use the right how to use the platform And if they have requests they are filling in the requests to the ops guys they talk to them So that the screaming got a lot a lot less noisy in the environment in the system, right? So that's about it big data and and deaf ops from a T system standpoint, which is called a batch up Thanks guys