 Okay, let's start another talk. The topic of this talk is HPC VOD transcoding on OpenShift. This talk describes setup for publishing subscribe application, deployment model usage of Gluster for persistent volumes and challenges for such non-standard scenario. Presented by Frederiko Naibolo and Alessandro Arcello. Sorry. Quite good. Hi everyone. This will be a talk quite after the course because it is not a developer talk. It is the transposition of a demonstration. We run to a customer for a real use case of implementation of OpenShift and Gluster storage. I will ask you to leave for a while the head of developer and think about the implementation of Gluster or OpenShift from an enterprise perspective which is quite challenging on the social part, mostly on the social part. The agenda is quite simple because we will analyze the five WAs who were, what we did and when and why we did it as an enterprise customer experience. The first thing, the first W is who. He is Eric Alessandro. Yeah, I'm a platform consultant. I'm actually switching to solution architect team. I basically work on configuration management at customer side with for example PAP at Foreman Ansible Technologies and now I'm focusing on OpenShift, OpenStack, CloudForms and so on. So basically with Frederiko we are at customer side letting work also configuring our technologies for customer purposes. Yeah, and I'm here for that and I'm working on, I've been working for distributed calculus at CERN and INFM and now I'm working on OpenShift, OpenStack and Gluster architectures where this project was built. It was built in one of the, in the Italian branch of one of the largest European broadcasters, which has about 21 million subscribers and is expanding, it's online streaming and on demand market with 10 million connected customers. So it was quite a huge architecture. Just to say, just to look at which kind of data we are running through, this is a Gluster storage console and for example they have that sort of media content on Gluster, which is quite a huge size. So for example if you think about issues that can arise, that can rise in, on Gluster and you have a failure on some of the bricks and you're moving something like happened a couple of months ago. You have to move, you have to recover after a failure 40 terabyte of storage. Assuming the hypothetical 10 gigabit network you will take about nine hours to move the data from one place to another. So it is and you cannot use of course the 10 gigabit at all because the environment is on production itself. And in the meantime the customer is complaining about it. Yeah, that's a side effect of working on that data. So this is the target size. But what we are talking about in this talk is about an OpenShift application which is used for customer encoding workflow. Summarizing that application in a few lines as I reported in the slide. The broadcasting content coming from broadcasting signal requires encoding activities to encode the content for all the different devices and size and frame rate in order to be dispatched to the customers. And this is an application developed by external system integrators. So the integration of OpenShift in that environment as you will see briefly was not so simple. The workflow itself for encoding resemble a traditional job scheduler. So we will see in the next slide there's this patcher that takes segments which are media content from the broadcasting signal and transformed it and encoded it with standard FFMPEG or similar programs. So the job scheduler is something like a traditional master agent architect or maybe some of you work with some of those technologies. Those are the technologies I've been working with in the high energy physics calculus. And those are the standard schedulers used in the last years. For example, MOAB, Torque, which was formerly known as OpenPBS, which was an open source fork from PBS or LSF if you use the IBM commercial product or HTConder if you use the one developed by the California University. So this is a traditional job scheduler software installed on a single machine where there's a concept of a master. And the concept is that the job is run by a small portion of the application which is run on an agent. The agent waits for comments from the master, executes the jobs, which in our case is an encoding activity and then returns the exit code back to the master. This is another graph showing you, you can't see the number, but this is a graph taken from the Kibana interface of the job, of the encoding job running per hour. Each segment is a media content that has to be transcoded. And as you can see, the lines are quite similar in the input and in the output. Job are quite unpredictable because they're coming from the broadcasting signal, so we have no control on the sort of jobs that are coming and the rate. The overall workflow summarized in a graph is that broadcasting, broadcasting publishes a content signal acquired by a traditional graphic card. Then there's a legacy Microsoft app because the driver for this card is, it exists only for Microsoft, which is called WFM, which realizes streaming segment, which is usually an MP4 content. Then this content is transferred to the hyper dispatcher, which is a legacy Java application, which was out of the scope of our port into OpenShift, and this was one of the challenges we have to cope with, because we cannot move this part of the application itself. Usually when you work in OpenShift, you're putting most of the workflow inside OpenShift. This was not the case, and we'll see soon how we did it. That the dispatcher split the encoding jobs, put the content on storage, and send it to the hyper agent, which encoded the segment, and most importantly is profite-based so that there's a JSON which loads the profiles for the frame rate, and the dimension, and all the other parameters for the encoding. So we had some challenges to work on, because the dispatcher was not in scope of the porting of this application. We have to first demonstrate to the broadcast that OpenShift scale better than traditional VMs or traditional physical hardware, and we have to work with legacy physical nodes still working, and they are working still now, because we have to work in parallel. We could not port the whole application to OpenShift. On the other side, the advantages of this, well, the factor that simplified this application, this porting was that the node agent is quite simple. It's a Node.js application which publishes a REST API, and that the performance improvements were immediately tangible. So as soon as we deployed the first containers or the first pods, the performance increase was of an order of magnitude with respect to the traditional VMs. So this is just another focus, another slide focusing on the workflow just for the dispatcher. I think we can skip it. Okay, and now we have the architecture. We have to work in order to have a legacy application running within OpenShift and communicating both with single pods and with external entities like VMs. Yeah, as you can see, we have a list of OpenShift nodes. Basically, all the nodes you see on this slide can be scheduled by OpenShift for running containers. The last one is the one containing the broker that actually is the legacy application, and this node is part of the OpenShift cluster so it can actually access to OpenShift network and run the legacy application. With this trick, the legacy application can contact the internal container that actually when loads up, contact back the broker, and the broker itself will schedule new encoding on the registered client. Yeah, the main reason is that in a published subscriber architecture, the dispatcher has to contact every single pod as itself, and usually in OpenShift, you mask the pod under a router or a load balancer or something else, while in a published dispatcher activity, you cannot rely on this, and the dispatcher has to contact every single pod asking for idleness, asking for job status, et cetera. So it has to be part of the network of the agent themselves, and at the same time part of the other network for the traditional VMs. This is another representation of the nodes in our environment, so we have seven Apollo nodes, so with tons of gigabytes, gigabytes of thumbs, there's a graph in a success slide. Another important thing is actually the customer agreed to use physical nodes to unlock the real power for the encoding jobs. So unlike the standard way of putting OpenShift over another virtualized infrastructure, in this case, customer wanted to run it on bare metal. Yeah. Then we have a simple, this is mostly customer-based, we have to explain to the operation team how to work with OpenShift, which is usually goes on the other way as a developer usually works, because on OpenShift you usually run from build, build configs, then you build images on top of that, and then you can see at the end the final pods for the operation team, which usually is not involved into the build part, into the deployment part, but one just to see the running environment and see the logs and access to the traditional chest have to work on the other way around. So we started with them from the single pods where they can log in with OCE remote shell, they could see the logs, and then we explained later that they can run a deployment, which represents all the pods running, and they can scale the deployment adding and loading pods at will. And then as a successive step, we explained them that these pods, this deployment are run on image streams, which are provided with the agent software and the standard Node.js framework with all the libraries included, and then at the end we explained them how to build this image stream, this image and the image stream related running from the source code coming from the developer, so we have to treat the topics the other way around. We have the encoder agent deployment with the memory defined by system integrator, this was another successful ship because we were able to demonstrate that they are over-sizing their requirements for the memory needed, and they can do the manual scaling adding and variables, and you can see again that there are mainly, those are the most important environment variables because the single pod has to announce itself to the dispatcher when it starts, so that the dispatcher now knows that there is one more pod and can schedule jobs on it, and this announcement is done on the agent dispatch URL, and then there's the profile git from which the profile JSON is loaded, so that the developer can modify the profile, the encoding, the frame rate, et cetera, and this is loaded when the pod starts, then we have the image stream coming from Node.js application with the IP version git code, we deployed it on top of Opyshift which is quite common, and then there's a build and how we run the build with a standard source to image workflow. This is another of the five W's, so when we did it, we were able to run the whole process in just a little more than a month, which is quite short timeframe for an enterprise and for a project like that because if you think of all the MIDI contents that have to be streamed and the constraint for a production environment, and then we are constantly upgrading the environment, managing it, and the next step, for example, would be to put the dispatcher inside Opyshift itself, which was a thing we couldn't do at the first time because of the constraint for the system integrator and on the fact that it did not exist concept like Petset or Demonset in order to have a single instance of a pod running, or another thing we missed for example was the egress router in order for the dispatcher to contact from the same source IP address the agents which are not inside Opyshift. This is a summary of why we did it, and we see that now we have OpenShift with a lot of resources available and they are present as working, they reduce their development cycle. If you think that previously they have to dispatch a single VM on the month and this action lasts at least one day or two, now they can just scale the pods quite easily. We augmented the physical hardware density and so they are running more pods than VMs, and then they also avoid the technology lock in management because now with environment variables the customer can modify profile by himself while before it was bound to the system integrator to modify those values. The project involved also Glaster as I told you, and for example this is an extract of the export Glaster as a storage was an external project already in place, and here you can see that, yeah, where is the last line? In this case Glaster is exported as an external endpoint and all the agents can write the media content directly on the Glaster FS, on the Glaster FS, on the Glaster FS cluster so that the content is available to be presented by external origin server, which are mostly NGINX web servers. So thank you. Sorry, when? Okay, we run just the production building of the image with standard OC build config as you saw, and somewhere like. With the S2I? Yeah, it is an S2I source, we take the code from the system integrator and we provided the Node.js version which they required with all the libraries included. We also deployed a Git server on OpenShift itself, Gox. Yeah, we took the occasion for them to deploy also an internal Git, which I just closed, because I clicked on, I don't know, reopen, because remember that the system integrator is providing us obfuscated code, which of course they do not want to show us all the internals of the application, so we have an already parsed artifact to be deployed on the image itself, so we deployed that on Git and provide the ImageStream Node.js which is version 6.2, which the system integrator required, and the build config is run through OpenShift itself from the UI or from CommonLiner. Yeah, more or less, we're still trying to collect data correctly, because we have a Kibana, we have an elastic search environment, which is an external one managed by another team, and they're not providing correctly all the data to us, but the customer told us that the project could continue because they saw a huge improvement in their performance, but we still do not have physical data to provide you. Yeah, because for example, you see that now we are running 84 plots, which was a requirement by the system integrator, and they had less than 84 VMs before starting to show, so we translated the physical hardware, and we replaced that from VMware, we uninstalled it, we formatted it, and put that into OpenShift. Yeah, they have more, they have half of virtual machines. But now, just to answer, we just saw that the memory they require is quite huge for tasks, so now we can downsize the single plots and increase the size, the number of them, which was not possible with standard VMs. Okay, thank you very much, and as well do not forget to post your feedback and rating of this talk, and who would like to sign up for tomorrow lighting talks, please do that at the white table right behind this wall.