 So Good morning tomorrow today where I am going to explain you where we build and deploy media salsa, which is Sir software services. We are deploying using triple It's super base. So I am Julien Pivotot. So I am a Belgian guy. I work for in which we are quite big open source company in Belgium the Netherlands and also in Kiev So let's start So Does someone know what is media Mosa? Okay, so media Mosa is a Drupal based digital asset management system. So basically Is to assets it can transcode videos and it can manage Everything about the metadata around it. So creation data and it's each where all the metadata you can Expect and it can also manage to do the search inside this metadata And it's also a service oriented So basically media Mosa is a back-end and then you can plug any front-end you want For example, we use a lot of seedbuilder stuff, which is another just for distribution that can just plug to media Mosa and and make YouTube like websites So media SELSA is just media Mosa as a service We do not use proprietary code. The code is pulled from GitHub basically. So it's just media Mosa as a service Nothing more nothing less So what is the infrastructure we are speaking about so that's only the media SELSA infrastructure but behind that we still have puppet servers and All the basic stuff like Merge server C12 but for media SELSA basically you have the back-end application Which is PHP for example PHP or paste So it's a Drupal distribution and then you can have front-ends that we speak to it and Then you have all the servers the web servers the database servers, which are uning mySQL The solar servers which to do the search By default Media Mosa doesn't use solar, but you can plug it to it So you can have all the advantage if of solar and then you have the trans coding servers Which will run FF MPEG to just one cause to unscode the video in the rights format And we have all of that for each environment and by environment I mean you have the development environment the user Acceptances Environment and then you have the production environments. We have several production environments So a commercial one an academic one and then we can make dedicated environment for some special customers that's once more security or things like that so each time we are reduplicate all that infrastructure for one of environment and Each environment is the same. I I mean The development environment is the same as the UAT environment and the production environments, which means that If we have performance issues on one of the environments, we will have the same performances on the other environments So what is DevOps about? I mean if you are in the DevOps track track, you might probably know but DevOps is about four things It's about cultural changes Automation measurement and sharing so we will go one step by one step inside this to see how we implemented The project the project is only one year and maybe six months old. So Let's see where we are at the moment So in which we are an open source company. We do only open source things and As it opens us we contribute back. We do not keep We do not keep source code for us. We just contribute back to the community or for example We publish prepet measures. We publish The media mosa code and things like that. So we are 40 people in three countries. So Belgium, the Netherlands and Kiev We have customers all around the world. So in France in the US and it's it were and With this we have one language for all the company. This is important because here when when you have a developer that Can speak English? It's like, okay, you can communicate with it directly. You need someone more It's important that we have only one language in the company And then what do we do? We do a lot of different things as well Just we are not a drop-out shop. We do consulting internal project about development system administration. So we do long-term contracts we do fixed price contracts and We do a lot of stuff. So that's why That's why we We are more than just group hours. We do a lot of stuff and then we deal with The ideal worlds which you have plenty of time plenty of women resources Plenty of money and there is the reality. There is a budget So you need to take the pragmatic approach. You need to make it just working and You don't want to deal with Someone doing something that you know in two months. You will need to do the opposite So you need to do it right in the first time So we are a distributed team Which means that we don't have Meetings physical meetings. We don't sit in the same room. We is just the communication is very difficult It's not software. It's not development. It's not fixed. Yeah, you need to talk with people So how can you improve the communication stuff? There are several areas of improvements. For example, we do daily virtual stand-ups It it can be a very example or overhang out so you can Actually see each other in the eyes and just talk freely then you have Well mind so everything is a task and RedMind manage the tasks the git repositories and the documentation so one project as one RedMind repository one pipette modules have one RedMind project and like that and You will also have mailman. So When you ask a question to the developers you just send to developers at blah blah dot in result you when you reach everyone that needs to be reached and Then you can also communicate with your tools Jenkins You can use example P mail mailing list and things like that. So when something happens everyone knows and Then there is also internal training. You can do them also Physically like okay, let's all meet in Belgium next week or you can also do it over hangout And you can also have external people just joining for the this training and then there is mainly two teams the developers and the operational team So the developers are paid to develop new features to write tests and the operation guide Just take care of the infrastructure and the continuous the delivery integration and all the other stuff And I could also have the monitoring and so the ops teach the developers to think about monitoring when they develop their application and think about distributed services or will it be deployed and Will it scale and things like that and the developers just teach the top the ops about okay, what do I need? What do I need for my test? What would I need for environment? What will be important and that's a continued process of the developers telling the operate the operational team Okay, what do I know? What do I need and the operation are just saying to them, okay? We have a performance issue there So please correct your code or please do something or tell me what I can do to help you and that's everyday communication between all the teams So that's the cultural things we still have a lot of things to do inside all teams to get it really done really right But they are we are working on it and it was it works quite Nicely for the moment Then we have the automation. So the goal of the automation is just You automate all the things you should not SSH to a server Ideally in the ideal world because if you do it SSH to a server You will probably need to SSH to a server in u80 in development in production one in production two in production and then you never end to do manual changes and when Someone figure out or why does it work on that environment and not on that other environment? It's like omit. Someone made a manual change there Why? Why do you need to do manual changes and when you need to act on a Server for example for madness or things like that you can use tools like mCollective and then you can say on Order trans coding servers. I want to update that package or it's really if you need to To do it at scale use just mCollective instead of SSH server and mCollective is the tool that is used for continuous integration and delivery. So So we do continuous delivery and continuous deployment So basically we do the same for the puppet code and the application code so back ends front ends So for the puppet code we deploy it directly to the development environment after making some tests around it So basically season mean just push the code Up it's deployed to the dev environment and you can check on foreman if the changes had applied Are some of you are using puppet maybe? No, so puppet is a tool that's really just help you to configure your servers And you can tell I want that package install and you can tell Put the configuration files inside it and each situa. So we use the We write down our infrastructure like it was PHP code basically that means that we can change the infrastructure using gate and things like that and then It's really easy after to have an overview of your environment because everything is in that there are five puppet files and It's a code so you can easily grab it find it and situa and Then one the puppet code is deployed to the development environment. You can Say, okay, I wanted to be applied to the UAT and to production So if you for example, if you want to install a solar server on a serve on a Server called media salsa zero three for example You can write it down to the puppet code and then it's installed automatically in the development environment Then you can click a button and have the same solar setup on the UAT environment and you click Another button and you have the same on the production environment and then All the environments are not really the same Some what sometimes you just want for example the replication be activated on a certain on On the development environment so you can add feature tracks in your puppet code So you can say okay on that environment You want to test that things and you can still play with the rest of the puppet code to the other environments And then we have the application code. So the developers just They do the magic code. Oh, sorry and Then they can they receive a melon Basically, they can just click a button to have it deployed to UAT if everything works and then The business can the business decision is okay. It's works on UAT Do we put it directly on production? Do we click the button to put it on production so the customers can have the new features? But for the for the developers is like, okay, I push I can I have the integration test in the development Environments and then I can just push it to UAT by myself So the tests Yeah, the developers they test a lot but Yeah, before the test didn't work or it works on their machine because they have their PHP version Which is not the same as in production or is the wrong platform because they are using Ubuntu And we are using CentOS in production or it's not the good PHP version Yeah, but now it's fixed thanks to Jenkins and also thanks to Vregrant I don't know if any of you know Vregrant, but Vregrant allows you to create a Test on environments for example in virtual box. So if you want a CentOS 5 To test something you can just Have it and destroy it rebuild it and things like that. So and Jenkins Run the test on the development environment Which is the same as the production environment So if the test works on the development environment They will work on production and you do not affect production by running all these tests that That are already working on the same environments except that it's testing environment and Then the code is under revision control. I think nowadays everyone do it Everyone do it, but yeah, it's what I think but I have already met people which are just Okay, no, it's not and the revision control and I need to SSH is just oh no, no, just use git or anyone you are on Mercury alone And then you need to prefer the small commits before because if you want to If we have a performance if you and you need to review 50 50 commits or one big commit. It's like okay Maybe we prefer to have small changes one after another and if I really want To have a new feature or a new version of my application Then I will probably make a second git repository and I will probably for example Make a new repository for the version 4 of my program instead of having another branch And we are just packaging the master branch so the developers Yeah, they can work on their own branches, but we are just using the The master branch on Jenkins and after that if you really you need to have several versions You have several projects who have several you have several other things like that and then with the The git stuff we just put it in a package. So all the code is an operating system package Some people just don't catch why so that's why we use operating system packaging system like Debian RPMs and things like that so It is important because it brings you all the features of the packaging system of the operating system for example, if you Have dependency to our projects and you don't have packages You will probably need to add the share script to it to install all the dependencies and things like that And when you are a file on your server, so where is that php file coming from you can just Know it just by looking at the OS provided tools And the source repository may not be reachable. Yeah to clone the git repository You need a SSH key or password and you need for maybe to be connected to a VPN So the social repository may not be reachable as it's really easy to build repository operating system repositories like with pull or will Debian create Reproposed stuff and things like that. It becomes really easy And when you automate when you use puppet when you use chef when you use unstable and if any other Automation tool. It's really a little of a way to add this to your automation. It's Easy to add a package that a git repository in or no it's just Easy to use the packaging system and you have tools like FPM that I can just do it for you and Then there is one important thing that the configuration does not belong to a package Yeah, so you can build packages, but do it the right way I mean if you put there my sequel configuration to the package and you want to use the same package on each Environment, it's gone not gonna work unless you have the same password on each environment, but basically the IP will change it So just separate the code and the configuration and then we use Jenkins So Jenkins is the software that will just Check out the code and do the test and things like that. So it's the main Is the main piece of software that we are using on an everyday base So Jenkins uses pipelines. So a pipeline is a collection of jobs. So a job is for example one test package check out I will explain that further but and They just run one after another. So first I check out if it succeeds. I will One start check if it succeeds if it succeeds And then you have all our beautiful pipeline and you can see where it's wrong So is it the test that are failing is it the packaging that's failed and For the developers It's really easy because they push their crowd and then Jenkins makes some magic and they get a mail back with the changes They made and the link to deploy it. So the deployment is also made inside Jenkins So there is one tool that there's a lot of different things using different Nondos like Duras and etc But the developers they push they get the mail. It was okay. It was not okay That's the reason why and that's the changelog. So, you know, so who to blame or who? You don't need to point fingers. But if you know, okay, that guy just make a changes He will push it maybe to you 80 and then you you know if you break it, you know, it's oh, it's really my god That's break it. I need to To correct it before I can it can go to develop panel you 80 environment And so our pipelines are privatized that means that they are automated So we do not go to the Jenkins shop interface to click 40 buttons to have a new job and to type everything by hand So each pipeline is identical to another pipeline. For example, the backend pipeline. Yeah We have several backends one for version 3 version 3.5. Yeah, it's just the same It's just the same pipeline does does not mean that it's the same code because we Specified the git repository for example, we specified the dashboard view We want in Jenkins for those who use Jenkins. We directly put the pipelines in the right view And then we also add the target URLs that that is used for testing. So for testing and for two or so Before yeah, we went to the server we went to rush by hand like to rush update the rush clean cache But now we just use the pipelines and them collective stuff to do it and we do not have manual work To do so if you need to do the clean cache on for several servers Yeah, it's not gonna do it and then we also create the views using puppet. So From the installation to of Jenkins to the creation of the pipelines everything is automated So what is a pipeline in a graphical way? That's a pipeline. So You have all the jobs That's our run one after another and you also have the history of the pipeline. So that's the last three pipelines You can see that This one just this job has just failed It's This job has just failed for example, and then it's really easy to to see okay That was wrong. That was good and it's always the same testing that is made. So What are what do we have in the pipeline? We have first the check out. So a developer pushed a code. We just pull it then we have the syntax check We just use php-l it's fast It's easy and it can just tell quickly if something is wrong You do not have to deploy a package and things like that if there is really a huge mistake You see it directly Then we do the style check using Drupal Coder I don't know if some of you are using it, but it's using php code sniffer and you can just plug it in you have The Drupal style get stuff and Jenkins allows you to have some graphics to see okay I added like 10 new warnings in my code and okay. I fixed 10 new warnings Then we do the package The place when where we do it is really important because we package it before we test it That's important because we will use the final packet during the test Which means that okay, maybe the development environment will be broken because there is a mistake in the code But at least we know that if the test succeed, we will have exactly the same code in production There will no be brush changes. There will no be fancy stuff. No, it will be the same code And then we deploy it to the development environment. So basically we use pull I don't know if someone you I know we do not use pull for that We use pull for the puppet packages, which just create a prm repositories easily and for the code We use dbm files because we have notes running dbm Then we deploy them using mcollective. So mcollective allows you to remotely deploy update packages doing the epiti epiti update and things like that And then you can we run the test in the development environment. So basically first We do a reinstall of the website, for example, we do a fresh rush it install Just to make sure that we didn't break it the installation and then we just go on with the test It's rush one test I do not know what's behind to be honest because I am not a developer But yeah, there's one test and we only pick the test related to our code because yeah, the Drupal test takes like Two or three hours and we do not modify the Drupal core code. We just modify the major Mosa modules and Then we publish the package to the other environments Which means that instead of using branches like one branch for the development one branch for the production environment We just use several repositories for example in the development repository you will have version 45 but in the production one you will have version 25 and That's why we don't use Get branches because we use we use the system packages and we just do it another way and Then the developers the operators and anyone can run the promotions Which mean okay, that package is good. Let's put it in production. Let's put it in UAT So at the end of the pipeline you just have that page you received it you receive the link by mail and then you have a Button Then you you can just deploy the demorphon 10 to UAT up click the button You can you want to deploy it to production? Okay, click another button. You want to deploy it to A chemical production. Yeah, there is a third button. So it's really Simple the developers doesn't need to learn to you how to use Jenkins or to O2-SSH to a server How to do to wash DB with our good parameters? No, it's just one button. It's really simple and You can also add some more coding Conditions to their promotions like okay, you can promote it to UAT. You can promote it To the production one. Yeah, but only if you have promoted it to the UAT first So you can ensure that their version on production will not be behind UAT There cannot be human mistake there It's kind of dependency between their version of production UAT and things like that The other side is that someone sometimes you just deploy it to UAT and UAT again and UAT again and you forget to Update the production one and your production one is one one behind but that's still something we need to figure out so Jenkins use a lot of different tools like Pulp to manage the RPM repositories and m-collective to update the packages and through and rush But that's completely transparent for the end user If you are using m-collective already, I have on my GitHub page a rush agent for m-collective Which can do the basic stuff like update DB and basically Basically clean cleaning the cache. And m-collective is important here because You probably don't want to Jenkins to be able to SSH the production machines their infrared of machines and the UAT machines and collective is Limited of what it can or cannot do and there is authentication there And you do not need to access full SSH access to your to all your different servers So that was the automation part Now let's go to the third part the measurements so in the DevOps In the DevOps ideas you try to monitor everything you try to have a Lot of matrix from any kind of stuff and it's important because Yeah, it's it's not when it has crashed that you will find the matrix from one hour ago No, you need to have everything Everything at no to be sure that when it will crash you will have the right things so you Create the more matrix you can and then you can you see okay That one is really useless or that one is not useful in details, but yeah The goal is to have the more relevant stuff inside your monitoring tools So the first thing Not the first but one things we do is we use lock stash. I don't know if some of you use lock stash But it's a small Application that will just collect the logs the Drupal logs the Apache logs the Development logs the system logs and then you can just take Take them from different sources like for example you then you can take it from the Drupal watchdog you can take it from syslog from And any kind of stuff and then you can export it to whatever you want So if you want to put it into graphite, I don't know if some of you are using graphite But graphite basically helps you to create great matrix in a really easy way for example you can just open a socket to power 200 2,222 and send something there and it will be a Matrix in your graph. You don't have any configuration to do to add a new matrix and every developer can basically do it and Log stash can do it for you for example when I have a fifth 500 error in Apache. I just create a matrix for for that and say, okay, there is There is a 500 error there and then you can just graph it if Okay, how much error do I have and you can say, okay at the same time I had the load that went up and I have the ffmpeg usage that went up and That's that helps you to find what goes up to others You can also do it with the deployment logs for example, you are deploying a new version and you probably want to show the management Okay, we have developed 10 new versions today. We have deployed 10 new versions today. So we are really We are working fluently and everything works like expected and you can tell your customers Okay, we have developed we have pushed five new features in production last month and it sees work And you also have the system looks at more not for developers, but developers can also just see okay That's the load average. That's and they start to Thinks like operating people because they do not group They do not need to assess the server to see the load average So if their tests are just making more load average and before they can say, okay I will try to find what what's the problem there in my code and They start learning with you and it's all going on And we also lock stash elastic search which allow you to search the logs and things like that Kibana, which is a front-end to elastic elastic search and lock stash which you can just show to our developers and they can Just say okay I can easily grab all the logs from all the platforms or all the environment so I can grab the locks of all the Transcoding servers and it's easy. You can take graph out of it and things like that You have also stats D and graphite, which as push I just explained So what do you want to monitor like I said you want to monitor everything and I mean, yeah, the basics The load average the number of user of user connected to the machine is Apache running is ntp running Is there my the my sequence running and then you want to go further You want to monitor each vehicles because we may be maybe the main the default wheels is fine but there is one wheels with Database connection trouble or there is a real status is a White page of the death or and you want to monitor the database is to can the user connect to the database Is on the database running out of space and things like that and you also want to monitor the corn jobs in the case of media Most I'd still more important because I think we are running the corn job Every one or two minutes if I do remember correctly, but if the corn job are taking too Too much time or if they are not running we really get into trouble So we you want that to be in your monitoring system. You don't want to go to each back-end and check Okay, when did the last one job run? No, you want to have it on the same place So for example in nicking Garvin Graphite and then we use graphite it and G dash so G dash allows you to build dashboards with the graphite tools So graphite basically it's take matrix out of a UDP Prot and then it just make graph is from it. So we have graph is for the basic stuff like The load average the number of users to disk space the disk rights the I own and We have more specific graph is like the ffmpeg usage We we needed them because yeah ffmpegs was crushing because There is no enough memory in the machine Then you build graph is you you build a dashboards and the next time you can just say, okay That's the problem. No load were affected and things like that. So that's some examples of ffmpeg Memory usage so you can see okay. That's that Christopher just Sounds coded five videos or one huge and you can start also knowing who is using your infrastructure when they are using your infrastructure and Do you have 10 times more videos this month than last month and yeah, you can just learn from your own infrastructure You can extract business value out of it That's another Graph of the number of manage assets So it's the number of video photograph and things like that on one day and You can this is like This is business valuable. You can show your manager the manager can see okay. We have so much video. We have so much photograph and it's It's important also for you Because you can see okay. I expect that customer to add for example 500 more video next month because they did it this month and you can have the evolution of your platform One of the difference between graphite and munin is that in graphite you can specify the rent the time range I know in munin. It might be possible with money 2.0 But in graphite it's building. It's really easy to do you can say I want from Monday to Tuesday or for Monday at noon to Choose that noon and you can start doing Transformation on your graph is you can do the average you can do the sum you can you can play with all the matrix if you want That's a third graph Showing the number of backend ID, so it's basically the number of customers and it's also Business variable to have it easily at the under the hand Because yeah You can't in there, but who cares about it. Sorry. You don't see it correctly, but we'll catch about that Hey, it's really important to know your infrastructure If you do not know it if you do not know okay, how many people are using it and yeah, it's like It's useless to do monitoring. It's useless. It's like doing monitoring and being just blind or Yeah, so it's important to have all sort of different Matrix so that's and the last point of the DevOps things is the sharing so Yeah, we just shared our experience. So if you have any questions You can just go to the microphone and ask me them No questions Okay Thank you Presentation it's it's really great that you share your experience and it seems that this Whole this stuff works for you and I'm curious Approximately how much time will take for a new company to adopt this way of working So we have been working on it for one year, but if you have already a Team on place and with a lot of people it thinks really a long time It's not a one day or one month change. For example, I think at Etsy they just took like 18 months to just go to a DevOps way So you need to change the way people think about it and you can just it's not good It's not something you would just say okay No, you do it like that because people will just say no I won't No, you need to show them that it's better for them and that by doing it you have more fun You enjoy your work and you have less risk if you deploy 10 times a day here Then you are not afraid of deploying if you deploy two times a year It's like each time. It's like will it work will it work? Yeah, so it's just take time because you need to change the way people think And yeah, it's about the way of people thinking because this This package deployment. It's very interesting. I think for a company which are like side factories Because it seems to be very specific like you should host yourself because this Package systems it won't work with all the hostings. It depends from the environment from the operating system Everything could you recommend this for the such kind of companies? Yeah, actually for the front ends. We don't have one One code base we use front ends with such builder use custom other custom front ends And you just need to change for example the test that are running or the target platform But as we have the pipeline perpetize it really takes a Really a little time to change to change it and to just say okay I have a new customer He needs a seed builder site or it needs a classic the parasite and it's really easy for us to just create it and So yeah, I would recommend it for every company. Okay. Thank you. Thank you. You're welcome Thank you