 We're gonna start the next session here. So if I take your seats, we will begin we have here today Martin kiss who's Is actually a helium MVP and he's gonna give you a presentation on the efficient Web deployment pipeline so a big round of applause for for our next presenter. Thanks Thank you Hello, everyone Thank you for the invitation for this each piece lot. Anyway, I'm not an HP employee, but everybody thinks that I basically and open stack ambassador from one way and I did a lot of interesting things Since the very early days of open stack so we basically built up the open stack community in the Central European region I'm located in in Hungary near Budapest and I Doing a lot of open stack related work specifically contributing back some site projects like the community portal of the open stack foundation and working on some question and answer sites and also Integrated the open stack ID authentication project into the open stack infrastructure so I have a lot of experience related the software development and I like to share this connection between the software development application word and Open stack and How can we operate properly and develop properly software for the cloud? Basically the thing that writing software is hard we can experience that every every day and if Somebody starts to develop a project you need to make some initial decisions. You need to select software project management methodology Need to find out the dependencies on third-party or the external models need to choose a platform or an application application stack so so it is not easy it I guess it was easier maybe five years or ten years ago and if we are thinking starting to think about Websites it seems to be a very simple thing but Making website is getting harder and harder those days because imagine we need to provide the multi device support we have very different frameworks and If we like to deliver some customer face application we need to think about the testing and user experience and and basically developing a website is is it's not a trivial thing if we are delivering it for the enterprise and Basically Open stack Is a very interesting infrastructure as a service solution? It is a very important part of the of the software operation But but it is an infrastructure so We need to we need to add some additional layer to it So basically If we want to operate in a in a cloud model or an edge highway we need to select a very flexible development model and For example, if I looking around in in Central Europe most of the startup companies are already Adopted this agile model But if I looking around in the telco sector or the large enterprise sector They are doing the software development in a very traditional way and it means very huge and long development cycles and They have some quality issues So basically it means for for this traditional model developing the software means a very huge cost for the development side and If I see for the some very interesting Hungary and start as for example Prezi Maybe you heard about them or or you stream or log mean they are very successful They have very small teams and they are very very agile and they can release software very early Basically, they adopted some rolling up grade models so they can release softwares even daily or hourly and We see the huge difference on the market between the enterprise and and the startup methodology and It is a very interesting thing because usually the enterprise have a lot of resource resources A lot of people and those small companies really acting with just a few few guys or few People and it is working. So basically I have experienced. This is the scrum model Where we have a so-called sprints and and we are planning in three different levels We have a product backlog via where we are collecting all of the issues or user stories or new things we like to solve and the team is selecting and prioritizing those tasks and allocating a two-week long period to develop those new features and And basically it is a development model. So it is not working for for customer support, but for development It makes a very nice and very friendly environment for for all of the team members for the Developers because for this two-week sprint we define the the tasks they need to need to finish and They are trying to do so it means it doesn't mean that they will finish everything at the end, but it is It is based on on the Team Collaboration and usually it is a progress a new team won't deliver very quickly Every solution but but maybe within a month. They they will So more and more problems. So it is very interesting to see how it works in the real life and At the end of the two-week spring Two-week spring they will deliver a so-called product increment a product that can be released for the customers and They can start everything Again, so basically it is it is a development model that we cannot use with the traditional Application and operating environment Basically a very important part of this model the last step when we need when we get a product increment and we need to release it somehow and Basically, we are holding some project repository Maybe you heard about this geek thing where we can hold the versions of the software We can hold the feature branches and the so-called release branches that we will release for the customers And and usually we have some operating issues Basically, we we get some hot fixes that must be released very very often or We we need to deliver some very important security fixes. I think it is a very current problem of The actual IT that we have solved the security problems. So We need to define a very proper release management policies So agile doesn't mean that we have a very free-form model agile means that we have some standards and policies We need we need to follow to deliver some very good quality software at the end and deliver in time so basically to support this type of Agile development model we need to build the proper tool set So basically open stack is using something similar or so we are using that for the open stack community portal that we have a repository where the developers can Commit changes or check out the current status of the project to make some patches or backfixes and and if the developer is finished his Work he is dropping that in the into the continuous integration system And it is very interesting that the continuous integration system will do a lot of automatized pretesting of the software so if the Scripps decides that my patch is wrong and and it And it is not working that I have a chance to to try again and and fix my bug and And pass this gating process and after that when the scripts and the robots decides and software decides that my Patches patch have the proper quality Then I'm so-called manual review happens So there is a team who can review my patch from human aspects because we don't have a well We don't have the proper Quality for the scripts or software yet So so finally the human decision is very important in in this step But it is a typical quality assurance process so we can raise the quality of the code that we are putting in the final release and If the month I get an acceptance on the manual review process then a so-called post building Parts starts and and this post building can do a lot of different things. For example, it can automatically create a Package for my my application and they deploy this package in the staging system Or if I'm not doing a simple backfix or a feature commit for example, I'm Tagging a new release it can deploy into the production system. So this process is is Entire automatica so it is automated after a manual approval No human will touch the the software and it is very important that we can We can remove a little bit the humans from from the project and and raise the quality of the entire entire software so it definitely works if we are doing a Frequent deploy and this agile methodology requires a very frequent deployment so imagine if you are start to do it by hand it can Take a lot of time and and I need to allocate a lot of resource for that and We have a very huge chance for the human errors. So Basically, this is the deployment model. We are using for the community portal actually and One other other very interesting aspect of the development model that we need to set up a very identical application stacks both for the software developers for the staging area and the production systems and Maybe you heard about this background story Vagrant is a very nice little tool that help us to set up a very similar Developer environment that we are using in in the production or the staging environment and Using this model. We can avoid a lot of issues related. Maybe Different package versions or different software stacks. It is very common I met with a lot of developers who are doing some web development even for larger companies that they are just Setting up some PHP framework or something similar on his own machines and doing the development and finally when they are trying to Deliver that for the production stage everything starts to fail and and this is this is the reason why we need to use Those tools and need to that deliver a very similar and identical environment for even for the developers and and this Vagrant is is just a small script that I can Small command line tool and I can hold all of the the configuration settings related to Vagrant in a very simple source code file and I can even hold it in a version Versioning system So basically this this is the ultimate answer to the worst for me problem I met with that a lot during the last last few years and And basically a real life real life example. We are doing some Drupal base website development and It is very interesting to see that Drupal is a is a web framework and We see this movement that a lot of enterprises Replacing the proprietary Web front ends with some open source based Solutions so basically we have this Drupal WordPress on jub jubla as an alternative and we choose the Drupal because it has a Very open and and strong ecosystem it based on open source and have a very nice community So it means that we can access all of the source code For even for the core Drupal project and for the external modules It has a large and active ecosystem and and basically it has a lot of tools we do not need to invent And from that aspect it some form supports or continuous development efforts and even it delivers some security support so every project or module I upload to the Drupal org website Get a review from the security guys So it means that I do not need to do anything Because I will get the security bulletins. I need to write good quality code. That's the basic but I will get some security support from from the Drupal org security team and Drupal even have very nice enterprise references even White House website was Based on Drupal. I don't I don't know the actual Status and we meet some government government project for the EU where we use the Drupal in Government environment So basically Drupal provides a lot of command line tools It has a very nice modular layout and have standard roles for development So we can find the role how to how we need to develop a website How we can separate the development into different parts for example backend development front-end development and and for the side builder role and Usually we think that the website is a very simple something but basically this is the very simple and big picture of an Drupal application stock and architecture how a real life Drupal Deployment looks like those days. So it is not not the webshop of the Neighborhood guy. It is it is the proper applications tech for an enterprise grade web solution and and basically at the top we usually have a system load balancer We have a varnish cache that can store and pre catch the the rendered web pages We have a so-called elastic pool of Apache and and Drupal Applications and We have a another cache layer that holds an internal key value for for example for session Data and internal hooks and and similar things. Of course, we have database at the end the standard traditional Relation database and if we are building something that requires Google-like search and content search Features that we are adding an Apache solar cluster and If we are thinking about cloud or we are thinking about the worldwide presence, which is common in those day where we need to Deliver an application in 24 per seven Usually we are storing all of the assets and and images in JavaScript and CSS in in an external object storage or a Content daily using a content delivery network. So we can We can distribute the the load of the website all around the world using those modern services and So basically it is not a trivial thing to set up our enterprise website if I'm checking the internally out of Drupal, this is the directory structure and We have a so-called Drupal core that the Drupal project delivers and and it is doing some very basic things and Of course if I'm doing a deployment I will have some moving parts like the files and some images and maybe the configuration is related for this specific deployment and of course We are not using the core module usually we need some special feature or some different content types or Similar things so we are starting to develop a so-called custom profile which holds our modules the web skins all the translations here if we are thinking about the multi-language site and and some custom libraries, so so this blue part is is our addition to the to the project So if you think start to think about the steps of the deployment how we need how we can to deploy that Into a staging or a production environment and and doing the release make basically building a release Will include the step where we will copy the files from the Drupal core and the settings the Configuration and all of the custom things into a single single directory structure So so if I like to deploy I have a so-called legacy legacy manual way where for example, I set up Apache and my SQL environment and Manually building the course and modules and finally when it is done and I have a configuration file and installing it but it has some disadvantage because It seems to be that it is as short it have a short deployment time It is true if I need to do it once But if I try to adopt those agi technologies and I need to do maybe a new release in every or deployment in every hour It is a huge disadvantage because it is very hard to hard to maintain or upgrade If I have some custom patches for upstream modules, it is also very hard to maintain So if I'm doing it frequently It is it is a huge pain I have a another Solution or option here because Drupal has a very nice tool set and it supports a so-called make file So in this make file it is an example of that I can define the number that the version number of the Drupal core. I can define the additional modules and usually Drupal is not perfect and sometimes I need to deliver some custom patches Which is not included in the upstream or it is related my corporate project I don't like to release back. So even I can define my custom patches in this make file so it is very easy because After using this drashmake tool, it will build the entire directory Directional structure and entire Drupal distribution Based on this make file. So basically it is very easy to upgrade. So if I see new module release, I simply need to rewrite this version number or if I Hear about a new security upgrade. I just need to upgrade my module this new version and rebuild the entire project so with this make file I can separate the custom code from the upstream and I can Build a very clean environment from scratch So it means I can repeat all of the deployment steps But it has a disadvantage. It is still a manual process I guess the ultimate solution we are using that We will combine this make file with the continuous integration and the continuous deployment systems so basically we are using that in the open stack infrastructure and After the end of this This process in the post building stage We are just using a make file and rebuilding the entire Portal from scratch and and deploying it into the staging system so It is definitely the ultimate way we need to we need to use Drupal enterprise environment and Basically, it means we have a very Short setup time after this drash make we are just building a Package even it can be and Debian or a redhead package We can publish a manifest file about the the latest version of The distribution we made and we can trigger and install and upgrade and the question came here how to install or upgrade this This Drupal site we we prepared and packaged because actually in in the for the community portal we have a lot of puppet scripts for that and we are still building this manifest and and package file and We have a very custom solutions of different scripts that is trying to to physically deploy into a instance of this Distribution we made so here comes this each parent development platform as a last mile of of this deployment process because it is based in in cloud foundry and and it is trying to do something similar we we did with open stack that it is an open source platform as a service and have a very similar governments and and Ecosystem like we have here in open stack and and it defines the rest API's so we can access the features through a user interface or or a command line tool or even we can write or on tools and drive this class foundry using using a rest API and the other point that it is very easy To learn how to use it in the real life. So basically the developers Doesn't need to be experts system administrators And don't need to know too much about the cloud because they are getting a very nice API or user interface to deploy their code if we are looking the layers of of the cloud foundry and how is Connected to open stack. Basically, we we have an open stack infrastructure that is working Usually if we are not using some private cloud Deployment we we can use a public cloud infrastructure based on open stack and and those cloud foundry nodes Sitting on top of this open stack infrastructure. So when I Like to start or deploy a new application cloud foundry will Or can create Even new nodes in this open stack infrastructure and all of the application will sits on top of that in in containers So it is a very effective way of using of the resources If you are looking a little bit deeper in this Cloud foundry and we compare how we could build up and Web application environment from scratch We can see very similar components because we have some router at the top So so cloud foundry can manage the even the auto scaling of instances or can build up an application pool So our application can be scale out into multiple containers So we can even build up some some load balance or or failover solution there. So The very important part is this application life cycle management a cloud controller and this application access execution part that can Provide or orchestrate our application into the plot into this platform as a service solution and if I rolling back to the previous Diagram where we had this Drupal architecture. We saw that basically our architecture Our application is just a very small part of the story and and we have a lot of External services we have an external service this varnish cache the memcached or ready's Backend even database is is an external service and Apache solar cluster is an external service and This platform as a service model separates the the components exactly that way because we have so-called service brokers and And for example, I can I can define my application that I needed the database And I do not need to deploy a database service. I I need just consume the the database service from from this cloud foundry or through this cloud foundry and it have a Lot of other services built in me for example a robot MQ Q or a or a shared file system and And we can even extend this part of the cloud foundry And and of course what we get we have a built-in matrix collector and very important parts We have a application log aggregator So if we are deploying our application into into this platform as a service We can get a feedback about the events that happening inside our application So if we are implementing proper logging, we we can see it through the cloud foundry So it is very nice even in an user interface if I launching or upgrading an application I can follow what happened inside. So if I like to implement Proper application environment without this I need to set up some some log aggregator system so it is a lot of work and If I like to do an application Deployment in the real life. It looks that way. Basically, I have a command line tool I need to log in there. I have a login account and Basically, I just pushing my application into the into the platform as a service and at the end I will Got a running running project. So there is an example of how to create my CQL database So basically it is deploying an application means one, two, three, four, five Commands, so so it is not bad if we are thinking about how to do it a manual way how much Comments I need to to issue To launch my application or upgrade and existing existing one. So it is five commands. It is it is not bad, but We have a very nice Solution to decrease this this number of commands. We can define a so-called manifest file Where we can define our application and all of the service relation we need for example, this Drupal application We are defining a so-called build pack. I will tell it Tell a few words about that and we can define all of the services or Website will consume for example this example it is consuming just a database search And this build pack a very interesting thing We have Build packs for Java PHP and different platforms. So we can reference in this manifest file to this Build pack and when I launching my application into the platform as a service Solution it will build up the application environment based on this build pack, but it can be anything So we have a lot of ready-made build packs It is open source and they will be on the on the github But if I have a very custom needs even I can write a very simple bash script So it is very extendable and if I have this main files if I had this manifest file to my Drupal distribution for example, I just need to do a hellion push command to upload it into for example a staging server so As I mentioned before we have an API for that and if we have a continuous deployment system We can include this single command and the login part Into the post building job and we can get a totally automated Solution for for application deployment without any human intervention So platform as a service build pipeline element why why it is good because we can standardize or Processes with that we can retink how we did the the deployment before We can use the common tools provided by the different frameworks. We can eliminate the manual work and We can lower the cost because we are building frequency. We are focusing on the quality and We need to focus on real problems we need to allocate resources for for deployment and at the end it means that we can deliver faster and we can win the market and I like to share two very short user stories regarding this We have this pencil company. They are Genev based digital communications agency and They are working mostly in the sport industry and and converting all proprietary Enterprise websites into into drop drop out and we started a pilot project with them and They are doing exactly that so they are standardizing their release management processes They are working in automation and of deployment to staging server Basically, they are using Jira and bamboo for continuous integration. But as a last my step in the build server, they are they are deploying this Cloud Foundry instance and Why they Need that because they are doing very frequent deployments. We started this pilot program to build up a staging server and They have very different products for for different sport associations But all of the applications tech because they are using group are is very identical and very similar So they can build up completely automated deployment System using those tools and I'm started on a proof-of-concept project I think it will be a little political challenge to push it through but As we are using group out for the community portal We have this last mile step to deploy a portal using puppet scripts and some additional magic and I like to start to standardize this process and use a platform as a Solution to host all of those services. We are using these days and Basically, we have everything else. We have the continuous integration system based on Jenkins the Similar to we are using to build the entire open stack project. Thank you