 So good afternoon everyone, thank you for coming to our session and today we will be discussing about developed tests and deployed at scale and the purpose of the session is to share feedbacks, tools and processes with you that are part of our daily use of Drupal and all the CI and CD stuff and especially for quality process. So first of all let's introduce ourselves. I'm Sylvain Moreau, I'm the CSO of Access, I'm also its founder in 2001 and it's a pleasure to be here because today is my 10th anniversary of being a Drupal code speaker. My first time was in Prague in 2013 so it's really an honor to do this again and thank you. And I'm going to hand it back to Anne-Sophie who's going to introduce herself. Thank you. Hi, I'm Anne-Sophie, I'm a project manager, I'm also an active member of the French Drupal Association, a member of French Translation and I love to organize Drupal Cums. Hi everybody, my name is Francis Le Maître, I'm project manager at FMSH. FMSH is a public institution which promotes research in social and human sciences and also we are leaders of CanalU. So I led the redesign of CanalU and I'm also responsible of the maintenance of the platform. So what will be covered in this session, this is the plan. So first of all we will present ourselves, the customer and the agency, so FMSH and Access. Then we'll talk a little bit why we made this session and TLDR. It's because we wanted to share our mistakes and the common tools that we use and then we will go through every detail of our tools and processes and see some use cases and we will conclude with the detail of not so good or we can say a bad experience. Even we have more processes, sometimes we fail and we like to learn and share while we fail on this one. So first of all to start we are Access, you've seen us, we are a sponsor, Platinum sponsor of this Drupal Cums and very proud of it. We've been doing Drupal since version 4.6, so that means 2006. I founded this company with some people in this room. We are 42 people including 26 developers, in-house developers friends and here at Drupal Cumlin we are 15 so don't hesitate to talk with us, come say hello and if you want to talk about projects we're here, everyone is a team, we'll be there for you. And we take care of our customer from D0, that means we do the strategy, the UX, UI, the development, but also the hosting, we have a French hosting, we do the maintenance and also the SEO, SEA analytics part and our customers are big public and cultural institutions, public and cultural and also NGOs like WWF UNESCO, we will talk about UNESCO tomorrow at 9.15 and some big private companies in the editing field for example. So now I hand it back to Francis to introduce Canalue. So Canalue is a public media platform which is financed by the French government. It is totally free, it's free for use and free for contribute. It was created more than 20 years ago and now we have more than 25,000 hours of video and podcast archives. We have 300 content providers. Actually I could translate Canalue as a university's channel so most of content providers are universities but we also have museums, public institutions, and research hubs as well. Most of our contents are recordings of conferences and seminars but more and more we have an interesting collection of interviews and documentaries. Canalue has its own dedicated infrastructure which is led by a hosting company and one important point is that we developed our own live streaming and video and demand services with companies specialized in audiovisual and also we have our own encoding service which is interfaced with our Drupal back office. So of course audiovisual aspects are very important in Canalue but also high level documentation is very important. When publishing a content, contributors can complete a lot of metadata such as author, scientific fields, chapters, copyrights, educational uses of the content and so on. For that we interfaced our taxonomy system with public entity service called IDRF and using IDRF our metadata are interfaced with RAMO which is the official taxonomy of the BNF, the French National Library and also it enables our authors to be interfaced with different authors or repositories such as archive or wiki data. So some numbers of Canalue so we have 17,000 Drupal users in the back office more than 47,000 video pages, more than 22,000 author pages, almost 7,000 taxonomy terms. We also have 2 million visitors per year, 200 new contribution every month and at least one major delivery in production every month so you can see that there are a lot of activity and data dealing with Canalue. So in this use case we as a team we are working with an architect and the main link developer, one, two, three, four or more developers back and front the same project manager since the beginning of the project and a lot of issues in Ticap and there's two sub-project on RedMinds the tool we are using for project management. On the customer side we have one project manager, Francis and other people and two technical partners for video integration and IDRF integration and in terms of organization we have four main branches and five instance and we'll talk about this a little bit later. What we wanted to share today was were good practices and good tools and the objective of putting such tools together and such practices together in quality processes. First of all it's to be confident in our practices and our deliveries because when you work on this kind of sites you want everything to be needed and everything to be perfect because things can go wrong and you want to limit that and be confident means being confident between the customer and between the agency and that helps to keep a good relationship and confidence with our customers it's the first part be sure that our deliveries won't fail and we also want to continue to work on the project from the time we put it online because if you can imagine putting a site online is only a birth and then you have to raise the child to another level so this is how we see big projects and some kind of actions that need to be taken right from the start is a clear communication maybe it's an evidence for you but we do reclimiting between our project manager and the customer project manager and the dev team we need to have clear and shared processes that means that every responsibility has to be detailed and attributed to someone that means you know what a project actor does in the project and then finally and that's we will detail just right now it's a bunch of tools and practices to establish a routine of work the more routine you give to all the whole team and the more quality you bring and the more confidence you bring so now I can let Ansofi present you all collection of tools with Francis so we start with the very beginning of course we are using it for code based storage and we have as I said previously we are four main branches named master main and other branches for evolution dedicated and preproduction and production production is live sites we are trained to avoid cherry picking and to follow the full process for deliveries in order to avoid mistakes and to keep something clear and simple to follow and and we also have name or branches with the same name of the environments in order to be out having something really clear for new people on steam and we have also some protection of the pre-prod and prod branches for avoiding mistakes on deliveries adding to that we have also implemented some tools added some tools on the CIP CIP and pipelines in order to have automatic testing and code checking using standard and open source tools and so I have also lists all the tools we have implemented on a slide in the end of the presentation so you would have the full list of links it will be easier for you to to check if you have the same list of me with me and we also are using matermost as a chat system we have multiple team on matermost one for us on the team for informal discussion and technical discussion and another space with the clients for more formal discussion having a good communication and visualization on what we are doing now and what will be done on the after we are also trying to make the tools talk together so Githlab is talking to matermost and me as a project manager I know when a commit is done and what is about and it's easier to know if it's something is going wrong or not and as well in this list of tools tool I have rediscovered preparing this presentation we have also unit tests we made this test for two reasons the first one is because Kanalu is like kind of a tube and you don't want a contributor can publish a video on another channel contributor so we have a unit test specifically specifically made for that for testing the sport and the other test is for being sure IDRF the external reference is still working with our system for deliveries we are using also Jenkins and matermost and these two tools are also talking together so in Jenkins we have one job for environments and most of this are automatic launch except one is my privilege as a project manager I can run a delivery on a staging environment it helped me to make a full delivery of tests for the client without site is rerunning and testing by the team when I have a lot of tickets to deliver to the client I make my own delivery based on the main branch and then after we'll start to launch a full delivery to production and for that we are using a no-made shell script really simple it make a good pool says compilation synchronization of your configuration and do a lot of dates kirkash and that's it we are also working with red mine as a project management tool and we have also created a custom field in order to have the same list of environments and to be able to create a specific list of issues which features is available on which environment and to and to make also specific tickets for deliveries and and I will just explain right after this is an example of a ticket for deliveries so if we are ready and we have a lot of things to deploy and we want to share a bunch of features we are creating me as a project manager I'm creating a ticket and I have two developers to make a diff between two branches so master and prep rod and and this makes something like this we have a specific commit message with the pattern with the idea of the ticket and it helps us to create this list and to make automatic link to the tickets so for me I can once the diff is made I can check each ticket I know the different status and only only if all the tickets are validated I can deploy easily the developer and and sometimes we need specific process because not all delivery are automatic and you can need to run a specific command or to add an item for a menu or do something manual it could happen so we have another custom fields for read this notes each developer is working on a ticket and explain what action has to be done manually for delivery on his ticket and we are when we are creating the full ticket for delivery each release note is regroup and the on the diff merge message and it's easy on your on your delivery on each environment to follow all these steps to be sure everything is available on each environment okay so first I would say that at Canal U as a customer we believe that we have to take a very active part of the quality approach our testing team is composed of three people one documentalist one webmaster and enemy so first of all each unit delivery is tested by by my my two colleagues and they work with access until each feature correspond to the requirements after they validated all the other tickets I check and validate the package of features to be deployed on preproduction after the preproduction deployment I first make a technical check I verify that each service of Canal U is a working well so for example I check that I can encode a video launch live streaming receive an email from Drupal write something on the database and so on so for that I use specific routines and also a grid the grid that you can see here on the slide this process takes me about 15 minutes if my technical check is okay then my colleagues start the operational check so for the check that each use case of Canal U is working well both for front and back office so for example the documentalist verify the results of the certain engine for specific carrots and check that the results are confirmed to the data also the webmaster verifies the whole process of creating modifying and deleting a media content for that they also use their own routines and grids and this process takes about 30 minutes if everything is validated by me and my colleagues I can give the go to deploy and production the deployment and production is not made by access it's led by our hosting company most of the time it's very quick it takes like five minutes and since production is a perfect copy of production we there is no need for operational check again at this stage I just make another technical check to make sure that every service is running and also a last word so a last word about encoding service because it adds some complexity to this process because for example when we want to make a deployment on production the day before we have to launch a script that we developed a script that oppose the encoding tasks because if we make a deployment in production when there are encoding tasks running it could we could have database conflicts so we also have to deal with this so all these routines and process are well documented all the documentation is shared on the wiki page of the renminer project and of course is shared with all actors of the platform so access but also the hosting company and the audiovisual company as well this documentation is quite easy to follow because we described step by step the process and also we describe the notification processes and also very important the wallback procedure when facing an incident so we try to keep this documentation updated all the time the better will be to anticipate the incident but it's not easy to do it every time but at least when every time we face an incident of deployment we update the documentation to make sure this incident will not happen again in the future this is not especially for deliveries but it's another example of tools and quality check we are running every day on project as we are working on many dribble sites we have created another dribble site it's an internal project with one main content content type named project and we have listed all our project inside and this tool help us to to check the the update we could run on sites for core for modules met also for libraries and other stuff like php etc and this help us to see what what we have to do and when and to have a good visibility on the maintenance of the site and for each site so it's not exactly the same process for deliveries but it's also an example of great tools we could implement as well so it's a slide with the list of tools we are using with the cut quality check also it's mainly I think only open source stuff it's common stuff and well-known it's also stuff we have learned to to to use in the in conference and rubicon as well and and you will find this in the presentation the site but one day even with tools and processes and good communication sometimes bad story could happen we had a bad experience maybe the only one in deliveries with this project and that's sorry we had a lot of bunch of tools to deploy and after our discussion and testing we decided to deliver on Monday at 2 p.m. but preparing the docker image we had an incident with git lab I didn't mention docker before because I think it's not specific tools in the in the delivery just a tool we are using this project but most of these tools we are using fork and alue with docker we're also using with other solution for deployment and but in this case we are the issue with docker so it was impossible to create the image and to prepare the delivery and for this time it was us as a technical team to to launch the deployment because we had some specific comment to to run after the deployment uh finally after discussion we were ready the day after on Tuesday and we finally launched the deployment at 4 p.m. and then it was a specific day I was in a train and the redeveloper had to leave early on this day and we had a missing communication between us and a customer and finally Francis has made his technical reset uh at 7 p.m. and of course not everything was working fine and we didn't have any video and streaming it it was not working at all so we had to do something and he had the the good contact with the hosting company they were able to to launch a wall back and to take time to relaunch the delivery after okay so we say that what we learned from these experiences that everything can everything bad can happen and so it's not a waste of time to imagine what you will do if you face an incident and it's better to to write a routine for every kind of scenario you can imagine it's better to anticipate so for me the main failure in this of this day is that and it was also my fault that to to allow to start a deployment using a process we never made and for which we didn't write any routine any wall back procedure and so on so for sure it's something we'll never do again if I don't have a routine a process for a deployment that just we just don't do it and first think about how we will do and what we will do if we face any incident so and also to conclude about that what is very important when of the documentation of a deployment process okay is the steps but is to think about checking availability of every care people concerned by the deployment uh having your wall back would since uh return and also very important to have a clear and quick communication uh uh process and uh quick notifications for for example here I did not I was not notified at 4 p.m. that there was a deployment if I would have been notified earlier maybe the situation would it would have been easier to to act for this situation but hopefully everything finally it's uh it's a story a story that Finney had had a good a good end because we could recover the seat uh quite quickly so I really hope this a lot of tools could help you to improve your deliveries and I'm really sure you have other tools and other experience too it's just an example of what we are doing in our team but there is a lot of solution of course and maybe the best quality tools are start start always at the beginning of the project when you start working with the clients and you keep to maintain a good communication and confidence and after be able to to speak to have a session and to become with your clients it's always good and and always trying to explain and to share what you are doing and when and maybe uh repeat say the same thing on different uh uh system on a ticket on matamos on a on most platform and uh it's also a big thank you to the community because all these tools we are using we are using now because we we are uh we have uh I tell many camps and conferences and we have experimented tools it's all open source tools and so thank you to all of you that's it so if you have any questions there is a no question the application hello thank you for the presentation so I have so many questions but I will try to um the first one would be do you adapt your process to the client or as your client something to say about the tools to use or is it something you impose at the start of the project or something like that most of the tools are the same sometimes we need to adjust sometimes there is specific uh ask for needs for specific clients but most of this we are trying to use it because we know the tools and the process and it's really easier to have the same things to do on all projects the best example is when you're the customer wants to work on JIRA and this is really a pain we try to fight about that not about every tool but JIRA against trade mine it's really a pain okay so um another one I don't remember what it oh yeah um you have a lot of tools regarding your process or your deployment do you have some quality tools for the front end like accessibility or performance or something like that or is it a secret or it's not a secret but maybe we don't have this for the moment until we have to work on this yes thank you for the presentation I have a question for the client if I may um so how uh as a client did you come to integrate and take time to um the test routine uh indeed we yeah we uh talked about it a few days ago in internally and um it's yeah uh in some ways uh the clients should yeah uh yeah uh suppose that as the uh aid provider the the thing everything has to be tested before delivery and it's not uh obviously the the the client's job if you if you understand thank you I hope I understood uh what your your question I suppose it depends on uh clients and on the business you have uh for canal you uh canal you we don't have uh we don't make money uh for it uh and so uh we can support uh longer delays and we don't need to make deployments every day and deployments don't need to be very quick and so on uh what we focus on is quality so um and it was my approach he uh my approach is if we uh if we are not sure we can wait it's not uh it's not a big deal uh so um uh for sure it was not easy internally to convince uh my colleagues to take the time to uh uh to do these routines and to take the time for those tests but I think it's also the role of uh project manager to uh to uh to uh to to teach them and uh I would say that at the start of uh when we started the first test of deployments actually I personally made most of the tests and time after time I show them how to do it and then finally I just had to make the technical test uh so this is our internal part and also for the I would say the deployment routines it's a work that we did with uh and Sophie we talked together and we uh imagined uh uh how the best process could be yes hi I just had a question around um I think I think it was implied that you're sharing uh your red mine access between perhaps all the parties and um is there like a line where you might stop uh sharing certain amounts or how transparent are you with what's in red mine and what's shared between each party and does everybody see everything or is there like a any complex there we we had an initial red mine project for for creating the project and after we have created another one for all the tests made before the first deliveries so uh it was in the initial project the old development team me and the client um was but uh there was no ticket assigned to you I think in the first initial project and after we have created a specific project for testing and it was just a discussion between Francis and your the team and me for testing um and no after there was no no specific project we we could um we could have we have specific role on red mine and uh not everyone is able to to do everything but uh that's it one way to keep things closed between the customer and and the agency is the only place where we do this is on matter most on matter most you have a channel dedicated to the developers so all the commit messages and they come to this channel but the customer doesn't have access unless he requires it but otherwise that's we we create a specific workplace in matter most for every customer so we can also communicate but on a different channel no more question uh yes of course I think uh the video will be available on the the comment site or sorry yeah I think in the next few days every session will be with a video and the slides are shared and maybe from our agency point of view the great thing having customers who understands the quality process and the importance of tests is that you always learn from them and some of uh as you asked before some other clients we are working right now with they are starting to challenge us with accessibility testing including in our pipeline so every new project is a way to add to our pipeline a new new tools and that can be also like ghost inspector for everything automated so thank you very much for coming