 Hello. So I'm gonna start. My name is Rolando Henry. I work for the Ontario Digital Service. This will be my little discussion about DevOps. I realized I might have been a little ambitious with my alliteration when I came up with my title for this. And so after seeing a couple of the other talks and I think they're gonna be going a little more deeper into some of the things that I will be talking about. So this is more kind of a for someone kind of learning how to scuba dive, deep dive. So kind of snorkeling in a pool at a resort kind of style. A little bit about me. Like I said I work for the Ontario Digital Service. I've been in government for about eight years now. I like patterned shirts. I sometimes democratically get them chosen for me the next day. And I have a cat. His name is Lord Perkins. He's quite good. Sometimes when I work from home he helps me solve some problems. So basically you know what is DevOps other than kind of a portmanteau of development and operations. And really the idea is it's kind of a philosophy and kind of culture about how to develop. You know in keeping in mind that you want to be kind of agile and agile. And you want to release your code faster, get more feedback faster, kind of resolve issues faster. And you know be a little more efficient about releasing your code with automation. You know continuous integration, delivering deployment. And so you know there's this kind of feedback loop where you get to you know you kind of plan, you create, you know you build, you test, deploy it, you know kind of monger it. And then take that feedback to back to the developers to kind of help them you know fix those problems. And you know that DevOps culture where you're kind of the developers are working directly with you know those operations people you know you make the developers life easier and kind of make the operators life easier too. You know the last thing is as kind of a sysadmin you know when a developer comes in like hey can you release my code and you're like okay how does it work. And then you try to do it and you know there's a long list of things you have to configure or change. You know this this process of kind of just having that you know integration with the the operator and the developer just makes you know life a lot easier. You know when you think of the development process a lot of times you can kind of get that oh well you know it worked on my sandbox or when I was working on it locally it worked perfectly. So you know it should be working you know when we deployed it to Sage or or to prod. And so this idea of DevOps in this culture is to kind of make that whole process consistent and you know iterable and so that when you commit some code and have it deployed either automatically or have the sysadmin do it you know the expectation that it should work will be there and you don't have that kind of you know oh it worked here but why isn't it working here. So for local development you know as a DevOps I kind of try to help our developers work better. There's been some other talks already about Drupal VM a colleague Nick who you know Drupal VM is a dream as he says which is you know and Drupal VM uses vagrant and Ansible and from a DevOps perspective that can be actually really great because that vagrant image can mimic you know your stage and production environments almost exactly and the the way Drupal VM works is actually with Ansible which is a whole bunch of playbooks that kind of automate and codify the entire build process which we can then actually use those exact playbooks and rebuild that in the same environment or even package up that you know image and use that as well. Another one where I think Derek is actually doing a deep dive on Lando right now and I approve of the name Lando because that's actually another nickname of mine but I haven't actually had a chance to work with it and that's more Docker based and that can be really great as well if your production environment or your staging environment is already using Docker. Lando is really just a wrapper for Docker compose and Docker makes things simple and you know if you can get your developers to kind of do things that are in the production environment but in a simpler way that they don't need all the nitty-gritty of all the Docker setup then that can make you know the system in slide is easy as well because you're basically just kind of taking that code probably into an image and doing that same thing and so it keeps that development release cycle you know consistent and you know you can always do it and expect the same results. Another great thing about kind of where things are going we've seen a lot of great tools about continuous integration and deployment is that you know developers can push the code and see results in you know whatever tool they're using and instead of you know making commands like hey can you launch this or put it right away even before it kind of gets there they can get that feedback. You know we at the Ontario Digital Service we're using GitLab so all our projects are in GitLab and as DevOps we try to you know also mentor our developers and kind of help them use good processes like GitFlow we use and we're kind of you know it becoming a little more agile as well like we're fairly new at some of this stuff and kind of that feedback loop that I showed before in a meta way also kind of applies to DevOps in that DevOps we're kind of building new features helping our developers getting feedback and kind of putting it back in and now we're kind of enforcing more of that kind of CI, CD, continuous integration and deployment for projects coming in and I'll show you an example GitLab file that we use for one of our projects that Nick is working on and I'll do a just I don't know if you can really see that so I'll just step kind of through this here so basically and we've seen this similar things with Travis and CircleCI with our GitLab you can do it's very it's great and that you can run just scripts you can set it up to run as like a you know docker in docker so more like so can either work kind of like Jenkins where you can just run scripts or you can kind of set it up to be kind of like CircleCI Travis where it's it's running based in docker and for our Drupal project you know this kind of round of work before it was we would commit some code we would tag it and then we would take that code and deploy it but we would do like the the composer update the theme node build as part of the deployed process but in kind of this round of our continuous integration we actually do the composer updates as a part of the process stage and the the node build so you know instead of in the deploy you know some composer dependency fails or something with node fails or having to worry about which version of node is on those boxes you know a developer can you know do some work and see those errors early that they may not have seen on their local machine because sometimes you know you can install a module locally but forget to commit something and then you do the actual commit and it goes and it's like well where's this dependency so now we you know that compose install you know we can use an image that's kind of fine-tuned for just that PHP portion and then do a node build where you know just use the specific version of node and then what we do right now is we kind of package that up kind of like an artifact and put that up to our code deploy and then that gets that code during our actual deploy gets pulled down we do a you know the actual up DB and all that as part of the deploy but that's the only part that we kind of have to worry about we don't have to worry about all those additional levels of waiting for dependencies to install it's all kind of prepackaged where did my presentation go it is gone okay this is me trying to use Google or this Google slides but it was there and let me just there it goes anyone there it is no sorry guys it was oh see this is when I let the the screen I'm like looking at my screen not seeing it okay great so I'm gonna go back to this sorry so in GitLab each commit has a pipeline and you can set that CI pipeline to only work on certain things so say the a testing portion you could have it do on every single commit the building of stage credentials can be done on develop not master or on release branches so you can kind of give it certain parameters and you can have a manual deploy for only master or just tags so you'll see a lot of you know control make it a lab which is great for for system that DevOps so you know certain stages of that development process you know is very automated and then certain steps where we may not want code to go directly to prod just yet where you know there may be additional steps that as DevOps we need to do we can do it a manual way it just kind of provides that level of control a lot of you know that automation is about hoots you know you think okay we've we've come from the days of in on-chain digital service where we would go on you know when we wanted to do a deploy we would literally go onto the boxes get pull you know do all the setup and it's like this is kind of ridiculous so let's start automated this but then you go down kind of rabbit hole of well I can automate this way and certain tools at the time let's try it with you know with salts or with puppets and then you start going in and you've spent you know a good amount of your day automated something that you could have done you know really quick but the idea is you don't want to always go onto those boxes and do it and so over the years we've kind of changed our our processes and we're you know really getting in line with this new kind of DevOps you kind of culture and a lot of the automation tools are already out there that we were writing our own batch scripts and Python scripts to do a lot of stuff that you know the CI does and what Ansible will do yeah okay that is our so right now we're also using Ansible to to deploy and let me try this again so basically what's nice about Ansible is it's all kind of based on modules and steps and it's very you know do this step and it's kind of self-explanatory of how you write it you know you kind of give a comment this does this this will do this and you know you basically just run a little command or we're starting to experiment with AWX which is an open source version of what's called tower for Ansible that basically you can create these playbooks and literally just push a button and if you run you know these really long playbooks or and you can configure them by you know adding tags to certain steps say okay on these machines do this step or only do this portion or something fails to go back but because we've gone in and fixed a certain thing you know start here specifically with our Drupal 8 deploys now is in our automation 9 times out of 10 it works great you know the pull down the code to the you know Drush up DB just in case through Drush CIM for our deploys we copy it and then once the configuration has kind of changed we just simulink so that's kind of the you know the live version but every now and then there will be you know you need to run up to twice or CIM twice or just you know something kind of craps out and that's so Nick and I will sometimes when we do a deploy is kind of sit together and kind of go through but you know most of the time now it's you know you can give it to kind of a junior they just go in just ask for the tag it goes and you know as this is kind of going and again in my kind of programming mind I want to automate a wave saying oh you need additional steps how can we add an additional layer to say I'm going to do a deploy but I'm also going to do these additional steps to kind of help out that automated process which kind of leases this general problem of you know you want to do one thing and you could probably do it really fast but you also want to set up a way so that the next time it's actually asked you know it's it's automated this a lot of my day is kind of is like oh I'm doing a lot of the same thing so let me find a way to do it but then you know you spent a lot of time ranking scripts another kind of portion of of that DevOps feedback loop is you know monitoring this code that goes out and right now we're starting to use the elastics back and sending like log metrics log metrics like engine X PHP and we're sending metric information like about how the instances are running and we're also actually sending packet information which can lead to you know some interesting debugging I'll do a demo soon for Drupal specifically you can set up you can even send watch dog logs to elastic search either setting up watch dog to go to syslog and then that syslog would automatically get picked up by file beat there's a module called JSON log where you can kind of take those dresh logs and you can you know set a filter say only do warnings and errors and it pre packages it as JSON and then that just gets picked up and you can do more analysis on that I'll do a quick demo here so this is kind of our production environment and the kind of basic engine X logs and so you can basically set you know time period so this is for the last seven days you can kind of see where traffic is coming from but what's kind of nice about this is you know you can also see kind of where the errors are coming in and it's like oh well this is you know a significant spike in errors and so I can go like that and kind of drill down to that time period and you know find out okay there's obviously something weird happened on the long weekend going on to those engine X logs and you can find the actual actual error so this looks like some upstream you can so we're gonna just look at response codes that should show something so we see a lot of 404s and you know you can see because it's Ontario.ca it's also used for like the actual LDAP so we get a lot of this auto discover but what's interesting is we also have machine learning on our engine X or logs and we have this thing called anomaly explorer and you know we can get down to showing basically you know it will look at your engine X logs and say oh these are you know kind of events that are out of you know normal normal range and then you can even go and you can explore that and what we found actually is during that time someone was actually spoofing a 10 dot IP address and hitting some weird there was just thousands of these just hashed sites to the point of you know normal traffic on a Sunday you know less than 10,000 hits to you know over 50,000 and so a lot of DevOps a lot of my day sometimes is just kind of coming in and taking a look at what has kind of gone on with the the site but or if like a site's been slow we can go in and look at this information and kind of debug you know was it was it PHP was it engine X sorry this one you know and you can even get down to my SQL performance I have good thing I have backup of images but so you know you can look at the logs and then also that the metric information about the actual instances that we deal with are there you know down to the information about you know what kind of instances they are which region this is the my SQL kind of performance control errors how much of you know that throughput there is there and you know to getting the developers access to those dashboards as well they can go in and kind of see information about you know if there was 500 was it something that was caused by the release because there's salt you know based on time you can kind of see oh I did a release and all of a sudden there's a huge spike in 500 errors we can give that feedback back to developers or they can see it themselves going through this these dashboards that we're starting to you know empower our developers with ways or our tools as well and so there's that you know with all that monitoring and kind of empowering developers we can help plan you know their strengths and what they need to develop or fix on their you know their projects sort of it for now but do you have any questions on some of the tools that we use yeah yeah so that's yeah so it's I think the elk was elastic search log stash and cabana and so cabana is the visualization portion of that stack elastic search was the search portion portion and before there's a service or application of log stash where basically just pump everything to log stash you create these pipelines to kind of enhance the data but now elastic has these things called beats which are subsets and kind of predefined pipelines so the file beat specifically only looks at file logs and has modules so there's like an engine x module and you can look at you know like drop rates putting things into categories yeah exactly so the categories and modules metric rate is specifically just for metrics that's all it's looking at packet beat looks at packets and it has specific modules like you can look at like you know Postgres mystero traffic model they're kind of separate and you can even technically use beats like so every instance we have has a little a beat Damon running that kind of processes that and right now they're just kind of going directly to elastic search because our structure isn't that large per se but if you have like a lot of services and especially if you're on a kubernetes or open check you have a lot of containers you can send those beats to say a log stash service that can then have like a queue and so it ensures that all information goes or you know it just helps the processing you can add additional metadata you can say you like pseudo-abstall yeah pretty much no so file beats say you go to be on stuff I'll be and file beat modules enable engine x and it just knows where engine x is what kind of information kind of comes through it adds and it processes everything and then elastic search itself self has these ingest pipelines that basically do the geo IP look up they you know can enhance that data and elastic itself also has a grok or debugger or you can say you have custom log messaging that you know isn't covered by any of those modules you can put a sample message and kind of the idea of what you want and it'll tell you you know how how to group that information or you know this is a time stamp this is the code it's a I actually had to add an additional pattern for PHP FPM logs because right now there isn't kind of a module specifically for PHP FPM but the laws are always there and they're fairly consistent so you can just be like if it's PHP FPM just apply these kind of rules and that information goes and another thing that we're going to be working on soon with a lot of our API is is called APM application performance metrics which basically is a little server that runs that sends information to that's the stack but then there's a node module that you install on an express app that kind of what's the word instruments all your files and so as you're running like any API call it'll go in and show you the how well it did but then also the window of say it took this much time to call my SQL took this much time to cause you know top to Mongo and the return response and if there's an error it actually goes and you can click on the error and it will open it'll show you the exact staff and line of code where that happened I'll show you now that right now we're not actually using it but we want to in our APM application performance metrics and it was a separate company but elastic prepared them like bottom and it's really cool and so there's for PHP API I think there's one for PHP there's one for Ruby there's one for JavaScript and the idea is send as much as possible to elastic search so you can send everything so that if something does happen you just close that like choose the window of time and you can drill down to like you know line of code type information any questions I can keep asking yeah I mean the DevOps discussion so I think we've been looking at setting up for some of our stuff is for me for monitoring stuff yeah no we had talked about from ETS for metrics for metrics and there's some overlap of what they kind of do yeah but we you know being kind of government we kind of have to watch out like for a lot of paid services that we kind of do and so this elastic step we're already on AWS and so we're using elastic cloud through AWS marketplace so it's already being applied to a subscription service that we pay for or our billing so we got to experiment and play with it and we're you know we started small like our team was you know six to twelve people before so a lot of it was can you learn it and do as much open source free kind of things that we can do but we did talk about using for me this but I think I was like really big into elastic search and kind of still am so it just kind of like so I was just looking at stuff actually the overlap and on the fact they age for me this it says hey how do I get logs and then it says don't use these something like the up stack yeah so yeah it's and I think there's even I think you can actually send some from ETS data into the last six back as well and so there's a small lap there I just find it's just nice to have everything in one spot yeah I think I mean that's it looks like that's where it's going people don't want to have five separate tools for all this yeah I mean you can you can go you know fairly deep into that and create you know logging you know say you do get commit you can send information to elastic search about the CI pipeline and so you know if you follow a sudden there was a spike in 500 errors you go down you look at your logs and say oh there was you know this pipeline in here and during that time this one machine you know there was a CPU spike and then you can go drill down I was like oh yeah and actually even in even in here some of the dashboards there's like SSH logins so you can actually see if there was you know an attempt to log in or if someone did log in that you didn't actually notice and you can see pseudo commands and it's like you can see the whole trail of everything that happens under instruction