 So my name is Jaco Krahling. I'm a DevOps director at NetBank and I'm super stoked to be here with my first time to India, first time speaking at Agile India conference. I have spoken at various other conferences worldwide including DevOps days as well but I'm really happy to be here with everyone. So a little bit about myself just quickly. I've got a little bit over 21 years experience in IT. Ten of those years I spent in Europe working for one of the largest investment banks and a little bit over a year ago I joined NetBank in there what we call the digital fast lane division. So it's a part of the bank that looks after or looks at disrupting the financial services industry. So we look at things like blockchain, IoT, machine learning and so on. So it's really an exciting space within the bank. But the last year I've been mostly responsible for creating a DevOps practice within NetBank and I'm going to talk about that with you all today. Just a little bit about NetBank because I'm sure no one has ever heard of this bank called NetBank. It's a South African bank established in 1831. So it's quite an old bank. Used to be called Cape of Good Hope Bank and then through a number of mergers and acquisition later became known as NetBank Group in 1903. So it's got a little bit over 137 billion rand in market capital, trillion rand worth of assets and we've got about 8 million customers. So it's probably the fourth largest bank in South Africa. Great. So today is all about DevOps and when we started on this journey a year ago, we really looked at three areas we want to tackle the spot of our DevOps adoption, being process, culture and tools. And the reason why I use this image of a bar stool is because if you think about it, if you cut off one of those legs, what's going to happen? You're going to fall over. The stool is going to fall over, right? You don't have stability and that's exactly how I think of DevOps as well. You know, you can't implement DevOps practices by just buying a tool and implementing a tool and think, great, now I'm doing DevOps. You need to have all three of these capabilities in equal measures for you to really have a sound DevOps practice within your organization. And so really what we're going to do today is we're going to take a look at each one of these process, culture and tools, one after the other and sort of see how NetBank has adopted this in their journey over the last a year or so. So hopefully it will be exciting. So the first one we're going to talk about is culture, but I've decided to add some memes to my decks. Hopefully it's funny to you. I know I had to find putting this together. Who can tell me from which movie this is? Lord of the Rings. Yes. So that's Borimer saying one does not simply do DevOps without culture chains and if he tells me that then I'll probably listen because he's a one badass guy. So what did we do around the culture element of DevOps in our whole journey over the last year? So within the bank and again, it's very important and I think the theme has also come through quite a lot in the last couple of days is you need to have buy-in from Exco level within your organization for this to work. You know, you you can try it out in pockets, but unless you have organizational buy-in it's going to be very hard to drive this culture change throughout the rest of the organization. So we established or we embarked on this journey about two years ago just before I joined and we called it EnWire, which is new ways of working basically and it's all about being more customer focused and again, that's also a theme that came through quite a lot in the last few days be more customer centric but also have a digital first strategy. So any new products we land we land on our digital channels first and then obviously to be more agile in the process in adopting agile practices more rigorously throughout this journey and ultimately to become more competitive and not just to catch up with the competition but to potentially leapfrog them as well in some of the products and services that we're taking to market. So we started to do this and as part of this we also had to restructure our teams to fit in with this model. Now I think this concept is not new to anyone here the idea of squads and tribes. I mean it's been it's been used elsewhere as well but I think the the important lesson to take away here is that you you can't just take a model as is and apply it and think you're going to get the economies of scale, right? So we had to change this to fit our needs and I think that's an important thing within your own organizations you need to understand what works and what doesn't work and to continuously change and improve on that. So for instance the one thing that we have focused on is the squat level construct. So you have a team of multidisciplinary individuals, your developers, your testers, your design leads, your business analyst, your product owner, your scrum master and your engineering leads all forming a team and being products focused. So each of these squads look at a particular product that we were building as part of this NWOW initiative that we launched within the bank and we started off with about 12 squads trying this out and see how how well it works for us and also each of these squads would take a product from cradle to grave so this team will build it and support it ultimately once in production. So there's no handoff to ops to say okay now it's your problem you need to deal with it everything is dealt with by the team as well. So it's really you know complete autonomous team while the team had options to choose some of the technologies they want there was still an element of enterprise architecture that will guide them through that tool selection process and technology stack selection process but ultimately they were responsible for for the application stacks that they used to develop within. Now you will see that the picture of the little bee hive and some bumblebees flying around. So we also came up with this idea of within the DevOps team which we call the best DevOps team to create a hive and the whole idea behind the hive is that you will have some of your DevOps engineers sitting in the hive looking at things like product selection rollout of new DevOps products upkeep maintenance upgrades and and those kind of day-to-day tasks that the engineers in the hive would typically take care of but then you have the little bees which is our DevOps engineers they will actually be in the field assigned to these squads so they will actually be part of a squad and helping the squad to build out these pipelines that that we are building for them effectively and in that process we are also cross-training the developers so that they can become self-sufficient in maintaining these pipelines and even build their own pipelines based on the skills transfer that happens between the DevOps engineers and the developers and the race of the team. The idea is that the team should become self-sufficient and for those engineers to then go on to other projects and squads and help out on those as well another auxiliary function of this is that those engineers can bring back learnings back to the hive so from that point of view if from a DevOps point of view they made a tool selection but it doesn't work out in the field the engineers can bring that learnings back and say actually this didn't work so well for us we need to go and see if there's another tool that can solve this better or maybe we've tried something completely new which we like and we can now adopt it across the rest of the enterprise so that was also part of the role of the DevOps engineers sitting within these squads and giving those valuable feedback continuous feedback all the time so from the best DevOps point of view the approach was to redefine the software delivery process through the colors might be a bit hard to see the six areas that that DevOps team would take care of the planning and development which is really so we use a clashing tool stack for that JIRA for issue tracking and user story and so the DevOps team would also create the workflows as you go if for you that know JIRA you have the concept of workflows so we would define the workflows for the squads up front so they can just use it almost like a template for how we create user stories and how we transition those stories through the different stages because our pipelines are fully automated and takes those into account as well when we build out the JIRA workflow as well and then they also work on the continuous integration deployment validation and also optimization of the tooling that we support within within organization one important thing to also remember or to to know is that the best DevOps team are actually sitting within the change management organization so why is that important because if you think about it change management is going to be one of your biggest inhibitors to adopting DevOps practices because there's always going to be a lot of governance checks and stages that will inhibit you from deploying as frequently as you like because you know you will have to go through things like change control management and open change requests and you can only deploy during certain change windows and those kind of stuff so by having the best DevOps team aligned with the change management team we were actually able to start breaking down those barriers because they are working very closely together and just to try and and remove those barriers and get to the point where we can continuously deploy all the time without having to manually raise change requests and get approvals and wait for change windows to open so that we can deploy and working collaboratively with that team we were progressively been able to to knock down those barriers as well which became a big part of the whole cultural aspect of it if you think about it you know it's it's unless we get that buy-in it's going to be very hard to adopt these kind of DevOps practices great so that was more on the cultural side things that we started to do in the last year or so if we moved to the processes this is a good definition of debugging being a detective in a crime movie where you are also the murderer and as a developer it you know it does happen a lot where you inadvertently introduce defects into your code and then you have to debug your own code as well so I find this to be quite quite true actually so what are some of the processes that we've introduced in our whole journey so the first one I want to talk about is a source code strategy and it's all about you know what kind of source code strategy you're going to use you know there's again this is also comes from the continuous delivery book the trunk-based development but there's also another one called git flow and so we've actually taken a combination of both the idea is that we want to get to the point where we do trunk-based development where every commit that the developer does happens onto the master branch but we also realize that that's not always feasible in in in our context so what we've come up with is short-lived feature branches so by using short-lived feature branches and again these feature branches shouldn't exist for more than two to three days you know we're not talking about weeks and months we're talking about days at a time that that we will skip and because we know that the longer we wait to merge code from a branch backing the more difficult it becomes you get into merge conflicts and all sorts of nasty stuff that takes away a lot of time and debugging and trying to resolve those issues so what we've said is okay we will use feature branches but it must fit within a sprint and especially within a couple of days if we can do that and then also really to be able to augment that with test-driven development where when we do merge that code into the master branch there's a certain number of checks that we're going to go through to make sure that the quality of the code you're committing is of a certain standard so we won't just allow anyone to commit code onto the master branch because that master branch is your path to production effectively any time you should be able to take whatever's on your master branch and deploy to production at the click of a button you know so you can't have broken codes set on your your your master branch and and we'll talk about that a little bit later as well we also do things like peer review so anytime you decide to commit code you go through a pull request and that gets reviewed by your peers and that's also part of the quality gates that we have in place before before we do that mergers then something we've started to play around with more recently is feature toggles so feature toggles is really interesting because especially where you have code sitting on your master branch that's maybe in production but you might work on new features that's not yet in production yet all your code ends up on this master branch every couple of days how do we prevent that unfinished code effectively from ending up in the hands of our customers so we use feature toggles for that so we can switch off those features whilst it's still in development and when we're ready we just you know flip a switch and that feature is available in production so we don't have to go and redeploy code or or merge code back into the master branch and now go and deploy it into production it's it's all there test driven development jess just spoke about it this morning in his keynotes as well that's actually something we do in force in the organization and it actually took a while for the developers to to start doing this because you know as a developer myself you know nothing's more annoying where you just want to start coding and writing code and now you can't because you need to write the test before you can start coding and it's actually quite frustrating but there is method to the madness in in in doing that and by doing so you actually make sure that the developer really understand what he or she's trying to bolt by thinking about it before you actually start coding which actually helps the thought process and the design process a lot um like i alluded to on the previous slide you know we won't allow a developer to commit code unless they're at the certain level of code coverage and those kind of checks we run on every commit so if you commit code without the necessary unit tests to support that we will block that commit and throw it back to the developer and by doing that you start to enforce these kind of processes that the you're trying to adopt cool continuous integration another process that we decided to to take quite seriously it's not just about compiling and shipping it although i wish it was like that sometimes again you know we have a lot of approval gates that we put in place from a continuous integration point of view to make sure that once the code lands on to or what we call our continuous deployment pipelines we know that it's reached a certain degree of quality before it goes on to the pipeline because remember that those pipelines that you create can run all the way through to production so again anything that lands on the pipeline you need to make sure it's of the highest quality otherwise you could introduce defects in production very quickly and bring down systems very easily so continuous integration does help with that and i'm going to talk about that a little bit later as well in more detail and really you know one of the key pillars to to building your DevOps practice is having a robust sound CI strategy in place and then the deployment part thereof the pipeline or continuous delivery whichever word you want to use for it it then becomes your your path to production right how you would jump through the different stages different organizations will have different stages that you need to jump through to get to production we have a number of environments that we need to go through before we can deploy into production and we need to prove that we've tested all our code in all of these environments before we can deploy and again the pipeline takes care of that for us right through automated testing and other practices this whole thing is fully automated right there's no manual intervention you don't have to install code manually you don't have to kick off tests manually everything is basically a zero touch effectively and you need to have that kind of process in place if you're going to make this really work for you because again there's no point in having agile development but it still takes you three months to four months to get it into production because you have manual testing and you have servers that takes two weeks to stand up and you have approvals that needs to get approved and firewall changes you know it just breaks down at that point and and again this helps with that which sort of brings me to my next one which is the continuous testing components as well so as part of these squads like I said we actually have the QA team sit within the squad so they are part and parcel of the team they sit with the developers actually literally next to the developers and as the developers start working on these user stories the testers are starting to write the the acceptance tests and regression tests alongside them already so so we use you know bdd development so serenity and cucumber for instance to do a lot of our automated testing in and and they will work very closely with the developers to make sure that they've got this so that by the time we deploy or commit code and we deploy through the pipeline that pipeline already has the automated tests stages or steps in place exercising the code as it runs through the pipeline so really that works quite well if you have your testers and your developers very closely aligned and that also gives you the fast feedback right because what happens is if it breaks you need to know sooner rather than later because from a developer point of view if you write a piece of code and that code only gets tested in a week's time is that contact switching problem again right so the developer might have gone and started working on another user story now a week later he's going to go back and rethink what did he do on that user story to solve it if you can do it within the same sprint and have your testing done within the same sprint then there's a better chance that you'll get that resolved much quicker than then delaying that process which brings us to our last part of the strategy it's the tools look we can't get away from it we do need tools i mean you can't get automation going without tools although hopefully as you can see it's not by far the the biggest component of of dev adoption but certainly one of the three important ones and as Darth Vader says i find your lack of tools very disturbing so you need any tools you need to have proper tools to do the job so i'll just run through a couple of examples of tools that we are using in our environment at the moment first one Jenkins from a continuous integration point of view and we so Jenkins has this concept of pipeline as code right so that's what we use when it was nice about pipeline as code is your your Jenkins file actually sets inside your source code repository alongside your code so anytime you need to deploy code that file defines your CI process effectively you know what stages you're going to go through in terms of getting your code out the door effectively and again it all lives in your source code repository and if you read the continuous delivery book that's also how they're recommended every configuration item needs to live in your source code repo and and Jenkins makes that quite easy to do so we've got a number of stages so for our continuous integration stage takes about three and a half minutes so that's everything from compiling the code to doing a full unit test run and to do we use sonar cube to do the code coverage analysis so we test the code coverage we also look at vulnerabilities on each commit to make sure that the developers haven't introduced new vulnerabilities in the process as well it's actually quite funny because this particular project is built on dotnet core which is microsoft but we compile it on linux slaves so it's actually quite a quite a joke in the office because the developers are working in visual studio but everything else is happening on on linux machines under the covers but dotnet core makes that possible right because it it's now supported across multiple platforms so all our slaves are running linux buntu and so we are able to actually build and test it in that fashion and then there's just some graphs around sonar cube as you can see for this particular project we have 80% code coverage on this project and like so over 300 unit tests that runs on every commit what you will find is that on the last step the publish step so there's a test scan and then publish so what publish does is it will now take the compiled code it will create artifacts so it will create the binary version of those source code and then store it in an artifact repository so we use nexus for that so we will store the artifact in nexus so that when we do the deployment during the continuous delivery portion those components that's responsible for deploying the code will go and grab it out of nexus so again it makes sure that no one can taint the the source code you know no one can slip in there and and recompile a component and then slip it into a particular environment if it's not in nexus then you can't get it onto that box effectively so again we put some controls in place there so one of the tools that we decided on on using is called excel release is from a company called Zebial apps so Zebial apps develop enterprise DevOps tools as well I know a lot of people like to do this through scripting and again there's multiple ways you can solve this you know there's not just one tool for the job but what I like about excel release is that it's it's very graphical you know you can visually see all of the phases within your pipeline so so they call each release or each iteration through the pipeline they call it release so although it's not a release from a production release perspective it still runs through the whole pipeline and as you can see is that we've got a number of phases pre-release develop e2e and qa you'll notice that there's no production phase here and that's only because our production release sits on a different pipeline and that's again down to governance right so the release managers whilst they were happy to break down all of the barriers between these environments because previously you had to have change requests even going from one environment to the next we've we've gone away with that but for production they much rather prefer still going through that manual approval process and it's a journey right you know we're not going to get day one all of these barriers removed you know and as they start to trust these pipelines have been building that hey wait a minute these things are actually delivering quality code we can start to trust the pipelines we will slowly but surely start removing those kind of obstacles as well to get to hopefully a true deployment pipeline all the way into production so again that's work in progress and what you'll see here is that we do a couple of things through our pipeline so firstly we set up test data so the pipeline will do things like reset the databases to a to a known state it will create new test profiles that we need to use for test injection test data injection into our tests and it will do all of that stuff right up front we also interface with JIRA so again what happens is as you kick off one of these pipelines we need to know well what defects or what user stories are we actually testing as part of this release so XR release talks back to Jenkins and we can actually pull a list of all the user stories that's related to this and we now have context so we know what are we doing in this particular release and therefore we can also start to do things like automatically transitioning those tasks in JIRA automatically and that's why I said in the beginning you know we also define the workflows within JIRA which makes this possible so that as the pipeline starts taking care of transitioning your applications from Dev to ETV to QA to prod you don't need a scrum master to go into JIRA and say okay we are now in QA okay now let's drag all the tasks up to QA because that again is a manual task that someone has to do the pipeline does all of that automatically there's no need for anyone to do that and then from a deployment point of view we use another component which I'll talk about now called Excel deploy which does the deployment of the physical code or the artifacts onto the environment so in this particular environment we were running on VMware and Windows with IIS and what it would do is it would set up the environment for us on each deploy we actually also have Ansible in here as well so what happens is at the very first step in each one of these phases is and you probably can't see it it's actually an Ansible cookbook that we run and that will ensure that the servers are in the right state known state for for that application to be deployed so we make sure that we have the right version of IIS installed make sure we've got the right patches installed all of these things because one thing that happens a lot in organizations it's what we call environment drift so environment drift is like you know people start making changes on particular hosts or servers and they don't replicate it across all all of the environments and then when you deploy something and it breaks you wonder why did it break you know it worked in one environment but it doesn't work in the next environment and that's largely due to environment drift so by making sure that all of these servers are in the right state we get consistency around that predictability and then I can say we have the full automated regression test as well as part of each each pipeline and then we also have the performance testing in here so that's performance testing is an interesting one right because performance testing used to be like an exclusive function of a performance test team within the organization where it's a very specific skill set if I can call it that that and the problem with that was that again if you wait till right at the end when you're ready to go to go live you then bring the performance test team into into the pro process but by then it's almost too late you know they have no context of the project and what have what happens if it fails a non-functional requirements do you not go live do you rush to try and get improvements and that becomes a very difficult situation to navigate so by having the performance test as part of our pipeline we can actually do performance testing on every bolt that we put through the system and that also gives us a very early indicator if things are starting to go wrong very quickly so the performance reports actually gives us that insight and again that closes that feedback loop that you need back to your developer so say wait a minute that new library that you introduced has created a 200 millisecond latency overall and you need to look at that for instance so again the commoditization of performance testing is really key to the strategy and then lastly we also have integration to chat ops so we use Slack every time we do deployments we publish it on Slack so we know exactly you know which bolt is is being deployed right now and in which environment is currently setting and all of that communications is pumped through to channels in Slack that that we use and we can do this whole process in 45 minutes and the only reason it's 45 minutes is because the performance tests run for half an hour out of that 45 minutes where we like really you know we we do about when we test this particular component we do about 300,000 requests in in a 30 minute time frame to see how the system behaves and that's all part of the pipeline so I spoke a little bit about Excel deploying which like I said is the the automation part of deploying your code onto onto one of more of those servers the only thing I wanted to maybe highlight here is that you can very easily see where in the process you are so you can see which versions of your application is deployed to dev, e2e, qa and and also into production so it's very nice to see which bolts are setting where in your environment but there's also this concept of dictionaries so dictionaries is again the idea externalizing your configuration management capabilities so you have different environments with different configuration properties and it's all kept in dictionaries even things like passwords are kept in here because it's masked so therefore you know from a security point of view you're not exposing any passwords in config files that sits in a source code repository and as you deploy into environment it will take those dictionary items and apply it onto the config files and then deploy it onto those servers so it works actually quite well but with that being said so that was based in 2018 where we started with this at the end of 2018 we decided to move this particular application to a microservice architecture because it was built on.net core it sort of was the precursor to our microservice adoption and it made it a lot easier to do so this year we actually started to change the architecture slightly where we are now running that component in a Kubernetes cluster effectively and so as you can see we still use Jenkins to do the bolts and then what Jenkins will do is it will create the image the Docker image which we then upload to Nexus so all of our Docker images sitting in Nexus at the moment we still use Excel release to talk to Kubernetes so Excel release will basically go tell Kubernetes to stand up the servers go stand up any other auxiliary services that we deploy as well to set up the ingress policies for this particular application and everything's happening through the API layers we're already in Python scripts effectively what it boils down to that interface with the Kubernetes APIs and we're able to to orchestrate that through Excel release what we are now looking at as as we go through our own continuous improvement process is to start looking at Argo CD so Argo CD is also an open source project on GitHub you can go and check it out it's really cool for deploying Kubernetes based workloads and again there you can define what that pipeline should look like so you can define in also in YAML files similar to how you structure your Kubernetes resource configurations you can do that in Argo as well and then it will take care of the deployment to your dev, test and prod clusters and so this is something we're starting to look at as well at the moment so the performance testing I touched on this a little bit earlier on and this is just an example so we use something called blaze meter I'm sure if you guys have heard of blaze meter so it's based on jmeter which is open source blaze meter just makes it more convenient right because they have put all of the plumbing in place that allows you to do massive parallel performance testing using their infrastructure basically they've got servers sitting in all the cloud providers that you can reuse but you can also have what they call private locations so private locations you can only test from within your organization you can actually have private locations that talks out to blaze meter so you can still run your tests from within the organization and what's also nice about it is based on YAML file definitions and the language they use is called torus so again I think it's toruscli.org if you want to go check it out but like I said it's a YAML based configuration which is really really nice because it allows you to define things like your concurrency your ramp up time and also your scenarios can be very very detailed in terms of you know what payloads you send to what requests what information do you want to harvest out of those responses and reuse in subsequent calls within your test and it's all done through through YAML configuration so I think it's really the time to value is very quick effectively and here you can just see an example of that report we actually wrote the plugin for this by the way so it didn't exist in excel release this particular plugin so we wrote and contributed back to the community what we've done so anyone else can can start using it if you have blaze meter as well so basically if we look at our base DevOps strategy these are a list of all the tools that we're using today it's quite a lot but I'm not going to go through all of them but basically you know we're continuously evolving continuously looking at new tools and again coming back to that construct of a beehive you know these guys are hard at work to evaluate new tools and come up with better ways to solve a lot of these problems and it gives you just a taste of some of the things that we are using today and we're also looking at things like chef as well to do configuration management as well going forward as well but really you know what it all boils down to is the lessons learned right so drink copious amount of coffee it's not going to happen overnight right it's not something that you know I hear a lot of people saying oh yeah I you know we bought chef we're doing DevOps but it's not that easy or it's not that simple you know there's a lot more to it the tooling side is certainly one aspect of it but there's a also aspect to it that's more around how you change the organization as you hopefully saw on the previous slides and it's going to take time this handbook which is also I think out in the bookstore the DevOps handbook was a very valuable asset in coming up with our own strategy for adopting DevOps and I would really recommend that and you can even get jays to sign it probably here on the floor as well and sell sell sell this might sound strange because I mean we're not in sales but actually we are right because even within the teams and projects that we're working you know we need to to take those learnings to the rest of the organization and if we don't do that then we're not going to be able to influence you know microchanges in the organization if we don't do that so even as engineers you need to take your learnings to you know to your sponsors business sponsors or to your colleagues that works probably in other teams and that's where the whole chapter idea is also quite neat because if you have a chapter set up you can get everyone that's got the same interest coming together like once a month and you can share those kind of learnings as well with your peers across the organization which is a good way to sell within your organization but above all just start doing it you know it's a lot of terms especially my consulting days where it was a situation of yes we are looking at how we can do this but we're not actually doing anything we are getting into this whole analysis paralysis phase where we just too scared to start exploring and start experimenting and I think the golden threat here is you need to start doing something and you won't learn until you start experimenting and like I said you're not going to get it right on day one but we are also going through that journey and learning new things and improving on that but you can't improve if you don't experiment and challenge the status quo you know a lot of times in organizations especially larger organizations like the banks a lot of people are set in their ways you know that's how they've been doing it for the last 10 years and who are you to challenge them on how it should be done you know so what if it takes two weeks to action a ticket that's how we do it you need to start challenging those kind of behaviors and and and start to maybe include them so that I can start seeing what's actually involved and so that they too can go go come on this journey and start improving their own practices because as we saw earlier this week as well you know it's great that from an engineering point of view we're doing all of this cool stuff but if everyone else in the organization has not bought into this then they always can have the struggle and so what if the color is pink I'm not wondering what on earth what does that mean so one thing my manager always says to me is you know it you know a lot of times as engineers we get sort of attached to certain tools or certain technologies and we like you know you know we might love Kubernetes but for some reason the organization might have chosen OpenShift but you're like I don't like OpenShift it doesn't matter you know make work what you've got in your organization you know you don't have to rip and replace everything to get this working you know if your organization has made some decisions around technologies use it if by using it you can come up with actual data to support that it can't do the job then you take the data back and say actually these tools are crap we can't use it we need to reinvase or re look at other tools and that's cool but again try to take the the the the subject of element out of the picture and say well let's use what's there and then come of data points to support our arguments going forward but above all just have fun it's that's what we're here for and if we don't have fun then why are we doing this and with that I wanted to close off and say thank you very much for for this opportunity. A few minutes for questions. Thanks thanks for the talk and sharing your experience I have one particular nugget that I was looking and I couldn't find when you were talking through is security now you know banking is is also a place where security takes a lot of precedence but I couldn't see that as a part of the pipeline or the things that you're using or the last one so do you want us to share so so security you know there's a lot of the things have entrenched security bolt into this into it right so for instance using xr release to do the deployments you know that's gone through rigorous security risk assessments as well to make sure that the tools we use are you know capable from a security risk point of view to safeguard the organization around things like passwords and stuff but then there's also things like penetration testing right which is more to do with okay so anytime we release new software we have to go through a level of penetration testing to make sure that you know we again haven't introduced new vulnerabilities that is not on here because that is the one element that's done manually quite right and that's something that we are obviously working on to see how we can automate that and make that part of the process yeah yeah and that's why I say you know that um so part of that is done through sonar cube so sonar cube can do security vulnerability but then there's the static application code analysis right or security testing static application security testing that but it's still manual hi jacko quick question I know you've mentioned no time so it's like understand what no time is and from quarterly to weekly no time right so and follow up the question to that is that like understand what's the ROI because right now looking at the tools I mean like it's quite many and it's a good amount of investment plus we also have the current running process to the mind shift context which and people probably learning and acting on it it's a cost as well so what's the ROI that currently that you see and I mean how soon you think you have found that you know for your organization yeah how soon you got the ROI cool good question so um so before like I said before we we started to bolt this this capability if I can call it that you know production releases only happened like I said once every three to four months okay so it was a very slow cadence at the time so we run two weekly sprints okay and every Friday we do a production release um so that's the cadence we're on now so every Friday of the week we do a production release and then every second week what will typically happen is if you do your um your your your demo days that you have with your business stakeholders at the end of the sprint to show what you've been doing typically after that show and tell we will go and press the button and deploy that code into production so if they are happy with what we've done um we will deploy the production so that's happening on a weekly basis now obviously there's there's times where we're not introducing any new functionality then we might skip a week but typically it's like every Friday and the idea is to bring that down to daily releases but like I said it's a journey we're not there yet but we would like to get to that point we can do daily releases um as for an ROI so to give you an idea is we bought a capability using this methodology which took us 16 weeks to get it into production and in front of customers and there was another project that's running in parallel that's been running in parallel to deliver on the same capability that's using traditional waterfall they've been working for two years they're not in production yet in 16 weeks we landed in production and within three months we've made over a hundred million revenue uh on that product by deploying it earlier so from an ROI point of view we were able to do to realize business value a lot sooner than some of the traditional IT projects that is still trying to catch up with with this project. If you have questions you're good at hanging them down. Cool. Awesome. Thank you very much.