 Yn ymwybod, mae'n cael ei cyfnod. Mae nhw Adam Moss, yng nghylchedd yng nghylchedd ym Mhau Argynfod, Fyeth ymwinell, i'r ymdweithio i'r Argynfod, y Ddweithio Wyrddol, ym Pen sy'n gyfle, mae'r ddweithio i fynd, yn ymgyrchu'r cyflwynt yn gwneud. oedd yw'r gwaith o'r rai gyrdd ymlaen o'r gweithio. O'ch gweithio, mae'r ddechrau'r ddechrau'r ddechrau. Mae'n ddweud o'r 22 miliwn o'r cyfnod o'r ddechrau. Mae'n gweithio o'r region o'r 84,000 o tyfu. Mae'r gweithio gweithio o'r Code 55 miliwn, oedd o'r gweithio, oed o'r ddweud o'r rhaglen. Mae'n gweithio o'r region o'r 10,000 o'r ddweud o'r systemu yma'r ffordd, y bydren i ni'r ffordd. Fodd ychynig ar y cyfnodio a'r ffordd oeddion oedd ar gyfer cyfnodio. So nad ydym ni'n defnyddio'n ymddangos. Rydyn ni wneud i'r gyfaradau gael yna, yn ymddangos yng Nghymru, ond rydym ni ar gyfnodio ddannu 200 miliwn gael y keeperiael. Felly dywedai fel ynghyluon gyda gwirio sydd gennych y cyfnodio. Felly na'r ffynu'r eu cyfl Fanonwyr, mae'n bwrdd hwn i ddechrau, mae'r llwyfod honno, gan gwych, â'r ddechrau tylfonnyn, erioedlau'r cyfloldiadau, ac iddo ond fe y llwyffydd o organisations oelod. Yn 10,000 ym 10000 ym 10000 ym 10000 ym 10000 ym 10000 ym 10000 ym 10000 mae'r syfwng yma... byddai gydag i fancyddau yma o hwn y living i'r sgwrth. A fyddech chi wedi wneud ydych chi'n havnod, roeddwn wedi gynnol. Now, this is driven by Core Central Government Policy, so we want to move to 24-7 services, people work different hours now. It's not a case of if you can just go into an office or a job centre or something. We really want to make stuff self-service so people can use it when they want to use it, how they want to use it. Obviously the web is great for doing that. We want to provide a more streamlined and cohesive user experience. That's not only for our services But also when we can point you to services in other government departments that are relevant to you. We wanted to provide a greater capabilities to inspect and adapt our processes and the requirements and the needs of citizens in government. And fundamentally what else do we want to do? We want to save money for the taxpayer. We want to reduce our operational costs. So we had a great idea. Let's do agile! And that was literally it. That was the statement. We are now agile. So obviously you can imagine. There's a massive cultural shift there. It's not just as simple as saying you're now agile. As an organisation I can guarantee you there are people sitting in this room that have not been alive as long as we have outsourced IT development. You're talking 30 years plus of outsourcing. So you can imagine what happens when you say it to someone. You do an agile. You're now accountable for this stuff. You don't have a contract that you can point back to and a supply you can point back to. It's you. You also have a business which has been basically for 30 years educated. If you don't get it in the first release you ain't getting another one because it's too expensive. So there's some of the agile concepts. Minimum viable product. In DWP to a lot of people now that's maximum viable product. So it can cause quite a few problems, right? And then you've also got the other thing, right? Staff performance reviews. Everyone gets a performance review, right? But most of the time that's an individual performance review. It's a complete antithesis of agile working because it actually generates a behaviours that can incentivise the individual over the team. So what did we do about that? Well actually we've scrapped individual performance assessments. As of April this year we only do team-based assessments. The team succeeds or the team fails. There is no individual. How did we go about it? Well we came up with an organisational blueprint. And this is basically the thing that defines our patterns. Defines architectures, microservices, immutable infrastructures, container first strategies, sidecar patterns, all that sort of stuff. Include security patterns, encryption and transit, encryption at rest, different cyffersweets, the order of cyffersweets that we would use and which circumstances we would use them in. Different application patterns, whether we're doing asynchronous stuff, push-pull subscriptions, semantic version, infrastructures code, configurationers code, and then various development practices, life cycles, test driven development, behaviour driven development, pair programming, secure code and automation, all that sort of stuff we document in our architectural blueprint. One of the other things we do is create what we call a tech radar. And this is technologies that we're thinking about using, technologies we do use, technologies we kind of don't want to use anymore, and technologies we absolutely definitely want to get rid of. And you can kind of see it there in the radar chart. Stuff on the outside, stuff we're thinking about, we might look at as you get closer to the red dot in the middle, red dot, we definitely want rid of all of that. But the stuff on here, the life cycle for stuff on here is not linear. We might reject something in trial, C-Shop on Linux on this one from when I took this screenshot, for example, but actually we'd accept that now, because six months later the ecosystems moved on, the commitments from the vendors have moved on, so we'd revisit that decision and something that we've previously rejected, we may quite happily then bring back in. But we basically have one of these tech radars for pretty much everything, whether it's app steps, tools, languages, hosting, monitoring services, management, security tools, all of that will be on one of those somewhere. Now one of the things we thought about was, okay, how do we get rid of that 10,000 challenge? Continuous delivery. We'll just do continuous all the things, which is pretty awesome, right? We just choose a tool, we put it in, whether it's GitLab, CI, Jenkins, Go CD, something like that. Sorry, something like that. I was saving the mic drop for the end. So it's not a technology problem, right? You can put a tool in, you can do stuff with it, some of it will go well, some of it will not go so well, but then what do you end up with? A git big gap and hole in your systems, which if compromised basically gives right access to everything that that thing's got right access to. So for a really risk averse organisation, which we are, that's a bit of a problem. So you start looking at things, okay, so let's look at risks. Risk owners, we've brought them all back in house, they're naturally risk averse because suddenly they don't have a supplier they can blame. It's not mitigated through a contract, they have to fully own it, so they have to think about it properly, okay? But there are some things we can do around that to improve the situation, yeah? So infrastructure is code, awesome. You know how infrastructure is being built, it's repeatable, you've removed human error. Configuration is code, similar thing. Improves the transparency and repeatability. It's easy to statically test in advance. And then of course if we move to immutability, we get rid of a lot of complexity with things like patching and server drift and all that sort of stuff because you just destroy it, rebuild it, okay? But when you're doing that you need to think about chains of trust, okay? So this is all the way from pre-development, all the way through to production. So you need to be asking yourself, is it enough to say, my staff wrote this thing, is it enough to say, this module has 30 million downloads per week from NPM or Maven Central or whatever it is, is it enough to say, but it's not used in production? Because some languages, Node for example, the default action will be to install all your dev dependencies unless you tell it not to. People, so people write, so one of the points there we said was, is it enough to say my staff wrote it? That's a decision you have to make because you need to engender trust. So trust goes two ways, so developers are typically privileged users in an organisation, they get more rights to things than other people do, right access for example into production. So you need to protect the environment from them, doing something wrong either maliciously or accidentally, but you need to protect them too. So make sure your policies and intent, specifically the intent, are really explicit and you're really transparent about it. Think about your disciplinary policies. If someone releases a piece of code that has a vulnerability in and it causes you a major problem, are you going to blame that individual or are you going to blame the process that they went through to get to that point? Because you can pretty much guarantee that one person might have wrote that code, but that's not the failure in the chain that let it get all the way through to the point where it became a problem with you. So think about how you're going to deal with things like that. Devices, so everything starts with a device. Most of our developers are on Mac, we do have a few on Windows for various other reasons. None of our developers have local admin access on their device. And we got a lot, a lot of grief when we said we were going to do this and I do mean a lot. And we said, OK, we're doing it anyway because there are a number of risks that this mitigates, mostly that you can't take off all the monitoring we need to do. So you can have pseudo rules. Sudo rules give you exactly what you need, we just have to be explicit, we're making it transparent. 18 months later, we have five pseudo rules in the system. But for two years before that, everyone needed admin access. So this is something you might want to look at. There's a bunch of tools there we use to manage it. There's a few security ones missing because security through obscurity, I had to take them off. You can look on Gartner, there'll be no surprises in what they are. But yeah, have the reality that developers, if you put a restriction in place, they're going to work around it. They're going to work with them to come up with an answer that actually works for you. So you can achieve your objective without some backdoor processes that completely undermine your nice pink and fluffy world. So we would really say consider auditing and monitoring rather than a more traditional blocking and whitelisting of things. Connectivity, that was the second stage. So we do all of our connectivity. We basically replaced a lot of connectivity with Zscaler. For those of you that aren't familiar, you can think of it kind of like a VPN, but for outbound traffic only. So it's all kind of centrally controlled and there is no inbound connections. And that does, we basically have private access, which is the equivalent of an on-premise VPN, and we have internet filtering and stuff all built in through that. It's all software as a service and all runs in a cloud, and we just connect it to everything basically. Dependencies are always a good one. So open source is fantastic, right? There's loads of opportunities for it. Increases your transparency. You don't have to reinvent the wheel. You should have a faster time to market or your MVP. Consistent and reusable, but as with anything, there's challenges. Modules that quite innocently are doing something else and then someone compromises and suddenly they're stealing credentials or other interesting files. That happens fairly regularly in the NPM ecosystem. Other things, slightly less malicious, but that you mightn't be happy with. So modules that phone home with usage statistics and give details about the environment they're running in, you mightn't be happy with sharing that if it's passing out the patch levels of your systems and stuff like that. And then the other one is good one is CVEs and how do you do continuous vulnerability monitoring? Source code controls, so we use GitLab. We've talked about the people, we've talked about the devices, we've talked about dependencies, so we can now comfortably code stuff. GitLab, why did we choose GitLab? Well, Git is just kind of the thing. It's why, wouldn't you? Don't want to go back to the days of SVN or worse. You know, some of the old Microsoft stuff. Source safe, anyone? No, thanks. But yeah, so lots of people want distributed version control, right? But then they never ever use it, because they use GitHub or they use GitLab, which is the central point of control. Not really a big deal. The one thing I would say about that, and this came up in one of the sessions earlier, actually, where someone said, you know, we're not backing it up, we have no disaster recovery. Well, you do, because every single laptop that's called in that repo has a copy on it. So if you ever do find yourself in that situation, just remember, you've probably got a developer code somewhere, your job is just finding it. The real beauty of having a centralized system like GitLab, it's a single source of the truth, but you have to enforce it, right? If you're an organisation that's run in GitLab community edition, and you're offering that as a central service, great, make sure you keep in pace with what the developers do, because if you're not, you can guarantee they'll spin up their own instance of it. I know this because we have 17, and we're trying to pick that problem currently. You know, so obviously, whenever you can, script it so it's fully automated. But when you're doing that, remember, one size does not fit all. We have a lot of code. 55 million lines give or take. Lots of different languages. 50-something on that slide there. That JavaScript line, this is all scaled properly, by the way, that JavaScript line, that represents about 9.5% of our code base. Guess which one's bigger? Language you haven't probably used for 30, 40 years. So it's quite naive to think you're going to have one tool, a process, or method of going from A to B for all the things. So just accept you're not going to achieve that. If that's your objective, you've already failed. So start thinking, what's going to give you some value, what's going to give you your most value, what's your real pain points? Is your pain point connectivity? Do you have a classic walled garden approach where you have air gaps between your environments? Where actually, connectivity is probably the thing you need to crack to give you a lot of value to move to continuous delivery. And then from there, build on that and start looking at other things. The cloud, it's great. Other people's computers, I think we've heard that one far too often. But using the public cloud, it may shift your risk profile. It's conceptually, it's different to on-prem. You cannot control all the things as much as you might like to think, so you cannot. Depending on the provider you're using, you might have additional risks that you now have to consider. The US Patriot Act, for example. So most people will say, oh, that's OK, I'll just run it in the UK data centre. That doesn't necessarily help you. Go have a look at the US Cloud Act and have a conversation about your UK data centre. So there's lots of things that can trip you up there. Automation, we like automation. Automation is good. It's repeatable. You can automate lots of things. Testing, unit testing, static tests, dynamic tests, environment testing, using terraform, using vaults for credential management, that sort of thing. Deployment, we tend to favour Ansible, you may be a popup shop, you may use other things. Test that as well, use something like InSpec to test it all. When you're looking at your automation, start concentrating on what's the low-hanging fruit. Because there'll always be stuff you cannot automate. So trying to start solving that problem, again, you've already failed. There are things in accessibility tests you will never be able to automate. Why? Because you actually need a proper accessibility user to do it. There's some security tests you're never going to be able to automate because it's not what the tool is doing, it's what the person is doing. Think about things that can adapt and respond. Perspective, so look at things with different lenses. Don't just look at it as, I am a business and I need to achieve this. Look at it as, I am the Chinese agent and I really want to compromise your thing because then I get this technology for free and I don't have to invent it. I am one of your competitors and I just want to steal all your stuff and rebrand it and sell it as my own. So start, if you're not using them already, start thinking about using things like threat models. Threat model is great. Microsoft do an absolutely fantastic free threat modelling tool and it will, you can draw an architectural diagram and it will start generating threats for you, some of which you might not have considered before. A lot of them around the boundaries and system interactions that you may not think about necessarily. Transparency, obviously I keep saying this. Think about the things that you need to demonstrate, how you will do it and how you're going to prevent repudiation. Dashboards and information radios can be quite useful. We've seen some products earlier on that do this sort of stuff. There's plenty out there you might build your own. CD, continuous delivery, treat it like any other product. Don't expect you're going to go from kind of nothing to the Ferrari on day one because you're not. Use the information you've already got. If you've got threat models, you've got different perspectives or postponers, identify what your real MVP is, what's your real pain point where you're going to get that most value straight away so that you can implement that. Start getting that return on investment and more importantly start bringing the community of people who are probably terrified about the fact that the developer can write something and two minutes later it might be in production. You can start bringing them on that journey because they can have as many checks and balances as they want as long as you automate it. It's when you start putting manual process and you start getting human error. Obviously if you do this you can iterate it. You can become more stringent or robust as you go along. You might decide you need to loosen something up. It's perfectly fine. Many bosses and end bosses. Think about your hard controls. Many bosses, the stuff that you should be doing. Stuff you're seeing your developers going to give you grief and not doing. Ultimately stuff that you can go, meh, doesn't really matter this time round. I'll let you off the first time. Maybe the second time. The third time I'm going to crucify you. You've got some flexibility on it. End bosses. This is the stuff that for you is not negotiable. It's either done or you don't pass go. You don't get that $200. You need to be really clear about what they are. They may vary by team based on the maturity of the team. You may have much more stringent controls in for a junior team than you do for a more experienced one. Or the other way around. Remember if you're doing this people are emotional. Try and codify it. If you codify it, it's repeatable, transparent. But it also gets rid of the perception it's an individual attack. My senior developer really doesn't like me because every time I submit code it gets rejected. There's tools out there that can help with that. Danger is one we're looking at. But there's plenty of others out there. Continuous doesn't always mean it has to be automatic. There is nothing wrong with having a manual process. As long as what it's doing is automated you can have a manual process to kick it off. That's perfectly fine. You may need that separation of duties, for example. You may need it because you have a risk level that says actually this is our core business database and we want two or three pairs of eyes on anything on that before it goes through. Even though we've got this whole pipeline that does all those checks as well we still need that additional thing. It's perfectly fine. Make things transparent. It's an older pipeline of ours but you can see we've broken things down into parallel stages, various different things, building things various different compliance checks or commits formatted to the style we do. Have we suddenly included a GPL licence that we really didn't want to? Unit tests, code quality, security tests all that sort of stuff. The great thing about this is you can just keep building it up and building it up so you can build it up and roll it out to everyone at once. But you can see there you can have the flexibility to say we'd really like these things to pass but actually we'll let the pipeline keep going if they don't because it's a slightly softer control. Accessibility testing for example you do get false positives in automated testing of that. You do get some really hard positives as well. Then you can see right at the end we've got pushing AWS that's a manually triggered task. So we want someone different to go in and press that button but it's all fully automated, it's all part of that pipeline. When you do this though think about how you're going to evolve it. This page from about.gitlab.com is great for that because you can see their future roadmap and all the little things that are on it as on our roadmap are really good to look at and say is that something that's going to give you value? Is that something that you want to add into your pipeline or consider doing? Because it may be something you just haven't thought about before. Are you doing security network monitoring? Are you doing monitoring of container networks? Make sure the ingress and egress of them is correct and not just the entire platform. Have you got them boundaries right? This is a really good thing just to take a look at it and say is there anything on there that actually we need to start thinking about that because we haven't come up with that problem yet. So things like that, you can look at all the Gartner reports and stuff like that but I actually quite like the layout of this one. And that's pretty much it. So do we have any questions? Don't start your penguin. Honestly every talk I've sat next to this lady all day and you can tell when the speakers come at the end and she starts. You can go right now. We have a question. I was just wondering about the interactions you have with security and compliance folks around this stuff. So you talked a lot about building that stuff in but in terms of the kind of objections and making the case for why these processes need to be automated. So it really depends on the security folk to be honest. A lot of it can be quite challenging. You know there are a lot of processes that are processes and they're there because they're processes. When you start unpicking some of them and it's like yeah we can automate those bits we can't automate those bits. Those bits don't actually add any value. Why are we doing them? That bit in particular can be quite challenging because then you're talking about changing an ingrain process. You know don't get me wrong we've got some really switched on security guys who are completely up with all of this sort of stuff. We have some really switched on security guys that are not up with this at all and mediating that discussion can be quite a challenge sometimes. It's an entertaining one. You know when you start having debates about cipher suits and stuff like that and why we're using a weaker one before we use a stronger one. And actually a lot of the time it's because we look at it and say that's wrong. We want to use the stronger one because of this. And then the security guy comes along and says yes but you have to talk to this other thing and the thing you don't know about this other thing is it has this limitation on it which means you can't do that. We have a lot of old code, you can't just say we're going to run everything over TLS 1.3 and use the latest elliptical curve cipher suite because actually that will just block everything for certain things. So a lot of it's getting that understanding and being able to have that conversation but being able to have that conversation in a way that is not adversarial and it's actually looking at it from the point of view okay so the thing I'll always do when I was a security human and say I want to do this, this is what I've thought about what have I missed and then we'll start a conversation based on what I've missed and I tend to find that works better than going along and saying I want to do this how do I do it because actually they can see and then I've put some thought into it I've put some consideration around what it is that I think is a risk and then they can start adding the value based on their experience of the rest of the estate and all the bits I don't get to see and say okay but you need to consider this, this, this and this and a lot of that will depend on your environment you know if you're a new start-up you're probably always in the cloud and you're probably doing zero trust by default and all this sort of stuff if you've got kind of a more heritage estate or a more on-premise based estate or a classic walled garden where you've got all your controls on the outside and nothing inside so you know depending on where you are in your journey as an organisation and what you have behind you may impact some of those decisions Very good thank you two more questions I was just going to ask you showed a lot of languages up there obviously you've got a very large engineering team what was the experience like when you brought GitLab in possibly enforcing this on top of the engineers to say actually you've got to use GitLab and use the CI tools as opposed to giving them the freedom to pick the right tools for their own sort of engineering experience So there is some conflict with that so what we tend to find is if it's a really experienced engineer and we've hired in from outside they'll ask those sort of questions if it's a junior engineer or an apprentice engineer we've actually brought on and we're training up we tend not to get them to be at a bit of a GitLab CI versus Jenkins in particular you know we do have another team based in London that is purely running on concord CI for example and that's quite where the team is at a maturity level where actually they can articulate why it's the right solution for them then I'm quite happy to have that discussion when it's just I used this thing in the past I've learned something new that's a very different conversation because that's not really the sort of engineer I want So yeah so basically the Jenkins things they all pull from GitLab so they're all integrated with a concourse the concourse one is slightly different because a lot of that stuff is just pushed directly to GitHub because that's a very mature team so a lot of that's just tied directly into GitHub but everything is proxied on premise we use like Nexus repository manager through that because that's part of our chain of trust and digital signage and all that sort of stuff in terms of the asset but the code itself it's pretty much everything is through GitLab very good we only have like one minute quick question you mentioned you like you love automation, automate everything just curious if you've done any work in terms of building some of the some of the security tools into your deployment into your commits if there's any thought of integration in that area? Yeah so we do have some built in so if I went back to the pipeline slide on our security side we've got things like black duck we've got things like check marks you know we do run Clamia V we run claw, we run a whole bunch of tools and a whole bunch more on that particular diagram one of the things we do tend to find is sometimes the integration can be a right pain in the butt to be honest check marks is great but it's kind of fire and forget so unless you constantly poke in the developer and saying have you actually checked the outcome of that report and you have a check further down your pipeline to go back and say what was the status of that sonar cube scan for example because a lot of the security tools they run asynchronously so if you have a synchronous pipeline and you're running an asynchronous tool at some point you've got to get that signal back now that was a conversation I had with GitLab actually around how we can do this and we have got some example code repos of how we can use the multi pipeline feature to fire off an asynchronous tool and then fire a result back into a pipeline and then potentially make a decision on that further down in the pipeline I think we can probably share those that would be a good idea but yeah we do have quite a lot in there well thank you thank you very much