 There's a little bit about me. So my name's James DeLuce, I'm with Ernst & Young, so one of the big four accounting firms. I have an interesting perspective. I lead our America Certification and Compliance Services. So technology companies that, for instance, have ISO certificates, such as Google and Amazon and Rackspace and those types of organizations. I oversee all that work, so it's been interesting to try to adjust our control programs to these technology organizations. I had the pleasure of writing and I tinker a bit in a kind of recovering coder. So the way I've broken out this presentation is to be very specific on facts, on what we see in an audit side, to kind of share some of the audit progression that happens so that you kind of understand how you can insert yourself into the process and make your environments kind of maybe not conform to the audit, but maybe communicate it and weave together better than before. So with that being said, I will kind of talk about audit speak a little bit and so my attempt will be to share some of the phraseology that we use on the audit side so that as you're a participant and not a victim of that process, perhaps you'll be able to better represent what's happening in the organization. And so when we go through this, if at any point, if something is a bit vague or it seems I guess too legal, please just kind of stand up and wave and I'll try to dive in a little bit more specific. My intent was to provide a format kind of the present as to the industry, business management and the requirements, kind of speak about hypothetical non-existent situations that they may or may not apply. And then of course, having analogies and translations as possible within the audit side and then kind of progress through there a couple of through some key indicators so that when you're looking at your organization, how does how that organization could look better from an audit perspective and then how it could perform and continuously improve. You'll find my hope is that this topic, it does not really steer us to how to just get by but through the audit, but perhaps how we can evolve our programs to continuously evolve better in that side. And I'm a little biased and come from the security side and we have things like the ISO standards where we talk about continuous improvement and so I'm really gonna try to advocate that flow through here. So right out of the box, when you think about audits, you have to think of it as assurance and so it's very particular if you've ever tried to engage an audit organization like an EY or another big four or another county, you were very particular in how we present the information that we execute and what those reports are used for. And the reason is if you've kind of seen the news over the last few weeks or over the last 10 years, you've noticed big four accounting firms that issue opinions that don't always line up to reality, either go out of business or people kind of lose their homes and houses as a result because they are directly personally liable for those statements that they made about the organization. So assurance is where our businesses are trying to form partnerships with other organizations, right? And so if you are running a grocery chain and you're trying to do business, you won't have trust with your suppliers. If you are running a technology platform, you're trying to have trust in that the people that are providing you maybe the power and pipe at the physical layer are doing that at a certain level of competency and consistency so you can meet your customer's requirements. And so when you think about the trust between the parties, you have to think, what is everybody asking of each other? And so when it comes to audit, that's primarily what we're aiming to solve is if you are looking at the public markets, they are looking for a financial integrity. They're looking to have absolute confidence that the numbers that they're being presented and trading on at that moment are whole and accurate. And so that's why you submit things like SOCs and the SEC issuing guidance as to how that should be done. When you work with your partners to do business, you have to worry about the trust in the delivery of the materials, the consistency of the services. And to give an example, a very specific example, if you were working with a partner who's providing you just-in-time equipment and so maybe you are a cloud provider and you're receiving goods or you are a manufacturer, it really works in both scenarios. It's been in the manufacturing, but this client in particular had a supply chain risk where they were producing at a just-in-time level and they were asking for assurance from their vendors. And when they were asking for assurance, they wanted to know the operations would be consistent, that they had security controls in place and such things. And the reason they needed that is that prior to that date, they had a supplier who was attacked by a competitor who modified their shipping manifests so that our business received product and didn't receive product when it was supposed to and failed to meet our client deliveries. And so if you think about it, if you are making a car and you rely on brakes from a vendor, the attack hit the brake provider and the brake providers shipping manifests were modified by the attacker so they were not sending us enough brake pads to build their cars. And so it was a very interesting attack structure and it related to a much more sophisticated supply chain risk program. But when it comes to assurance, it is how do you trust the third parties and what are you representing in those reports? So you have to realize is that when you are a part of an audit, there are particular tasks that are being sought out in order to present that information to those third parties. So really think about it at that level and I try to, it's really OAuth for business, right? So how do I know I'm really handling how can I pass the tokens and how can I trust that that's the correct information and the information that I'm sharing at that time? So really think about the role of assurance and how it meets your business. The type of audit can vary. Like within an organization, you'll audit yourself, right? That's continuous improvement that will allow you to have integrity of the books. It allows for programs to mature. And also it allows for management to understand what's really happening, right? Because all the lights are on, does that mean it's all working, right? And you would say well just because the lights are on doesn't mean it's working, we have other ways of measuring that. And so as you go through the organization, you see that statutory regulatory items come into play. Scoping bounds is very important. So when you deal with financial reporting, you have to deal with the financial integrity of the systems, what's contributing to those systems in order to determine transactions. So it's very explicit, it's very specific hardware, specific systems, specific people. When you work on internal and external audits, third parties, when they're shipping manifests and product quality and all those things, those all come together. The recipient of the report is very important. Number one to the auditor, right? Because we have to conform to certain public requirements. And the reason I say it that way is when we put reports together, those reports have to match a quality standard and then we are audited nonstop by the quality boards to review our papers, our decisions, how they came to bear. And I actually included some of the observations that later on in this presentation to share with you to show the rigor and the adaptations, the adaptations that our programs take as a result of that feedback. Because it directly affects organizations as yourselves and ours and how we represent other organizations. So each type of audit has a unique concern and I say a right. And it's a right as in if you're representing the security or privacy of your organization is presented in that information. So really focus on the type of audits and how it fits your organization. When you think about the pieces coming together though, unfortunately or fortunately, depending on what side of the coin you're looking at, audit report trends, it's shifting. So when you work with organizations, we've done a couple surveys and we found that three fourths of organizations see an increased request of reports. So different types of reports, different integrities that are happening. And the bottom stat here is actually very scary. 45% of the existing audit reports do not satisfy the objective of the person requesting it. So think about that for a moment. You work very hard, you put together your control programs, you represent the evidence. Somebody comes in, puts their name on it, pushes the report out and 45% of the time it doesn't satisfy the objectives. That needs to shift as far as an operational change. And what's interesting is that there's another stat that we found that when we did the survey is that it's on average 40 hours per response time. For a per third party asking you for a question over, hey, can you represent how you have security? Or hey, can you represent that my data is safe or secure or integrity is there or I can trust this partnership? Average is 40 hours minimum. So that, and I say average minimum is, when you did the stats out, 40 hours is the lowest amount that on average we saw people having to reply to it. And so usually folks like yourself, people in the operations and in leadership positions that are responsible for building those systems. And so it's a big hit to the operation. We are seeing regulations and quality standards improve to catch up. As technology improves, we're seeing things come out more clearly. And again, we'll talk about a couple of those near the end. But as the standards are evolving to reflect, for instance, if you are familiar with ISO 27001, which is a security management program, they just released a new version in the end of September. Otherwise it was eight years and unchanged. So from 2005 to now the standard hadn't changed. And so we didn't have iPhones, we didn't have Facebook, we didn't have half the technology we have in this room. So think about the magnitude of shift that's happened. Supply chain has become much more important online applications when you weren't even called out in the old standards. So it's very interesting to think how we fit those together. And so why are the auditors asking for these things? Well, the birth of the audit requirements, I try to be a little coy here a little bit, but the idea is that regulations and other state risks and concerns and then they draw up control objectives. So when you are looking at your organization, there are control objectives and then procedures. Control objectives are thou shall do good. And then how are you going to do that? And then there's going to be sub parts behind that. The control objectives are very important because that is what you will use when you're trying to communicate to the business as to how your continuous improved and continuously deploying operations fit into this model. So when you're being audited, you need to think about it from that level. To be more specific, just Gene. Testing. One of the reasons why we're so excited about this presentation is that nothing freaks auditors more out than seeing developers do their own deploys. So in the absence of change manager and approval processes, separation of duties. And so James is a practicing auditor and we've asked him to actually establish from first principles not only why audit, but then go through risks to control objectives and controls and have to show how does an organization show that they have a control environment. They can prevent bad things from happening. They can't prevent, at least be able to detect and correct. And how do you evidence that so that you can show that to another auditor and have them, like James, be able to say, yeah, that's perfect. That's what I was asking for. So this is the reason for the first principles discussion. So thank you, James. Sorry to interrupt. No, no, it's good. It's actually allowing me for a technical difficulty, so. Oh, I see. Yeah, that's technical. No, it's good. Let's give this a moment, shall we? There we go. But yeah, good, clear, beautiful. Yeah, no? That's kind of interesting. Yeah, all right, that's good, beautiful. So separation of duties and development testing of operational environments. The control objectives here further elaborated out. Development tests and operations should be separated. What does it mean for you? And so I have to say this is for you to decide. And one thing we have to understand is that from an auditor's perspective, I'm auditing what management feels is absolutely required. And so the control objective that's designed here is unique. The procedures are put in place to document that control. So that is what the auditor is looking for. And so what I'm trying to show here is the linkage of how it impacts your organization. So when you sit there and you're being audited, they say, I need to see all the access logs for your control code. And I need to see that all the pushes are happening by people approved and change control happening every time. What they're really asking is that, help me understand this linkage? And how does it fit in your business? And so when you look at it from this level, you design the control objective. The procedures say the developers are not going to deploy code or they are going to deploy code, but they're going to do it in this manner. And then this is what the auditor needs in order to represent it. Because remember, when we're putting together these reports, we're trying to say, everything is working the way you say it's working. Think of it that way. So what do the auditors need? They need demonstrative evidence that it satisfies in a statistically relevant level. So think of it this way. When you have to show evidence, it needs to be something that you can preferably not print out the 300 pages of code, which has happened, which is a really fulfilling experience. Didn't work, but it was really fulfilling. Think about how you're representing that you are satisfying it and providing clarity to the operations. And it's a statistically relevant level. You have to understand is that when we decide on what evidence it's the population and the sample rate, if there are errors in the sample rate, then we have to ask for more evidence. And so it's a published standard. There's no magic behind it. We have to all follow the same rules. So when you're in an audit, they say, well, I need five pieces of evidence. Great. Five, what's the population you're working from? Work with them to understand and get that visibility. Because there's no secrets. And there's no hidden agendas during the audits. You can ask the questions the other way and say, what was your population? Why did you choose this sample? Because hopefully you'll come back and say, that doesn't make any sense. You're not even checking the control of the way it should be checked. And then the audit was like, fantastic. Help me test this correctly so there's no downwind problem. Because if you think about it, if we give them the wrong evidence, and it gets all the way to the end of the report, and then somebody in management says, that doesn't make any sense. I disagree with those findings. Then we start the cycle over again. And it's kind of like a debt spiral of pain. And we can get out of it just by being more clear and kind of go back to the communication. So I want to just kind of poke up. Yeah, please. Yeah. So James, in that last example with the separation of concerns, are you saying that as a client running a service, we are able to say, OK, out of so many hundreds or thousands of deploys to so many customers, that's your denominator for the statistical base. And here are the logs of what happened during those things where these procedures were followed. And here are the greens and here are the reds. Correct, yes. And we can provide that evidence to the auditor and say, so this is the demonstration that we are in compliance of the control objective? You're going to put me right in the box right away, aren't you? I like that. So any good person would say, well, it depends. And that's completely unhelpful. So the answer is, in certain scenarios, it's that easy. And so I would draw a couple and I like to say, well, it depends. But here are the variables that make it depend. It depends on the types of pushes we're speaking about. And I have one slide we'll get to. It'll be a nice review when we get there. But specifically, it says, usually I see people tear out the type of deploys and where you will come back and say, well, yeah, we do hundreds of thousands of deploys throughout a year. But 80% of them are minor tweaks. And so therefore, here's that population. And then 10% of this, 5% of this, 2% and 3%. And so the idea would you would present and say, the total population of what matters to this scope, for this control objective is 1,000, or it's 500 or it's 20. It may be very minor. And so the answer is, yes, you can represent, here's the total population, here's how we rolled it out. And here are the control procedures as it relates to that control objective to satisfy it. So I want to say yes. But I think there's some variables that have to be mindful of because I think everybody's organization might be slightly different. You want to help? So just to rephrase, you're saying, one is an argument of, is this a statistically relevant population? Yes. Right. That's what you're saying. And the second is, once you identify the population, then what is the statistically relevant sample size? Yeah. The sample size is actually a very specific formula that we're bound by. So that's much easier a question. It's the first, what is the population? And I would advise tearing out the populations so that you're not, for instance, looking at stuff that's just, oh, we just updated a piece of content or small push that was in production effective. Maybe this session should have been called ask an audit or anything. All right. That's between six and seven. All right. So let's talk about trains. I tried to, again, put down some analogies here. I've heard of anyone else that had any questions or were getting lost in jargon. I would say just pipe up and find the fastest way to get to the desired outcomes. Yeah. That is ideal. So jumping right in, again, I talked pretty fast because I wanted to try to give you guys as much content as possible. And since we're going to make the content available, it should be ideal. So key activities, coming down the tracks. So if you had to train, we have three different scenarios. We talked about the control objectives, the procedures, and the audit activities that result. But the risk assessments and the threats and vulnerabilities are very important. So think about your organization and how you quantify risk. What matters for you on a risk basis? And so the risk assessments help design the security program and the compliance program together. And risk assessments are very important. So a lot of organizations have a much larger risk program, and then they have sub-risk programs to evaluate kind of tactical areas. And it's important, I'll draw this for two points. One, it's very leading practice, and it's very in tune to not over-auditing and not over-putting safeguards in place that are unnecessary. But secondly, it actually aligns up to some recent audit guidance that just came out that speaks directly to auditors practicing this approach with you. And so that way, you're not being, I have to say, kind of getting audit fatigue because people are asking questions that are unnecessary. So when the train is coming down the tracks, meaning you've got a bit of runway, it's not audit season, it's not busy season, you guys can reach out to the internal audit department. You can reach out at the security group. You can work with management. Identify what are the risks that the business is trying to address at the program and the audit level. You have to understand that the audit plans are built a year in advance. So we're talking about the ploys, how fast it is and how much better of its smaller chunks when we go faster and its lower cost is more efficient. Audit programs have to be built a year in advance. So when they're already having it planned out based on last year's risk and they're coming to see you. So that's already too late. So you really want to plan this a year in advance of really trying to reach out to them, understand what the control objectives are and what risks do they assign? Because again, business assigns what are the control objectives to the procedures? So if you can become part of that process, the procedures that they're asking you to execute against are in line with your own business and your own responsive development structures. So really think of it that way. So if the train's already left the station, now we have to worry about things that are approaching us at this moment. And so I like to say, I wrote up here as a team, seek out the regulatory and common control objectives that we're managing against. And so, I think the last slide had it maybe. Yeah, so a common control objectives would be if you have PCI, HIPAA, FISMA, FedRAMP and all these other access control, segregation of environments, there's some consistency. And so usually your departments will have them pre-built. And so you don't need to try to manage against seven different standards internally on the development side. You should be able to reach out to those teams and say, what is the consolidated requirement you need me to achieve? And the scope and balance. And now what you're doing with one, let them come bring to bear the legal and the audit experience to say, here's the seven we need to do because we're in these seven markets and strategically we're gonna go in this and expand globally. So all those factors should already be considered by this point. So you should be able to reach out to them and say, I need a consolidated scope and balance of what we care about so I can make sure our deploy environments fit that structure. So it's really careful to think about that. And then when you started thinking about, I took about here procedures on already committed and developed the translation of those procedures to your environment, meaning that if we've already, we've already baked the procedures and we will represent our control environment by these 17 things. Work with internally to draw up how you're gonna reflect those and have management sign them off. What you've already done by doing that is now you have visibility across. And so when there is an audit, they'll say, well, how do we know that this was okay when we come back and we come with these findings? Well, management review will say we reviewed that. We already reviewed that. We understand that that was a variance and we've already addressed that. And so that's a very positive effect for the business. So can you get some examples of what those 17 control objectives might sound like? So that would access control, management signed off, things are finalized, things are published, control objectives. If you were in the government space with 853A, please. Oh, it was my five minute warning. So we're gonna talk faster if that's even possible. So I'll come back to break on that question. So finally, during the audit, it's now possible to change the audit control procedures. And it's very important that you focus on representing as carefully as you can without creating a negative event during an audit. Now, that doesn't mean come up with fake evidence. That means just be very specific as to what you're responding to, and no more, no less. But the idea here is the audit, you want to be a success with a high quality and just avoid any negative impacts. Negative impacts would be misrepresenting the control environment that then requires suddenly an SSC filing that we can't issue or an audit report, because the control environment wasn't correctly audited. We don't, that's a negative event. That's a lot of noise, we don't need more. Even if it's a great environment, it doesn't change. It's still a negative event. So really focus on that. I'm gonna jump to the challenging environments. So factor fiction, utilize the audit as part of your continuous improvement. I always say you should require the audit to come back to you with feedback as to what could be improved on operational improvements on the control procedures. After the audit, completely valid to request that. And it's very much in line with kind of the risk assessment approach that's expected. I'm actually gonna probably do a slide 30 seconds and then we can have a Q and A and we can go back. Change management, preventive versus detective controls, tear out your changes. So based on what? Based on your organization. So if it's only, if we're doing dev pushes and it's real minor, it's not gonna affect certain broader systems. That's one category. Maybe it doesn't require any sign-offs. Maybe there's very limited automated testing. As you tear up into the larger areas, I had a privilege of speaking with a lot of individuals in the last 24 hours about how they roll this tech out and everybody has automated testing and automated checks within the system. Even though they're pushing it directly to production, there's almost this magic middle system that's handling that negotiation and the risk and the canary scenario. So it's important to communicate that while dev is pushing to production, there is this intelligence in the middle that you've built and you have to take credit for that because otherwise people just think, so you're literally just pushing into production? No, obviously not. No, I would never do that. So you have to help with that concept. So change management, visibility and then how would you change an organization from two week change to two hour change has to do with that automated integration. Separation of duties, it is possible to have both worlds where you have separation and you do high throughput. We had a great example this morning with Netflix where they talked about multi-level counts and monitoring or you need a wicked good application in the middle to act as the promoter, which is what I was just referring to moments ago. I always highlight that a lot of the standards don't require separation of duties. So just be, again, get in front of the train. Why are you asking for separation duties? Rationalize that and close that out. All right, so just two more to go. I know we got the clock ticking in the back. PCA updated on 1024 as relates to financial reporting systems. I put this in here for your review later on. Risk assessments and the monitoring controls and the determination of the control procedures is what they actually advised that based on reviews of public accounting audits, this is what should change in our audits. How do they expect that? They want to try to make sure that we can test across a location and extrapolate that they're designed and implemented consistently across all the locations. So think about your systems and this statement. Electronically, how you're deploying it across large dispersed environments, this statement applies directly to you and this is how you translate these high deployment environments and these cloud and abstracted systems using the public labs and the other scenarios and technology we have in place. So really keep an eye on that. And then finally, what is a clean, audible world? Like what would you do to help the auditor and help yourself? Clarity, a lot of times, some of the stuff just doesn't make sense, right? Sometimes you are working at a light speed that some people just don't quite grasp and they want to walk in some place and see a wall of servers, right? Don't walk them into a room of wall servers and then say, oh, we also have 1,000 someplace else in the cloud because that's confusing. Give clarity, 100% awareness. An auditor is trying to gain comfort that you are doing exactly what you need to do for that time period, right? And so if you can confidently say, well, here are all the employees we do, here's all the people involved, here's how we monitor it, we have very little preventive controls except for this amazing intelligence in the middle and this is how we have confidence it's all working based on this trending. That is what they need. They need awareness that you can bring those pieces together and help line up the linkages to the risks in your own program. Multiple competencies are usually involved, reproducible, so that if you, this year, present something and the next year, present something different, last year's results are looked at, right? You review the stuff every year, so we have to review it too. So if something has changed year over year, that's a really big problem, especially if the facts go like this, okay? So be very clear, reproducible on how you put things together and then comparable and I put integrity and validity, nobody's challenging the evidence as it has integrity to it, but sometimes it doesn't measure up to what you're doing totally and so that's sometimes either communication with the audit program to the findings or just how the evidence was presented in the scope and balance. So that's it at a very rapid pace. I'll do questions afterwards, I know we're out of time, but thank you very much for your time.