 It's 1.15 the rest told me this was the best speaking spot of the conference because I said Man, I don't know if anybody's gonna come after lunch on a Sunday at 1, you know 1.15 the rest said no No, that's the best spot of the conference, right? So I think he was building my confidence, but We'll take it. I'll take it. Let's get everybody up because we got to clear just a little bit of space in our In our mind to cover this kind of heady topic agile practices proven in high assurance environments Highly regulated. Okay, so everybody stand up Okay, and do whatever you do whatever you do to stretch a little bit and just feel like you got some Room to get something in your in your mind. We can fit a little bit more in Listen to this guy for an hour and then Scott comes back on so Got somebody who's somebody who's good up here. All right, so clear just a little bit of space in your mind Okay Yep, we're ready so I Asked you to do that because I'm I'm really not the expert in this area, right? This is an expert talk they say was on the expert track But you all are the experts. You're the experts in your particular field not me. Okay, so as we go through this today I'm gonna ask that you guys share your experiences and when there's questions if you can answer it, you know as per your Particular industry or environment feel free to speak up. Okay, you guys are the experts I've come to India and learned a lot in the last four days. This is my first time here. I've been you know Wanting to visit India for about 15 years since I I started my career. Yeah, I look a little bit young So I've been wanting to come to India and I've got to experience a lot Like I said in the last two days of the conference three days of being here Now is my chance to share a little bit of the work that I've been doing not in isolation but with customers and practitioners and consultants in the industry and Now I can share that with you guys and help you guys Build a community around agile practices in highly regulated environments if you haven't already started building that community here, right? All right, well, let's start with just the definition of high assurance so you can decide right, you know after this if you're in the right room or Potentially you want to get up and see another talk after this right because I'm not going to get to my agenda until We're 15 minutes in and you're already settled in okay So high assurance high assurance software systems are unique because they must satisfy basic functional service Properties that the system intends to deliver okay, as well as guarantee desirable system properties such as security safety Timeliness and reliability Some of the industries that are covered here and I did a little bit of a scan of the crowd to see We kind of which companies that you guys were working for but some of the industries that are often talked about when we talked talk about high assurance are You know military and defense command and control systems aerospace Could be automated manufacturing could be be automated banking systems that have high risk Okay, we're also look at things like the development of nuclear power plants And then the example that I'm going to you know talk a lot about today is The health care industry specifically medical devices Okay, so I won't you know talk about a broad spectrum of industries and I actually Just give you some examples of regulating bodies that you guys have probably heard about whether you develop health care products or Or whether you just use them in your daily lives But some examples of regular regulating bodies include the FDA to the US Food and Drug Administration ISO standards European Union or Med Dev There's one here in India the drug controller general of India that you guys Are probably familiar with but to my understanding that's fairly recent as of 2005 and Is just building their Standards organization if you will or their regulatory organization Okay, we also have health Canada, and then of course ourselves we can be harder on ourselves right then Then some regular regulatory agencies, so we'll talk a little bit about that now. There's also Organizations like the global harmonization task force that release guidance documents that combine a lot of the different regulations across different regions of the world and Start to standardize that as well as the IEC So although the IEC is not a regulation. It's recognized as a good standard when developing medical instrumentation So those are a couple of the examples of regulating bodies that we might have if we were to look at this medical device example Okay, so I'm gonna ask everybody to stand up one more time. Okay. I know it's terrible, isn't it? Last day of the conference and this guy's got a standing up all the time Okay, so by the definition that I gave of high assurance I want you to stay standing if you work either in your daily lives or Within a company that provides products or services that are delivered in high assurance environments So stay standing. Wow everyone All right pretty close now. I'm not encouraging anybody to sit down stay standing Okay, now if you're already using And I'm just gonna use the term agile practices, right? And this isn't because it's an agile conference now you could call it more advanced software practices You could call it new software development or lean startup practices or lean practices So if you're using those things in the development of this high assurance product stay standing Okay, so some a lot of you are already sort of practicing right okay now if you've read your regulatory guiding documents For your specific industry whether those are internal whether they're an FDA document or an ISO document Or something else for your industry stay standing if you've read those in the last let's say if you've read them in the last two years We'll be pretty lenient All right We got we got a couple yep if you've read them in the last year if you've just taken a look at them and maybe it's your annual New Year's resolution that you're gonna reread those docs right okay, so we got a couple people standing Thank you, and these are yeah These documents right here That'll be part of what I'm asking you to do today. You saw a lot of people sat down, right? so one of my asks of you is going to be that you go back in and Next week or the week after make a little bit of time and actually understand your regulation Doesn't matter what your role on the software team is You can't challenge status quo and you You can't deliver better products if you don't understand what you're actually complying with so take a minute and go back and read those things I would set some time aside so you can get a nap in if you need to or Sleep actually that a lot of them that I've read aren't actually that bad. They're pretty readable Okay Now would you be surprised to see this is in with regard to the FDA and medical devices Although the waterfall model is a useful tool for introducing design controls It's usefulness in practice is limited for more complex devices a concurrent engineering model is more representative of the design processes in use in the industry And that's from an FDA FDA guidance document that that dates back to 1997 That's why I say kind of go back and reread these things reread them with a different lens right so It's before the signing of the agile manifesto even right and it's saying you know Although the waterfall model is useful You know there are other methods out there and you're gonna see that if you look at the guidance and the regulations When I went back and read these things I would find a number of places at least where you can draw statements out like this right now You have to go back and read it in context to make sure that that's Applicable but things like it is important to note that neither this document nor CFR 8 20 30 itself constrains development To single-pass stage gated waterfall activities or specifically in IEC 62 3 304 where it states The medical device standard states these activities and tasks may overlap and interact and may be performed Iteratively or recursively. It's not the intent to imply that a waterfall model should be used Okay Like 62 304 Yep Okay So I'm not here to bash the waterfall model and likes I'm Scott mentioned this morning there You know it has its place and and its purpose what I'm here to to challenge you is that? We're not limited or restricted to the waterfall model because we're in High assurance or a regulated environment. Okay. I'm here to help dispel the math the industry myth That's kind of perpetuated by our own past right that we can challenge these things You know if we want to and we think that there's a better way We can actually challenge the the status quo oftentimes. It's not our regulators who are Mandating this on us and if we work closely with our regulating bodies We can figure out ways to be in harmony and develop better software Practices if you will agile being one of those That we can use within our own organization to help with things like quality safety security Also to help with innovation and speeding Our products to market as well. There's no reason we can't take advantage of those things within our own industries Now that said I'm gonna ask two things from you today and I've already asked both of them. So one is going to be that you participate you guys are the experts So if there's questions, I'm Not going to be shy to ask if anybody else would like to to field that or is run into it in your industry one of the things that I've done is to help build a little bit of a community where Healthcare organizations could actually get on the phone with each other sit at a round table not talk about their specific Their specific company and practices, but just talk about best practices for software development within whatever guidelines and regulations So you guys start to build that community yourselves here as well and the second thing would be of course go back and understand your regulations if you haven't already Okay, so if I didn't meet you I tried to get around and meet as many people as I could before the room filled in but I'm Craig Lungenfeld and I've worked with rally software now for a little over five years I've been lucky enough that my background is always included either agile and lean or software in it Okay, so there's some lean stuff within manufacturing before I went and and started as a software engineer and worked heavily within the software development lifecycle and really within regulated environments I worked with Monsanto, which is a global biotech company and at Monsanto I helped to build the document management systems that actually controlled a lot of our regulating Documents across across the world for different regulating agencies Now I've been lucky enough to work with a lot of great people in my days and on this specific project I've worked closely with Dean Luffingwell and many of you may know Dean as an author a Methodologist an executive he most recently came out with his book Just about a year ago. I think agile software requirements Okay, so Dean and I teamed together not just with ourselves but with several others and We did come out with a writing or a white paper if you will about using agile practices in High assurance and highly regulated environments Okay, and we didn't do this in isolation at agile 2010 We brought together several of the companies that were coming to me and asking hey I know you work with these other companies in our same industry How are they doing verification and in validation? What are they doing now that they're building user stories? What are they doing with their prds and SRS is how is that actually working? Okay? I said I don't know the answer to that right. I know what this company is doing and that company is doing But I don't know how you come together So what Dean and I did was to build a community where we could start to share those ideas and come up with some suggested ways Okay, now you'll have to apply that to your own specific environment If I go fast enough, I'll get it Let you guys have an example. I'm from a health care company. That's actually applying some of those practices so Everybody's got their waterfall story, right? so everybody here is practicing agile remembers the days of waterfall, right and Mine's gonna I guess date back to when I initially started as a software Engineer working for a large insurance company largest home and auto insurer in the u.s. Okay, and of course, this is the proverbial my friend, right? so this is my friend's story really my friend was working on a software project where They the application was was gonna send out the billing statements to you know a specific state for a specific line of business, right and so they were updating this application and Like many of the projects at that You know specific organization. It was it was late and it was over budget and what did we do? We had and we followed PMI processes, you know to the T straight out of the book And so what did we do? We rushed into testing, right? And we act we they so they rushed into testing, right? And what happened was they were able to get it through all these stage gates and into production very quickly Which oftentimes didn't happen, but in this case it did So got into production and when the application ran for the first time in production all the English-speaking policy holders that preferred English got their bills in Spanish all the Spanish-speaking policy holders got their bills in English and We all laughed about it, right? He got a little bit of a talking to in that project, of course Who had sent out all these paper bills? Got a little bit of a hand slapping But the point being is that I think if they were practicing Agile act in fact, you know, you can be pretty sure that a Defect that is that big would have never made it out of an iteration yet alone into a production life cycle and for the products that we build in the medical device space and in others where aircrafts rely on our software to stay in the air and nations require our software for defense and Humans actually go into medical devices or rely on our tests from the software that those medical devices produce Those things I got to think at a minimum if you were to make a mistake like that You'd lose your job. Okay in a worst case you might actually injure somebody Or cause some other high risk That's why I feel like Agile is important or better software practices Other than the traditional ways that we we worked like the example. I just gave Okay, so where are we going with this? That's part of the show too So where are we going with this? I want to just give you guys a little bit more fodder Many of you are already practicing Agile and high assurance So you might run into these things within your organization people asking questions people asking if other companies are actually doing this other Organizations are actually practicing practicing this so I'm going to tell you You know how Agile is proven within high assurance using that health care example and then I want to talk about four key points Which is you know a proposed Agile framework and this isn't something that's prescriptive You know this is just a suggested way that I've seen some companies work The Agile high assurance requirements model again They say framework and model right says working with with Dean and and that's what he does frameworks and models and stuff Right, but these are proposed ways that you could work and they answer questions or help to start the conversation around What do we do with PRDs and SRS's? Then we have when do we do artifact generation right? We have verification and validation artifacts When do we do those things and lastly? An updated QMS system. How's our QMS system intertwined with this and what do we actually? Need to do in terms of updates So that's where we're gone And then at the end or close to the end I'd like to have a guest speaker up here to talk about their implementation Of Agile within a health care environment Okay, so just some case studies the first one I could find and the first thing you do when you start an effort like this Right you go out and Google Agile and FDA and you go out and Google Agile high assurance Agile regulated environments All these things and the case study that kept coming up was one that dates back to I think the project or the case study was actually like 2004 when Abbott laboratories compared a traditional project to an Agile project and they actually presented these facts at Agile 2009 But they came up with 20 to 30 percent fewer defects found right we've all heard things like that But in conclusion and you have to go out and get the case study yourself and read it But in conclusion they say this experience has convinced us that an Agile approach is The approach best suited for development of FDA regulated Devices, okay, so that was was kind of the first one. I could kind of find a more recent example Would be GE healthcare and a Dr. Dobbs article GE healthcare goes Agile that was submitted Or published I guess in 2010 Okay, and in that in conclusion of that and they give a lot of examples within this publication of Things to watch out for and things that maybe didn't quite fit quite right So it's a good article because I think it's just really transparent but even in conclusion Andy Deitch and just to give you some context they I was familiar with this because Worked with GE healthcare a bit. It's probably some GE healthcare folks in the rooms I have a large contingency in India and Israel In Europe and then as well in the US. So this was a global Agile initiative within their imaging solutions group. It's it's since went broader, but Andy Deitch their VP of engineering says we're making progress and and feel the benefits that our agile adoption have Been worth the effort because of this we're rolling out agile Globally within GE healthcare. Okay, so it's not a real bold statement But if you actually read you'll see some evidence of you know Some of the struggles Andy went through and and they believed that was a better way Even within a regulated environment to deliver software Okay, and then you could use this example from the US Department of Defense right who held their first ever Agile conference. So it's hit the government, but You could if you guys were here this morning Evan Lieborn if I have his name right We spoke about how the Australian government and using agile within the Australian government as well So Evan's here at the conference and actually had a chance to work with agile within a very Prince to environment within the Australian government and An interesting fact that he noted was that agile project actually won an award over the Prince to projects, right? And then there's white papers and references so a very good reference that's going to come out that I Read which I think is going to be the final draft Mike Russell's in the back of the room And has been heavily involved in this is the association for advancement of medical instrumentation and they're gonna Be publishing soon. I think in the next month or two a technical information report Okay, and it's entitled guidance on the use of agile practices and the development of medical device software there was contributors from all sorts of health care companies as well as FDA auditors and So forth on this kind of 90 page reference and it does a really good job of blending what agile's principles are with With what the health care regulations are okay, so that will be coming out soon I'd write that down and check out their website if you're not a member that would be a great Reference to get and they make a statement as well since agile is a highly incremental and evolutionary approach It can therefore be mistakenly assumed that agile is incompatible with the expectations of the medical device software process And they draw out several examples as As I mentioned so I would look at that And then we have any number of blogs that are popping up and Scott yours is one of the first ones that I saw And you'll you know I'm squeezed in in between Scott Amler today And so he has the his agile scaling model talks a little bit about Regulatory are using agile and regulatory environments Tom Grant from Forrester did a whole series on it last year He gets a chance to talk to a number of companies as well as Dean laughing while who I mentioned already if you go out to his scaling software agility blog You'll see a whole series of about 30 posts on agile in regulated environments okay So is that proof I guess some some case studies some places that you guys can go it is being used right and We can we can say that we've crossed the chasm or at agile's hit mainstream and oftentimes We'll say that when it actually hits these more complex environments like that that it's actually being used on a broader scale Why is that why in the last couple of years? Have we really seen a lot of people try to tackle this and and seeing all these reference materials and we see big companies that are creating these complex and High-risk devices using agile software for the same reasons That everyone wants to look at better ways to deliver software, right? And when we look at agile it provides just a better working environment in a more collaborative culture better morale, right? Productivity time to market is huge right now in this environment. So people want to take advantage of those Benefits that allow them to get to market quicker and time to market now The main one is quality, right? We've all experienced Hopefully better quality in our products. Maybe some of you haven't but there are many principles within agile that just Inherently provide us with better quality, right? So this is an example, you know agile does drive quality safety and efficacy and some of the things like setting up a CI environment Looking at automated testing The user story itself, which I'll go over a little bit deeper into the presentation And so let's look at these kind of the four points that I mentioned earlier Agile high assurance lifecycle framework if you will so why do we need anything different than what we already have? Okay, doesn't agile already you know, isn't it? Don't we just do it within our engineering teams? Why do we need to work differently because I hear this right with some engineering teams that are kind of working on Under the radar, why do we need to do anything different? They can do their regulatory stuff and we'll do our agile stuff, right? But it is different, right? It has additional requirements when you're when you're within a regulated environment Okay, those and this comes from an FDA guidance and who wouldn't think it's applying implying waterfall, right with a graphic like this Okay, but you'll see those two words verification and validation that came up all the time for me and Verification, you know when put simply you built it right and validate validation is you built the right thing, right? Verification provides objective evidence that design outputs of a particular phase of the software development Lifecycle meet all the specified requirements of that phase Okay, verification design outputs meet the requirements of that phase validation confirmation that our product meets the intended use of the system, right? Includes evidence that all software requirements have been implemented correctly completely and are traceable to the system requirements Okay, I apologize for reading that. That's not something that I'm gonna memorize for sure but that's verification and validation and One of the questions that I often got is how does that play in and in addition to that? We have things that regulations Either strongly suggest us to create or just straight out mandate You have to have a systems systems requirements, right? You have to have a traceability if you want to say matrix, but Really we have to provide requirements specifications and we have to make sure that Those are traceable And there's different ways to do that. I'll give you guys an example Let's first look at kind of just the what you guys have probably seen is the agile framework or the scrum framework Okay, what does that have in it that actually? Works very well and this is teams can do this pretty well when they get started if they're kind of if they're doing it under the radar or not They actually can comply with a lot of regulatory things just by doing an iterations It fits very well with things like verification. Okay, what do we have on an iteration that? Makes us good. Well, we have an iteration backlog, which is our inputs in that iteration backlog We have these things called user stories User stories if we're doing this correctly, right have a very strict definition of done They also have acceptance criteria hanging off them. Okay, so the acceptance criteria is going to become the verify piece of this Right, the definition of done is going to include those items that we need To actually exit or get the output from the sprint that can include things like, you know updating Regulatory artifacts if we need to when we enter the sprint we're in this constant define build and verify cycle Okay, and every day we're updating our user stories We're running our actual test cases and those things become very traceable items if we use automated systems You can sometimes set up a sophisticated system that will allow you to automate this stuff, right from code or being able to trace down to your Code being able to trace to your test cases and so forth the fine build verify. Okay at the end We're not only going to deliver output, but we're also I'm going to add an additional step, which is You know the the demo at the end, so you're going to let Allow your business stakeholders or actual proxies for your product to view what you've actually built So you can get eyes on it early Let's scale that to an agile Project life cycle so like I said the verification iteration was something that I often seen teams be able to implement and be pretty Successful with it. It was when did we do validation though in the validation activities because we can't keep up with all the Validation stuff during our daily sprint especially if we're just starting as an agile team and a lot of those things are manual, right So when do we actually do these things and don't we need to do it every sprint because a sprint is a phase, right? An iteration is a phase, okay, and what what I found is the best way is actually You are going to have to accept that if you're kicking off a new project in an environment like this You're gonna have some additional planning some additional analysis You might have to do some work on your quality management system You might have to set up some additional architecture. Okay, you might do requirements definitions like create PRDs and Break those into SRS is up front, right? We're gonna accept that. We're not gonna say we're just gonna get started Then we would move into verification iterations and that's the iteration I just explained in that iteration framework where it allows us to be very traceable but at the end what I found is a lot of the Organizations that I worked with at the end they would verify verify verify So we did three iterations was highly traceable and at the end we'd have some big lump of scope Right all these artifacts that needed to be updated that didn't blend any of their QMS really with their actual agile teams and The complaint I heard was yeah, we're very predictable during our verification iterations are our regular sprints, right? But when we get to the end we have no idea how long it's gonna take us to get this thing validated So we can get some system increment into an environment that you know the FDA is blessed if you will and What I saw is those organizations come back and then try to address that particular issue with validation sprints and they might have any number, you know, I've Seen three and four validation sprints So it still takes two months at the end to go through validation But by chopping it up into sprints We retrospect every two weeks and we decide if the things that we are doing are valuable Okay, and we try to pull that forward so we become predictable if we know it takes four validation sprints Maybe we try to get it down to three and try to get it down to two So we don't have this huge pile of scope, and we're unpredictable at the end and Throughout this life cycle then we're working on continuous verification validation Activities this has to be driven in tandem with our quality management system, right? We can't do this separate from our quality management system And when we scale this Okay, if you're familiar with Dean his big picture Model is one that he often used to explain how to scale to a complex organization And I know that in many of your environments you're creating products that take teams of teams and programs and Entire portfolios entire organizations to build right so he uses this model to explain it And it might be hard to see from the back But some of the things that are important when we looked at this in the context of high assurances Although you have architectural governance that sits above The the program maybe at the portfolio organizational level you also need your compliance activities Okay, or you're regulating bodies you need those to work in an agile life cycle as well so you need they need to be educated on what the teams are doing and The updating their quality management system they need to do that in tandem now those people It's got to be a cross-functional team where you have team members that are actually working within that system working with the compliance team and Vice-versa those people have to funnel down into the actual teams similar to how an architect would work and you also have at the At the release level or program level you have prds right so the big scary thing for a lot of organizations was Yeah, we do agile within these little increments But what do we do because we then go outside of agile and create a prd and srs document, okay? So what we you know suggested was hey you're still gonna create your product requirements document You just might not do it to the same level of detail You're gonna do it to the detail that's needed to kick off this particular phase of the project And then the teams will work on the srs is which will eventually be broken our system requirements Which will be broken into the user stories that the teams will work on in the sprints And you'll have a system increment, which is a validation increment Okay, now the input so going from that big picture model back to kind of the input of an individual iteration I talked about this. It's still a user story at least in our context Others might others might use different artifacts But a user story at a minimum is just a story card, right? Which is pretty scary especially in a regulated environment, but it offers two pretty valuable properties Which are the definition of done and acceptance criteria, okay? and So those two acceptance criteria in the definition of done can help drive that input And you could also add other things to a user story if you needed to for your environment. Yep question You're gonna yep, you're gonna have a definition of done And you're gonna revise that definition of done for each sprint, right? And all the stories have to Meet that definition of done criteria It's for the sprint that each user so that that graphic is probably Yep Okay, so the question was is the definition of done specific to each user story or specific to a sprint and definition of done Typically is specific to a sprint, right? And then you're gonna iterate on that definition of done that will include those regulatory things Okay, now user stories really good because it like I said, it's got good traceability And I mentioned this before but a user story can trace via the task down to the code level and even down to the unit Test level if you want and a user story can trace to the acceptance test So it creates a lot of traceability within the iteration that can then scale up beyond iterations into validation Okay, we don't want to let those you know one of like I said one of the things that I often heard was Hey, we're using user stories on the agile teams and then when we get to regulatory or validation claims We're going back to our traditional prds and SRS's and we're doing a lot of manual manipulation Okay, well in an advanced environment Hopefully that your user stories because the user story is good. It's not what you planned to build six months ago It's not a requirement that says hey We plan to build this six month ago user stories actually what you built, right? It's it is the detail of what went into what you built And if you can get that to then trace back up to your system requirement which traced up to your Product requirement you can build those documents then or even have those documents automated Based on what you actually built okay, and it will fill that definition in as you go Okay, so if you don't want to change the labels and the product requirements doc might be called a marketing document Or a system intent don't feel like you have to shock your organization and change those things You might just start to change the way that you produce them And then if you go out to the scaling software agility blog you can see Some kind of more detailed Requirements models out there. I won't go into those in specific detail Now the quality management system, and I'm certainly no quality management system expert this comes from a friend of mine at Omnix, which is a Joint venture with GE and the University of Pittsburgh And what he said in terms of consideration for your quality management system I saw this across the board right you have to involve the folks that are actually creating and maintaining your quality management system in your Agilit initiative, okay, and We have you have to look is this is this something that we can iterate on or is it a rewrite? Is it an opportunity for us to rewrite our quality management system? Okay, you have to establish that quality management system cross-functional team Okay, so you have to have people that are actually Working within the framework of that system sitting on the team and people sitting on the call the on the team and people sitting on the call the Compliance team that are creating that system sitting down with the actual teams that are using it Treat it like a product run releases and sprints on your QMS system I'm just like you would any other product and iterate on it improve on it make it better Okay, and then my last point was a Lot of questions about what actually happens in the validation sprint Okay, we got we got the picture on the verification sprint and you're kind of asking us to move some things out of that So we don't weigh ourselves down with those two week iterations But when do we do the validation? activities that that we're used to doing and that we need to do right to comply and So I know you can't read through each one of these but moving those activities out into the validation sprint to anything from Updating traceability matrixes and like I said automation is your friend when you get into very fast-moving environment like this But updating traceability matrixes like PRD to SRS and product requirements doc to feature tests Down to the code and so forth any documentation that you need to update so You have to allow yourself time to actually if you need to manually generate that and to Have sign-offs and get it into the appropriate system. That's what the validation Sprint is there for as you The first time that you go through this like I said you may have several validation sprints until you can work on Understanding and work with your auditors on understanding. What is actually required and you can start to get the right set of Documents versus just giving them everything that you have in the past okay, so I'm seeing some yawns out there. So I'm gonna invite my guest speaker up and Change it up a little bit that Matt Anderson who has contributed a lot to this conference And so I appreciate him being willing to just step up at the last minute I got a mic up here for you Matt. I appreciate him being able to step up and Give you guys an example of how Cernar health care was able to implement agile within a highly regulated environment Yeah, and I'll say that looking at the slides. There's some things we do differently All right, but and feel free to stop me as you kind of look at what we're going through So go ahead and go on to the next slide here So when we look at our quality process our QSM process We call it method Q method quality or we also call it slim Right because we're trying to say it's an agile process We actually as part of our agile rollout spend a significant amount of time Making sure we had a quality process that worked with the agile environment So when we looked at the FDA requirements and we worked with them what we realized is What the FDA wants is sign-offs at the release level they want us to validate when we are sending software out That's when most of your signatures and most your artifacts have to be in place So what that enabled us to do is say you know what? Let's talk about what we intend to do at a high level and we showed them our roadmaps We showed them some kind of our epic level stories Those of us to know Cernar we don't like to say epic because that's our chief competitor So we use capability instead But so we use these capabilities and some of the acceptance criteria They're based on that to meet our high level requirements to say hey, we're not coding in a vacuum We know what we're going to be doing But we don't know it in such detail that I have to have a detailed requirements document already in place Then we move into our iteration phase and we do our user stories And we'll have some future stories on that and then just before Final release we do a final design input signature Which is our final system level requirements document that's been updated and Then we do our final design output signature Which is all of the artifacts that are produced by the coding system including our code reviews and what have you the key? Is when we are going through our iterations We are updating a solution level or system level document as opposed to My individual user story or I'm creating a document per project The FDA actually does not want a Project document level documentation. They only want a system level documentation and for us That was a big area where we were stumbling all the way along and now we realized hit system level that gives us a little bit more flexibility So When we do any work we look at everything as a change record we have released change records So we would sign off our design input at This release CR level we would then go down and when we're going to release we sign off again design input design output And you'll see the various things we would have their solution level requirements our traceability matrix Whatever we choose that to be those are all included in those If we go on and look at what we do during the iteration Our initial design input is those user stories and because we have that acceptance criteria on those user stories Maybe we have some visual design mock-ups that go with it that shows us again the intent for the change record that we're working on Then we would break down the tasks for our user stories, and I teach this in our classes I don't want to see And most electronic tools have the templates that allow you to quickly break out tasks and almost everybody Immediately goes into well. I got requirements and design and coding and whatever When I talk to my teams and Mike that doesn't help us right tell me what you're going to do And then one of the tasks that I want at the end I don't want to see a create or write requirements I want to see update requirements, and I want it to be one of the last ones We do because all we're doing is we use the user story for all that conversation that we want to take place And then one of our final tasks is let's just go update that requirement at the end with What we actually did not necessarily what we intended to do So we get rid of the marking things as future and tagging something that's not quite done yet something along those lines But we update that requirement and we sign off on that CR as as we are ready for it to release So we do it incrementally every single time With that in mind you might have kind of a hierarchy of CRs and so I can have a release CR way up top which is again where our final signatures And that's where we expect to be audited But I can have these children CRs and they can even have children grandchildren so on so forth and Kind of the philosophy behind it is My parent if I've signed off on it can actually that can serve as the initial design input for any of the children And from a traceability standpoint was we've worked with the FDA. They've been fine with this The big thing is is anytime we have a capability right something that we're considering to be big and epic or what have you It's releasable so each one of those CRs They have to be able to stand by themselves and they cannot span releases right if I have to try and span releases I create a new change record and we go from there Questions yes I Yeah, so what we have is we have a tool That we actually do our electronic signatures in and when we do those electronic signatures We note what the change records are and for us would note the user story numbers Here's all the user stories that we reviewed with this and and we attach the updated system level document or a link to the updated system Level document depending on what tool we're using Right, so it's just a one-stop shop for all of our Traceability and every every artifact we need to have and it's it has a single record and when the order comes in We're like here you go. We'll go look at this and you can see this project from start to finish No, it's part of the development team So the whole development team and that's part of our definition of done you have to include all of these things Right now is that heavier than your standard agile? Yeah, but it's not that much heavier And one of the things that we talked about yesterday was just the have making sure that that's versioned Right and allowing yourself to update that and know that we're going to update it and continually update it And one of the challenges we have we actually tried to go just using user stories And the challenge we have is what happens when I have a future story that invalidates a past story Right and if I go with just user stories that doesn't quite work for us But with the systems local requirements document I can still handle that scenario because again when you get audited It's always point in time. What was your process at that point in time? And what were you developing at that point in time to produce code? I? Typically it from neither actually it really comes from our roadmap right we determine what we are going to do and we decompose that work down and Then we'll sign off. We'll create the CRs based on what we expect to be the releasable components So again, we really want to go from vision to roadmap down to The release planning down to the iteration planning We let that flow through and just our release plans are what are the capabilities we plan to release and those are change records And then we bundle it up under a parent For our ottering purposes So so repeat that again. I want to make sure I understood it if you're in a regular environment You cannot get away from documentation, but what you can't get away from is how detailed the documentation is So for our I'll just share our experience. We used to document down to in our requirements things that I All the system shall make sure that all required fields are bolded and highlighted in yellow before you populate them And then they stay bolded and have different color whatever after the fact and we would list that for each and every field Right so when I'm audited I'm going to have struggles to make sure oh if I missed one How would I get through down to that well now? We pulled it up a level and said okay? The system shall honor required fields how we choose to do that is up to us Right, but we're going to make sure that it it meets our usability Standards and whatever our guidelines are and we can be audited at our guideline level And so it keeps it at a much higher level as opposed to getting down into the weeds no Because we're doing a sign-off at the end It's not fixed and that's the beauty of the fit the so what I can say is we plan to do these five Guess what the one over there on the far right didn't get done We just pull it out and when our final design input signature is done. It has what we actually did not what we intended to do Yeah, so I'd say my answer is gonna be the agile it depends We have a hundred and forty plus teams, so I can't say definitively what it is most of our teams try and fix Release on a monthly basis We do have some that release like every six months And others that release every two to three weeks and so what they define as release We just are giving them a pattern of here's how you need to follow it But make sure when you sign off your design input at the end. It's what you did not what you intended to do Okay, so that's why we love we like these parent child CRs so I can have a parent CR and Then have a child CR for each team that spans off of it because in reality We'll get we have to release the thing together and we have to make sure that when we release Software that works all the teams do their job Now if I have the middle one and one team gets done and the other one doesn't get done. Guess what we pull it out It's not in that release We move it into the next release Yeah, we also tend to assign an owner So one team is the one that's responsible overall For the project that we're working on and the other teams are supplementary to add into it But if they have their own artifacts we expect them to have change records for those artifacts Okay, all right. Thank you Matt and as Matt mentioned right we use the words framework and model, right? But I don't know if there is a framework and model For the way that you do this stuff just trying to inspire some some thought that you can go back in and challenge status quo, right? So Anything that you might do in one industry or one particular organization Probably isn't going to equate directly to another organization But trying to share ideas so that we can take those things back and challenge our own status quo Appreciate you coming up and sharing that Matt Okay, just in closing a couple of key points to remember is you know Agile extremism does not help in this case. It doesn't help to really say well, you know as an engineering team It's working software over documentation. So we don't have to do this, right? We've went to Agile We don't have to do this. So that's not helpful. You do have Documentation you do have to comply with regulations if your products get pulled off the shelf or You know you get your executive team in trouble because you didn't comply with something and now there's a risk in your product, right? That isn't going to help anyone, right? Agile or yeah agile and most regulating bodies are not at odds and that's where I had mentioned if you go actually take a look It's the the regulations themselves or read the guidelines. You'll see that Most regulating bodies aren't at odds with Agile and we can play very well together And one of the things that we can get that we can do is get to know your your regulators, right? Make sure that you go to them and ask them questions if you're transitioning to a different process to make sure that you're satisfying them first of all And like I said make sure that you're satisfying this person first you need to Satisfy compliance while preserving as much and injecting as much agility and innovation into your process as you can And make sure to implement the appropriate degree of rigor As needed by your product or what you're building in your environment. Okay. I've seen an organization That had their internal IT organization building or implementing an ERP system that was holding themselves to a greater degree of rigor Then their product side that was actually building cat scan machines, right? And that internal IT function was holding themselves to a very high degree of rigor So make sure that you're implementing the appropriate degree of rigor for what you need and in closing I think you know my point is pretty clear that I feel and have seen Agile being used in many of these Highly regulated environments, especially within health care So as I mentioned, you know I challenge you guys all to go back take a look at your own environments and see if there are Ways that you can inject some of the practices that you learned at the conference here into your own environment and Just a quick statement from Tom Grant of the Forrester group Like I said, he did a blog series last year and did a lot of research on this has done some speaking on the topic And he says you know Agile isn't just good for high assurance development. It's better than traditional methods Thank you for attending. I Think are we right at the time box? Because we'll take questions if we can if not be happy to kind of move toward the back