 Thanks guys for being here today and thank you to Lollie for letting us shake up the format a little. I wanted to make sure that we could showcase the work that we've been doing across government and also help to introduce other ways that you guys can get involved other than just working for the U.S. Digital Service. So as to Lollie said, I am the, in a very government-y title, Executive Director of the Digital Service at the Department of Health and Human Services. My team is chartered between the Centers for Medicare and Medicaid as David highlighted our work on Medicare payments as well as being able to work on other issues that come up across health and human services. Prior to joining U.S. Digital Service, I actually had started my career in government work to something called a contracting officer, which is the person that awards contracts. So I've managed a lot of really large, fucked up IT systems and overpaid for most of them. So I exited that space and really wanted to move into an opportunity to bring great technology to the table. So I worked for startups and a number of companies really making sure that companies like the ones you guys work for could come and do business with the government and that system is still broken, right? So I'm back here trying to make sure that healthcare and the way we are building and buying technology and government is changing. So right now our team has been working on something called the Quality Payment Program, which as David alluded to is an opportunity for Medicare to move away from fever service payments for healthcare. So every time you get an X-ray, a doctor is able to charge for that specific test, right? Whereas if it's outcomes driven, their ability to just continue offering the same service over and over again without being focused on actually making you better changes. So we've been doing that in conjunction with one of our partners, which is the Nava Public Benefit Corporation and Ivana is here today. And David Ko has been serving as our CTO of that program. So I'm going to let them take an opportunity to introduce themselves and then we'll start with some questions. Yes, you can all hear me. Hi everyone, I'm Ivana and I'm a product manager at Nava Public Benefit Corporation. So we partner with government agencies to radically improve government services. And in addition to working with Shannon's team at CMS, HHS, we also work with the Department of Veteran Affairs and other state and federal agencies. And currently I'm a product manager working on the QPP architecture. And I got a chance to introduce myself just a second ago. But my name is David Ko and I'm an engineer at the U.S. Digital Service. I'm currently serving as the CTO of the Quality Payment Program that Shannon just introduced to you all. And I also joined the U.S. Digital Service from a little company called OKCupid where I was the director of engineering. So if any of you are familiar with that. Nobody's outing themselves. Awesome. So talking a little, diving a little bit more into the Quality Payment Program and what that meant for Medicare. So basically Congress passed a piece of legislation called MACRA, which tasks CMS for implementing this concept called value-based care. When Congress did that it was pretty loosely defined. The actual implementation falls on CMS to figure out how to do that. Often what happens is that an agency equipped with people who don't have experience building modern systems are thinking about user experience. Start to just pull together pieces of the policy and basically create a pretty shitty user experience that all of us have experienced in our time interacting with government, right? So MACRA actually, MACRA and the Quality Payment Program gave CMS the ability to basically take a few disjointed complicated systems that made for providers needing to enter information into a number of disparate places and combine them into one. And so Kavana is going to chat a little bit about what it took to kind of get us there and what that experience was like. So as Shannon mentioned, the Quality Payment Program, QPP, isn't, I mean it's not the first time that CMS has tried to do value-based care. It's really hard as you can imagine. There's a lot of data that CMS needs to collect from doctors to be able to measure how they're actually doing and to be able to measure patient outcomes. And so we were tasked with figuring out how do we get that data from doctors without adding additional burden to doctors already who spent a lot of their time already filing paperwork but also their main job is to take care of patients. And so what we did is we're approaching it from an API-first approach and so we're building the Submissions API to enable both the internal systems that allow doctors to submit this data to QPP but also enable third parties who build the software that doctors are already using to track their patient care. So I'm going to pass it over to David so you can talk more about that. Yeah, so an API-first approach, which probably doesn't sound particularly revolutionary to anybody in this room, is actually incredibly important and revolutionary for the government because basically you have the system whereby lots of people have to submit information to the government and it's complicated. Like if you've ever filled out a government form you have a sense of how frustrating this was, except times a bunch, and then a lot of other people end up paying other people to actually submit information to the government for them. So there's all these companies whose business models exist around giving the state of the government, figuring out how to fill this form out. And we thought well, they would be able to get it to us a lot easier and a lot faster if they could rather than having to go through what was maybe an unimaginably manual process for myself and many others in this room, I would bet we were like we could build you an API and you could submit that data directly to our API and we could then make that data back available to you via that same API to make sure you submitted the right thing or that you could show what you submitted to your customers. So having this API previously once you submitted something it was a bit of a black box. Again if you've interacted with the government you probably know what I mean. We're trying to break open that black box and give people some visibility into what they're doing and also give people an efficient way to actually do what it is that they need to do the government is asking them to do. So this API first approach is actually even though it's something very simple that we're all very familiar with is something that is actually really important and really revolutionizes the way people interact with the government. Yeah it's always a little bit funny to talk about some of the quote-unquote tough tech problems that we solve in government in part because to an audience like this it feels so simple right like that's the obvious answer but getting to that point took a ton of persuading and a lot of when you mentioned the word API people would say no no no no and then you would realize that actually they didn't even know what that meant so there was a lot of conversation around what is an API that had to happen which was exciting to get to this place to help them understand their impact on the market and the way that we could start to improve that process for physicians. So one of the other interesting things that's happened in my team's time working on the quality payment program is that I'm not sure how many of you have actually had interactions as a contractor trying to do work with the government but as I alluded to it's unbelievably difficult not only the process of actually getting awarded work and selling your services to the government but actually implementing the work and having to deal with the federal process or the way that we write requirements and tell engineers and people to do things is unbelievably difficult and a super smart technical person in the room will struggle to be able to say you're doing it wrong because that's the person that's writing their check. So we took a really different approach with the quality payment program and we really started to rethink the way that we could run true agile development in government not this like all the requirements are up front and you're sprinting and somehow that's agile like we wanted to actually do this and teach people teach federal employees people that were working at CMS how to do this again and so Yvonne is going to chat a little bit about what it was like to be a contractor coming into that experience and also what it meant to define a product vision for the first time and teach people at CMS how to do that and what that meant so if you want to talk a little bit about that. Thanks Shannon yeah you're definitely preaching to the choir coming in as a contractor was definitely new for me having come from the private sector before this I was at SockDoc in New York and before that an e-commerce startup so this was totally new I came in and there were a ton of stakeholders and it wasn't ever clear still isn't clear sometimes who's who and like what role they play but before I before I you know take let's take a step back and talk about product vision you know it's product vision is really important to a product team because it's sort of your sort of North Star and that's that's your like guiding principle and your goal for so that it serves as an anchoring point when you're making decisions about what goes into your product roadmap and making tough prioritization issues so it's especially important when there are a lot of cooks in the kitchen a lot of different stakeholders both on the government side and the contractor side and so out on top of that that we're implementing legislation so we're implementing MACRA QPP is they publish 2,000 pages of policy every single year for the program so that's you know that's part of that's that's another stakeholder in itself and so these are all these different pieces and it can be a real challenge so it's really important to have that product vision and so as a contractor we work alongside our government partners as well as other contractors who are building other parts of the QPP ecosystem to align on that product vision so it has its challenges but it's really important to have a very collaborative environment where everyone feels heard and we're we have that we're all aligning towards the same goal so as Shannon mentioned before the goal of quality the quality payment program is to really incentivize quality care and at the same time decrease reporting burden for doctors and so how for for my team in particular we're building the submissions API which is core to the system how do we support that goal and so to get to that point we partner with our product owners and product strategists and engineers both at CMS as well as USDS we talked to the other contractors who who are building the the other systems that that allow doctors to submit data to understand like what what what is the goal what should be the goal so and for the API that was to get as many vendors on the integrating with the API as possible and that took a lot of collaboration a lot of discussions and in addition to that something that we're really passionate about I think at USDS and Nava is approaching problems from a user-centered perspective so always talking to users feeding that feedback back into our roadmap showing our stakeholders both government and contractor side just what our users are saying and what the data says because it's hard to argue with that you know yeah and of honest being pretty modest there's not just a few other contracting teams there's 13 so there's 13 different companies with different approaches to technology building in the same system it has a lot of benefits and a lot of downsides and things that have had to be managed but it's created this amazing opportunity for us to really deliver a great product so yeah there's a there's a tremendous amount that goes into being a contractor with all these extenuating factors that could impact what you're deploying so speaking of that kind of more in-depth technology and Ivana mentioned policy so there's program policy but there's also the policy that governs the way that technology is built in government so I'm gonna let David chat a little bit about what it's like to navigate things like technical reference architecture that hasn't been updated since 2003 or requirements around security yeah so you want to talk about those my favorite topic security compliance in the government so yeah I like to kind of think that when we're solving problems and building systems in the government building software is not an easy thing to do and the government basically says every time you know outside of the government you would have one problem we're gonna give you a second problem to deal with at the same time so one thing that we we had to deal with was for instance if you think about CICD right that was something that we wanted to implement that we have implemented you normally would just you know have to go through the trouble and like Shannon said we have 13 different teams working on this project so CICD having that be effective and work consistently despite the fact there are a bunch of different teams is not a trivial thing to start with but on top of implementing that there are issues such as there's a thing you may probably never heard of if you have I'm sorry called an ATO which just stands for authority to operate basically it's a it's a thing that you have to have in the government if you want to run your system in production someone has to sign off that your system is secure that you can run it and won't be it won't get you know compromised which having someone sign a piece of paper helps a lot with that but in many places in the government people are just used to you know you can talk to them and say well we want to do CICD and it's all great you kind of know that they didn't know what you just said but then they'll say but we also want you to make sure you because you have this ATO updated every time you release so you can go through this month long documentation process every time you do a release and so then I'm like I know I'm sure you didn't know what I meant when I said CICD so some part of this is that we have to just explain and reset where the government lands with with regard to these but in other places we do have to figure out ways to meet compliance which means getting a little bit creative running NESIS scans as part of our CICD pipe on in a in a regular basis so there are ways to kind of come and meet in the middle there but it sets an interesting set of new technical problems just to do the things that we we kind of all want to do on a regular on a regular basis yeah do you want to talk a little bit about duplicative encryption and some of the run-ins that we had with that oh yeah so sometimes it's actually worth not fighting the battle and just doing what people are asking for to check off the box I have a bunch of stories like that one of the the good ones that I worked very closely with Devana and her team on was that we needed to encrypt every every taxpayer identifying our number which is basically a social security number in our it has to be encrypted everywhere and so one of the places that they wanted to be encrypted was well it needs to be encrypted in the database and we're like okay well our database is encrypted at rest so we're okay right well well no we want to make sure that you encrypt the the tin itself so we're like well okay we kind of talked about a little like this doesn't make that much sense and it's gonna make our lives a little bit harder because we're gonna have to do searches on this and but at the end of the day right you can spend two or three or four weeks trying to fight a battle or you can just put a nice little encryption hash before you insert in database something that you can still search on and it makes everyone happy so sometimes sometimes that's the battles that you don't fight that end up end up being important but yeah a lot more stories like that if you want to come find me afterwards yeah there's it's always interesting to have to do a little bit of trade-off analysis to figure out what you want to push really hard on and what you don't and a lot of it has to do with effort right it's gonna take the team a ton of time to do something we'll technically you know we'll try to push back more but some stuff is just not quite worth it so speaking of continuing to kind of move stuff forward the quality payment program our first submission window is ending this month and so I'm gonna let Yvonne attack a little bit about what the future roadmap looks like and some of the other problems that they'll be tackling in the next coming years so I'm going back to that product vision we want to get more vendors using the API so we're approaching that from sort of a few angles the first is what's the market size and how can we increase that so we're looking at different re revamping our authentication process so that more vendors like electronic health record software and other health IT vendors can use our API and then the second thing is like once they get in the door what what barriers do they have to integrating with our API and you know a few examples based on user research not having a real a true integration environment is is a real pain point so that's something that we're gonna be focusing on next quarter and then finally in terms of like sort of longer term we're continuing to work with our government partners and contractors to better understand how our team can support those overarching goals of QPP which is ultimately to deliver more real-time feedback to doctors and faster so that they can continue to improve the care that they deliver without being burdened by too much reporting and administrative stuff yeah so obviously we're up here telling you fun stories about the work that we do and how amusing it sometimes is and hard it sometimes is one of the things that we wanted to make sure to leave everybody in this room with is an understanding of why we've chosen to do this work so I know for me personally as I highlighted I had the opportunity early in my career to really understand what government dysfunction looked like and really kind of caught this bug of wanting to make it better like this is my government this is our government we all have the choice to get involved through a multitude of pathways and my passion for health care I think has really been driving my time at CMS staying in this role has made me recognize that CMS is the largest payer they pay for the most amount of health care in this entire country and the things that CMS does sets the standard for the rest of the industry which is a little scary at times when we talk about the fact that these are the technical issues we're running into when other payers say that they're waiting to build a new system until CMS does and so there's this drive to show up every day to make sure not only that government can function better but that health care itself can start to really move to a place where I think all of us want health care in this country to be and so obviously I don't think that government is anybody's really first choice when you've got a technical background or interested in building cool things I know for me those are the reasons that I'm here but I wanted to make sure that Ivana and David who both came from private industry companies can chat a little bit too about their drive to be here and show up every day yeah I want to echo what Shannon what you've shared with us about the major impact that CMS and some of these agencies have on our everyday lives Medicare and Medicaid I think combined treat benefit over a hundred million beneficiaries other projects that we work on at NAVA we work with the Department of Veteran Affairs to modernize the appeals process when a veteran files an appeal for for benefits they wait on average five years some folks like don't don't survive those five years while they're waiting for that and so those are problems that you know when you read about them it's heartbreaking but then take it back to a personal level and I've I mean I think every one of us have experienced government services and how many of us have low expectations you know when you go to USPS and the line takes an hour and there's only one window open or you go or me personally I when I when my startup shut down I had to file for unemployment or you know two years ago I was unemployed for a little bit and I had to sign up for Medicaid and I kept getting PDFs from the New York State Health Exchange telling me that because I made zero dollars it was over the limit of 60,000 so I didn't qualify like how to and I couldn't it took forever to figure out how to get someone on the phone to anyways so a lot of personal experiences but think about the communities you move through to right you know I'm probably gonna have to help my parents who are retiring soon to sign up for Medicare you know we have tons of people I have a brother-in-law who who's a veteran and he may have to tap into these benefits so I think we're all privileged in a lot of different ways but think about the communities who for whom like the most vulnerable parts of our our nation that these little problems that we have where a website has business hours or the line at USPS it's like an hour some people can't afford that so that's why I come in every day yeah definitely seconding everything that Ivana said and and Shin said I think from my perspective the one thing I would want to add is I definitely knew you know that I wanted to make a difference in the world make a positive difference and there are a lot of ways to do that volunteering for various organizations but I knew also that I had a particular skill like one thing that I was good at and I was building software and so you know how how are you going to use that to make the world a better place there are a lot of ways for sure but hearing about you know USDS when I learned about it for the first time and heard about these systems that you know if what you can do is keep a server up running or build an infrastructure that allows someone to develop a big complex system that's there's there's a group of people where they're like well we would really like to make the entire American healthcare system better but we just don't have anyone with the expertise that knows how to like build a big complicated software system if only we had that person although people have a lot of obviously a lot of different opinions about what government and what it does I think there are a lot of people who just want to help and there's not a lot of expertise so government runs a ton of software systems and it does it generally a pretty poor job of keeping them online of making them usable and of building them in a way that allows the government people to make the best use of them so and that's mostly just because the people with the expertise to do that aren't aren't around to help to help accomplish that the people who do build these systems are very hard working great people to work with but there's there's definitely a need for expertise so that was one thing that drew me very much to this job I think just echoing what David said I know we spent some time today making fun of the way that government works but there's a ton of hard-working civil servants who show up every single day to keep it running and they spend 20 plus years of their career in the same roles and being able to bring new expertise in has been transformative I've watched employees at CMS who now all they talk about is human-centered design and they want to apply it to policy making to regulation and that is really that that is the offering that we bring right scrub in with us for a project and leave and they will continue to do great work so in that van if anybody is interested in applying as David plugged earlier we have a table in the corner usds.gov slash join please consider submitting an application there's a number of other programs and in that same note companies like Nava are what end up driving a lot of the work forward so if any of you have not done business with the government before I would highly encourage you to try it I know it's arduous and painful there's a lot of people trying to make that better but you should definitely consider it because it's great engineers and technical talent that come in as contractors that actually are the ones who end up implementing the work so thank you guys all so much for listening to us today I appreciate it appreciate all of you guys being here thanks you guys have some questions for us we have a couple minutes clearly I explained how government works in its entirety oh there's a few back there can we take them a mic to Lolly thanks I'm gonna give this one to them to these two maybe David go first well there was I don't know if this is the worst acronym but we definitely sometimes if you want to make a project happen in the government you actually have to name it with an acronym otherwise no one will take it seriously so this was this was not a CMS but we did have to name a project flash which I believe was a procurement vehicle at DHS and it stood for flexible agile support for the homeland I may have missed something in there but yeah I think lean was the L my example is pretty close to QPP but we have a system called the web interface which is the most generic name ever and it's CMS W I and I'll also add that we I think actually multiple contractors probably have this but we have like github repos where we can like or spreadsheets where and plugins where you can double click on on acronyms and it'll show you what it is because we have so many that we need to keep track of so can you give an example of you say would be a great provider in industry for health care that you work with as part of CMS some story about it can you repeat the question could you give an example of a provider who was an excellent integrator with CMS as a shining example of best practice to make your lives easier so one one good example of I think you're asking about the integration with potentially our API Athena health who's a both an EHR and a registry for has done a very good job of of really making good use of our of our API they've really taken they they were like we really want to have this and actually the first day that we had launched it in production all of a sudden 30,000 hits came to one of the endpoints because they were they were technically we weren't expecting that but luckily we system was fine with it but so they were very quick out of the gate sure I would also well I would add that they're also a really great partner for CMS too because they they really understand the value of human-centered design and designing with users not for them and so we we actually have a I have a monthly check-in with them and we often like share feedback with each other that's feeding back into the product so they've been a really great partner in that sense I'm just gonna give the very governmentee disclosure of we called out Athena health but we have lots of other great amazing partners that we share very similar experiences with so yeah so my question is more related what you're saying earlier about to trying to get more more more companies and more people that are involved in kind of DevOps practices involved with the work that you do what are some of the lessons learned that you might share like a couple two or three key things that people should kind of come prepared to do or recognize as they as they try to work with HHS or CMS yeah so I would say kind of product market fit is a really important thing here paying attention to what it is that the government is actually asking for and also providing consistent feedback a lot of a lot of contractors want to just write to the language that a federal agency put out right and that's how we win business we basically word for word regurgitate the thing that they said and promise them that we can do it but I think that there's an important piece there of actually pushing back a little and making sure to differentiate yourself by saying you're asking me to do this but if you do that it's gonna give you a bad result so that's one of the things that I would always recommend doing I think the second piece is partnering with businesses who have established channels for business development a contractor who's got a few contracts and maybe you add a skill set that's another great one and the third one is also just continuing to be involved in the conversations that governments having like if people are putting out RFI as if there's request for comment on proposed policy or regulation like ensuring that your company is consistently involved in those communications is the best way to get FaceTime with federal employees who are making decisions last one so I know the QPP program and the Blue Button API reside today on GitHub can you talk about the process of getting the open source community involved in those projects how like how does that work I think that in many ways it's it's very similar to I guess the process of getting the the open source community involved with any any new API that you're launching so a lot of the things that we do are very similar we are basically building a software system at the end of the day so you know doing the outreach to people who might be interested hearing their feedback seeing what they would build on your API and making sure you can support that I think the other the other layer to that is to make sure that we know what is the most important thing to open source for for people who are out there because there is a lot more work involved in getting that public to private the public flip switch flipped on GitHub that in the government then there normally is you know a bunch of people are gonna have to sign off on that and you know you're really it doesn't make people who don't fully understand technology very comfortable say oh we're giving them all of the source code to our system that sounds dangerous to a lot of people especially people who are used to software development and who may be comfortable with the idea of security is obscurity of as obscurity so that doesn't prevent us from open sourcing it just makes everything harder and it does mean that we have to pick and choose our battles a little bit more so especially if you are on the open source side and you're integrating with these it's very important to figure out what it is that is most important to you because even though things will move a little bit slower we would like to get you what is most important as soon as we can so I will great well thank you so much I appreciate David I'm sitting here looking at your scarf I'm super jealous so I'm putting mine back on but I appreciate thank you guys so much hopefully you'll stick around