 Good morning. Sound off, Jonathan. It's great to see so many folks. Great to see so many folks here standing remotely in the back. So we're here from the government and we're here to help. My name's Jonathan Mayer. I'm Chief Technologist for the Federal Communications Commission's Enforcement Bureau. I'm joined by wonderful colleagues throughout the federal government. We have Lori Craner, who's Chief Technologist of the Federal Trade Commission. Eric Mill, who's a senior engineer at 18F in the General Services Administration. And Alan Friedman, who's Director of Cybersecurity Initiatives at the National Telecommunications and Information Administration. Alright, let me try a little louder. And with apologies to the folks who are near speakers. And Alan Friedman down at the far end, who's Director of Cybersecurity Initiatives at NTIA within the Department of Commerce. So it's a great cross-section of different technology policy roles within the federal government. We have working on cybersecurity in the main component of the executive branch. We have delivering services throughout the government. We have an independent agency working on security and privacy issues, real thought leader within the government. And then we've got the FCC. So I'm going to ask each of our participants to say a few words about their agency and what the agency has been working on in technology policy and the role in the agency. And then for the most part this is going to be an extended Q&A session. It's up to you to lead the conversation. There are microphones at the left and the right of the room. Please line up and have your questions ready. And my game plan is to just alternate and we'll do our best to answer. Okay, so Lori, could you start? Hi, good morning. So the Federal Trade Commission focuses on consumer protection. And in the security and privacy space, we're very much interested in protecting consumers from having their private information breached and leaked. We're interested in protecting them from unfair and deceptive practices. Scams, fraud also spam, robocalls, these are all things that we are very interested in investigating and in finding ways to protect consumers. We also do outreach to consumers about how they can protect themselves and avoid falling for some of these scams. We are very interested in talking to researchers and if you come to our session at 1pm we will get into more detail about that. But we would like to work with the researcher and hacker community to identify vulnerabilities that we need to be aware of, to identify tools that we can use, that consumers can use. And so we encourage you to come talk to us. We've also set up an email address research at FTC.gov where you can send us the things that you've discovered that you would like us to know about. We also have a number of events that are coming up this fall that we are going to be interested in having researchers attend. They're going to be open to the public as well as webcast. So we have a workshop coming up on drones. We have one coming up on smart TVs. We have one coming up on disclosures. So privacy notices are one form of disclosure, but other kinds of disclosures as well. A bunch of things. And then in January we will have our privacy con event. You can read about all of these things at FTC.gov slash tech. Thank you. I'm Eric Mill and I'm with a group, a relatively new group in the federal government called 18F. It's like one 8F. And we're about two years old and we're housed in the most excitingly named agency in the federal government, the General Services Administration. So we're about a couple of hundred people, mostly not in DC, and we are trying to do technology right in the federal government. So we actually have dozens of engineers, product people, UX people, design folks, security people. And we are trying to make sure the government can do well for itself on technology and to make sure the government can do things the way that we all want to do things. We can embrace the cloud, use open source, do agile development. We can bake in privacy and security from the beginning of the development process. So we try to lead through implementation and delivery. We do some of the things that are maybe most relevant to this crowd. So we're currently working on getting a bug bounty program started on a number of our public facing systems. Shout out to DOD who just did the first government bug bounty program fairly recently hacked the Pentagon. Definitely learned a lot from them. We're also trying to hire, the government does in fact employ information security professionals. We're trying to do it a little bit differently and hire people that are senior technical implementers that don't require security clearance and put them at work on a variety of government-wide systems that make the country work better. And in particular part of my work there is focused a lot on encryption and in particular, AT&F has been a really animating force on web encryption, on HTTPS in particular. There actually is a federal policy mandate right now in the executive branch to move everything to HTTPS only with strict transport. That's something that our organization has animated and put a lot of energy into. Something I personally work my tail off on and that's gotten me the chance to meet a lot of the good folks here and hopefully we're making the government a better place. I'm Alan Friedman. I'm with the US Department of Commerce in the National Telecommunications and Information Administration. We're part of the administration. We are the president's advisor on telecom and internet policy. You may have heard of my boss, Assistant Secretary Larry Strickling who's trying to keep the internet free and open with the ICANN IANA transition. And thank you. And in fact that's a large part of what our organization does is represent the equities of a free and open internet both inside government discussions as we sort through policy as a giant complex government and also across the private sector. In fact we take this notion of multi stakeholder engagement quite seriously. Too often there are policy issues where if we wait around for legislation it's going to take too long and it may not be great because there's going to be lots of complexities and legislation is a pretty big hammer. Regulation also takes a while and when Jonathan isn't writing it himself it's not always optimal. So what's the tool we have left and our tool is we try to get the right people in the room and say guys let's solve this among ourselves with the right engineers in the room from all the different relevant stakeholders so that we can demonstrate that this is a solution that can be solved by coordination collaboration rather than waiting around for long drawn out legal processes or regulation. We have two ongoing initiatives that might be of interest to you guys right now one is on everyone's favorite topic vulnerability disclosure. We know that this is not a new issue but on the other hand the cliche that everyone is now a software vendor really is true and there are a lot of organizations that have never had to work with researchers before so we're bringing together security researchers, vendors middlemen, everyone possible and saying hey how can we equip companies and organizations around the country and even around the world to know what to do when someone knocks on their door and says hey there's a big problem in your system and we can help you solve it how do we get people along that path. The second initiative which we've just announced is around everyone's buzzword favorite IoT security. It's sort of recognize that one everything is going to be connected sooner or later and two security is a giant flaw. No one's really building it in right now. How do we start that process? How can the government promote a better marketplace for that? So we're starting with a small debate and say hey patching seems like an important issue but there isn't really a universal definition for what it means to be patchable. So let's get some tech engineers, people who make products, consumer representatives, security researchers in a room and say what are the different dimensions of patchability. Here are all of the technical details. Here's the user experience here's the connectivity issues. Let's build a taxonomy and then try to collapse that down to a small set of definitions. There is no one size fits all. And from those definitions have a couple of words that we can tell consumers don't buy a smart widget without this on the box. Voluntarily. This isn't the government saying don't do this. This would be consumer reports or a budge or someone else saying hey look for these words but these words are backed by a couple of paragraphs of technical specifications. And by the way if you lie about what's in your box we have some colleagues in government who know how to take care of that. So if you're interested in talking about IoT security or patchability disclosure we're very happy to have you engage. And we hope you do because when we meet it we say it's multi-stakeholder. Whoever shows up gets a voice to weigh in and make sure that everyone else can hear what you have to say. Thanks. So let me touch on the FCC briefly and then again open it up for questions from the microphones. So the FCC is the federal regulatory agency for communications, infrastructure and services and that includes communication, security and privacy. So in the US legal system the FTC is sort of the closest we have to a catch all data regulator but there's a lot of sector specific regulation. So for instance our colleagues at the Department of Health and Human Services deal with medical security and privacy. FCC does communication, security and privacy. And it's an independent agency in two senses. The first is we're not within one of the cabinet departments and the second is the commissioners are nominated by the president and confirmed by the senate. But they don't report to the president. So if the president wants to provide input on an FCC proceeding he writes a comment to the agency just like any of you can. So the independence is very real. The FCC's core function is independently proposing enacting and enforcing rules. So it's a little bit of a blend of the three branches of government. We say first here is what we think the law should be on this issue and then put it out for comment and any of you can write in and say why were right or wrong. Then the FCC finalizes those rules and then ultimately it becomes a job for where I set primarily the enforcement bureau to make sure those rules are followed. So the FCC's authority in communications covers a range of technologies. Radio and all sorts of RF emissions. Television by their broadcast or cable or satellite or fiber or whatever is next. Telephone of course and the agency's recent focus has been especially on broadband internet. So you may have seen the term net neutrality somewhere or other. I'm just going to guess this crowd's heard that one before. So the FCC proposed strong rules to protect the open internet and just a month and a half ago the DC circuit concluded that those rules were consistent with federal law and net neutrality is the law of the land. So much of the net neutrality proceeding focused on the kind of economics of innovation online but at the time the commission said we know we're going to come back and look at security and privacy. Do more rules on security and privacy left that open for another day. And so earlier this year the commission proposed rules for ISP and security and privacy saying ISP should be transparent about their practices should have reasonable security protections in place for your data and that you should usually have choice and opt in choice if your ISP wants to repurpose your data for advertising or anything else. We've also been vigorously enforcing security and privacy protections that are already on the books. So for instance earlier this year we settled with Verizon for tampering with their customers internet traffic to insert their unique identifiers that made them trackable online. There was over a million dollar fine but more importantly Verizon agreed to make the practice opt in for any of these headers going out to third party businesses. We've also done data breach cases against AT&T in Cox and just earlier this week you may have seen we reached a settlement with TP link a router vendor over selling some routers that could be modified to create radio interference but an important part of that settlement was TP link committing to working with the open source community and chipset manufacturers towards bringing Linux support, custom firmware support onto their routers. So even when we're kind of operating in one of our kind of classic areas we're trying to make sure to promote innovation and make sure that sort of the freedom to tinker is protected. The freedom to lawfully tinker my boss likes to point out. Let me close by touching on some of the exciting work in progress we have. So just recently the commission set up the sort of licensing infrastructure for upper microwave spectrum. Now upper microwave spectrum I knew nothing about before coming to the agency. Turns out the technology is now there to make this very useful spectrum and it's widely believed to be an important component of 5G wireless technologies. So the commission set out its security expectations for the spectrum and plans to address 5G more fully soon. Those expectations include that there be a routing security and for voice calls and text messages, security from one communications device to another communications device. We think that's what the use of the spectrum should look like. And if you'd like to hear more about it one of my colleagues Admiral Simpson is going to be doing a presentation at the Internet of Things workshop here at DEF CON. He's also done quite a lot of work recently to address robo calls. The chairman sent out letters to the major telecom firms saying he expects immediate action. AT&T has taken up the charge. They're leading a new multi-industry working group to deliver actionable results including new deployments of call authentication standards, new efforts to make sure phone numbers like the IRS mainline can't be easily spoofed and efforts to build compatibility interfaces so folks can bring filtering like spam button technology into the phone system. And then I mentioned earlier the agency has been working on security and privacy rules for ISPs. We proposed those in March the comment period closed recently and so that remains work in progress. So that covers what I wanted to cover for the FCC and again this is your session. It's going to be mainly Q&A. So there's a microphone there and a microphone there and by all means line up and we'll take your questions. Thanks. Start over here. Yes I imagine this is for the FCC. I'm wondering what is your time frame for the telecoms to harden their system 7 vulnerabilities? So we work closely with the telecoms to implement better protections across their networks including SS7. The commission hasn't put out a firm timeline on that particular issue but an important part of the 5G communications work that the commission is doing is saying here's the way we think the world has to look going forward obviously we're not going to tell companies how to build their networks but we're going to set expectations and we're going to work with them to make sure they meet those expectations and for now that's an ongoing conversation but the commission does have regulatory authority and can always be firmer if that becomes necessary. Over on the right. Hi. This question is for the FCC. What are some bits of advice you can give to private citizens so that we can be impactful during the request for comment stage? Lately it's becoming an increasingly politicized event with large corporations lobbying excessively hard and we don't have the monetary resources to have our voices heard and we as technologists know that some of the things that they're doing have led to stagnation of broadband in rural areas increasingly nasty behaviors like with Cox trying to do the opt-in service for additional privacy and it's just it seems that it's getting worse in some ways. How can we have our voices heard? Thank you. So let me start with the FCC component of this then I'm going to hand it off to Alan and Eric to address getting your voices heard in the processes they work on. FCC's usual process for doing a rulemaking is we issue something called an NPRM, a notice of proposed rulemaking where we say here's what we think the law should be in this area and then there is a usually about 30, 45 day comment period then another equally long reply comment period then there's some period of internal decision making, stakeholders can continue to come in and meet with the commission, continue to write letters to the commission and then ultimately the commission proposes final rules then usually someone sues and then finally after a judicial review the matter is settled. So that's the process as we're making sure your voices are heard we'll have to be careful not to comment on any ongoing proceeding. I think it's fair to say that I've been really heartened to see how the process works being in the agency. Smart comments get noticed and if you come to the conversation with something new to say and especially if you have some real data to bring to bear it gets noticed and so sort of the best advice I can give on how to contribute to the debate is make sure that what you write is not duplicative, ideally doesn't use curse words at us or something like that and gives us some really constructive input. Those comments get singled out. Let me also add just as a purely procedural matter, make sure you're commenting on the right proceeding. Every so often folks will file comments in the wrong place and the system at FCC has recently gotten a lot better for filing comments. We have a whole new online comment filing system but make sure you file in the right docket and make sure the issue you're writing in about it is appropriate for that docket. So sometimes folks will have really smart things to say and really great data but it's just not germane to the specific issue in front of the agency. By all means call that to our attention. Feel free to kind of reach out to who you think is the appropriate contact at the agency but it's easy for it to get buried in a docket if it's not germane because someone will review the comment and say it just doesn't bear on this particular proceeding. So that's a kind of procedural note. Okay, so now over to Alan and then Eric. So as an example of a comment process that I don't know how effective it was, a few months after I joined the Department of Commerce last year I get a call from one of my colleagues in a different part of commerce called the Bureau of Industry and Security. He says, hey, we're about to release a proposed rule based on this arms control agreement known as Vasinar. So we had some discussions and we helped prepare them for the fact that they were going to get strong responses. And we did get a lot of responses and many of those were really helpful. This was a case where industry and the security community were on the same side but they brought two very different perspectives that was very helpful. It's challenging because often people were commenting based on news stories that were based on other news stories and so by the time they filed their comments it wasn't something that was directly related to the regulation because a lot of this stuff is quite technical. So as Jonathan said, make sure, have as much preparation as you can. But this is an area where we got the comments and they were overwhelmingly negative. I think there was one comment in favor out of over 200. And so the US Department of Commerce worked with our government colleagues and has gone back to Vasinar to try to renegotiate. And so that I think is an example of feedback from the security community driving policy in the direction that it should. And so as you are preparing to engage, it helps to talk to other people. If you have colleagues or friends who are engaged in the policy network they'll be able to give you a little bit of background. If you're curious at least in our case, I don't know if the FCC can do it, but in commerce we'll talk to you about what we're looking for so that you can tailor your feedback to give us the insight that we need to make good decisions. There are lots of organizations out there that are engaged in a lot of these issues, whether it's EFF or I am the Cavalry we need more advocates for security as a unique value. So please try to engage and learn as much as you can and then give us as much feedback as possible. So I'll just briefly add on, it's actually a bit of an outside perspective. So I'm not in a regulatory agency now at GSA. Before this I was at an NGO and non-profit called the Sunlight Foundation that does open government and transparency work for about five years. And I worked a lot on trying to make the regulatory process more accessible to people because I watched many different times where people leave the opportunity on the table to comment on a regulation. And I'll tell you that the people who will always comment on a regulation that affects them are affected businesses or the private sector, not very often comparably do you get real public constructive input on things. And it's not always well known that, and this is distinguishing from a lot of other countries in the world that in the US executive agencies that are issuing regulations must respond to every unique comment they get. They have to at least acknowledge it in some way. And I've read many final regulations that went down and addressed all the different groups and notable comments that they got and changed their minds on small and large things as they went. You don't always get your way, but when you participate showing up really does matter. And that was my personal experience as an advocate and like open government lobbyist working on these issues that showing up is everything. So I really do encourage you to, I mean the Federal Register, if you go to federalregister.gov, they actually added in the last few years a number of really great alerting and feeds systems for you to follow things more easily. It's actually a really great team that built federalregister.gov they were invited by OFR to do it after they did an app contest as an outside group of developers trying to reimagine what federal regulation and commenting should look like. And that is and there are other services that will help you do that. I just strongly encourage you to take that seriously. I'll just add that at the FTC we often are looking for public input usually when we announce that we're having a workshop there are opportunities to comment both before and potentially get on the agenda as well as after the workshop. And we are very much interested in people who bring us data. We want data, we want empirical results, not just the opinions which are nice too but if you are a researcher who can bring us data that's something that we are going to be very interested in seeing. Let me amplify that point before moving on to the next question. We hear a lot from lawyers in the government. We don't hear so much from technical experts and so that sort of input is incredibly valuable and it gets noticed. So you mentioned that the DOD now has a bug bounty but for sort of an opposite perspective one of the things that I do is run census.io and other scanning for security things. And five years ago when we started the DOD sent us a very strongly worded email saying you'd better stop scanning us. That means we can't participate them with them, we can't tell them about vulnerable TLS implementations. So how do you engage with the DOD beyond just submitting to their bug bounty? That's a difficult question given that none of us are from the DOD. We're probably not going to be able to give you the answer that you're looking for but in general the closer you get to communicating with subject matter experts inside different agencies the more you get answers that make sense and creative solutions to different problems. The DOD Hack the Pentagon program was started by the Department of Defense Digital Service which is a relatively new team inside DOD. It's part of the US Digital Service which is a White House initiative that has created digital service teams in a few different agencies but that's about the best I'm going to hire. Maybe anybody here is going to be able to give an answer to that. I think just large organizations are not monolithic and so as we said the closer you can get to the people who engage the better in the private sector we work with large companies inside our process on vulnerability disclosure who are trying to figure out how can we work with researchers even as their general counsel's office is writing comments about how we need to bring back DMCA controls on their products. So the trick is to find the allies in any organization that you can. I think this panel probably is a great way to start to find the right people so good luck and thank you for reaching out. Thank you also for running census.io. GSA 18F uses that data in our work all the time I personally use it in my work to understand the government surface area and to report things to other agencies as necessary and then to work with them to fix it. So I'm a student who's going into my senior year at high school and I was just wondering how did you guys get into the federal government and how could a prospective student also get in. Thank you. Go down the line. So one, I think there I'm going to speak for everyone to say we desperately need smart, passionate and technically aware people in government. Desperately need them. And the advice I would give is it is fairly easy right now to go from the technical world into a policy track. My background is in computer science, wasn't very good at it so I have my Ph.D. in policy and when you're meeting, so it means I'm a mediocre economist and a mediocre coder and when you're mediocre that many things end up in Washington. And I was an academic and then someone talked me in but I think the advice I would give is stay on the technical side as much as possible but engage in policy in your spare time and eventually you'll find an issue where you can find the right person and weigh in and they'll say we need you on our team. Yeah, I mean so as somebody who went primarily my background in software engineering I have a CS degree but I work a ton on policy day to day now. It's really as simple as becoming an expert in something and being willing to talk about it publicly, privately, leadership without fear and have confidence in what you say and really develop your skills as a communicator. Being a good writer is just a universal skill that will make you more effective at bringing people into your way of thinking projecting that you know what you're talking about and that's something that even if it's not going to be for even if you don't end up working on policy for some amount of years take the time to keep exercising those muscles to keep writing and to keep getting feedback on that and to keep becoming a good communicator. So I started my career working in AT&T and was doing research on privacy mostly and I actually presented research to the FTC 20 years ago. I went to their workshops and when FTC staff said can someone explain again how third party cookies work? I would take time from my day to call them back and to explain it yet again and basically became known to them as someone who is willing to explain these technical concepts in plain language. I then became a professor at Carnegie Mellon and have steered my students and their research to trying to make our research relevant to some of the policy needs and submitting our results to government agencies. Until right now I'm actually on leave from Carnegie Mellon and the chief technologist position at the FTC tends to be an academic who comes in for a year or two. The other point I want to make for our high school student friend is that if you know that you're interested in government service there are scholarship opportunities for you. So scholarship for service, basically if you are a US citizen and have technical interests you can get the government to basically pay your tuition in exchange for you than committing to do some work for the government and so it's a great opportunity. To amplify something Lori said it's about explaining things to other people. The community that we're all a part of here, this conference is tremendously huge even just this room is filled with people. This is a large amazing community and you could spend years, you could spend your entire career communicating to and within this community and go very far but there are certain kinds of things and certain kinds of impacts that require you to speak outside this community and to make your work accessible and approachable to a larger set of people because even a lot of people who aren't professional information security folks, rational privacy folks have an interest in that aren't dumb and are intellectually curious and are willing to apply and integrate that stuff into their work. So it's something to remember too that even though you may not ever have to you may never be confronted in your life with the time when you have to communicate with your community, there are certain kinds of work that you really should do that. So I'm also a loner from academia I'm on loan to the FCC from Stanford. Go Bears! We can't all go to school at a country club so I'm at a different stage in my career from Laurie of course. I hope to be faculty in the not too distant future but I'm just rotating out from grad school and so I want to note their opportunity is absolutely at that stage of your career coming out of academia. If you don't know what you want to do next you're going to take a little gap between what you're doing in academia and whatever comes next. The government has great roles there there are a bunch of great opportunities straight out of college, straight out of grad school there are programs to support that more programs are coming online all the time. They're also wonderful internship and fellowship opportunities to explore. Even with a six month or one year stint in government you can have a tremendous amount of impact. Or a summer internship which we actually have three summer interns at the FTC in technology roles this summer. I really want to emphasize Eric's point about communicating with folks in government. I think having worked on both sides and I guess I should come clean I'm also a lawyer. The way in which folks communicate in the hacker community is very different from the way folks communicate in government. And better for worse but learning how to sort of speak Washington is really really important. That's something you can learn in advance of coming to the government and it's a great skill set you can pick up if you spend some time inside the government. Yeah over on the left. Great thanks Dan Tynan for The Guardian. I have a question for all the panel members and it's kind of a general one. There's been a lot of speculation lately given the hacks for the DNC and Hillary Clinton's campaign that the actual election could be hacked. In particular by a certain nation state whose name begins with R. So I'm going to ask you to rate on a scale of one being not a big deal ten being holy shit. How worried you are about this happening and if so what worries you most? So remember when I was talking about learning how to speak Washington is? No comment? Yeah it's not I mean I don't think it's not really any of our certainly not my area of expertise here. So I'm going to use this as a pivot which is the other aspect of engaging with policy is to know when to say that's a great question but I don't know. Let's bring in actual experts and fortunately since 2000 there's been a lot of great research on security of electronic voting machines and I don't know the obvious people here but there are a couple of great professors out there. The other lesson I would take away that's highly policy relevant is if you really are interested in this go and volunteer for your local elections board. You will be the only person there under 70. The 70 year olds are wonderful and it is a great way to learn how complex technology and the bureaucracy and the ideal high level goals of democracy all work together. So if you are interested in understanding the security of the election system get some on the ground experience while you're hacking your election device as well. I've been an election judge in Pittsburgh for the past 10 years and it's a really interesting and eye-opening experience I definitely recommend that. First off thank you all for coming here today I can't imagine it's exciting to be told you're going to be at DEF CON and it's representing the feds but thank you for coming. Appreciate that. And that being said I did two questions mainly for the FTC. Where do you see the breach insurance industry going and do you see that's going to drive private sector up in their cybersecurity game because we know legislation ain't going to do it and is that a growing stagnating industry? So that's my first question. The second question is you said a minute ago you want smart and passionate people but the government culture tends to bring out a least performance necessary attitude. Is there anything being done at the executive level to change that culture? Yeah, so on breach insurance once again I would say that I'm not an expert in breach insurance and I'm not really sure. On the issue of getting smart people to want to come to government I think that the administration has made a number of pronouncements about wanting to do this saying that you can wear t-shirts and jeans to work it's a good start but that's not enough just along those lines. I think that within our agency we're an agency that's mostly attorneys and it's set up to work the way attorneys work and as we are hiring more technical people we're saying wait we may need to do things a little bit differently for our technical folks so that this becomes the kind of place that they want to work and where they can thrive and I think the leadership is very much open to that. I want to add something on the culture change. So 18F is a new office we're about two years old in the GSA and one of our missions there and the rest of the government is to work on that cultural problem to attract people to government and also to make it a great place to work for people. I'm actually I really enjoy my job at GSA it's actually the nicest, most humane place I've ever worked in terms of remote work, in terms of having being in the cloud for email and docs and calendar for having really nice people to work around me to have computers to deploy things to etc and that is that's a really valuable thing. There's something that's really dangerous though that I know we have encountered and I have encountered is that it's very tempting to talk about culture change as people change and to talk about problems that you perceive in the government as problems with the people but it's really not the case and the government turns out to be filled with a lot of really smart, well-meaning people in some really terrible incentive structures with a lot of fear that drives executive level decisions like fear of being criticized, fear of being punished, fear of being hauled in front of whoever and that is it's that thing that you have to attack through transparency, through a little bit of courage through changing incentive structures as necessary to re-interpreting or rewiring rules around hiring, all those things and yes those things are all being worked on at the executive level and at the rank and file level in different ways it is just a big problem. The US government is the largest organization in the history of mankind and it's very decentralized but it is being worked on all over. The only thing I would add to this is somebody who's quite new to government is some advice that was given to me when I was first approached is your first boss is really helpful and I'm lucky and I think many of us are lucky to have fantastic supervisors who recognize that doing Meet the Feds and something like that is really important to the missions of the policy that we're trying to change and so if you are contemplating joining government think a lot about your supervisor and what that relationship is going to look like because a great supervisor just makes your job a lot more fun. Hey, my question is what kind of metrics or data points do you guys capture to make sure that your organization is safe or secure on the right track? Sorry, could you repeat the question or Jonathan maybe? Yeah, so what kind of metrics or data points do you guys capture to make sure that your organization is safe or let you guys are on the right track? Yeah, sure, so it varies. In terms of monitoring your own systems, people use all sorts of different scanning tools, people use all sorts of different metrics at the kind of costs that are incurred on those systems. I know that one of the things I work on is measuring encryption presence and quality around the government and around 18Fs and GSA's systems especially using all the same tools that you all probably use, things that are based in libcurl, things that are based in SSLIs, we use data from Z-Map scans of the internet, we're running in UNIX-based environments and doing that same sort of work and so we use the same tools that you all do and use that to improve our work. So I believe we're getting the signal from the goons that it's time to wrap up. Thank you all for your questions. We're going to stick around for a few minutes to allow additional questions outside if you're not inside. Please go on that exit door on that side of the room. Thanks again. Thank you.