 I think that aspect you're mentioning there about measurement of what works is so critical and it's hard You know the federal level it may be more monolithic, but you look at states and cities you can say oh wow This city is doing something that's very different from That city and it's working You know Cory Booker was on stage with me at the personal democracy forum last week And you know he was just talking about reducing crime rates in Newark and you know their metric They actually have a metric they're trying to achieve and you know it was very similar in a way to the way You know web developers think about you know some metric they're trying to drive on their website I guess I just want to say even though it's an imperfect analogy event To between laws and programs I think there's a lot of fruitful work that we can do in in working that analogy You know I mean as I just started with a riff on you know Carl's idea that law is the source code of our democracy But this idea that we can in fact start to test and measure the outcomes And I think some of the work that Vivek has been doing is starting to show us how we actually are producing the data that creates The potential for feedback loops about we we did this program and it's working We did this other program and it's not working and the back at least in the it part of the federal government is Starting to close those loops and to kill things that aren't working And I think you know that's part of of where we can take this analogy and Said that the community of people who will study the law the community of people that will implement the law I think it needs to become Sort of it needs to move into the 21st century, you know with with you know with some of these practices of Open access and then as a feedback loops about measuring outcomes just to sorry just to add to the The ability to actually shine light to give you an example I remember when I first came on the job I got handed a PDF document and I was told welcome to your new job and here are 27 billion dollars worth of IT projects that are way behind schedule an over budget out of $76 billion worth of IT projects. I mean that's what I decided. There's no way you could manage By PDFs that are generated on an annual basis. So the analogy I use is very much like a Precision-guided missile where you're getting feedback loops every millisecond in terms of where you are in relation to your target In terms of time and space in the same way with a lot of these IT projects as soon as we started shining light And we made sure that every CIO was telling us every 30 days Where they were in terms of cost schedule and their rating as a result of that We've already seen results as far as at the VA for example We halted 45 IT projects Terminated 12 of the ones that were not performing. We've done a number of tech set sessions where we've seen For example at the SBA. They were spending $1,614 per card. This is your identification card the smart cards were the same exact card You could get a GSA for $240 and What we learned was a lot of people just don't know where they are in relation to the rest of the government or other entities and the ability to really shine light and focus on performance focus on being relentless about Evaluating how we're doing has a huge impact on implementation and The ability to actually think through what is a new legislative process or management processes that we need to implement with Security for example we inert lots of issues and that's why we're engaging with Congress on Rethinking the legislative process So actually I thought this on you know, there we go I actually thought you were going to say that when you took on the job They handed you your head as opposed to the PDF I'm now I'm levitating a little bit because I'm thinking about programs as analogies to The the law and thinking how buggy programs are and how I'm thinking how buggy law must be If the analogy holds up and I suspect it does Do you think that it can be made politically fashionable to have more openness and accountability I think certainly the president has used that Tool in that that sense very successfully so far But it seems to me that success in this domain is going to rely on the public deciding That the political process demands this kind of visibility and that the elected officials can't get elected Unless they subscribe to this notion of open and very visible process So that's an important thing an outcome that I think we need otherwise this idea may not hold up well The other question a very simple technical one. I've noticed that companies are being asked by the SEC To produce their financial information in a particular form called XBRL It's another markup language It's conceivable that if we adopt practices along those lines that more and more information will become machinably visible And it could have two very valuable outcomes. One of them is merely openness and visibility But especially with regard to the risks that people take when they invest in public companies But the other thing is that we may actually be able to use the information in ways We never could use it before because we didn't have the ability to use software to analyze what that's telling us So I'm actually analogy. There is a really good one You know with what Google has done with the Google book scanning project. I know it's been somewhat controversial But there's some fundamental point that was made is that this isn't necessarily about Humans reading all those books. It's about what can you do when computers can read all the books? So, you know, what kind of textual analysis can we do? What kinds of programmatic Comparisons and I think there's a similar potential when all the law is online, you know We actually, you know, you know what we have seen is this explosion of complexity, which makes it impossible I think for the average citizen to understand the laws that govern them and we need We either need radical simplification or lacking that we need some power tools to help us, you know Understand the impact of the laws that we produce and and so I kind of what I'm imagining here is this is the beginning of A new kind of legal science, you know, where we can in fact Study the law using the powerful tools that are at our disposal where we can look and we can discover the conflicts, you know by Scanning and digesting all those laws and saying well this one over here conflicts with that one What are we going to do and we can start to surface those issues we can We can you know build simulations of what are the outcomes, you know There's I think there's a lot of things downstream that would come out of this effort that in fact this also relates to the idea that You know if we make all this free aren't we taking away the economic opportunity of these companies that publish this stuff now? And I think well no actually there's much bigger economic opportunity in creating new kinds of value-added tools for You know studying the law Can I yeah, please yeah, we should really make this interactive with everyone No, no, you guys fascinating absolutely I just wanted to comment on two things one is on the research applications and I think one of the most compelling Cases out there is that legal research today is unable to access the corpus And at a couple of the Laudak of workshops in Colorado and Texas We had some guys from Michigan come down that our doctoral students and lawyers and they demonstrated a class of Applications that have not been possible before like Deciding which legal cases are important based on citation analysis And it was fascinating because they showed us a visualization of Marbury versus Madison over time And it wasn't very important and then about 50 years later It started to get cited and cited and cited and it became one of the landmark cases now with Marbury versus Madison We know that sort of intuitively but with some of the appellate decisions. Maybe we don't There's other classes of Research that can't be done in the law school such as analysis of patent litigation to see whether it varies across districts So that is one of the most compelling use cases I think of making the corpus more broadly available is is to begin analyzing that let me make one more other brief Comment though and that's on this bugs in the legal code issue The vast majority of the law are not Supreme Court decisions. They're not court cases They're not murder cases The vast majority of the law are things like the standard for the amount of lead you can have in paint or What is a toxic chemical or how do you define whether a school bus is driving safely and I think some of those Regulations and technical standards are much more amenable to this this debugging process that That Tim has envisioned so that that's my interjection and I'll let you get back Actually, it's a very it's a very interesting interjection the first Immediate reaction I have is that when you have the corpus in machinable form This still leaves you with the semantic problem of representing what's in that corpus And this is something we're still struggling with if you look at Google Bing and the other search engines They are mostly matching text against text without any real understanding of what any of the words mean Tim Berners-Lee has been struggling with semantic web ideas for about a decade now, and it's still hard I wanted to go back to jump in real quickly before you do that. Yeah, but we also have amazing breakthroughs There's spotty to be sure but where we find that there is some hidden semantic and so for example location great great Example we are now able to surface all kinds of data around Locations and our phones are now able to report our location and therefore we can make all these things relevant And I think in a similar way we don't have to understand everything to be able to understand You know some key elements and so it may be that for example, we're able to identify an outcome You know whether it's you know reduced crime rate Let's say we have laws around incarceration or criminalization of this activity versus that activity we can actually go and say well Let's look at the presence of these laws in these 50 states and then looks let's look at the outcomes and the meaning can come Sometimes work back from the outcome No disagreement with that at all it when you where you have a an organizing principle Like location or time You can apply it in very very general ways to the data you have I'd be a little concerned about correlation versus causation Sorry, you should get back to the original point I but I do want to pick up on something that Vivek mentioned and be very specific about it You mentioned wine is a subject about which I'm quite interested because I have a modest wine cellar Which is intended to be consumed As opposed to admired right so the point I want to make is that I tried to ship some wine to a friend of mine in Chicago Earlier in the year and I made the mistake of going on the web to find out what the rules were For movement of this product from one state to another and I discovered an entire matrix It's a pair wise thing and the states have different rules Paralyze rules so it's an n-squared problem So I was going to suggest to you that if we were looking for a Particular poster child problem space to deal with it would be Wineshipments from state to state and if we figured out a way to simplify that we would have done a really great thing It would be a work thing a worked example of simplification So people would be able to share this wonderful product I think we ought to hear from people sitting around the table who've been thinking about this stuff Mike We have microphones. This is the controlled substance so to speak This gentleman had his hand up right there. Yes Well, just to carry your theme I'm paul ver kyle from the administrative conference And I'm gonna be summing up a little in a little while Which is gonna be an impossible task, but I think these are brilliant presentations and for someone who is a new government Official and trying to restart an agency who's concerned about this. It's it's quite gratifying to hear from you With respect to law and how you can determine whether a law is effective or whether you can debug it You do have to use your wine example the 21st amendment. We decided long, you know 1918-19 that we were going to debug the problem by eliminating it Well, that was not very successful and it has taken us and then we changed our mind, you know after we after the the ineffectiveness of the law became so obvious Where we created a whole criminal enterprise? We decided we would went on do it and then we're back and it only is last year or two years ago in the court 5-4 they decided you could send your wine To another state and the state laws couldn't prevent you from doing so so I don't think it's so complicated Wow, that's just me but it's another problem with law just to make the point the law isn't only about what's on the books, right? It's about enforcement and it's about a discretion in enforcement and so that is why it's so hard to analyze It's not about getting everything on the you know It'd be nice if we could have all every law and we could look at them and find out flaws But that only begins opens the discussion the real discussion the discussion we have to have in policy making You know in government is how do you enforce the laws and who does? There's a beautiful example of a case in the Civil Rights Commission where there is in Louisiana a justice of the peace who refuses to marry mixed-race couples and He was told that in 1967 the Supreme Court decided that was Impermissible and he's still doing it this gentleman needs a microphone John stolen the mic. So there he is It's like I'm round. I'll pretend to be Ralph Reagan for the moment So I have a couple of thoughts from really great presentation One is that I think some of the complexity in the law Is a reflection of the fact that in governmental systems There are values being expressed that are not just about efficiency then raise the SEC as a as a as an example of having the corpus in machine readable form in terms of of the of the Filings that are being taken for public companies, but the SEC now right now is struggling with flash trading Which is a different sort of problem of corpus in machine readable form, which used to be I think in in in sort of in in the pre ability to really analyze Data in in in virtual micro seconds used to be thought of but perhaps as trading in front of your clients now Enterprises are creating vass numbers of trades and and wealth to their to their own companies To their own trading systems by do by executing flash trading programs But I think to some extent there's a systemic risk that comes from that and to a different extent there's a sort of a Latent unfairness to to average Traders in a system that permits the ability to to Capture those rents by being analyzed that that analyze that corpus so in government. I think a lot of times the complexity is added by by a set of values that aren't just aimed at New product or efficiency, and I think we need to kind of introduce that into into the conversation and The other Observation I think I would make is that raised the issue of sort of doing what's what works in the internal government systems To try to enhance the productivity of the enterprise of government itself, but ultimately I think the real test Is for the clients receipt or the delivery of government service, which I think is even a more challenging Problem or more challenging question it lends itself. I think to what Tim was talking about If you imagine the education reform movement and the use of data, etc. It lends itself to trying to debug What's working and what's not working, but it's a it's a harder problem I think when you're when you're thinking about the Delivery of government service at the consumer side than in terms of in using better business practice Practices on the inside of government to enhance and make more efficient The way the government is actually working on a day-to-day basis And and I think Vivek you and and the crew over there are really I think operating at both Levels, and I wonder whether you share the thought that one's easier or more difficult or what your experience is on Actually trying to see whether The delivery on the consumer level side is being enhanced So so I think If we look at healthcare for example one of the things we realized what was this principle of Shining light on the performance was What's game-changing in terms of allowing policymakers managers to make different decisions? What was interesting is as we're looking at the healthcare system one of the challenges we found was that there was more Transparency around actually buying a camera online and the ability to compare aperture size or cars online and the ability to compare Speed zero to sixty then there was one hospital to another one doctor to the other And so we're looking at a body of work on how how can we rethink? CMS Literally the concept of dashboard applied to that model or what happened with a community health data initiative That we're working on at HHS where literally we just put the data out there and third parties Started taking that data and finding intersection And I'm a true believer that intersection lies at the value of whether it's multiple Disciplines or data set so they literally took the geodata Combined it with funding data and layered on top of it outcomes data and suddenly this map emerged around poverty levels in Texas and Which county how much money we were spending? What were the policy decisions that were being made? I agree with you. That's it's a harder issue to solve But I think the only way we're going to be able to solve it is by unlocking a lot of this public data And I also think there's some really interesting Advances that have been made geolocation being one of those because it's very important It grounds you the other advance that needs to be made is around this common architecture on funding where are we spending money as a society and Laws play obviously the role in terms of how money is allocated to solve some of the toughest problems We face from public safety to education to health care But but it is a more difficult problem because we don't necessarily always own all the information that we need And we don't have what I call Transparency at the atomic level which is at the point of delivery. How do we get that? Atomic data to be able to then make general Observations whether a program is working or not and where we should be doubling down versus where should we be divesting? Could I know you have a hand up so we need to get the microphone down there? I wanted to make one small observation a lot of the phrase transparency. I think is often interpreted as How to expose those buggers who make bad decisions or do nefarious things? There's another utility and transparency that showed up in the internet related to something that Tim said And I think Vivek also and that's where when you share information You allow others to help you evaluate analyze it and create tools for using it So there is a very positive potential That arises out of making information available and that is that third parties can't help Evaluate it better. So there's a collaborative element in all of this Which has been fundamental to the internet's evolution the millions of people who do things? To make information available or make it more easily discoverable or more easily used is what has allowed the internet to become What it what it is today from its very humble origins? You have a question? Yeah, hi Rob Schecter As far on the issue of now that we have these Statutes and codes machine readable, you know, what can really happen besides just presenting it nicely Something that I've been really interested in implementing and working with is the Library of Congress subject headings which people are beginning to discover and think about new purposes for and This is just a great it's online in a couple different machine forms XML and other And so basically most people see it as a really interesting controlled vocabulary How do you describe law? What are the various subtopics and subcategories and for me? I find it really interesting because every state Oregon we break up the laws and we talk about them in different ways and It's every state has done that in New York. They have something called mental hygiene law I don't maybe a New York lawyer can explain that one to me Yeah, is that right we don't have that in Oregon so Yeah, so so what I imagine is and this wouldn't even be very hard like an afternoon project per state like a per like software engineering terms a per state adapter Translate mental hygiene in New York into the Library of Congress term and then if that existed for each state all of a sudden You have cross-state Searching and knowledge really and there and people have already made I believe this kind of adapter for France and for Canada Canadian law as well into Library of Congress subject headings our Library of Congress for these other countries or I believe these other countries have actually adopted them Which is interesting And I had one other as I had a concrete Example of debugging the law which yeah can be hard and there's there's many subtle problems And I think this will become more common is our sorts of tools become more common on the web one issue in law is Civil penalties that sometimes accrue to someone not just criminal penalties when they've done something bad if you've committed been Can if you've been committed for you've found guilty of robbery? maybe your landlord is allowed to kick you out now, but that's like a civil sort of a law and Sometimes due process concerns can come up because the burden of proof is different well Once you have a collection of statutes and laws and each one knows into what sort of category it is you can begin doing searches and comparing civil section of penalties versus criminal penalties And or white collar versus other kinds of crimes And I found that the this data is already there in the sense that the legislature for a hundred years in Oregon at least they've created this wonderful index with terms like if you want to search for dog You'll say ck9 and to me this is a tag cloud They've been working on a tag cloud for a hundred years, but they didn't consider it like that It's this awesome ontology and so this lets nice intelligent queries be done You know how many links in and out are there to the to civil penalties versus criminal, etc. You know this whole Direction you're talking about makes me think a little bit of what we see in the private sector with things like Tax you know where turbo tax Let's you know an ordinary person File their taxes more efficiently because somebody has done the job of understanding The laws and regulations and turning it into an application And obviously tax professionals still you know work the more complex cases But you've actually automated some of the simpler cases and I do wonder if there's some real interesting applications You know once we start to be able to look at all these laws, you know, can you actually build? better Interfaces for citizens to actually do things that right now are very burdensome, you know getting permits You know, why do they take you know two months six months, you know Because there's a long complicated manual process and it may well be that there is an automated process where you go Okay, we have understood all the the things that we have to do here And can we now build a better interface to all the different places? I know there's one group for example that's working on this in the In the social sector, you know, you're eligible for 40 different poverty reduction programs And you have to apply 40 different times nobody does and they're trying to you know The project called intake one where you know, you basically fill out one form and it figures out how to navigate the You know the the various agencies to get you the benefits that you're entitled to and I think it's sort of a whole set of applications I think that start to fall out of thinking about this as a System that you can start to engage with Via computing as well as as via, you know people sitting down and just reading it There was a series of presentations on access to justice particularly at the Chicago Kent Law Dock of workshop and that's one of the things that they were most interested in you know The vast majority of Legal proceedings are not done by lawyers. They're done by individuals trying to get into the system I believe in in Florida more than 50% of divorces are not involving lawyers or their pro se and the access to justice people are Very interested in making the law available and then simplifying the process of letting people deal with the legal system And that's one reason and I think people like professor tribe We're here this morning and a lot of the access to justice people are very Interested I hate to cut this off because again we could go all day, but I think we're ready for our final panel at this point I want to thank you all very much. This is really fascinating. Thank you very much Out of the lawyers in Florida would be involved in divorce cases of their own