 I used to be CIO of the Dutch Ministry of Health. Now I got worse. I'm deputy general for digitalization of the Dutch government. So I'm paid to look out of windows during office hours. I'll do this together with Browno and a lot of us at the first row. It's about how we combined looking out of windows with hackers mentality to fight the pandemic. And this is his laptop. So this is Browno. Browno, please introduce yourself. Hi, I'm Browno. I'm a hacker. I use Linux Mint. He is a government worker. He's boring. He uses Linux Arch. Sorry. And they develop software for Debian, which I don't have. Fighting infectious diseases is ages old. And we still do it the same way. We identify people who got sick. We trace people who were in contact and isolate them. That's it. It was so for the plague. It was so for the Spanish flu. And it's the same when fighting the current pandemic. But in the old days, you knew everyone you met. You didn't forget them. Nowadays, you meet a lot of people. And most of them you don't know. And if you're as old as I am, the rest you don't remember. So the National Institute for Public Health in the Netherlands as well as abroad said, please do two things, try two things using digital means. One is, please warn in context that you don't remember because you're old. And secondly, see if you can help contact tracing experts in doing their work. So what we did, this kind of stupid, I had to stay at home. And I was watching television to a press conference on April 6, 2020. And there was this minister of health who was telling this story about cell phones knowing exactly how far they were apart with Bluetooth. And that was one of those fine moments that I used Twitter to the max. And it became even funnier because he started an epithon. A what? An epithon. What the fuck is that? Yeah, what we did, we had a weekend. We prepared that in two weeks. And over the weekend, we tested seven apps available on the market. Two of those, I don't say that often, were rolled out in countries. Two separate European Union member states. But after the weekend, seven apps contested. After the weekend, we concluded none of them was good enough. So I made a lot of fun about that. And I called it to say the hoogmiss of the techno-optimism. The high mass of the techno-optimism. So Ron let me know. Well, I called him and I called some others that were, to say the least, a bit skeptical on the internet on social media. And I said, if you know best, join me. And he did. Ho, ho, ho. There are some non-negotiables here. First, privacy is secured. Second one, security is non-negotiable. And third, if you build something, it should be available to everybody. And then Ron said, of course, OK. And then there were a whole bunch of people. What we did, instead of doing it with people that stare out of windows, sorry, we did it also with people from the outside. So I had the honor to work with a team that would never work with me, that would always work against me, amongst which are large names. And I said, well, you may say that I should take privacy seriously. And I should take security seriously. I should take accessibility seriously. But those are my failures too. So we aimed for the highest. And now this shit. Let's see. Oh. Please work, yeah. This is one of our blind testers. We had blind people working on the team. We had blind people testing everything we did on a daily or weekly basis. This was at the same speed. She read the screen, which is a lot quicker than we would read it. She just heard it. So everything we did, we tested to the max. And yes, it is fully accessible what we develop. We were the first government website with no cookies at all. Yeah, that's actually, oh, that was actually kind of a thing because normally you have to have cookies on the government website. And we promised not to track anybody. So what we said is if you do cookies, then basically you're tracking people. Let's not do that. It's your part. Yeah, that's true. So let's not do that. So then people called me working against government standards. So I went like, yes, I know. And that was basically the end of the discussion. We did a lot of other things. Some were basically part of the framework that we used. But for instance, one of the things that we did, we made it a crime if you try to force somebody to use this app. And only a few countries that did so. You have to exchange keys. Well, if you're sorted by alphabet, you basically have a way that you can't trace where these keys are coming from. We made sure that the municipal health services couldn't really see anything about if somebody was really uploading. So it's OK that if you're uploading keys or saying, yes, I am uploading keys, it's OK to lie so that you remain anonymous. And we had a lot of privacy features built in that basically helped out a big time. One of the fun things was as well, and we underestimated that one a little bit, we didn't make any backups because backups is data that you keep on, for instance, Google drives and that type of stuff. You don't want that. So then some people lost their phones and went like, yeah, I've lost my keys as well. Yep, that's true. So we can't recover that. Nope, you can't. So that was one of the things. And if there's abuse, we won't allow me to have a team to go after people that tried to be funny and say like, OK, let's see if we can trace people. Then we also had to assess, OK, what's the risk really and who are we fighting against? And basically, there were some people within the government that said like, you're fighting state actors. Good luck with that. And very vocal people, I can basically make it very easy to say, everybody hates us. And that's what you have to secure against. The next one, thank you. Then we did something else. One of the things to gain trust is to show that things are OK. How can you show things are OK by showing the researchers that have been done in that? So all the pentests, code reviews, et cetera, have been sent to parliament and made publicly available. That was really the first time, I believe, that it ever happened within the Dutch government. And proactively, not wait for a FOIA request, do it proactively. The verification, that is honestly something that I didn't do, but Dirk Willem van Gullijk, who is here somewhere as well, thought of. But if you publish code, how do I prove that that code is really the same software that is planted on your cell phone? So we made the process to certify that by notary. So that, well, I think most of you wouldn't trust the government fully. So the only thing to do that is, how can I show that the GitHub code is the same code as in the Apple App Store? Having a notary proof that compiling that code leads to the app that's in the App Store. Well, opening that all up basically helps you to prove that you're really up for good. And one of the funny things was when we passed the law in parliament, privacy and security were not a concern anymore, because they had all the documentations there. And we were transparent to the max. How many people were contacting the public health institutes because they were warrant? We published that every day, daily, on a daily basis. All the scientific research into does it work or not? We published every single research report we got. So everything we did was not only open by code, open by process, but also open by everything we got from research or reports. So we know who used, used. We stopped it. We stopped the app because it wasn't needed anymore. People didn't think we would. We stopped it, but we knew who were using it. And when we stopped it, we hired auditors to verify that we really stopped the app. And not only the code is open, the designs are open, the documentation is open, the UX research is open, the translations are open, and made by the public also or improved by the public. And all discussions were open. So press had to get used to it. We published the first code, and the morning papers opened with first app version published, which was not true. It was just the first code. The first commit. It was the first commit. Press had to get used to doing that. We had designs for the icons. And Yellow Prince, one of the main designers for Uber, for example, did that and did that on Twitter. One of the things that I believe in for years is that making mistakes is cool. Because if things go wrong, you can do better next time and learn. But you can also, by going by mistakes, assess risks easier. It's far easier to say what can go wrong than to say what is the risk. So I introduced the fill your mode effect analysis as risk assessment methodology. And now, like two years later, people, before they call me, they say automatically, yeah, we should do an FMEA. So they already know that, but whatever question they are going to ask if it's a bit complex, you start to think on how you do things wrong. And it helps, oh, one last thing. It helps also in making design more robust. And of course, you don't punish people. You're just being proud. That's one good thing of wrong, by the way. Just one. Yeah, that's only one. Just one. Is that whenever something went wrong, one would say, OK, this is my mistake. I own it. Instead of go like, how could you do this wrong? And that gives, of course, the confidence that it works. Then I was lucky. I was just there. And I saw on the news that one of the subsidiaries of the ministry basically had a very basic issue in their website, URL manipulation. So why wasn't this in the pentest? So I wrote around an email, hey, Ron, don't you people do pentests in this building? So yes, of course they did. And then I looked at it. And basically didn't contain the test they did. Most pentests are basically a list of things that the researcher thinks is not OK. So when we started with procurement for pentests, I realized I need to be able to follow the entire process. So what we did is basically have standards. If you find something, I want to understand how serious it is not by a thermometer or whatever, no, by a standard. And if we're talking about testing, let at first at least do the OWAS top 10 on the WSTG or the MSSTG. Because then at least I know which tests have been done. Yes, it's OK to do more, but that. And then, of course, we wrote the report will be public. So we asked 10 companies to help us. Six of them said like, oh, no, with public reports, we don't. So that was kind of interesting. And recently, we adopted the policy that our internal pentesters do a pentest and an external auditor will verify the pentest by the documents alone. So they need to be written in such a way that the auditor can understand each and every test that has been done. And you see a car here. If you have an old car, you have to have it inspected once a year or else you're not allowed to drive it in the Netherlands. That's basically what we do. And then some companies go like, no, no, no, pentests are magic. No, they're not. Most issues are with the basics. And then the vaccinations came. Liberation had lost. Erron, how do they register this vaccination thing to the EHR so that the National Institute of Public Health knows about the vaccination? And then we discussed eight weeks before the first vaccination round started that around 15% of doctors don't use a system that can report about vaccinations to the national government after they have been done. So we just had eight weeks, six weeks. No, we had three weeks. It was December 15. So what did we do? Well, we were at lunch. So some people started chatting about it after that question, like, oh, fuck. Then some people started to escalate. Then we stopped the blah. Yeah, then some people escalated. We stopped hiring. And then I said, OK, I'll help you out here. So let's start a project on this. And then the manager came into my office and said like, it might be the case that the Ministry of Defense starts to vaccinate. And then the project needs to have a code name. So I went like, OK, if this is really what you're worried about, meet my cat, Brani. And by now I killed her. I call her Brani-Banani. So the project was called Brani-Banani. Then we went to basically I phoned a mental phone. And we phoned all sorts of hackerspaces and basically said, like, help. So they came to help. And then, are you busy on the project? Yeah, is that more important than COVID? No, then you work for me. And on the other hand, OK, next. So before you come to The Hague, just run into a supermarket and make a picture of a bunch of bananas. So we built a website, brani-banani.nl. This will be the webshop for all your bananas. And behind that was the vaccination registration system. So if you wonder why I have a t-shirt with BRBA, you cannot name a formal official government website, brani-banani.nl. But you can name it brba.nl. And in official documentation to the Dutch parliament, it has a total different acronym. We made up afterwards de beveiligte registratie voor bijzondere assets in Dutch. So if you look back in government files, if you look at the parliament files, if you look back on technical briefings, they'll use brba.nl. So now you know it's about cats. It's Hiro, Brani en Kijkho. En Kijkho, it can complain a lot. So I called her Kijkho de Zijkho as well. And this is how it looks in a government system. Brani, banana, Zijkho, Kijkho, and Hiro. And afterwards what we did, anytime we met, for example, a nurse in Rotterdam that helped us, we asked, do you have a cat? And now every part of the system has a cat name in it from one of the participants in working with us. Some people thought it was a stupid idea, like that manager, and then the minister said, like, you know what, let's stick to the cat names. That's kind of cool. So I don't want to. I got to. So if you now see Red Hat, for example, using animal names, I don't see anything. No, that's it. I stopped there. One of the things that I did with all the projects is use hardware security modules. For those that don't know what is that, that's a cryptography thing. And Ron always says it's a big working prison for keys that will be born there and die there. But the issue with this is that it takes a couple of weeks before you have them, and we had only three weeks. So somebody said, you know what? Use a software HSM, a software hardware security module. No. On Windows. No. So I phoned Mendel and said, OK, I don't know what to do, but think of something. So the next morning, Mendel said, you know what we could do? We could do an HSM on a YubiKey. So I went, oh, yeah, that sounds cool. You should try that. And go do that. And he said, nope, I already did that. So there we were on January 2 in a data center putting YubiKey into a service with a ceremony that Derk Wilhelm von Gulich wrote himself in, I believe, four or five hours doing an official HSM ceremony with everything and made together with Anne-Yan and made a total chaos out of it. But that's a whole different lecture. Oh, yeah. And then, of course, the credentials had been distributed. You could have heard it yesterday already, but we used the army for that. And that was kind of cool because within two hours you started to see on LinkedIn all happy nurses and doctors that all of a sudden could log into the system. Because with hospitals, we've got 118 in the Netherlands. So 118 times two credentials, that's easy, you know, for the defense to distribute. But then I asked Ron, how many other doctors are there that would need the system? And then we were silent again. And then Ron looked at me and was like, OK, I declare this that we are not in control. Let's have lunch. So then the Anne-Yan's and Mendel and thought of things again like, you know what? Let's use this card that doctors have to log in. And then Ron said, can't do that. Well, at that time, I was responsible for that card. It's a card distributed by the ministry to doctors to log in to systems to identify patients and communicate securely. And we didn't have an open source implementation how to interact with that card. We now have. It's the Uzi card and the project has been called Puzi. And yes. Then people started like, OK, we've grown bigger as a team. So you have to have a security team. So you will be audio security. I mean, like, no. So I thought like, what is important in life? Well, cats and Indonesian food. So I like Indonesian food a lot. So we called the team Red Team in Indonesia, which is Tim Abang. So our team is now called Tim Abang. And I had a nice photo of Kaiko the Zyco. And we transformed that into a logo. And then, of course, we did all the marketing and branding that you need to do within the ministry. Almost everything we do now is open source and on GitHub. And we had the first minister worldwide to do a commitment on GitHub. Never done before. How many repos do we have now? Someone? Anyone doesn't even know. Dozens. More than 50 publicly repos. And then, something else happened. And you probably know when you traveled within Europe or had to enter a promise showing a QR code. We were asked to try out what was needed if you would ask people if they were tested negatively or not when entering, for example, a restaurant and in a pilot phase. And that was before the European DCC. We started before. And we said, if we do it the same way we do contact tracing, for example, you don't want people to be linked between two events. So you walk from one restaurant to another. You do not want to be linkable between the two. You do not want people to be able to register that you were there. And you wanted to be accessible too. So if you and I skip something, I go to this. We developed a Dutch QR code. On the left is the European QR code. Within it, all the data itself about vaccination, test, and recovery. The Dutch QR code, which is now stopped, by the way, because it's not needed anymore, has just the first letter of surname and forename and day and month of birth. If you have that code, you are safe. Not knowing who you are. Not knowing why you have the code. Not knowing why you're safe. It's just a proof that you were eligible to enter. So when we had our own QR code, it had no data. Not about why you were safe and not about who you were. And we went at length doing that. We even added to the code we used, for example, that if you were safe because you had had a vaccine, you were safe for about six months. A test is 24 hours. You didn't get more than 24 hours in the QR code. So no one could see if that was because of having been tested or having had a vaccination. That's what you get when working with hackers. And it is fair to say that, especially thanks to Dirk Wilhelm von Gullig, that the European list is smaller than they initially wanted to do. So we developed three apps. The holder app, the scanner app for domestic use, and the scanner app for international use. So we had our own domestic QR. And based on how many people had Q as a first name, you wouldn't show the Q. So when a first name was not used too much in combination with the last name, for example, there wouldn't be a letter. There would be a dash. So we developed the apps. They were accessible, tested, and many languages. No credentials in the QR code. And just first name, last name, day of birth, and month of birth. And I say that again, that's what you get when you work with people who normally work against you. And you don't get a reason why the code is not accepted. You just get a not valid. The portfolio behind this, and Anjan just said, over 50 public repos is huge. Built by just a dozen, just some dozens of people in the past two years. And that's what I have to say, thank you. It has been a wonderful journey. And it's an honor to be working with people like you. People that normally work against me, now they work with me and brought something on a table that you normally would not expect. People would expect hackers bring hackability. They bring privacy. They bring security. And they bring it in a way that most of society wouldn't expect. So thank you. And there is someone here in the room that managed to get a hammock into the ministry. So he deserves his own slide. Thank you, Mendel. We managed to have fun. Lots of fun. My fellow workers at the ministry didn't always recognize what we were doing. But we started a book bound to 1337 on the 10th of January 2021, which is at sub-242. No one discovered that. So it went, it got into parliament and we did it at specifically that time. We started a book bounty for one of our apps. And we had fun. We had lots of fun. And the minister of justice said, this is a team of wizards with the heart at the right place and still have fun. The soap dispensers at the ministry now have stickers stating that they are emptied at every end of the day. And that licking there leads to herd immunity. It is possible to have people like you, and thank you for that, working in a place where you would normally not expect them, and excel and do that together. And from my new job, you all want it. We need people like you doing that. And I don't think Bert is in the room, but the main point I learned, and for whom of you knows the block about the toaster from Bert Hubert? For the others, read it. It's about if you don't have people who know what they're doing, you're not able to drive, change, and have others do what they need to do. It's a block you should read about the toaster of Bert Hubert. So are there any questions? We stole that from people yesterday. So we have some sample questions like, have you considered naming things after dogs? Or why did you use Comic Sans on one slide? Sorry, Jan Jan. Yeah. Hello. Thank you for the excellent talk. I have a small question, and especially for Ron. Did I give you more of a headache in your old job or in your new job? At least that's much fun. For the people not aware, follow him on Twitter, my minister said in parliament that source code of the government can be made public, should be made or can be made public because the law states so, and he just demanded the source code for one of our identity teams. Yeah. I wanted to know if you also will open source all the hidden jokes because all the applications are now done. Can you also open source the jokes that were hidden because I know there were many more. I'd love to do so. I'm not responsible anymore, but you can. The microphone in the back, please. Hello, I'm Thijs. I reported the vulnerability in this app. I tried to email it to you, but the email bounced. Could you get a bit closer to the microphone, please? Oh, we know that. I tried to report the vulnerability in your app, the Corona Check App, but the email bounced, and eventually we reached out to you. And then, Brenno, you promised us tickets for MCH. Yes. We didn't get them yet. Don't want to be impatient. No, no, no, no, no, no, no, no. Yeah, no. Hold on, hold on. Can you frame this as a question, please? Excuse me, I didn't get the question. When can we expect them? Aha. As I have told you and mailed you, probably, but I also told him, let me know where to send them, and I will send them. And you never did. OK, wait, wait, wait. Have a beer. Maybe this is good for a discussion after. There are five tickets that have been handed out to people, so we did do what we promised. I think it's best to come over after the talk. Yeah, yeah. We'll make something up, probably. Yeah, for example. That's nice. Microphone in the front, please. It's not on. Oh, yes, it will come on. Just keep on talking. Yeah. My question is, what assumptions did you have about the other party? It's a question for both of you that got invalidated when working together. OK, for me, the one that comes to mind first is, and how is there a good Dutch-to-English translator in the room? It's the word burger hat. What would that be in English? Yeah, I know, it's not that. The amount of time people put into helping every other citizen having problem with everything we did, I wouldn't expect that. So the amount of time spent at helping people who were not able to get a QR code and not wanting them to be left behind, I was touched. That you would never see around places where I could normally. And then, Brenno, I don't look out of the windows after 5 o'clock, that's it, right? Yeah, what for me was, what's called really invalidated is that I had always an image that people were anti-transparency. And I just noticed that it wasn't really that, but that people find it scary and just need to be helped a little bit. And that's one of the things that you can make a change there. And that people like me use arch and augment, right? Yeah, that was a disappointment. That was a big disappointment, yeah. Thank you. Thanks for everything, especially for doing, I guess, decent job, even if you didn't get all the tickets separately, I don't know. What is the biggest failure that you had, and how would you do it differently next time? What's your biggest failure? Yeah, I know one. At a certain point, we started getting people who were defrauding the system. And at a certain point, we had one testing company that we closed down for a day because we thought they were committing fraud and they weren't. That's, for me, the one mistake where I thought that I should have been sharper on the whole process myself and be more aggressive in how to deal with that. We closed it down and we shouldn't have done that. That was my biggest, that I should have been a lot more aggressive on the processes. And, of course, we changed that. But that was my biggest mistake. For me, it was always balancing parliamentary language and internal language. And I had several occasions where I used too many parliamentary language internally. And numerous where I used non-parliamentary languages where I should have. I won't go into details about that. So does everyone have to be bilingual in government and informatics? Yeah, it's translating every day, every minute. I used to, I've studied computer science. And I always, in my mind, it buckles when one, two languages are not easily translatable. And those two are not. So, hey, thanks for the talk. So, in your view, what needs to change at other ministries for them to adopt this mentality? And how will you do that? Oh, wow. For one, Mendel said, his first question was how many, what was it about having, how are the differences between how I interact with him now? They never heard of Club Mathe. So they hadn't had the right rings. One of my co-workers bought a crate of Club Mathe to have the right discussions with Mendel. It's a culture clash. And I don't know yet how to combine the two. What we sure have to. Need more hackers, please. We need more hackers, yeah. Hey, the Dutch government does a lot of work with big consultancy firms. And they spend a lot of taxpayers' money on it. And as a Dutch taxpayer, I would like to understand, are there any learnings from this process that you're going to take into when the next request for proposal is being sent out for a company to bid to? Yes, the parliament asked for those learnings. And for the Dutch-speaking people amongst you, the bid had bid, or the Advisculation ECT, the Council for ICT, I'm tested. Don't know how to say it in English. Made up with all the. And the CIO of the Dutch government will implement those. And there amongst them are about how to hire the right people, for example. Yeah. And thank you for visiting. Please stand close to the microphone. And you can pull it down if you want. Don't worry. Thanks, Koen. Somewhere in the beginning, you mentioned that you enforce and or approve that the code is actually the same on the app as in the repo. How does it work? That's a very good question. I'm just looking if the Dirk Willem, where are you? Because you invented this. And also, I'm just taking your credits. Dirk Willem. Dirk Willem. Well, he invented it. So, yeah. William, please. Right. Could you keep the microphone close to your? Dirk Willem. To your mouth, not to your stomach. All right. So basically, I think the real proper answer, I think it's probably best to go tomorrow, or I think it's today, to the airmark talk. Because basically, they'll explain a lot more about the technology and the cryptography behind it, which is basically something called EDEMIX, which is essentially something called a zero-knowledge proof where you prove to the other side that there are certain properties that are true at the time of you showing it. And that property can be like, I'm vaccinated, or I'm OK to enter here, or my first initial is a D. My month is February. Oh, sorry. It was about the notary explanation of having this. That one. Oh, sorry about that. All right. I missed that. So one of the big issues we have is that at some point, you've got a chunk of source code. It's in GitHub. It's tagged. And you basically build something which goes on to the app store, and you ultimately fetch it from the app store. And what you really want to know, of course, is that as a citizen, that piece of code which ends up in the app store is really the piece of code you're actually basically you've seen going there in public. And the reason you want that is for many, many reasons, because basically there may be all sorts of threat actors and parties which hopefully can change that, including, of course, people in the government themselves. Even the government themselves could sort of be entrusted and be unfair. So basically what we did there is we did two things. Firstly, we basically used an s-crow service to absolutely document exactly what the source code was which was going into the build. So you really sort of know this is sort of like, whatever went into this build process is basically corresponds to this GipTec and these are the shahashes and everything else. So basically at that point you can reproduce it. And the second thing we did is that we had this observed by a notary. So someone entirely independent outside the government which would basically observe this process and sort of like see through it that this entire process onto the app store basically was essentially sort of like handled as agreed and so on. And then after each of those releases we publish those statements basically sort of like a record by the notary as well as the s-crow statement about what that was in GitHub so basically anyone can verify that. As a hacker I have to say, it's a little bit sad we didn't get the level of doing proper reproducible builds because that would be sort of like the next level up but that was simply not possible with technologies available. And there was this one person who said like, I don't trust notaries. I want to decompile the app and check it myself and I invited him to do so but unfortunately he never came around and I wrote another email like, what can we do to facilitate this? Did sort of like do that internally and if you look at some of the earlier pentests you'll actually see that these pentests started at the sort of like the submission to the app store and sort of like worked back to actually confirm that was in order. Thanks. The main goal here was trust. Hi, thank you and your team for your work and the talk. Thank you also for these example questions and now since there seems to be time anyway why did you use Comic Sans on one slide? Because it's a very interesting one. Yeah, yeah, because I wanted to take the pisset on a young. Just because we could. Yeah. Any last question? There's a long coming up. It's going to be a dangerous one. No, no, my name is Niels from ZZ. Why did you develop OpenCAD and what is it? Come to my talk tomorrow at 8 p.m. Thank you. No, no, no, I love this. I think you have a more extensive answer to that. No, it does. Now it does. I have a teaser. Um, we have extensive testing to do ourselves with numerous testing providers, for example, that provided Corona tests and we had to make sure that all the network was secure. So what we did was build a open source tool when we just open source that on the Ministry's GitHub to do that. And that's CAD and it's OpenCAD. And for people enjoying Annie Allen's graphics, it's FreeCAD.NL. But tomorrow there's a talk about that. Cads, OpenCAD. It's about catacombs, octopus. Cats. Cats. OK, this is now I'm going to end this. This is going nowhere now. Ladies and gentlemen, please a big hand for these guys here. Rond Rosenaal, Renner de Winter.