 Hello everyone, for our last talk of the day We have Anthony Kava Who is a lifelong hacker and works for the sheriff's office in Iowa doing digital forensics Investigating the geekier aspects of crime and fighting with vendors convince them to improve security of their products and One another thing that seems interesting that he has no police powers in Nevada so No problems He he's going to talk about how bad could it be inside law enforcement and local government upset Give him a round of applause, please All right, I think think this is working Good afternoon. Thank you for choosing this or maybe there were no other seats available at other ones. That's cool, too I'll take it either way Let's get you can see all that right good in a step. I probably should have taken earlier We didn't test the audio don't worry about that. It was supposed to be the final countdown. So if anyone could sing that Or at least oh Right, there might be an audio plug. We won't worry. We won't worry about that. It'll be all right probably All right, so you saw in the front slide. You've got my name and my handled Carver So that's kind of a self-doc side, you know who I am now. I've got my picture on here, too I look like this The most important thing I want to stress as we begin this is that I am off-duty at this time. It's actually 1,800 back where I am I'm off-duty. So Yeah, hang on Thanks for the oh, I sit there man. Give me away. All right. So In order to to make it official though, I'm gonna do something real quick. Let's see if we can point this out County 7875 1042. All right. I don't know if you heard that or not. I Had to get a dispatcher to do that so I feel like it have to play it even if I screwed up the audio here, so I'm off-duty. I have effective limits on my freedom of speech So we'll try to stay within those because I happen to love my job and my employer especially since it's recorded Here's the disclaimer These are my views. They don't belong to my employer anyone else. I'm not gonna be as you I will tell you the truth even if it hurts me or anyone else Please don't get me fired or sued That's my request Full disclosure in the interest of full disclosure since it'll be a lot disclosure here This talk or a version of it was rejected by black hat and def con and rom-con. So You know what you're getting This slide is just as a right reminder to myself if you're familiar with the lyrics of Drake It's because I need a picture for our local paper So I don't know you're taking still photos during this or anything if not if you take one and send it to my Send me that on Twitter. I can send it to my paper and then that'll make them happy and they're 500 readers They'll see that recording too I Really like the idea the the theme this year versus last year last year was kind of a downer this year It's technology's promise and I kind of I guess try to imbibe that that theme over the last six months I've listened to nothing but Kraftwerk and You know, I've been to Vegas a few times. I want to remind you stay hydrated. This is my safety slide This is my fourth def con. So we already had the poll question earlier. I went to 13 14 I took 12 years off came here last year and I'm just really thankful to be part of the inaugural app sec village. This is kind of amazing I thought it was a joke when I got the acceptance email So I did check the headers and all that Here's our agenda your experience in the intro right now. We're gonna get into the things you see here. Let's get into them I usually have a lot of detours and tangents in my talks I can't do that because they're gonna flash signs at me and tell me I have to get off So I'll be careful about that Again full disclosure. I'll tell you I work in law enforcement and I'm a hacker. I am a hacker and I carry a badge and So I know that there are some of you who probably subscribe to our gatekeeping that will say that's impossible You can't be a hacker and a cop or you can't be a copy as a hacker and I I say I think you can and we already made it clear. I don't have any police powers here in Nevada. So, you know, do what you need to do The idea for this talk Kind of came to me last year I was at the loft panel I was listening to them and they were talking about ways that we can engage With the government and engage with authorities which is kind of like, you know That's a big difference after missing deaf comfort a dozen years. It felt a little bit different to hear that You know, I've got a quote up here from space rogue and I won't lead it read it to you verbatim but the idea is He was calling you to get involved in the legislative process talk to your lawmakers at the state level at the federal level And try to give them the hacker's perspective on some of these things that they're enacting. I Think that was important work DC 21 and I wasn't there, but I saw it on YouTube But Mudge actually spoke about being a hacker in the federal government and had some experiences there And he quoted this tweet that was sent to him He kind of tweeted asking the Twitterverse about whether I should do this talk or not and I thought this is great I think communication understanding. This is gonna join communities together. You can join hacking and law enforcement together I think it's possible. It's gonna be a heck of a struggle, but I think we can do it For those of you don't know who Mudge is This is a recent photo. I thought that would get more laughs. Did you not see this on Twitter? In any case, he's not actually at Thunderdown ARPA over at the Excalibur I'm gonna give you some reasons to hate me so we can get him out of the way I do work in law enforcement. It's been established. Yeah, this is a bit cyberpunk for this year's theme So I have an alternate version How many of you in here work in law enforcement is there anyone in here? All right you all wonderful This is amazing. How many of you are working you see? No, you cow. Yeah, you cowards You can tell by the cruddy facial hair because they haven't had a lot of practice with it If you do work in law enforcement Or you're somebody who wants who's interested in the dialogue with law enforcement. There's a website hackers.blue go there There's a form you can fill out. I'm trying to get people talking to each other If you're interested in that go for it if you're not ignore And then this slide is because there are literally dozens of us here And so I'm gonna talk a little bit more about that stuff on Sunday at Sky Talks I'm a pearl monger and have been for years and I know that's an old person's language developers in here, right? So none of you I mean everybody knows PCRE, but you don't actually do pearl. I hope I hold some of those certifications that you hate I've actually been on Dateline once it's my claim to fame I guess but Dateline has a weird relationship with DEF CON as you know, I did not try to infiltrate DEF CON I'm here, you know I also didn't realize that the guy interviewing me is the popcorn meme guy So I wasn't star struck I'm not a fed asterisk. I do work federal cases and the coolest part of federal testimony I've ever given was this one where I was on the stand being cross-examined and I the defense attorney started saying Deputy Kava you're an expert in pornography and before he could finish his question I just said thank you because then it goes in the transcript and now it's on my CV So, you know where I'm coming from I'm not gonna get super political here come Sunday But I'm not gonna get super political here, but I am firmly against cryptographic backdoors I would love it if I could dump every phone that I get in the lab, but I don't want to give up or Privacies for that. I also don't think this should be a political statement, but all people deserve To feel safe to be fed to be taken care of so that's where I'm coming from in this I'm trying to convince you. I'm not a jerk, but we'll see I Worked for a county sheriff. So I work for a sheriff's office in a certain state not shown here It's redacted in that state in that state. I'm in part of the I I'm sorry I'm part of the internet crimes against children task force. We'll have to redact that part and So when I go and talk to kids at schools, I tell them that we go after people who go after Bad guys on the internet they go after kids. So we're the good guys Sneakers reference for those of you of a certain age So anyway, I spent in our county's IT department. I spent about 15 years working in the IT department I did some security type stuff there towards the last half of that and then the last eight years of that I moonlighted as a special deputy It's a reserve deputy in our sheriff's office. So for a dollar a year in addition to my day job I did digital forensics and then for the last two years I've done that full time as my day job and then I go out and do some special deputy stuff on the site Yeah, I redacted the county, but you know, even the most cursory internet search. This isn't the SE village. So I Still do a little bit of light IT work Digital forensics though is my day job. I have dumped a lot of Nexus phones. I do not retire replicants. So This is this is all introductory because I feel like it's There is a natural animosity between hacking community law enforcement accused me some community sometimes for good reason And I'm hoping that we can we can get past that because I think that we can be the same community or some of us are thinking that way Anyway, it'll take some winning over of the the others, but We're really not that different you and I we have a lot in common I think first game console was an Atari 2600 I started out with Commodore 64 Apple to my first computer that could actually call my own was a PC junior And if you haven't heard of that, it's a crap. I Ended up with three of them a few years ago And I gave two of them to a local high school and they've it's in their computer lab now And the first thing we hooked up the kids asked, where's the mouse? And And the third one went to Foon. So if you were into retro computing stuff, this is my Friday follow follow Foon on the Twitter Yeah, so this kid spent a lot of time on computers probably wasted this kid This is how bad security was back in the 80s and 90s like late 80s early 90s is that this kid got rude on a lot of stuff Spent a lot of time on bulletin board systems red anarchist text files Aren't they all the Jolly Rogers cookbook was the knockoff Anarchists cookbook and all that I had available to me I ended up on some bulletin boards underground ones the ones that where there's a door game That looks like a door game, but it's actually taking to the real bulletin board until I got busted because I didn't realize how phone traces work and Result in the band for a couple years then I got my first paper out so that I could buy a 486 So I could run cracker jack crack hashes Spent more time doing lamb parties than doing homework, which explains why I barely graduated and all that led to The the ability to do the job that I love today So I think now that we've got the common ground laid out here, and there's only 10 minutes left, right? I Want to give you some of the commonalities some of the general principles in local gov app sec That's my fun name for local government until the somebody actually registered the domain then I'll screw that up Your local governments everybody's got one right you live in a city or a town Or a county somewhere and they vary in size So you've got the small ones, you know population of like a thousand people or 500 something like that You've got big ones at Los Angeles County We've got populations of millions and so the thing is that the smallest organizations They they can get by on Excel spreadsheets and access databases for their information systems sometimes pen and paper even today And the largest ones their true enterprises So they might have something developed in house or they've got some really expensive thing and there's people stuck in The middle my county stuck in the middle. We were a population of 90,000 and so we're too big to get by on Excel spreadsheets Not that you should do that anyway But we're too small to use the really cool enterprise stuff we can't afford it So you end up in this weird you give it this niche software that's made just for these These size organizations and the problem is there's not a lot of incentive on the vendors to look at application security By the way imposter syndrome. I don't actually know what app sec means I'm talking about vulnerabilities and applications. Is that qualify you're the professionals Okay, you're gonna let it go. All right good. Is that because the rest of this will be anyway Midsize governments the applications don't get looked at because they're it's not like it's Microsoft office where there's a thousand thousand There's thousands and thousands of installs out there and there's people actively looking for in this bug bounty programs or whatever else There's none of that people don't look at what's under the hood a lot of these governments Even if they have an IT department, they don't have a security function There's no one there that's looking into security actively and in our state We're lucky our state actually will come in and give you like an IDS and we'll come in and do vulnerability Scanning for you and stuff like that. They'll help you out if you ask but The problem is that the places that are lucky enough to have IT that's great Some places their IT person is like there's an auditors clerk who happens to have a copy of Norton Utilities and so True story and so So that person ends up being your IT person being your go-to person and that person is also at the same time trying to You know run elections and finances and real estate stuff And in their spare time they try to hold off, you know nation-state actors, so It's a losing battle All these organizations need help and need advice with technology because they don't have the internal resources for it and The reality is that a lot of the advice that they do get is gonna come from vendors Anyone here sell stuff for a living? Okay In one way or another man, I didn't this is this this isn't ethics or philosophy But yeah, you're right in one way or another but the vendors that are giving a sales pitch oftentimes recommend a product sometimes It's their own That's where they get most of their advice and that can be kind of a trap So in terms of the technology's promise imagine if citizens were providing that kind of advice just food for thought And and here's another one and I'll answer this I guess for you in a minute here But how much how many vulnerabilities could you find if you were looking for them in your local government software? If you spent like a day or two with it, you know, just looked at the cursory stuff We're gonna find out So I'm not a professional at this and it's not even the major part of my job So if I the things I'll show you if I could do them then you know anybody could find it But I want to go over why this is important You're you paid for it your tax dollars paid for all this software they pay for the services They pay for these things if there is if there is a ransomware Infection or some other problem with a system your tax dollars will also pay for the fix for that They're gonna pay for the ransom. They're gonna pay for the incident response Yeah These are systems that handle things like 911 calls Dispatching firefighters and medics and police to emergencies is a life-and-death stuff They also record the worst days of people's lives I mean this is where a police report goes that's gonna have a narrative that describes, you know bad stuff They haven't people call 911. It's not usually just to say hello. It's because something's going on and If you think if you think about it, there's crime scene photos all sorts of stuff that You wouldn't want it to get out. It doesn't need to be on pay spin And I know that those of you who read those anarchist text files Might consider that blowing all that up would be kind of cool because then you know You can't prosecute people but you know you would throw out the good with the bad and there are people that actually really need prison So anyway, I think this is important stuff to watch Every vendor that you work with in this space and probably all spaces They tell us that we need to have a secure environment and what that is code for is that like hey We didn't do anything with security on this. So you'd better be watching everything for us If they have cryptography, it's going to be zero to bad somewhere in there There's a lot of hard-coded credentials There's a client side enforcement of privileges stuff like that and client side vulnerabilities not considered to be a problem because Well as we know in 2019 all attacks are straight at the firewall towards the internet everything's you know a Direct attack. There's no phishing anything like that The thing is that they tell you that if an attacker got to your client And I've heard this from multiple people that that's your fault you didn't your client got you know Somebody's on there and now they're looking at our software and they find the hole Well, it's your fault for letting them on there. So no reason to secure the application Who's in charge of security? There are some places when we ask they say we can't tell you like it's some sort of secret It's a trade secret. We can't tell you in charge of security. I'll get into that. Yeah and Yeah, we take security seriously. I if I took a drink every time I wouldn't have made it to the talk today Let's talk about the dangers So we've already kind of established these local governments if you've got one IT person If you got one IT person no IT person they can't look under the hood They just don't have the capabilities on a skill set. They don't have the resources to do it even if they want to it's too small Yeah, and a lot of these places are lucky to even have an IT person So we need people that can and we'll look under the hood and I'll give you some particular cases to try to show that I've redacted the names because I love my job and I love not being sued These vulnerabilities though have all been reported one to five years ago. So some of these are pretty ancient They've either been addressed notice. It doesn't say fix says addressed Either addressed sometimes they decided we should accept the risk. They just tell us that But they've either been addressed that they've been retired the products have been retired assume that Yeah, how bad could it be? Oh And by the way, this is what I was getting as I'm not an apsec professional I didn't I had to look it up on Wikipedia to know what it was before this put the CFP in but I Did this in my spare time like I've been I've been for the last like I guess 17 years I've worked on a salary so I get to work as many hours as I want and so after I So so after I've done all the stuff that's supposed to be part of my primary part of my job This is where I was like I'd really like to know what the heck is going on on these things and start to look into them So if you did this as a job, imagine what you'd find I didn't do any real reverse engineering a because I'm not smart enough Be because it might violate our ULAs and I didn't want to do that because I don't want to get sued I love my job and the people of my county Yeah, so the tools they used were really advanced stuff like strings grep some pearl obviously I Spied on some API calls to think count is okay. I used a wire shark and burp sweet and looked at some traffic Didn't do any real reverse engineering stuff So now part three a tale of three vendors notice the triforce Start off with vendor a vendor a Sells a suite of software to local governments and it runs all sorts of stuff. It'll do your finances So you can't payable accounts receivable your direct deposit for banking Human resources stuff payroll your real estate taxes all that stuff's in there And they will sell you law enforcement manager stuff too. So your records and everything else So when we start looking at this the immediate things that pop out are the sequel creds that are baked into the client install So there's an XML. Yeah, there's an XML config file and it has a username and password for sequel The password is not plain text the password is encrypted but to use it. It's got to be reversible encryption, right? And it turns out that the username Well, the password is a leapspeak version of the username So, you know, you could probably pretty well guess it and it has SA on the sequel server So every client that's installed includes a free login to your sequel server with SA And it's a this is the bonus part. It's the same at every customer site I verified this I was a little bit afraid to do it But I slacked somebody at a different county and I said hey just real quick do you think your password is this and they said holy crap That wasn't me that was the computer the The API traffic this thing has a client server infrastructure, which is more than I can say for some of these apps It's plain text or at least it was it was initially and the thing it sucks about that is it also had Authentication tokens when you log in you get a token that never expires or I don't it can't say never but I tried it for weeks So, you know like six weeks later that off the off token still worked. Another nice thing is that there's roles-based Permissions, but they're not enforced at the API level. They're only enforced the client So if you get that token which is sent in the clear you can reuse it for weeks and weeks Maybe years and and do anything you want on the system So I can get into my that system and I can go look at my payroll and see what my paystubs gonna look like But once I get the token I can look at everyone's payroll and maybe pull their bank account numbers That's a problem, but wait, there's more What about their payroll website? So this company also hosts a Payroll website where you can put turn in your timesheet Internet-facing website and there's an API that connects between Your local app server and their website and that API traffic when I went to go look at it Turn out to be in the clear they were doing HTTP and XML that was not encrypted Has all sorts of PII in there and I'll tell you a little bit more about that in a second And it was that way for weeks now what happened was when we first implemented it I did check and it was HTTPS and you know I felt okay about that and then when I went to check back on it to look at the API traffic I I thought I'd have to intercept it. I didn't I mean I thought I'd have to man in the middle But in any case they turned it off for debugging They were debugging one site and they turned it off for every customer as far as I can tell and Forgot to turn it back on And so I brought it to their attention and they did turn it back on Also, there's cross-site scripting because there's like announcements in there I can go put an announcement in for all employees when you log in to do your timesheet and it was vulnerable to cross-site scripting They said those aren't security vulnerabilities Even though it pops up when you log in So I made some requests that this particular vendor asked him Would you please tell customers about that time when their stuff was not encrypted and I think they deserve to know Then I got that's a locus, but it's supposed to be a grasshopper. That was what I heard grasshoppers after that Sorry crickets crickets I also asked them if they would stop using Google Analytics in the inside of this website I don't know. Does this feel weird? Do you I need a professional opinion? But when you log into this payroll thing the internal parts have good Google analytics So they're bringing in JavaScript from an outside site. I know it's Google and you can trust Google but They said no because they use it to decide what pages you're looking at and I thought well Couldn't you parse the logs or do something that didn't require that but anyway, I didn't get any motion on that They did have our social security numbers on there and dates of birth So when you log in to do your timesheet you can also click little buttons as profile And it'll show me my full name my home address my emergency contact my date of birth my social security number my ethnicity my race I mean it's like I don't know what they didn't have on there and We asked could we please at least get the social security number at full social security number By the way, not just last four and the first they said some people like it So no and then eventually they gave us a little checkbox so we could say no we don't want to show So we suppressed that but they didn't let us take off the date of birth and all the other stuff My my argument was that if I don't know my date of birth, you know, that's I don't need I don't go to the website to check it We we brought that that thing about the hard-coded login that's the same everywhere with an essay account We asked them to change that and they did change that they revamped their software updated their client updater And they pushed out a big update to every site that broke some of them and I told them that's my fault but And they changed the passwords at every site for this account and they took away essay The account still exists and still does some stuff that I guess they needed to do They need to re-architect the whole thing to get rid of it But at least it's not essay at least the password is different different sites. So, yeah The payroll website with the time sheets did not support password complexity expiry or lockouts and when they set up they they come out to your place and they set you up They give you four-digit pins and without lockouts, you know If you can guess my first initial last name you can try it 10,000 combinations and you'll get into my payroll account So that's cool, too with all my social security numeralism The other the last thing I asked and this is the biggest ask is put somebody in charge of security, you know and audit your stuff and Believe it or not this story vendor a has a happy ending This is the timeline for this story. I first reported The vulnerabilities back in March of 2014 and I gave them here's what I have so far and I think there's gonna be more and they said, you know, okay, we'll look into it but it doesn't sound very serious and They say why don't you just give us a list of every vulnerability you find Why don't you do a free audit of our software and so I did in a time frame that worked for me, which is about a year And I didn't do that purpose. I really wanted to get it done quicker But about a year so July 2015 when I came back to them said here's a list of like two dozen things then some of them are really bad and They again weren't super excited about it. What I thought was funny I went back to my emails to look at this to see what happened But I sent that they weren't super excited about it and then a month later I started getting automated emails from them telling me to do Microsoft patches as new CVs were released and stuff So it was like I thought they were taunting me and Then a year goes on every few months. I'm pestering them. I'm trying to go up the chain I'm going to LinkedIn and looking up, you know, the boss of the boss of the boss and I'm reaching out to anybody I can and I didn't think anything was really gonna happen at that point We weren't getting a lot of traction Until a month later August 2016. This was great. We found the one person in their organization This is a big company fortune. Whatever one person that cared about the software and the security of it And they weren't even in that side of it They were in the operations IT side running this website and they started looking at all these things And they started actually doing stuff with it and they were able to get an actual audit started a few months later So by December 2016 is still two and a half years after the initial reports They started auditing you know stuff They even found vulnerabilities I couldn't find because they got access to the source code in the logs and everything And then the real happy end into this is that the next year that one person who cared and pull my Paul Harvey voice Was promoted into their first security position Auditing this stuff regularly and so good outcome. It took three years, but good outcome A lot of these things were fixed and they put somebody in charge who actually make sure that they don't happen in the future That's a good story. Let's do some other stories How am I in time? I Think I'm okay. I don't All right, we're gonna speed up. I drank a lot of coffee, so we should be good Vendor B. They do a 911 dispatch system. It's called CAD computer aided dispatch That's where you take your 911 calls you send out police firefighter medics that sort of thing Records management jail management all your inmate information including medical information and mobile CAD stuff and communications So that's in your car. You've got this computer in there usually a pass-off tough book They've cornered the market But anyway that thing interfaces with the CAD system where your dispatchers are I'd say it's 90% serverless way ahead of its time since the 90s because they didn't do a client server architecture All the clients do direct sequel direct sequel and every user has to have a sequel account This was a management nightmare. We already gave them AD accounts Why do we also have to have a sequel user for them local sequel user? But we did and each one of the sequel users gets DBO On all their stuff database owner can delete everything whatever so if you use the GUI It doesn't let you do stuff. You're not supposed to do but if you open up Microsoft access and make an ODBC connection You can do what the hell you want They have one component in this suite that has its own users table It doesn't rely on these sequel users and of course the passwords restore plain text That particular component is a web based records management interface It's for you to go log in they said put it on the internet we did not put it on the internet and They said you have your people log in you can look at reports you can look at in main information all that stuff It's really cool plain text passwords. We've complained about that. They upgraded x-word Trivial sequel injection I mean like single quote or one and you're in to bypass the authentication on this thing Cross-site scripting is not a vulnerability. We know that Bonus ampersands break your system So because they were displaying an HTML if you put an ampersand in a record field it broke the display on the page So I did write a pearl script that every five minutes checked for ampersands and changed them to the word and true story Their mobile CAD software in the cars it goes over a radio network So you want it to be secured right so they said don't worry about VPN. We rolled our own crypto and So we used open VPN on top of theirs Their crypto is based on 64 bit blocks I know that much and there's some questionable padding stuff going on I'll tell you a little bit more of that in the next slide because they also use it for passwords and here's what's interesting about I mean, it's all interesting but to me anyway But the the key that they use for this mobile communication is hard-coded in the install You can't I shouldn't say hard-coded you can change it But most people don't and so we asked to change it And it was a big pain in the neck to hit every client after the deployment to do this But we changed ours and I know they have hundreds of other customers that did not so if you know the password that comes with the client Installer you can read all their traffic So the passwords for the mobile login, they're stored with this reversible encryption not hashed And it's based on 64 bit blocks, which you know eight characters So if you and they give you up to 12 for a password I think they did varkar 12 back in the 90s and they just stuck with it for 20 or 20 years But anyway, if your password is longer than eight characters the last four Characters are in plain text. It's it's padded with spaces, but those are plain text, too So we wanted help we wanted a tool. This is related to this We wanted a tool in our IT department so that our IT people could go to an internet page and reset passwords unlock accounts in the system But we didn't know how the encryption worked. So I went to them and I said well You should know this thing has a console application that stays up all the time We had to lock the screen and use vnc to get to this thing and that's how you set passwords And so we went to them and said hey, could you give us something to help us make this internet feature? We they just updated their password expiry feature and all of our users passwords expired the same day So that first 90-day cycle three shifts is law enforcement usually works 24 hours. It was supposed to Three shifts of lockouts and on-call IT stuff So wanted to give our guys something and guys and girls something to To change those I asked the vendor. Can we get a recipe? Just tell us how does encryption work? You know, what's the cipher mode and give us the parameters we'll do it ourselves or give us a little bit of code I know they're gonna do that but gives a little bit code will do we'll redevelop it or give us a tool command line tool That we can use that would work. They said, okay, we can't do the first two. We can develop the tool We're doing a feasibility study now. We're going to have a project manager assigned Estimate six months development ten thousand dollars give or take probably give So that I got to thinking I looked at and like they have a DLL that does all this stuff, right? So What would it cost us ten thousand dollars or about a thousand dollars per line of Python? We can do it and it doesn't lines of Python and just use their DLL I still don't to this day know how their encryption works But I don't need to because I'll just use their library to do it and you can see the hard-coded key up there With the B. Can you okay? That is the actual key. I I obfuscated some of the other stuff, but that's the actual key Good news is they retired this product. This is a football reference I'm not a sports person, but I understand this guy's retired Vendor see our final vendor does the same kind of stuff and I own dispatch mobile everything We were shopping for vendors and I asked the sales engineer dude I said how are the cred stored and he said we're not Dumb, but he said it a different more demeaning way. I thought was terrible But then I went to look because I thought okay Let's see what they do unsalted MD 5 hashes in 20 something At least they can do active directory right so we don't worry too much about that oops We turned on active directory they said okay active directory is now enabled login with your AD accounts, and I didn't But I did ask our IT department at this point I didn't work in our IT department anymore, so I couldn't do the wire shark You know stuff myself, so I said could you guys look at the LDAP traffic and just make sure that they Encrypted that and sure enough it was it's not they forgot the s In LDAPs so plain text passwords going across our IT department did better disclosure than some of these vendors because I said Hey, could you go look at the logs and see which users have logged in since they turn this on and let them know to change their password, and they did So 80 works and that mitigates some of the problems once you put the s in LDAPs It's at least it's encrypted on the wire But for some reason they started storing our passwords in MD 5 hashes anyway I went and looked at the database after that and I found mine in there and I started cracking some of them And I found something in common they all they're all lower case I'm like what the heck's happening here my passwords mixed case It has to be for the AD rules, but it's stored in this hash. That's all lower case Well turns out they're doing stirred a lower MD 5 no no salt and throwing it in the database This dog looks so sad, so I actually I got a happier dog for this But I also found out that they don't validate the TLS certs So when they do that after dark connection or when you do an API connection they take any cert That's why I can look at their stuff They have this messaging system built in this messaging platform in case you don't want to do email and It's it's rich text if you put a hyperlink in there you put the mouse over it. It does not show you the URL You're about to go to it's blind, so I thought that was kind of dangerous. They took it under advisement 80 ops only done once well any off if you log into the thing So if my accounts locked out or expires or something while I'm logged in as long as I never log out of their software I'm good. I've been logged in for three years now Their updater doesn't need admin rights, that's good And they have a really decent updater actually for their system And but the reason it works is because everyone is given full control on a program files directory for their clients So any you know you get into a shared computer with 20 users any of them could infect your exe and then you run it under your Profile they use rsync. It's kind of cool that they use some open-source stuff But they only check the size and the modify time of the file And also rsync was statically linked Which would upset Stolman, but they fix that Client surprise prize does direct sequel for off this surprise the heck out of me because they didn't need to they have a server component in API Why does it have to do this well as some sort of backwards compatibility for something? Whatever? The the sequel user is rot 13 protected, so we weren't too worried and The client not only does its own authentication. It also makes its own inserts for the audit trail To log your login So if you're a pearl ish or you're in the PCRE as you know, I am you can write a pearl script that Patches a DLL with this regex and this will get you in with any password you want You don't need admin rights because everybody can write to that DLL. I'm a 10. Oh, we're gonna be good You can be question time and everything. I'm gonna slow down You can do this by hand with a hex editor the first time I did it that way or you can use the pearl script You're just changing that sequel query The size of the file has to be the same so pad it out with nulls and the modify time has to be the same or rsync Overwrite it so you can just change the mod time Then you can bypass authentication you can bypass auditing and I have a redacted demo for you It's about a minute and five seconds The music may be difficult to hear But it doesn't add to the things This is when you first this is when you first start the program and it starts updating by my music It starts doing its update. This is rsync running and getting my client up to date except for the DLL It's not going to overwrite that I modified You're right. I probably could have edited this to make it shorter But here we go it takes about half a minute or whatever and then I'm going to get this cool Log imprompt with logos for all of the agencies that use it because it's a multi agency system And my logins there first name first initial last my password is not a and yet I'm in So that's cool, but There is a super user used by the vendor whose passwords also not a And it can do things I can't do with my account and now I'm in as the super user Yeah So here's the that video was I hope you liked it and I know some people that hated it because All these things have been reported, but nobody was doing anything about it Nobody was really interested at the vendor until I sent that video out to them and copied all of our commanders from these various agencies I love this isn't these you're my people and I love being here I don't have to explain what sequel means and stuff like that But to our commanders in that until they saw this, you know that video it was hard to understand all the stupid nerd stuff I was saying Soon thereafter stuff starts getting fixed I did a presentation to our commanders and and the first slide was this There's some other stuff they did they decided not to resolve or whatever I was impressed I reported them that all lowercase md5 hash it was in them the database was also in memory and I said Hey, I can proc dump this process and get that hash out of it and crack it pretty easily They got rid of that that doesn't get in the dump anymore. So pretty cool the downside to this vendor is that I ask who's in charge of security and They've initially they say all the product managers for all these different products. They're in charge of their own security They're QA people do it And then I say well is there one person, you know who's who's a seesaw or whatever They said we cannot tell you like it's a trade secret and I kept bugging I see you got to have somebody who's in charge of this stuff. Is it the president? And then they said we've convened a special committee So after all those vulnerabilities they can meet convened a committee and it's a secret committee because I asked who's on it I cannot tell you I said, okay, how often do they meet can't tell you that can you give me any concrete? Initiatives that they've undertaken this secret committee and I said no trade secrets So this committee is you might have figured out probably doesn't exist But every time I send an email now I say please forward this to the secret committee members. I Also asked them to audit their software. I said if I I'm not in my five five five ish all right, so I Told him I'm not this isn't what I do for a living if I can find this stuff in my spare time What could somebody who I was actually good at this do look at everybody in this room do? Time to conclude because I've got five minutes left and part four is that So anyway niche software doesn't get a lot of eyes on it You could put your eyes on this stuff I think because the thing is that these your local government if they're you know if they don't have the resources internally There's no reason you couldn't reach out to a little elected official and if you don't want to look at this up directly You're afraid to get ensued or whatever Giving that get asking them questions or giving them questions to ask their people and their vendors and that might help out and The leverage that we had we don't we're not a big place We're not a big customer We're one of a lot of customers and we don't put a lot of money in for them So we didn't have a lot of leverage on this vendor, but the bulldogging worked They wanted to ignore it and stop responding to stuff and they went a year that one vendor if you keep on them Eventually, they either have to do something or they get tired of listening to you and they do The magic part of this is the herd immunity because if you can fix these vulnerabilities just like anywhere else You fix the vulnerability at one of these governments. There's a thousand other customers out there running the same software They're gonna get the update fixes it to you fix stuff across the country This is personally if you were interested in this stuff and you were looking for you know I want to run ideas by anybody or you're looking for anybody to help you with an intro or something like that Non-monetary because I work in government my whole life. I don't know how to make money or what to do with it So I can't help you there, but I am I would love to talk to you about it if you're into this sort of thing and There we go. The special thanks to the people that Give me notes or encouragement on CFPs that got rejected and got accepted. So thank you to all them And here's my contact info and thank you for listening