 Okay. So I'm Katrina Hayes. I currently work for a company called Mandova Network Services. We help support ISPs. So we provide the back end services for ISPs that otherwise don't have the same sort of resources as Time Warner and ANT and those companies. So why logging? It gives your code a temporal dimension which as the internet is coming of age we've got like 10 years worth of stuff that's built up. And as you're sort of going through these processes, much like baking, I don't know if you can tell that's a picture of a pie, and it's a pie that's built up by layers. So the idea is that you sort of have this like crumble nut layer on the bottom. But then when you go ahead and add the next layer on top of it, you can't see the nuts anymore. So you're going to get people that are walking through. Now if the pie is originally like, it could be strawberry or rhubarb. It could be a memorandum or it could be turtle. You're going to end up, if the label is just there and it just says pie, nobody's going to know that there's nuts in it until somebody has an allergic reaction. So that's why you start adding logging because if your original system wasn't documented very well, you aren't going to know that there's nuts in there until somebody goes up and crashes. So by logging, if you weren't standing there next to the pie and watched somebody fall over, you weren't going to know that until afterwards. So some general things to this talk. Like I said, as I was looking at it, I was looking at these two particular case studies of bugs. As I was going through it, I realized that both of these types of bugs are particularly in studios because they build up over years. You don't really know that they're initially there and they aren't initially there. The two general types of bugs that have come up. The first one sort of builds out of this process. You start with a very simple system 10 years ago. Over the years, you're sort of adding all these different things and somebody comes in and is like, oh, I'd like to be able to send things by email instead of mail. So you have to go in and add this extra system. And as you add all these extra things, you get different inputs and stuff and nobody actually knows how the entire system works anymore. The other type of bug that I was encountering while we're doing this cooking, each person comes in and is like, I'm going to build my own type of pie. And so you just end up throwing all these recipes at them and forgetting where everything went in the first place. So the first type of bug, that color's on again. I kicked something, I guess. So the first type of bug was exemplified by, this was on a site where you could navigate. You just had some sort of static content and you navigated from page to page. Over the course of the years, different people came in with different requirements. You're sort of like, okay, this particular course needs to navigate in this order and this different person needs to go through these slides in a different order. And so you end up, instead of just saying like, okay, if the person clicks the next button, you go to the next page, you have all these different roles that get in with getting layered in. So what we ended up having to do with this one is that every time someone clicked on those buttons, we had to go in and add a logging step. We had somebody describe it as an open stage of mass visualization, right? So each person that's going through the site now has their own unique place that they're supposed to be going to. But when you end up with, you know, there's a giant pile of data like this, right? Okay, so you sort of add things to your logs as you can. And then you can track people like, you can look at what we were initially looking at. See what we can see? Well, this isn't actually a data set, but if you track different things, you can see like, okay, here's how the person is going through the site. You get the different timestamps and we have the different user that it is. And so if you're going through all of these different things, you can get, it's when they were initially setting up, they weren't going to want to click. You weren't going to log every time someone clicks on it because they're just going to the same page, right? But now that we've got all of this storage space, we can start adding data sets like this. So you can watch the person as they actually go through the site in real time and you know what page they're at now. And so for this bug, it saved it because we were able to see where the person is supposed to be going. So if they got stuck in a little loop that just kept sending me back to the same page or something, we would see that happening to just kick them over to the next page. Now that we have this log with all this information in, we also start to save some other bugs that we didn't even realize were bugs in the first place. So we have the networking timestamp issues that people are calling like, hey, you know, I'm supposed to have, I've been on the site for half an hour, but I didn't always say I've been on your site for 10 minutes. And we can look at it and be like, okay, well that's because one time this time seems to be showing up on your computer through JavaScript and the timestamp that's on our servers is saying, you know, Sam, think that's great. We also started tracking some stuff. Like if somebody called in and they were still running Internet Explorer 6, we would say, oh, that's why this isn't working on your site. Instead of just trying to hear them describe it over the phone. Am I talking too fast? Am I going to be addressed a lot more by open source, I think. So this is what happens when each person comes in and says, like, I've found the solution. I'm going to throw out everything else that's come before me, except that I don't actually know how to get to the flower anymore. So this is too many cooks in the kitchen, right? Each time somebody comes in and decides, like, I'm just, I don't know what's going on, so I'm just going to throw everything out, you also still don't know why it was there in the first place. At the same time, if you aren't too careful about throwing out everything, you end up with these layers, like, oh yeah, where's this coming from? So in this particular bug, what was going on was that certain cases were getting turned out. It turns out that there was a previous layer that was adding a marker to those that nobody knew about anymore. So we had to, the way that we ended up tracing this bug was that we just had to sort of keep adding new logging points at each join that we could find. So at each point that we would get this data dump, and then it would point to another place or something that was going on. So we'd have to sort of keep adding new logging points at each join that we could find between the different coders. And eventually it just ended up being the first being a flag stuck to these cases that was causing them to get thrown out, and nobody knew where those flags were being stuck to them. In the first place, we just sort of like, so we go in and out at another point, and we're like, okay, erase that flag. And yeah, that's just sort of another layer of crux that's getting added, but see, I don't know why. The Internet today, it's a lot more complicated than it was 10 years ago. So nobody can understand everything anymore. So each time you come into a project, you know, don't just try to throw it away and write it off the scratch, which is kind of where that was. So I guess this is sort of where I was going with this. I mean, does everything anymore? Yes, we have these giant data acts and stuff that we can sort of toss everything into. As long as you can go back and find it, what you're looking for when you need it, you'll think about it. That's why I thought I was talking too fast. Add an additional login. You try and keep it all in a standard log format across all your systems, or is it sort of ad hoc? Like sort of throw in just whatever texturing makes sense. That's human readable. So I guess for a lot of these, before I was at Neo Nova, I was an independent contractor, so I was going into a bunch of different small businesses, and sort of, you know, okay, we're going to throw a code at it, and that'll fix everything, and then we're going to not have a code anymore, which is why you can't necessarily talk to people that came before you. Right. So do you have like tools? I mean, do you use tools to like, parts of those logs? You know, like when you come across a bug, going to the log file is trying to find out are you using like Elk stack or anything like that to like index the logs, or are you just going in with like wrapping? So it depended on the type of access that I had at the system. Then in some systems I didn't have real-level access, so I wouldn't be able to actually look at the log files themselves, which is why I would try to add some sort of like a big table or something that I could look at in access, even if I didn't have real-level access. So I guess I'm using logging in a sort of loose sense here. Gotcha. Okay. Thank you. Any other questions? Who has a question? Was it you? Oh, yeah. So if you're logging everything, how long would you retain these logs? Eventually you're probably going to have to purge. Yeah. So you probably don't need logs from 10 years ago. I guess it'll depend on the level of resources that you have. So the longer you can keep your log files, the more likely it is that you'll need to go back and look at them. I guess in one particular case we had people that were from, they hadn't realized that they'd been hacked three and a half months before or something, and our logging went to about three months before. So that's something I'm from. So I guess as long as you can keep backups you want to, at the same time I know that a lot of small businesses don't start restricted in the level of resources. So they have to store what you can. Yeah, and especially now that we have these giant data centers and stuff.