 Basically, what we're going to talk about is who we are, a little bit about what we do ourselves, some of our goals for the network, some of our challenges, a high-level discussion of what we've got going on in the network here, and just have basically a Q&A. We want to get feedback from the audience to really understand what you would like us to do, where you want us to go, things you'd like us to see, or like to have us do. Has anybody heard about this? It's a new meme we're starting. It's called Pentes John. Enables SSH, can't get in. All right, so let's start off with Heather, who's going to be introducing Lockheed. I'm also representing Lockheed today. Some of you, if you've gone to the DEF CON networking site and seen some of the updates and the graphs, he is the mastermind behind creating that content and keeping that up-to-date. Lockheed has been with DEF CON since, I believe, DC3, and he's been running the network. He is the primary lead for the network team, and I am the second. He also built the mobile agenda. I'm not sure if any of you used that this year, but it was up both this year and last year. Pretty handy, you know, quick reference guide. And I've been with DEF CON since DC9, so this is about my 10th year. And I do a lot of the tactical operations on the ground. All right. So next would be Mac. Yep, I'm Mac. I handle a large part of the wired infrastructure and servers and services for the network. Make sure that the baseline connectivity is there, uplinks, wired jacks for the speakers, goons, vendors, contests, et cetera, et cetera. All right. I'm Videoman. I've been doing this since DEF CON 6, basically. It's been really fun and challenging seeing things grow over the years. You know, we went from an infrastructure at the Alexis Park where literally we would get up on roofs and run cable, you know, in 100-degree weather, or run cables across the parking lot. And, you know, seeing this network mature and seeing it grow into what it is today has been really fun, really great. And hopefully it provides value to the conference attendees. For sure, it's definitely provided value from the perspective that, you know, updates.defcon.org. How many people use that site? A few of you? Okay. I mean, that's the info booth. They set that up. They come in and do that. And we provide the network for that. If that network wasn't there, that wouldn't be there. All right. Next, Louise. I'm Louise. I've been doing the wireless network here for, I think this is my sixth year. So I've done one year at the Alexis Park, another year, or all the years at the Riviera. And now we're here. I'm Derek. I'm the guy who runs DCTV. I've been with Defcon since Defcon. Wow. So out of curiosity, how many of your drunk hungover asses were sitting in bed watching DCTV this morning? I just want to see if she has. That's awesome. Sorry for the outage. That one was my fault. DCTV changed a lot over the years, especially the ones where we didn't have it. And it's been a long time since we had all the pure analog balance and old analog cameras switching over to what we have now. We're now running MPEG 4 over IP, over a 4,000 foot fiber trunk all the way to the cable TV head end, and the other total end of the building. So that was a bit of a challenge to set up. We had a lot of last minute changes to the infrastructure, but it looks like it worked out pretty good. Special thanks to the Sonar Knowledge folks. They helped pull it together. Yes, props to Sonar Knowledge. Sparky. My name is Michael or Sparky. I guess I'm essentially the DC tech for the knock. Everything you see taped to the floor. My blood, sweat, and tears went into that fine, fine work. Resident bartender. Yeah, that's about it. I've been with Defcon since Defcon 8. Without Sparky, there would be no network. Ribchi, who is down at the end. Yes, my name is Ribchi. My responsibilities basically are very similar to Sparky's. We run as a team. We run around. We make sure everything is working the last minute. Do repairs if necessary. Attack down table. Put it in switches. Anything you guys need. We make sure it happens. And Eric, who's a new addition to our team. Hello. I'm Eric and what I've been doing all along this week is to try to ensure you have the wireless coverage and reachable for almost all the time. So he lost 10 pounds during the whole week. Pretty much. Pretty much. That's a lot of weight for a little man. Yeah, I mean, so basically Louise and Eric are in charge of the wireless infrastructure. And, you know, we arrived on site on Sunday and started setting up the infrastructure. And a few arrived on Saturday as well. But Louise and Eric actually go through and started reassigning all the access points. We did a little bit of an upgrade. We'll talk about it a little bit later. And they're that team. The last person on the list is T's, who actually wasn't able to join us. He just got a new job, which is good. But unfortunately couldn't take the time off. But what he does is a lot of the content updates. So yeah, we've got the mobile edition, me.defconn.org, that he helped put together. He's helped with a lot of our maintenance of our wiki that we're using. What else, what other duties has he put together? I think that's... General cable, bitch. Yeah. General bitch. All right. So what is the network, right? Some of our goals is reliable connectivity. You know, if you're going to come to a network or come to a conference and have a network connection, it needs to be somewhat reliable. Secure-ish, because we don't know if someone has a zero day for the wireless network, right? We say secure-ish, because now we have WPA with enterprise management on it basically allows you to log into that. How many people here actually created a login before they came? All right, how did you find out about it? Twitter. Twitter. Was that primary? All right. Then that's great, because that's something that we've kind of used as a team. If we know something, we try and tweet it out like, hey, go do this. And then usually someone retweets it for us, which is good. I was going to say we also pay attention to the Twitter stream a lot. So if some people are complaining about particular problems in an area or something, that's one of the ways we've been trying, you know, that's our poor man's ticketing system. So the other thing, does anybody remember at the Alexis park? I think it was the first or second year. Basically, Ghent, who is no longer with the team, due to, well, anyway, he brought along a switch from one of his work, like, what was it, a 4,000 or 6,000 series switch. And we went, oh, we've got, you know, 100 and some odd ports. Let's plug them all in. So we plugged all the ports in, made all the ports in the conference center active. And what you'd have happen is people would come along and plug in and go, oh, I can get access here. Great. And then people would be like, oh, do you have a hub or a switch? Oh, sure. They'd plug that in. And all of a sudden you'd have this train of people going down the hall. It was kind of funny. I mean, cool but funny. Except for the fire marshal. Yeah, yeah. So the fire marshal, I don't know if anybody remembers that, but we had problems with the fire marshal at the Alexis park. And so we said, okay, we can't do this. People need to be able to be mobile and be able to get access. So at that point we started really deploying a wireless network. You know, the wireless network at that time was in no way secure at all. It was all just open wireless access points. Cool. And so basically, all right, let's work towards something that people can get access to anywhere. The other thing, we have a boatload of segmentation. We have speakers, press, all the speaking tracks. So like this podium has its own VLAN, has its own network. Each of the five tracks that we have out there have their own VLAN. We decided to change that up. Last year it was one VLAN for all the speaking tracks. And then someone decided to do some man in the middle on the speaker network. And we were like, yeah, maybe we should change that. So we did. You know, it's constantly a growing process, a learning process. And you know, that's really what this is about. But right now I think we're at 199 VLANs that we've configured on the back end. So, I mean, that's pretty cool. Was it 140 or for wireless? 140 or for wireless and 59 are for... Wired drops. And we had 69 actual drops into the spaces. So only 10 of those drops shared VLAN segmentation with anybody else. Yeah, and those are primarily the contests. And unless we get a specific requirement that it needs to be on its own network, we kind of say, all right, maybe some people can share. It reduces the load that we have. I don't know what our back ends switches can hold off the top of my head. It's like 300 VLANs I think is what we max out at. But all right, so some of our challenges, I mean, we don't have an infinite budget, right? We actually, I mean, when we started out doing this, we said, all right, what can we get off at eBay? What do we have that we can borrow? And that worked out well for quite a while. Unfortunately, we can't go borrow Rubik controllers because sometimes they're in use. Well, most of the time they're in use. So what was that? Defcon 13? 13. 13 that we actually went out and purchased the wireless controller, the Aruba wireless controller, primarily because it would have some IDF built into it, you know, intrusion detection, be able to detect rogue APs. I think that was the year that there was some contest that was doing something in the contest area, like a king of the hill access point or something. And we would see it and we'd doze it. We'd say, all right, no, it's not on the network. Gone, you know, five minutes later, access point would disappear. And the next day it would show up again. We're like, come on, guys. And finally, on Sunday afternoon, I think they came back into the knock. And we're like, yeah, we tried to do this contest, but unfortunately our access point kept dying. We're like, oh, yeah, that was us. So coordination also helps with us. Hotel, does anybody want to speak about the hotel? Sure. Are you looking for space usage or? The logistics here. Yeah. You know, obviously you've been walking the half a kilometer back and forth through the convention space. You know what the walk is like. Actually, I think the space this year is incredible. Our wireless design has improved greatly, just due to the way that we've been able to use the space. The hotel staff are fantastic. We've had just a great partnership with them this year. And while we had a good partnership with the Riviera, things are just really smooth here. And, you know, frankly, it's a nice way to transition into a new hotel. We have, and I don't know if you want me to get into this right now, but we have better bandwidth options here. They have metro area ethernet here. So we have some flexibility to, you know, move up in bandwidth. The infrastructure itself, while different from the Riviera, has worked out well for our needs. And really, it seems like even the traffic flow has been better for the humans. So the other thing is when we're at the Alexis Park, it was a non-union hotel, which meant we would grab a cherry picker and literally at like three o'clock in the morning be driving around the hotel spaces, driving this cherry picker, going zip tying the access points, running the cable down. We all did that, like everybody pitched in. You know, we'd get the network set up just before the attendees would come in because we really didn't have the space beforehand. But when dealing with union hotels, we now can't touch certain things. You know, the lighting grid, you know, electrical things, hanging things on ceilings, hanging things on walls, sometimes has to be done by a union staff. So that in itself was also a challenge for us because we didn't think like that. We had to change our way that we were dealing with the hotel. The infrastructure here, Mack, do you want to talk about the fiber? Yeah, so moving back, moving from the Riviera to the Rio is really, it is a different, you know, stage of hardware infrastructure, a different level of IDFs, different connectivity between them. Here we have a lot more option, you know, well, actually we have the same option, but it's a standard option. There's fiber between everything. Versus at the Rio, we actually had some fiber patches. We had some copper patches. We had some long range ethernet patches, which were very interesting to some of the spaces. Sorry, at the Riv. So it's, here is very different. I mean, we, you know, the hotel is just significantly different from that standpoint. In general, one of the things that we have to consider with our infrastructure is it has to be spun up and torn down in a matter of days, and it's only used once a year. So we tend to, you know, going back to the money item, we tend to, you know, get what we can, but we can't really justify something because we're not seeing a 360 day use out of it. So a lot of our infrastructure tends to be fairly, you know, fairly older equipment. We're still getting stuff off eBay. There's certain, you know, purchases which we make of newer gear just for targeted applications. The Aruba, the core for, to a certain degree, and a lot of stuff like that. But, you know, we have, you know, sitting in our, sitting as part of our infrastructure, I keep looking at this and I want to say, here's what our infrastructure is versus here's the challenges. I'm going to go over the actual infrastructure in a bit. So I'll leave that. So bandwidth, we actually have 100 megs bandwidth this year. It's a metro area Ethernet. It's not a Wi-Fi point to point. So. I was going to say, so that's the uplink we have is 100 meg. Between all of the IDFs, you know, to the Aruba gear, et cetera, we're sitting on gig backbones. You know, we could argue, hey, we should have done this earlier, but realistically looking at some of the graphic trends, or looking at some of the traffic trends, we haven't really been pushing more than a couple hundred meg. So it's, I mean, I think it's high for a convention usage, but it's not high for an overall usage, especially compared to the gear we have. I remember a couple years ago, there was Dave Bullock whose photos are in a lot of these here, but Dave Bullock did an article on the network, and in the comments below, people are like, oh, why don't you have gigabit? Because at CCC, we have 10 gig. Wait a minute, we don't need 10 gig. We can't justify it. We have to be able to justify what we're going to pay for. And, you know, in Europe, they may actually have companies that will donate bandwidth to them. We don't have that luxury. I mean, we're in a hotel. We have to pay the proper channels and get the proper things. Encore is actually the one who provided that to us. So, as a convention. When we say justify it, we're talking about you guys. You're really what justify it. So if you find something you want to use more bandwidth for outside of torrenting, because a lot of people just see that as not easily justified, but if you have some really application, something you want to play with and test with here, now, this is a good network to do it on. So think about that, especially for next year. And the other thing that I think we kind of do well is the rumor that, oh, don't go on the DEF CON network, because you'll get hacked. Right? Because if you don't use the network, we don't justify that we can add more bandwidth, that we can add more things. So it's kind of a chicken and egg. All right. That's my job. So keep pulling down stuff, would you? Let's see. The device, so wireless. Do you want to talk about wireless? Yeah, sure. So for the wireless, for any wireless implementation, there are two major concerns. One is, of course, coverage. Right? You want access wherever you are, in this case, in the common areas. But also here we have, or any convention, you have a problem that is user density. So I can put one access point here, and it's going to cover this whole huge space, but once we have a lot of people, it goes away really fast. So technology evolved a lot on the wireless market and wireless devices. The thing is that you're still limited to a certain number of users that are going to share that one access point. So then it comes to the user density is going, the only way to solve that is that you're going to need more access points. And once you have more access points, you're going to start interfering with each other. And that's not good either. So wireless, in the U.S. usually we use channels one, six and eleven. That's what we should use. And so if you have more than three APs in the same place, they're going to interfere with each other. So it's good to have a solution that deals with that automatically. The other problem that we had in, I was going to say in the past, but we still have some funny things happening. This device compatibility. This used to happen a lot for the past few years. It didn't happen. But last year, and I think we're going to talk a little bit about that, we had an issue with iPads. And it came back this year. Once it came to, you put that in sleep mode, it wouldn't come back. So you had to, there was a workaround for that. But again, it's always like it takes a couple people to say, I can't get on the network. We hear that really fast and we try to fix it. The other problem we have is time. I mean, we do not have an infinite amount of time. We have basically three to four days to do setup. We're here for a week on site. We have lots and lots of gear that we bring with. Either gets thrown in a car or gets shipped from some place. All of us come from across the country and maybe even outside of the country. We have two Canadians here up on stage who can represent, right? We're a team of ten people, essentially that work throughout the year. We'll have a pre-con meeting where we'll come out to the hotel and make sure everything is the same, make sure things are working, and then just do things over email every once in a while or have a conference call. The team is kind of broken up into the infrastructure, which is Mack and I, the Wi-Fi, which is Louise, Eric, and Heather, and then the video, which is Derek. And then we've got the two managers, so Heather and Locke, as well. Don't forget about our ground pounders. They're pretty critical. Don't forget about the ground support. They're pretty critical. Yeah, you've got to have the bean counters. All right, so this is the map, essentially, right? This is all the areas that we were expecting. And we weren't really expecting, I think, that we were going to be able to do the reg desk or the swag area, just from the perspective that we hadn't gotten a request for it. And Penn and Teller Theater, I think, was that also... It was the last minute. Yeah. So we kind of had to scrounge this year because we were expecting X and we got X plus. But, you know, we do with what we can. And this is a lot of ground to cover. If you look at this map, it's a really large space. It's covering... This is like five or six IDFs, right? Oh, three, four, five, six. Six IDFs, one catwalk space and one... Sorry, one... Under theater space and one cable TV head in space. So we had to put equipment into switches and all those. Nine different cabinets. Yeah. And IDF is interdomain feed, right? I always had an intermediate distribution frame. There you go. Versus master local. Israeli defense force? Israeli defense force. Basically, it's a wire... It's a telco cabinet or, sorry, telco closet or where, you know, where you drop all of your network... So all of your wall drops come back to what is considered an IDF and then that will link up to a central IDF, which is usually called the master. And this year we didn't have to work through any bathrooms to get to ours? Yeah. I still wish we had better pictures from that one. Yeah. There is an IDF in the bathroom in the RIV. It's awesome. Yeah. It's like, oh, this is where our... Oh, that's never mind. Anyway. So this is kind of basically the structure of the network. I'm not going to spend too much time on this because it's a slide and you're not going to be able to see anything. But it at least gives you a concept of this was pre-putting the stuff in place. Post is a little different because we had to do some work grounds, but it's almost where we're at right now. This is a map of where all the access points are, essentially. Do you want to talk about this one a little bit, Luis? What is important here to see is that we cannot. So we have tools that say this is the optimal location for the access points according to RF logic and whatever the smart people put into softwares that calculate that. But of course we depend on many things including where the drops are, how high we can put the access points and all that stuff. So this might not look optimal, but that's how we have it. And then we deal in configuration with basic rates and transmit rates and all that good stuff to make number one coverage work all over the place, as you can see on that one. And also avoiding interference and roaming because roaming is quite important. You start, you turn on your device here in this room and you're going to be walking around. You don't want to drop whatever you're doing on your device. So that's important. Here's a graph that we just pulled at about 2 o'clock today. As you can see we did peak at some point on our 100 meg connection. There's two pretty big spikes there. And apparently last night y'all went out and partied. Because you can see that there's a big traffic drop. All right, so we're going to talk a little bit about what worked for us in this space. We'll talk also a little bit about what we have to move on. So secure wi-fi, I think, does everybody like the secure wi-fi? Yes. All right. So I mean you can thank Louise, Eric, Locke, those guys for setting all that stuff up. Locke actually did the radius database back end. And the, I think T's also did the front end for that so you could go in and enter in credentials off site. We actually have that server living here and then when we pop up our internet connectivity that's what you're talking to is directly into our network on that machine. I think it's very important that we have that because it allows for some sense of privacy. I mean from the perspective that we don't actually ship those VLANs off to the wall of sheep. We made that conscious decision last year. We basically said, all right, if we're going to give people a secure network there's got to be some semblance that there's an understanding that it's secure. Which means that yeah, maybe you have pop three still enabled maybe you have to tell them it to a router maybe you have to do something like that. But we don't want the wall of sheep to grab it. It's just the wrong thing to do. We don't want to give it explicitly to the wall of sheep. I think it's the best way to put it because I mean the traffic once it leaves our uplink is going across the internet anyone can man in the middle at that point but we're not going to explicitly go out of our way and do it you know at the beginning we want to make sure you have some semblance it's still up to you to secure probably end to end just from the standpoint of you don't know who else is out on the internet so you do need to protect yourself but we're going to do what we can in that space. And the other thing that we've done is we've disallowed peer to peer traffic so if you get on the Wi-Fi you're only going to see the MAC address you're talking to you're not going to be able to see anybody else's traffic you're not going to be able to map other people on the network because you only have internet outbound does that make sense it's a good thing I mean it's how we're trying to protect people from getting pwn I mean that's why you don't use the open Wi-Fi so you can actually use the open Wi-Fi you're more than welcome to check all your email and do all your banking on the open Wi-Fi the wall of sheep gets that traffic so does everybody else around you exactly yes fire sheaf right internet connection 100 meg recovered that did anybody realize that we had IPv6 yeah alright has anybody gone to an IPv6 site no fail no I was kidding yeah from what we saw you know .2% of you did something with IPv6 that which actually mirrors the rest of the general internet traffic so it's not too surprising but once the power base which is over everybody will let the traffic go up and it's true we should talk to them about that one so what was the we were tunneling outbound right yeah we're using a tunnel broker or sorry tunnel with tunnel broker slash hurricane electric so good company if you want to go with IPv6 stuff go with them let's see the Wi-Fi was updated we had a software update on the controller that allowed us to do some good stuff we were able to extend the network out with just a couple of access points using some mesh protocols where was it to Starbucks basically yeah we're trying to go to all the way to the casino but it really didn't work well there were some issues that the casino had too much no Wi-Fi is when you gamble I don't know it would be nice to sit in a casino and drink a beer we'll see what our expansion limit is for next year so big thing DCTV to the hotel rooms do you want to talk about that Derek sure it's been a few years since we've had DCTV in the hotel rooms just because in the Riviera to their cable TV head-end so we really miss that and obviously a lot of people got sore feet over it so we're really happy to be able to get back in there again this year but we're still at a challenge and that was we had to pipe video like you know over a mile of cable effectively so the systems we were using before which ran a lot balance weren't going to work because you just get a big blob of noise at the end in audio so this year we're running now the equipment to do this stuff is crazy expensive at the professional level so we're actually using some again eBay finds really inexpensive MPEG-4 and AC encoders that are designed for video surveillance and we ran in some challenges actually finding decoding equipment that will play it back on the other end and those challenges came up right near the end so again thanks to SOK for supplying us with some scan converters just to retrofit what we brought that cable TV head end working but with five channels it would have been I think it's three thousand dollars for the two ends of the encoders for a professional level equipment so a fifteen thousand budget for DCTV that wouldn't have gone over well so instead we did it for basically a shoestring budget and I think it worked pretty good I mean the original plan was to use PlayStation 3 actually as decoders and we found out unfortunately once they have a network in between them they started crapping out after about three to five minutes of streaming so we were scrambling trying to find boxes that we could run this stuff on and basically Derek re-engineered it at the last minute and got it working you know our plan was to have a little bit more content on the TV channels but unfortunately we had to forgo that just to get the feeds there for now we were literally plugging them in during the first talk wandering around here as people were arriving in we were plugging it in so that's how close we cut it with DCTV this year finding the servers to plug in as well since we didn't have the PlayStation we had to scramble to get computers so now my work laptop is up in the CATV head end I can't get it back without a guy who's licensed by the gaming commission I think to actually unplug it from their network and same thing I'm sorry Mac but I have using this wonderful bandwidth is now serving videos too so he doesn't get any new television to watch when he goes back home so you can thank these guys for giving up their laptops so you can have TV in your room okay alright so we also had workshops I don't know if anybody heard about that yeah he's fine he's fine it's Defconn what do you expect he actually works trust me he's now sleeping yeah I can hear him snoring alright good good he's fine alright so we tend to we tend to party just as hard as you do so and go to work every morning yeah we have to get up at 8 our call is at 8 so sometimes that's pretty I'm a slave driver yeah I'm a slave driver I don't know how they do it oh that's right they go to bed at 10 oh I'm just kidding when I leave here I go on vacation that's not true just one night so workshops we had workshops so we had to deploy some switches there that was new to us kind of a new new thing which I think will be good I think we'll see what we can do maybe we'll have more stuff going on there alright issues POE we bought a bunch of POE switches off at ebay unfortunately they are Cisco proprietary POE switches we got some power injected switches not some POE or sorry power inline switches not POE yeah we have IP phones but no access points that wouldn't help yeah so we could run Cisco IP phones on them all day long but not wireless access points that are to the 802.3 AFs back you said IOS devices I heard Cisco IOS I don't know what we're talking about no one told me Apple apparently overnight the DHCP stopped working we're not sure about that we're going to look into that so that's an issue to figure out if someone was dossing it or not we don't know to be fair people were at parties no one should be on overnight last night so it's fine like I said everybody's out drinking PS3s we talked about new infrastructure new hotel just having to figure out exactly what hooks up a lot of the time sight unseen et cetera and there are more challenges that come with casino hotel fortunately the good hotels have their infrastructure designs that the convention services area is actually a purely separate network from the hotel and casino side and this hotel is no exception they have their network very well separated there's also well anyway Wi-Fi penetration so just being able to penetrate into the spaces I think there's some potential dead zones but you know as we figure that out we're going to get better at okay we need more access points or maybe we need to put access points in key areas so that people get a consistent signal so yeah we plan for the signal to bleed over to the walk away there to the hallway and it didn't do as well as we thought so that's that's something to get better next year I think it's the big giant metal shield yeah I'm kidding now I'm looking at it something about an RF cage fair day cage alright so now is the A and Q right we give answers you ask questions there's a microphone up front here people want to come up and ask a question I don't know if it's up don't all rush up at once or just yell it out we'll repeat the questions if they're not on mic don't worry about it I'm curious what the cost is for the internet access for the duration of the event and who actually are you paying the hotel an ISP what are some numbers there Heather actually I don't have the contracted price in front of me and that happened months ago but it is done through the hotel bandwith set aside and they bill based on ballpark I'm not asking to the it was honestly I don't have it right now it was mixed in with a bunch of other contract stuff months ago so I do apologize but that is actually bandwith that the hotel has for all conventions we can go up to 200 meg on that bandwith beyond that they would need to bring an additional super quick question on the diagrams you had up there the whole application to map out the wireless ranges what is that called I think that go ahead Luis it is embedded on the Aruba system so it's called RSPlan and you just throw there like the size is it open for other AP types or just no, no, it's specific for the AP so the controller itself has a management interface and that's where it's actually pulling that from you go in and you go them right and put them on that map. So it allows us to import a map essentially into that. It takes all of the fun out of the job. Thank you. Quick favor. Can you guys move the line over right up here? We're not the TSA. We're doing this the wrong way. We're supposed to use a Q&A room but stuff starts at 6.30 so it's kind of weird. Hush. Go. Not really a question. Just a comment for consideration for next year. You guys mentioned that you have a lot of equipment that you transport out here. Possibly a solution to save on transportation costs if it's just equipment that you only use once a year. Just get a local climate controlled facility. Yeah, we have talked about that and for some things that will work but we do have the challenge of having to pre-configure and test and update and do that sort of thing. This year we actually, there's three of us that came out Saturday night and we had already pre-configured a lot of the equipment and updated a lot of the equipment to, for instance, work with IBV6 and give us the additional features that we needed. So we do a lot of that kind of remotely so it'd be hard to do that in a storage situation but for some of our equipment we are considering. Yeah, cables and stuff we don't need to necessarily haul out but we need to update firmwares and whatnot. Just wanted to see if you guys had any recommendations for a new V setting up an 802.1x network? On the cheap I guess. For 802.1x what, on the server side or? The whole thing. The whole thing? Well we actually use today any enterprise or not really enterprise type of solution supports 802.1x, WPA, WPA2 with 802.1x authentication. You can get, we use free radius. You can use free radius and sometimes it's, could be a pain about to configure but it's well documented actually these days for the most, for the commonly used applications. Alright so I have two. One I was just curious for the wireless stuff. It's like in the program and like you guys mentioned it goes straight from the, like wireless to the firewall. Is that just by way of the VLANs? Because you said they have like access point isolation so like two clients can't talk to each other or do some fancy stuff like L2TP tunnels or anything like that. So for each AP we have two ESS IDs. Each ESS ID on each AP has one VLAN. So if you hop in AP1 you jump on the secure network. Once you run to another AP you're going to be on another VLAN and then internally all the traffic is actually encapsulated through GRE from the AP to the controller. The controller makes decisions as for what VLAN that user belongs and what policies we do and we actually bridge everything up to the firewall. So all the routing is done on the firewall. Very cool. The second one is since you guys mentioned it I actually fired up TCP up on my phone which usually does IPv6. I didn't actually receive any RAs. Are you on the secure or on the secure? Yeah, probably on the secure. I'm not going to see anything because that's the whole idea. Because on top of it, on top of the VLAN I was going to say for the RA, the RA you should have been seeing. It comes from the firewall side. Yeah, so that, I mean I was seeing it plenty on our stuff. So we were having better, I was seeing better RA, you know, traffic coming through than from the DHCP just because of the way V6 does its address allocation. So I'm curious if we could see it afterwards. All right, cool. Again, just one thing regarding traffic management, traffic shaping. Have you guys played around at all with dynamic percentage based PERMAC traffic shaping at all? Or is it just FIFO? Not PERMAC. We do have outbound traffic shaping on the firewall. We tend to give priority stage, GTP, SSH. So it's protocol based? It's protocol based, yeah, at the firewall level, not anything at a station level or you know, PERMAC. Okay, cool. Are you asking us if we want to prioritize our traffic? No, no, no, no, no. It was just because I've been investigating some of that for a long time, you know, when bottleneck alleviation. Okay. Yeah. A couple things. One, it seems to me the equipment you're using is not exactly specialized for DEF CON. Have you considered maybe working with UNLV or community college in the area to have them maybe split cost for equipment with them? They use it during the year. DEF CON isn't during the traditional school year. Maybe you guys can take the plan. Something like that. So part of that comes down to whatever negotiations and contracts we can do with those individuals and I don't know if we have any particular relationship with UNLV particular, but it's an idea. One of the things we have been avoiding in the last few years is getting any kind of specific vendor promise or any specific relation promise that is not ours because given the timing, given the config and given this resource allocation, if all of a sudden they decided they didn't want to let us use it one year and we're borrowing equipment, we're really in the hole for that. So we try to make sure those are good. There's a political reason they didn't want to give us the equipment. I mean, it's a really great idea, but we are a hacking conference. Yeah. Well, I mean, if you owned a share of the equipment, that might kind of cut the legs out from under them as far as holding back goes. And the second question I had is that guy still breathing? Yeah, he is. Okay. He fought a good fight. He's had a very long week, but he's been up in the morning and helping us out every day, so. Hi. Do you guys offer any type of service consultation or anything to the host hotel to help mitigate some of the computer malfunctions that might be happening during the conference? No comment. No. Typically we don't. Unless they come to us and say, hey, we're having a problem. Do they ever do that? What's that? We did with the RIV. They did ask us to, for whatever reason, block outbound connections from us to the RIV websites, for instance. So it's a delicate line when working with the hotel that we, there's some back and forth, but when they ask for something, we see what we can do. But typically it's the, we're segmented off. If you think something's happening in the network because of us, then we can talk about that specifically, but we are segmented off. It can only come out through our head end. So we have an idea of what's going on there. This is very different from what we experienced at the Alexis Park where we pretty much owned their entire network and would come in and they, I mean, it was great. From the perspective at the time, we were very small or, well, smaller. And we were able to go in there and just sort of take over the network and do what we needed to do. And now we're dealing with much larger network, much larger venue and then the whole issue between the casino, the casino gaming issues and the convention center. So it's a totally different ballgame now. But we do, we have meetings throughout DEF CON. We're constantly in contact with hotel management. We have very open communication about what's going on and what they're experiencing. And while we don't out and out offer to help secure their network per say, we do make ourselves available for questions and if they do need something, you know, they know we're willing to help. Okay. Thanks. And thanks for all you do. Thank you. I just had a question. You guys showed some graphs and metrics. Do you guys keep track of that and make it publicly available? Yeah. You can actually see it on DEF CON networking.org. Okay. Great. Thank you. Yeah, we usually will put together a presentation and then put all those graphs up afterwards. The graphs are actually up already. Most of us eventually made it back to our hotel rooms. And when you flip through the TV, we basically saw the desktop of your laptops or machines or whatever. I don't know if you record those, but would it be possible to record the talks and then like queue them all up and like for the talks for the day, just play them in a loop at night? You mean like a PBR? Yeah, right. Actually, so the situation is such that we can't do that partially because the talks are recorded and sold. And also because with this hotel, we have agreed not to replay talks due to since we're putting it on a cable system that we can't lock down. That means anybody who's in the hotel can see those talks. So by not replaying them, we're providing them with a sort of plausible deniability. Yep, they said something in the talk that the customer didn't like, but it's not being replayed. Yes, they asked us if the speakers wouldn't swear. We said we cannot. This is our compromise. What made me think about it, I had to do some work and I was sitting there flipping through the channels, waiting for stuff to happen, and I watched part of Jeopardy. I'm thinking like if other people in the hotel who have nothing to do with Def Con are sitting there watching Jeopardy, we might as well show them something more informational. It was fun, but you know. Now next year, we're looking at putting something in between the talks as well so that there's not that, you know, the VLC desktop sitting on the screen in between the talks. But again, this year, while we did get DCTV up and it was, you know, it was a win, it was still a challenge, so we were working with what we could. Look out for next year, because it'll be bitter. Hi guys, I was wondering if you guys were planning to put up maybe like documentation of how you guys did the things you did or maybe some like best practices for people that want to set up, you know, in other environments that are hostile as well. I don't know if best practices is the right term to use on this network. But yeah, so we've been debating about it, well, not really debating about it. We've been lazy about it for a few years. This year I did like for particularly the firewall configs, the switch configs, those are actually stored in a Git repository. Once I scrub it, I'm going to make it available on GitHub, scrub it and get some sign-offs on it. I want to make sure everyone's cool on it. But yeah, we're going to be putting that together and putting it available. So check out defconnetworking.org and we should have some stuff up there, hopefully in a couple of weeks. Thanks for all you do. No problem. Thank you. I said one thing that would have been, I don't know if it's doable or not, but with the TV it'd be kind of cool if we had like the power points in the corner as they're speaking. And I mean, I know it's obviously probably a big challenge more to be mixed. Yeah, actually what we actually wanted to do that, we wanted to combine some of that stuff. It's an equipment challenge that we weren't ready to face this year. We may have time and planning to actually do that next year. So yes, we agree. We actually wanted to switch it over to just the power points, but again, we'd have to have scan converters and stuff that we didn't have hardware access to. If we hadn't had the challenges that we did have, I think there was a pretty good chance we would have gotten to doing that this year, but we just ran out of time and we would have been changing things halfway through the talks if we did. Two quick questions. I hear you use PF firewall, is that correct? Yes. I think you're specifically in charge of it. Is it fair to say that you think the same of PF, GUIs like PF since in the same sense that a lot of IP tables folks think of IP tables GUIs. Is it kind of looked down upon over just PF itself from the command line? Is it one big... In a lot of ways it depends on the tool you have, or depends on your level and the tool you have. We're pretty comfortable with a lot of the interfaces and we do a lot of things that the interfaces don't necessarily want to do, so we need to be able to customize it. So again, tools like PF since are good for an end user type person, but when they get into our realm, we've got 200 VLANs which would all have really goofy rules to find on it. If you go in and actually do a listing of the PF rule set on a PF sense box, there's things that they do, rules that they put in there to make it easy so you don't stomp on yourself, screw yourself up essentially. We don't want that. We want full control of it. We screw ourselves a lot. We do it just fine without having to have a GUI to troubleshoot. But we're done. Yeah, so we're going to... But PF is much better than IP tables. Yeah, PF's good. Real fast, we're going to sneak it in. We want to hear your feedback. We want to know what you think, what improvements you want to see, et cetera, et cetera, so at least I'm going to be available and a couple other people are going to be available over there for at least a little bit if we have to, we'll move to the real track one Q&A space, so come talk to us. And you can always email knock at defconnetworking.org too with your questions and feedback. Thank you. Thanks.