 Hey everyone, so This will almost be the same talk as last year except for the questions, which hopefully you will have lots of which is why I'll be hurrying which is a really really good thing because You'll see this in a few places our our infrastructure is becoming more and more stable and less and less panic on fire Which is really really nice There's one major or minor difference between the two This year is the first year. We had more clients on the IPv6 only network than on the dual stack network So IPv6 is here We had a lot more VPN traffic on the dual stack network And if you are familiar with most VPN solutions, they don't really deal with stuff nicely Especially net six four is is I mean net six four is lying to you on a networking level. So VPN solutions tend to not really like this long story short If not for this issue a lot more people would probably be on the v6 only network Of course if their VPN works, then they would probably be using that which is really really nice So for the second time Last year was the first time We actually had some time to sit down and breathe instead of just running around fixing all the things and all the times and and keeling over and dying in a corner after so Last year the running joke was something will happen something will explode every second because This this silence it can't be good something must be happening and we just don't see it yet This year people did not even talk about this anymore So they didn't even think about like it was two years ago or ten years ago. They were just this was normal Stuff being boring was normal, which is totally awesome. Of course Things work instead of us always fixing stuff We have place reach this place of stability. We are just doing minor changes. We are doing evolutionary changes We don't like toss everything out and start anew as we as we did a few years ago People are actually getting sleep. I think I got six hours of sleep last night. That's Way more than we had a few years back. So this is really really nice Our setup is pretty much still the same. We have our ASR 1k, which does the routing It does the it does the next six four. It does not do the NSX for all those things It does not do DHCP anymore So we're starting to move services towards our servers, which are redundant and the ASR is not All monitoring is done with Prometheus and Grafana and we are emitting all that data to grafana.com On their Cortex cluster. So if you go to dashboard foster mark, you will be seeing their stuff So you so you don't hammer our servers into into submission Of course peak yesterday when I when I tweeted the dashboard was like I think 80 queries per second, which is quite a bit so What we do with video is still the same I Here I can't really open it But we have those video boxes which are new and there will be a link later Which where you can we can clone those and and redo those everything here is is open source All these streams are sent to the rendering farm, which is made of old laptops And then we are setting an off-site for for streaming and such So if you're looking at the stream, it's actually coming from the outside into into the building And we've got our own review system and people are already reviewing videos and speakers are already reviewing their content and things are being transcoded so we can upload the final versions relatively soon. Hopefully This is a picture of our render farm So every single year what we do is we grab a bunch of thinkpads off of eBay And we sell them on site after the event. So for us this render form is basically free Of course, we are just reselling them at at cost We are using new model every year. So two years ago. We had the x220 then we had the x230 this year We had the x240 you can kind of guess what we'll have next year probably of course We're just sticking to that price point and it just goes up by 10 every year. So that's nice and If you think hey, I want one of these you're way too late Because people are are standing in line to grab those so if you want those First they go to info desk grab a voucher else. You can't get any timelines Installation of router we did this a lot earlier for quite some time already Network up we actually had most of the network up on Thursday There were some miners some major hiccups, but basically stuff was working Friday night Which was later like in totally working state than last year But in in partially working state we were a lot earlier which again is good because it gives us more time We don't have to do everything in a panic If you look at 2015 this was a really really really really shitty year for me Monitoring again, we are having monitoring we are talking to you'll be to maybe even be allowed to run those servers All year and they get access to our monitoring so they see what we do on their WLC So they see what's happening in their Wi-Fi blah blah blah blah blah, so maybe we can even run this year on video We actually managed to set this up on finish on Friday, which is the first forced them ever so again We became quicker at what we do because we do reuse more code so Reuse there is one issue Obviously you need to know your automation if you just log into a machine and you fix stuff and everything's happy and then things explode And it's shite This workflow is normal if you do automation if you do ansible you have extra complexity Which you need to deal with in order to to to not have things explode and when you're on the stress This is extra extra work, so you really really have to be committed to doing this Of course, else people will start working around it last year. We messed this up We left old backgrounds in our video config We had the wrong date in our t-shirt tracker all those things this year None of this happened because people got used to reusing the stuff from former years, so live and learn basically We will probably be using next book and that box for for RCM DB for next year We will most likely be installed this in a few weeks and then we will start emitting emitting DHCP config and DNS config and all these things from a database So it's not just random files, which you can which you have to look for you have one single sort of truth For everything our dashboard is still the same Hammer at it try to bring it down If it explodes then Grafana has something to debug so they're happy and If you want to look at what we do in our conference stuff everything is open source Just clone that and you might need to do a few tweaks, but we keep all this in the open for obvious reasons Same as those boxes So this is already half of my time and I was really quick about this course. I am sure you have questions Shoot away So the question was if we are running free software on our airs are 1k and the answer is And now it's iOS it's iOS XE Of course, you can't really do anything We would like to to use Free software in the space I come from the networking world. So it's really really hard even network vendors who use like Silicon they just buy and bulk like from Broadcom Run into really really icky bucks and they have teams upon teams of people just dealing with this So doing this in open source will not happen in the next few years unfortunately, so one of the reasons that IPv6 only used to be a big problem was that it doesn't Didn't used to work with Android because they were being annoying about the HCP 26 and the lights But it seemed to work on my phone this year So what did you change so that Android suddenly started working with the network here? That's through to advertisements and router advertisements worked with Android for ages I will not go into the details of of Microsoft versus Google and through to announcement versus the HCP v6 Both sides are wrong No, I mean If you would have a system where just here your end client would support both and The farther you go into into infrastructure, which is more or less static and there you only have router announcements That would actually make sense because there is nothing to do like tell me what printer I have you cannot do this without Announcements you can do it which the HCP v6 so there is an obvious use case for for doing this at the edge But people are being really stubborn about it And they just don't want to because they need to be right and they need to win and they need to be slapped Yes for obvious reasons Anyone else I'll only bite if you don't ask so This is your chance So the question was what service we are using Those used to be database service like 10 years ago or so so random old crap It doesn't really matter it used to matter when we did monitoring differently course This was a huge drain on on our resources on the servers with permissives. It doesn't matter anymore, of course It's just so efficient The one funny thing guess what the number one user agent on our network is on the whole Campus of of you'll be during this event The the number one user agent Not quite as permissives. Yes, of course we we we scrape everything every 15 seconds And we're just hammer away at all the infrastructure. You saw the rendering farm. We have tons more of infrastructure And it's still so lightweight. It doesn't matter So we use random old machines and I have I don't even know how much RAM and how many CPU course they have Of course, we stopped caring. We'll be putting SSDs in them next year and else whatever it just works So the question was if we're using our own infrastructure, you'll be infrastructure or what we are doing the answer is in between We are using fiber runs between the buildings by you'll be we are using their WLC as in their wireless access controller We are using their access points and we are putting more access points Which automatically register with the WLC and they then sorry will be Managed by the WLC automatically, so we just put more Access points for example in Jean-Saint. We put a few we don't need to put as many as we used to of course Things settled down and stuff got better For the video we do everything ourselves For all the cables you see here for the camera back there and all the other rooms we do everything ourselves But we are trying to use more and more you'll be infrastructure But obviously in a way which makes sense for them and for us and so we can rely on this being still here and working next year Which obviously when here when you're here once a year is kind of an issue, but for wireless it would be it would be Close to a week of extra effort to put Wi-Fi in all the places So we have to use that and it's really good that we are allowed to three more minutes Can you know on the website to Live to live feedbacks to the different tools It seems to me that also last year RECAPCHA was used as a method to verify the Humanity of users. There is any plan to Substitute these things with an external services with an internal services Because I think it's a bit worried to that the fact that I have to contribute to some machine learning things I recognizing traffic lights and so Okay, leave my feet back. Yeah, gotcha. So the question just repeat was We are using RECAPCHA on on the feedback side to verify that humans are humans Obviously Google has been using this for training their AI systems for years like identify a storefront identify a car identify a dog Which is just training data for them to to use for the for their AI And this is basically giving stuff away for free at a free and open source conference And it's on giving to one entity and the question was if we would be able to host us ourselves The answer is probably yes if this assists I have no idea to be honest We kind of need it because people were being nasty about about that form it's The extra effort to really do it from scratch is too much for us to handle But if there's some drop-in replacement, we would most likely be using this instead So if you have anything send it to feedback at FOSDM org and we can have a look Anyone else One minute the question was if we ever had any security incidents and how did we deal with them? Yes and privately So I mean I have this in the closing talk slides That in 2017 we felt the need to to tell people which which SSIDs we are using Last year we felt the need to talk about how how access points look and not look this year someone tried to ARPS both our spoof or our poison the the local Wi-Fi caches for for Putting basically they took the MAC address of our default gateway and used ARPS Moving to to grab the traffic and just see if any random passwords floated by It's random bullshit from script kitties, so If any of you is in the room or you know one of those people it's Yes, we can also do it and it's boring and it's shite and it's a free and open source conference Just don't do it. It's it's bullshit Thank you. Thank you