 All right, I think we're gonna go ahead and get started. Yeah, that, Mike, you call or are you gonna? Yeah, sorry, we've got the red dot. I was looking for where that was. So yeah, I'd like to thank everyone who's joining us today. Welcome to today's CNCF webinar, use open source bare metal and 5G to achieve autonomous drone delivery. That's exciting. I'm Paul Burt. I'm a technical product marketing engineer at NetApp and cloud native ambassador. So I'll be moderating today's webinar. I'd like to welcome our presenter today, Cody Hill, Field CTO at Packet. And a couple of housekeeping items up top here. During the webinar, you're not able to talk as an attendee. There is a Q&A box at the bottom of your screen though. Please feel free to drop your questions in there and we'll get to as many as we can. Cody sounds like he's up to the challenge of answering things as they come in, but if your question doesn't get answered live, I'll sift through them and make sure he gets a glance at them at the end. And this is an official webinar of the CNCF. And as such, it is subject to the CNCF code of conduct. So please do not add anything to the chat or questions that would be in violation of that code of conduct. Basically just be respectful to your fellow participants and presenters. The recordings and slides will be posted later today on the CNCF webinar page. That is cncf.io slash webinars. And with that, I'll hand it over to Cody to kick off today's presentation. Yeah, thank you very much, Paul, really appreciate it. So as Paul said, I'm Cody Hill with Packet, Field CTO. And we'll be talking about the work that we've done today, that we've done with Sprint on their IoT curiosity platform and kind of diving into a use case that we have with drone delivery. So a little bit about me, it's a classic slide that all presenters give. I've been in the tech industry for 15 years, started out at an ISP, I was the guy telling you to unplug it and plug it back in, and kind of worked my way up into being a cloud architect at General Electric. And then I have been on a few startups here and there. And, hey, Mark, if you wanna go on mute, that'd be great. Thanks, yeah, so I've been in a few startups here and there and I joined Packet in the fall of 2019, but I've been a customer of Packet since 2015. So great product and so I'm happy to be a part of it. So as Paul mentioned, I'd encourage you to ask me questions, right? The Q&A inside Zoom is there. So go ahead and drop questions in and I'll go ahead and try to answer them as I go. I'm gonna get somewhat geeky and techy on some of this stuff. So if you have questions, I wanna make sure we catch everybody up before we move forward. So go ahead and do that. So here's the agenda for today, right? We have a quick introduction about Packet, a little bit of talking about what the Sprint Curiosity Network is and how that kind of pertains to Packet. And then we wanna talk about how you could deploy physical servers on the edge of Sprint's Curiosity Network. So you get extremely low latency to the end devices there. And then once you have these servers, right? Nobody really cares about physical servers, right? They care about the applications that run on those servers. So what does it take to turn a normal server into an IoT warehouse and analytics tool so that you can start analyzing your data from all your IoT devices? And then we're gonna basically show you how to put that software on the hardware and do a live demo. And then everyone's gonna leave today with the GitHub link to all of the code. I'll be demoing today, as well as a promo code to get into Packet and get some free compute credits and try it out yourself, okay? So a little bit about Packet, right? So as of last week, we have just been acquired by Equinux, right? So we're no longer a small startup company. We're now part of Equinux and we're really excited about that. And we have about 20 plus cloud locations, about 130 members on the packet side of the house. We were founded back in 2014 by our CEO and a few others that have been in the infrastructure space for quite a while. And the cool thing about our platform is it's bare metal that you can spin up just like a cloud, right? So you get a server in as little as 60 seconds and we have over 60,000 servers being deployed and destroyed every month, right? So quite a bit of where I'm chairing that hardware, spinning it up, tearing it down. So it's pretty cool. One of our marquee customers is Sprint, right? And Sprint had the opportunity to say, let's throw out everything that we know about a cellular network and let's start from scratch and really build a network that's for the modern day. And let's focus this around IoT and an edge use case, right? So they came out with what they're calling their Curiosity Core and they could have gone to any of the public clouds or try to build their own data centers and do all of that, right? Cause they're Sprint, they're massive, they can do it. And what they realized is that their core competency isn't hosting data centers or dealing with servers. Their core competency is that networking side of things. So they partnered up with Packet and what Packet is allowing them to do is they put their finger on a map anywhere in the US and they say, I wanna have a 5G network in this city. And Packet will do all of the things necessary to deliver them a REST API with physical servers that they can provision with all the networking they need in less than 90 days, right? So they can go into any market, anywhere in the US and Packet's on the hook to deliver that. This has become a great partnership and I think we're live in 22 cities around the US now and more to come in 2020, right? So that's what Sprint's doing with Packet. And so we decided, let's show some people how to utilize this IoT network and leverage Packet and all of that, right? So this is, you can have it however you like, right? So you can create your own servers on Packet. We have a nice web UI off to the right here. You can use a REST API, that's an actual curl command that will spin up a server for you or use the automation framework or language that you want, Terraform, Ansible, GoLang, Python, right? Whatever language or tool you're comfortable with, you could spin up physical servers on the edge of the Sprint Curiosity network that will get you extremely low latency access to your end devices, right? One of those are drones that are transmitting a lot of data or standard warehouse machinery that are sending temperature and things like that, right? So you could spin up those servers on the edge. But servers aren't, like I said earlier, aren't the most exciting thing about this, right? A server without an application is useless, right? So it's really about the IoT software that goes into it. So we worked with some friends of Packet and really smart folks, and we put together what we think the best of breed open source technology is that will allow you to build an end-to-end IoT pipeline and really capture, store, visualize all of that data so that you can really leverage all of the data coming off in this case, drones, right? So you can see the drones at the bottom and really what they're doing is they are transmitting a JSON payload to emitter, which is an open source MQTT broker that is then tied into the OpenFaz and Alex Ellis helped us out with this from OpenFaz and it's tied into different topics inside of that. And then certain functions inside of OpenFaz are doing inserts in the databases, we have functions to render a map so we can see where these drones are live on a map and I'll show you that in a demo in a minute. And then we can actually read the data out of the Postgres database as well. And then kind of for free, some of the things that you get with OpenFaz is you get Prometheus monitoring of all of your functions and all of that. So every time you adjust something that goes into Prometheus, you get some nice Grafana charts to show you how that ingestion rate and all of that is working. And then we've layered on Metabase which is a business intelligence tool and Mapbox which is a visualization tool. So this is all kind of layered together to kind of get you a lot of visualization and some intelligence into that data. So it's kind of break these things down for a minute, right? So the first piece of this is Kubernetes, right? So all of this runs on top of Kubernetes. In this example we chose K3S because K3S was specifically designed to be the Kubernetes at the edge, very minimal installation of Kubernetes, very lightweight. So we said, hey, that's a great use case to deploy that, right? So I assume everyone on this call is familiar with Kubernetes, right? But we can cover it a little bit, right? So Kubernetes is an open source container orchestration framework, right? So whether you're using Docker or just container D, it basically orchestrates deploying your container workloads across one or multiple servers and then handles all of the plumbing from getting you services, your services delivered to it, you know, ingress access. It can handle scaling those in an automated fashion and really allows you to care less about your physical infrastructure and kind of let Kubernetes orchestrate all of that for you, right? So pretty cool application framework works really good with all of the components that we have here and it's really the way the industry's going. Everything is being built to run on top of Kubernetes so why not build ours that way as well, okay? So the next piece of code that we need is emitter, right? So emitter is an open source MQTT broker and one of the things that we really liked about emitter was that it's horizontally scalable which you don't get in a lot of MQTT brokers, right? So as you need to scale up the amount of ingestion points you can just create more and more replicas of emitter and it'll handle that scaling for you. And then it was built from the beginning to be super fast and secure, right? So every single topic inside of the MQTT broker has the ability to be protected by an authentication token. So it really keeps you from, you know it keeps your data secure and your ingestion points secure. So we like emitter and then to kind of the glue and kind of the heart of this that kind of puts everything together was open fast. We said, we could possibly write some custom code and build our own containers and build our application to ingest the data from emitter and dump it into a SQL database and then build some code inside of our app to render a database map for Mapbox and stuff like that. And we said, you know, why not use a serverless framework such as OpenFaz? And so that worked out really well. And with OpenFaz, it kind of office gates all of the complex things inside of Kubernetes and allows you to just kind of focus on, you know your code, the functions that you want to deploy and distills everything down to the Docker container level, right? It makes the barrier to entry to consume Kubernetes pretty low, right? So we really like OpenFaz so we decided to use that here. Next is Prometheus and we kind of talked about that, right? So if you're not familiar with Prometheus, right? It's a cloud native compute foundation project, right? So go CNCF and it's a monitoring system that will basically scrape your data in point. So it actually reaches out and collects data for you, right, other than you having to shove data or push data into it. So all you have to do is host a REST API and it'll start scraping that REST API for information, right? So it's gonna be pulling in your metrics and then underneath it ultimately it's a time series database. So it's very efficient to store all of the metrics that are coming in and then display them on a graph and do those types of things, right? So Prometheus is probably, I think has won the format war for modern monitoring systems and it's great in this scenario and we'll kind of show a little bit of that in a minute. And then Grafana, if there was a war for visualizing all of this data in an open source way, Grafana definitely has won that. And Grafana is a great visualization and analytics engine. It allows you to tie in almost any data source in this case for using Prometheus, we could use Postgres, but almost any data source and then you could build beautiful graphs and charts and all of the things to kind of visualize that data and kind of do trends and all of those things. So it's really great to store all of this data. We chose Postgres and we chose Postgres because it's been around for a long time. It's really stable and we didn't want to get extremely creative on that aspect of it. We could have chose Cockroach EB or Ugabyte or some of the newer cloud native scale out databases, but we just wanted to keep it simple in this example. So we chose just a standard Helm chart for Postgres and we went and deployed that. And if you're not familiar with Postgres, it's been around for a long time at an open source relational database and has come a long way and has been adopted by quite a few cloud thinking and scale out database companies such as Cockroach and Ugabyte DB, they've standardized on Postgres QL in that. So in this example, if you did need to spread this data into multiple geographic locations, all of this code would be easily replaced by just replacing that database with Ugabyte and now you have a distributed system as well. So Postgres is a good option here. And then we have Mapbox. So Mapbox is a really cool piece of software and really what it allows you to do is visualize any type of geographic map data and it kind of overlays it on top of Google Maps. You can then put different types of points and things on that map. So in this case, we want to visualize where our drones are going. So our drones are starting at the hangar. They're then going to make their deliveries at a couple separate warehouses and they're going to come back to the hangar. And we kind of want to see what that looks like in real time and where those drones are. And Mapbox really helps out with that. So great tool there. And then lastly is Metabase. So Metabase is a business intelligence tool, right? It's open source. And one of the cool things about it is that it starts making some intelligent decisions for you, right? So it starts correlating data. It says, hey, this looks like coordinates data. Let me build a map and put that on there. Hey, does this map look good to you? Oh yeah, that's great. I didn't need to build that map, right? And it's going to start seeing different trend data with maybe the battery is starting to deplete automatically on these drones. And it sees the same pattern. It's like, hey, you probably want to see this on a chart next to all these drones. And it'll build that chart automatically for you. And so you just need to tweak a couple things here and there, but it's actually an intelligent business intelligence tool, right? So you don't have to be the only one bringing intelligence to the party. So we really like Metabase for this. It was great. So now we talked about the software. We talked about deploying servers at the edge in the past. So how do you actually get this thing deployed all together and marry this stuff together, right? So what we did is we built a Git repository, right? You guys can all go to this thing. It's github.com slash packet labs slash IoT. And in there, there's a Kates directory, right? So inside that Kates directory, there's some Terraform scripts. And off to the right, the screenshot of what it looks like when the Terraform playbook is finished. So basically, you go in and say Terraform Apply. Get to fill out a couple variables, right? But Terraform Apply, it kicks this thing off. In the end, you now have the exact stack I'm talking about up and running, ready for you. And you can start sending metrics to it and start visualizing it. So anybody that wants to play or test with this after the call can definitely do this. And at the end, we'll be giving out a free promo code on packet. It'll get you one of our small servers for about 60 days, right? So it's a decent amount of money to go play with some of these servers, right? So we actually jump into a live demo now and kind of show you what all this looks like. And I don't see any questions yet. So if you guys have any questions so far, right? Make sure you're using the Q&A channel. I'm not looking at the chat. I'm dropping any of your questions into that Q&A box and I'd be happy to answer them, okay? All right, live demo, let's do it. So the first thing I'm going to do is I'm going to deploy a new software stack, right? So this is our software stack. We have Terraform Apply and the question, I just got a question in the chat is why use Trafic instead of Istio? And the short answer is that Trafic is built into K3S. It fit all of our needs and completely worked. So we didn't decide to deploy K3S without our ingress controller and replace it, right? Istio is great, but Trafic work good for our use case. Cool. So let's go ahead and deploy Terraform Apply and we're going to do an auto approve here. So the thing is going to spin up, create a new system and we're going to flip over to the packet UI and kind of watch this thing happen, right? So here's our CNCF project inside a packet and we should see a new server pop in here in just a second. There it is. So we're building K3S01 and so this thing is going to spin up and while that spins up, we do have another question. Let's see, correctly in itself, defending since there is no mention of security. Yeah, so I didn't really dive into the security aspect of this, we do have a cert manager on this so you can, you know, we do have TLS encryption over this thing, but yeah, we didn't really dive into protecting this pipeline, right? There's a lot of things that you can do, you know, Istio from a service mesh standpoint would be great. Really we, inside of the presentation we threw in the cert bot and all of that to keep the edge kind of secure and at least SSL encrypted, but yeah, there's definitely a lot you can do from a security standpoint to keep this app more secure. No questions out there. As well as the emitter, we chose the emitter that all of your MQTT data is authenticated as well. Great, so we have this system up and it's spinning up. So we have another environment that's already up and running and I kind of want to drop you through all of these things, right? So first is the OpenFAS portal, right? So jumping into OpenFAS, you could see we have the render map function, we have our database reader function, we have our database insert function. We also have our MQTT publisher function that's bringing the data from emitter and shoving that into the database or into the queue so that we can insert it into the database. So that's great. You can see that some of these functions have been called 24 times, database readers have called quite a bit more than that. And then all of this data goes into Prometheus and can then be visualized right here, right? So you can see we've had almost 72,000 total requests through OpenFAS, right? And so we allow the quick visualization, you get this for free using OpenFAS, what's your function execution rate? How many replicas do you have? How long does it take to the duration of this? All of that. We have got another question in the chat. How do you remote boot edge hosts with an operating system, right? So that's what Packet does. The entire company was based on let's build a bare metal cloud. And so there's really no secret sauce to provisioning a physical server, right? You need to connect to it via IPMI and power it on. It's then going to get a DHCP IP address. And in the DHCP tags, it's going to send it to connect to a Pixi server, right? In our case, I Pixi. At that point, everyone knows how to do that and you can then load up Red Hat has instructions, Ubuntu has instructions, all those folks have instructions on using an image from them and using a Kickstarter file to deploy an operating system. What we've really done is put a lot of our intelligence in to that boot environment that comes in. We've basically made it a dockerized environment that will allow you to make a lot of really smart intelligent decisions at that point to deploy the server. And then we built a metadata service that coincides with that so that you can use things like cloud init and user data to actually turn bare metal into a cloud, right? So that's kind of the secret sauce behind packet and what we do and that's how we provision those servers on the edge. So once you have your Grafana up and running, hey, we have a lot of cool stuff, we can actually jump in here and visualize what we have for Metbox, right? So you can see here we have Northwest Vegas, we have a warehouse, we have a warehouse over here in Northeast Vegas. And we have a cluster of drones here on top of the hangar, right? That's where all those guys are. So if we wanted to send those drones on their mission, let's go ahead and kick this guy off, right? So we're gonna kick off our test pipeline to deploy these drones. And we're gonna see in just a minute, it takes a minute to build the flight plans and send all the instructions out. But these drones are gonna start taking their flights over to their respective warehouses to make their deliveries, okay? So that takes just a second, so we'll come back to that. These are the instructions that are being sent back and forth, right? So it's telling everything where to go and it should be refreshing if that's going. Let me just refresh this real quick. I don't know if I gave enough to the demo gods this morning. Oh no, oh no. Great, so these guys should be painting on the map. They're not, at least close this warehouse and see if this still works, it does not. Okay, so that's great. So this is supposed to be showing drones flying from this hangar to the Northwest Vegas deal and then from this hangar to the Northeast. You can then close the warehouse and all these drones will come back home or you can grab an individual drone and click on it. You can see the battery life, where it's going, its speed, where it's coming from and you could send it back to the warehouse if you needed to. So it's unfortunate that this isn't working. So it says they'll return to hangar and let me try this one more time and I'm gonna give up on it. So while that's going on, we'll check on it in a minute. This is all the visualization data from when the drones are making their run, right? So this is what Metabase does for you. The previous deal was Mapbox. And it basically shows, it starts building graphs for you that are pretty smart, right? So this is one that we didn't have to really mess with and it's just any warnings or errors that drones are sending out and where they were when those errors or warnings happen. So you could see that we have quite a bit in this area here and wondering what's going on. Well, if we go up here and look at our high wind map, this is a very high wind area inside of Vegas. So these drones are having issues staying on course. There's a lot of wind going on there, right? You could see the average payload versus battery drain, right? So if the payload is much higher than the battery drain and they might end up running out of battery life before they can make their delivery, right? So this is part of understanding these and that's why the controller is a bi-directional communication, right? So it's sending information again, the map's still not working. And so we can actually, with that data, we can make that quick analytics and say, hey, this drone's not going to make its delivery because its battery is draining way too quickly. Let's bring it back to the hangar before it has an issue. So one of the other things that the controller does is say these two drones are on a collision course. Let's avoid that. Let's move them out of the way so they don't collide with each other, okay? And so you get all of that and this all basically comes for free with very little work inside of Metabase, right? So you can see all of this. Pretty cool charting. There's a few other things that you can dive into here, right? You can go directly into Postgres, look at the drone position table. Sorry, I shouldn't have clicked on that. There's this lightning bolt to X-ray this table. That's something that they call. And so there's just more analytics that they're building on the fly on just a database, right? So you can start seeing the drone's position by bearing, drone's position by battery drain. All of these things are just charts that they're just building for you thinking, this might be useful. I see some trend data here and they're putting them on a chart in an automated way for you. So pretty cool stuff that Metabase does. And you could see that this environment is now done. We've spun up a brand new environment. If I jump over to packet, you can see the servers online. And let me cross my fingers for the demo gods. Hey, we have a new map rendered for that new endpoint, okay? So that works. I am going to try to make this one work and pray that it works. So let's try that. So let me SSH into, you can do it. I know you can. I don't want to do too much of this live. It looks like I might be having an issue with some of my environments. So that's great. It's up here though. So I'm not sure why. Oh, and now it's not. Great. So it looks like I'm having an issue. We can figure that out later. I would like to go back to any questions you guys have and back to the slide deck here. So basically, it's the end of the demo there, right? So what we do want to do is for anybody else to try this out and we might need to do a couple pull requests to get this thing up and running is go over to bearmet.al slash IOT that'll take you to that same GitHub repo that I was showing. And when you sign up for your packet account to deploy this, if you use promo code curiosity 100, it's going to give you $100 in free cloud credits so that you can either try this or any other project that you're working on. You can actually spin up some free compute credits and there are some free servers and test them out $100 worth and do your thing. So that's all I have for today. Any other questions, go ahead and throw them into the Q&A window. Yeah, thanks, Cody, for the great presentation. So we do have some time for some questions. So there's a Q&A tab at the bottom of your screen there and you can enter your questions in there. I see one question or more statements that are questions like demo gods, exclamation point. So since that came up multiple times during the talk here, well, people think of real questions. What are the traditional sacrifices that you make to the demo gods, Cody? Yeah, you know, normally I get down on my knees and beg them to make it go well. That's normally my repertoire. That sounds like a good one, maybe RMRF, some directories here and there, that sort of thing. Yeah, I fully tested the demo into this morning and it worked without a hitch. So why wouldn't it work live? No, not sure. Yeah, it happens to all of us. I think it happened to me on my last live demo. So I think you have more support for this sort of stuff than may feel like in the moment. But we do have one question that's come in. So how do you manage multiple edge clusters? That's from Binu. Yeah, so it's a good question and there's a lot of different thoughts out there. So in my opinion, I think my best efforts are spent higher up the application stack than managing Kubernetes itself. So I would go with a vendor-supported model, right? So folks like Rancher, folks like Platform 9, anybody else that's in the Kubernetes ecosystem and participating, I would go with those guys to help me kind of manage those clusters around the globe so I don't have to build a huge operations team to do that. I think a different question might be, how do you interconnect those clusters together and how you allow them to communicate inside of Kubernetes? So Istio is a great option for a service mesh and they now have the ability to, I think it's called Istio gateways, where you can have multiple separate clusters that are then, you can then map pods so that they can talk to each other through that Istio gateway and a shared service discovery across that. HashiCorp Console Connect can do that as well, right? So they have a great option there to do multi-cluster Kubernetes to allow your services from both of them to talk in a secure fashion. And then a new project that Rancher just came out with is Submariner, which is really cool and interesting. I like it a lot. And Submariner basically creates IPsec tunnels between an IPsec mesh between one to many clusters and allows all service discovery and pod discovery across to all of them. So it's a pretty cool tech out there to help you interconnect these clusters but actually doing the day-to-day operations and if you're doing upgrades and all of that, I'd be looking for a Kubernetes control plane which there's quite a few available out there. And the Rancher one's free, you just pay for support. That's a great point on thinking beyond just the install and looking at the day-to-and ongoing maintenance and all that. We have some other requests in the chat. Number one is, what is that GitHub URL that's from Ed? Yep, so the quick one, the short link to get there is bearmet.al. So bear, like bearmet.al, but there's a dot right here, slash IOT. So that's a short link to that GitHub, yeah. Great, and then we have another request for a slide previous in the deck. Cedric has requested, you can throw the chart up showing the architecture. Yeah, so this is also on that GitHub link. You can see that picture. Let me present this again and make it a little larger so that you can see it. I've got a few talks. And then we have another question from Binu. He has a question about, is AMQP possible in K3S, going for the older protocol to MQTT? Yeah, so K3S isn't what really cares or manages the AMQT broke or AMQP or MQTT. It's more of a plug-in to OpenFaz. And to be honest, I don't know for sure if OpenFaz has an AMQP broker off the top of my head. We'd have to Google OpenFaz AMQP. I'm sure it's there. I do know for sure that like another serverless framework that's called Fission, fission.io, which is a Kubernetes based serverless platform that's fully open source. I do know they have an AMQP broker for sure. So if you are using AMQP and OpenFaz doesn't have the connector for you, check out Fission. They might be a good option. Very cool. We have a question from Prakash. OpenFaz, how does it help you simplify your Kate's cluster implementation? Yeah, so the Kate's implementation doesn't make Kubernetes any more, the implementation of Kubernetes any more easy or difficult. What it really makes is the learning all of the constructs of Kubernetes, right? Deployments and auto scalers and services and building your Docker containers and all of the things that you have to the barrier to entry that a developer needs to figure out to consume Kubernetes is not low, right? It's not just, hey, here's Kubernetes go and they're fine. It's definitely not that way. And so what OpenFaz does for you is it gives you a much more graded way to get into consuming Kubernetes where you're really just caring about the functions that you're writing. This is my code. I'm going to write that code. I'm going to deploy to OpenFaz. And OpenFaz is going to figure out how to distribute it for me. It's going to give me an ingress point so I can execute that code. I could do all of those things and I don't need to figure out how Kubernetes specifically does all that for you. Very cool. We'll close on this last question. Cy asks, what is the latency across the emitter and drone? So it obviously depends on the distance from the drone to the tower and then from the tower to the server, right? So that's the big thing that we're, that packet and Sprint are shoring up is we're putting these servers as close to the tower as possible so that you can really get sub millisecond latency from the termination of the tower over to your server. Now the drone's transmitting data. You're probably dealing with a couple milliseconds there to depending on, are they a mile away from the tower or five miles from the tower? It's going to change things, but not dramatically. So round trip, we're definitely talking sub five milliseconds, right? Which is pretty good, right? And in a lot of cases, you're sub three milliseconds. That is very cool. Well, I need to prep my scripts but you had a code at the end there for everyone to go check this out and get some credits on packet. I love stuff like this where we get to see tech kind of interact with the real world and I'm actually personally going to bookmark this code because I'm excited about the ARM 64 stuff packet has available for me to tinker with. So I'll be using it on that. So with that, we'll be closing things out here. So great, thank you, Cody, for the great presentation. And all right, that's all the questions we have time for today. Thanks for joining us today. The webinar recording and slides will be on later today on the CNCF website and we're looking forward to seeing you at a future CNCF webinar. Have a great day, everyone. Thank you.