 All right, I guess we'll go and get started. How's everybody doing? Good, yes? All right, I'm Dustin Ruhl. I'm from ECS team, or as some folks have said to me, EC Steam. It's kind of funny. This is Tom Collings. We're both senior cloud architects in our cloud practice. And we're here to talk about today the fire hose a little bit. One of the things that we've kind of noticed in our cloud foundry journey, our cloud native journey as a company, is when we interact with folks, there's a lot of people that are doing a lot of really cool things, as you've heard here, with cloud foundry. Everybody says, and they nod their heads, yeah, the fire hose. I know what that is. I know exactly what that is. But then you start to peel back the onion, and it's like, hey, wait a minute. And I went through this myself. Well, I don't really know that much about it. I know what exists. I kind of know what goes into it. So hopefully this talk kind of either reiterates what you already know, maybe a little bit different. You learn some nuggets. Or this is totally cool because you learned something that you didn't know before you started listening to us talk. All right, well, hey, you've got cloud foundry. You've got your cloud native developers. They're building apps. They're building a lot of apps, hopefully. Because as you probably noticed, this platform is very, very good at allowing you to go fast, easy, and safe. Maybe you're a little more mature. You even got a CI CD pipeline, several of them. And your apps are going from foundation to foundation, different orgs, different spaces, lots of testing going on. So the whole point here is you've got a lot of activity. So how does that make you feel? Feel a little bit like gridlock? I mean, you've got all these new platform components. You hopefully have all these new and old apps that look a different, the kind of behave in a different way, cloud native. Maybe you heard a lot about that, maybe you haven't. And you've got a lot of users. And it's just, what do I do? How do I, you know, you feel a little bit more like I'm in a jungle. And then you start thinking, how do I know the health of my platform? How do I know the health of my apps? What if I want some statistics on some of these things? Right? How many people have asked these very same questions? Show of hands. Come on, it's interactive. It's almost lunch. We're not in a food coma yet. Right, so good. How many people have actually gotten into some detail on the fire hose? Oh, it's great. All right, so hopefully we'll be teaching a lot of people some things today. Well, you've all heard what the loggergator is. That's one big giant subsystem, if you will, of Cloud Foundry. And it has a bunch of components. Okay, confused yet? Well, really what you have is a bunch of sources. And those are apps. Those are, you know, things like the Diego Brain, the UAA, the Cloud Control, a lot of different components, all the components of Cloud Foundry, and apps are considered sources. All right, well, what's next in this chain? How do my metrics and logs flow through this loggergator? Well, for every job, or VM, whether you like to talk in Bosch terms or IaaS terms, you have a Metron agent, and that's the collector of all of these sources, multiple sources on a cell, for example. Then you have another kind of collection point. Think of that across AZs, okay? You have Dopplers, all right? And then on top of that, you have Traffic Controllers. How many layers of abstraction here? Well, actually, they're all pretty cool because at the end of it, there it is. There's a fire hose. So you hear loggergator, like what is that? And you hear a bunch of other little components, little agents here and there. They're all basically coming down, down to the end there at the fire hose, okay? Which pill you wanna take right now? For those of you that's already, done a lot with the fire hose, you wanna take the blue pill? Or do all of you wanna take the red pill? How many reds? Really? Oh, come on. How many reds? How many blues? How many people don't care? Awesome. All right, Tom, make a note, we're gonna have to make this a little bit more interesting. So the fire hose is just a big, giant multiplexer, right? You got a bunch of inputs going in there and really what happens is at those sources, it gets turned into what's called the drop-sondie protocol. Who here has heard of protocol buffers from Google? Excellent. All the drop-sondie protocol is on top of protocol buffers, binary wire format. So it's very efficient, very compressed and messages are flowing through and drop-sondie, okay? Well, you kinda hear that there's a lot of stuff in the fire hose, right? How many here think everything's in the fire hose? How many think most is in the fire hose? Oh, good. Well, that's actually true. So while it's a journey for us, it's also a journey for the teams that are building these, they're continuing to put more and more in the fire hose. So for example, right now, any of the syslog components that live, or excuse me, any of the events that go into syslog on a VM, right? Those are actually coming out of our syslog. So you won't see those in the fire hose. So if I'm going to SSH into the cloud controller and I have a failed login attempt, right? That's a security event that a lot of people wanna see. That's not gonna show up in the fire hose yet. And that's why I have the roadmap on here. How many people have actually been on the website and seen the loggerator roadmap? Yeah, we got the product manager in the back. I would hope he's seen that. Well, on there, I encourage you to go on the loggerator site, there's things that loggerator roadmap, and Jim is very good at showing what they're working on. And one of the top items that I saw they're working on or about to work on is actually getting that R-Syslog information into the fire hose, which is really cool, okay? So, how many people knew that's where, if I did cflogged my app, that's where that came from? All right, see, a lot of you guys learned something new. So you sit down, just like you're normally developing, you do cflogs, that's another source. It doesn't attach to the fire hose, at least not yet, but right now it comes off of the traffic controller. Now, the key point on looking at this traffic controller before we continue on, notice it's a web socket. We're gonna come back to that in a little bit when we talk about consuming the fire hose. Okay, so let's take a look. What is that? Oh my gosh, there's no apps in this. This is just the platform. Look at all that. That's the fire hose right there. Lot of messages. That feel a little bit like this? Thought so. Who hasn't drank from the fire hose yet? Exactly. Well, I know, and this is a little tidbit of information, you can walk out of here that's useless. I have a lot of that in my head, right? So that was what you saw right there, came from Bosch Light running on this laptop, and I have Diego on there, so it's about 19 VMs, containers inside that vagrant VM, okay? 60 seconds. Anybody wanna take a guess about how many megabytes that was in that minute? Come on, let's hear some numbers. Two, 16, 20, 200, wow. Okay, well, that may be true. I'd be a little nervous if it was 200 meg on my laptop. What's that? Ah, nice, all right. Touche, well, on my laptop, without debugging turned on, running on Bosch Light with those 19 VMs, it was actually about 10 megabytes a minute. That's nothing. Why is that important? There's so much information in there that anything that you wanna find is gonna be in there. And at some point in your maturity path through the cloud native journey, you're actually gonna find that a lot of that information is gonna become useful at some point, okay? All right, so there's a few of you here that have dealt with the fire hose. You already know the answers. How do you consume this? I've got this web socket endpoint. It's throwing a lot of information. Well, the answer is nozzles, okay? We as our inner nerd, we like to put a lot of nice metaphors around things. I happen to think this is a great metaphor, okay? So, if I'm connecting to the fire hose via nozzle, there's kind of two main kind of configuration that you need. One, you have to have a user, because it's privileged access, that has a fire hose Doppler scope, right? A lot of people when they first do it, they just hook up admin. I probably wouldn't do that. So if you're operationalizing yourself and you haven't already done this, you should create a separate user for anything that's consuming the fire hose. You also have the endpoint, the web socket endpoint that connects to the traffic controller. And it's on my Bosch light. It looks a little bit like that. I'm gonna interrupt you there. Don't wanna glaze over the fire hose user. A lot of information has come through this fire hose here. There could be some PII, PCI kind of stuff coming through here. So you wanna guard this fire hose Doppler user pretty well, because that application's gonna have access to some information that you may not want leaking out. See, leak, fire hose, nozzle. Hey, there's the metaphor. Nice. So a little nugget, that's all discoverable via the CFAPI, right? And there's the endpoint right there. So if you're doing something programmatically connecting, right, you don't have to feed it all these environment variables. You can do some discovery, which is kinda cool. All right, so what now? I know how to connect. I got a rudimentary nozzle that I'm gonna connect. Well, remember we talked about drubsondi is the thing that's flowing across there. So what is flowing across? Really, it's envelopes containing events. Okay, that doesn't sound very exciting. They come from the traffic controller. Again, not very exciting. All right, getting a little more interesting. I got to un-martial that from that drubsondi format. How do I do that? We're not gonna go into details here, but what we're gonna demonstrate a little bit later, we encourage you to go, and what we're talking about a little bit later, we encourage you to go look at the source code. It's actually pretty straightforward. They've done a great job on that team, writing clients to allow us as a community to consume the fire hose and do something useful. Well, these are the event types. A lot of those have a lot of interesting aspects in their own context. Ones that are going to be, we're gonna look at today are log message and container metrics. Okay, and I'll get into those here in a minute. So now let's kinda talk a little bit about the life cycle of a nozzle. How do I, what do I do with these nozzles? How do they live out there? Well, kinda there's three primary ways that you can do this. One, you can create a nozzle as a Bosch deployment, right? Getting your manifest, Bosch deploy, Bosch release, create a release, upload the release, deploy it. Another cool thing you can do is you can turn it into an app, and then just CF push, and we love that. Well, there's also some plugins, and we're gonna show you that actually the little video that was showing you all the nozzle right there, excuse me, all the fire hose messages, that was the fire hose plugin, and from the command line, CF nozzle, and I'll show you that here a little bit more in detail. Leading into, here's some examples, right? So the demo that we're gonna show you and the source code that you hopefully go look at after this at some point, you're gonna see that we based what we did, and I hope a lot of people do in the open source community based on what other people do. I don't wanna redo the same work that somebody's done. I'm lazy, or at least that's what everybody tells me. So a couple of very cool nozzles. I like our NOAA, that's kind of the base one, and then fire hose to syslog, and I'm gonna show you why it is one of the very reasons that it's cool. There's a lot of reasons, but it's a very cool nozzle. All right, so let's look at a message or two. So we do our CF logs, spring music, this is just a router message, I get it back at 200, tells me I'm successful, right? Well down below is me, CF nozzle, filter, log message. Now doing this, I'm gonna see all the app log messages, just so it happens on my laptop, I only had spring music running. So you can imagine how much information's coming in here. You can start to think about, if I wanna operationalize this, I have to segregate my log data and create security models around it so that only certain people can see that data for the very reason Tom pointed out, PII, PHI, hopefully not PCI, but hopefully you can protect that information. Well, let's kinda look at some of these, how are they made different? Maybe you kinda saw that, maybe you didn't, but looking in the red, you see a nice little RTR, you can kind of, all right, it's router, that's what I'm used to seeing when I'm a developer and I see if logs. Well, you get the job name, for example, in the firehose message, but you see there are a lot of things that are also very similar, right? I see the same get and the same URL and the same HTTP response, they're in both messages. So it's not magic, it's not voodoo, right? It's just a lot of information. All right, let's look at an app log message, and this is leading into one of the cool features that I love about firehose to syslog, and hopefully you will too, and maybe you guys have already experienced this pain, right? But if you notice in the log message for an app, you see that big, long good, right? Everybody see that? You kinda see the start of it right after app ID. So what do you think happens to that in, say, a blue-green deploy? Anybody wanna guess? Changes, that's right. So now if I wanna look at something a couple weeks ago, I have to remember that very long alphanumeric ID or create a meta table, right? That's all well and good, and there's very good reasons for doing that, but on an operational side, how do we solve that? Use the firehose to syslog nozzle, right? You're still gonna put that into your syslog aggregator, a splunk, a log insight, whatever, and it injects the app name into that very spot, which is very cool, all right? So how many here have experienced that pain where that change of the good? You're like, how do I go back? There's a few. Well, I encourage you to look at and implement firehose to syslog, because that will fix that for you. All right, we've done a lot of talking about how do messages flow through the log aggregator into the firehose? How do we consume the firehose? It's really easy to kind of think of a use case that, hey, I wanna look at my logs, right? Everybody wants to look at logs, but where the power of this firehose comes from, we wanted to kind of, you know, yeah, we could go down that rabbit hole and kind of be monotonic on some of that stuff, but what we really wanted to kind of open your mind to is feedback control loops. This firehose is great for feedback control loops. Anybody have any background in here in manufacturing? All right, a little bit, that's good. So in manufacturing, you're producing something, you measure something, oh, it's out of my tolerances. The cereal's too sweet or too big or too small. Now I gotta go back to where the thing that's producing and I turn some knobs and I turn that into an improved process, right? So it comes out the other end in a more controlled way. Well, the same thing can apply here to your apps. Okay? And we've all probably heard of, you know, some of the feedback control loop systems that are out there. There's a CPU auto-scaling, right? That's feedback control for your apps. There's, what is Kevin Hoffman's nozzle that you built? Oh, the app usage nozzle. If it's been an idle app for a long time, we'll go ahead and delete it for you to keep your AI accounts down. Yep, so there's another type of feedback control loop. Well, today, we're actually gonna demonstrate memory-based auto-scaling, okay? And in this demo, there's kind of a couple of main components. One, as you see, is the memory-based auto-scaler nozzle and we deployed that as a CF app on Bosch Lite on my laptop, okay? And why I have that dash line coming out of that green plus, that's the control point, right? That's where we're gonna take some information coming out of that output. And the B is an algorithm that says I need to do something because I've exceeded some threshold and what's that something? I'm gonna scale my app. So really, I've got a container metric coming out for a dummy app, okay? I've got the average memory that I'm gonna look at for all instances and I'm gonna look for them to exceed a specific threshold for a duration. And algorithms are all tuning. What you're gonna see here is a prototype and feel free to use that as a seed to start. And afterwards, that component's just, that nozzle's gonna go back and say scale my app. I've reached some memory threshold over an average that makes me uncomfortable, right? Because you can't be perfect in everything. You don't wanna stop deploying something just because it's gonna run out of memory, right? You'd like to get that all cleaned up, but maybe it's slow enough and hard enough to find, you can still get some business value by deploying that. And we have some kind of cover and a feedback control loop that allows me to take care of that safely by scaling as I am filling up that bathtub of memory. Okay? And at this point, now we're gonna go to demo and I'm gonna turn this over to Tom and drive the demo. Sure, so like Dustin said, we've got two basic applications deployed to this space right now. The first is just a dummy application. Once it starts, it immediately starts allocating maps of maps. It just wastes a bunch of memory. The second is the nozzle that we're attaching to the fire hose. And through a few environment variables, we've set some parameters. One is the memory threshold. We set that to be 100 megabytes. So once you cross that, we're gonna start looking at possibly scaling the application. Yeah, so there they are. Memory-based autoscaler and the dummy app. So let's go ahead and start the memory-based autoscaler. And we just wanted to demonstrate by starting this that the very first thing it does is connect to the fire hose. And we just put a log message in there because it's not very interesting that, you know, hey, I'm connected, great. Let's go ahead and start that guy. So we wrote some filters in there. It's only looking for container metrics. And of those container metrics, it's using the same syslog to fire hose style code to annotate those events with app name and turn those container metrics into useful things like memory usage of the container. So it started up and you can see it connected to the fire hose. So let's go ahead and start this dummy application and go back and tail the logs over here. So there we go. We got our very first event and right now we're using one megabyte, which is well under our threshold of 100 megabytes. So the way this is coming out right now, it's about 30 seconds between events. There's probably ways to tune that to get quicker feedback. If you're doing this kind of system, 30 seconds is probably appropriate. So here we go. It's coming along, it's coming along. So this dummy app, all it is is a map of maps that just keeps allocating and allocating until it gets higher and higher and runs out. Right, so here we go. After 30 seconds it went to 124 megabytes and we've crossed our threshold of 100 megabytes. So what we wanted to account for is just spikes in memory. A lot of times dynamically working heaps can clean up their own memory. So if you get a spike, you don't wanna scale on a spike. You wanna make sure, all right, it's been above this threshold for a while. We're probably getting into a dangerous situation. Let's consider scaling this guy. So once this next event comes out, any time now, come on, there it goes. All right, so yep, we're at 221 megs. We've been over that threshold for a while, so it's time to scale. We went ahead and issued the command to scale list to two instances, and if you do a CF apps, we'll see now there are two instances of dummy running. So let's come on back. One of the things we do in terms of bookkeeping for this application is keep an average of all memory uses across instances. And so the next time this comes up, you'll see that there were two instances that were keeping track of memory for a minute and will calculate the average there. There we go. 332 and 105 megabytes for the instances. Average is 218, and we're still over the threshold, but we're also keeping track of the time that we last scaled because we want to give the applications a little bit of time to come up and handle some of the load. I mean, obviously this app is killing us, so we'd want to be scaling pretty quickly, and that's just another dial we can tune to keep things going. So that's basically how that thing works. It's a pretty simple prototype, but if you wanted, you could jump in, tune a lot more of these dials. You could do some kind of composition pattern, so you're looking at HTTP response times as well as your memory, as well as CPU to give yourself really complicated scaling algorithms. So where can I get this code? All right, there's links right there on ECS Team's GitHub. Encourage it, yes, Jim. Yeah, absolutely, and of course, I've uploaded, we've uploaded this presentation, so feel free to do that. Also, come on by to the booth this afternoon. All right, we'll talk more Firehose Madness, and hey, you can register to win an Amazon Echo or a Garmin Vivo Smart, and the drawing's at 320-day. And at this point, I want to leave it open for questions and start with Jim, go ahead. I do, let me give me a second. I can, I actually do have that up. So while he's coming up with that, one of the things you can do with these nozzles is attach little service endpoints on there, so. Let me go. Is he doing this real-time on us? Hold on, hold on, refresh. All right, you guys, hopefully you all heard that. Great, any other questions? We've got a few minutes left here, no questions. Oh, yes, in the back. A little bit, what you mean by attach as a service? You absolutely could do that because it's just a web socket endpoint that's protected by a privileged user, and so, absolutely. And yes, Jim, go ahead. Perfect, did that answer your question? There was one right here, yes. Yes, yes, we just chose because there's, you know, people have probably already seen a CPU autoscaler. We decided to do one that was memory-based, but you could absolutely grab the different container metrics, CPU being one of them, and actually doing some action on that. And then you could also do it on any other type of data, right? And the other point of the feedback control loop is to open up your mind of how can I self-tune my app? How can I make my user experience react to how users are interacting with my application? Right, there's, think of the possibilities, yes. So the question was how do you scale out the firehose? You can do that a couple of ways. You're gonna scale out the traffic controllers, and then your nozzles you can scale out and buy the subscription ID, so one of the things we kind of glossed over is you connect with a subscription ID and then as you scale those nozzles, that becomes a pool, think of like a database pool as an analogy, and then those connections to the traffic controller will use that pool and spread it across. Say again? Correct, correct, it's not partitioned. Any other questions? Yes. So just to tag on what I was just saying there, if you have the same subscription ID, then it should be load balancing those messages, excuse me, across that subscription to the number of nozzles. So I have a different subscription ID, then it's actually getting, both of them are getting the messages that they are signed up and filtering on. Right, and like in this example here, we ignored any other apps, and we ignored any other, excuse me, app of, excuse me, message types, event types, and only looked at container metrics for the dummy app. Yes. Yes, actually in our, you can use the CF API because we use it to scale. So we're in this code here, we're using a couple of different things. We're using Dr. Nick's Go CF client, and in there there's some get methods to get origin spaces based on your app ID, which is app GUID, which is what we use actually to do the scale, so absolutely. So you should have all the information just falling through the code that we have there, and it'll show you how to do that. So we do not have a nozzle for Splunk. One of the ways that you could, start using Splunk right away with this, is set up a syslog listener on Splunk. I know UDP, TCP, no, you can't do that? Yeah, you can't. No, no, I know you can do that. I'm saying he is restricted from doing that in his Splunk system. Sure, yeah, but yeah. I've personally been on projects where we sent it directly there, and it was fine. So I think that's probably a good safety thing, and maybe they're worried about dropping messages or something because most of the time it's UDP, but that's where you gotta weigh, if I have business data, and it's the most critical thing in my application that I'm doing, probably don't wanna send it through there and rely on it, because you could have some drop messages. Any other questions? Awesome, thank you guys for coming.