 Hello, good afternoon everybody, I'm glad you all made it, I know it's the last session on a Wednesday, afterwards been a pretty busy week, so I'm kind of impressed to see there's many people here, I wasn't actually expecting that many people, so good job. So my name is Andy, I'm an engineer on the Rackspace private cloud engineering and development team, and along with my colleague Jesse, who's currently sitting, we're going to talk to you about logging, and how you can manage it in your open stack environment. The way that the architecture has changed for kind of cloud and cloud applications and infrastructure as well, means that the way we were doing things before with logging doesn't work now, and we've needed to address that, the issues that have arisen from that, like very recently for our team, and so we've literally just gone on this path in the last six months or so, and so we've learned quite a bit and we'd like to share that with you. So I work on a pretty big team of really good guys, and like we're all developers at the moment, working on the OpenStack Ansible Deployment project primarily, but what it does mean is that if I say anything that's completely wrong, or a bit weird, the shepherd cricks are probably going to come out, and the talk might end a little bit early, so hopefully that doesn't happen and I stay on stage the whole time, and then everyone can be happy. So why should we care about logging at all? Logging is kind of the ugly cousin of monitoring and metrics. You don't know you need them until you really need them. What we get a lot of the time is product comes to us asking for metrics and things that they can measure, things like monitoring and metrics are very tangible, so when stuff's down a customer wants to know that something's down, and that's where monitoring comes in, and what the product wants to show customers are graphs and pretty pictures that customers can look at and go, wow, this is awesome, and those are the metrics, but customers don't really care about logging in the sense that they never look at it, and why should I care? It's just a bunch of lines of meaningless text to me, and that's a problem for us because developers also don't care about logs in the same way as supports and operations do, and developers tend to care more about stack traces as my application running or things working, is it being deployed properly, but supports and operations need the logs at critical moments when things are not going well, and at that point it's too late to say, oh, you know what, our logging solution is kind of bad, your site's done, give me a couple weeks and I'll fix the logging and then maybe we'll get back to you. So logs aren't just for support people, other people should care regardless. A couple other use cases like auditors need logs, anyone in compliance needs to store the logs, you know, you can use them for, aside from troubleshoots, you can use them for trending, access correlation, there's a whole bunch of other use cases that like I'm not even going to go into really, oh, I forgot, wait too many, back, sorry about that, there we go, okay. So yeah, like a lot of the use cases we get are like, hey, show us metrics, show us monitoring, but you kind of need the logs and the trick is also that even auditors who need the logs and need to store them, they don't actually care what the logs say, they just care that, hey, I've got logs and they stored, like are these logs being stored? I don't really care what they say, but as a customer when your site's down and you're on the line with a support guy who can't fix your stuff because your logging is bad, you're going to care about the logs, you just didn't know you cared about the logs and as a customer representative for a company, you're going to care about the logs when your customer calls you up and says, hey, your support sucked, like you couldn't fix this thing, like we were down for five hours, what's the deal? And almost all of that boils down to logs. So we all have logs on our servers anyway, so what's the big deal? Like why do we care about how they're managed? And that's great, so if we look at the old-day solutions of like we have one server with a web server, an application, a database and they're all on the same server and we've got one system administrator and the system administrator logs on to the server and we've got a site down issue and he looks in the Apache logs and he looks in the database logs and maybe there's an application log and we'll say a system log as well. So he's got four logs to look in but that's you know that's pretty doable, I can look in those four logs it'll maybe take me 15 minutes, find out what the problem is, I fix the problem, everyone's happy. What if we had two servers now and what if we separate our app and our web servers out into separate servers? So now we've got four servers, one running, two running application, a web server and maybe a database server and each of them have system logs. Now we've got eight logs across four servers and now it's going to take me a lot longer than 15 minutes to log into all the servers, look at all the logs and find out what the issue is. Now we move that forward to what we're looking at now which is that the explosion of kind of containers and microservices has meant that your logs are all over the place. They're on cloud instances and applications, for infrastructure they're often inside containers, they're everywhere and some of these things don't last long so I can spin up a cloud server and spin it down and then the logs are gone and then how do I troubleshoot that and then on top of that the sheer volume of logs, like I've got 100 logs on 100 different containers and I need to look inside all of them to find the problem but this just doesn't scale and it's a pretty common problem for like most things in in ITN cloud is that you can't look at the problems that you have now and solve them the way you were but just more of that and so you need to get smarter and you need to get more efficient so it's kind of like in the olden days before cars and bikes and planes you could walk around your town because in order to get food and water it was only a kilometer away but as soon as you needed something outside of your town you might need to get a bike because it's maybe 20 kilometers away and walking's not adequate for that but then if it's 100 kilometers I now need a car and if it's thousands of kilometers I need a plane and that's kind of the way that ITN pretty much everything else is is evolving like as soon as things start to scale exponentially we just can't keep doing the same solutions and this is kind of how the error of the log gripper began I don't know I don't know if Siri's doing my presentation for me but it actually would be useful so if Siri's in the audience like feel free to come up so I don't know if there are any John Oliver fans who watched last week tonight but he's got a segment that's called how's this still a thing and so tonight I kind of want to ask the log gripper how is this still a thing so I don't know if you are aware but there are people in organizations all across the world whose job it is literally to sit there and grab logs and it's normally for like cloud infrastructure we've got massive public clouds in various companies and I'm going to sit there and I'm going to grab logs all day and it's also normally the junior guy like you can be the log gripper enjoy that like Jesse's actually our log guy so he's going to do the demo later so you can look forward to like first-hand experience from a log gripper so for those of you that don't know grep is a Linux command line tool that's it's a pattern searcher so it's commonly used to search logs and other files I don't want to give the impression that it's not useful at all as a tool I mean it completely is I use it on a daily basis I think pretty much any developer or system administrator does but as a tool to search through thousands of lines of logs at a time it's it's not adequate it's it's simply not I really don't think this should be a job in in the modern cloud era and I think that the time for gripping through logs needs to go back to you know seven years ago when I had a one server infrastructure or two server infrastructure and and things weren't distributed properly it was it was just I have a load balancer and a couple servers behind it and we're all happy now I'm pretty sure they don't recognize this on their LinkedIn profile or their CV but uh you know look out for some key terms and I'm sure you'll spot them in fact anything that says junior cloud something is probably the log gripper a friend of mine put it like this it's like a pigeon looking for a solid dirty seed in a pile of wood chips and as the infrastructure scales you could say it's like a pigeon looking for a seed in one of several hundred piles of wood chips the end result is that you get disengaged unhappy employees you don't like the jobs they're doing and look to go elsewhere so recruiters if you find a junior cloud person you should probably ask them if they want a job they almost certainly do we have to get more efficient we have to keep these people engaged there's like smart people doing stupid jobs and we need to stop that so a little bit of an ad break I'm going to do a bit of a plug so apologies in advance but it does relate back to the log talk so I'll dovetail it back in one of the projects that we work mainly on is the open stack ansible deployment project my colleague Kevin did a talk yesterday I'm not sure if if any of you were at it it was in the stream actually and I just wanted to to give a quick plug to it because as somebody works pretty extensively on it it's something that I'm pretty proud of I think the work that that has gone into it is is really great and it works a lot better than any of the other deployment tools that I've used at least as with a lot of great things it started off with rack space but we've now made it into a community project so I'm a little bit biased but definitely check it out we're on ISE on on hash open stack ansible and it's up on Stackforge on github so if you're interested in deploying cloud or want to play around with like a container architecture for deploying open stack then definitely take a look and the reason it's kind of important to the log talk is mostly because the change in architecture that we took was from everything deployed on metal to all the open stack services deployed in containers with the exception of compute and swift storage they're all installed on containers and that's important because the logs for everything are now distributed across probably 40 containers for an install and are and are still in in different locations so if you take nova as an example you've got a nova scheduler container you've got a nova api container you've got a nova metadata container and that means I've got three places to look in for nova logs you know I mean that's not even including like the scheduler and all sorts of other services for nova so it just scales and makes the log problem much bigger and and makes a solution for that problem a lot more important and that's actually the reason that we've now addressed this and that we're giving this talk is because before when we had all the services on physical boxes it wasn't smart and it wasn't good but the problem wasn't big enough for us to address it and this moved it it just it just made it so much bigger that we couldn't ignore it anymore so one of the problems I've been talking about is that the logs are distributed across many containers in in many many places so a pretty simple solution would just be to create a centralized log server and that's actually one of the things we have done we have a centralized access log server but if we take a centralized access log server as the example it's great right we have all the we have all the logs in one place and you know that we can now search them in one place instead of having to log into into other servers but aside from skipping the step where we log into a whole bunch of servers what problem does it actually solve like we still have the issue that we've got you know thousands of individual logs they're just in one place um and like what if that server were to die like if we've got one server and it dies we just lost all our logs I mean that's way worse than any problem we had before actually um and can we scale it like if I've got this many logs coming through to one server that server is probably gonna like fall over under the weight of all the logs that are pushing through it we could crash our entire infrastructure really and then like backing up the logs like how do I ensure that they're not going to just disappear so we we solved that which is with a pretty common application that pretty much everyone is using who's doing log management for open stack and well any large application really by using elastic search so elastic search is kind of like a database where we can send the logs and they're then searchable and they're indexed and we can search via tags and fields on the logs that we've sent through and you can use command line new tools or there's a GUI called Kibana which you can use to to search it the one of the benefits of Kibana is it also gives the the pretty graphs and the pretty pictures that product are after and it's also useful in terms of just showing the the kind of logs that you have in and correlating logs elastic search can also be clustered so that kind of handles the performance and resilience issues and you can also do some cool things with archiving for example you can archive off to an existing Swift cluster for example or another object store so why is elastic search not enough so we've got all the logs in one place now and it's not in separate log files they're all actually in the same searchable place so everyone's happy well it's not quite the end of the problem so when you put a whole bunch of logs from different locations in one place what you're really doing is comparing oranges to apples to beach balls they're all round and but they don't all fit in the exact same way they have different characteristics and different use cases but really i can't search them to be more concrete in terms of what you do in OpenStack the application logs the database logs the system logs they're all in different formats in fact the OpenStack logs themselves i think have around four to five different log formats so just in OpenStack itself we're talking of like four to five different things that don't match up if i were to just throw them all in elastic search this leads to loads and loads of entries in elastic search but they're not searchable so really what all we've done is create a more jumbled view of hundreds and hundreds of logs that isn't really useful to anyone because now i can't even really see where they're from or what what the message is or anything really it's actually counterproductive and like for that reason you need to have a system where containing all the logs just isn't enough so how do we make the logs searchable how do we put this together so one of the things that you have to do is you have to tag and format all the logs and filter out the logs that you don't care about so a lot of logs at the moment in OpenStack are not set to the right priority so there was actually an ops meetup yesterday for logs which was incredibly useful but one of the takeaways from that is that most people who are doing logging from an ops perspective are doing all their logs on debug mode now that sounds really dumb because if anyone's looked at debug mode the logs that you get out are ridiculously verbose there's a ton that no one really cares about but the problem is that there's some logs that people do care about that only come out in debug mode and so what ends up happening is I ignore all the logs I don't care about and I have to filter through tons to get the ones that I actually do care about but the point is you can filter you can format logs and you can tag logs and for this we use LogStash which is also really common I think most people are using LogStash to to do the kind of log filtering formatting and tagging and and yeah you can basically like tag and and format logs so that all the logs are of the same type so if I take a log from patchier I take a log from like the nova console for example I have I set them out so that the time field is the same so now I can search on time field or the the source IP or something like that is is the same so a couple of the things that we've learned so as I mentioned we originally started looking at changing the way we do the logging because of the change to our architecture so we like to do as much as we can open source so the whole OpenStack Ansible Deployment Project is is open source and even the add-on stuff like the logging which isn't technically in the project is open source it's available on our Rackspace RCB Ops it's on GitHub so there's a repository called RPC Extras which has all these kind of things and a couple other tools that we use specifically for Rackspace we move them out of OS Ansible Deployment because it's they're like Rackspace specific things and we think that our opinionated version of what you should do should be separate and we want the OS Ansible Deployment project to just be community driven specific to OpenStack but one of the things we wanted to do originally was look at what other people had done with this and use it or fix it or make it better or contribute to it but what we found was that there were three implementations that were public they were all really out of date or didn't work quite how they should work and and we're now not useless in terms of they had no merit but we're not fit for what they were originally purpose for and were not being maintained or updated and there was no real community around it so there's no benefit to use these the funny thing is though that as an example in the ops meetup yesterday there there were at least three companies that are all using elastic search and log stash and are all doing filters and are all doing tags and formatting and and all the other things that you need to do and if you looked at them they're probably almost exactly the same but they've all done them by themselves and that's a little bit weird we've obviously gone and done the exact same thing so you know hands up to that as well but we have put it on our RPC extras thing and we'd love to to have something where you know there's a there's a single set of like these kind of formats and filters and all sorts of other things for logs dash one of the issues again is that the devs see the logs differently to to like ops and it's a community problem so the people are going to fix the problem on the devs but they really need the ops guys to get involved and say these are the things we don't like these are things we do like these logs shouldn't be debug mode these logs should be important these logs aren't required at all they should just be removed and and various other things and i do want to give a shout out to the open stack community for that because when i first started working on open stack in in 2012 i went to my first conference in san francisco and there was a obsession on logging and the obsession on logging in san francisco was a little bit different to the obsession yesterday in san francisco there was room full of ops guys and a handful of devs and they were essentially hurling words at each other that were not quite so friendly so the the ops perspective was this stuff sucks and all this stuff is terrible and you guys have done a bad job and like this whole thing is rubbish and the dev response was essentially well patch is welcome like you guys can do this stuff yourself and blah blah blah whatever which is just like this is not healthy conversation right and nothing changed in the logging in the following couple months which is pretty much a takeaway from the whole session when i compare that to yesterday's session when we had dev dev people saying hey listen give us give us examples of what logs you think are not worthwhile give us examples of how we can make this better like what would be useful to you how can we make this better for you like we want to help but you need to tell us what we need to do and i think that that's been very prevalent throughout open stack over the years how the the dev community has adopted you know the requirements from ops and tried to to engage and i think that at some point it needs to be a 50-50 so the ops guys need to come to the table and the dev guys but they're waiting to help and and so that's one of the one of the main issues and even within our own organization there's resistance to change right like as a sysad i know how to do grep i've looked through logs using grep for you know years like this is what i do so you're not telling me to throw that out the window and use this new swanky tool that you've created or you know implemented like why am i going to do that like i don't know how to use this thing it seems that's efficient i really know how to use grep i'll just do that and so it's hard to get people to start using the tools even when you've built them or they think they know the answer to the problem that they have and so they're telling you what they want and what they think the answer is rather than what they actually need and and so that's a couple of of the issues that we've had and then another thing is that what tends to happen with ops is that they don't think they can talk to the development team well or they don't really have a good feedback loop and so they end up taking whatever the devs have built and just building some stuff on top of it to make it look like how they think it should look and that's not really an effective way to do it i mean a perfect example is the the kind of filters that people have in general to remove logs like the correct approaches to go to the dev team and say hey these logs aren't needed like let's get rid of them or these logs are needed and and we're not getting them in the right velocity so let's like add those ones in not to add filters that just hide the fact that the stuff is broken so that was a pretty just general overview of the the kind of things that we've done uh my lovely assistant Jesse is now going to do the hard part which is the demo so fingers crossed it goes okay um but yeah he's going to run through some of the the tooling that i've talked about and and just kind of reinforce some of the things that that i've spoken about hopefully i might go and have a drink though so i might not be able to see it hardy all right this is uh stuff visible visible enough so welcome to the world of the log gripper yeah the number of knocks i've walked into um where this kind of screen is showing up and there's just lines and lines of text going through every now and again there's a color here and a color there those are the smart ones that have figured out how to sort of focus issues but generally speaking the knock guys sit there and and when the lines start to exhibit a pattern of passing much more quickly than usual then they know something's wrong that's basically the technique which is actually just ridiculous um that's it's not useful information and only exhibits actually one kind of problem the other thing is that logs um the using logs and analyzing logs gives you the capability to spot things that can't really be solved by alerting alerting tell you the known tell you problems with unknown thresholds and known issues that you have whereas logs give you the capability to not only dig into issues and and find um problems that way but they also allow you to spot trends and trends are very key especially when you're talking about and looking at uh environments at scale so let me just uh think about where my next step is oh yes not that one let me try and go over here done right so a couple of things become important um if I step through like the log gripper's world is horrible and um that really should die um and so we introduce tooling like elastic search elastic search allows you to have something that can receive the logs um there are various ways of getting the logs into elastic search which I won't go into um we're taking a pretty simple less moving parts um uh mechanism which is literally just to ship logs with a decent log shipper that can do multi-line stuff at the source because that becomes a big problem and uh we put it in elastic search elastic search then does the indexing and makes the stuff available where you can actually search and look at it and there are wonderful little gooey tools which you can show to your managers because that's awesome and uh they do little things like this present pictures of your cluster and my huge cluster of one elastic search server um is happily behaving itself and that's wonderful but it really comes down to the same kind of thing how do I get my tail in a in a more manageable way and uh for uh for the sake of demonstration because command lines don't do very well in presentations we like to do uh we like to use something like Kibana which is um provides an interface on top of elastic search to do um the gripping and the tailing in uh in a very easy and very easy manner so what you're seeing here is um it's an all-in-one open stack server but it happens to be containerized so every server's in a container so there are a lot of logs going around we're not we don't have every single log on the planner coming through but there are enough logs as you can see what is this 1500 events um for each bar here so that's quite a bit of events coming through but this isn't exactly manageable either so if I stop it and I start to think about okay what am I actually looking for am I looking for something along the lines of um all logs so let's see if I can get this right no that's not gonna work so obviously one of the the common things you might want to take a look at is uh you know our script kitty is trying to get access into my back end I'm actually trying to um figure out what my passwords are and get in so doing this kind of thing is pretty much like a log rebus job it's just got a nice little bit of makeup on it this is not very exciting either and uh as you can see there's not a heck of a lot of metadata going on here and this is all that's like if you only have some sort of log shipping and you only have elastic search on the back end this is kind of the basically all you're doing is dressing up the log rep is job but at least you're doing it at a bit more scale and log stash puts you in a position where you're able to do slightly more funky things essentially you're putting log stash in the position of doing the log rep as job so let's find a more interesting log nope can you try and don't log today let's go back to your long list I'm sure we'll find one so there we go so what you're seeing here is all of a sudden the log has a lot more metadata log level the hcdp version given by the client you're looking at the receive date request time request or ip but more importantly for we're not going to see it here for for some sort of metric something along the line of hcdp time that's the kind of metadata that that might be useful but let me do quest I think it is yeah so in your typical troubleshooting process when it comes to something along the lines of figuring out what's going on in the event you can track it by request ID across many logs and this is where it really becomes useful because now I'm seeing everything every log and every event that happens to relate to your request ID so this is not just the load balancer queries that are checking whether stuff is still live these are actual requests being sent through and if i'm lucky I will find the request ID that also has there we go no it's not it there's another one over here I'm missing it let me skip that but what you'll end up is effectively the log rep's job is being done so your queries become a lot simpler a lot more accessible to people who don't know the complexities of regex and and you have the ability to get access to a different set of logs and a different set of metadata but then you can go one step further and you can do the really cool thing that everybody loves to do in their offices these days and you can have a screen on the wall that shows pretty pictures to the managers when they walk past and the managers love to see things like this which actually is a very very simple depiction of what's going on in an environment but it's also a really good way of being able to spot an unusual trend so if I do something like this and I do a couple of goodies just to create some traffic what you will spot all of a sudden is the response time goes up this is an all-in-one server with eight gigs of ram it's not a very efficient open-side environment but this is the kind of thing that you actually want to see and you're not necessarily going to pick up in your alerting infrastructure you want to see that there's a change in trend you want to see that all of a sudden a particular service is executing a lot more queries than it has before you want to see that perhaps some of your underlying infrastructure is doing something unusual you want to perhaps see that there are a lot of delete events and why is that going on some of the dashboards that I've looked at have built dashboards around you know what's going on with instances at this moment in time other ones have done what what's going on with the storage at this moment in time you know there are different focuses that you can do with little dashboards like this and this is this is information that you really can only derive from logs and this is why you use cabana instead instead of trawling through logs instead of relying on alerts anyway that's me thank you very much I think we have time for some questions um you can reach out to us on isc or twitter our details are up there if if you need to but yeah it's other time for questions or beer can you would you mind using the mic yes so I have one question or maybe one with a follow-up so one of the things that's interesting about logs is you know the information that you get out of them but one of the things that's painful about logs is saying how much information you want so in the sense that I you know often don't really care to run my logs at a high logging level but if I start to hear about problems I want to be able to tune the log file so just as difficult as it is to receive logs from lots of different places it's equally difficult to control logs that are in lots of different places so that I can turn the knob up and say hey give me a little bit more volume on my nova compute log show are you all looking at that particular problem or yeah so I'd say that problem is is handled by the kind of deployment mechanisms that we use so like we can control the verbosity or the the log level of the individual services based on on how we deploy using ansible so it is something that we're looking at I don't actually think the functionality's in yet to control on a per-service basis but it's saying that we would definitely handle with with the ansible deployment tools that we're using but there's we are as a as a team as the RPC specific team we're also looking at logging on the same levels that with the log working group we're taking a look at the problem from many layers so as for for RPC or or OS ansible deployment we're looking at what there is now and we're trying to work with what there is now but parts of our team are also looking looking upstream at the open stack projects and trying to also solve their problem at a higher layer and then as that evolves we obviously can evolve the output of the product as well okay thank you okay I'm not sure to have understood the complete picture so basically you are using tell me if I'm wrong um lock stash as lock shipper to to a central logings thing that after afterward goes to elastic search and have a nice dashboard with keybatter or I use using RCS log to send logs to a central location that is afterwards uh processed by uh lock stash sure so those are actually uh two separate things really so we we have a centralized RCS log server mostly because of one of the points I mentioned was that um when support comes to us they think they know what they want so they ask for a specific thing they don't say we have these issues and can you address them and one of the things they asked for was a centralized RCS log server um so that's reasonably simple to implement so we said sure we can do that for you and we implemented it um it it's not a great solution and we wouldn't advocate using it um but but it's there and then the the shipping to log stash uh the shipping using log stash and then to elastic searches is kind of separate so it doesn't take the logs from the centralized RCS log server um we are actually I think you just set up the Python beaver to use uh log shipping for the logs themselves um so we're using uh Python beaver to do uh kind of multi-line logging um for using the log shipping to do multi-line logging and various other things uh which then since it's a log stash that has uh some of the filters which we were actually able to cut down a bit because of the way we were using the log shipper and then through to elastic search yeah we're finding that uh RCS log doesn't behave well when you're talking large scale um for some reason um queues build up and it's queues build up on the sending side and on the receiving side the the server just can't keep up when you're talking large classes so um within OS sensible deployment we've provided the capability to do your syslog to whatever uh it doesn't have to necessarily be an actual syslog server you could be using Splunk or log stash or whatever you want um but um for for the derivative RPC product we're um we're looking at basically using beaver um not the uh not the uh log stash shipper um and that's because we want to be able to handle multi-line at the source multi-line stack traces um must equal slow logs, rabid logs, all that kind of stuff uh when you rely on the log stash server to do that you can only have one worker and your log stash server dies um there are other ways to do it you could be dropping everything in in readers and having your log stash sit on the side and do the processing um we're trying to not have so many moving parts um and beaver's really simple to to configure and it's working really well um but the way that we're doing it there is each each container, each host in the environment will run the beaver servers it does the multi-line processing in a in a few tags right at the source sends it through via TCP and JSON um to the log stash server and the log stash server then does a bit more analysis and drops it into elastic search so the RSS log thing is a different path so that we have two paths to also store logs okay thank you sure so um in your ELK stack what versions have you tried and what works the best this question one and second is I believe on the on the log stash side you mentioned some of the drawbacks but from the elastic search and the kibana point of view have you experienced the drawbacks um so the drawback on the elastic search and log stash side is really Java um elastic search is reasonably manageable um because there are plugins that expose a lot of the underlying stuff so you really can look at it but log stash is a bit of a black box so when it doesn't work it gets very frustrating um the debug longing is a little over the top um you know you see every event in troubleshooting it becomes a a right to roll pain but um and we've we've done a bit of research into replacing log stash with hacker for instance as an alternative um which can do the same job can can operate the same way but has a little more insight into what it's running for the moment though we're we're sticking with the ELK stack as it is um but yeah just even in install time running a jvm um and having to maintain the jvm and the memory it uses that that took a little bit of doing to manage it but it's it's working okay now and it processes quickly enough the other drawbacks really were around the the main issues we've had so far in full scale is log shipping it's getting the logs to the log stash server the other issue was handling multi-line because you really want those multi-line events that's one event otherwise it's pretty much useless log stash can do it but then you have to implement uh when you implement multi-line codecs within log stash filters you um you have to implement only one worker it can't it can't implement multiple workers so what happens is your log processing grinds to halt you just can't funnel a log through quickly enough so you can work around that by having multiple log stash instances and then push your logs to those instances but it it's a messy way of doing it and we've taken the approach of rather trying to handle that at the source it works a lot better python beaver you'll find that on github there is another one which is a fork from the log stash order called log courier it's also written in go does all the bits that you would ever want it to do and more but it's not as simple an installation you'd have to download it compile it produce packages yourself that is just an overhead that we don't want to handle so i think we're out of time now okay if you want to come ask questions off to us feel free to step up and uh we're hiring i'm obliged to say that