 Hey, everybody. Welcome to our panel on observability in the cloud-native era. And I couldn't be more proud to have such an esteemed, you know, awesome panel of superstars here. Charity majors from Honeycomb, ETL Schwartz from Commodore, and Jason from Datadog. And they are going to demystify all of the things you want to know about observability in the cloud-native era. So I'm just going to give a little bit of framing to the discussion. We have prepared some questions, but we are more than happy to receive questions from the audience. You have three people that you really want to tap into and make it challenging, you know, roast them. And yeah, you're definitely, so there's a mic in the center. You can stand up and ask your questions. So we want to talk about how, when systems are now getting more complex and downtime is more painful and costly, it's becoming mission critical to have actionable data about your systems, to be able to make the right decisions in real time, and particularly when managing high-stress situations like production incidents. However, like all things system engineering, this is easier said than done. Not only is there a jungle of tooling out there, from open source to commercial, but understanding what to monitor and even what not to monitor, how to build your monitoring and observability, the building blocks, and really ensure that everything, eventually when you need it, your dashboards will reflect the business critical issues you need to know about. This takes years of experience. It's not easy. That's why I'd like to tap into some of these monitoring and observability experts for our very own community who can share some insights from their hard earned experience, what pains and friction they see and have seen with their user bases today, and particularly with the complex cloud-native systems that people are building. What are some good tips to overcome some of the common challenges? Without further ado, I'm actually going to let you guys introduce yourself and tell a little bit of a fun fact. Get started, Jason. Tell people about yourself. I guess I'm first. I'm Jason. I'm a staff developer advocate at Datadog. I've worked at a number of places that you can read up there. I don't know that they really matter, I guess. I was at Gremlin. I did a bunch of chaos engineering thing because it turns out that it's a good way to test your observability is to just break things and hopefully you can see what happens. Before that, I was a software engineer at a bunch of places, MongoDB, things like that. Fun fact, I used to be a chef, so I would work for 10-hour days, Monday through Thursday, and then Friday, Saturday, Sunday, worked in a restaurant. Yeah. I'm still waiting for you to cook for me. All right, ETL. So, hey, it's a bit loud. ETL Schwartz, city of Commodore. What we do, it's less known than Datadog, a bit less known. We're a Kubernetes real-heavility platform. Basically, we try to make Kubernetes simpler. But in my past life, I was mainly like a back-end infragile, big companies like eBay, small companies, and a lot of startups. So that's my background. And like a fun fact, I really like pandas, like the animal, right? Like not the Python package. Okay, with the package. So, yeah, that's like my mascot for a lot of time. And my name is Charity. I'm the co-founder and CTO of Honeycomb.io. And I identify as an operations engineer before starting a company, which was never on my radar. Before doing that, my niche was really being the first info person who would come and join a bunch of software engineers when they thought they had something real that users might actually want to use. My fun fact is, after a lifetime, I grew up on a farm and we ate most of our pets. I was very utilitarian when it came to animals. After a lifetime of this, I just got my first foster cat. And she has this dislocated jaw so her tongue is sticking out and she eats by mashing her face in the food. She's disgusting and I love her so much. Amazing. So it's awesome to have three of you here. Sorry. So, based on that and your short bios that you gave us, you all come with both hands-on experience and now you have more of an industry perspective as vendors building tooling, observability tooling for cloud-native systems. I'd like you to share a little bit about what you've experienced as a practitioner that helps you understand what users need today. And you can get started. I mean, literally, this is why Honeycomb was started because, you know, back in the days of P.A.R.S., my last job as an ops engineer, it was kind of, we were doing a lot of, we're doing like microservices before. There were microservices, a lot of like cutting edge stuff that we didn't really have the tooling for. And as a reliability engineer, I was professionally humiliated because every day the thing was going down. It was just like every day a new app would hit the top 10 in the iTunes store and down would go P.A.R.S. And I was just like, what is happening? Like I tried everything. I tried all the logging tools and all the monitoring tools. And like, you know, what they all had in common was once you figured it out, yeah, you could add a metric, you could add a dashboard, then you could find that problem immediately the next time. But these problems weren't happening over and over again. Every time it was something different. And like the last generation of tools just isn't equipped for that. Like you really have to like have a sort of paradigm shift to like away from basic metrics and logs to more of an event-based trace-based fundamental. You see all your thoughts? Yeah, sure. So like, I started my career working for a company that does like a fraud detection. And that basically means that our KPI was finding bad people that are using stolen credit cards. Every time our system was done, we basically lost a lot of money to bad people. And this in turn really forces you to be like, quite good or at least not really bad. Because I was the one writing the mails to the management every time we had a downtime. And the first time is how much money did the company lost? We lost 100K, half a million. So this like in turns really made me, first of all, not be too worried about downtime, because you know, it happens to everyone on one hand. But on the other hand, like being obsessed around what's the best way to prevent downtime and even more importantly, learning from incident and from like past issues. So when we started Commodore, like that was the main thing that I always had in mind, like this guy or girl in the end, that is super sad right now, because all of his production is down, everyone is shouting at him. And all he wants to do is you know, like to find the right switch to turn on in order to make sure everything works like works. And I think to this day, you know, talking here with a lot of like production engineer, platform engineers, I can really relate to the pain because in the end of the day, it's like, it's hard, like it's hard to be like the one in charge. So that's pretty much it. Yeah, my foray actually into this whole space is actually slightly different, because I think most people come at it as a, you know, I have a service or I have a website and is it up or is it down? And way, way back, I worked at consultancy and one of our clients was this church in a small rural town in Midwestern US. And every year they would have this Christmas show. And so it's probably right about this time of the year, someone else is dealing with us now at that company. But they would literally have like everybody in this town would go and try to buy tickets for their Christmas show, which was like 100,000 people in the span of like they would open up at like 10am. And so it was this thing of like, I'm a web developer, I wrote PHP code. And it became the thing of it's not whether it's up or down, it's the performance, right? Can we scale and handle this? Because it's going to be down, right? Like that's a guaranteed if you don't have that scale. And so mine was kind of interesting, because it was mostly that got me into web performance and started to think about like, how do you get things faster? Which is now very common with what we do, right? Everyone's thinking of performance. Slowly the new down. Yeah, that's true. It's interesting. People have zero tolerance anymore to kind of wait around for the website to load or anything else. Okay, so on that note, you guys spoke about the things that kind of brought you to build the tools that you have been building and also the things that you learned on the way. I think one of the questions that often comes up that I've heard when I whenever I've been kind of in the in this area is that is it even possible to have visibility into all the bits and bytes that systems have today from the millions of services communicating with each other, get CICD, shift left, all the engineers touching everything all at once. It's built upon different stacks, tools, clouds, and they all behave differently. How do you find a needle in a haystack when it really matters? I mean, first of all, the answer is no, of course, it's not possible. It's not possible. I mean, I mean, in theory, it might be possible, but in reality, nobody is going to pay for an observability bill, an OLLI bill. That is 20 times that as big as their entire production systems. Nobody's going to do it. There are diminishing returns that kick in very quickly, which is why like all of this is why I get really pissed off whenever people like we can't sample every line is sacred. And it's like, do you know how much shit you are not gathering right now and you're totally happy about it? There's a lot of this that is visible to us, but most of it isn't because it just works. And people who are as old as, I don't know about you, but Jason immediately, we remember when it didn't just work. Remember, a curl crashes being like just a thing that happened every week? Yeah, doesn't happen anymore. We don't have to monitor that for that for the most part. We can just be resilient to it. I could keep talking, but I'll leave I think, you know, like the the base premise of it's impossible. I think it's right. Like when we see when I see like, modern development, so much moving pieces. It's impossible. And even trying to do it, I think like, it's set up for failure. Like my, my like two cents here is that and then it's hard, right? Is building the application with the mindset that things are going to crash. Like everything you do, you add a feature flag, it's going to mess with your production environment. You are adding like another step over the pipeline, what will happen when this step won't work? And you need to deploy something fast. Like there are so many failure scenarios. But every time we add another layer, it's like just adding more and more like failure scenarios. And we need to have that in mind. I think the thing that is super hard for most people is they are like optimistic in nature. I don't know. Like I was a developer, right? Like, and I wrote code and I hope it worked like most of the time. And it took me a lot of time to understand when I was a junior developer, that like nothing is going to work. And even if it does like it looks like it's going to work, it's going to fail sometime. And I think like even the most trustworthy things which is like obviously S3. Like I remember the S3 like downtime. And you know, even the AWS, right? Like those guys are good, right? And they store the icon of like S3 being down on S3 because they didn't talk about like what's going to happen here. So I think like having like failure mindset is the key thing to have. But on the other hand, I will say it's very hard to achieve that. Like you do need some maturity and some understanding of failure scenarios. Yeah, I mean, to the just underscore the point, right? Like, yeah, you probably can instrument everything. If your app is small, if it's very, very static and you don't plan to change everything, right? Like if I had a time machine went back to myself 20 years ago, building simple PHP apps or Perl scripts, like, yeah, I couldn't monitor that whole thing because I had one Linux server that I compiled the kernel for and had a stupid Perl script and hasn't changed, right? But let's face it, we're here at this conference and it's cloud native con. If you're doing anything in the cloud, you're doing things dynamically, you're probably doing at least one deploy a day or even one deploy a week. It's changing enough that it's just impossible. Everything breaks. And like to the entire, this is the entire point of this is like, why is everything so hard now that we live in the future and everything's great? Well, it's because it's because honestly, like, you can't think about your systems. You can't think about your software is just the code, right? This is why unit tests are not enough, right? Your unit test driven development is great, but all it tells you is that it logically probably works. But that doesn't actually mean it's going to work or do what you think it is because these systems are emergent, they're complex. You don't, this is why I feel like observability driven development. Like you don't know if your code works until you've watched it in production as real users are using it. And then all you know is that it works for now for some people. Okay. I think those were kind of depressing answers. I'll be frank. All right. Provide some tips and truth is depressing. Like, you know, everyone that tells you that is now yeah, you can monitor everything expecting more optimism and more. The optimistic part is it means that you all will have jobs, right? It gives you a chance to, well, there's round of applause. But there's that. But it's also for me, it's you have a chance to learn. You have a chance to improve your skills and get better and develop depth in what you do, right? Because nobody actually, if you're a developer, you don't want to be the person that just writes code and says, I have unit tests, it works for me, right? You want to build something that's solid and reliable and you want to build systems that are more complex and you want to get better at these things and build these really cool things. And you can't do that if you're writing stupid, simple scripts that don't break because they, they're not pushing innovation. Everything's about risk management, you know, and I don't think this is depressing at all. I think it's hilarious. I think that you just have to have the right sense of humor. It depends on what period of time. I think when you're in real time, it's probably depressing, but afterwards you can laugh about it. I disagree. But then I come from operations where, you know, I don't know, being down is funny. It's funny. All right, so following up on the previous question, maybe give us like some practical tips. Like you started to dig in there, so maybe we'll dig in a little bit further, double click on what you were saying. Like what are some good tips that you would give in order to kind of overcome the common challenges and do your best at finding kind of the issues as close as possible to the incident? Yeah, well, I mean, how do you get better? Practice makes perfect, right? I mean, this is, for better or worse, like I took that deviation and went to Gremlin because I still believe in chaos engineering, right? And per charity shirt, right? Like you either test in prod or you live a lie, right? And so, well, you could wait for those actual production outages, but hopefully you're not having those regular enough that that is actually practice, right? So you have to set up scenarios. So chaos engineering, right? Like start testing your apps, try to break them and see what you learn. Yeah, I would say that, you know, like the thing with chaos engineering that always, you know, on the, I understand chaos engineering, but my feeling, especially as a vendor, right, but also as a practitioner, there's a lot of chaos always happening. Like, I think most people have enough production issues as it is, like to, you know, they don't need to add more production issues. I think the play of like, how do I learn from one production issue to another? Like this is, this is my answer for that. I think, again, most people don't really know how to understand, like how to learn from one production issue to another. Usually, usually, like for every production issue, like two things went wrong. Like there's the actual problem. I don't know where I wrote the bed call or something like that. And something around the processes, processes or observability was broken as well. Because if not, maybe I would have detected sooner. And I think like doing those post-mortems, learning from one production issue to another, same if you're using chaos engineering, like, okay, now I know things are failing. What's going to happen next time? I think like this is something that not a lot of companies are doing and can really help them elevate their game as time progresses. Yeah, I mean, you could say it's depressing, but I actually think it's not, because there are so many things that are broken in your systems right now, and it's fine. Like, we're resilient to most things failing. Like, there's little, like, you know, bits getting flipped on disks all the time from solar rays. We're resilient to them, right? There are Linux kernel problems happening all the time. We're resilient to them, right? The thing about cloud native is that the, we're able to have a level of abstraction now that we can more or less stick around it and build our apps without having to deal. Everything breaks eventually, but we are getting better over time. Like, there is so much software now that you can just take for granted, and you don't even think about, which lets you, which frees up your time to actually focus on the code that matters, right? Your software. Which is why I feel like, your first question about, do you have to instrument and understand everything? No. I think about it kind of like a headlamp. You need to be able to instrument, you know, computers are having these blips and, you know, spikes all the time in errors. And until someone comes along and goes, that's meaningful, I need to understand that, you have the luxury of ignoring it. Which is the other reason that I think that, like, a lot of the LLM stuff is kind of getting it upside down because, you know, only people can attach meaning to things, and computers can tell you when there are spikes, but like, they can't tell you if it means anything. I will select the most, like, unpopular answer, and the real answer is, like, use less technology, right? Like, it's maybe not the best thing to say in a cloud-native, like, conference, right? But, you know, like, in the end of the day, like, I'm really trying to help, like, people, right? Developers should be faster and so on. But if I could, I would build all of my product on Excel or something like that, something that never breaks. Super simple. Now there's Python. I like Python. And, like, if it would be a possibility, like, I really try to keep things as simple as possible. Like, again, you know, like, it did work for, like, a lot of time. But I think, like, people are, like, really don't underestimate, like, how simplicity and less moving parts and less technology really helps down your business, like, doing less. If you can build your app in a lampstack, by God, do it. So now I'm going to ask the controversial question. Then why Kubernetes? I think the answer is because, like, life is complex, right? Like, again, I would really wouldn't want, like, anything no microservices. And this is how commas were started, like, one monolith, one micro, like, no microservice, right? Like, it was a monolith and one repo and one database and that's it. But, you know, it's life. It's, to be honest, like, it's life. Then you want disability and disability. And those two don't act well together. So I think it's always, like, on one hand, like, you know, I always love the new shiny thing. There's something inside of me that likes, like, testing your technologies. But finding this balance is, I think, the most critical thing. And I think a lot of the people also are right in the room, in the conference. We like those, like, being on the bleeding edge. There's something super fun around that. But then you are on the bleeding edge and, like, suffering. So I think, like, finding this balance is super hard. And a lot of the times, like, one of the questions that we ask all of our customers is, why did you migrate to Kubernetes? Like, you know, like, you had a setup. It worked, like, your company is making a lot of money. Why Kubernetes? And some of them have really good black reasons. And some of them are, like, yeah, everyone is moving to Kubernetes. And we're also moving to Kubernetes. Someone in the management said that. And, like, now we're stuck on a seven-year migration plan. So I think, like, that's the reality, right? Like, again, maybe unpopular. But I think that's reality. Well, I think there's two interesting points, though, between what you said and what Charity said, right? Charity mentioned that there's layers of the abstractions and things that keep working. And there is simplicity. So it's not to say that Kubernetes itself is simple, right? We've all worked with it. It can get very, very complex. But with these abstractions, you're essentially, you want to simplify the concerns that you have. And so a lot of what we're doing, yes, Kubernetes is not simple at this point, but it makes life simple for your developers in some ways, right? Because they can just take their app, put in a container and deploy it, and it generally works, right? And so a lot of what we're doing with technology these days is just shifting where that simplicity is. And so, again, you know, keep things as simple as you can for what you need. And for what you need, you're all here. It's probably Kubernetes to make developer life simple. Unfortunately, it probably makes your life a little more complex. Yeah. You know what else wasn't simple? Having to orchestrate all the different HAProxy and NGX and Chefrolls and everything to do all this shit that Kubernetes now does for us. Like you're sort of localized in complexity in one little place. I'm not sure if like Kubernetes is like the first thing. I really like Kubernetes, right? Like, obviously, I started the Kubernetes company, right? But I'm not sure if like developers is the first thing because for me, like personally, I had Chef, I had Puppet, I had like all of those, sorry, tool when I needed to work with them. And then I found out about Kubernetes like seven years ago and everything is YAML. I don't really like YAML, but it's much better again than the alternative. So for me, it's like more, oh, this is a much easier way of saying what do I need as an infra guy, not necessarily as a developer, like I worked with containers like a variable form containers and I had operation guys like doing the Chef for me and it was like good for most most time. We shouldn't be making these decisions for technical reasons for the most part. They should be business differentiators, you know, they should be driven by where are we going to spend our innovation tokens? And, you know, the only, the only like other reason is if you're trying to hire people and they all want to work on Kubernetes or they all have that skill set. That's also a super valid reason to do it, but we shouldn't really be like sitting around here talking about which is more complicated is probably the wrong lens to be making technical decisions. Okay, solid enough. I'm going to take a pause and see if anybody from the audience wants to ask anything. Any questions? Go for it. Stand up and ask your question at the mic. Thank you. Did we get way off topic? No. We like it when you go off on a tangent. It's fun. I'm glad to. Observability things are easier in that context. Great question. Did everybody hear it? Fantastic. I will say that like in general Kubernetes makes a lot of things much simpler. Like adding like fluent D or like agent and things like that. Kubernetes makes standardization much easier so you can add all of those things even without working like again like fluent D, elastic, Rafaana and so on. So on one thing it makes adding those observability layers much simpler. And by itself you would have expected things are actually easier. I think the main downside of Kubernetes and again I love Kubernetes is it makes things so simple that it's very easy to add more and more like stuff. It's very easy and cheap to add another service. So I think that like most problem with like Kubernetes observability is it allow an easy standardization for everything which is great but it allows everyone to start his own container or cluster or namespace. And in reality because you already have Kubernetes you are like yeah sure I'll add another service and another service. And before you know it you have like 500 services running out in the wild. So I think from an in-front perspective it's much easier. But from like you need someone strong saying we shouldn't like not everything needs to be a service, right? So that's my answer here. Yeah I completely agree. It's trade-offs, right? Some things have become easier and those because the scale have led to other problems. I think the number one thing that I think makes the most sense to focus on is making sure that we're increasingly collecting our telemetry through the lens of the application not from through the lens of the system. Because system level metrics you know they're powerful if your job is running infrastructure and if it is running those level systems but for most of us our job is not that it is making our end users happy and so you need to be able to capture all your telemetry in ways that you slice and dice and say what was this user's experience like? What was this group of users experience like? Because it's completely possible for the infrastructure to be healthy quote unquote and your users to be very unhappy or for the infrastructure to be unhealthy and your users to be happy and which one do you care about most? Well you always care about the users the most. Yeah that makes sense. Anybody else have any questions? Any other questions from the audience? It's a lot simpler but oftentimes the reality is like we operate pretty much everything on public cloud that have a lot of managed moving parts that do not expose somebody data. So I see it like in the long run we kind of get more less visibility compared to what we had when we were running on our own. So what are your thoughts? Because that I feel that I'm feeling the pain already but I feel it's going to be accelerating a lot in the coming days. I think like it's a great point because and I think everyone here talked about like the abstraction part right like on one hand you don't really want to care about the fact that there is like some Linux somewhere in EKS right like you don't want to care about this until you really want to know about this because you are using too much like I don't know a bandwidth or whatever I know through I'm not sure what. So as I see it like most companies don't really care about again from what we see from our customers they don't really care about the underlying infrastructure for big companies it's a huge pain it's a huge problem I think it's more like of a learning curve for the cloud providers like I do hope that like in five years no one will really need to know because EKS and GKE and EKS like all of them will be good enough I think we are currently living in a state that the abstraction is not complete and this in turn just like surface the pain on one hand no easy way to understanding on the other side which is like bad but it's mainly for from what we see like for very big enterprises for like 95% of the users they don't really need to care So I know we touched on this a little bit but I guess let's expand on it a little bit what problems are still unsolved and what is the industry doing to solve them? That's a very broad question Yes I think one of the biggest sources of pain continues to be just having so many different sources of truth when you've got like one thing over here for your metrics and dashboards you store that same data again for your logs you store it again for your traces and you store it again for your profiling you store it again for your security tool and like the only thing that nits these things together is you the person in the middle who's like just like trying to eyeball time stamps and go well this probably the same thing as that it's a huge problem you should have a single source of truth from which you can derive the metrics from which you can derive the but obviously I'm a vendor so I have a vested interest in this but I think it's a huge problem Yeah I think I will also like give my side which is basically and I think you said it like perfectly there are like two kinds of problems on a high level on Kubernetes there is the end users problem which are the developers or that are doing the things that are really moving the business forward right and there are the info people info guys people problems that is like I have problem with the nodes with the memory and and so on and so on and I think like currently there is like like a very very like two different camps like the developers and the ops people and something somehow need to bridge that it can be that the developers will be more like Opsi the Opsi will be like more developers or some tools will like be the bridge but I think where the industry is currently like heading towards is two very very different like type of users that are using Kubernetes and each one is cursing the other user because he's not happy with that and somehow they need to coexist and I think like this is what we see the most like we try to bridge that gap but in reality like you know like I've been I've talked with people here for like the last five hours or something like that developers are complaining on the platform team not like understanding them and the other way around and I think like this is something that really holds back our industry like those camps yeah I'd say that you know charity's point is great because I do find a lot of customers that are like it's it is that correlation thankfully like at Datadogs are you set me up not to be a bitch but we have all of those in one place and what I found is our problem is actually when I talk with customers it is become the thing of you know we talked about DevOps and moving developers and operations closer together and yet how many of you work at a place where you have like the observability team right and it's like your developers don't care your ops people have stopped caring because now it's that observability teams problem and so we keep having these silos in these divisions and really that's being the issue not a problem and I would note that they're in the same place but there are different data sources you can't actually correlate them but not to be a bitch but the observability team thing I think is actually a really great pattern in as far as they're like the they're doing vendor engineering in a way you know because I think that there are lots of powerful vendors out there who have these very powerful powerful tools but you need a team kind of be that that that layer did to be making for example some libraries that everyone can use that are that are very you know idiomatic that you know fit in with your tooling and all the languages that you need and all that stuff and and I feel like I feel like the observability teams are doing that are really doing God's work yeah I mean there's definitely like a mix right there I'm not saying that observability teams are bad by any means but there is a natural tendency especially in some large organizations where they become those people right and there's this disconnect and then people stop to care because that's not my OKRs or my KPIs for the monitoring and somebody else is watching that so I can just ignore it and obviously like that's when it becomes pretty bad you can't be an engineer if you don't understand how your code is working in production you can't call yourself an engineer I don't think I feel like we should coin a new discipline like DevOps or something like that like that's something that we should start discussing I actually think that DevOps is on its way out and here's why I think that things we didn't ask charity go for it in the beginning there were engineers who wrote their code and owned it in production and then systems started to get super complicated and we're like ah shit there's too much for one person to do I know let's slice it up so that half of us write the code and half of us understand the code that was a terrible idea DevOps is like trying to knit these things back together and it's a value it's a very you know valid you know thing to be doing but I really feel like that we're all moving back to a world where everyone writes code and everyone is responsible for owning their code in production some of this is more infrastructure code some of this is more you know application code whatever but like our systems are actually getting so complex that if you didn't write the code you're not going to be able to debug it and operate it and if you're writing the code but you're not operating it you like this is all about feedback loops right software engineering is all about having a feedback loop but if you're not subjecting yourself to the output of the code the shitty code that you put out there into the world then you're not you're not actually ever going to understand it in order in the way that you need to in order to write good code yeah all right I'm gonna we have two minutes left if there's any more questions from the audience yes can you talk to me about like costs and using observability tools to measure costs and like define value for not just observability teams but tools as well great we always love cost questions yeah we actually have some great blog posts in the honeycomb blog about we use our own observability tooling to do costs and cloud costs and everything I'm so glad you asked that I feel like I for one am glad for the death of the zero interest interest rate phenomenon I think that it's forcing us as engineers to pay attention to costs and real business problems I think that like there's been this pathology since the beginning for a very long time where CTOs, VPs of engineering are considered like the junior participants in the sea level suites because because we have historically not known how to represent our work in terms of that the you know the universal denominator of of dollars and cents so I don't know I've got to let Jason take some time to talk to this too but like you're on the you're on the right path we absolutely need to be asking those questions yeah I would say you know when I think of costs think of it as any other resource right and I say this because with things getting more expensive so many organizations are like spend less spend less right but if you were to deploy an app and someone from the sea level came to you and like less CPU use less CPU like like what do you do with that like you can't just continue to dial that down in the same with costs so my notion with cost is track it like any other resource and focus on like the effectiveness and and less on the driving down cost it's about the value that you created at the end of the day yeah I think like there are really hard thing about cost is you need to find some balance between like the cost and the reliability or like the how the code is actually running I do think like the industry is currently like lacking something in that area like there are a lot of cost tools there's a lot of observability tools but finding this balance is really really hard like I think it's it's like one of the biggest like problems I think currently in the industry we try to solve part of it but I think that like something do need to change in that regards like not sure what okay so I think we're just about out of time thank you all for this really excellent panel I think we got to touch on a lot of things I didn't actually get to ask the last question which I think was the probably the one that everyone wanted to hear was about like kind of open source versus commercial tooling a little bit on in the space but we're done we're out of time so we'll have to do it next time better luck next time but all three of them are here on the ground you can you know talk to them Honeycomb has a booth Commodore has a booth do you guys have a booth? They probably has a booth everybody has a booth everybody gets a booth so you can continue the conversations with them there thank you so much for attending thank you Sean thank you