 Hi everyone. Welcome to today's webinar, the State of Observability and Log Management in 2022. We're currently in the midst of a perfect storm of massive data growth and a need for innovation. In order to shed light on key trends, observability challenges, and some of the approaches to resolving those observability challenges, we went ahead and surveyed over 315 IT professionals across a variety of industries. In this survey, we got their perspectives on the current state of exploding data and their struggle to gather valuable insights from that data. Our goal in running the survey was one, to understand what's driving the massive growth of observability data, and two, what approaches will help teams be more productive. Today, we're excited to share the results of the survey and some of the data points that help us answer those questions. I'm Richard Gibson. I work on the marketing team here at Aira Software. There's a few items I'd like to address before we dive in. First, we'll be sharing the full presentation after the webinar, so no need to worry about taking notes or screenshots. Second, there will be some attachments you can open and download during our presentation. And finally, if there's any questions that come to mind during the presentation, go ahead and submit them, and we'll save some time at the end to answer those questions. Before we get into the webinar, I'd like to introduce you to our two speakers. First up, we have Todd Person, who is the co-founder and CEO of Aira Software, and presenting with him is Stella Udovicic, who is the Senior Vice President of Marketing at Aira Software. Stella, the floor is all yours. Thank you very much, Richard. It is my pleasure to share very exclusive findings from our first-ever state of observability and log management reports with you. As Richard mentioned, here's some of our goals. The primary research goal was to capture very hard data on trends, observability, and log management trends, and understand what's happening in the market going forward in 2022 and beyond. We ran this survey in February 2022, and as Richard mentioned, we surveyed more than 300 professionals, both executives and individual contributors. And all professionals had responsibility for managing availability and cloud environments, application, and infrastructures. So here's some of our demographics. We divided demographics in this volume of data and industry. We gathered data from across the spectrum on industries from fintech, technology, transportation, energy, et cetera. And also, we targeted customers, enterprises, and organizations that have at least 10 terabyte of log data to manage. And when it comes to different role levels, we had almost a third of distribution of IT executives, a third of DevOps and SRE practitioners, and also cloud and application and enterprise architects consisted for about 34%. Regionally, this survey focused on North America with about 75% of people coming from United States and Canada, some in Europe, about fifth, and a little bit in Asia Pacific. So there are three big groups of trends that emerged when we ran our survey. The first group is that overall, there's a huge volume in growth in observability data. So the first question that we have asked is, how does your IT organization deal with all that data? Majority of users, and what are the uses that organizations have? And majority of users are in troubleshooting and monitoring, the performance of applications and infrastructures, improving security, and also supporting IT audits. There's one very interesting finding that we had captured, and there's also that log data is still very heavily used for understanding not only user experience, but both user and product experience. And that's very, very interesting, despite some of the over the years investments made in more specialist tooling, such as application performance monitoring. And one of the reasons that could be the case is because APM tools are seen as very, very expensive. So people still resort to use logs to understand user experience. So we expected that IT has variety of users from log data, but we were also pleasantly surprised to see that insights from understanding, from analyzing log data is seen across the organizations and also within the business, in lines of business stakeholders, for a variety of use cases. One is understanding customer activities, compliance reporting is big, and also, as I mentioned, improving product and user experience. One anecdotal finding is that less than 1% also commented that they used insights to predict failures. So that's interesting to find, well, I kind of expected to see more of using logs and applying AI and ML techniques on top to predict failures. And this is certainly something that we'll expect to see in the future, due to massive amount of log data, we're going to see more application of machine learning and AI technologies on log data to make this insights available for all in an accessible manner. And in this next finding, there is almost unanimous consensus that from our server responded that all log data is important for IT outcomes, as well as vast majority of our respondents said that log data is also essential for business outcomes. And about 70% state that it's critical and very, very important. And the larger the organization that has more log data to manage, they're seeing that IT outcomes is even more important in those larger organizations with lots of data to manage. And one of the reasons is that when an organization thinks that they will get useful information for critical insights from log data, then they will work to harness that information from log data, and they have volumes of log data. So still very, very interesting finding. And there's also universal agreement across all that overall in the coming years, we'll see that log data volumes will just keep continuing to grow. So what are some of the data sources that are driving that growth? Within top three data sources are infrastructure, security, and cloud services logs. And that's very closely followed by application development, containers, environments. And as a standalone sources, we're seeing content delivery network accounts for 22% of growth of log data that people selected as a major source, such as Cloudflare. And I find that's really, really interesting. So what do we think about content delivery networks as a major source for log data harvesting? Yeah, I think one of the things that we've seen is that, you know, I think you highlighted the places that people tend to use log management heavily for infrastructure, security, and cloud services. Those are seen as more critical logs. And I think as we look at part of the volume explosion, it seems like teams have to make decisions about which logs are the most valuable. So I think CDNs are kind of lower on the percentage spectrum, primarily because they tend to be one of the first ones to get tossed to the side. So I think we've actually seen a lot of interest, you know, especially with services like Cloudflare where you can actually get, use their log push service to receive those CDN logs. I think as we see more affordable and more efficient ways to manage logs at scale, I think we're starting to see CDNs as one of those easy wins for some of these teams to be able to get back insights into what's coming into their network. And I think in particular, as we start to look at the rise of the importance of cybersecurity, we're seeing CDN as kind of a leading edge for being able to assess threats, you know, look for unwanted behavior before it comes into the applications. So I think, you know, while it may be a lower percentage on this graph, I think it's a place where we see a lot of a lot of room for growth in the future. Thank you very much, Todd. Now, moving on to the next question, the set of finding is that IT executives and enterprise architects report the most types of log data growth when compared to DevOps, SREs, and operations professionals. And one thing it could be that because of their position in the organization, but also interestingly is that if you look at security data growth here, there is a discrepancy between executives and the rest of the roles we've pulled. So as executives consider that security source as a source for data growth is going to be for 78% while the rest is kind of at least like almost 10% is a lower. And it could be that IT executives also need to keep a close track of that data growth, so they can manage and then security is seen as a big source for data growth for them. So it's interesting finding when you analyze different, the realest differences in responses between different roles. Now, when it comes to how much log data is expected to grow, there is not a consensus whether it's going to be 25% and everybody said that, but also there is something that is going to be two to five times growth, for instance, 20% of people, something that there's 50% of growth. But still, the overall trend is that data will continue to grow. And as you compound that information across several five years, this is just in 2022, we are seeing kind of skyrocketing trend in overall growth of this data source, data type. So even as log data is growing, it seems that not everyone is happy about that growth. And in fact, the majority see that while it is very necessary for log data to grow, they have a very mixed feelings about this growth because log data is seen as kind of massive and coming variety of forms. So it's not easy to reap the or glean insights from massive amount of data, at least not with existing methods. So before I dive into the next group of research findings, what do we think about this kind of set of findings that they're summarizing in the first part? I think that the thing that's most interesting is that there's basically a, everybody assumes that these data volumes are going to keep growing. We're seeing come from a lot of different places. And I think really the only question is, depending on each individual respondents sort of perspective, like, do they think it's going to grow a medium amount or a huge amount? And so I think really we're just seeing, I think we're seeing kind of the result of a lot of pushes towards moving things to the cloud, scaling up infrastructure. I mean, I think a lot of this even started with kind of the rise of containers five to seven years ago. We're just seeing a lot more a lot more systems, a lot more microservices and a lot more infrastructure. And as a result, you know, monitoring these systems, keeping them healthy, I think is just it's starting to show us that the data volume required to really manage these systems well is going to be pretty massive. And I think, you know, with the pie chart you showed about people's feelings about the log growth, I think we're starting to really see that practitioners are struggling to figure out how to make use of all this data that they're generating. So I think it's I think it's going to be a continued challenge as these volumes just continue to grow year over year, if the tooling doesn't change. Thank you. That seems like a perfect segue into our next set of findings. And then as Richard highlighted, it's kind of a perfect storm of changes in the application architectures and more massive adoption of clouds. So so let's see how it practitioners and it executives deal with all that data growth. So 76% of survey responses found that enterprises do take steps to minimize the overall log data volume and log data growth. And the biggest one is that they just store the first group is 62% said they just store the only critical data. And some of them decide to quickly erase data within 24 hours. And what's really amazing to me is to see that almost 20%, 80% of IT teams we pull choose to disable logging, which could be quite dangerous, right? You know, incidents may happen right at that time. And you're choosing essentially to find blind. And another answer which we found is an anecdotal but frequent response is that enterprises do erase at some point, but they keep log data for longer times as the log data is needed for compliance for audits and for forensics analysis. So delete log data, but after it's useful then is still needed for troubleshooting and all these analysis that they just mentioned. And it seems also interesting that IT executives are far less likely to report all the efforts to minimize log data volumes. And one thing it could be either like those efforts really don't reach them or it's something that they're not reporting on, but it's something that essentially one needs to keep tabs on because of all the data proliferation also incurs rising costs. And then when it comes to those costs, almost 80% state that they are trying to minimize costs because there's various methods that they're trying to minimize costs. One can use like an offline storage method like S3 or one approach is that to try to reduce licensing at least costs for commercial vendors is to use open source tools. And some of the danger is that like there's still costs associated but it just may not be tracked in a more traditional way. There is still infrastructure costs that need to be taken into advantage, but it's interesting to see that there's a huge group of people about 40% that chooses to route data to less expensive at all. So there's a lots of effort invested into figuring out costs and not just simply managing volumes. And there are also mixed results when it comes to the success of this effort. So some of the survey respondents said that they wish they had data they erased, or it's very difficult to access data once you store it to some offline method like a cold storage. And only 12% think that all these efforts they're making to reduce the volumes and costs are effective. What do you think about this kind of success of existing efforts or moving data to things such as S3 for instance? Yeah, I mean I think the use of S3 or GCS just you know object storage in general is it's interesting. I mean I think it kind of indicates that those logs have some value and while there may be some artificial limits for storing in primary log management systems, whether it's you know a cost or just hardware limits, they can't store it all in the systems that they want to. They don't want to get rid of them, they don't want to lose them forever. So they sort of put them in a holding place you know it's sort of like you know basically like a data lake like a low quality data lake you can just send archives of logs there. But you know it means that there's probably a desire for those logs to be useful at some point in the future. But the really the trouble comes in when you're trying to balance the growing data volumes we were talking about earlier with the kind of fixed costs of existing systems and those those costs generally aren't going down but the data volumes are going up. So at a certain point you kind of hit that decision point we have to figure out what to do. And I think again it just it comes back to kind of what we were saying at the beginning of this section which is it feels like there's a need for new tools that can kind of change that cost value paradigm a bit because we're seeing that folks are trying a bunch of strategies to reduce costs. But ultimately it feels like they're they're having to take a path that really minimizes the value that they can get from logs because things become considerably less search searchable when they're just you know static archive left on object storage. Yeah lots of lots more lots of fascinating findings here which brings us to the next next set of questions. Next question is we wanted to understand what are some of the challenges in dealing with log data and our survey respondents identify all those challenges and they choose and what are some of the steps they find really hard to do. And it turns out that preparing filtering and cleaning data as seen as so hard a step which is also storing data followed by the storing data in the cost efficient way. And also another another response that we captured that is not in this chart is that people find the event correlation not easy which also underlines the fact that you just cannot dump data to things such as S3 but you have to have the data accessible and easy to consume and correlate and then figure out insights and queries that we need to run on those on those data. And then moving to the next one 97% of IT practitioners report that existing tools have challenges in particular because they're not built to handle this huge massive amount of log data. And there are a variety of things that those challenges are. The first one is it just takes too much time to analyze data that are coming from a variety of different tools like a traditional tool proliferation is still not sold. Then you see the resources that are dedicated to manage all these tools the larger organization the more resources they need to manage these cool tools. And then some of the also very answers that are captured is that different departments have different needs and want different tools. So when that's the case how do you see the complete picture across all the departments and there's no there's lack of logging standards which also makes ingestion pretty hard which goes in alignment with the point what we just made earlier that preparing filtering and making data easy for ingestion is not seen as easy. And also another one is there's no central place to capture all those data. So there's lots of challenges also scaling a price with the scaling of data volumes is also seen as another challenge. And this is where kind of skill ability of tools comes into picture so that we ask teams to evaluate the risks they face when you grow your data volumes but your tools may not scale with the amount of data that's coming at you as Todd mentioned earlier from this kind of adoption of different application architectures like all different sources of data that you need to handle that you didn't think about which also can have impacts such as one obvious one is troubleshooting but a troubleshooting in kind of long time it takes for incidents to result because you need to deal with all these data sources so troubleshooting takes longer but also one interesting response that we got is that insecurity risks are larger and people complain that they can expose and accidentally log per PII data as well as credentials which kind of brings again into the mix that especially for security scaling all the data and making sure the tool support the massive data growth is essential because your security interests are going to be higher if you don't think about the massive data growth so the other risks are applications become less reliable you may lose some monitoring data and then ultimately all that could lead to loss of customers and direct revenue impacts and when it comes to executives it's also interesting that executives are more aware than you would potentially expect of resolution times and loss of revenue impact when compared to the operations or IT practitioners and it's very always interesting to see what is those discrepancy between different roles and you know it's in the second one incidents take a longer time to resolve that seems to be a top of mind for executives because that ultimately impacts customers and then I'll turn to Todd to share our third group of findings here so I think really just looking at the I think basically what we're seeing here and I'm trying to think about how to recap everything we've already said is that you know we see that the amount of data that people are handling that are going to have to manage going forward is going to grow the costs are rising and we're seeing a lot of the tooling choices that I think folks have made are directly impacting how much they're spending just managing these log volumes and so essentially what I think we're proposing I think what we're advocating for here is that we we see that there's a necessary change that's coming in the log management space that I think is going to need to happen in order for these log volumes to be able to be handled with the you know we at one end we're talking about 10 to 50 percent year over year growth and as high as you know 5x year over year growth and so I think in order for companies to be able to really get value out of the logs that they're creating the logs that we've you know kind of already articulated that we know there's value in them the tooling really needs to evolve to get to a point where not only are the costs manageable but are the insights that people want to get out of those logs manageable to make it you know worth worth collecting and keeping that data so that that you know the entire business can get use out of it and I think Stella one thing you were you were saying on the the slide about the the awareness you know it's it's not it's not too surprising to me that you see the the IT executives are the ones that are like most concerned about revenue loss of revenue they're most concerned about MTTR and whether it's you know reality that it takes a long time to them I think when they're looking at business impacts it feels like a long time and so I think the more the harder it gets as the data volumes grow to get insights the more value there is going to be further and further up the business to have tooling that really gets you to you know ingesting the data volumes that that need to be ingested and getting insights regardless of how that continues to scale year over year and then I think the the other thing that is really interesting is that you know observability has been around I mean as a term let's say less than a decade maybe seven years or so I think the thing that we're seeing is that there's still a lot of folks that are early in their in their observability journey so I think looking at this slide you know only 11 percent feel like they have a mature observability implementation I think you know it really just says that there are a lot of tools out there but there are still a lot of people who are evaluating and adopting tools but really figuring out how that works into a full full-blown organization wide strategy and so I think if you look at the folks that are you know to the right on this graph that have either never really embraced observability or heard of it or you know just haven't picked any tooling yet there's still a pretty big subset of folks out there who are early in that journey I think are really trying to figure out what this means for their business at large and so I think what we'll see is as these as these folks continue to move to the left you know there's going to be a lot more opportunities to bring new companies into the observability space but I think also we're going to just continue to see those folks who are new to the journey be surprised by the amount of data that they're generating whether it's you know logs metrics or traces and so I think it's you know while it feels like a relatively widespread concept I think this data is telling us that there's still a lot a lot yet to go before we reach maturity in the observability space and I did similar survey last year and it's very interesting even though it's still early that compared to 2021 we see the overall growth of observability data adoption is really staggering 180 percent so it's it's growing but we still have a room to grow overall observability and Todd moving on to a little bit kind of if you can walk us through some of the next findings and the kind of the types of data and what are the different varieties and etc. Yeah definitely and I think the you know as we as we talk about observability we think about kind of there are three you know traditional pillars logs metrics and traces and so you know logs have probably been around for the longest or the most ubiquitous like they were they were kind of the original form of getting data out of out of these systems and so it's kind of not surprising that logs are the most data but also you know as data volumes continue to grow logging is really the easiest easiest to add easiest for application developers and I think the other thing to keep in mind is that the way that most of these metrics are generated logs are generally generated you know per request you know like the the more users interact with systems the more logs are generated you know in in direct correlation and there are logs from other things as well but I think metrics generally tend to have a relatively stable output cadence so if you look at you know kind of pre-media standard it's you know data gets reported once every 10 seconds or every 30 seconds depending on how you how you have it configured so those generally are decoupled from you know growing user volumes so I think we'll continue to see log data be be the driver I think in terms of of most data and also if we're looking at it in terms of just you know you know gigabytes of whichever data format logs also continue to get more and more of our boasts we can keep adding more tags and more fields and more descriptive modifiers and I think logs see that a lot more than I think the other other data formats you know and then coming back to the data variety you know we mentioned earlier like it can be cloud cloud native deployments it can be kubernetes infrastructure it can be cdn it can be application logs they're all these different places where logs come from and I think sort of by almost by design you know logs have that flexibility to really be able to just be inserted anywhere metrics continue to be relatively well structured and the places that metrics are exposed are also well structured so I think we're going to we'll just see you know logs come from a wide variety of sources contain a wide variety of information within them and you know continue to be you know for better or worse less less structured than the other types of metrics and then when it comes down to cost you know I think these other two you know drivers really lead to cost having more data having more variety you know obviously the higher data volumes is just expensive because it's more data but I think logs because they are so they have so much variety they're harder to index on they're harder to scale they're harder to get value out of the queries end up being more complex and so I think kind of all all three these pieces here really feed in together to just say you know logs are a place where I think companies are going to continue to invest heavily I think they're going to continue to expect that there's a lot of rich data in those logs and I don't see that trend changing anytime soon if anything I think in some of these we might see you know logs continue to grow more potentially that's that's something that on this graph also is that you know the cost related to managing tracing is also very interesting just by the sheer amount of traces it could be billions of traces recorded so it seems also that sampling needs to further grow to get you know get the cost down because it's it's it's a it's this proportional amount of costs that's associated with tracing as well compared to the other methods it's kind of very fascinating definitely I think that's a great observation um thought moving to the wide variety of tools to manage it yeah I think we you know we talked a little bit about some of these the offline cold storage options and I think those make sense you know I mean in terms of you know cost per unit of storage it's it's hard to get much cheaper than than S3 you know for the especially for the durability that you get but I think what we're seeing here is that as these data volumes grow that we're talking about you know we see companies that have been traditionally heavy users of tools like Splunk start to look for more affordable options for for some of these bigger data volumes potentially data that's less mission critical and so we see the adoption of open source tools I think elastic search is very common in this space and we see the you know kind of that stratification of cost and we see it sort of get pushed down to lower tiers but the thing that I think companies are starting to realize also is that even as as those tools are embraced and start to grow now someone has to maintain those and someone has to manage a new piece of infrastructure which has its own cost it has its own operational overhead so they're sort of the you know cloud based blog management products that are that are in the mix and there obviously are a bunch of folks out there and I think what we're seeing also is that those costs have continued to grow because as as these you know largely sass players I think in the log management space see their their data volumes grow you know that costs get passed on to customers and so I think whether it's you know using traditional log management products or kind of doing a DIY manner hosting your own elastic search cluster or pushing this cost to the cloud I think what we're seeing is that it's becoming a big expense and I think with the data volumes that I think a lot of companies are starting to approach now it's becoming I think very difficult to to fit into the traditional I think budgets that I think people have a plan for for observability and so I think really what we're what we're you know looking at I think is the big sea change that's coming is finding a way to straddle some of those traditional costs whether it's you know on-prem or in the cloud versus the the low cost of object storage and finding a middle ground where you can leverage some of those cost savings but still get the you know the queryability and data insights that you want from that log data while while finding ways to over time kind of bring those costs down I think this this slide also is interesting just kind of looking at how many how many companies actually have a team that can manage some of these these tools and so you know 37 37 percent say that they don't have any dedicated resources so it almost is a given that they're going to have to use a cloud service or something that that's managed by another vendor but 20 percent have a dedicated team a full team to manage their log tooling and so you know whether it's that 19 percent or the other 44 percent that say they have some some individuals that are responsible you know there there's a significant investment there because generally these are going to be you know technical employees they're going to be you know relatively advanced with the ability to manage these tools you know there's a significant cost there that goes above and beyond any software licenses or any hardware that you're having to pay for to manage these platforms and my guess is that if we took data took the same data from a year ago these numbers would have expanded that we would see more dedicated teams more dedicated individuals spending their time on this and I think really just you know whether it's logs metrics or traces I think a lot it's been the values is increasing to companies but also so is the investment in keeping these systems running and you know the I don't know unsurprising part of it is the more these are used the more they become part of the critical path for uptime incident resolution and so it's it's not acceptable for those systems to be down or to fail so having those folks in charge of keeping those systems up and running you know is it becomes a critical business function and I think this is just showing us that the bigger the company the more data the more likely they are to have a dedicated team you know our dedicated set of individuals that are managing this and this comes back to I think kind of what we were saying about you know business outages and loss of revenue I think as these systems become more critical more complex and more a focus of you know how really how outages affect the business you know we see more and more resources from a human capital perspective and also from a you know just infrastructure and hardware expenditure on on managing this data and getting value out of it so yeah I think with all the things that kind of led up to this this slide isn't super surprising to me it probably isn't to most people and then I think one other thing that we've been starting to see a little bit and I guess this feeds feeds a bit into sort of the overall narrative here about growth of observability data how this gets managed in larger companies by looking at you know stream processing or streaming observability data where we're starting to see some of the early phases of this and I think this is kind of lagging the overall observability market a bit because it's it's becoming more important more valuable I think as these data volumes have been growing and so I think probably going back a bit a lot of the a lot of existing observability pipeline work is probably built around something like Kafka or RabbitMQ or some of the other kind of existing open source just general general purpose message queues but I think what we're starting to see is that other solutions are emerging that give you the ability to have something that's more specifically focused on observability data and primarily recognizing that that data is it's time series data there are certain things you want to be able to do with logs or with metrics whether it's from you know the systems that you support and route data to and from the ways that you want to you know clean that data up the way that you want to be able to potentially look for redact PII I think a lot of that data becomes very specific in the way that it needs to be handled and so I think this is really showing you kind of can see almost like two two cohorts in here and the first is folks that have some sort of a of a tool or in the process of picking a tool and my guess is that most of the folks that say they have a fully deployed solution are probably running Kafka or something like it under the hood with some newer vendors in the space that we see some adoption for but then you know you do see almost a quarter of respondents are evaluating options and then another 14% are considering evaluating options so there's sort of becoming this set of problems that I think observability pipelines are made to handle that are becoming more important to companies and I think a lot of this kind of the subtext I think is that a lot of this comes back to the things we talked about previously which are managing the complexity of data managing the cost of the data figuring out how to optimize your you know infrastructure resources to get the most value out of this data and so I think it's it's still very early but I think observability pipelines are going to be a thing that make more and more appearances in the overall observability space and start to become that connective tissue between all these different tools that I think people are using and then again this this kind of breaks down you know by role whether we're looking at IT executives architects or DevOps where the observability pipeline kind of investment fits in unsurprisingly you know the architect role is largely I think pushing this as they're starting to think about you know all the different tools all the different observability tools that they may be using all the different data sources where the data is coming from and and how they think about you know across a large organization how do they manage that data and how are they thinking about the growth of that data and what they're going to need to have in place to make that something that's scalable down the road and then you know I think this one's super fun everybody thinks there's value in innovating for in observability and you know whether it's in the the what the tools are capable of or the type of data that can be supported you know I think it's it's recognized I think pretty broadly now that observability is is potentially a huge could bring huge value to its organizations and I think you know as as the the proliferation of tools continues as the data volumes continue I think it's becoming clear that there's no there's no perfect solution out there there's no there's really no single tool that I think does everything that teams want and so I think really what will continue to see over I think the next next few years is that we'll see probably more folks move towards and style I know this is your one of your favorite terms that the single pane of glass like I think there there's a desire for more teams to be able to see more sort of cohesive views into this data and it's not it's not a super easy thing to do because I think we're you know especially when we're talking about data in you know the petabyte scale there's a lot there there's a lot going on and a lot of different sources and a lot of different stakeholders but I think we'll you know we'll continue to see more and more I think interesting innovations in not only just how we store this data at scale but also how we just how we get value out of those tools and how we start to pass that value further further up the chain within these organizations so that more more people higher up can can get value out of the the insights that they're gathering from observability data and then yeah I think this goes back to another slide earlier that you had Stella but I think having having tools that I think do a good job of supporting this observability data growth gets people more excited about what's possible I think a lot of what's happening for the teams that are not interested is it becomes so much about managing the infrastructure managing the costs and the pain associated it becomes almost this I don't know constant constant stress that you're trying to manage these things that you that are growing out of control and you can't actually get to a point where you can sit back and enjoy having having really good tools in front of you and so I think really this is just showing us the more mature the observability infrastructure is at an organization the the more exciting I think the insights are the more value they get from the data and the more I think the organization feels like more data is better and I think that's that's what we've we've all wanted to believe but I think we've been all we've been hindered a bit by by tooling in the observability space and yeah I think this is just some some hope hopefully for those of you out there that have been struggling that modernizing your observability infrastructure can can be a good thing can be achieved now moving on to more of a summary of our discussions we you know there's some there's lots of excitement out there and to underline there's still innovation is urgently needed how do you see these percentages it's very high percentages that are reporting some of the kind of big statements here yeah definitely I think I think this this fits into a bunch of the things we were already saying but you know you know just sticking your log data in s3 you know you've achieved the goal of storing it but you haven't achieved the goal of getting any business value out of it and so I think really it's it's you know maybe the last slide was the perfect one to kind of fit into this so you know 100% of people think that the yeah sorry the previous yeah 100% of people think that there's you know value and innovating in observability you know it's not surprising that that we see these these high percentages so yeah we need to be able to find new ways of storing this data at scale that are that are queryable that are actionable we want to be able to make the tools actually solve the problem rather than just becoming a pain point for engineering teams and you know sometimes you just take the problem and you just push it onto you know kind of those dedicated teams we were talking about before and then yeah and I think you know innovation I think is going to be key here moving away from some traditional traditional tools and I'll I'll throw I'll throw elastic search out here you know it's it's been around for quite a while it's it's a relatively mature tool it gets used a lot in in log management use cases but the the kind of my my summary of it is that it's it wasn't built for that it happens to be a good fit for some of those use cases but ultimately it tends to become more and more painful and more and more expensive as the data volumes grow and so I think we're seeing a lot of that right now I'm starting to play out in the market and so yeah it's it's comforting to me to see that everybody seems to agree that there's some innovation needed in that I think there there is a possibility to I think do better with observability through a lot of the I think the new products that are starting to come to market now that we covered quite a lot of research we gleaned some new insights some expected some unexpected thought could you share with us a little bit about what are some of the recommendations we can have for people out there in the space yeah so I think some of these are going to be relatively straightforward but I think you know as you start to look at your at your options at your at your existing choices for observability there are you know products that are becoming available that are you know leveraging cloud native architectures that are you know sitting on top of things like s3 or gcs that are are finding a better way to sort of dynamically stratify these these log layers that I think people are starting are trying to figure out themselves so I think really you know keep an eye out for for tools I think help with that cost management I think we'll start to see a lot more of those out there in the market and I think right now a lot of the tools that folks are using I would say are relatively I guess are sitting on the kind of a previous generation of of technology and so I think a lot of the costs that are out there in the market right now are artificially high you know I think and along with that I think there's there's a desire to keep especially with logs not so much with metrics a desire to keep data around for longer periods of time so I think you know as we see desires grow to have you know audit and compliance ready storage for logging there's also the ability to be able to go back and run potentially you know machine learning models on on historical log data to get customer insights you know I think starting to factor those those storage terms into your plans and then figure out how you can how you can leverage your tools to be able to to make that storage work in a cost effective way it's going to be important finding ways to expose some of this observability data to other parts of your organization I think largely it's been mostly restricted to dev ops and sre but I think as we start to see you know more of these mature type of observability infrastructures we discussed I think a big part of that is starting to surface that data surface those insights surface the you know the the MTTR is the the incident responses to other stakeholders is important you know I think paired up with the the low cost and and long-term storage just finding finding tools that give you the power to get extract data from your from your historical archives whether it's being able to have it online and queryable or it's you know being able to have a set of of tools that can quickly look over some of that cold data without having to you know delegate individual engineers to go pull that data down and and and look through it I think there are a bunch of different ways to go about about getting there and then I think kind of onto the innovations in observability pipelines and observability data management looking for tools that can help you with some preprocessing some you know overall reduction of data volumes I think I think that's going to be something as I said that is is growing a lot and gets a lot of innovation going forward and we'll start to see that appear in a lot more organizations and then yeah just integrating with with other existing tools and so I think a lot of this is as you look at potentially observability pipelines you know you don't necessarily have to replace your tools I think especially if you look at some of these tools that are deeply embedded in you know security organizations or have business critical insights you don't need to replace them you can find tools that can be operated alongside them and you can use observability pipelines and things like that to figure out which data goes into which tool and how do you kind of shuffle that that data in an automated way and make it possible to you know essentially let the tools rebalance that data for you so that you can you know more effectively optimize the costs of the tools that you're choosing to put data into and let the the use cases or the needs from that data drive you know which which tool and which cost profile is associated with that data so they kind of you know all these things together I think are things that will start to see more and more as these log lives continue to grow but I think the upside is that there are ways and tools and strategies I think to get to a manageable profile and I think again going back to one of the earlier slides the more mature organizations that have gotten to a place where they figured these things out they do actually start to get value out of that observability data and get excited about having more of it available but I think a lot of it comes down to picking the right tools and strategies to to make sure that your organization can be successful thank you very much thought now moving into the the q&a portion of our presentation we covered quite a lot of insights and if you'd like to get the full report here's the link you can download it for free and read the details and keep it for you and kind of see if you if you find this summary this report something that you've seen in real in practice so over to Richard to see if there are any questions on the line yep we do have a few questions the first question is based on findings what do you see for observability going beyond 2022 this is for bud yeah happy happy to answer that one um yeah so I mean I think we we talked about it in a bunch of different ways but I think the biggest piece of it is that you know observability is here to stay I think we're we're just going to continue to see data volumes grow I think jumping into a parallel space you know IOT data is another it's another you know time series use case that's similar to observability you know that that's also been continuing to grow and largely fueled by the number of devices that are out there the number of data points that you want to be able to get insight on and I think what we're really seeing is is a lot of these parallel evolutions where you know that the compute resources are becoming more ubiquitous the number of whether it's microservices or containers or individual processes that are being monitored you know we're seeing that hardware become hardware and sort of the the low-level software becoming commoditized to a point where it's just becoming utility and I think the the growth of observability data is really kind of just the the tail end of that of that growth and now everyone expects there to be um logs and metrics and charts and graphs and insights into all of this stuff and I think that's just going to continue to grow um probably not you know I think we had some of the initial some of the respondents that you know thought that their organizations might see you know 5x growth or more um but I think my guess is that a lot of those a lot of those companies are seeing that growth because they're in the middle of their you know kind of cloud migration journey or embracing tools like Kubernetes and they're starting to see that but I think the the industry as a whole I think we'll see pretty healthy year-over-year growth for quite a while um yeah and I think just going back to the things we said before it's going to come down to tooling choices to I think really get value as that stuff grows um and yeah I think there's going to be a lot of a lot of cost explosions and a lot of pain for some of these teams that are managing these tools but I think we're I think we're in a place now we're starting to see the rise of a of a new set of tools that will will make these challenges easier we have more questions Richard or yep we do thank you Todd the next question is were there any surprise findings in this year's survey I think that's okay go to Stella um yes um I spent last 10 years in you know monitoring observability log management space um some of the some of the responses are quite expected such as you know troubleshooting times take longer or proliferation of tools but some of the findings such as if you don't scale with the growth your tool doesn't scale with the growth of data um that you will be exposing um yourself to PII risks or accidentally exposing credentials um and some of the kind of the the the more urgent needs we have in security space to to take care to take uh you know harness all the data is very very interesting to see especially as the amount of data volume grows that kind of keeping tabs on incidents and potential risks and security is becoming more urgent than ever in this space so that's something that was very surprising for me because this survey specifically targeted IT organizations and those survey the security realm is kind of shining through even for for for even though we didn't pull directly security professionals so that's very very interesting finding for me all right we have another question here does AERA software offer a self-managed deployment yeah great question and uh we didn't we didn't talk too much about the specific product offerings but yeah I think um our our AERA search product is definitely deployable on-prem um or or in a you know a self-managed cloud um I think it's it's been it's something I think is somewhat unique to log management versus metrics and traces because I think to Stella's point just a minute ago um logs tend to have a lot of risk for having PII and so I think for a lot of our customers being able to take our product and run it in their own infrastructure whether it's it's cloud or actually like physical hardware um is a relatively big selling point and I think there are a lot of other SaaS vendors that you know they're SaaS only and so it becomes very difficult to actually have full control over that infrastructure so um I think given given the potential for you know security breaches and things like that I think being able to go um on-prem for for log management is actually really important I think it's going to be a kind of a a key part of our business going forward um and I think it also just gives it gives those customers the ability to run the software they want and control how they manage costs and whether they want to you know move to uh you know a different hosting provider or negotiate discounts on hardware um you know they're they're free to do that and the you know the price of the software can be a bit decoupled from the actual hardware that's being run on it. All right thank you and then this question can go to you as well Todd what are your expectations for streaming pipelines adoption? Yeah awesome um I get pretty excited about this because I see that we've had um you know there there are lots of logs coming from lots of different places I think going to some of the charts we highlighted earlier um and I think you know we've seen a bit of um my new favorite term agent sprawl uh kind of in recent years and it becomes very hard to get the the right logs from the right places in the right format deal with back pressure deal with failures and you know especially as we've got um you know really infrastructure running all these different clouds all these different places you know downtime happens all the time and so I think finding a way to remove the reliance on on agents to be responsible for um you know really data data cleansing data formatting data transformation um and pushing that to to a more centralized place um you know whether it's you know just uh a very generic message queue something like that or more um you know uh specific type of observability pipeline I think we're going to see that that's going to become more common and it's going to fit better with the the tooling and trends that I think modern development teams are starting to embrace whether it's you know get ops style workflows uh being able to push configuration changes um more you know more dynamically um whereas I think rolling out configuration changes to thousands of agents across your entire infrastructure is actually pretty pretty difficult to do pretty difficult to coordinate um and I think that that evolution of the observability pipelines and the broader kind of observability data management concept I think we're going to see become something that more more organizations and more teams are going to become reliant on and it's actually going to start to be a pretty big part of how um those orgs manage these huge data volumes across lots of different tools and have the have the confidence that things are working the way that they want um and that they can continue to make dynamic changes as you know as new products ship as the company changes as as teams grow and as as volume scale um so yeah I think we'll start to see I think it's it's still really early in that space I think we'll start to see a lot more of that over the next year or two for sure um we're coming to almost four hours so if there are more questions please reach out to us directly and with that I turn to Richard to say through your closing words yep thank you thank you Todd thank you Stella thanks everyone for participating in today's session keep an eye on your email we'll send your recording of the webinar if you have any questions or want a personal demo you could reach out to us by visiting era.coco slash contact and filling out the quick form that's there you can also sign up to try air cloud completely free by going to era.co and clicking the free trial button in the upper right corner and you'll get your first terabyte of log data for free I think thank you from all of us here at era for being part of this webinar we look forward to seeing you at our next event thank you Richard thank you thank you thanks everybody