 This is your host up in Bhaktia and welcome to another episode of TFR. Let's talk and today we have with us Josh Williams Head of engineering at Observe IQ Josh. It's good to have you on the show. Thank you for having me today We're gonna talk a bit about you know, the whole evolution of Observability space but before we go there just to remind our viewers What is observe IQ all about? Observe IQ is a company completely centered around open telemetry and collecting just in general, I would say facilitating collection for customers in regards to telemetry both logs traces and metrics and Really enabling customers to send those that telemetry to any platform or destination of their choice And can you also talk a bit about how have you seen the evolution of Observability itself in the cloud-centric world ever since the not only foundation of the company But if you look at open telemetry also, it was the merger of two projects, you know open sensors for there, you know So so talk a bit about that Yeah, absolutely. So, um, I have been working in the space for probably I would say seven or eight years now And it's been a very interesting journey for me to watch how Telemetry the telemetry industry has evolved over time. So obviously very early on All of these different platforms and vendors had very specific API is very specific requirements for collecting data We've seen that transform over the last year or two with the advent of open telemetry Where there is a now consistent model that is being used the one place that I have probably seen the most evolution in my opinion is Not just the standardization of how that data is collected But also in terms of how that data is ingested you see quite a few vendors recently are starting to switch over to Injusting what's known as OTLP format, which is the standard format in open telemetry And that seems to be the big trend right now that I'm observing What is driving this trend what I would say that is driving it would be so the first thing that was standardized with open telemetry was the actual collection of the telemetry and What ends up happening is every single one of these vendors or platforms has to support their own model within say the open telemetry Collector to send this telemetry to their platform so One thing that you would see change over time is that then requires support for what are known as exporters in the collector So they their API as it changes or requirements change The way they ingest that data is going to have to be Maintained over a long period of time Whereas in contrast if you were to accept the data and say OTLP format This is now a standardized ingestion point and you no longer have to support all of these different disparate APIs That accept the data encoded say in a certain format expected to be sent up in a certain particular manner one of the ways I see this is You know some some telemetry platforms will ingest labels For their metrics in a certain format compared to other telemetry platforms. It's actually been a little bit of a Learning curve for us at Observe IQ when it comes to the open telemetry collector So for instance when collecting data and sending to a platform like GCP That data might have to be Massaged or altered slightly differently than the data that is being sent to say another platform like New Relic And that and that is because there are different expectations between the two there are different expectations for their API There are different expectations in regards to how the open telemetry Formatted data is actually transposed and mapped over to the data format that the platforms are expecting So when that changes to instead natively ingesting say OTLP The the user the customer no longer has to reason through that in regards to how they send their data Whereas right now what we're encountering with several customers is they may have an open telemetry Pipeline that they have configured that would work generically But they still have to maybe alter or move fields around in order to facilitate the data to be used Say to its fullest of those platforms. What kind of adoption you have seen of Observe IQ blind plane you can it would be great to see you know some of the either big use cases or some of the Exciting unique use cases. They're like hey, we were not on expecting that use case there. Yeah, absolutely. So As I mentioned before it's been a really interesting journey for us I would say early on in our company's history We were very much focused on collecting the data and so a lot of our efforts were we're centered around say integrations So for instance, how do we collect metrics or logs from Postgres? how do we collect metrics or logs from engine X and that is transformed over time because that is a More or less solved problem when it comes to the old and so much of community once that integration is built once that receiver is built That's a solved problem. What we're now experiencing is instead focusing on cost reduction for our customers So whereas we facilitated the collection of the data It's now then taking that data for our customers and helping them reduce the cost to the platforms that they're sending to and This could be done in any myriad of ways. For instance, we could Reduce the number of logs that are being sent by deduplicating, you know common logs that maybe you don't have to be sent as often as they're being sent or For instance, one of the big efforts that we did recently with bind plane is we actually created a processor and open geometry that allows you to take Logs and convert them into metrics because different platforms may have a very costly log model where it costs quite a bit of money to ingest say large amounts of logs and so what we can do instead for the customers is we can take those logs We can extract data from it and then turn those into metrics a great example would be say Apache HTTP request logs You could turn those into metrics that could be say a metric that is a number of bad requests So rather than having every single log for every single bad request that you've observed and paying for the storage of those logs You can then turn that into a data point that drastically reduces the amount of data that you're sending as you said You know the the adoption is growing but as that option is going and we are also seeing you know, of course The use cases of Kubernetes is also growing beyond what was you know expected which also means that The customers are running into problems challenges that we did not see before or you're like Hey, yeah, this is what we learned to so what are the challenges that you see? The the customers are facing today, which you it's not that hey, you know, we have solid But you're like this is still a problem. We have to solve one of the problems. We're facing That Kubernetes definitely brings to mind Is in regards to our collector deployments that we do for customers So for instance when they're monitoring an application within Kubernetes that requires to deploy the collectors in a somewhat ephemeral state and so with our buying plane platform that allows us to Manage these deployments. We have an issue where say collectors are coming online or going offline as the Kubernetes environment is scaling and because of that We have to somewhat help our customers reason through say One of our main value props is showing the amount of data that's flowing through their system But if those agents are disappearing over time, we don't want to just throw those measurements away And who knows how quickly or that that system is scaling or scaling down And so that's been a challenge that we've encountered recently is we have to look at Our deployment model slightly differently when it comes to Kubernetes and somewhat reconcile that with say our non-kubernetes deployments It's mostly the ephemeral state is the way I would summarize it and how well prepared is observed by Q or The community to address some of these challenges because one one more thing that we've seen that loudly were is Things get complicated very quickly. There are so many projects There are so much overlap even if you look at their fully dedicated, you know project And then there are other projects which are expanding their scope. So how do you look at that? Yes So that's very interesting for us because open telemetry I would say Because it is a large community open-source project one of the the complications that you'll see is there are several different voices in the community that are pushing for say different things that they would like to prioritize and One of the things I have noticed in regards to open telemetry is the complexity of the configurations for customers They're very complex There are several smart individuals working on this project and because of that there are just so many lovers so many things that you can Slightly tweak in regards to batching in regards to how you're handling your memory like limiting that memory So that is one of the challenges we face However, something that I saw very in the slightest side But something that I see that is very interesting around this is with the recent advent of say AI or things like chat GPT One of the cool things we've seen customers do is Feed in to chat GPT The requirements they have for their config that they would like to put together because it could be somewhat Intimidating to build one from scratch if you've never worked with open telemetry before and we have seen customers Be successful in doing that creating quite complex configs that solve their use case by feeding it in so It necessarily isn't a solution to the issue that I see but it is an interesting use of technology to use another technology Josh, thank you so much for taking time out today and of course talk about not only company Ecosystem and of course how you folks are solving this problem for the larger ecosystem there Thanks for your insights and as usual, I would love to have you back on the show. Thank you Thank you very much for your time today too