 So if you can do it, if not, it doesn't matter. So welcome, everyone. We are Bartek and Richie, as you probably know. As per usual, we are trying to optimize these four questions, which also means that if someone can go on Slack and check online, if there is any questions in the Tech Observability channel, for everyone who is following online, please ask questions in the Tech Observability channel. And I presume someone up here is going to volunteer to just read them out. OK, Ian is volunteering. Perfect. So housekeeping aside. So, buddy. Hello, person, please. Thank you. Yeah, so welcome to the Tech Observability thing. What is Tech? Technical advisory group. Renamed from you as well. No, behind you. The person behind you. Person? Hello? No? Yeah. Includes a no. Thank you. No, I'm not taking off the mask. I'm a decent person. I'm keeping the mask on. Anyway, so Tech is the technical advisory group. And observability is kind of obvious, I hope. Your mask. This is highly, highly unfair to everyone else. It's like peeing into the pool. It's simply unfair. You legally agreed to wear a mask. I hate this thing. Just do it. Anyway, as there is huge confusion about all of the structure of CNCF, let's walk through this quickly. And with my governing board head on, we created this really, really pretty slide. But it gives you an overview. So you have the governing board, which has a few subcommittees, like legal committee, like the marketing committee. There's also special communications committee recently being introduced. There is a budget subcommittee introduced. So that's the governing board, which basically sets the whole structure of CNCF. Under the governing board, you have the technical oversight committee, which I'm also part of. Under those, you have the tags. And the technical oversight committee basically is about the lifecycle of the project. Oh, sorry. Oh, sorry. Yes, thank you. I thought you, OK. Thank you. So under this, you have the tags. Recently, we named from special interest groups, which used to be called SIG, security, observability, blah, blah, blah. And then you have the end user community, which also has their SIGs, with a still name overlap of the Kubernetes SIGs, which is why the CNCF SIGs got renamed to tags. And yes, this is all confusing. And this is really, really ugly. But this should at least give you the overview from the amount of people who took pictures of this. I guess this is good content. Anyway, so from our charter, basically, I mean, you can read this yourself, we're about observability and about producing more content for observability. We are about trying to get people on board, but onto observability. Basically, anyone who cares about observability is well placed within this tag. It's not, and that's kind of why tag is a misnomer. It's not just that we are advising the TOC. It is also that we are advising and helping end users. And in particular, we need end users to join to help us hear content and such for end users. There's two common definitions. Like there's 10,000 definitions of observability, because everyone who has something to sell will redefine it, so it suits whatever they're currently selling. Did you change the order? No, you didn't. OK, so like from the initial control theory, coming from math and having mathematical models of industrial control processes, observability is defined as being able to determine the internal state of a system simply by looking at its inputs and its outputs. There's some discussion on if the original quote includes the inputs, but that is the control theory definition. Basically, you're able to ask questions which you didn't know that you wanted to ask before. Put differently. And you can use those terms interchangeably, in my mind. But this other quote also pulls it very nicely that observability allows you to ask new questions while you're debugging, while you're doing your thing, as opposed to just having data which you can't do anything. Like best examples permit this, of course, you can do actual math on the data. It's not just a graph. And done, you can actually do analysis and math on your data. There's the three pillars of observability, not called so because they're the only thing in town, but like they are the thing which everyone in this room is most likely going to start with either trade logs or with metrics and will then work their way towards the other two. Of course, this is like the common thing and that's why these are the pillars, but there's more. There's continuous profiling. There's crush dumps. There's many other things. We see more and more that we have overlap, as we have new people coming into the observability space who define this wider and wider and wider and wider to basically give you more insight into your complete system. Of course, if you already have certain things about like your performance metrics or your API contracts and you already know that this thing is in this and that state, you can use this as part of your whole observability story. Thank you, Regi. So now comes the difficult part, right? So we know theory, but are there any projects that help us to fulfill those rules around finding those unknowns and unknown unknowns, right? So how to find those? We have this landscape is very complex. As you can see, there are memes about that, so it's kind of clear, but there is some rule to this madness and this is why we try to aggregate those projects into groups of things that have in common. So we have this observability group that you can zoom in. There are lots of projects that are kind of members of direct members of the CNCF, either Sunbox incubated or graduated or there are kind of projects related in the open source that are somewhat connected to the observability. And kind of hopefully, I mean, I'm pretty sure you know Prometheus already because that's kind of a default monitoring tool for communities, but there are more, right? As Regi mentioned, that there are signals are growing, growing, growing. There are more types of the information, categories of information that we can present and use in various ways. So there are more tools than just Prometheus to kind of leverage those. And with the stack, we try to ensure, there is some kind of map towards how to use those and which you should use based on your use cases. So let's focus on the CNCF projects now, right? So in the strict kind of CNCF landscape around, Sunbox incubated and graduated projects, we have a few of those graduated, Prometheus, Jagger and Fluenty. So really representative from metrics, logging and tracing. So kind of already those pillars, the basic pillars that we discuss. But of course, we have other projects that represents maybe extensions that more some kind of solutions around that really it's not easy to make sure they cooperate with each other, all those standards and APIs and technologies, but we try our best to make sure those teams talk to each other and there is like a common pattern. And way of using those together in this stack, right? So now with those projects, let's see what we can do with those. So at the end, project is one thing, but what tech is responsible for? What is our kind of future steps? So we want more content, right? So as you can see, the space is very complex. We know the basics like Prometheus, but what's next? How to really have this observability story concretely figured out? To do that, we need to have more educational content and kind of sort of, I would call it an index of, how to start with observability in the cloud native space. So we need more help and feedback and kind of more hands to really create those videos, blog posts, tutorials and demos, right? So there are lots of good contents that someone did over time in this space, but it's hard to find those. So we have to make sure it's really easy to consume. Furthermore, we want to focus on white papers. We already have one, which is really 99% done and it just really missing like last touches, which already explains this landscape in details, those signals, how to potentially use them, what are the kind of tricks and tips and kind of where to start. So you could, I'll tell you where you can find it already, but we need more of those, right? And we need more hands to help with it. And by the way, the white paper that was created was jointly created by at least like dozen people, right? So it wasn't like one person responsible of that because that's not the point. Our point is to build community and make sure we can work together. So of course, part of it best practices, demos, and yeah, really getting more end users, right? So I think we touched that on our meeting on Monday here in KubeCon, but essentially, we really want to make sure that the questions and the direction of the topic is influenced by the end users. So someone that doesn't directly sell observability because then there might be bias, right? And there might be other, you know, they might not see the problems that end users are seeing, right? So especially if you are end user or you know, maybe end users interesting in observability, just direct them to tag observability, please. That would be very, very helpful. So this is where you can really find us. We have meetings, which are fortnightly, Tuesday evenings, evenings, of course, European time. And we have repository, GitHub repository, which is kind of the center of what we do. We put issues there for, you know, anything that we want to do in future. Maybe you want to assign, maybe you want to assign yourself and help with that. That would be amazing. Maybe you have ideas to propose. Do that, do it there. The white paper, you can find in this repository as well. Everything is like in one single place. We also are in the Slack, so please join us. Say hello, and we have mailing list as well. And with that, it's all what we prepared, but we are ready for some questions if you have any. Hopefully you do. Maybe as an icebreaker, we don't bite. Who here considers themselves an end user? Ah, yeah, sorry. Who here considers themselves an end user? All right. Also, of course, I did the mic thing wrong. Again, thank you for pointing this out. For anyone listening to this online, please feel free to go on cncfslack. Hashtag dash observability, ask questions. I think there is a first question. Perfect. Sorry, I was more saying that I'm watching the Slack channel. Okay, cool. But I do have a question now. Shoot. In both of your opinions, obviously we have the state of observability where we are at the moment, but kind of looking one to five years in the future, what do each of you think is gonna be the next best addition to the observability tool belt that we're not using today, that we may be using in three or five years' time? I think we're just going to go both. But my own personal opinion is we don't really lack anything like there's, we have new stuff on purpose and there will new things which pop up, but at a very fundamental level, they are just three instantiations of the original thing. You can see that humanity has always run on events first and then they started to do math and check, oh, even better, summarize them. And this is what metrics are. You can argue that even traces are just a specialized form of events of logs. Like it's not a clear map, but it is a decent map. So what I personally expect is more distillation of all those signals back again into metrics because those are the cheapest way and the most efficient and the most effective way to work with insane amounts of data and allow you to just get more insight into your system with less cost. The other thing is integration, integration, integration. Those are all tools. They're not ready-made solutions. Yeah, and from my side, I think for the future, we definitely, I would say we see more of correlations side of the story, which means that you might have one signal, but how to cooperate with other signals, right? And it is especially important because people will realize, or I mean, hopefully they realize right now that you cannot solve all the use cases with fully tracing and only tracing and then deriving metrics and all of this because of the cost reasons, right? So you really have to look on diversifying your observability to make sure you really kind of be, you are pragmatic in what you do and maybe you use more expensive signals like event logging or tracing or even more gradual event observability. You do that only ad hoc, only for dynamically enable that or disable that. So I would see all those solutions to work better together and I hope for that as well. Awesome. I know we already covered quite a bit on Monday. I don't know who of you were here on Monday. So this is kind of an overlap also. It's the last talk of the day. Everyone is tired. It's still, we don't bite. Questions? Hi, recently Graphana Labs introduced Mimir. It's kind of fork of Cortex. So what will be the future of Cortex? Is it will be still developed and still in the sensors landscape or it will be eventually replaced by Mimir? Are you asking? So I also work for Graphana Labs. So for the people who are not aware of this. So like with my Cortex is a healthy project. Yes, Graphana Labs decided to fork away from it and maintain its own thing. And I like many properties about Mimir. Personally speaking, I even considered not joining Prometheus in 2015, 2016 because it was Apache 2 and not AGPL. And personally I prefer reciprocal licenses over non reciprocal ones. Cortex is healthy. Graphana Labs did quite some work on getting new people, new maintainers, new contributors into Cortex over the last year or so. It's basically for, but it also stopped investing or people's time into Cortex. So it's for the new maintainers. It's for the new contributors who carry the project forward. But it's healthy and it doesn't have any major bugs or anything. So yeah. Okay, thank you. Any more questions? Arthur. Hello. So I was working on the white paper some time ago. And I remember some discussions about the KAL's engineering and how it correlates with observability. And I see a lot of tools listed on the observability landscape that's related to KAL's engineering. But we never got to point to agreement that KAL's belongs to observability and how they relate. Were there many more discussions around this area? Because I'm sometimes away. So I think you mean the KAL's tooling for example, right? And I think there is a revelation in this space because I think they propose to create a tag. I don't know if you remember if they actually did it but I'm pretty sure. I mean, I think it would make sense to kind of separate this kind of KAL's monkey solutions to be a separate, very focused group that really focuses on that. Of course, everything is really connected to observing this KAL's. This is like plant KAL's and really learn from that. But in the same way, security is really connected to observability. So we should have a ways to cooperate between those groups. But at the end, it makes so much sense to group them. I think that's the solution for now. So we can essentially hopefully contribute the content we put in white paper there or maybe lack of it. I think there is some kind of content there to write to their specific white paper. So I think it will be better even for community. But great question. Thank you. And maybe to add to this, we see this more and more and again and again that you have all those different aspects of technology and they have huge overlap. You can easily argue that KAL's testing is just a way to introduce more input, more signals for observability. Which isn't the definition I would choose myself, like initially, but it is a valid and logically consistent definition of what KAL's testing is. And that's where this thing is coming on. This is all related to making the code work. And it's just how you look at it and how you approach it. The commercial or enterprise term is shift left, when you like before you even run the code, after you, sorry, before you write the code, after you write the code. And this is just a shift right thing from that perspective. But it's basically all the same general intention. Any questions? Yeah, great presentation. What would you be looking for in end user involvement in the tag? Basically ask them what they need. The white paper is one of those initiatives. The other thing is I, myself, I have a pretty old talk about like intro to observability. And I've been giving this again and again and again, also in the context of tag observability. Because the thing is the fundamentals don't change, just the audience changes. But several people in this working group on Monday, because again, we had this other meeting for tag observability on Monday, weren't even aware this existed. So, oh, sorry, sorry, sorry, I got it completely the other way around, sorry. Show up, voice your problems, work with others on solving those problems when you have solved your problems. Maybe write a blog post or something like, teach others about how you did it. Case studies are one of those types of talks which are super well received at QPON as well, where end users just walk through how they actually achieve their goals. And again, the thing is the fundamentals don't change. And bluntly speaking, your case study from next year, when you submit it, hopefully, will be, or this year, will be largely the same as the thing which someone else did two years ago. That's not the point. The point is it's new and the audience changes. And the main thing is show up to the cause, get involved. Thank you. Any questions? I have a question because we have, let's say a lot of components in the observability tooling, right? And what I observe, we have a lot of logs and the teams logging many things and with the metrics, we have a lot of exporter exporting everything. And then we just try to react on what crashed our application and try to alert on that. But it's always a bit reactive, yes? And we end up with a lot of data without tracing and so on. So we end up with the big cost of that. So what is the, I don't know, maybe it's naive, but what is the good way to, I don't know, choose what is valuable for us to observe, right? Maybe not from business point of view because it's always a bit related with what the company is doing but from technical point of view. First of all, I would like to say that it's not always reactive. Like you can totally build automation that really reacts to metrics and logs. Like one of this, one of kind of very useful example that you can do right now is to use horizontal auto-scaler or vertical auto-scalers and hook that into Prometheus metrics that will, for example, your application will tell that the CPU is too high and you have to scale it. So that's another, there are many, many more kind of solutions around proactive observability. So that's I wanted to add and you do want to answer this. One aspect about being proactive is as soon as you understand the things which can happen, as soon as you know patterns which may emerge, I strongly believe it's time to move those from observability back into automation. Of course, if you know that you have this and that thing on your one-scaler and like this and that load pattern, if you can detect it, if you can predict it, it's time to automate it the way. Yeah, that's the only addition really. Thank you. All right. Okay, I mean, we can have Erie Barty as well. To say maybe one last thing, maybe instead of asking YouTube questions, maybe we could ask the audience questions do some end user research of who uses metrics today in production. Okay, who uses logs in production? About traces? Okay. Oh yeah, we are there. Maybe then like who believes that their organization understands the value that observability brings to their organization? Okay, who doesn't believe, who believes, who doesn't think that their organization understands the value? Because it's quite, it's expensive. It's not cheap. Who, the companies who work at, which companies do not really believe in the value of observability? Okay. Yeah, that's totally fine. That's why we're asking. That's exactly the questions. I mean, when you look back and this, this is a kind of, I think it comes from IBM and I think it comes from like the 70s or so. Again, none of what we're doing is new. It is not new. It's just like more modern technology, different API interfaces, HTTP replacing TCP, blah, blah, blah, blah, blah, but it is not fundamentally new. And like for literally 50 years in computer science, the running wisdom was that you should invest roughly 15 to 20% of your resources into the monitoring of the actual software. And oh, like if you read the actual literature and the actual research over the years, over the decades, this number didn't really change. So like this is kind of the sweet spot where we as humanity landed on as the overhead or the cost of doing business, which might also be a good way to tell your manager, hey, if we have, I don't know, 20 engineers, maybe four of them should actually work on platform observability, blah, blah, blah, blah, blah. Could you, you said there was like literature about that. Could you post that in the tag observability channel? That'd be very interesting to see. It's been years since I read this. Okay. But it's one of those things which you read and it never leaves you. It's like one of those bam arguments. I guess it depends, you know, there are different companies and some companies are high and some companies are used, you know, maybe rely not on observability signals, maybe rely just on the calls, someone will call and say, oh, it's done. Yeah. It's everything about knowledge. We, like you said, as end users and also as community, we educate our companies, educate our users and situation changes. Of course I saw on the conference observability is very popular topic here, a lot of different solutions. So yeah, it's endless, almost endless. I even didn't visit all those solutions, stands and so on. So it depends. Would you like to ask the audience any more questions before we wrap up? Last call for questions. Again, we don't bite. Or your intentions within observability or something? We still have five minutes, we can use them. It's really because I was reviewing the white paper and I really like it as a resource. So I'm looking for when it's released and like I can refer to it when someone asks me on observability. Now I would like for, now I think I understand better what you wanna do, but again, a short term do have already some other outputs like in the pipeline or there's something that you're already working on that there will be some outputs of. Short term is the white paper. Two of us are going to sit down probably next week or the week after and just finish it. Of course it's one of those things where again, it's like almost done and you just need to. But he broke his foot, so we kind of waited some time, but it doesn't matter. The other thing which we will be doing in the short term is resurfacing older content, which is still valid and still useful. So people see it again or this audience sees it for the first time. Of course, again, the content, like the underlying things don't change, it's the audience which changes. Like this thing from going from logs to metrics and having metrics, if you go back to the internet, 20 years ago they made the same move. If you go to power grids, 70 years ago they made the same move. If you go back to ships where the name logbook and logs is coming from, hundreds of years ago they made the same move. Humanity has done this again and again and again when they needed to get an overview over something, they started writing things down, first as individual events, linking those events and then extracting numbers from them so it's easier to work with them, again and again and again. So the short term thing is resurface old content. It's worth to mention that there is also a nice demo in building, right? So I think we have even one, Henrik who helped with that. So look forward to have like public UI that you can just view and play around some example observability system and if you are passionate, let please come and add more stuff to that, right? There's one really good point here as you play with this. Like for example, you introduce chaos testing to the stack upstream this, send a PR. Like you played with it, you manage to break it reliably like in this case. Send a PR, send this back. Of course, this allows the demo to grow. Also it might help you within your internal company to get more time for your stuff if you can show them, hey, I did this thing and now this is part of the official CNCF tag observability demo kit for observability. Give me more time to work on this kind of thing or send me to the next conference or maybe give a talk about it next year or whatever. One question is the white paper, the PR is under my GitHub account and do you need any help to get the PR merged so you can get more control over the white paper? It's probably just a case of sitting down and doing it. It's not a case of needing the access.