 All right. Hi, everyone. How are you? Good good good good to see a good crowd today and and Super happy that you know, we at least get a decent time slot instead of in the evening, right? But very happy to see all you see all again just Thank you for joining in. We're going to you know, we're We'll just introduce ourselves very quickly. Matt. Did you want to go first? Sure? Hi, my name is Matt and I'm one of the co-chairs for the technical advisory group on observability and I'm Alulita Sharma. I am also a co-chair for the tag work closely with Matt and our Technical lead for the Tag observability also He's around someplace Bartek But he might be he has a tag right. He has a talk right after so you might be preparing for it, but Just to welcome everyone. Thank you for coming in. Please again. I we would like to see this as an interactive discussion Because as you know, the tag is like a community and user group, you know And a user group where all of us together working on different projects different open source projects, you know Kind of talk about observability. What are some of the latest projects that are ongoing in the CNCF space? but also, you know, what we can introduce as Areas that are important to Developers and users alike and and also work closely with the projects to make that Help them make that happen all right, so I think just a sneak peek wanted to call out and Your attention to a super cool logo that we actually Finally adopted this year. It's the owl and Matt has a very interesting story about that later on in the talk But again, just wanted to Call this out because you know our is known for wisdom and observation They're keen observers. So I think it's befitting to to the as the logo of the tag Just wanted to start off with the charter of the tag, you know, the CNCF has several Technical groups which again work very closely with the TOC Which is the technical oversight committee for the CNCF and the charter of the observability tag group is again to foster review and grow the ecosystem of open source projects for observability and That comes in different shapes and forms as you have seen many of you might have seen the landscape graph There are several categories of observability projects Ranging from monitoring to logging to actual cost and cost management and several other categories If you haven't seen it again, go to landscape.cncf.io and it's a really nice You know the chart of all the projects open source projects within CNCF's domain and also an other allied projects that work together and collaborate So going back to the charter again, we work very closely with several of the projects to help grow the ecosystem and Create awareness. We also identify and report gaps in the you know, if there are user requests, so we are seeing and great You know development of an of a cool project or a cool feature set Which is then aligning with an observability project again continuing to grow the project CNCF portfolio share patterns and good practices for observability all of you know, many of us are working in the space where we actually build tools as well as build solutions and pipelines for observing our systems and Be able to be able to share good practices that many of our teams have identified or developed or what works for specific use cases If this is a forum to do so We also work a lot with the education and Really, you know inviting different speakers to be able to share their knowledge and we all learn from that and also providing a vendor neutral venue for Validation discussion of project features and introducing Sometimes different features that then go and land up in different projects And last but not least working closely with the TOC on sandbox incubation and graduation reviews for different projects that are going through the CNCF graduation process and Our role again is very much to provide feedback and be able to provide user feedback Just to help the TOC the TOC is the final You know reviewer and and decision maker here So with that said again, I'd like to kind of move on and just highlight the CNCF structure Many of you may have already seen this slide For those of you who had no, you know, and are deeply involved There's a governing board for the CNCF the technical oversight committee that I referred to and I think Alina's here who is over She used to be on the TOC. She's now on the governing board And there are the end-user, you know, community work groups which are now tags Right and and as you can see there different, you know areas that kind of are focuses for each of these types of groups and And the end result which I think is not called out here What should be is the projects themselves right where we all work on the open source projects and You know form And our members of these different groups All right, so with that said I want to kind of hand it over to Matt to step through some of the 2022 accomplishments of what we did as a tag team this year for observability and Here over to you. Cool. Hi everybody So we thought we would walk through just a few examples Some of the more notable ones of the type of activities that that happen in the technical advisory group and in our meetings And as out as I will either mentioned, you know, we are not as a group an arbiter or a decider or an approver Right. The role is really to facilitate All of these groups, you know, productively engaging together and driving forward initiatives that make sense and Requisite so as an example as you know, as many of you may know in the open telemetry project There have traditionally been, you know, three signals, right? Anybody want to guess, you know logs and metrics traces At Kubcon EU just about half a year ago at the open telemetry meeting You know some members of the community and others raised an idea that you know continuous profiling Should be added as a signal type, you know, as I was at EBPF day a few days ago And if you've been watching the news or if you're a nerd for profiling in general You can there's a huge amount of innovation happening using this quite This quite old piece of tech that's been you know revitalized and used for for these purposes. So As a tag the way we could facilitate moving this forward What was as follows one, you know a lot of discussion, you know So we started this in the late spring and throughout the summer one of the community members Ryan Perry CEO of a company called pyroscope Organized and orchestrated about a dozen meetings for anyone who wanted to come, you know The first meeting had I think 45 or 50 people across a broad Swat of you know end users to vendors to Nerds I am one And we talked about you know, what does this look like right and there are different use cases, you know As a small example, you know some teams or some some projects might want to collect Profiling data at the edge on lower powered devices You know and only collect the data and stream it back, you know to some back ends that will then do analysis Whereas other devices or projects might want to do the exact opposite They might want to be doing aggregation and processing of profiling data at the edge and then you know transmitting quite a lot less So in in figuring out, you know, what should it be? What should the proposal be? You know those kinds of discussions were were facilitated by the tag But really in the background, you know like we're facilitator or a facilitator not a dictator So the the general process was by mid-summer, you know a lot of these ideas started coalescing So we started a document And then by mid-August we had a vision document to say hey Here's what a group of folks that have come together that are interested in this topic, you know We haven't we have reached consensus on What it is we want to do so once we had that we could go to the open telemetry group, right as a group and Open and Otepp and open telemetry enhancement proposal They're modeled after the KEP's the Kubernetes enhancement proposals as a process And so that was put out for community review the real difference and I think where the tag can help is You know anyone can propose an Otepp and and there are many of them and people do As an individual, but you know if you can come together and say hey We 20 or 30 people who you know have various expertise in this area We all agree that this is sort of a great path forward and a vision so that that gives more weight to the proposal Right because now open telemetry says well, they're not proposing a specific implementation You know, this is just a vision document that says like here's what we want to accomplish And there's lots of ways we can do it Maybe we can do it multiple ways at the same time for different folks But this is what we're going to engage in and so just recently I believe in September that was merged Right and so now as part of open telemetry. There is an actual initiative to go add continuous profiling data as a signal Type to make a fourth signal if you will and So at the open telemetry meeting yesterday where you know, they're typically a coup cons This was a topic and the next topic was okay Well, what happens next and the community comes together and agrees that okay We're going to go do some deep work on this and actually start implementing some things and we don't all decide You know how it's going to happen but we decide that it is something that people are engaging in and and It leaves plenty of room for folks to come in and engage in the process Right, so that's really the point so that's one of the things that we did over the course of this year or that we kind of softly allowed to happen or Facilitated in some ways, but really there's a community that did it But the tag was a place where they could come in that neutral setting and have those discussions in the first place, right that were Neutral so yeah, and just to add to that, you know again often there are many areas of you know Observability which intersect with other disciplines or you know related areas security is one of them You know networking is another one of them operating systems and container, you know storage is another area so You know that and then we'll dig into this a little bit more later in the in the slides where cross-collaboration across tags in terms of discussions and you know really vetting out the use cases for a particular proposal happens a fair bit and and it really is you know the space where you can kind of think through different ideas and you know which those use what use cases they are and You know what makes sense to introduce as a particular feature then that you can take up to a particular project and And submit a proposal so this is a good use case of you know what like the profiling Which is a cool thing that's going on all over many projects at at CNCF. You see these different features kind of landing in Back as feature proposals into the project which happens to be a hotel hotel in this case The second initiative. I think that we wanted to highlight on this cortex So did you want to Just brief show of hands non-binding who's familiar with cortex the project just so I know okay I would say that looks like maybe 20% of the room. So you fifth of the room bear with me Cortex is in short a durable Back-end durable storage for time series metrics some of the basic at a high level. It's multi-tenant It's designed to be running cloud environments on top of you know s3 or Google Cloud storage or min.io or some sort of object store It's designed to be horizontally scalable, right? So, you know if we need to handle more loads and and most interestingly I think as well the read and the right path so the right path being you know ingesting metrics from all of the various sources Being sent to it and the read path being querying data out the read and the right path are Independently horizontally scalable right so you can actually it's quite a flexible system that was designed to be cloud native from the get-go All Projects kind of evolve over time right and and and they have new new new ideas and new folks And then some folks you know rotate out So what one of the things that happened with the cortex project is some of the original the authors and maintainers Cycled out of the project to do other things right so the TOC and others were saying hey Well, this is a you know a lot of people are running this in production today No cortex is an incubating project. So it's in that that middle zone where it's seriously being run at serious scale So it's important that this this project remain healthy And so what could we as a you know really again? We don't have any power really nobody works for us We're not really in charge of anything, but what can we do to help? Make this you know a successful project as it's in our domain to be interested in and so we you know talked about What could be done? It's a talk talking with the project maintainers to say hey What what would help the project you know because and of course is contribution So if you're looking for a place to get down and nerdy with like TRPC time series metrics Some really fantastic stuff at scale. I don't know if anybody remembers and if anybody remembers when Docker Last year or two years ago. They started rate limiting Authenticated unauthenticated Docker pulls right when that happened our infrastructure where I was working when Kind of face planted a little bit. Well, I didn't face it more like paused, but you know We ran H.A. Prometheus pairs and all of our clusters, right? And so the right-of-head log started to really fill up because you know the durable storage is down And we over provisioned things so we had like a couple weeks of right-of-head log space really Discs are cheap but when everything got turned back on suddenly our whole infrastructure needed to send the Cortex stuff and again because we could Dynamically you know in a pretty easy way scale up and down stuff We could stick scale stuff up and start you know bringing back all of the back metrics and at one point I think it was like two in the morning and we hit like 20 something thousand art requests per second requests per second Rather request per second So and then you know it just handled it just great So it's a really fun project if you're into distributed systems and time series that they could use your help in a variety of roles, so You know Bringing visibility to that and really kind of helping communicate out to other projects to the community and to the leadership of the CNCF Like this is what's up with the project, you know, and we're going to help them We can't do it for them, but we're going to help share in their success right or help facilitate their success So it's sort of another type of activity that we do in the group I think we have a third one to talk about as well We thought it would be fun to have an observability speaker series Right, so you know if we've got people that show up because they want to nerd out and talk about Observability what might they like to hear who might they like to hear it from so we've had to so far and It's almost done processing on YouTube So by the time you actually maybe see these slides, there'll be a link there might appear But in September we had Liz Liz Fung Jones from honeycone come and do a talk on You know hybridized signal types and really a lot about how eventing and events really is a really cool talk. You should check it out Yuri from Who as many of you know had some small part, you know to do with with the Jaeger project? He came and talked about some of his thoughts around Observability and tracing and all of that and I've put a link there is a little paper that they've done a positional paper on Schema first application telemetry So the title is sort of self-describing. It's a fascinating read and I think you'll be talking about it sometime this week or next So and then we've got Jonah Cowell coming Who's CEO of logs.io and has been involved in various communities in observability He'll be coming to share some thoughts and I don't think we have a title yet, but we will soon So if you're interested in speaking to the observability community if you know someone or have a suggestion Reach out on slack The process is basically pretty lightweight so and I think just to add to that again the Tag sessions as many of you know it are published on the Google calendar for CNCF and and also They're typically on the first and third Tuesdays of the month so You know again at least joining in once a month is a you know We'd really love for more folks to join in and have those discussions that then you know carry over into the projects themselves and other strategic initiatives and features that you want to see happen, right? So I invite you again to come and speak or you know come and share your thoughts there So we want to talk next about 2023 and what's next again, you know, this is a great time to kind of discuss that because as we go into the next year typically, you know, there are a lot of air Lot of projects that come in on it, you know interrupt driven But but then there's some of that we do intentionally and try to plan for so Again, you know, if there are areas that you'd like to see happen we this is what we have kind of collected from the different discussions and Suggestions that folks have made as they participated in the tag or You know, there's different feedback that's coming in from different projects And and here's some of the areas that we want to kind of You know support and make happen and drive forward in the next in the coming year and Again, if this is just, you know work in progress if there are other topics You'd like to cover or see more, you know work done. I'm happy to you know, please please share The first area of course is Matt, you know Discussed earlier is profiling and as you can see that the profiling enhancement proposal for open telemetry landed You know earlier this month But we would like to see more, you know detailed use cases outlined there and Always, you know as all of us discuss these use cases kind of taking that feedback Back to the project hotel as well as others, right working closely with other projects and other projects that are actually working on ebpf and getting a Morche of a shared vision if you will towards that the Second area which has actually been super interesting in observability, you know as all of us kind of work on complex workflows Building in different observability solutions is wraps, right? Like how do you envision? dependencies when you are actually deploying You know server-side infrastructure or public cloud infrastructure Or even client infrastructure, right where there are several interdependencies in the Systems that we are, you know observing right and graphs is a great way to envision that right those interdependencies whether those are Learning and understanding different systems and the correlations between them Or even understanding the data that flows across those and that is going to kind of go into a bit more detail on You know what? We are looking at in that space another area that Came up during discussions, and this was something that even Yuri highlighted in his talk Earlier this month was to look at telemetry data You know in a in a more diverse way, right? That is it's not always about metrics logs and traces as we always hear about right that different kinds of Telemetry data to tell us about, you know, how are our systems behaving, you know, and Different types of data gives us that, you know insight into understanding the pulse and health of the system in any given Finding time and systems can be General it's a general, you know definition for it could be an application It could be a component within an application or it could be infrastructure itself whether that's an router or and you know Something that such as Kubernetes that you're running as an you know containerized workload. So Again looking at exceptions as another data type is something that has Been in in discussion now through this year and it's an interesting idea because you know all of us work on applications that run on infrastructure cloud infrastructure and At the end of the day, we'd like to understand the behavior of systems end-to-end what is what are Not only our applications doing but how is the system interacting with them and what you know Can we actually deliver the quality of service at the end of the day, which is what observability is all about? So looking at exceptions and more detail is something that's interesting because we want to know about the failures ahead of time in our applications and later another area which has been, you know, very much discussed in the in specific, you know as many of us were run infrastructure on cloud native Public cloud or you know, otherwise, how many of you use cloud native infrastructure? I Hope everyone But if you don't you're probably using something on prem, but it is a hybrid world today, right and Understanding as more and more of us run applications on public cloud infrastructure. We really need to understand What the usage is right and and how much? Does your organization really spend in Different categories right whether that's engineering whether that's actual, you know builds to an Actual vendor or whether that is, you know again planning for what kind of growth you need to have for infrastructure in the next year so cost measurement and understanding the actual usage of systems And system components is also something that Observability use cases are starting to look at and there are several projects within the CNCF Observability umbrella that actually address this a couple of them are open cost as well as cube cost Which many of you may have heard but having more integration across, you know Providing that data through a standard observability pipelines is also another area that you know is something That people who care about it are looking at How many of you care about that? Cost you do right So this should be an interesting one and again, I invite you to come over to the CNCF to the tag meetings So that you know, we can all discuss and actually capture some of these use cases because I'd like to better define a Baseline for what those metrics are what those traces are what do we need to collect from an observability standpoint that we can Standardize and tag, you know collect in a standardized way, which is then built into the collection mechanisms You know that you use to be able to pull from infrastructure, right? So that's another area that I think as actually Has a lot of lot of areas we need to can kind of deep dive into Another area I'd like to kind of call out is correlation across telemetry data signals And this is again an area where sampling for example is a you know specific use case as many of us Instrument our applications and enable tracing for those applications What does correlation mean here because how many of you use sampling today? Many of you right. There's no way that you could handle the kind of cardinality and Throughput that you get from a hundred percent of tracing turned on or profiling turned on or metrics turned out And even for that matter our logs, right? so again in one way or the other we do do a fair bit of sampling and Correlation then understanding, you know, how our data is Actually providing the right signal from an observability standpoint is important So these are some of the initiatives that you know, we have kind of collected feedback on and suggestions But please again if you see there's something missing Let's talk about it Other groups and activities. I just wanted to call out before we switch over to Matt is we actually seeded and observe Okay, Kate's work group in within the tag where it is actually looking at Kubernetes observability specifically and there's some collaboration that we have done with the instrumentation group was saved in the Kubernetes group, but also at the same time Kind of digging in into some of the demos and implement use cases of you know, what does that mean? So again, anyone who's interested in getting involved in that please, you know, please join in Secondly, we're looking for tech leads if you want to lead a specific area, you know You know, we do a fair bit of analysis tech and as well as technical white papers So if you are interested in working in any of those areas, please join in And it is very simple. It's like, you know, we really invite everybody to kind of You know use this as your group use a group and then again also contribute to the tag We'd like to see some reference tools recommendations And there is a GitHub repo for the CNCF The tag observability under the CNCF domain which you can you know come and take a look at there several issues that you Could work on and you can also ask for requests, you know, they are very easily and that said Again, I want to turn it over to you Matt. So you wanted to kind of dig into these Okay, so These are some challenging questions that are kind of hard to answer, right? I don't have time to go through all of them, but you know You know, some of them are a little cheeky, but real right, you know, our GitHub stars and you know popularity and market cap Actually correlated a more interesting one might be, you know, I'm a Say you're an IT decision maker like who's had to pick what open source projects to work with, right? Let's say you wanted to answer the question Well before I go dive in with my teams or by myself and put this project into, you know Something that is directly revenue impactful Who all contributed to it and who did they work for and many of those companies are probably very small So who funded those companies and what else did they fund show me clusters, right? That will make most relational database Systems just kind of weep It's more like a social networking question that you really need graphs to help Reason on or you know, I'm running even not very big Not even that much of a cluster in terms of load, but there's still thousands of pods say right What CVEs came out this morning and which of my workloads are they in and I'm doing say canaries or AB testing Type thing and I probably have a bunch of different versions of a bunch of different things running Right and the transit of dependencies alone get you to the scale of you know We kind of need a graph to efficiently answer questions like that. There are more but in the interest of time I want to fast forward a little bit to the next slide so Over the summer we've kind of launched a project it's nascent And we've done a good deal of work around project planning So there's you know a whole bunch of issues that are marked as help wanted or good first issue And there's a lot of room, you know to go innovate but in a nutshell It's a project that is making a labeled property graph. So think Neo4j D graph Redis graph, you know any type of labeled property graph that speaks open cypher, which is a Sort of like sequel for graphs if you will and it's a federated Supergraph made up of a bunch of sub graph modules so that people can work on things independently And the sub graph modules are things like CNCF projects and get all of it and Twitter and LinkedIn and CVEs and all packages Etc right so in order to kind of work out the individual data models for these sub graph pieces That's a job by itself, but having some way to coordinate all this Is imperative so we're using graph QL as an interface definition language And to get some federation support so we can you know mix and match these different sub graphs So you know we can have some things that are using public data sets But maybe you want to make a hybrid where you're correlating things that are out in the open with things that are behind your firewalls Or that are sensitive right so we wanted to have a project that was sort of Interactive and and mix and match and so some of these collaborations that we could I could briefly highlight Are other examples of the opportunity that you all have and your friends to To engage in the tag because sort of from that position you can launch off and do all kinds of stuff So in short what tag is observability tag s tag is security tag right so the packaging format Metadata for you know MPM and crates like does anyone know quick show it like just shout it out how many How many MPM packages exist someone know? Seriously, just just shout out. We've got like a minute here No one okay. Well on github. There's a little under 1.4 million in the public in public No, I'm sorry not on github on MPM on get a proper 2.1 4 million when I when I looked last We're approaching half a million maven packages This time last year there were 40,000 rust crates now There's like 80 something thousand of them and on and on and on right these are big big possibilities for what could be running on my Work in my workload because something pulled in something pulled in something pulled in something that pulled this in right and I might not even know Because depend about didn't tell me that my package that Jason pulled it in but something pulled in something pulled in something And I have a bug or risk So Collaborating with domain experts in in tag security to define what those graph data models are is an example to kind of cross-tag Collaboration where we can leverage domain experts from different areas Another is the business value subcommittee. You might not know them by their by their name that name But it's glossary.cncf.io right that's a bunch of definitions They're doing other stuff like providing updated views of the CNCF landscape that are augmented with category information Right, so what projects are similar to this or I want to use this GPL thing What's similar to it, right? That's the kind of recommender systems that we can build with graph tech or graph data science and Sort of the mother of a rolodex is really You know, you know, sometimes there are so many working groups and so many tags In the CNCF alone, you know, they just keeping track of who's who and who's been elected to what is a challenge sometimes But when you zoom out a little bit to the to the entire Linux foundation set of landscapes You know if the CNCF craft if the CNCF ecosystem in whole is a graph You can almost think of all the landscapes and aggregate as a hypergraph or you know where nodes exist in multiple graphs The point being those other organizations have their own types of user groups their own tags But they're called something else, right? So we definitely don't cover it, but you know if you build ontologies, right the kind of an ontology in short is What are the kinds of things that are in a graph and how do they relate to each other? You know as classes of things well now we can start to identify similar groups that might be called differently structured differently But are looking at the same topics for example all the Kubernetes things or any of the other dozen Landscapes at the Linux Foundation managers that are not the CNCF but have our colleagues in them, right? So I'm being told we need to go There's some other work that other examples of you know collaborating around secure use cases and I'll just the last one kind of I don't need to go into but I've made implications to it or inferences to it supply chain security Just understanding what's running. You know much of the work and observability I'll close by saying has been all around observing workloads, but we need to also think systemically about what's running in general and Owls why owls? It's not the overwatch fleet although. That's an excellent guess for me particular Owls fly softly. They have little they have little feathers on their leading wings So, you know Heisenberg there's a tie in there, you know, we want to observe things without making too much noise while doing it But I can assure you Matt's gonna do a different presentation. Yeah, this is a Penocular vision it's cool Their ears are offset like this so sound hits their skulls just a little bit differently So they can directionally tell where things are coming from again sort of an observational thing And they have grippy feet and talents, you know If you I'm guessing most of the people in this room have had the experience of digging into a bug or something That's not quite working right and you just don't let go once you full pull thread, right owls are that way when they're Getting what they want so That's about it. You can find us here. We meet the first and third Tuesdays of every month. Yes Hope to see you at our next meeting. All are welcome. Thanks And You know where to find us so please give us your Yeah, yeah, I'm sorry. I thought we're being kicked out. It's weird slightly over time We learn all about observability and owls if you have one question. We have time for one question Okay, you ask one tough question Stump the chance. Yes Hello, there are a lot of projects in the CNCF tied to the ingestion of data telemetry data. Yes, and comparatively fewer on the consumption The end user side. I'm kind of curious as you look through the gaps in your portfolio Is that a priority for your group or no, I think I think that's a great suggestion because you're right that you know there is and there is a Kind of an imbalance there, right and and definitely Let's talk about it because I think that there are several storage solutions as well as Analysis comma, you know plus storage solutions because I think you know when you're looking at different types of data Is there a single storage solution today and there probably isn't right? So again But I'd like to welcome you to kind of maybe join in for one of the tag sessions and we can chat another brief observation to augment that I Should is you know open telemetry is really you know open census and open tracing merged a little while ago And now we have open telemetry really focusing on number of things but primarily the collection of telemetry Right and that's all fairly recent and so like as we as we kind of shift our focus if you will as a community now That we've kind of got you know language independent ways to collect data in a non crazy way And then with the you know the collector or other mechanisms, you know handle it and get it to a place We can do that analysis Maybe like some of those bright minds and coffee induced nights You know we'll be focused on on exactly what you say like now that we have all this how do we juggle it? How do we analysis? How do we analyze it? How do we slice it dice it all of that? So I think it is an astute observation though that there has been a disproportionate You know view on the collection side, but I would think that's going to change as a observer right, but thank you again everyone and See you at the tag meetings We're around here too