 And we're live with Spotlight. This is Dan Papandrea. We have a special show today. We're talking Thanos. It's highly available, Prometheus set up. It's an incubator project from the CNCF, long-term storage capabilities with Bartek Biotka. Okay, and Spotlight starts now. All right, Ibarthek, welcome to Spotlight Live. How are you doing? Thank you for having me. Yeah, I'm fine. Thank you. It was a long day. So before we get started, I'm gonna do some housekeeping. Is that okay? You know? Sure. Awesome. So we are, this is the CNCF disclaimer. This is an official live stream of the CNCF and it's such a subject to the CNCF Code of Conduct. Please do not do anything in the chat or questions that would be in violation of that Code of Conduct. Please be respectful of one another. Another subject that's very close to our heart, Spotlight, is KubeCon. KubeCon North America. We're gonna talk about that in a little bit, folks. Registration for KubeCon and CloudNativeCon North America 2021 is now open for in-person and virtual, explore all the registration options. Go ahead and, you know, there's a link. You'll probably put the link up, hold down one second and boom, there you go. I'll leave that up for a little bit. So, Bartik, we're talking to Thanos today. I wanna ask you a serious, serious question. First question, and I know, look, I put your face on the guy. Is there inspiration from Thanos or it's just you all just went back to like Greek times, like Greek gods times? Like, I wanna understand this whole Thanos name. Where'd it come from? Nice, nice, good question. And I think there was a time in the very beginning of this project where we were afraid to tell the truth. I talked to my manager. We were trying to make some Marvel jokes, right? And my manager was that, from legal perspective, do not poke the beer. Do not poke the beer. All right, okay, we had like amazing jokes and whatever. So, at the end, we didn't joke too much. We were silent, but actually, when we proposed, donated the project to the CNCF, they made the proper for copyright, trademark checks, and apparently there is no trademark on the Thanos. So, we can joke around, just fine. And so, I can tell you the truth and the truth is that our, in my past job at the startup, it was called Improbable, amazing startup around games and distributed systems. And we were part of the infrastructure team. We were looking for some monitoring metric kind of solution that will solve our needs. And we were... Can we talk about Improbable real quick? I'm sorry to interrupt you, but let me ask this question. But maybe, yeah, maybe let me finish the naming and then I can talk about it. All right, all right, go ahead. At the end, I want to just quickly go into the topic why we even named Thanos. And the reason is that we were having like Kubernetes microservices and each of those services used to have some names, right? We have like a Scarlett, Ironman, Aquaman, and like Tor and like all of, and those we are trying to find a meaning, right? So, every time you create a new microservice, you were responsible to find a correct name, right? And we were kind of playing around different names and that's why it came with Thanos. And I think it, to be honest, it might find, it might be kind of feel silly, but actually I think it contributed to some popularity, right? So, let me ask about Improbable though. What was the major game that Improbable? Was there a game that was in the studio that like, what was the major game from Improbable? Sure, so now we used to have like a minor, I mean, many contracts with like smaller indie games and they were actually pretty playable. And the kind of idea behind the whole game studio, or like it's not a game studio, by the way, Improbable is like just a platform that allows to game studio to actually develop games easier, right? And actually it is still ongoing and the newest game, the Scavenger, you can kind of Google it around, it's actually, I think it's a beta so you can actually either buy or even try it. So the whole general idea is that you make a game in a very easy way, but actually you distribute this, you kind of write a game like it will be on one computer for one user and actually upload to our cloud and it will just distribute into many machines and run a persistent game with like capabilities of like scaling to millions players. So we used to have a game called the World Adrift, which was actually a pretty fine game where there was like ships and you could like build those ships and they were flying around the islands. At the end, if the ship was broken, the one piece of the ship that you could just harvest fell down in some island, in one year, you will play this game, one year of real lifetime, you will go to this place, if no one touched this part, it will be still there. That's what persistence really means, right? So the idea was cool, but very ambitious. Let me ask you something like, now you asked me like, I have two questions at this point. Like, are you a comics fan like, or was that forced upon you? Like, were you like, oh, Phaedos, I'm pretty, that's awesome. Scarlet Witch, amazing. Or were you like, oh, Dorky, I'm not into it. Yeah, you kind of were, you had to like it, I guess, where you were with us. Yeah, no, I like, I don't like, I mean, I would love to have time for comics, but I really like the movies and like this kind of, also Star Wars, also other stuff. So I guess typical, typical person. But yeah, you have to have some passion towards that, right? Got it. So let me, let's ask this. Let me, for the uninitiated, somebody in this community right now or an end user that's looking at us like, okay, what is Thanos? What does it do? What is it? Sure, so Thanos is a monitoring system. At the end, what it does, it allows you to collect metrics, which is a certain observability signal, among others. So one way of observing your software, how, what is the state of your software? What is the healthness? Maybe you want to trigger some notification if something goes wrong and how to define those things. All of this is part of the monitoring system. So we allow it to kind of collect those metrics and store them. So it's at the end, it's a distributed database, you can call it. And, you know, obviously as a database, you can access this data and you can, you know, run some, I don't know, alerting crews, recording crews and really store it efficiently. So the goal of Thanos was to really do what Prometheus is doing, but just on scale and then cheap and then continue the same amazing functionality through like a very stable promql language. It's like a language of querying those data, which is really, really loved by many, many thinking, many people in the community. So what we did in the, you know, in the essence, what we did with Thanos, we kind of took the Prometheus, we split into pieces and put into microservices, right? And we have similar functionality just scaled and this is what makes it powerful, right? And also highly available and so on. One additional thing that were, we added on top of, let's say this setup is, you know, we had the goal to make sure this is cheap long-term. So, you know, at the end, when you collect lots of metrics, you are having lots of data, right? So you need to have like a cheap storage and we thought that object storage, which will, which is available on every possible cloud provider and it's super cheap to store data there. We believe that, you know, it is actually amazing, you know, place where you can store all of this data, especially because you don't really use all of metrics, to be honest. Like you want to cover them all but that actually you use them, you know, rarely. So for those cases, it is actually very efficient to use and very cheap, right? I've learned a couple of things I was looking at the website and I know of you all because of, you know, what I work on, right? And so in terms of like unlimited retention, that's what we were talking about, the cheap aspect that you can throw this in that S3 bucket, the global query thing from PromQL was genius, all right? It's the way that you all interact so directly with PromQL, it's phenomenal because I can literally write one PromQL query and query all of these things versus one disparate exporter and all of these different nodes, which to me is pretty goddamn awesome. Like that to me is the big draw of this. And so like talk to me about like, again, is what was the problem, the original problem you were trying to solve with, honestly, like, it's not like, hey, you know, you just want to make this distributed thing. It's like, oh, what was the problem you wanted to solve here? That's a very good question, right? Like especially just doing something distributed for fun is the biggest anti-pattern ever. You should avoid doing this. Like it's a complexity. It is actually complex. Any distributed system can have complex situations, right? And to be honest, yeah, Global View was the main goal here. And I think the fortunate situation or like unfortunate that triggered this project was that it improbable to accommodate those different game studios and maybe players on different continents and different locations. We were running like 30 Kubernetes clusters in the very beginning of Kubernetes, to be honest. Like we were running Kubernetes ourselves and that was a problem at the end for a metric system because like from materials that was supposed to be localized is that we suddenly have 30 clusters. So you need to have a global view, bird eye view, whatever you call it, to gather all those information. And we really wanted to aggregate so some applications, some game could flow between clusters and get migrated. And all of this is part of the same application. So I would love to have informations and aggregations, no matter where it is, all right? So, and to be honest, it was like four, five years ago almost, not almost like literally five years ago when I joined improbable and we had those problems. This is, this was like very beginning of, you know, the whole world starting to noticing those problems, right? Because when I'm now kind of architect in Red Hat for observability things, then the main problem right now are not only observability problems, but anything else. Hey, we suddenly have multiple clusters. We treat clusters as a cattle, not as a pet, which means that we kind of create them and then destroy them in a seconds, right? So this was never a problem before. And now everyone is moving in this direction. So those federated solutions are must have. And this is why, you know, Thanos were born in a, burned in a, or like, born in a correct moment, right? And because we maybe were having those issues prematurely than other companies, right? That's incredible. And again, it's like you were definitely the early adoption aspect and then being able to like take that and then, you know, something you gave to the masses. Let's talk about that for a second. When did you decide, hey, you know what? Let's take this thing and open improbable, like you all open sourced it pretty much. Was that an early decision? Were you like, let's do this. Like, who helped with that? Oh yeah, that's extremely good question because having an open source project and like doing open source things for, especially a startup, a very focused startup with minimal amount of engineers has, it's like a huge investment. And to be honest, it would not happen if not somehow primitive community, let's say, because how the project started is that we have this problem of like, hey, we need federated monitoring system. And we want to have long-term storage and cheap operational cost and high availability. And we already used Prometheus. So that's great system that we just want to build upon. And we tried different things. And at the end, it was Fabian Reynard, like amazing engineer now working at Google. He had this idea that maybe we can use object storage and we can use exactly the same format as Prometheus. I don't know how it looks like, but I feel this will work. Who wants to pay for this work? And improv was like, yeah, let's meet. We kind of met in the KubeCon at some point. 2016 or 17. And yeah, we just started collaboration. So he kind of stopped his work at CoreS and for three months got contracted by Improbable. And I was kind of the engineer working on observability. So I kind of joined this effort. And his main kind of, because he was part of Prometheus team, his main criteria for even joining this collaboration was, hey, this has to be open source from scratch. So to be honest, Improbable was not even thinking twice because it was very, I mean, well, maybe they were thinking about this. But what I mean is that it was not like a huge conversation because there was a, you know, someone willing to do it, someone capable of doing and some good idea. And his requirements was to have it open source and there was not strong feelings to make it known. But like that's why it was, I'm super grateful that Improbable accepted that offer for and kind of we collaborated, we designed everything. And in three months, we shipped this to be honest, like in a obviously like basic form. But the truth is like he left like this contractor thing went to Google and for legal reasons, he couldn't even contribute again to Thanos. So I was kind of continuing this work. And really this is where the true investment from Improbable actually, you know, appear, right? Because this is the moment where you need to ongoing, do ongoing investment on community, getting more people involved and just paying this engineer to kind of look on those, you know, contributions and really open source factors. And this is where Improbable were kind of doing some of it and I mean, super grateful. And this is why this is super successful. But at the end, I had to sacrifice a lot of free time too. You know, that's the, nobody talks a lot about that. Like in terms of like open source, like, you know, like it ends up, there's the superstars in the core maintainers of certain projects. You know, I work on Falco, right? And so like there's, there's parts of this where it's literally like, you know, there's core maintainers like Lorenzo and Leo and folks on my side that are just, you know, killing and doing so much work and stuff like that. But then there's, we have to kind of take that step back and let others kind of come in and let them contribute. Talk to me a little bit about that. Like where you were like, look, I've done all this thing and now I'm like nurturing other talent to bring, take the mantle and take finals forward. Yeah, that's a very good point. It's all about- I'm nailing it today, Bartek. I'm nailing it. We're doing it together. Let's do it. All the points are good. Yeah. Let me think about that. So I think, you know, first of all it was definitely very important to start like building community. So I know I spent many, many, I know hours to just talk to different people and collaborate and really establish some kind of process that will just gather more people, gather more community because it looks like that the project got some, you know, movement, people started to use it, adopted but you know, it's hard to motivate anyone from those companies to actually even mention they are using it. That was really a blocker for some people. So I knew that internally, you know, you know huge projects are, huge companies are using TANOS but I couldn't even speak about that because legally, whatever. So at the end it was very hard work to and like maybe, you know, a little bit of strategy, you know, tactical thinking and maybe some discussions around how to get those people, those companies to contribute or even share that they are using or share some feedback. That was the first step. The second step was to actually have and start really a secure environment for people to speak up. And I was really, again, I was like, you know true open source developer, asynchronous work, no Slack no chat, like this is like not efficient. I don't want to do it. And at some point I was like, okay, whatever let's create the Slack channel. And actually that was actually the stepping that the trigger for actual contributor because there are so many people who are afraid to ask maybe silly question in GitHub issue because for them, GitHub issue is like, wow, a huge proposal. No, it's not for us. It would be fine if they would just ask this question but they were just looking for this easy, you know small step where which was just Slack and just, hey, just say something. And then with this momentum people started to help each other and this was a trigger for actual community. And I cannot say this, you know, enough just start this damn chat. Like you have to have it. Like there's no, you might not like it. You might like IRC who by the way no one use IRC this time around. So you need to have a Slack, right? There's no other way. And or whatever is the most popular kind of chatting platform but in the end that was a major goal and of course a major service step. And of course, you know, trying to get maintainers. So having contributors is one thing but then how to get that step. And there was this balance between, hey, I want to have some bar of the maintainer that he has to know about, you know, this stuff. They'd have to be friendly. There was so much. It's like an, it's like a hard interview to pass except that you don't have money after that. So it's really challenging but you just need to show value. I think mentoring help as well. We have lots of maintainers based on the mentorship we do and literally all of our mentees have like a nice job. So I'm super happy with that, with the effect of it. So, you know, mentorships, community and kind of having a safe space. So people can see the value. Like they see, oh, I will now maintain. Oh, now many companies want to hire me because I have this skill. And that was the trigger of, you know, oh, this is actually valuable to be like a maintainer. And sometimes we, what I recommend is that, you know it's rarely the case that the person is familiar with the whole code base, try to figure out like one portion. Okay, you are a maintainer for this particular part. So at least you help on this position like front end, for example. And you don't need to know about like everything else but you at least meet like those criteria. And we have the occasion when, you know we joined, well, we get someone as a maintainer and he immediately got the job, right? Like in the SRE space because, you know he knew he was part of the Thanos. So someone told me once because there was some interview of someone and then in some company and he didn't have a lot of even questions. He just said, I'm Thanos contributor or something and or like maintainer and someone said from this interview or this company said we don't need to check you that much. You maintain Thanos. We know you cannot be a moron. That's a good actually value as a maintainer, I guess. So this is those are some motivation points where you can bring, right? If there's projects and again, so, you know Thanos is an incubated project the Falco is an incubated project. There's projects, anybody's watching this I think exactly what he said it's like, you know having that being able to ask those questions showing gratitude, I think is the biggest thing is like, Hey, thank you for doing that. Thank you for putting this in here. We send, we have a contributor of the month is what we do. And basically we send some swag out to them and we say, thank you so much. Because, you know, people are taking time out of their day. This isn't their day job. Their day job isn't maintaining Thanos or Falco or Cube. It's not out of that. And that's the part I think that, you know any burgeoning projects in the sandbox you're watching this, I'm sure, you know but that could agree with me show gratitude to the people that are taking time out of their day. Totally, I want to add to this point because we had for some time and it was like a contributor idea to every Friday, just do a shout out and like shout out on Twitter and on Slack and literally traverse through all the weeks Slack and get all the people who contributed and like name it, name them and just say, thank you. And it works so well. We actually have an issue even open to create a bot for this but probably bot will be not like rewarding but still I think there is enormous help that people do and it allowed me to, for example focus on delegating and teaching and all of the stuff and I don't need to be constantly on the main Thanos user channel because people are helping each other. So it's enormous, right? And it was helped. I'm writing this down. Okay, sure. Let's create a bot together. Again, both will be nice still. Some kind of kudos bot, whatever, right? Yeah, man. I like it. Carlos, Pannado, if you're watching this, get that bot ready. Come on, got some work for you, my man. All right, so I'm going to ask you this because, you know, again, I'm in this space. You know, I work for Cystic. You know, we have like a back in store for metrics and also like, you know obviously run time security stuff with Falka and Cystic secure but there's one thing is like I've seen there's a lot of other projects that have very similar kind of, you know traits, right? It's kind of a store for metrics, right? We think of like Cortex or M3. What do you think like, but I always hear Thanos is kind of, this is the thing. This is the key kind of flagship store. So my question to you is, what difference? What makes, what's the difference? What makes you stand out besides those other tools? Yeah, that's a, you know, tough space to be in, right? Like there is so much demand that, by the way there is a place for everyone. Like if every of those projects will have like a, I don't know, like found a company behind that like we would still have customers, enough customers, right? So that's a good thing. Like there are still a place for this competition. It's not like it's a very niche. However, you know, there are still differences and first of all, yeah, there is demands. There are different people having different ideas about how the backend should work, right? So, and I think what is very unique in Thanos is that we really grown the solution from Prometheus project itself. So we use, you know, native protocols. We stick to the native APIs, but even on storage, everything is the same. So you, what we do literally in Thanos is like, we combine everything into those TSDB, whatever time series database of Prometheus blocks, which are directory of files, and we literally kind of put it in object storage. So if you would download the thing and run Prometheus on top of it, it would just work, right? It's exactly the same. And this is super powerful because we could leverage the same tooling, the same knowledge that Prometheus spent six years or seven to teach people all about, right? So PromQL is one of the most, you know, the most powerful and flexible, you know, metric querying language, but it's also, it has a steep curve of learning, like as every, you know, sequel or whatever other languages, right? So since people already adopted that, it was much easier to use this one. So that's why all those decisions were optimized for this, you know, integrity. And also it's worth mentioning that, you know, we treat this as a family. So for example, when our students or our, you know, maintainers on contributor, especially mentees, like they want to, you know, they have a project and task to do a feature to do on Thanos. We say, hey, contribute this to Prometheus first. Maybe we can share that, we can reuse that with Prometheus code, there's community, there's other people who can take a look on that. And if you contribute during this time into Prometheus, we count this as a contribution to Thanos because we're literally importing the same code. So this reusability was very important for us because we don't, you know, program the promo code engine from scratch, like a compaction from scratch, storage format, reading and writing. Those are very complex pieces of code, like literally unreadable sometimes because it's so optimized, it's memory mapping, it's so, so much stuff. And we directly import this code, right? And this is very unique, right? Because there are so many companies that M3DB, like Amazing System came from Uber. So, you know, Uber has to create something like, I don't know, like five years ago from scratch because there was nothing there. So no one blamed them to have like a solid system they want to share an open source. So it's great. And actually they are very, you know, happy to collaborate and they are adding integration points. So they are supporting Prometheus, they are supporting a write protocol so we can send data from Prometheus to M3DB, not only to Thanos. And the same with Cortex. Cortex is special because it kind of grown from the Prometheus maintainers, just different Prometheus maintainers and it was actually created one year before Thanos. And, you know, we would love to use it and we tried in Tim Provol to use it and it didn't work. But we are, I think it's okay to say that because, you know, we were very beginning of this project as well. And we used to have different paradigms as well, different thinking. We said, hey, we want object storage support. We want to have the same storage support. We didn't want to have push replication. There's some technical things that we want to try out. And in practice, what we see is that our teams collaborate very much. Like we have, you know, Cortex maintainer, Marco as Thanos maintainer, for example. I was contributing to Cortex many times. I know it's codebase because they actually import Thanos code and we import their code. And like we, this is great collaboration but end up with like massive goal. Let me ask this. I'm going to ask this. I want to ask you a question. Sorry, I'm going to interrupt you. Why have multiple projects? If you know what I mean? Like it's because, and I understand this and, you know, this is a controversial thing. I'm going to ask like, but like why have a Cortex and why have a Prometheus, why haven't M3, or excuse me, a Thanos when you can kind of integrate these if everybody's working on them. Because then it almost confuses the end user. But is it because of, like you said, it's the context each one is providing. Each one has a different thing that it's doing to a certain degree. At the end, they are doing similar thing. There are minor feature differences but at the end, especially Cortex and Thanos we really maintain the same APIs and stuff. And nowadays we have the same storage with some differences. And so this is a very good question but I will ask you, you know would you like to, you know your focus project like Cortex or Thanos and would you like to suddenly adopt, you know 100,000 lines of TechDepth, right? And for Cortex team, it will be TechDepth to have Thanos kind of components and code. For Thanos, it will be, you know, Cortex code. So at the end, this is a significant work and to merge those things. And we kind of, you know, talk about many, many times but I want to show the advantage of all of this. Like we learn from each other so much, right? We, and I was talking about many times is that we collaborate. I know what they are working on. They already know what we work on. And, you know, for example, it allowed Cortex to totally not work on down sampling, right? Because they, hey, Thanos have it and we will kind of integrate that if it will be needed. So they can experiment on that, right? We are not doing any shuffle sharding, like the Cortex is doing. And maybe they kind of chosen a different direction of compaction. I don't want to go to low level details but essentially they are using small blocks. We are doing large blocks. And, you know, those are different trade-offs. So being able to simultaneously kind of experiment with those two paths in the same time, it was actually enormous. Enormous, you know, a value for all of that. So we make different decisions because our teams have a different culture. We are diverse. So it actually, you know, because we treat each other friendly and we kind of reuse each other code if it's useful, this is super, super massive, massive help so far, right? Nowadays, nowadays I see Cortex maintainers very busy, right? Like they got this Amazon kind of user and I don't think they kind of contribute back. So Cortex maintainers are trying to hide things. So there are some interesting stuff going on. And I think it's worth to see that as a lesson as well, how that really evolves. But so far it's a super useful project. And yeah, just check it out. What do you need from the user perspective? No worries, yeah, it makes complete sense. I'm gonna ask a question about just a lake, somebody, and this is probably something you should probably go on raw codes, LGTM, but how does somebody contribute and get involved in Thanos at some point? Shameless plug. I shouldn't be plugging that guy, we're mortal enemies, but go ahead. You're a competitor, right? We're not competitors, I've won. That's it. So anyway, moving on, let's go. Yeah, so it's actually very easy. Just go to the Thanos IO and click the community button and you should have beautiful documentation, what steps you could do to maybe pick your first issue or maybe you want to give a nice feedback or maybe you want to help us to create tutorial or maybe you want to share your case study or blog post or any talk that you made around Thanos and our project and what we do and maybe you want to be MNT. We are heavily investing in mentorship and we are doing those sessions every three times a year and this year we have four MNTs and now we have another four so it's kind of madness but it's super useful for everyone, I think. So what you can do, just really apply for those programs because by the way, those are not for students. Like you might be a very skilled smart engineer who is like in like some close source company which doesn't give you lots of opportunity to be part of the community and you want to just try to join open source effort and maybe, I don't know, switch a little bit. The focus, we would like to show you the value. We have so many senior engineers, for example, Jessica on the Prometheus mentorship program, like she was working full-time and still participating as MNT and I was super useful for her to learn how open source looks like, right? So this is another way how we can get involved in the community. So yeah, just go to this website, find yourself the documentation and just ask questions to the like channel, yeah? If you have any. Fantastic, and again, no rest for the wicked here. You're so busy, man. By the way, I want to tell you, I appreciate you coming on here. I know you've been super, super busy. Let's talk about this. You're writing a, dude, what don't you do? You're writing a damn book? Tell me about that. Yeah, why not, right? I think, yeah, exactly. The question is why not? Like, I think, you know, people still learn a lot from books, like technical books. I love reading them. And especially if it's about like generic programming, whatever, like maybe technology that doesn't really get obsolete that quick. It's worth reading, but no one really writes books. Like we have only those fancy people who are doing streams, right, Pop? And wait, hold on. What? Go ahead, I personally attacked on that one, man. Just kidding. Like I think it's just, you know, more rewarding to do a video, more consumable for users. So I really understand this effort, but I think it's not many people write book, right? And it's not sexy anymore. And I talked, I really wanted to find a co-author and I really found, I mean, Fred Erick is helping me, Fred Erick Brancic, who is now... Shout out, shout out. Yeah, I super... Paul Polar wins. Coral Eskai. Shout out to... Amazing person, yeah. He's been so supportive of just everybody. He's just a wonderful person, dude. Shout out to you, Fred Erick. Yeah, I don't know anyone who doesn't like Fred Erick. I don't think anyone such like that exists. So Fred Erick's an amazing person and he agreed to write one chapter with me. So that's amazing. About profiling, by the way, because he's creating this... He's the CEO of the Polar Signals company behind profiling, continuous profiling and stuff. So anyway, and I was trying to find a co-author and it was so hard to, because everyone was calculating money and I was like, oh, this will take me 500 hours of my... And this means this money. And oh, I don't see this happening, right? So I really, if you do it, usually you really do it for just maybe values of teaching others and yeah, other stuff. So to be honest, my other motivation I want to share is that maybe people from me probably will laugh, but there was this kind of spreadsheet and probably is like many of us are having this in companies where we want to share who is good at what thing. And there was a spreadsheet with like a scale from zero to 10 where zero is like, I have no idea what this technology is about. And 10 is, I wrote a damn book about it. So I want to be 10. I was always like eight and like in GoLang, whatever. I was, and I write book about GoLang and performance. I didn't share the title. I don't, I didn't even share really publicly what it is about. So I slowly started to talk about that, but I really, really wanted to try this out. Well, best of luck and I'll probably end up having you on the podcast and we can talk about it live when it's about to launch. I'll probably buy like 10 copies and give them away. See, that's just, that's the thing I do, man. Anyway, so I get this question from my man, Narin. Thanks for joining the stream, but can newcomers to KDK utilize Thanos or is it focused on the enterprise use case? Yeah, good question. Actually, I would say it's mainly for Cubanist customers because well, it's in the CNCF kind of ecosystem. So it's kind of communities is our main targeted orchestration system. Of course, we make it sure it's like a containerized so you can run everywhere, but we have some special discovery mechanism and things that makes it much easier to work on communities. And it is actually used a lot in the smaller companies. And I seen this and I will tell you why. The reason is that it has a very flexible deployment model so you can install, you have your Prometheus stack. Usually they start, like if you want to start with observability and metrics, just run Prometheus. It's a single binary, just learn how to do that first. Then you can gradually install stuff. You can install just a sidecar and a courier and suddenly you have a global view because you can connect many of those Prometheus under the same and a Promql engine. And then if that feels all right to you and you say, oh, deploying is kind of nice, but I want to add as object storage support and like a cheap storage. So then you install additional two microservices. And then if you say, oh, now I'm a super user and I want like multi tenant and like amazing scalability, then I can think about maybe receiver and maybe remote write replication. And there is like lots of steps you can do. You can mix all of this. So it's very actually welcoming for just experimentation different parts and not implementing everything and getting migrating and like all those big step we are iterating, right? Trying to have this process of iteration. Alrighty, let me ask you this. I mean, this is kind of the last question I have for you, but what's next for Thanos? And what's next for, let me break it into two questions. What's next for Bartek and what's next for Thanos? Oh man, I think it's really something related to what you said that it's very important to delegate at some point, right, to the work. And I think we build like amazing team right now in Thanos where I can go to vacations finally, right? And I can focus on maybe some more advanced stuff and maybe I'm active in the seek or like tag observability now as a tech lead. I'm writing book and I'm doing lots of stuff around observability for different signals. I, yeah, I have passion to kind of improve this step and make use of pool-based kind of observability collection mechanics better and so on. So that's the future for me. I would say I would love to explore those things while making sure Thanos can integrate into that well. For example, we created a new project called Observatorium. I can plug that, right? And observatorium.io just literally go there and you can see this is like a platform which allows you to deploy Thanos with other observability signals because at the end you want metrics, sure, but there are other cool stuff like logging, everyone logs, everyone do at some point some tracing and continuous profiling. And to be honest, you can totally combine all of this and we are making sure as a Thanos we integrate with other signals, making sure the correlation story makes sense. You can jump from the metric, you can say you can have an alert and you have some amount of errors for some service you are on call and there is incidents and then you know, but what are those errors like? What's going on? You click the exemplar which is kind of the way to link into the tracing. Example situation that happened in your actual container in actual binary thanks of the tracing and this exemplar's kind of support. And then you jump into trace and see exactly who and what error happened and in what kind of circumstances. So those correlation capabilities is when you have like a consistent model of configuring different systems. So that's why observatorium was created. And we know it takes some time to integrate Thanos with those other systems making sure for example, exemplars are supported and they are in some way but we want to kind of extend this. So all of this is kind of grown out of Thanos and this is really also best practices grown, right? We are starting Thanos, I mean, CNCF meetups like literally today we have the first one. So mentees meetups. What we mean CNCF mentees meetup what we mean is that it's a series of meetups where we want to make sure mentees have like a way to practice public speaking but also learn, I mean, teach them about learnings they get and we get from the mentoring as well. So we want other CNCF project to also participate in this group. And I had a link somewhere but I don't have it right now. So maybe later I can send it. But at the end we are doing those things. You know, I mean, there are so many good best practices that came out of Thanos like including book, right? This is why I write this book because like not many people know how to optimize go, right? So this is another stuff that I want to focus on. So this is for me and in some way for Thanos, right? To make sure it works with other of the signals it is just even more scalable and secure, right? So all of those productionize topics really productionization topics. Well, you snapped your finger. You changed the universe. You made Thanos. He did some great stuff today. We talked about a bunch of things. Pardon me for a second. I'm going to speak some Polish to my friend Bartek right now. Thank you Barzo, Suham, Dwa, Bivo, me and you. Skupkan, good? No, it's a little bit of a pro. Let's do it, that's it. All right. All right, thank you so much Bartek for being on the spotlight. I'm going to do some housekeeping and I'll basically go, here we are everyone. So next week on Monday, we have Duffy Cooley. He's this week in Cloud Native. We're going to be talking also Tim. Tim Banks is on solid state and then we have Cloud Native Latinx. A bunch of shows next week. A lot of fun you all. Thank you so much. Remember to follow. Please make sure to follow the Cloud Native here the Twitch channel. We really enjoy them. We really appreciate you all being here enjoying the shows we're putting together. A lot more common everyone. It's going to be really cool. And remember everyone in the community, spotlight is always on you.