 All right, we should begin. We got the sign. Hello, everyone. Welcome to our SICK-SED sessions today. And the topic is forging a stronger bump between SED and Kubernetes. My name is Wen Jia from Google. I am one of the SED project maintainers. And now I'm a co-chair of SICK-SED. And it's my pleasure to have Merrick and James to co-speak with me. Merrick, do you want to introduce yourself? Sure. Hey, I'm Mark Sharkovich. I've been an SED maintainer for the last two years. And with the creation of the SICK, I am the tech lead for it. And I'm very happy to have you here. Thanks, Merrick. Hey, team. So my name's James Blair. I work at Red Hat. And I'm a reviewer for SED along with a co-chair for SED as well with Wen Jia. That's me. All right. Now let's get into our agenda. Today we'll first start with sharing a great news about SED. Or like we haven't mentioned SICK-SED millions of times. And then Merrick will give us a deep dive about the API contract between Kubernetes and SED. Followed by that, James will give us detailed introduction of the new community model and the recent SED releases, features, announcements. We'll close this talk by providing a list of SED opportunities or activities here at Kukang. And then we'll open up for questions. So it's been many years, actually almost close to a decade now that SED had been playing a very critical role in Kubernetes. SED is the data store for configuration data, status data, metadata in Kubernetes control plane. So if we take a quick look at the brief history of Kubernetes, back in 2013, actually a decade ago, the very first SED PR was committed to the SED Ripple. And then very soon after that, we had SED v0.2 released. At the time, Kubernetes is at 0.4 release. And baby Kubernetes adopted baby SED already. Then we released version 2.0. And then we rewrite the API released to v3. And when time comes to 2018, in KubeCon Seattle on a raining day, SED was officially accepted by CNCF as an incubating project. And there, we were all getting together with the founder of SED, maintainers of SED to celebrate that moment. And then two years after 2020, SED moved to graduated level as a CNCF project. And yes, SED is a pandemic graduate. And we can blame all the data inconsistency to that. And then three years later, March this year, Hong Kong and Merrick proposed a case of making SED a special interest group in Kubernetes. And it took a little while to finalize all the details. But we did it on September 12th. That was when SED officially become a special interest group in Kubernetes. And we have the blog post on both SED and Kubernetes website. You can go to the link and find more details. And here's when you should clap. All right. So in the new SED, we adopt the Kubernetes governance. And James and I are honored to serve as a chair of the SED. And Merrick and Benjamin Wong from Vimo are the tech leads. So why do we do that? The goal was very clearly defined in the charter and vision. We do want to build the best production data store for building cloud-native distributed system, like Kubernetes. And we aim to have the most reliable distributed key value store that is simple to maintain and scales for Kubernetes and beyond. Well, how do we do that? In another word, what change does SIC-SED bring to the community? From the Kubernetes perspective, SIC-SED represents, from SED perspective, SIC-SED represents the SED interest in the Kubernetes community. And from the Kubernetes perspective, now SED adopting Kubernetes governments would make it easy to delegate the decision making to SIC-SED. And also we will adopt the consistent process from Kubernetes, like CAPS and PR reviews that will bring the SED code base to the next level, the quality of the code base to the next level. And then we will leverage the Kubernetes release machinery, security responses, and all that. So going forward, there will be continuous investment on some ongoing works. And then there are also some new initiatives associated with SIC-SED. So to name a few, we will continue building the robustness test to avoid data inconsistency. And we will codify the API contract between Kubernetes and SED. And we will continue work on the downgrade story, recovery from the backups. And we are actually planning to provide the atomics to extend SED past current dimensions. Thanks to Hong, now we have a working prototype, actually. What I do want to emphasize the most now is that without the community, without you guys working with us, it will be really, really difficult to get any of this or to get any of this done in the speed that we want. So Special Interest Group is not a magic. Nothing automatically gets better just because we are sick now. It is definitely a great milestone, but it's more of a start of a greater journey ahead of us. And we are all in this to make a difference. So how do we get involved? We do have weekly SED meetings, or donating between community meetings and bug triage meetings, like some of the other six. And we have various channels for offline conversations. We do have an ongoing mentorship program. We cannot add any more people anymore, but we are aiming to provide opportunities like this more in the future. So stay tuned. And you can find the information in the SED.io community. If you cannot find that, please submit a PR to fix the website. With that, I will give it to Marek. Yeah, thanks, Wenja. So I would now want to go more deeply into the Kubernetes HCD contract. The reason is that even though the projects have been for a really, really long time present, the contract itself was never well-defined. And there are still a lot of gaps and correctness, things that we could improve, leading to better performance and better resulting community by allowing further extension of Kubernetes CRD, allowing better extension and better performance of Kubernetes. So API machinery in Kubernetes is pretty complicated. You may start your journey on simple keep CTL get or create and learn about the reconciliation loop. But this is really just a tip of the iceberg. The most of the work to make Kubernetes more performant and scale better is done on the dark bottom on the end. And we have seen a huge improvement in performance because of this. Like there is two, the two on the bottom watch list and consistent cache reads will really reimagine the performance that we can reach. But it makes things pretty complicated. So to be sure that the whole system behaves correctly, you need to understand each and every layer. And we have made some historical mistakes here. No single person is aware of everything. And we have documented cases where Kubernetes just got it wrong. But yeah, so starting from the top, Kubernetes uses leases to manage events. And for events to have limited time spent, we assign to events a lease, which after some time will delete the event. But the problem that it creates is that Kubernetes, for performance reasons, Kubernetes assigns multiple keys or events to the lease. And this breaks Kubernetes watch because when lease expires and events are deleted, they are all deleted in single transaction. And this leading to multiple revisions pointing to the same delete event. And this never happens normally in Kubernetes. And it breaks the assumptions built by the watch. So even as of now, when you are watching Kubernetes events, the protocol can have errors and miss some, you can miss some events or delete events. So it's something that all communities are aware of. And we are working about the fix. But no one spotted it for a very, very long time. Second, HCD wanted to do a performance improvement and how to best improve a list request when you have pagination is stop looking at the old objects and skip the limit or use the limit and skip reading all the objects. But this broke Kubernetes remaining items count feature, which allows clients to know how many objects are still left in HCD to be read after multiple page requests. And unfortunately, we only cut it after HCD release. And the third was HCD watch notification implementation. So for watch to be performant for resources that don't get updated very often, we sent notification about how resource version is progressing to those watchers. So that when they disconnect, they will get much instead of reading from very, very long revision, they will be more up to date. And this, unfortunately, was broken and very broken by allowing a race between an event and watch notification update. So for Kubernetes to really optimize watch and move to the second step, we need this to work. So the story begins with HCD data inconsistencies that really made us look into how we can test and validate the correctness of the API. So it's not about simple requests going into HCD and getting a response. It's about whole distributed system providing the guarantees that we would expect. And testing this is very complicated. So even HCD has done it wrong. And we've released a 3.5. We had a couple of those issues. And this motivated a full effort to build a robustness framework which up to which can take any generic traffic, any HCD setup, and even inject failures into specific places of the code base or simulate any network failure. And we can take all this history of all this experiment, collect all the events, and then validate them for correctness, allowing us to find not only react to bugs or react to failures, we can proactively discover and hunt for the new issues. That way protecting the project before discovery. And this made us look into the Kubernetes HCD contract because if you're validating correctness and trying to explore full space of how we can use API, the best way to start is from exactly how your most popular user or your most popular users are using the API. So when investigating Kubernetes, how Kubernetes uses HCD, we found that no single person was aware of all the behaviors and all the details were hard to find because they're old code base and the code is spread around. So we did a full collaboration of creating a document that everyone could put in their ideas and their thoughts. And we compiled it in and made it public and available. Two interesting things that we discovered is we formalized how Kubernetes Sharding works. If you didn't know, Kubernetes doesn't need to run on a single HCD. You can delegate each resource to a separate HCD instance or a cluster. And this was never really defined how it works, why it works. So we tried to get it right and made everyone aware of it so Kubernetes can utilize it because some people were using it for a very long time, but most of the community was not aware of this. Second one is watch bookmarks. When the feature was implemented, we had some intention how it should work, but it was never really written down and tested for. So by defining this guarantee, we could write specific tests or verification logic that allowed us to find multiple issues with it. What are the results? We discovered that watch in a network partition can travel back in time. So if there was a long enough time for HCD member to be disconnected from a cluster so that it will not catch up by changes but request a snapshot, we found that applying the snapshot will reset the watch. And sometimes the snapshot can be older than the member where the member has already, where it was before, leading to the watch jumping back in time. So that was totally surprising find that required a very detailed definition of watch guarantees and how it behaves, and then writing a dedicated code that can validate those guarantees. Second, we discovered a duplicated event. It mostly goes into HCD client logic for resetting or retrying to get a watch stream. So if there is a network failure, HCD has some logic to try again and restore the stream, and we found that there was, in some specific case, off by one error. Yeah, so with the contract defined and the testing available, we are looking into the future to make the contract something really solid and well tested. We are still missing a couple of things that are harder to implement from the contract. For example, compaction, because compaction at HCD cuts the history, making validation of it a little bit harder. So we still need some work on that. We have a lot of failure conditions, like network partition, like panics, like delay in requests, or delay in specific place of draft. But we need to do more. For example, we are really interested in utilizing the library for injecting disk failures, so to be able to inject any arbitrary disk failure and see how HCD really responds. We technically know how it should respond. We have some hopes that it will do that, but no one ever tested it. So let's find out how to be able to announce everyone that yes, we are testing, and any bug will not pass our reviewers. Going further outside of HCD, by defining a contract, we can implement a fake implementation. It means that when Kubernetes tests run its own tests, it talks to HCD, real HCD, and this limits what we can validate. But if we had a fake that behaves like HCD, we can then bring it to Kubernetes testing framework and replace it with it and cross validate whether HCD Kubernetes works the same as on the fake and versus the real thing and see if there's any inconsistencies. And then we can do arbitrary changes in the fake to find it's called mutative testing to verify if something would change in the HCD implementation, would it break Kubernetes contract? Thus solidifying what we know that is a guarantee that might stand versus what is not. And the last thing is validating the Kubernetes storage layer. So Kubernetes has pretty, like I described, the code that is used to define contract is very spread around and it makes testing pretty hard. If we could bring the same testing, like instead of testing everything on HCD layer, test it on full Kubernetes layer and then validate that Kubernetes behavior or an API server behavior, we should be able to prove and solidify the contract on the correctness or our assumption about correctness of Kubernetes even more and then be sure that whatever performance optimization we do will not cause any correctness issue, allowing us to really go beyond in that. And that's all about the HCD Kubernetes contract. Let me go to James. Thanks, Merrick. So to kind of round out the story of closer integration between Kubernetes and HCD, we have to talk about the humans. We have to talk about the community. So fundamentally, the way we approach community membership directly impacts how our community can grow. So with that in mind and taking on all of the lessons from Kubernetes where we can, in April this year, we formally adopted a new membership model. So loosely based on the Kubernetes membership model, one of the big changes there was the new member role. So sort of like that first rung on the ladder when you're sort of interested in getting into contributing to a project. And we've also sort of clarified and tidied up the roles and responsibilities for a reviewer and maintainer as well. And if we just have a quick look at the requirements for that member role that are up on the screen there, they are really straightforward team. So just where we're deliberately vague about multiple contributions, we haven't put a number there. And essentially, as long as you've got some PRs merged and you've been involved maybe in issues, perhaps, or contributing to discussions, you just need some sponsorship from a maintainer or a couple of reviewers. And just like Kubernetes with Kubernetes slash org where you open up an issue to add yourself as a member, we now have a similar process there for LCDs. So that entry level rung is now there. And for anyone that's really keen to get involved in the project, I really invite you to take that up. Jumping forward. So I just want to highlight a couple of numbers. So there's sort of a historic trend on how the LCD project, total contributions by month have looked. And we're pretty stoked to report things are going in a pretty positive direction of late over the last sort of year and a bit. So let's hope that continues. And we'd love to have you adding to those numbers and joining the team. Looking at PR engagement. So let's say you propose a change to LCD. If we were back in July 2021, you were waiting maybe almost a week to even get a first comment on that PR, which is not too flash. But as you can see, we've been pretty hard at work and the team are doing a great job getting that down. So we're pretty happy about that. Jumping forward. So quick results. We've had this new model for six months now. We've got seven new members in the team. And we've got two new reviewers from those members so that the progression is working. It's been awesome. So we've also now got, thanks to Wengere and the team, the mentoring program underway with the first cohort started. So just know that there's going to be future iterations of that program. And it's another way to help kind of get over those hurdles of understanding the deep complexities of both its city and Kubernetes and contributing. Jumping forward. So very quickly, that kind of rounds out the story there. We'll have a look at some recent releases and some of the features. So since Amsterdam, we've had three patch releases for our 3.5 stable release branch and same for 3.4. I think we've got another 3.4 patch release coming out very shortly. So as Merrick was mentioning, robustness testing is a huge focus for us. If anyone was paying attention, there's actually a data inconsistency between Merrick's slide and my slide as well in terms of the numbers up on the screen. But we've added some more features there. So we've now got support for LazyFS, which is a Fuse-based file system that allows us to, as Merrick was talking about, inject disk failures. We now need to actually pick up on that feature and start adding a lot of instances of using it. We've now got Kubernetes-specific traffic generation for those robustness tests so we can model what Kubernetes would look like when it interacts with XED in these tests. And we've got some more network failure injection happening, which is awesome. Another big one. So if I rewind, it was just a couple of months ago, we announced tier one support for ARM64. So there was a fair bit of work, kind of, wrangling tests and workflows and trying to get equivalents between AMD64 and ARM64. So for anyone in the audience that is running your clusters on ARM, you can sleep easier at night now. So funnily enough, as part of this, we kind of rewrote a whole bunch of documentation and kind of strengthened a bunch of things. And then shortly after, had to throw that all away after we moved from static ARM64 CI to on-demand runners. So yeah, good times. Lastly, GRPC. So we've done a migration from GRPC Gateway V1 to V2. Huge bit of work, and the team did a great job with that. So this kind of simplifies the dependency tree for Kubernetes and SED. Got us off a deprecated Gateway version, sort of unblocked our updates for OpenTelemetry and GRPC itself, and that was a pretty big roadmap item for 3.6, our next minor, like next minor release. So that was awesome. Very quickly, there is the summarized view of the roadmap for 3.6. You can see we've got a few in the completed column already, which is awesome. But as you can see, still plenty to do. So we need all the help we can get. And there's issues that kind of walk through this. And the roadmap is on GitHub, if anyone is keen to kind of take a closer look at what those involve. So to finish off, we've got a couple of things still happening at KubeCon. So tomorrow, we've got a couple of talks. Now, unfortunately, they are at the exact same time. So you'll have to pick one if you're keen to learn a little bit more about EtsyD. Maybe I'm biased, but we've got a maintainer talk here from Merrick on secrets of running EtsyD. So for any cluster operators in the room, that'll be the one for you. And we've got the booth. So in the top right-hand corner of the solution showcase, we've got the EtsyD project booth and the project pavilion. So we're staffing that full-time, come and hang out, and ask questions if you're curious on anything. And just very quickly, after KubeCon finishes, we've got the SIG EtsyD Slack channel in Kubernetes Slack. We've obviously got the GitHub link there and the mailing list, so you can reach out to us there. And with that, I think we have almost perfectly five minutes for questions. So if anyone has anything on their mind, take it away. Looks like. I think this is already answered, is the plan still to stick with BeBolt for the foreseeable future? So there is ideas to replace BeBolt as part of the proposed vision for the project because we want to play to EtsyD's trends, which for now like storage was not historically great. We adapted a project that was somewhat deprecated and it both costs, like we are bound to that now to the person maintaining it. So it caused a lot of, it requires maintainer involvement and we are not necessarily specialists. So we want to play to EtsyD's trends, which is watch and reliable replication by raft, and possibly think about opening the door and making, instead of extending, replacing full EtsyD, replacing the storage layer does, opening the idea of larger class or larger storage and larger, maybe larger size and maybe multi-tenant EtsyD. Maybe just to add one very quick thing to that. So there's been a bit of discussion about whether or not you sort of have an interface for Kubernetes itself in terms of the storage layer, right? And I think rather than throwing the baby out with the bathwater, as Merrick says, stick to the strengths of EtsyD around distributed consensus and just have an interface at that lower level for the underlying storage, yeah. Thank you. Then one other question. How solidified is the interface with Kubernetes at this point? Are there a lot of feature additions in the pipeline? So the con, like we are working, we have a document that describes all the behaviors that we expect. And we are working on defining a full Golang interface that we could, would be owned by EtsyD project, but would be used to ensure that we have it solidified that we know when some, there is a new change, that's how we extend and any extension is properly tested and verified. And now what was the last part of that? I guess more simply, are there any major breaking changes in the pipeline? Breaking changes? No, no, no breaking, like breaking changes, the main thing we will never, as long as there is version three, we will not break. It's mostly about utilizing, better utilizing watch and better utilizing watch bookmarking to make Kubernetes watch cache better, work better and serve more traffic instead of always going to EtsyD. So there will be, as long as there is version three, there will be no breaking changes. Kubernetes is really strict about this. Thank you. Just add to your B-bolt thing. The part that I talk about, about the extensibility, that's actually, we do have a prototype that building the interface between SED and B-bolt. So basically, hopefully we can swap the B-bolt with other things. There is a working prototype with SQLite underneath. So we don't have enough people to work on it because we have a list of other priorities, but if you're interested or anyone here is interested, it's a very interesting project to work on. Yeah. Hello. First of all, thank you for the effort on ARM support. Since it became a tier one support, I'm from OK, which is Oracle Cloud. We started rolling out EtsyD clusters on ARM and they are doing great so far and we are slowly rolling out across the regions. Regarding the API layer between Kubernetes and EtsyD, maybe I missed your point earlier. So is it intended to have additional implementation just like EtsyD because now it's tightly supported? Once you formalize the interface between Kubernetes and EtsyD, there can be additional implementations, maybe Postgres or... Is that one of the intentions of formalizing the interface? I can quickly respond because we are out of time and now the API machinery in Kubernetes is really strict. Our goal is not to replace EtsyD. The reason is as long as we have EtsyD as a performance goal, everyone is in the same bucket that we need to... We can optimize Kubernetes really for EtsyD and having an alternative implementation would bring more harm to Kubernetes by sharding the community too much. So that's why we are thinking about which layer to do that and it doesn't seem like at least for Kubernetes perspective that the whole EtsyD replacement would be helpful. Thank you, thanks for your contributions. Okay, team, with that I think we're out of time. Huge thanks for everyone for coming along. Great to see you all here, thanks.