 Okay, so you should be streaming now and you are the host again. Awesome. So should that make it open up on the website? Yes, yeah. Does that take a couple of minutes or should I? About 30 seconds, but I'm going to log in and double check it here. Hey, David. Hi, Lars. We're trying to iron out some technical problems here. Okay. David, you joined via the zoom link I posted, right? Not via the website. Yeah, Jimmy, the website still isn't showing the feed. Yeah, I'm checking on it right now. I see the problem. Your sessions aren't supposed to start until. 1215. 1230. Well, right, which were well past and I had it at 1245 in the schedule. I thought that's the central time. I thought I saw that Iran had, sorry, maybe off then. I thought that it was 1145 central and that it was still 45 us. Am I, am I in fact have the wrong thing here? It's so yeah, you're about 40 minutes early. Okay. Well, that's fine then. But that's okay. All right. We can, we can trim this off. So this will be streaming, but as soon as the time rolls around, it will appear on the schedule. Okay. So we can just leave it, let it sit here and it's all set up at this point and just come back to it. Right. Yep. Correct. Okay. Well, thank you for that. And I apologize for screwing the time up there. No worries. Listen, if there's, if you have any problems, feel free to, to hit me up on, on email or, or speaker support. And we'll get you. All right. Thanks y'all. Thank you. All right. Well, Naved come back in 40 minutes. Yeah. That's what I just messaged on IRC. Sorry. I, where's, I, I even like, I was like, Aaron, helpfully converted this into Eastern standard time and it's all right there. And I think I must have misread it. That's fine. Yes. Because he included three time zones in his email. No, he screwed up the time. I am going by the schedule on the website. We'd say 30 PM. Yeah, no, but Iran posted 1245. 1245 is not. He posted 1245 Eastern. In the email that he sent out to everybody. Perhaps Michael and Iran can then join our ESI forum because they're mistaken, right? Yeah. It's going to clash with something where it's not. So this is Wednesday, the 21st at 1145. Okay. So if you come down here, coral reef magic. Yeah. Okay. That's fine. I will let Michael and Iran know that his schedule was off and I wonder, I hope no one's missing it. I don't expect we're going to get a lot of people. Anyway, I'll see you in like 40 minutes. Cool. All right. Let's try this again. Yes. I see we have a visitor. Pierre. Hello. Get the ether pad open here in case we need it. Hello, man. How are you doing? I'm good. How are you? I'm fine. We have one non ESI person here, but he has been silent so far. Pierre, if you're there. Yes. Hello. There we go. Hello. I'll turn on the video. Yes. Hi. Sorry. I am also following a talk of mine, which is happening for 15 minutes. I'm just looking at any potential questions. Okay. I'll be multitasking here, but do, do get started. Well, so I will tell you right now you're the only person on this forum who is not one of the ESI developers. So if you have questions for us, this is the perfect time to ask them. If you don't, we're all just going to sit here and stare uncomfortably at the camera. Okay. Just a sec. No pressure. No pressure though. No one asking questions. Well, so if, if you don't know me, I'm Pierre Rito. I work for a stack HPC. I've been involved in the blaza project. Ever since it was revived. So it blaza originally was created. With the name climate back in 2013. And that was by a completely different. A set of people, including Sylvan Bosa, who is one of the core reviewers in Nova. And then it became inactive. And then we, we revived it. On my side, it was part of the, the chameleon cloud. And there were also people from entity in Japan. So obviously we, I looked a bit at ESI over the last few days when I saw it on the, on the schedule. I think what, what you've done looks, looks great. Obviously you have different requirements to, to what we're originally the blaza requirement. But it, it would be good to see where, where we can collaborate with some, we can share some ideas and things like that. I think having bare metal support directly in blaza, I think would be interesting for us. So if there's anything that's a, we can share between the two projects that, yeah, I think that would be variable. Min, I know you had looked at blazer more thoroughly than I think any of the rest of us, other than not having support for ironic at the time. Were there issues with the model that made it a poor match for what you're trying to do. I think the primary thing besides, well, the, the biggest thing was the lack of running support. Yeah. But that's something that I think that it could be added to blazer really easily now. It's a simple matter of just flipping, of just setting the, the less the ID on the, on the bare metal note now. And I assume that would be pretty easily done through blazer. The other thing I think was the owner less the model where it controls when their machine is, is actually available. Right. I, I, I could be wrong. So if you're please correct me if I'm wrong, but I don't think blazer quite supported that. If I remember correctly. Yeah, because it's, it's not being built. With those requirements in mind, I think technically you could do anything like that as a, as just a new plugin inside blazer. Because the, the plugins themselves, they can provide new API on endpoints and new manager code. So essentially you can reuse the, the kind of the shell provides all the, all the usual open stack things like database access and API shared code and things like that. And I'm just have your, your own custom behavior in there. So technically I think it would have been possible as, as a blazer plugin. I, I, I might take a large, if you think it's worth the time, I might actually take another look at the blazer code in the, today and tomorrow is almost down anyway. It should, it should be, it should be not down by the end of the day today. Yeah. So the power, the power outages that they've done their maintenance work, everything's coming back up. Pierre, have there been a lot of changes to the blazer code base in like the past year? Has it been under fairly active development? No, not very much. Unfortunately they, so there are still some people active mostly in Chameleon because they, they use blazer in production every, every day on the, have quite a large user community. The people from entity Japan are not as active as they used to be. So I'm also trying to recruit people. I'm encouraged them to, to contribute. Right. But you, I see that no, you've, you've quite, you've got quite a lot of code in ESI. You've built a whole new project. I was wondering, do you have plans of integrating into the OpenStack ecosystem? Or do you, do you think you're going to stay separate? Well, I mean, a large portion of the work is an upstream ironic. So it's already part of the OpenStack ecosystem. I don't feel that we've really thought about the long-term plans for the leasing service that we are working on. We have some immediate needs that we need to meet, that we want to actually get that code sitting in front of people and have people start interacting with it. And I think after we achieve that milestone, we'll have more time to spend thinking about like longer term aspects of the project. Right. But as, as kind of large kind of alluded to, like the, the ESI leasing service is pretty replaceable. Like we can replace it with a different leasing service and everything we've done in ironic would still apply. And like user, our users will still be able to interact with the bare metal machines in the same way. So we're pretty flexible on that front. Okay. Well, whenever I get some time, I'd like to look at what you've done for the leasing service and see if there are opportunities to, to get the two project closer. Because obviously it looks like a lot of the logic, you've actually merged into ironic. So this is something that we could, that could be reused by Blaser. Just like you need with the ESI lease. Yep. 100%. Like the, the integration between the, our leasing service, and ironic it just, it just sets a project ID in for a bare metal node in ironic. That's all it does. So blazer can do the same thing than, you know, blazer could also take advantage of our, of what we're doing in the multi-tenant ironic space. It's that simple. So it occurs to me, man, that would be something that would be an interesting project for someone would be to put together a, a portable ESI demo environment for people who want to like get their hands on it seeing it works. I see how it works. Like a, like a vagrant configuration that gives you a controller and two or three nodes and lets you experiment with the command lines and things like that. So that for people who are coming to the project and like, what exactly does it do for me? That there'd be a quick way of getting that up and running so that people could play with it. Okay. Yeah. No, that, that makes sense. Like a death stack. That's something, ideally something even more opinionated the death stack. So that it's basically run a single command and you have a cluster of machines to work with. Yeah. Just to try to lower the bar as far as possible for people who just want to get their hands on and like see how it operates. And so one, one thing I'm curious about is that, so do you have actual end users of ESI at the moment already? No, we do not. That's why, that's why we're kind of really focused on that initial milestone. Cause once we get our goal is to get this in place at the mass open cloud to manage bare metal hardware there at that point, we will have actual consumers of the code and we'll be using it ourselves also to operate that environment. So that's, we're really hoping to get there soon. Yeah. I'm just asking because I remember seeing in your slides that they, you only have a CLI. I'm wondering how much that is a barrier for, for end users, but from my experience on, on chameleon people in majority use the, the horizon interface. Right. It's a fair point. So what we're aiming to replace at the MOC currently only has a CLI. We're trying to replace the existing system that only has a CLI. So for now, I think a CLI would be sufficient, but we definitely would be interested in like a GUI at some point. Yeah. But, and initially the primary consumer of the system will be operators at the mass open cloud as opposed to arbitrary end users who are looking for hardware. So that's kind of going to be a slightly more distant goal, I think. But yeah, I think we all recognize that we need a GUI at some point. Yeah. It's a big difference if the intended audience. Yeah. System administrators. Yeah. You know, it looks like we've got a really full forum here, but honestly. It's still really just Pierre, who is the only person who's not involved in the project in some fashion. Hey guys. We're on. Hey, Ron. The opening for labs. When just went a little long. So just ended. So. That's right. Ron, we have Pierre here from the blazer project who has been talking to a little bit about that. Or on here is the. PI, the faculty at. Boston University, who's. Been behind a lot of the research that eventually turned into the CSI project. Yeah. Hello. Hey. So are you familiar actually with, I guess, chameleon is using blazer. Very heavily, right? Yeah. I used to work on the project. Closely with Kate. I think, you know. So, so yeah, I was the. The lead DevOps engineer from. I mean, it's inception to. Me. 2018. Great. Yeah. So we're meeting with Kate soon. Just being talking with her and trying to understand what they. What is it that in that environment it was prescriptive where. Sorry. Is it that chameleon interacted with ironic as a. As an administrative user essentially. Is that. Is that how things have worked till now without the multi-tenancy. So do you mean multi-tenancy. Of hardware providers. So the capabilities that, that, you know, these guys have actually implemented inside ironic now. That didn't exist till now so that you could come in. Lisa know to attend it with a particular. With their own credentials against the base environment. Like, so that seems to be like a fundamental requirement. But Camille clearly has worked without it. So how have they worked on that? Well, it. It kind of provides the same. The same end goal, but you don't get access to the ironic. API itself as a user. In Camille and you can, you can reserve nodes. Through blazor, but that gives you a reservation of an actual. One or many. Bermitt or hypervisals. And then you use know about to instantiate. Well, Bermitt or instances on them. And so it's still multi-tenant. But you, you don't get access to the ironic API. Like you can do in, in ESI as far as. If that model, your multi-tenancy is provided by Nova. Yes. Yeah. Yeah. So that's, that's the main difference in design. Cool. That makes sense to me. So it does make sense that. I apologize. They had an urgent call. So it makes sense. The blazer could be layered on top of what we're doing in the long-term. I suppose so. Yes. Yeah. Cool. I just want to make sure that we weren't reproducing something like it, it always makes sense eventually to push these features into the base platform, which is the way that I think, you know, has been doing this is that, you know, so good. I wanted to make sure we weren't missing something because we, I sort of felt bad. I hadn't reached out to Kate. Well, I, I don't exactly what you talked about. But because I haven't been involving Camilla on for the last couple of years, although I've been following more or less what, what they've been doing. But yeah, I think there's some complimentary approaches that can be done. Excellent. Good. So I have questions then since. Go ahead. The real work. This is the place for it. So do we have. So basically, if I look at what I understand of the project is, is, um, We've provided the mechanism so now we can properly authenticate things so that only users that have rights to a machine can actually operate on that. You know, through these multi-tensity capabilities. Right. The, and now we're going to go and examine the performance, which I'm really excited to find out how, how much better or how much worse this does when the research book has we had before. Um, but the, um, in some sense it's a very, very rich API. Right. That you have, if you want to go, you know, you're talking to neutron with, um, for the networking side and, and to ironic for the provisioning side and, and, um, is that's part of the thing that worries me is that really rich API. Now it sounds like you guys have addressed that part of the pipeline layer. Um, do we think we need to address that the API layer as well, where there's a sort of limited API that things can programmatically interact with. It's not exposing all of open stack. I'm not sure because one of the motivations for trying to move these features into ironic is to be able to take advantage of this established ecosystem that already exists and there's tools that can work with it. And people who are familiar with it and there are libraries that support it. And if we replace the open stack reference API is with something special, we have, we've discarded most of that advantage. Um, and it helps us still because we can take advantage of those things in our code, but it makes it more difficult for people to integrate with ESI then because then they have to do special things other than just talking to open stack. And maybe, maybe we do some, maybe we make that an option. Maybe there is some kind of front end service that prevents that provides a minimal API for specific situations, but then you'd have to write tooling that talks to that. And it seems like it seems difficult to meet like both sets of requirements to be able to offer a standard suite of APIs that are well documented and people use and also offer the simplified one in a way that is useful. I don't know. I'm curious. I'm sorry. Do you have the same view? Um, I kind of, well, I, I kind of had a different understanding of a question, which was whether, whether it would be possible to push some of the logic in art in like the ESI CLI directly into ironic or neutron and then, you know, and have that logic that API layer instead of at the CLI layer. My, my impression, what, from what I've, my impression of the projects is they're not really that interested in converging to that extent to, and took, yeah, to converging to that extent. They kind of want to remain separate. So that's, that's why I think we took the approach we did. I think in Iran, you can correct me if I'm wrong, that Iran's concerns more around the security surface involved when you're exposing the entire open stack API to consumers versus if you can present a more limited API that just lets you do the particular things that, you know, we want people to do. Oh, I see. Yeah. I mean, I can see, I believe there are already tools out there that let you set up a proxy in front of an API with rules applied to specific routes and, you know, rate limiting and don't expose particular things without having to reimplement the back end service. So maybe that would be an option to look at some kind of front end rule engine that can restrict access to only specific API endpoints. So that rather than having to write a service, you are configuring an existing tool and you can provide it with some kind of, you know, input that says here are the things that should be allowed. Yeah. Yeah, so there's definitely the attack surface is one element that might partially address it. It's, but, you know, just even exposing a richer API than you need to is concerning to me. There's, in some sense, I guess the leasing stuff you've done that is, how's that connected now? Maybe I didn't quite understand is that that's itself as a separate service? Yeah, that's correct. It's a separate service. And how's that? So how does that turn into its own project? Or is it has that being integrated the opens with the broad open stack community there? Um, it's, I think that's, that's, uh, that's open to discussion. I think we're kind of waiting to see how well, how well it fits the needs of the MLC of the, of the help be my replacement first to kind of make to kind of make sure that the requirements and the maturity is where we want it to be. And then we would take a look after that. Okay. I think that, that maybe the next layer to think about that is, um, it's just right now we have one ESI instance, right? That we're, or we have multiple, but I mean, in, in, like, we were sort of thinking about how different consumers of one ESI instance. But as we go down this, there has to be something that arbitrates when you have multiple ESI instances. So Harvard has its pool of a thousand, 10,000 nodes that's putting in their ESI and BU has it. So we want the MLC to be able to, or the open cloud test, but the graduates from each of us and have networks that spend them, right? Um, and in that model, there's always going to have to be some intermediary, some intermediary that's trusted for each ESI that, um, for example, knows that it can plumb through this VLAN, right? So you have to know, you have to come to a prior agreement on which VLANs are owned by which ESI instance and then say, yes, I have, he's giving me blessing to use that VLAN on these machines in this cluster, right? And that's almost more of a neutron thing because neutron is the tool that's aware of things like VLANs and switches and topologies. Yes. So what I guess I'm saying is that, that there's a, you right now have this intermediary for the leasing, but in terms as we need an intermediary for networks. Um, and maybe that's what ESI is, is this intermediary that handles the bridging for multiple different things? Does that make sense? Sure. Because I don't think neutron would want to, in the same way you're saying that, you know, they don't want to necessarily integrate the leasing service into ironic and it's spanning multiple things. Maybe that's the same thing for networking. Maybe I can see, I mean, so for, for the stuff we're doing with ironic, I mean, ironic has a very kind of specific goal, which is being the thing that manages hardware and the stuff that we're doing with leasing and ownership and whatnot are kind of a separate thing. So it has to be a separate project, but with neutron, I can see outside of ESI, I can see people, why people would want to have site to site connections. Um, just for purposes of, um, integration or disaster recovery or, you know, distributed services, all of those things, similar requirements. I wouldn't be surprised to find if someone has already started trying to solve at least part of this problem. Um, how do you connect to open stack deployments in a useful fashion? Um, Yeah. So I don't know if that means, I don't know if we'll have to do the same kind of intermediary service that we've done with leasing and ironic, or if we'll be able to piggyback on top of something that already exists or that somebody is already working on. I think that's definitely an interesting area of research though. Figure out what's out there and how we can utilize it. Yeah, I think Christie looked at some of the projects there that were in that space of network Federation specifically. So he may have some visibility. There was a project at the time, like an upstream open stack project called a tricycle, but nothing came out of it eventually. Okay. I don't think it's still active. There were some ways to federate contrail networks. But I haven't investigated much into those because it was a pain to set up at the time. This was three years ago, I think. And, uh, one solution that we tried was just a stitching the open, which matches together, which works in the very small scale. But it's a mess when you go into a bigger scale. Well, I know this is the extent of that. They've replaced OVS by itself with OVN to try to solve some of the management issues with OVS. And I don't know if that would have implications for what you were just talking about either. But, you know, they've, they've been able to reduce the dependency unlike the Neutron L3 agents, which has performance benefits for the networking, for the operation of the networking and things like that. I see. Well, I haven't looked into this in the last two years. So things may be a lot different. And there's also projects like a, I think Calico, which just does away with L2 at all. And you only have to use IP addressing, which might also be something to look at into in this case. So I think that's all fine and dandy for stitching things at the high level. But this has a requirement essentially to look at it at the layer two level where we want to kind of create these overlay networks with VLANs. And we want to do it as rapidly as possible. We have the capability, for example, in our infrastructure to come to agreement about fully trumped network scoops. Speaking of which, I have to run to a Federation meeting. I'm sorry. All right. I have to run as well. That's totally fine. Thank you for, thank you for dropping by. We're still in Europe here. So it's the time for me. Have a good day. All right. You too, Pierre. Well, I think that's all the excitement. Yeah. I think that's all the excitement, but I'm just curious on the call is just basically. Yes. So how are you guys doing? Well, hey, we made it a half hour. Actually, that's not so bad. Honestly, that's better than I expected. Okay. I'm sorry that I joined later earlier one ran over. Look at you and your clever background. How are you, man? I mean, you know, hate stupid backgrounds for a year. Well, I think we give this a couple more minutes just to see if any random people shows up and then we'll just close it at close it at two o'clock. Maybe maybe. Yeah, I have one thing to discuss maybe go ahead. Yeah, so I. So the thing I think we test this briefly in one of the previous meetings I joined was the security aspects of things right. Yeah, so like, like, is there a design document or a plan for that, and how are we going to go about it. We have a specific design document addressing security right now and that's largely because we're relying primarily on existing open stack projects to handle almost everything with the exception of the leasing component. Are there, what are what are the kind of the specific areas that you want to see addressed the secure boot component that's that's one of the critical things and station. And do we have design documents on the key lime stuff yet. I know that that's on the roadmap. So are you talking with Mike Peters there. I'm Mike Peters. Mike Peters is at red hat right. He's working with Luke Heinz on the key lime stuff. I know, I know Danny and Luke, sorry, and Leo, Danny and Leo have been talking, I've been talking to Luke. Have you talked to Mike Peters. And not yet. Okay. So yeah, we, we at IBM have been having like continuous meetings with both Luke and Danny, and also looking into also looking into the secure boot aspect so maybe we should we should start having combined meetings on that like, maybe what whatever you guys have been doing can be simply reused right. And try to integrate that directly to open stack setup that we are trying. Yep. So, um, Sorry, go ahead Danny. Anyway, go ahead. I was about to say like so, most of the work that Danny and Leo have been doing they've been working directly with ironic team so it's, if you want to integrate secure boot with, you know, into ironic that's probably the best place to start to be honest to talk with them on how they might approach it. Sorry, go ahead, man. No, I was about to say, but like, that's the stuff Leo and Danny are working on right now they've, they've come up with a spec that's with ironic team ironic team has looked at it they're iterating over it. So, we expect that you know we're going to this this entire integration process is going to take place in upstream ironic and we're just following the procedure there. One of our questions around that whole process is where it makes sense to perform attestation. Because if we roll that into ironic itself and it becomes part of ESI it doesn't solve the trust problem for someone consuming the hardware. Because if we run, if we run the attestation on the server side, then they're trusting us to do that. So we've we have often viewed the attestation is something that needs to happen. It needs to be controlled by the consumer of the hardware. Exactly, exactly. That that's the that that has been always the initiative like started like when we started designing ESI initially that was always in our mind and the key is that the that's why key line is the key component because it enables remote attestation and where that remote attestation aside it has to be completely tenant control. Yeah, and trusted, not just control but trusted as well. So if we integrate at the stations ironic then listen to just leave it up to the consumer when they want to run it how are they want to run it and yes I really doesn't need to have much say about that. Sure, but the provisioning service has to comply to whatever pros or protocol is required for secure boot right. Wherever like remote attestation can exist anywhere in the world, maybe maybe over a secure channel, even over the internet we don't know, but how that has to interact has to be like clearly defined in ironic. So sorry. So you said that ironic, like probably we should be having talking to ironic team about that. That's that's what are you saying like I didn't catch it. Yeah, we talked to ironic upstream and proposed spec on the security interface and key line interface in order to have a key line driver to do the attestation. Do you mean this thing. I see. So is that document publicly available. It's still on the review at review open stack, I think. But you can take a look at the, at the review, let me link it on the etherpad I guess. If you take a look on the etherpad line 17. That that's, can you, can you please share the link for etherpad I don't have it sorry. Oh, sorry. In the place. Oh, it's in this other chat. Okay. I think they didn't chat. So language. Okay review. I see. I see. So yeah, I definitely have a look at it. And yeah, thanks thanks for sharing this. Well, I'm going to declare this over unless we have anyone else who's got some questions going once. Going twice. All right. Well, thank you everyone for dropping by. Thank you guys for the great, great presentation. It was a good presentation men. Thank you. And thank you for leaving this forum Lars. It's been really hard work. All right. See you folks. All right.