 Streaming is going and then I will return the host duties to you. Are we sure streaming is actually working? Yeah. Because in Safari, which has always worked so at least worked for streaming. It says the media cannot be viewed. We had a couple problems with like two sessions yesterday. But I'm double checking. Yeah. Apparently, one other session didn't get it recorded. Diversity inclusion, but. And I'm frozen on checking credentials. Yay. There we go. Okay. Okay. Streaming is. Streaming is working. Okay. Okay. Awesome. Thank you. And Julia, if you want the session recorded separately. You can hit the record button down at the bottom is the host. And just ping me afterwards and I can share it with you. Oh, interesting. Okay. I think people are thinking that because it was being streamed that it would automatically be recorded. We do have them on just if you want an additional one, it'll record the chat and things like that as well. Okay. All right. Thanks everybody. Thank you. Good morning. This is our operator feedback session. Should we go ahead and get started? Unfortunately. I see two operators. So operators click the. Take the virtual mic and participate button and join now. Just out of curiosity. How was the. Like the zoom link communication. Because I couldn't find it. So. So the zoom link only becomes available at the exact time when the session is supposed to start. Okay. Yeah. Interesting. We have someone's iPhone here. Sweet. I'd love more operators. I know people use our software. I know our software is getting used. Like every telecom. Just perfect. Nobody needs to discuss anything. This reminds me of the operator sessions in Shanghai. Your complaints and you know what everybody was like. Silence. It was like that. I think it was after we excluded the sync with Nova. Okay. We got a couple more people appearing. So I guess. Ether pad. Where's the path link? Here's the path link. Okay. You could have link is. In the chat. In zoom. And is also in the page. So go ahead and add myself. So I guess. Does anyone want to share. What has worked well. Is this going to be repeated Shanghai. Either silence means everything or nothing. How we are going to know. I don't know. How many people here are operators versus upstream contributors. I mean, That's a great question. Maybe people that are operating can introduce themselves and talk about how they use ironic. Yeah. I think that's a great question. Yeah. Or use the rate of the zoom. Raise hand. The participants list. And everyone should be able to unmute themselves and join the conversation. I think part of the problem is. A lot of people may not realize to join zoom to actually join the conversation. I have to click the link. On the page below the streaming video. Which makes a little more difficult. It's like me at someone. Someone else joined. So maybe. Someone's got operator acceptance. So it seems like people are using. Ironic. Yay. What does not work well. Someone put user acceptance. Does anyone want to share that thought. Both of this was me. So I don't know if you want to like turn this into like on a therapy session once more. We need to schedule a separate therapy session. Yeah. Maybe. Yeah. What I tried to say about this is like. Like for the people operating the infrastructure. Like preparing hardware handing out hardware. Dealing with retirement. The acceptance is good. Maybe I just scare them. That's also possible. But the acceptance is good. It's like integrated with various tools and like workflows. So there the acceptance is high. Like, I mean, as I like online and talks, we use this for for benchmarking or for burn in or for like health checking for retirement for the handing, handing out resources to users. What does not well work that well is that the users accept this way of how we do this. I mean, it's not necessarily an issue with ironic or ironic concept solve. It's just share to share the experience because users, for users, this is kind of an additional layer between them and the hardware. Like something where they lose a little bit of control because our use case is maybe different from others. It's not like we have a like use, you know, like a big cloud and people come and go with their physical instances. It's more like an abstraction layer that allows us to do proper accounting. It allows us to have proper workflows and to retire notes properly to move notes from one user to another. And but for some users, they just basically order hardware. It arrives with us. It has to go to an ironic and they have got to go to ironic or noble in that case or open second or to get it. So for them, it's just an additional layer. So the main thing I'm struggling with is like having users accept this as a layer, especially if something goes wrong, like, I don't know. They try to create an instance. It fails. Note goes into clean failed. They retry. They get no valid host. They can't do anything. They have to open a ticket and ask what's wrong with my machine. While if they had like full control, they would like probably figure out themselves. So again, it's not necessarily something that ironic and can serve. So for me, it's just something that I struggled with. Like sell this to users. While on the other side, like everyone like on the hardware side that has to do all the acceptance, they can't do it. They can't do it. They can't do it. They can't do it. They can't burn in and so on. It's much easier because I provide them with a framework with, they can just hook in the various things they want to do into the client cleaning framework, for instance. And yeah, they are happy about someone else taking care of this rather than maintaining. Their own tools in order to do the burning or something. And also to use the same tool for everything. So that's kind of the, like difference between behind the scenes and like the user side. So that's kind of the difference. Is there anything that could be done to. Well, I guess two questions. One, have you considered granting users API access directly to ironic? I'm taking the look is no. But maybe something to explore moving forward since the, there's been improved support for those kinds of functions. Right. They may find. Yeah. I was going to say they may find using, not using it for some things using Nova for some things as an easier interaction. Grant, if you lose the accounting that comes out of Nova. Can we read only just for debugging? This is true. Provide information at least. He said that the users are happy, kind of happy that they're using like the same tooling. And the things that a lot of things are taking care of for them. Yeah. Sorry. I'm kind of wondering it. I shut up. Okay. I'm kind of wondering if, if there's any alternative means that they could get to, to claim the hardware and then enable use of another tool they're more comfortable with for provisioning. Maybe that's a path we could enable. Yeah. Yeah. I don't know. I mean, even like read only access or something. If they see a node is in clean failed, that doesn't help them very much. Right. At least they'll have some details on what is going on. And maybe they see last error. Maybe to make some sense to them. At least they won't just come to you saying, oh, something happens. I cannot find anything. Okay. I have no, then clean failed with this error message. It will save you time. It will save them some frustration from not understanding what's going on. Right. Yeah. That should be an option that's available today on. Train, right? You're running train. Yeah. Okay. Yeah. Well, the other thing that this, the other way that this could be done is like we, you know, you know, you know, you know, you know, we anyway have like, for instance, for repairs, we have like a repair team that's like dealing with hardware. They also interact with ironic. And they go through some system that we have. In order, for instance, to set without admin credentials to set maintenance on notes so that it doesn't. Appear in Nova as available resource. So basically what they have is like a, a tool that they can, they can use to perform certain actions on ironic and ironic hosts when they need to intervene without giving them. Full admin credentials, something similar could be done for users, right? They could go somewhere and say like, okay, very my host, what's wrong with it? Trigger. I don't know. A recleaning or something that's in clean failed and only that fails like open a ticket or something like this. So men did go through the end. And so I don't know, I don't know, I don't know, I don't know if it potentially in certain cases, if the data is there and present. That users could have implied rights and against the API, if they're doing a machine. Or they're using machine. But the kind of the whole ownership use model in your environment needs to be established. Right. Yeah, exactly. I don't think we're. Go ahead. This is the other thing. Because when they don't have an instance, they don't have an instance. Yeah, exactly. This is why this is like, I was thinking about this multi-tenancy thing at some point where basically we have a like strong correlation between physical nodes and the tenant, even if there are no instances. So that's not necessarily true because the owner field is intended for just that. And that. It was longer than the actual. We see field. I think that's what it's called. Right. So the, the field indicating who you're leasing it to is, should be the actual user. The owner is who has the powers of. To do whatever they want to do the machine. So when they, they feel angry, want to destroy the universe they can. And that was actually a. Sorry, go ahead. To be a bit more precise, it depends on how you set it up. True. True. It's, it all depends on the policy you set forth because the default policy doesn't allow this. Because. We want to be most restrictive by default. Well, it can't be me. Go ahead. The other issue that this could solve is this like, like recurring issues that I'm raising since, since the beginning is that users have no idea what kind of resources they can use. They need to be told. And if they lose this information, they have no idea. So. I know also this is not like an ironic. Issue per se. But we have projects that have access to, I don't know, 30 or 40 different flavors. And if they don't instantiate their machines right away, they have no idea how many of which they can. They can still instantiate. So if there was some ownership. Where they could basically carry and say like, okay, show me all my machines that are like. Available. Not instantiate yet. That would at least help them. Because otherwise I have to resort to things like I have to, I know, keep project metadata and sync and like, say, okay, you have 20 of these and 30 of these check and then like, see how many instances you have and do the delta or something. So I think. I think some of the more advanced fields that we've added in the past are exactly what you need. That way you can correlate and track and figure out who, what, where, how, and why. What we may be missing as a tiny like mechanism here or there to record some data and wipe out some data. What was that? Which field is that? You said there's a few. The owner field. Okay. Yeah. So that could be an option. I'm not, I don't think. Nova touches that field. Right. The least he feels could be useful. And I also don't remember if the Nova client. Touches that field, but those two fields could be now. Okay. It would be useful. You could actually then track that and correlate it and. This would probably be good enough because what I could do is like the moment I have these hundred nodes for a specific project. Like the project name or whatever. Okay. Rename projects and I have another problem, but let's assume the project name at least is like constant. I could put this there and then have something in this tool that I mentioned. Where they could escalate the privileges in order to get a list of all nodes that they are owner of with the flavor and what with the provisioning state, for instance, in this way they could get an overview. So I mean, so basically they would have a list of, okay, they could get a list of all nodes that they are owner of with the provisioning state. And then they could have a list of all nodes that they have been instantiated. So I should be able to do 50 more. Oh, no, it's only 40 because 10 and clean fade or something like this gives them some. But they could also like then query. Ironic directly. The question is whether this make will like help with the, with the original problem that they just perceived as an additional layer. Now they have to like talk to some other API. In order to find out what they actually have. So I'm not sure. I'm not sure this likes heads with the overall acceptance, but there's other reasons why we have this not necessarily use a convenience. Okay, but I would have a look at the owner fear that's a good, good point. Thanks. Speaking of lack of flavor clarity. I know we've kind of said that's really into the Nova discussion. I know that there has been some discussion in Nova regarding that in the past. Oh, it looks like. John and mentioned. Not managed to finish that yet, but getting closer, I hope. So I guess there's light on the horizon for that. Part of the unified. Hey, how's it going? Hey, not too bad. Not too bad. I just had a cook cake. So I'm feeling better now. So I'm feeling very positive now because I'm full of sugar. And no, I think we have got funding that I'm hoping means I can go back and work on that. We've got a lot of agreement on unified limits. The basic idea is that anything that is in placement. You can have a quarter on it. When you're starting the instance. So. Custom resource class being a good example of that. Whether we influence it first pass through like that is still under discussion, but certainly it's heading that way. The main, the main. Okay. The main or every time this comes up, I raised the point that users have no idea where the resource class is, right? Or resource. The users don't see it though. It depends on what you mean here. No, in order to solve the problem, if we say like this quota on resource classes, how is that exposed to users? What do users see? Yeah, that's fine. They see crazy resource classes. Well, maybe it's baby steps. First have implemented and then provide an API. There is a, I mean, one of the big. Open questions is how do we actually. Expose that properly? Because we keep. Whether we should just expose. Whether you should expose placement, a placement API that could tell you that kind of thing. Because, you know, what I would like, I mean, in an ideal world is something very similar to what. Cinder Manila do where you have like per type quotas and the user can actually see how many of this type. Can I still create how much space do I still have of this specific share type or volume type? Right. I was thinking the exact same thing. What. Has always been useful for me is to be able to load a page or load something and go. Oh, I can create 10 more instances, but I need to create 20. And that way I know that way. I'm not hitting something in advance. That way I can plan and act accordingly. So some three, I would be useful on placement. What I always kept thinking about is. Again, lunch, but pizza slicing, because in the Nova world, obviously things are more divisible. So usually what you kind of want to say is you can have four slices, you know, you could have to, you can have four, four slices individually or one big pizza or four slices, which is why we always get down a rat hole with the flavor per flavor thing. So the unified limits. And I guess. I'm guessing I'm saying plus one to the baby steps. Right. If we get to that point, we could. Without trying to make this up on the fly rate. I have wondered if we should have on those resource classes like descriptions. If there is a better, a better way of exposing that to the end user. But maybe in the end it's up to. There's something in Nova that should be able to say given your quota, how many of this flavor can you have, right? That's a reasonable question to ask. Is a reasonable question to ask something. Not necessarily another API, some client. You should be able to make that call, right? Programmatically. Exactly. It would help tremendously. And if you think about it, users could actually have their monitoring systems also monitor that. And if the number suddenly changes. Unexpectedly, then that could be an alarm and that could mean they've been compromised or someone's doing something. Yeah, no, that's true. Yeah, it's a useful thing to monitor, isn't it? Very useful. The current plan is to put this on the client side, just because we can't agree who the hell should own it. But again, it's baby steps. Like if we can prove that these four things need to happen for this answer to come out, then we can talk about who should own that API. I think in this case it's Nova, but I don't think we want to own it. So if we create the client, we can create the use case and decide whether it's a good idea or not. It's the whole proxy API thing. It's just a nightmare on the server side. What are your thoughts in terms of when this might actually appear? Do you think it'll be Wallaby or Zed or beyond? I always keep saying the next release and it never happens. I generally do hope it would be Wallaby. Okay, but don't count on it. Yeah. Okay. I mean, we have something hacked. We've had it hacked and I want it, you know, we're still on Alcada and I want to upgrade and change how we do quotas. But it's not available. So we're going to have to keep our hack. Yeah. I think that would be a fair way to, a fair summary of the current state. The, the honesty, honesty of the situation is that we do keep, when we get into reviewing, get into the detail, we do keep on covering things that are completely landmines. I'm kind of hopeful we've got most of them now, but I was at the beginning, right? It's just an honest take on the way we're at. Yeah. So it's, it's good to know. Let's be realistic about it. I suppose to, to hopeful. Yeah. I'm in it. The plan is that it would go in an experimental phase as kind of as an alternative quota driver. So you opt out of DB quotas and opt into unified limits. The bit that's probably the messiest is the transition tooling around that. Just because making that work with, you know, whatever you call the upgrades that don't do all the steps, but do. Yeah. That's a bit tricky. So we, of course, we have limited time. We should probably move on to. Next topic. Any ambitious features that anyone would like to see. Someone has suggested form and cop slash cop or as a back end. I'm sorry. I interrupted for a second, please. I'm sure we asked everyone here in the chat that they're had the chance to talk about their pain points. We've been asking for a while and. I can maybe raise one that is actually both topics. Which is. Okay. So speaking of snapshots there. Is a spec that was written by typing. It's up in review. In ironic specs. We. Urgently need to review it and provide feedback. And I think that would help. And at least it's the first step forward. Just like raid support. We have to make an initial step forward. It's a little bit more nebulous and also kind of risky because. If we're going to try and capture everything that could be attached. That could be quite a bit. And of course we lose memory state and all that. Unless. Unless we've got a micro hypervisor to talk to. So in terms of like parity with a VM. The VMs wouldn't capture the ephemeral discs. Now. That's a. Thorny a problem in ironic to be fair, but. Yeah. Can ironic actually deploy on. Disks that are not labeled as the root disk. Can you deploy. Yeah, I'm trying to see why, why would the snapshot operation. If the deployment operation actually just uses one disk, which is the root disk. Well, I think right now it would use what it, what it's determined to be the root desk or to pierce be the root desk. But if you were to expand that out to be all, all storage discs. That may be attached. Then it becomes much thornier problem as John was kind of pointing out. Especially if you have a node with. A hundred terabytes of storage locally. And then all of a sudden you're having to load upload a hundred terabytes for the data into a cloud. And compress that. It could take a very long time. Yeah. One petabyte. Please. Don't know. Yes. Yes. I'm afraid. Same problem with VMs, but. The only other thing I was going to say is that the whole. From an over API perspective. It's not meant to reboot. I guess it has to. Right. If the general case. Yeah. And I think maybe what the. What we could potentially do is. Make it an opt-in only feature. And if it's not explicitly enabled. That way the operator basic, at least the operator can go. It's like, well. Or we could make it such that you try it once on Nova. It fails horribly returning an error message. And second time I'll work. Or something along those lines. It might be navigatable. Yeah, it's tricky. Yeah. I mean it. It's because there's not a full hypervisor there. It's almost like I'm. Doing evil thoughts about the shell feature now. I'm going to shut up before I cause a problem. It's just because shelve off loaded is technically the state that you'd be in. But I don't think that's a. I don't think you want to do it that way. Yeah. That's a good point in question. I think we've reached that discussion with Nova in the past. Yeah. I can never remember what we come up with. I don't think we did. I think it was basically like. It should probably still be a snapshot, but basically functionality is like shelf. Yeah. It comes back. Well, yeah. You've got the choice to not do that, I suppose, but yeah. True. True. It'd be weird. I guess. Shall we move on to. Any ambitious features. Someone has proposed. Use of form and slash cobbler as a backend. There's a little bit discussion need to pad. I'm guessing by the lack of comments, but maybe the group's not really interested in that. Okay. The idea. I mean, why would people suggest that as the option? I don't know. And I think it's because they're the pain point. It is not strong enough for them to switch away from their existing tooling. And. Okay. So it's like, right API to existing. Yeah. Yeah. And it's. It's the, the mechanism workflow is. There's such a dramatic difference there. Interaction. It's. Not really usable. But a migration slash conversion guy would be useful and we should write something. Anyway, next topic that's on the list is something never, you never dared ask about ironic. Yeah. Yeah. Yeah. Of course we, someone actually put. Isn't it ironic? It's called ironic. Yes, it is. Intentionally. So this conquering the world thingy. When. Wait. Wait. Wait. When are we conquering the world? When are we conquering the world? Uh, Next Friday. I think works for me. Okay. I may have some bouldering in the morning, but that's way too early for you anyway. Yeah. I, it's actually kind of funny for you to mention that just because the number of. Of cases where ironic is being used as explosion last year. And it's really a testament to everyone's work in this project over the years. By the fact that it's. It's a great way to look at things that are, that are being leveraged and used in what, in some ways we never expected some ways that. Are excellent and great at adaptations and. Others just kind of get the job done. So. Go ahead. Sorry. Some of the things we've been discussing involved Nova. And I know we've talked in the past about how we have more users. Wanting sent on ironic. So how was that working out? that are specific to standalone? Yeah, the pain points really seem to be mostly Nova based interaction because I guess two reasons. One, it's incredibly painful for us to add anything into Nova or historically it's because the Nova team culture and the hurdles we have to jump through to get anything merged. So that's one thing. And also there's, that culture also tends to wanna try and design a thing to be cloud-like and bare metal is just not cloud-like in many cases. So there's just a point which we kind of stopped adding things into Nova to interact with the bare metal. Also most of the changes have been more operator focused. So I think it's probably worth revisiting and going back to, I'm just not sure what it will actually yield in terms of fruitful results. Beyond the needful that it can be identified such as understanding what's in use or snapshot at some point. Right, that makes sense. So another item, ironic as an interface layer for multi-tenant cloud computing versus ironic as general purpose tool in an enterprise is it both for good? Which direction should it, should slash will it take? So it's bottom of the ether pad. Do you meet your notes? He's not an operator. But he thinks that we need a more consumer friendly way to use the API. Yeah, I'm trying to guess what would it take to make it more friendly specific as a general purpose tool. Probably an easier to use API, installing tooling, which is something we are working on. So it's not like a new requirement just to emphasize that's sewer priority. It's a simple of GUI what's actually needed. Maybe if we had somebody to work on existing GUI. Yeah, I mean, yeah, all those things, but just from a, if you'd want to drive your infrastructure, so lots of the customers we work with are kind of wanting to transition to, you know, infrastructure as a code and that kind of thing. So in that adopting of those different pieces, like they don't see that hurdle as a big one because they're sort of really bought into that. But for the more general use case of the just give me a thing, we're generally not a very friendly user interface because we've exposed quite a lot of the features. And there's some interesting presentations from some people at one of the UK Research Council's Jasmine, like they wrote like a little shim that just does the very basic give me a VM thing with a flavor option. That's a really good point because there's a disconnect in between the development community in OpenStack and the operator and user communities in that a lot of people do actually want to just give me a thing. But some of the developers seem to want you to have to define all these options and pass all these parameters. And oh, you what? The special flags that we had, you've passed these extra arguments since just that's not user friendly. That's at all. I have recently been sticking several pins in my eyes when trying to do anything to do with like SRV, for example. I don't mean to throw a shoe at a particular feature but that is horrendous for, no, no, but you create the nick with all of these special properties. Yeah. Sorry, my ear's just burnt when you said the SR-IOV and I still have PTSD. Yeah, I can believe it. It's more just as a backup to what Julia said, right? Can I have an orange, a banana or an apple, please? Is the kind of question that people want to be asked? And that's what flavors are meant to be for, right? It's just, sometimes we get lost in the weeds, but that is UX, not API, as I should say. So I have brought this up before, but I usually get, say, a tumbleweed in response, but I've used mass a couple of times. Now, I've never actually set the thing up, so I don't know what it's like from that perspective, but in terms of actually using it, it's a lot nicer than ironic in terms of something that you point and click at. If you want something that you're more advanced to use it and you want to start driving things programmatically, then, yeah, ironic all the way. But for someone who wants to manage, say, a hundred servers or less, being able to do that in a web UI wins. It's just really quite easy to do. I know people have had their own complaints about it, but, yeah, what I've seen of it, it's been pretty slick. It's a really good point. We focused on a scale that is huge and we kind of left the enterprise out of the equation of any sensors. And at that scale, if you're not doing infrastructure as code, it's a lot easier to do. If you're not doing infrastructure as code, it's a lot more appealing just to have point, click, give me my banana or give me my orange or apple today, not have to a bunch of different work. We did have some work on an initial UI, but unfortunately, that all ceased with a departure of a certain company that shall not be named. So I'm the perpetrator as that Foreman-Cobbler question there. And Ironic tends to provide cloud APIs which are nice for if your workloads are running either in virtual bare metal. But in our case, we've got a significant amount of tech already sunk into things like Kickstart and old school bare metal provisioning. So I wonder if there's a... I mean, it's a layer on top of actually on the image, literally, that wouldn't be needed here. Oh, that's a different question. Do you mean you don't want to use cloud in it, basically? It's literally cloud in it. Cloud in it is beautiful in everything, but it's not something that was developed in 2000. So... You're actually raising a really good point. And there is someone working on Kickstart support. So that could be helpful. Like explicit support to be able to say, here's my Kickstart file. Go make the thing happen. Yeah, I mean, a lot of our workflow is sort of encapsulated in somebody hands you a machine that you give the Kickstart URL to, and then the machine exists afterwards. Does that give any plans? What do you mean by a Kickstart URL? So you can give... When you boot a machine on a CentOS CD, you can give a kernel parameter with a URL. But what does that URL do? Sorry. It's a Kickstart file, which is basically the CentOS installer Anaconda. Oh, yeah. Or so that... It would be good if you could take a look at that Kickstart Anaconda driver spec to see if that would help. Or if there's something missing there that... Very likely that is exactly what I'm missing. So let me grab that from review. So you have the link... Let's... You know where to find it. So I think we're almost out of time. There is an additional question regarding RAID, which might be crazy. I don't know, which is layering RAID types. I know we've had a couple people come into our scene and go, how do I do this? I want to do this. But I don't think we've had a huge demand for it. Any thoughts? I guess not. Well, any ending thoughts before we all wrap up and go to our next session? Well, thank you, everyone. Thank you. Have a wonderful day. Thank you. Have a good one. Thanks. Bye.