 new release. I'm going to go back here and go to the Google groups and yeah so there was a new stable release that was published a few days ago and according to Vadim this release should have all the necessary fixes to install upgrade on vSphere and this was just happened on the 12th so if you haven't tested it yet I'm not holding you accountable because today is just a few days after that and I think that was a weekend so weekend release but I think we have have fixed the problem good run that you might have done so it might be worth trying again and there was one other note here I fixed the issue I had and then I'll do a the only kind of do about skin she was a little so I'm gonna say at least on the upgrade side John you the installation issues seem to be so John you I'm gonna switch computers yeah I would ask if you could that would be great yeah I'm gonna switch computers because I'm not sure what's going on here hold on I'll dial back in yeah the internet's not been great today for some a few people so it's it's not happy it wants to go on vacation there's Vadim hey Vadim before you start talking we're gonna let John Fortin reboot his machine and come in on another machine but good one can you tell us a little bit about your configuration on vSphere we've heard from Bruce and John previously so I know so my my setup is user provided infrastructure and it's all in one we learn behind an F5 load balancer on vSphere and net up I don't as a storage vacant the valide is just joined us to so hey will lead I've unmuted everybody hi hi everyone you should be on you should be able to unmute and turn your video on if you are if you so desire we're just getting started we have one person I had to reboot and he should be back shortly will lead can you tell us you I know lead did the the vSphere okd deployment demo a while back and do you have are you running okd in production or as a POC somewhere will lead and can you tell us a little bit about your vSphere deployment we are running open shift on vSphere at work and I wanted to run okd on test environment however one of the supervisor doesn't want to yet so we are actually running just open shift on the production so the vSphere on okd environment is on on a on a on a home server on the R620 other than that I didn't manage to get it to get it approved at work yet okay now and that that's alright and here comes John Fortin back in and John let's try and see if we can hear you again yeah how's that not so much better so John yeah something's going on with my computer my regular one so okay I'm on my laptop see perfect so John we've just gone around around the table here if you could tell us at Market America what your okd deployment looks like today on vSphere so in regards to the 4.5 4.6 or in general so we're running 3.10 and 3.11 in production but we're looking at you know 4.5 4.6 you know 4.X and you know to migrate first quarter so did a successful 4.5 install you know a couple of glitches that Vadim helped me through but the 4.6 was problematic and over the last couple of weeks I think we fixed all of those I did successful upgrades and fresh installs over the weekend with the new release from that dirt day was that the 12th whatever whatever day the 12th was and that was that was really successful the only issue and I hate to call it an issue because I don't know the cause actually two issues one is VMs not turning on by themselves but that just turn them on we don't know if that's a infrastructure issue on my end or if there's something else going on and we're sort of discussing that but also the network stack seems to get really busy after a 4.5 to 4.6 install but other than that everything I've seen so far has been very very clean and I see that Jamie just just joined us from you Mish and so Jamie I'm going to put you on the spot if you can unmute yourself we're making everybody go around and talk through what their vSphere OKD deployment looks like at the moment so if you're sure so I am at the moment in the process of just about to rebuild my 4.6 with the latest release that came out the other day I have not yet and basically I got burned by the hostname issue last time I attempted to build so I had an incomplete build so my goal this time will be a complete build awesome awesome so thanks and that that got us up to speed I think on where everybody's at Vadim if I think Gudrun is I haven't seen Gudrun on this call before but if Vadim if you could give an update on the the latest release and what that issue was in case she's seen that and then maybe we can and I'll let Mike and run a round table we could talk about other issues you might have or wishlist links feedback sure so when we first release a good e46 this has been a bunch of changes introduced by Fedora 33 content most of them have been related to DNS resolution however by now all issues I think have been fixed so you can now successfully install and upgrade as John has mentioned today there has been quite a lot of changes in for six though as well which is a busy networking from OVN that's basically an open OCP buck we filed this in Brazil waiting for network folks to pick it up another important problem we're hitting right now is that you cannot mirror the Koi images the key images back to your machine so you cannot do an air gift install this is caused by again for six upgraded by a for six change but it has hit us silently when we when our build farm on CI has been updated for six as well so in order to fix that we need a release which has this bug fix and also all images for built probably not gonna happen in the next couple of weeks because it's quite a lot of effort so we that's the first thing we'll fix in in the new year unfortunately is that specific is that specific to okd or is that an OCP OCP wide issue it's an OCP wide issue with hits okd most but you still can release OCP because internally we rebuild all components using older clusters so that's time they are lucky but again it's a pretty huge issue she's gonna take some time to actually get it fixed another medium-sized let's say issues that our cluster signature keys have been updated behind our backs I have a fix to push it but the next couple of releases next two table releases probably would have to be updated without the signature verification we have steps how to do this manually but in cluster verification is messed up because it uses official right ahead keys instead of the okd specific keys but these are the largest problem I've seen so far there are a couple of unexplained problems like open shift API server hangs up on upgrade but we don't have a clear reproducer and it's also affecting OCP meaning it would get a bit more attention I think that's all I have for the status I don't know if any these year specific problems John has hit a couple but we don't we can't reproduce them on our CI so really hard to find out what's wrong there so this this meeting is for vSphere people so we wanted to hear from you about any issues or feedback or things that you are running into so I would like to open open the floor and the only thing that I was thinking since we have well eat here is is asking the question is the demo that well eat did deploying vSphere that we recorded for the okd deployment marathon does that change drastically in any way shape or form in other words should we redo a deployment video for vSphere to help people out and is it possible to create a vSphere deployment recipe and that's the one thing that that I was curious about from other folks here so I have a UPI recipe that's going up I think I shared it with you last week it's almost done will be done in the next couple days IPI it probably would be helpful still to have someone do an IPI version of the vSphere install video so I can do a video and I can do the recipe for UPI but I think IPI would be helpful as well and we'll lead did you do IPI when when you were deploying I can't remember no I didn't do IPI the IPI failed we couldn't get the IPI working but it was the it was at that time it was just the IPI was just working so maybe now with 4.6 I need to give it a try okay well if you can do that in the next week or so and give us your feedback I I promise not to go on holidays until the 20th of December here and and you post it to the working group that would be gray and that would be that would be very helpful and I'm just adding the issues here I mean that for me that that's those those are the kinds of things that that I worry about is sort of the community lead is to make sure that the instructions are there so someone coming in like Good Render or elsewise has has the tools and the documentation to do what they need to do and maybe a blog post or something like that on on this new release I think now it's safe to do Vadim a blog post about the 4.6 or do you should I wait until those images are already yeah I think the mirroring and signatures problems are pretty significant okay so wait I don't mind waiting because basically it's more work for me that's I'm fine with that I have a couple questions about the IPI recipe stuff that we just brought up so like I mean in general like the IPI installs should be fairly boring you know kind of start the command line and let it go in terms of creating like a recipe or materials for people what you know what what is kind of the expectation here do you want to document how vSphere is configured and then you run IPI on top of it or what what are we kind of looking for there well I put that to the group is like what is most useful an IPI generic one or a vSphere generic one with IPI layered on top that's okay that's that's that's really up to the group here I'm not the arbitrary or trader of what should be in a recipe um well IPI itself is relatively easy the the interesting parts are when you set up the config file and if you want to do something like change the default keep you in memory and stuff and the documentation online is fairly good but you know somebody going through and actually doing that you know as a video would you know would probably be helpful for people that are doing it for the first time you're talking about you know the DNS requirements the fact that you have to have you know the API and the the I'm sorry my mind just went blank the wildcard DNS you know first up for the ingress you're talking about a little bit because some people are going to like I don't know what that means so I think that would be one on from a general point of view from anybody using you know Sector out for whatever but for vSphere in particular from an IPI side the user provision stuff is a it's a whole different animal because there's a lot of different flavors of that yeah and I you know I the expectation for UPI I think is really different you know we could make a video about it but it's going to be different for everybody I think you know something like an IPI video and you know I'm just thinking about this from the issues that we see you know hearing what users are saying and whatnot like I think there'd be a lot of value in being able to first lay out what the expectation for an IPI install is like this is what we expect from vSphere and then the second part would kind of get into some of the things that John's talking about like not just running the installer but what is the prep work that you need to do before you run the installer like how do you set up whatever configuration on the outside allows you to point to your vSphere cluster because once all that's set up you know the installer should set up the load balancers for you and everything like every you know once you've got it to the cluster the installer should do all that work for you so I would think having a good standard set of known good setups for vSphere that people could run IPI on that's a really big first step from where I'm sitting yeah that makes sense yeah I think you need to clearly describe the permissions required too because that was one of the reasons why I didn't do IPI like our you know like my server is one of a whole bunch all running vSphere and my admins don't want me to touch it really at that level and if I could you know if I could have given them a set of well these are the permissions I need then I might have gone that route although still might not have succeeded those sort of UPIs the only way yeah but those permissions are in the documentation exactly what's needed in the in the okd documentation it says exactly what yeah last last I looked was in July and there were some things there but it wasn't clear that they were comprehensive so the permission it could be just my could be just me the permissions is mostly for the for the storage integration yes for the vSphere storage integration is that correct well then being able to create VMs and all that other kind of stuff and you know so you have to have you know particular pieces of you know authorization to be able to do that easiest is just to have an admin ID but you know if you want to get a user that's given all those pieces it should work it worked for me but I don't know if my admins gave me extra stuff I gave them what I needed they gave me an ID yeah I mean basically you need to write to the data store to be able to copy like there's a whole series of them and just a simple table I think would be helpful to to show people that yeah so my reference for this is usually the VMware like wiki for the vSAN CSI integration and it has changed since July I remember because another another colleague had an issue and he asked me how did he do the permissions and when I looked it up it has it has changed since then it's getting easier so basically there was some vagueness before but now it's actually I think it's a little bit easier so I guess yes a video for this is maybe it takes like maybe two five minutes just how to do the permissions and roles and how to bind them together I think I think what the context of a recipe though is to have the video but then to also have all of the links all of these things that people are talking about in some in the recipe so that that's I love the videos because they're they're great and they're useful and that's what but if you don't have the the documentation have difficulty tracing down some of these bits so just to be you can't you can't cut and paste from a video yeah I'm always trying to figure out how I can annotate my videos I just haven't mastered that part yet so yeah I'm learning Camtasia Camtasia is pretty good but it's sort of the MS paint of video editing but I think if if Jamie if you can create some you know a short the short video and the recipe to go whether I will happily make the and make a PR request to the okd.io site I and wherever you host it if you look at the couple of examples that Charo did already for recipes they're they're not hosted on in the open shift repo they're hosted in his personal repo I'm happy to make a umish or your own personal repo as well yeah it's an it's an actually uh I think it's in a umish repo but I can put it in a umish repo either way yeah either either way um so other other than that um so I think you did in John I might meet you a little bit because you've got some noise going on in the background so yeah like yeah you get you're living a real life um you've got real people in your house yeah my grandson's in class and he's in the same office yeah that's great okay um so uh if are there other things that are stopping you I mean a well-eaten um and this probably isn't a v sphere specific thing but you mentioned that your management didn't want to deploy okd yet is that just because it's um hasn't been mature and stable for a while or because it's the open source stuff or what were the were were there any things that we can help with that they are interested on the on the operators and I mentioned that that I cannot guarantee that all operators that will be uh one-to-one availability and okd as their open shift I'm not sure about the open data hub because I saw I saw that they have done some work on the offline disconnected operator availability but I haven't tested it yet also and I'm not sure if it's available in okd or not I haven't I haven't played with okd for the last two three months yeah so to answer this point there are two types of operators in okd which you can install so the first one is shipped right in our release payload and release payload for okd is exactly the same as it is for ocp except some tweaks we need to do for instance machine config operator is slightly tweaked to work with fedora croess and samples operator is tweaked to ship uh community samples so that's the first part these are identical but some parts are slightly different because fedora croess is slightly different than arcos and then there is optional operators lied by olm and here is the difference because okd is shipping with community operators enabled only and ocp is shipping with both sources like redhead and community available only however in okd if you have a valid full secret you can enable both of them we have this documented in in FAQ and well yeah that's the that's the difference these two sources are controlled by you if you don't want to use a full secret from redhead you will have just the community operators if you know what you're doing and you have a valid full secret you can enable both sources and there is no secret source which we hide somewhere in in ocp that's that's a story i would be more concerned about questions of support but there is not much we can do on okd side it's community supported then it's just not going to change ever other than that there shouldn't be any difference if there is it's about which we're about to fix like bare metal support for instance so when i use the redhead full secret i think i'm entitled only for two months or i'm entitled for more um so the answer is you when you create an account you will be given a temporary subscription two months but if you change it to self support later on you can still use it and you it doesn't break any kind of a clause far as you know again officially it needs to go through legal but if you're using it for your personal sources that lino going to bother anyone and if you're using it for testing it's definitely not going to be a threat but the problem is that uh the gray area here is that if you would run okd with red heads full secret and red heads um official supported operators on production there's probably a gray area you would need to clarify that with i don't know with legal or somebody else um i don't think anybody has tried that so far now the target is test and deficit i guess it should be fine correct yeah yeah it shouldn't be a problem so i have a question and i haven't i haven't been asked this by management yet but i'm uh i'm almost expecting it um with with the announcement that um redhead is made with centos and how they're changing centos um and you know the fact that you know there are a lot of people who are using centos in production and stuff like that is there a concern or should there be a concern that the same type of thing that could happen with okd um you know versus your regular open shift as centos versus rel i think um we we're talking apples and oranges that's what i think but i i just kind of want to hear it yeah you can hear it from me it may be mandarin oranges even um it's very it's a very different um relationship between the two products than rel is to centos and centos stream is to rel might be closer to what okd is to open shift in some ways we always talk about okd as being the sibling sibling open source upstream not quite upstream to ocp so it is the the issue for us i think is that okd lags behind ocp and so if and we've had this conversation at the last okd working group if you wanted to contribute to open shift in okd you're always kind of contributing to the prior release or trying that that's more of our our issue rather than the the centos um relationship to rel um issue it's it's apples and oranges it's not the same thing you you you'll always have that and go ahead jamie so i one of the things that i've been doing is i've been actually using it as an opportunity to promote fedora core os and promote okd um basically folks are saying well we've been using this in university setting for research um machines for cluster nodes etc and so what i've been doing is sort of pivoting on that and saying well if you have fedora core os you can run these things in containers and get that same sort of um isolated functionality that researchers like for particular libraries etc and if you want to go the cluster route instead of doing hbc stuff like centos nodes you could actually do an okd cluster with fedora core os nodes and run matlab jobs run like whatever it is that you want to do actually in containers so i've actually been taking it as an opportunity and i'll uh forward to the community email list uh email that i sent out about this last week that maybe can give folks some talking points that would be great that might be fodder fodder for my blog post meet you co-author that blog post um with me so i think that that's that is the position that i like about okd and a lot of mine is 10 is 1 all right i'm gonna mute john again uh or let there is also another significant difference that uh in okd everything is controlled by the cluster itself and if you make an at city snapshot you can revert back to whatever state you have and using rpmo3 you can revert your post back to whatever state you have so this is why we're not really bothering with publishing versions of kernel and exact list of rpms we're shipping in every okd release if you're hitting a problem you probably should revert and we'll look into that later on that has hit us a lot during four to five to four to six upgrade but that's how it works we don't have to bother with kernel api and stability things which we do we have ci which ensures that and if you're hitting a problem during upgrade one of your masters may go broken but we have ways how to revert it back to stable position report a bug and move on it's a little bit different than a single centOS machine upgrading from one version to the other so we're much more safer in this regard when we're having a full cluster agreed i'm looking forward to that because upgrades of 3 10 and 3 11 yeah i'm also looking forward to that oh go ahead to answer no no i i'm just was trying to move the conversation forward so go for it okay so uh i'm throwing a uh bug for fedora coro west in here because it it it seems to affect check that out in the chat it seems to affect uh vSphere folks um i've had it happen to me a couple times when booting up nodes so if you run into this folks when you're doing ipi or upi let me know this is the io control uh issue that pops up and then sometimes if you reboot the node it'll work fine or not so this is going to be a showstopper for folks doing ipi or upi uh vSphere installs a vocad so uh i'm gonna chime in on on this and if other folks run across it definitely chime in and let's see if we can nail down what it is that's happening hmm i haven't seen that happens to me you know because i'm rebuilding my one test cluster a lot it'll happen maybe on one node um each time i do an okd install uh and so it happens uh on that reboot uh basically um after the update so that's on vSphere yeah and if you reboot again uh it tends to be okay i've had one where i had to reboot like three times and then the node finally booted properly so i'll do some debugging next time it happens um but i thought it was good that someone else had the same experience and so i was kind of happy to see this my day job is watching ocp upgrades and we're seeing it's the scariest bug of all we're seeing this symptom quite often on rel as well on across all of the platforms the problem is that there is no way to extract any useful information from it it's just kernel uh breaking for whatever reason um i think the smartest way would be i don't think we can fetch any additional information from my provider right at least tracking the vSphere terminal relationship there might be i can look into that there might be but i don't know i haven't gone that deep into uh but i agree let's start with tracking it in fedora cross and see where it takes us because i've seen it happening across all the platforms including bare metal uh some system du the fhu which can happen due to a variety of hardware or hypervisor um i'm not sure if it's a great idea to have one umbrella issue for everything but um that's at least a start at least something we could point other people and have a workaround that you just need to reboot it's it's pretty bad but it's not like end of the world yeah it's good if we just put something up and maybe the faqs or something like that that just says hey if this happens you know reboot and you should be okay just because you don't want someone to think that they're dead in the water i'm in okd installed because of that yeah probably that's a good idea we should have a list of known issues um i'm trying to keep a list of banned issues in okd updated but uh there's only three and apparently okd is becoming more and more widespread so uh we have many many more um yeah having a list of known bugs which people actively hit every day probably a better idea than me trying to to find out how widespread each bug we get so let's start with that yeah i think that the tough part is we need like we need some sort of like living list right you know the code that or a list that can kind of live with each versioned release or whatever can we use the the projects page um the community projects as a way to track those i mean so that we're not at least have a link to them i can create in if that's what what works best for you videum um to see them and to review them i don't want to create more issues i just want to have a in my mind i just want to have a list of them that we have links to elsewhere i think a better idea would be a markdown list as a file in the repo so that we could control it we could suggest changes and uh it will be visible to anybody visiting in hub open shift okd if you create an empty one um i will happily promote that or and promote it on the the mailing list i'll happily promote that theory at the next okd working group and in between um if that works for folks i just like my community page an addiction to it um other things i know i'm going to put a link from the community page to the file in the repo so that's in that first column of the community page i would just put that in um and i might put a column in that's v sphere specific to that community page so that we can link these notes if this meeting is going to be reoccurring um i will link to the the working group notes and i've put all the notes from today into the main okd working group meeting so this is not so people will have visibility of it um as a guest has just joined us but that could be anybody that's james hi james um i will lead you said one thing in um in passing really quickly that cut my attention um odh open data hub on okd and i do not know if we've tested it um on the deployment of open data hub on okd yet so i will um see if i can find someone in the odh team to talk about that um in an upcoming uh working group or on the mailing list at least um there is a gchat internal channel for that so someone i i would guess that someone has already done it but i'm not sure whether they've done it on v sphere or what the configuration is so i'll try and hook you up with that and maybe um even do some sort of briefing on that um because that would be a good thing on january 28th i'm doing a um open shift commons gathering on um data science um and the open data hub folks will be there um in in in mass shall we say um so if you see that fly through your your radar that's a good place also to ask that question come in to see see what they're doing and i'll try and get them to address that in their um their slide deck on their demos as well maybe i could even make them demo it on okd though that probably um is a lot to ask as the christmas holidays are coming up and i bet their demos are all fully baked already and they're off baking gingerbread instead um yeah i mean the odh should run on like 4.5 or 4.6 okd like i i don't i follow you know i used to work on that stuff and now i kind of follow it at arms reach um but as long as the cluster is 4.5 or 4.6 i would imagine the odh stuff would work but i i don't think they have a deep dependency on something coming out of the operator that that would be the only issue i i don't know i i can't imagine and i there's something tickling in the back of my brain that that somebody's already done a demo of odh on okd and um i have to look through the three or four hundred videos that we've recorded this past year and see if somebody did that already and get that to you if that helps move you forward i know james is you just get that that would help i think that would be good one the other um trying to think the other thing um the other people um that i'm going to try and get in here but they're not running on vSphere that's i was thinking about CERN they're on open stack that's a different deployment i don't think it matters the infrastructure what mattered at that time it was the latest version i believe was 0.7 and until 0.7 the open data hub was not supporting offline restricted network installation okay so when you have when you put basically your operator open open data hub you will not see you will not any actually AI ML operators at all there but i saw there was a github issue and i saw that they started working on it the target was 0.11 so i'm not sure which version they are right now well if you can dig up the um the github issue and throw it in the chat i will track it down and and james if you wanted james castle just join so if kept if you have a vSphere specific james i know you're i'm pretty sure you're a red hatter um you're probably just listening in but we're making everybody introduce themselves so if you want to do so or if you don't have a microphone nope he doesn't want to introduce himself that's okay introverts are welcome um and so there's so um we've got about yeah 12 minutes left are there other vSphere or even generic 4.6 questions you guys are having yeah can i ask a couple of questions please then yeah so um just because i think we um like i i did uh i guess yesterday uh tee up the uh upgrade from 4.5 to 4.6 and uh uh went through actually i did it from the console and i didn't have any trouble with the uh i didn't have to force it or anything it just you know started right away no problem uh and it went uh you know relatively through updated anything the machine config ones seem to always be at the end and then it will go back and do a lot of networking things at the same time uh and uh the uh everything was going smoothly until the very end where um one of the uh worker nodes uh uh was uh you know having massive cpu the point of where i got an alarm on vSphere on it uh and uh the net result was that uh that one worker node uh failed the upgrade and uh uh uh when i when i looked uh later on it had an ovs configuration service failure uh so that's okay uh it uh then the net result was that uh i lined up with uh three control plane and five compute uh and uh all of them are you know successfully upgraded except for the one compute which is now ready scheduling disabled okay so that seems reasonable uh and then uh but subsequent to that then all then uh my cluster operators had all been uh available but uh at this point the uh authentication console image registry monitoring and api server are all unavailable and so i have no console and uh uh the i can't get into the machine from outside the only thing that uh works at the moment is with the uh the original uh admin key uh and uh i can get into it from uh a machine that's on the cluster and uh i would have i guess hoped that if that uh losing one worker node wouldn't have such a big impact uh especially since it's managed to realize that it shouldn't schedule it you know so i would have thought that everything else would be up and and running fine but that wasn't the case yeah i don't think it's related the worker nodes should not block the upgrade in fact the worker node upgrades happens right after the upgrade is complete um there is an odd issue with open shift api server which is critical i think let me find it yeah so 395 um this one may bring it down a lot of operators and it's usually because open shift api server gets stuck for some reason um i don't think it has a workaround right now but it certainly affects ocp as well so i suggest we could we could start with filing it back to okiri we need mas together which is probably i don't know hopefully you would be able to collect it well i just did collect i just i just did collect one uh and uh i was just looking i was doing it while we were we were talking and uh it you know there were a number of services that weren't available as it was going through but it did gather a lot of stuff it looks like in retrospect so yeah so i have a mas gather cool um if you could have a look into details of open shift api server if it still throws 503 for everything that's probably the bug we're talking about and in the musketeer yeah it should be somewhere in namespaces uh open shift okay i've never looked like that wait wait wait or or you can just tell me we would have a look oh no i'm happy to look uh so in the uh the quay open shift musketeer folder let me just go there i suppose uh and okay so i've got namespaces you said uh the namespace is folder okay and then 5000 of those and then you want the open shift api server one or server operator so so which uh the api server operator we need the api server yeah and then we would have pods and we should have three pods there in in each of those we would have containers called open shift api server and shift api server so eventually we'll reach the logs and the current log is what we need okay so i need to go down to pods and so i got three pods yes api servers so pick one yeah i think it would plague every single one of them okay and so i've got so inside of one of the api server pods and i've got the ic2 okay the api server something yaml fix audit permissions open shift api server open shift api server check in points uh so so i'm in one of the pods where should i go next uh there should be a folder called open shift api server and this should also have a child called open shift api server and deep in one of those there would be a current log which is log of the spot and there's a logs yeah okay great and then current log previous and secure log previous log so you want current log no wait that's all you should be yeah current right um yeah and it ends with something like transport closing okay so tail the second because i i did more and i got zero percent okay so let's see so then transport is closing uh before that handle sub constate change and switching balancer to pick first and then earlier there's transports closing and i'm just looking so what should i look for i think it looks very similar to what we have in the bug 395 unfortunately i don't know what else to collect we need some input from api server team so there is a link to bugzilla which is striking the ocp side of this bug i think we should tropic yeah there's the link um we need to comment there and attach the must gather mentioned that it started happening on four six okay so attach the bugzilla bug yeah well good i put i put most of the links that we've talked about today in the meeting notes too so if you're you're scrambling okay this is so oh okay so this is so developer thread hat that's your work right no not that one yeah okay uh too many passwords too many passwords oh never enough just use the same password everywhere right yeah so it's a list is there anything else that folks want to bring up before we uh disband while he's doing that uh yeah so well i can update the once i figure out how to log into bugzilla again which i hardly ever do uh i can um well i i guess because i'll have to update i'll have to push the must gather somewhere so that i can link it to bugzilla so that's not instantaneous um so at the uh uh now from the other chat what it sound like it might be worth rebooting the worker node just see what happens but it sounds like that is unlikely to change anything so i should just wait or since it's happening on open shift api server it runs on masters so workers are just a symptom of the problem uh right right we could give a master reboot a try but again i haven't seen any work around or any more information to come up with something that that's really a positive bug i was just wondering in terms of work around it do you still have that one worker that's causing you problems like is that is that one worker still in a weird state or whatever yeah it's it's sitting back on the uh july kernel and hasn't you know it's it's not upgraded yeah because i would think you should be able to delete that machine from the machine set and let let the machine api create a new machine for you it you should be okay to delete it it'll drain and evacuate that node for you and then it'll create a new node at the right level so i mean it's not a great work around but it's an easy worker how do you how do you let it recreate it uh like it's not so i'm not you're not creating any machines because this is upi great right so you already have a you're you have a worker node right that's in a bad state right now but the cluster still has that worker node attached to a machine set right like so it still thinks though so you could get the machine object you know from the open shift machine api namespace and then you could delete that machine object and what will happen is the machine api will try to recreate the machine for and it would just create a new one at the same level that the machine set is set for so it would create one that fits the machine set basically right okay all right folks we're we're at the top of the hour um and it sounds like this was was useful um to do um and and so if we want to do another one sometime maybe quarterly or something like that v sphere specific or when the need arises i'm happy to do that especially on the opposite week of the actual working group this fits into my prime slot to host if you have other questions i'll post the video and the links and the notes into the the google group probably later today i've got to catch up on a couple of other videos from working groups so i'm behind i think two now so we'll get that going but if there's anything if this was useful was that useful to everybody to do this um there weren't as many v sphere people as i thought come into the forefront um a lot of the usual suspects so um other than goodrin who i um i don't think i've heard her speak up before so i'm happy to have you here um and we'll we'll do it again um as needed how's that yeah please please keep me in the loop dian yeah to help out yeah that'd be cool and um well i'm going to share this with the the pms a couple of pms maria brocco and um andrew sullivan as well so they can listen into our chatter and hopefully move some some of the other v sphere customers forward too so thanks for your input today yeah i mean if this if this becomes popular i wonder if we might you know we might think about just having like an okd provider on this hour people could just come and ask provider questions or something or bring up their issue i mean we probably don't need one for every provider but like a general session just about provider topics would probably be worthwhile yeah no i think i think you're right um there's there's something to this um so we'll talk about it again at the next okd working group and see if we want to have like a subgroup or something a reoccurring meeting but happy to do so so thanks again everybody for joining and if i don't see you again have a great holiday break and um now i've got my notification after yeah and james welcome to my world times time changes um so i you you should have been me like three weeks ago when it first changed i missed i think two meetings in a row with the wrong time so um but luckily charo chimed in and covered my covered for me i don't feel that anyways um if this becomes a reoccurring one we'll put it in the fedora calendar so thanks everybody um and take care thanks diane thanks thanks for the thanks for the