 It had been a great flight with a great entry and landing and the team was preparing to hand over the vehicle to the KSC ground folks. One of the last orders of business in this process is to transition from internal vehicle cooling to ground cooling. Now in orbit, cooling is accomplished through a combination of the radiators and flash evaporator system. As shuttle equipment heats up, the free-end loops and water loops carry this heat to the flash evaporator and radiators to be dispersed into space. For entry, the radiators are cold soaked and then bypassed so that that cooling can be used on the ground. Then post-landing, the shuttle uses its cold soaked radiators for a few minutes and then as they heat up, the team activates the ammonia system which flashes ammonia onto the free-end loops to provide cooling. In order to have a safe transition to ground cooling, the ammonia system has to be deactivated and confirmed to have no leaks and then a go is given so that the ground team can activate ground cooling. At least that's the way it's supposed to go. Now there are some critical elements to consider regarding cooling. Obviously, you have to keep cooling going so nothing overheats, but at the same time you can't get too cold or you could freeze the interchanger. The interchanger is the heat exchanger where the water loops dump the cabin heat onto the free-end loops. All the shuttle cooling loops meet there. And since we're dealing with water, we can't let it get below 32 degrees Fahrenheit or the water loops could freeze. And as you know, when water freezes, it expands. And crack, you've just ruptured a piece of equipment that could cost over a million dollars to replace. This is a very efficient heat exchanger so this would only take a matter of seconds. On STS-27, the ECOM team was faced with just this scenario. It's post-landing and everything's going normally. You're on the ground. You've made it, right? What could go wrong here? The team is even starting the post-landing tradition of hanging the plaque in which one of the flight controllers is honored with special recognition. However, not everything is done. Handover to ground cooling hasn't been completed. And the ECOM team is watching ammonia performance as suddenly the free-end loop temperatures start diving. So why don't you tell me, first of all, we've got Queen Keralock and Chuck Dingell for STS-27. Why don't you tell me your positions for this flight and what the responsibilities those positions were. So start with the leader, Chuck, of the team. OK, so it's been a while, but I think it was Aston Entry and Orbit 1 ECOM for the flight so we were responsible for the life support, thermal control, and also at that time the electrical power systems on board. OK, and Quinn? I was the Aston Entry thermal, specifically responsible for the active cooling and passive cooling systems on the shuttle, including the ammonia boiler, freon, and water loops. OK, so what were you looking at there that cued you into the fact, hey, I got a problem going on here. Was that that bell that went off, the alarm that went off earlier, probably? Actually, what happened was, the first thing I saw was we have an indication on the ammonia system, the primary controller, if something goes along with the logic or goes under temp, it automatically switches to the secondary controller, trying to go to a different controller to control the system. That was the first insight, as I saw, is this indication that we just switch from the primary controller to the secondary controller. It generally means under temp. That was the only indication at the time was seeing the switch. And having said that, it's upstream of the Avapout T-ducers, so the temperatures hadn't arrived yet. Yeah. How many seconds is that from ammonia to? I forgot. I do know. Yeah, it's about 45 seconds or so. OK. Yeah. So, you guys knew something switched over, but you probably didn't have an idea at that point. It was an under temp, right? Right. OK. OK. And I think you may have just called, hey, I think you called an under temp switch over, maybe. Right. OK. 33.3, 32.3. You better get a call here when we're ready to issue. 31.2, are you ready? 1.2, you're in the range. Are you ready? Turn off the ammonia system. Turn off the ammonia system. Now the team has confirmed there's an under temp problem. The cold slug has now shown up on the downstream temperature ducers, and the extra cold freon is racing towards the interchanger, which could crack if faced with these temperatures. How long does the team have to respond? What is the correct course of action to take? OK. So, at this point, you know that you've got the evaporate tea low. And what are you thinking? What am I thinking at the time? There was a hairy edge where, you know, it's the, I knew that maybe the initiated ground coolant, there was some speck leakage to the ammonia boiler that it leaks ammonia. So, you're getting some cooling from it, whether it's on or when it's on, no matter what, if it's not required or not. So, it's providing extra coolant to the loop. Initial thought was, OK, we're down, just broke 32 degrees. Maybe they're coolant, and the cooling cards adjusted to low. Am I ready to take this drastic action yet? Or is this, maybe we can take the ammonia boiler off and that would be sufficient to bring it above freezing again. Was my initial thoughts. It was not, it was pretty getting pretty excited. As you can hear, it was the fact that, you know, we were actually below the freezing point of water and we needed to do something quick, so that was the initial reaction. Just get it off and see if that will bring the temperature back up above freezing. So, what is the issue here with getting too cold? What's the problem with doing that? I should try to talk to her. Well, the interchanger is a very efficient heat exchanger and it's got a free on one side and water on the other. And if you put in free on below 32 degrees in pretty short order, you're going to end up freezing water. Turn the water into ice, it expands and volume changing the heat exchanger can break the heat exchanger. That'd be pretty expensive to try to replace this. And one of the big things was that we have done that before in SDS-3. There were some data points. The big thing is, I don't think they'd have any spares and I think the big thing is it just puts not only the cost, which they estimate at $1 million, but getting a new one and installing it in the downtown of the vehicle was a big impact. The impact of schedule would have been pretty enormous. Right. I mean, basically you can't power up the vehicle with the, you know, loops or water loops are not active. So, knowing that, why didn't you just go take the drastic? What is drastic about turning the fan loops off? Why don't you just say, let's go ahead and turn them off. What's the issue there? You know, so we got on one side, we don't want to freeze the interchanger. Why don't we want to just turn the fan loops off? What's the issue there? Well, I mean, to me it was big to you. It hadn't hit that threshold. I kind of had my line of sand. It was below 30 degrees. That's when my line of my sand weren't playing anymore. And as you're going to hear in a little bit, it happened fairly quickly from the drop-down and it was very obvious that once it got to off-skill low of the dooser, which was 24.9, you'll hear that several times repeated. Off-skill low. Yeah, it was off-skill low. It was very obvious what we needed to do. But it was, to me, was a drastic. I mean, the fact that we could potentially shut it down, we may not bring it back up, or we may do something wrong where we lose some science because we lose cooling to the vehicle. And I don't know, that's a DOD flag. I don't know if that was a suit up. Right, that's okay. So basically, you turn off the fan loops, you're losing cooling to the vehicle. Right. The potential emergency power down. And as you hear later, I would point out, I didn't care if Chuck was being the brave person. Let's get it back up. I was like, shut it down. Just forget it. Okay. Let's play a little bit more. PlinyCom deactivate ammonia system. Everybody stand by just a minute. We're going to break PlinyCom. Go ahead. Pliny system, meet off. We're under-temping. PlinyCom, we'd like you to take the ammonia to off. Activate it. PlinyCom, both free on loops off. PlinyCom, both free on loops off. Good ladders, we'd like both free on loops to off. Okay, ammonia ball or B just went off. Okay, so you can see the MCC on command. You're at 24.9 and 24.9. So 7.9 and 24.9. So 7.9, 7.9, 7.9, 7.9, 7.9, 7.9, 7.9. So it sounds like nothing, you've made the call for the free on loops. Nothing happened, right? Initially, and I'm still not clear, you know, we, they're actually, the troll team was basically in the, like I said, the post-mission mode where they're doing the post-black hanging and stuff. And I think even GC got to the point where actually we, temporarily they shut down calm between the vehicles, what I heard later in their first initial call to the vehicle to turn on both free on loops. And I don't think they ever left the house. They're probably thinking the rule, the 1.1, hey, the crew's gone. Why should I keep calm up? I mean, we're already doing the plaque hanging. Let's just, let's just shut this thing down. So, okay. Yeah, I don't know what, I don't know what all the factors were in that listening to that recording, which I don't think I've ever listened to. Probably wanted to block all that out. But just being kind of stepping back as an observer to myself, if I would have had the wherewithal, I think it would have helped to be more exact in the call, especially knowing that the flight crew was not in there, who probably could have responded very well to deactivate both free on loops with a support person in there instead of a more precise call like on panel L1, free on pump 1 to off and loop 2 to off. I think that in hindsight would have helped get that done much faster. If I was a ASP in the cockpit trying to interpret that call, I would probably have trouble myself. Right, you're an MS-1, you haven't simmed in a year. Where are those switches? Yeah, that's good, that's a very good point. Yeah, you want to go to Interchanger? I wanted at least one off. Okay. So what was going on there? We made the call for the friend loops, and now Quinn's calling again. Let's get one off. Yeah, so I can tell you from my perspective, I was trying to find the procedure for a VAP out-temp low. I think in the entry checklist. Right. Okay, so I was stumbling through the book to try and find that procedure to save myself. And so after making the call to turn both free on loops off, I think I was under the assumption that that would happen quickly. I'm going to find the procedure to look for subsequent steps to guide us. And so Quinn called and I'm asking questions, I think out of that procedure. And Quinn says, hey, I just please turn at least one off. So it hit me at that point. Oh, they didn't respond to the call. Right. And you and Quinn can't see each other here, right? One's in one room, one's in the other room. And sometimes there's over the airway of conversation that goes on. And so you didn't know why they weren't turning the friend loops off, right? Right. And I'll tell you from the MIP perspective, one thing that really taught me was, okay, I've already made the call. I'm watching. Chuck is watching this data, too. Don't need to bother. He sees what's happening. I'm kind of in the back room dancing, panicking, cussing. Yeah. So you get the loop off. But I am not. I learned after the fact that when he started talking about what he wanted to do next, it would put a fear through me that he doesn't realize they're still not off yet. And then I'm questioning, why didn't they go off? Why didn't they actually do it? So, but the point in the lesson for future guys is, you know, don't make assumption that you're frick-fickers looking at data. This is all Vox, so it's happening much quicker than it really is. Right. But we had, I want to say, correct me, from long 60 to 90 seconds somewhere before we would have damaged the interchanger. Yes, another 45. I have about 10. There's another 45. And we're ticking along. It's 20 seconds, 30 seconds. And I should have been going, Chuck! We're still on off. I should have been more proactive. I made the assumption that he's looking at the data. He's seeing it too. He's cognizant to put the fear of God to me when he realized he made a call like, what are we going to do next? And I'm realizing he's not looking at the data. And I'm realizing I should have been pushing it. Well, I certainly should have been paying attention to get the primary action to ensure that it was done. That was one of the things that I wish I had done better back at the time, was to be more cognizant to making sure that they took the action that we asked. He's taking too much. It was my job to watch the system, so I should have kept him informed. It's a team effort both ways. Absolutely. It's a team effort all the way around. Good and bad. We work as a team. Let's turn both three on loop, sir. Keep calm, both three on loops, sir. Line up for the ad. We'd like both three on loops to off. Keep calm. What's happening to you? We're under 10th. Sound like he's getting a little stressed out there. Yeah, I think the seriousness of it was probably starting to soak in. Yeah. You probably, I'm just guessing, you just realized I made a call and I'm looking at the procedures and my call didn't go up. And now I'm realizing I need to push this thing a little harder. Yeah, something like that. Yeah, still I think the lesson to be learned there was give a more exact call. Yeah. It's likely to get a quick response. Sophie to call it, because like you said, there's no panel there. It's just the pumps. And if it was an ASP, they might be looking. He's looking, yeah. M-O-10-W or something. Is this on the mid-deck? You know. Yeah. That's good. Okay. The free-in loops are off. The cold slug has stopped, but it's still sitting there. A silent menace ready to wreak havoc. The team knows this, but how does the team keep the interchange from cracking and still re-establish cooling? Now how much time do they have? So you got both free-in loops off and you know that's a drastic action. How long could the free-in loops be off? And how do you gauge that? It's a big deal to have in the free-in loops off. Are you starting another timer? Because you were starting your interchange, your interchange or freeze timer. It's like I know I got so much time before that cold slug hits. Now I've turned the free-in loops off. You got another timer going now? I didn't have a timer. I had the general knowledge that things don't melt down instantly. Right. To me, probably the fuel cells are of the biggest concern. And I know that for fuel cells what we could go, for loss of coolant pumps on fuel cells, go like nine minutes or thereabouts. So I think you have probably at least that order of magnitude before something catastrophic happens. So clearly I think you have not a lot of time but an order of magnitude more time to deal with them being off than to deal with the cold slug coming around. Right. Okay. Did you have a clock going in your mind at all? Well, the good thing is we had the EPS guys under the fuel cells. I knew it was the short pole. We had the EPS guy watching the fuel cells right across there. And he would be the one to make the recommendation whether we had to terminate the emergency, do it basically a power down on the vehicle if we couldn't regain it. Now let me ask you this. Had you guys simmed this case basically prior to this? No. Go ahead. I don't remember. Not really. And the other thing, even when they did start simulating this and doing a sims of this later after this event for future teams, one of the things about the post-landing was always it's very stressful for me is you can't recreate and train all the noises, the ground convoys, the confusion of the additional loops, the communication that's going, all the loops you're listening to. It was very noisy. It was always a very stressful time for me when I became an e-com. When I asked an entry was, because we didn't train it. And it was just, not only that, even afterwards was this chaos everyone was getting done with their job in the control center. It was noisy in the control center. It was noisy on the loops. And you had this unknown factor we don't train with, which is the KSC guys very often. We did start using direct com with the guys trying to minimize any kind of problems. We could directly talk to the guys at the convoy. But no, it was the training. We sat down and walked through this procedure. There was also a lot of lessons learned, like I said, from SDS3 with things that they had done. After we handed over, they did some additional ammonia boiler tests that they undertemped. There were some lessons learned from that. And the fact that that happened there made it very cognizant. It happened once. It could happen again. Yeah, I think I just can't emphasize enough the few minutes of time that we took preparing and walking through that was just enormous help. I think it's as much better as I could have done in that situation. And I certainly wish that I had. I think I could have done a lot worse had we not done that preparation because turning off both freon loops is you want to do what? Right. You want to turn off all the clothing with the vehicle? Yeah, the fact that we had talked through it, at least it resonated with me that, yeah, that's what we have to do. We talked through that. We talked about the urgency of time and how that has to be done to preserve that heat exchanger. So that, I at least never questioned that aspect of it. Had we not talked through that case is as deep a respect as I have for Quinn and knew that I had the best thermal guy in the back room. I don't think I would have just said, okay, whatever you say. I think I would have at least questioned some, and that might have made the difference in the case. So that preparation we did was just worth invaluable. Now, was that SOP for you? Like, if you had an event coming up that you would talk to the back room and say, hey, this is the things that could go wrong. This is what we're going to do. Was that just SOP or you just kind of got lucky that time that you did that? You know, I really don't know. It's probably SOP when you feel like you don't know as much as you should know about something. And again, coming from the electrical side of the house and trying to pick up the thermal and life support stuff, you don't pick up all the knowledge you need in a matter of a couple of years, which is probably as long as I had been thinking about it. So it's many years of preparation to really understand those systems. So to me, it was basically all that I could do, personally in limited time, in order to be the most prepared that I could. So I had five or ten years of experience in that I might have been more complacent and not felt the need to do that, but I did it out of a personal need to cover myself. Right, gotcha. Excellent. Let's keep going here. So we got that right now. The free-in loops are off. Stand by. We're going to exercise the procedure for the five-to-six entry checklist. Okay, you want to take the free-in loop one-to-eight, wait one minute, then go two-to-eight. That's correct. Okay. Now, is that something, it sounds like you talked about this previously, so had you already stepped kind of through the recovery of the scenario as well? When we went to our time frame, we actually went through all the contingency aspects, three steps, and kind of walking through it. It was really, so we kind of knew the plan that we were going to do. Again, I was at the point, I would have been happy if you said it, it's too emerging. At that point, so let's get this vehicle back right in a good config to hand over. So I don't remember, were those steps in the entry checklist about T-LO to reactivate like that? Yes, they were. Okay, that's good. So it sounds like you're doing one loop at a time. Why don't you just turn them, under temps, we've stopped it, let's go ahead and turn them both on. Why did you do it that way instead of one loop at a time? Oh, it was actually part of the procedure, but the real reason is you're only activating one, so half the, I don't know what to say, you're only dealing with half the coal slug at a time. And the vehicles being hot right now and the water loops have gotten, the interchange has been hot since it's been running without any cooling. So you just look it over, and one of the coal slugs at a time and then evaluate how the system is working. You get the insight into the situation, whether or not your under-temp situation is basically remedied or not, you could look at it as half the risk, half the heat rejection capability of the heat exchanger. I wasn't paying attention, did y'all pick, were you picking the freon loop with the ammonia system? I wasn't paying attention to that. Actually, both of the loops were off at the time. I mean, to turn it back, both ammonia systems were off, it didn't matter. Okay, that's right. Did we have an additional measurement on one loop over the other? Yeah, you had an interchange or in on loop one. So I don't know if that's why you might have picked that one first. I just did what he said. Well, it wasn't, he had a little bit of heat. Chuck does it good. Have you already gone flow proportional valve to payload at this point? Actually, believe it or not, the old procedure was go to interchanger. The assumption was the payload loop was going to be the slower flowing loop, so the procedure written generically always go to interchanger. Oh. It's been changed since then. Oh, right, right, right. So if you had a payload, you would freeze that payload's water loop if it was there. That's normal as before. Okay. So take free on loop one to A at this point? That is correct. You want to do the procedure on five dash six? That's a firm flight. We want to take free on loop one to pump A right now. Okay, captain, we'd like to go to five dash six. Yes, water loop one. Yeah, that's firm. Is that you guys? Water loop one is out of page five dash six. Entry checklist. Let's see. Did I catch that right? Y'all didn't turn the water loop off. They must have said it wrong. He may have said on. One of the points too, is when you listen to this loop, it's amazing to me after listening to this thing how I knew exactly what I wanted to do, but I was under the stress and I don't know, panic, but under the stress. Right, he never panicked. It's amazing. I was communicating exactly. I was having trouble between loop one and loop two and what pump I was feeling to communicate. Chuck did an excellent job, if okay. I heard what you said. Is this really what you wanted? What were you saying? And it was very easy to go back and say, yes, that's what I meant to say, but it was amazing how simple things like loop one and loop two, even though I had the procedure over, I was saying the wrong thing. So I turned the free and loose off. Now you had kind of an order of magnitude more so you could slow down a little bit, right? A little, yeah. Not a lot, you got the fuel cells out there heating up, but you're able to slow down a little. Does that go through your mind? Okay, I can take a little bit more time, make sure I'm understanding the call instead of just, you know, pedal to the metal. Is that in your mind? I think it was in mine, yeah. I know it's a way back. Yeah, it was. I was going through it in my mind. Free on top, loop one to A. That's fine. In Capcom, we want him to start with free on pump, loop one to alpha. We want him to press on through the procedure. Is that all right? Let's start out with free on loop one to A and look it over. Okay. Capcom, all we want is free on loop one. Holy cow. Yeah, is that what you want? Free on loop one to A and watch it for a minute. Yeah, sir. Okay. Both free on loop two down. That's complete. That's read up to A. We will do that. Just said that. You just say entry checklist is not available now? I think I just heard that. I never heard that before until you mentioned the FDF's not there. I never put that together before. Free on loop one is running again. Okay. Okay, they're back up now on loop one. Yeah, they're back up on loop one. I think he come, are you noticing any ground cooling now? He's still down with it. I can't answer that right now. At that point you had one free on loop up and I heard the EPS as mentor, right, with that Larry Mentor back there, saying the fuel cell return temp starting to skyrocket. At that point with one free on loop up, were you all feeling pretty good about yourselves or did you need to keep going and get both loops up to get yourself in a really good config or what was your clock like or what were your thinking? We're not over tipping them when we got one loop up. I was feeling pretty good at that point. That's good, so you're starting to get over the hump here, gotcha. It went from me dancing to seeing Larry start dancing. It's not a dance of joy either, right? Gotcha. Karen, what is your pop out at this time? Okay, about that. Here's 42.5. Do you think we under camp duty ground or cooling? Initiating ground cooling, in fact, without deactivating the ammonia system and having it cranked down rather rapidly. So I know you just said it there but just to put it out there. So at that point, what was the problem? How did this whole thing kind of evolve? Okay, maybe you can remember better. I thought they did a new system and they did a new procedure at on the cooling cart where they re-circ-fluid. I believe is that correct? They re-circulated the fluid in the cooling cart. It would allow this cold slug to build up. Okay. So when they initiated they got this really cold slug to occur and I think that was it. But there was a new procedure. They implemented it for 27 and changed it back after 27 to kind of more of a standard procedure they had. Back then was it standard to have the ammonia in the ground cooling but at the same time? They both came up at the same time, right? You had ammonia and they hooked up. Was that standard back then? Normally they would notify us ahead of time that they were ready to initiate. At that point we would make the call to deactivate the ammonia because the new procedure re-circ had this cold slug when they initiated it up but we were not notified that they were about to initiate it. Also there was some pressure that they knew that we were getting near the end of our second ammonia tank. They're feeding a schedule crunch there to get they initiated so I think that they got in a hurry and they initiated without a continental procedure with a contact to initiate. There's another piece of an error change. You've got some guys on the other end are feeling some schedule pressure. You've got some procedures that weren't synced up, some calm that wasn't synced up. People starting to get distracted with the plaque hanging. There's a lot of variables going on in there. You captured that really well. I didn't realize that all that was going on and even listening to the recording you guys had obstacles out in front of you didn't know about, right? That's really good. In hindsight it's always 2020 I wish that I would have done more coordination with the people on the KSC end to have a better handoff and I think we did improve on that a lot after that event. Right, let's see what else we got. EPS is reversing. I hope it comes on down. Here's our threes under four kilowatts. It should be above one ten on it. So it sounds like EPS is freaking out a little bit but you know it's going to get better here, right? I don't know if it was out of ignorance or not but I was less concerned about it than Larry was at the time. You had one family pup, you remember what the KW was about that time? I think he was saying four for that fuel cell. If there are evens it's probably about 12 or so, right? So one friend should be able to handle 12 pretty well. Okay. We're stuck. My dog appears very taste back to normal. We're going to slow down. We'll get you back home. Thank you. I'll go with you. This is ground cooling. This is ground cooling. So we're activating this one system. Okay. Okay. We see good ground cooling. We're going to go to take free on loop two to Pompey. Okay. We'd like free on loop two to Pompey. I'd like you to take free on loop two to Alma. Sounds like you're done. Yeah. So as quickly as it came it almost subsided and forget all about you, right? Yeah, like Quinn said that since that was Vox I think the pain lasted a little bit longer. Yeah. Vox, this is what we got about nine and a half minutes there so unboxed. How long do you think the situation really went? About double that maybe? Yeah. That's excellent. Any other summary thoughts about the situation? You gave a lot of hindsight while we were going but any other thoughts about the situation? I guess some of the things that came from this mission like we talked about one-to-bash-one got changed we started to train post-landing we spent a lot of time with the KSC folks trying to establish better protocols and communication between the ECOM and the ground convoy guys. We improved the Vapauti low procedure when we came a lot of improvements on how to go through it what else did we improve? You reminded me of stuff I had long because there was a lot of improvements and the communications with the flight directors stressing with them that hey look the vehicle here's what under temp occurs this is why we're so scared of it and the flight directors did a lot better job post-landing to make sure the team is cognizant we didn't hand it over until keep the room quiet until the crew's established and the crew's out Sometimes, even when the mission seemingly over, it may not be over in the midst of distraction the ECOM team identified the under temp situation diagnosed the cause of two cooling systems running simultaneously then resolved the situation while preserving crew safety and all vehicle hardware the MCC flight controller is one who must always be aware of what is happening and we may find ourselves in a role where our performance has ultimate consequences discipline competence, confidence responsibility toughness, teamwork and vigilance these are our core values values that have served us well values that have allowed us to slip the surly bonds of earth and perform our dance above the skies