 Hi, my name is Chelsea Troy. I am a software engineer and data scientist. I'm currently teaching a Python programming course at the master's programming computer science at the University of Chicago as well. And you can find me online at ChelseaTroy.com or I'm also hey ChelseaTroy everywhere, including on Twitter, which you have a link to a tweet in the chat. So you can take a look at it there. So I'll be honest, most of the time when I speak and teach, I stick to purely technical topics, but today I'll be talking about two important professional skills, which are giving and receiving feedback. What I'm gonna do is I'm gonna tell you some of the things that I wish that someone had told me about giving and receiving feedback when I was first starting out in tech, because I think it would have made the first decade of my career a lot less stressful. And I think that these things would also make somebody's second, third, or fourth decade of their career much more productive, particularly as they have gained some power in the workplace. Why when they've gained some power in the workplace, we are gonna talk about that today. So first, let's discuss what we mean by feedback. What is feedback? It's a signal. There's a useful tidbit of information in there, as opposed to noise that doesn't have a tidbit of useful information. From external sources, this isn't our self assessment, which can be biased and can have things in it that we miss. This is instead supplementary signal that we're getting from somewhere else. Sometimes that's our test output when we're working on an app, or sometimes it's our system logs for working on a program that has logs. Sometimes it's another person, and that's the kind that we're focused on today. It's a signal from external sources about past decisions. This isn't about us as people. It's not personal. Sometimes it can sound personal, and it's up to us to tease out of that what decisions we might have made that resulted in this signal we're getting. We'll talk more about this later. And it informs future decisions. Feedback helps us make future decisions. In machine learning, it's how we update the weights in our models to make better predictions. In human life, it allows us to gauge the effects of our future decisions by remembering what we've learned about the impact of our past decisions. Now, this can go either way. We can get feedback that encourages us to keep doing something, or we can also get feedback that encourages us to change something. But this is gonna be our working definition of feedback for our talk today. It is a signal from external sources about past decisions that informs future decisions. So, here's the tricky thing about feedback. The point of feedback is to help a person or organization improve. And from that perspective, we should look forward to it, right? But a lot of times we don't. Instead, feedback ends up producing a lot of anxiety in the workplace. Why is that? Well, a couple of reasons. First, when we imagine receiving feedback, what we're picturing is usually criticism. That happens because we focus more on negative feedback than we do on positive feedback. It also happens because when we're angry or frustrated with someone, we don't have many options for expressing that in the workplace because it's not considered professional etiquette to ever be angry or frustrated or sad or any of those pesky negative emotions. So we bubble up those negative emotions as feedback because that's the only avenue into which we can shoehorn the airing of grievances. That sets up a picture of feedback as you over on one side of a table and your goal on the other side of the table, on the opposite wall, and then the feedback giver standing across from you at the table, blocking the way between you and your goal of getting your perfect score or your raise or your promotion or your bonus money. Something like this. Now, when we're seeing it like this, it's scary to get feedback because it is scary to hear criticism. And it's scary to give feedback too because we get worried that maybe we'll hurt someone and maybe we're even worried that the feedback recipient will retaliate against us. Because we have this vision of the feedback giver and the feedback recipient on opposite sides of the table, it's very hard to get out of that mode of thinking. And conventional advice doesn't shift that perspective. It just sort of tiptoes around it. Let's talk about some examples. So what's some conventional feedback advice? Constructive. People will say this or they'll sometimes say opportunity feedback, give constructive feedback or opportunity feedback. And they're replacing the term negative and negative feedback, but it doesn't actually change the way anybody thinks about the feedback. Instead, everyone hears constructive feedback or opportunity feedback and they back translate it to negative. So this technique doesn't work. Second, the poop sandwich. So we sandwich a piece of negative feedback in between two pieces of positive feedback. There are a couple of failure modes for this. The first one is people know about the sandwich and they ignore the positive bread to only focus on the negative part of the middle or people don't know about the sandwich and they fail to understand the urgency of the feedback since it seems like it's mostly positive, two positive pieces with one negative piece. So this doesn't work that great either. Finally, some of the conventional feedback advice now includes make it actionable, specific and kind. This is getting closer to a paradigm for how to deliver feedback, but it assumes that the feedback giver has control over whether the feedback is perceived as kind and they don't. So why would the feedback giver not have control over whether their feedback is perceived as kind? I'll give you a couple of examples so you can see what I mean. First, if the recipient did not consent to the feedback, then maybe it's receipt is unwelcome and it won't be received as kind no matter how it's said. Or maybe the recipient doesn't prioritize the skill that this feedback is about. So the receipt of this feedback feels like a request that they change their priorities. Great example here. For folks in marginalized groups who are looking for feedback on their technical capacity, they'll often be given personality feedback. Oh, you should smile more. You should speak more softly in meetings. You should speak more directly in meetings. This kind of feedback that doesn't help them advance their technical skills and is focused specifically on how other people perceive their personality, which somehow for folks from marginalized groups can never be right in a workplace setting no matter how it is. It's a great example of a situation where feedback assumes priorities that might be different from the actual person's priorities and won't be perceived as kind. Another way that feedback might not be perceived as kind is if the recipient was not expecting negative feedback and the receipt of this feedback was a surprise. This sometimes even happens when people ask for feedback. For example, when someone asks for feedback and the thing they really want is validation. We'll talk a little bit more about that case in a minute. So if all this bad stuff is gonna happen, why bother giving feedback at all? Few reasons. First of all, to improve the team and the work that affects your day-to-day experience at work. So it's a good reason. Second, to build a positive working relationship. If you provide people with feedback that helps them, that helps you build your relationships, which is great for your network and your work experience. To be seen as a mentor or advisor, which is a necessary part of assuming a leadership or expert role. Or to get reciprocity when you ask for feedback. Because folks are more likely to help you out with feedback when you want to improve, if you've provided them with feedback that has helped them improve. So if we wanna give feedback, how do we mitigate the risks associated with it? There are a couple of ways that we can do this. First of all, I don't give feedback unless explicitly asked for that feedback. And second, this is where we're getting back to something that we discussed a little bit earlier. I determine, does this person want feedback? Or do they want validation? So if they want, oops, if they want validation and I tell them that something is wrong, my feedback can be seen as unwelcome or unkind even if I explicitly asked for it. We talked about that a little bit earlier. How do we determine whether and where somebody wants feedback? So first, I'll ask the person for two or three goals that they have. This helps me understand what are they trying to get better at? And therefore, in what areas might my feedback for them be welcome? So let's talk a little more about goals. What might goals mean in the context of a code review, for example? Well, usually when we write software, the main things that we're thinking about depends on the app and the project. Some of the options might be maintainability or documentation or resilience, no errors as a result of inadvertent bad inputs, for example, or security against deliberate bad inputs or speed, those are all qualities that we might have as goals in our software and things we might be looking for feedback on in a pull request review. What might goals mean outside the context of a code review? Well, outside the context of a code review, our colleagues might have all kinds of technical or interpersonal goals. For a while, for example, I had a goal of noticing when my less outspoken colleagues wanted to talk in a meeting and helping them get the floor. So that was a goal that I had. Second, after I get the person's goals, I get their self-assessment. And this is how I determine whether they're looking for feedback or rather whether they're looking for feedback on opportunities to improve or whether they're looking specifically for validation. If their self-assessment is purely positive, they might be looking for validation, which is sometimes fine and I can provide it to build a relationship with this person to the extent that I authentically can. So for example, if somebody puts some code in front of me and they're like, oh my gosh, I just did this, I worked really hard on it, I worked all day yesterday and I think it's awesome. Isn't it awesome? In that case, they're looking for validation and I'll agree with them to the extent that I authentically can. Yes, Joe, this is absolutely awesome. This is so cool. That can help me build my relationship with my colleague by validating them and what they're working hard on. So if, however, their self-assessment includes some concerns, it's probably safer for me to provide something other than congratulatory comments. For example, if that same colleague comes to me and shows me some code and says, hey, I'd really like your thoughts on this code, I'm proud of the elegance with which it does whatever, but I'm concerned about the legibility of this part right here. So that tells me that this person is already thinking critically about this code and it also tells me something about the specific goals that this person has with regard to improving it. That's helpful for me in knowing that I can say something other than a validating comment and have a chance that it will be seen as helpful. So once I have the self-assessment, if that self-assessment demonstrates that this person is not necessarily specifically looking only for validation, I now make two lists. For each goal, I list first things that max out the opportunity to reach that goal. This is things that this person is already doing that work towards their goal. There are things that this person should keep doing the way that they are doing it to get closer to that goal, things they don't have to change. So second, places where there is still additional opportunity to reach the goal. This is things that the person can change to get closer to their goals. These are opportunities for me to help them get closer to the goal that they may not have seen. And by framing it that way, what I can do is move my chair in the feedback conversation around the table so that the recipient and I are now both on the same side of the table facing their goal on the other side and conspiring to help the feedback recipient get to those goals. It changes the relationship between the giver and the recipient of the feedback from an adversarial one into a collaborative one where they are both facing a common challenge together. Now let's talk a little bit about receiving feedback. When we're thinking of feedback as this adversarial thing, giving feedback is scary, but receiving feedback is also scary. We're gonna come back to the scary part, I promise, but first we need to talk about soliciting feedback because we need feedback to get better. To accelerate our careers, we want to absorb the knowledge and perspectives of other people, including about us specifically. But in the working worlds where the peer reviews are not required, far more often than scary feedback, what we're getting is no feedback or insufficient feedback. And without feedback, we can't get better. So how do we incentivize people to give us feedback? By the way, this becomes more and more important as we gain power in the workplace because the more power we have, the more possible it is for us to retaliate against a feedback giver in a way that would impact their careers or their lives. This is part of the reason that as people rise up the power structure, they complain about getting less and less feedback. It's not that that person became a different person, but rather that the amount of power that they had increased such that the risk of telling them something that might upset them also increased for people at the organization. So how do we deal with that challenge? First week's breastly solicit feedback from people rather than hoping it's just gonna roll in unprompted. I find that if I gave someone thoughtful feedback in the past, they are more eager to reciprocate than when I am looking for feedback. Now that means that I have to undertake the risk of giving them feedback, but there is a trick that I use to get around this. Even if the only feedback I've ever given someone in the past is validating feedback, they tend to be quicker to give me whatever kind of feedback I'm looking for, including critical feedback. So if I need feedback from someone who is themselves not particularly skilled at receiving feedback, I can give them validating feedback at opportune moments. And that will help to, it'll help to establish that reciprocity without incurring the risk that critical feedback forces me to incur in an adversarial feedback relationship. So that's first, expressly solicit feedback. Second, we give them our goals. We were talking about asking for goals and we were talking about giving feedback. Similarly, when we want to receive feedback, we can provide those goals up front. This does a number of things for us. First of all, it makes us safer to give feedback to. It makes us safer to give feedback to because it demonstrates that we have already thought about what we want feedback on. And thus that that feedback is more likely for us to perceive as helpful. Second, it controls coworkers' focus. So if somebody's been working with us for 10 or 15 years, it's gonna be really hard for them to filter back over all of their experiences with us to get all of the feedback that they might have for us. But if we have a difficult goal, then we can use that to focus their thinking and consider their specific interactions with us that might have related to those goals. It also allows us to avoid situations like the one where we were talking about before where somebody is looking for technical feedback and what they get is personality feedback. When we control coworkers' focus, we reduce the likelihood that we're gonna get feedback on topics that are not our priority. It makes it easier to think of examples having that specific goal in mind helps folks consider when we might have done something to improve at that goal or when we might be doing something that's preventing us from reaching that goal. And finally, it makes it easier for folks to frame their feedback as an attempt to help. So we just talked in giving feedback about making two lists for each goal, the ones where the person should keep doing these things to achieve the goal and the ones where the person should change something that they're doing to reach the goal. But if somebody hasn't necessarily seen that framework for providing feedback, having the goals can help guide them to it anyway. And they can start with, I understand that you have this goal of blah. I think that you would have an easier time getting to it if you change this, this and this. Folks stress out a lot about framing feedback precisely because no matter how you frame and welcome feedback, it tends to be viewed as unwelcome but the feedback on the feedback will never be about you gave me feedback I didn't want. It'll always be about the framing because it's not considered appropriate to say I didn't want that feedback in the workplace. And so the framing causes a lot of stress and giving folks guidance with regard to how to frame that feedback will be helpful for getting them to give it in the first place. Okay, so third, after we receive the feedback, we follow the steps. What are the steps? First, thank them. This establishes that it is safe to give us feedback that we probably won't retaliate. Second, explain how we will act. So we get credit for responding to the feedback even before we have done anything with it. Third, follow through. By using the feedback, we demonstrate to the giver that we value what they have to say. And fourth, publicly praise the feedback giver for their help. So they receive a professional reward for helping us. This makes them more likely to give us and others feedback in the future because what they're doing is they're taking time away from the code they could be writing or the job function that they could be performing that's gonna get them a raise or a promotion and they are instead spending that time helping us get better. So here's a question. What if it hurts? What if they have something bad to say? So in a former life before I was a software engineer, I was a coxswain, which is the person in a rowing shell who sits in the back of the boat and yells at the rowers and tells them what to do and steers. Now, when I was doing that job, sometimes rowers would fill out anonymous evaluation forms about my performance. I hated it. I felt nervous every time. A few years in, I started reminding myself, people think what they think about you so you might as well know what it is. And that's helped me to receive feedback ever since. By not asking people what they think, I'm not making them think more highly of me. I'm just cutting myself off from the opportunity to fix it. So keeping that in mind is helpful for me when I'm anticipating feedback that's not congratulatory. But we also have some actionable advice here too. First of all, if it hurts, follow the steps. So having steps to follow in the event of tough feedback makes the feedback a little easier to receive because it reduces the panic of not knowing what to do or say when you get that feedback. You start with thanking them. You know that. You have a framework for what to do with feedback. Second, a night of sleep has prevented me from making many a rash decision. I tend to be able to distance myself after a night of sleep and think about things a little bit more clearly. Third, research. When I get feedback, I look it up. Have others received this feedback? How common or rare is it? Who tends to get it and why? And how do they fix it? This can be really valuable if you've received feedback about, that includes information that you completely weren't aware of. For example, you received feedback on some code that had to do with a performance concern that you were not aware was there. Or you said something to someone in a meeting and it came off as belittling them based on their age, but you weren't aware that that was where the context of the phrase that you used came from. You just heard it around and you'd used it. It's helpful to do research in those circumstances to give you an idea of the gravity of the feedback. So finally, and this isn't always, but this is if the feedback is particularly difficult to process. I'll find someone who can help me process that feedback. Now I wanna mention here that I will specifically find someone who has more institutional power than me so that they can feel safe saying no if they want to. Being a processing buddy is essentially helping somebody with emotional labor. And so I wanna make sure that I'm not asking somebody for free emotional labor in a situation where they can't say no to me. This person can help me work through my feedback and figure out what I'm going to do with it. I can say to them, I got this feedback. This is how it made me feel. I did some research. This is what I found out. And this person can help validate my feelings sort of and help me get from, I got this feedback and I feel really bad, to this is my action plan that I'm going to use to get closer to my goals with the help of this feedback. And finally, we make that plan. We take those action steps. We put them in a list. Maybe we decide in what order we're going to tackle them whether we wanna tackle a few of them at the same time or we wanna save some for later. Having that plan is what turns feedback that hurts from that hurtful experience into the positive experience that it becomes once we act on it. So in conclusion, we talked about the adversarial model for feedback and where it comes from. We also talked about a more cooperative model for feedback. We talked about how to ensure our own safety before giving feedback and how to give useful feedback. We also talked about how to attract feedback, how to respond to feedback and how to turn the feedback that we need to change something into an actionable plan. So here's my closing salvo on why this topic is so important. Not only do we know we need feedback and theoretically, we want feedback, but also feedback is the only mechanism by which we can access a perspective other than our own about our work. And that capitalizing on perspectives other than our own is exactly what leadership is. Without it, what we're doing isn't leadership. It's acting alone, which is rarely successful and I would argue never sustainable. Nobody has the energy for that. And no matter what field we're in or what role we play there or what title we hold, if we do it long enough, we'll eventually have to step up and lead something. My hope is that this talk provides you with some tools to take on that challenge with a lot less stress. Thank you. With that, I think we can go to Q&A. Let me stop sharing. I found it really helpful, especially in looking for my own growth kind of ways of seeking out feedback, which I think maybe I don't do enough, so that's good. I have some questions. The top voted question here is by someone named Matthew Miller. So I'm gonna start with that one. Hi, Matthew. Yeah. What do you do when you are in a mentor or leadership role and something happens where someone needs to hear something but they aren't asking for the feedback? Maybe not like a code of conduct incident but just something that really should have been happened better in the community. And they're not asking for the feedback or they might not even seem like they're aware of the situation and you are in a leadership role or maybe you're the sponsor for that person and the project or something like that. I think that here having a personal relationship with the person and having some awareness of how they might respond to the feedback can be helpful. For me, if I don't know anything about the person, I usually take a very risk averse approach to providing that feedback. The interesting thing about feedback is that it is a lot less risky to transmit feedback down a power gradient. And so if you're a mentor to this person or you're somebody that this person needs in order to have access to something, you have more freedom to provide that feedback that's not being asked for because it is down a power gradient. And we'll see the same thing in the workplace. So for example, if I have a colleague who really, really needs to receive some feedback, it's very, very urgent that they receive this feedback and they have not asked me for it. And I'm not confident that they're going to respond positively to this feedback. Generally what I will do is I will talk to their manager because the manager is the person with the power to fire this person. And therefore the power gradient is large enough that they can give the feedback down the power gradient and they can do it with a lot less risk than I can. Now, in reality, this gets a little bit fuzzy, right? Because in general, another thing that I try to do with feedback in the workplace is I try to the extent that I can to give it directly to the person before I end up going up some chain with it. Because I also find that one-on-one conversations are the most productive venues for disagreement or for difficult conversations like that. I don't think that I have ever seen success by surprising people in a large meeting or anything like that with like an idea that's wildly different than what we were trying to do. In general, anytime I'm trying to change things and the bigger the change, the more I do this, I try to socialize it in a one-on-one situation first. I do have to counterbalance that with the risk of retaliation when it comes to feedback specifically. So suppose that I need to give feedback to like, this is not the situation by the way where I currently work, but suppose that I need to give feedback to like a tyrannical CEO who is 100% convinced that they do everything absolutely right. In this situation, having a direct conversation with that person is not going to work. And so in those situations, I need to figure out a different way to do it. Now, there's another thing here in that example, which is that if the person is the CEO, there isn't really anyone with the power to fire them with the exception of maybe the board. And going to the board is like a whole thing. So whatever they did probably has to be really, really bad before I'm going to bother with something like that. But it happens. What do you do if you need to provide feedback in the presence of a really, really large power gradient? And in those situations, basically your options are collective action or disengagement. And we can see this in a lot of examples. We see this in the way that people engage with their government about things that the government is doing. We see it in the way that folks engage sometimes even at the company level. We're seeing in tech right now, particularly in the United States, a large movement for tech workers to organize or unionize relative to their employers. And part of that is that over a really large power gradient, sometimes you need that collective power in order to be able to provide feedback in a way that's gonna keep the feedback giver safe from retaliation. Unfortunately, the other option than collective action is kind of disengagement, looking for a new place to work, looking for a new opportunity somewhere else, or deprioritizing that thing such that you can sort of live with the person continuing to do it the way that they do it now, which is of course not ideal. But luckily we have a number of options before that and hopefully if we're working at an organization that can sort of start to systemically deal with feedback more the way we've talked about in this talk, unless the way we typically do it, then we reduce the number of situations where we end up in that difficult place. Yeah, thanks. The next one here is what are some ways to move your chair to the same side of the table when it's actually like a video call situation like this or it's text, like you're not really, you're not literally in the room. How do you, how can you do that in a metaphorical way? Yeah, absolutely. Well, so conveniently the chair thing is a figurative example. We don't ever have to literally actually move our chair all the way around the table. Instead, using that as a visual to help demonstrate kind of the way that feedback can work. Now I will say, I believe some studies have been done that indicate that that actual physical juxtaposition of people does make a difference in how things are perceived. So I don't want to discount that. But even if you can't move and I don't usually physically move my chair, I can still figuratively move the chair by asking somebody about their goals and then framing my feedback to them in terms of how do they reach that goal? Both, what do they need to keep the same in order to reach that goal? And what do they need to change in order to reach that goal? And as I'm talking to them about that, I will continue to bring it back to their goals. And I can say, I know that you're trying to get this project pushed by this state. I think that you need to keep doing what you're doing with regard to commitment to your own like focus time sessions and those kinds of things. Another thing that might be able to help you reach that goal is when folks are putting meetings on your calendar all the time, you might consider trying to ask for an agenda ahead of time. That way your meetings will end up running over a lot less and you'll be able to stay true to that focus time that you wanna spend working on this thing. So that's an example where somebody maybe asked me for feedback. I'm trying to get my project out. I can't seem to find time to work on it. How do I do it? And I have kept the entire thing focused on their goal and how they might reach that goal. And that's very different than if I had come in and I had said something like, well, you've got this project, it's not reaching the goal. Maybe you're not putting enough time into figuring into getting the project done. And by focusing on the goal and how to help them reach it, I can avoid a situation where it seems like I am saying that they don't do enough work or something like that, for example. Or if somebody comes to me and they're just vaguely like, I do you have any feedback for me? Like if I go to them and I'm like, hey, I know you're supposed to be a programmer but I notice you're in meetings all the time. Like a lot of times that's not a programmer's fault but that can look very adverse. That can sound very adversarial, right? And so focusing specifically on this is the part where I am helping you reach this goal. You've communicated this goal to me. I wanna help you reach it. I see you're doing these things. They already are very helpful towards reaching your goal. Here are some additional things to think about that framing reduces the risk in providing feedback a lot because it stays centered on a collaborative, even conspiratorial relationship between you and the person to help them achieve what they wanna achieve. I think you actually kind of answered the next question here as part of that but maybe can elaborate a little more. What can you recommend as good practice to avoid honest, possibly negative feedback being taken as personal criticism? Yes, so this is where the two lists is extremely, extremely helpful because those two lists are broken down around each goal. And so you can say, I know you have this goal. Here are things that you're already doing to contribute to that goal, keep doing them. So what you've done is first, you have focused on the goal. This is not about the person, this is about the goal. Second, you have started with anything that they are already doing that helps achieve this goal which means that you are starting with some validation which is helpful. And then third, you have here are some additional things to think about to reach that goal or here are some things that you could change that would help you reach that goal more quickly or in the timeframe that you want to. And that way, the focus is the entire time on helping the person do something that they have already expressed to you that they want to do which makes it a lot easier to establish that feedback as something that you're doing to try to help them. Okay, let's see. How do you feel about the adage, praise publicly, criticize privately? How do I feel about the adage praise publicly and criticize privately? So this harkens back a little bit to something that we were talking about before, a couple of things that we were talking about before. Praise publicly, yes, not everybody loves being singled out in a large group so that's something to keep in mind. In general, public praise definitely, large groups not always necessarily the way to do it but praise someone to their manager or to people who are responsible for tracking their growth in an organization, absolutely. So in terms of criticize privately, in general, if I can and if I think it is safe, I will try to give feedback to the individual in a one-on-one setting first before I will go over their head with it. I will only go over their head with it in two circumstances. The first one's much more common. The first one is we already talked about this in a one-on-one setting a few times and nothing is changing and now what we've got is not a single incident but rather a pattern. And I define a pattern as at least three instances where something is demonstrated. That ends up being really helpful to me because then if a manager sort of like is busy and doesn't wanna deal with it and tries to write it off by saying it's not a big deal, I could say this is not about the latest incident, it's not about any of the incidents, it's about the pattern and if the pattern continues, then that's gonna be an issue. So that's the first one is where individual wasn't effective. The second one, and this happens much more rarely, is where I already know from having watched this person receive feedback in the past that even my best feedback techniques are not going to make this person see this feedback as kind or necessary or any of those things. Maybe they just straight up don't respect my opinion and what I have to say in which case they really don't wanna hear anything for me. No feedback that I provide is going to be considered useful to them. And I don't have control over how they perceive that feedback, right? That's the second situation where I'll talk to somebody other than directly kind of the feedback recipient. And I will say that happens very, very rarely. And I will say for what it's worth that happens even more rarely now than it used to given that I've sort of gained some institutional power in the industry, which means people are a lot less likely to write me off than they were early in their career, unfortunately. Okay, this next one is about closed beta testing and feedback on apps and software. I'm not sure that's super relevant so I'm gonna skip it. Question? I won't say about that one. No, no, it's okay. I think you're right. I think that's a good call. That having been said, this talk is mostly about interpersonal feedback. However, I have a couple of blog posts about feedback on code specifically. So what I'm going to do is let me pull that up right now and then I'm gonna drop a link in the chat because feedback. So feedback is this interesting thing, right? Where there are a couple of different ways that we needed in the workplace and the interpersonal is kind of what we've talked about but we also have this very unique situation where we talk a lot about code. So this one is about reviewing poll requests. It sounds like this with closed beta testing, another thing that we're talking about here is sort of feedback from large groups and how you parse feedback from large groups. So I don't have anything specifically on that on my, well, I might, but I don't think that it's 100% related. I think something that is more related is understanding how to parse data from a large number of people in order to figure out how to change something. And for that, the pieces that demonstrate how I do that are actually about teaching where I have a large class of people and I have to design my class based on how they are going to respond to things. And so I have two pieces on teaching and feedback that I'm also gonna share because I think that those, a lot of the skills that I talk about in those is also gonna translate to something like a beta test. And so the first one is just about iterating on a class based on course feedback. It talks about the questions that I use in surveys. It talks about how I evaluate the responses to surveys. And finally, I'm gonna share a piece called Designing Assignments for Programming Classes. And here's that one. And what this one does is it says, how do we take a bunch of amorphous data in bar charts and figure out from that how to change our assignments in very amorphous ways? So hopefully either of those will be helpful. Awesome. I hope somebody is saving these links so we can attach them to the video later. Action, be cotton. I'm glad you, that was a really useful thing out of something I thought was out of context. So awesome, that was really helpful. And actually we just did a survey of I got about 800 responses back about the Fedora project in general. And so we've got a lot of that collective feedback to parse. Sweet, you have a case study. Exactly. All right, the next one is, can AI assistance be helpful to predict the outcome of the feedback and warn of something bad is going to happen? Okay, sorry, can you repeat the question? Can AI assistance help you evaluate feedback before you give it to see if it's going to be helpful? I have not tried that. I would hesitate. Here's why. What computers do is they attempt to approximate, computers cannot innovate. What AI does is it can take, well, I will say what, yeah, what computers do is either we tell them how to make a decision or they use data about decisions we made in order to attempt to replicate the decisions we made. And for that reason, unless the issue is specifically a scaling issue, computers are bounded in their ability to accurately make a decision by how accurately we make that decision. They can outdo our scale. They cannot outdo our judgment currently. And I would say that when it comes to how and whether to give feedback, collectively our judgment is not great. And that's not because of any one person. It's because we have a culture that doesn't really teach us how to give or receive feedback. And because of that, there's still a lot more work for us to do around learning those skills to be able to make those decisions ourselves. So AI is probably not where I would look to improve feedback mechanisms right now. The place that I would look is to changing the way that we approach feedback as an organization from the leadership level down. Because this works at the IC level, but if the CEO is still like, does anybody have any feedback for me? No, I guess I'm not getting any feedback. Then that's creating an uphill battle for the modeling of mechanisms of effective feedback. So it's sort of us to start at the leadership, at the leadership level. This is amazing because I saved for last this question, which you've just segued nicely into, which is how can we, as a project, build a culture where exchanging this kind of supportive feedback is a norm? Yeah, so this is an excellent question because it brings in this whole other skill set call that I'll refer to as socializing changes at work. Because the thing is that at most organizations, even at the leadership level, it's giving and receiving feedback is extremely stressful. And so the ways to start to change that culture and organization are either be in a position of immense power so that you can push people to do it unilaterally, or develop consensus around it and start to build sort of at the workplace collective desire to do something. And that can, of course, include leadership. It doesn't have to be adversarial. But to do that, we have to be able to socialize changes at work. So I wrote a series about this that I'm also gonna link in the chat right now. It is a three-part series. It's a little bit older. Let's see. And it is about how to socialize big changes at work. It's a three-part series. I'm only gonna link the first piece in this series. And I know it seems like I'm just like putting all my blog post links in here, but... No, no, thank you for sharing. Yeah, absolutely. So here's socializing changes at work, part one. And so that'll be really helpful for sure. And the final thing that I will link is... Actually, no, I'm not even gonna link that. It's fine. Yes, so socializing changes at work. As we... If we wanna approach this feedback culture question from an organizational perspective, we have to start by sort of getting people on board with it. Similar to what we were talking about before. I find that that works best first in kind of a one-on-one capacity. Start talking to my colleagues about this, see how they're feeling about feedback, ask them questions, find out what their pain points are, right? And once we have that, we can start to collectively kind of figure out what feedback mechanisms would work for us to reduce that stress. And I think that hopefully this talk can provide a starting point for a lot of those. And also the individual culture of the organization will have a say in that. The biggest thing that I find myself thinking about when having a difficult conversation at work or having a conversation where people disagree, I try to remember that capitalizing on alternate perspectives is exactly how an organization differentiates itself competitively. Because if everyone agrees so much that one person could be making all of the decisions, that organization will only ever be able to gather as much leverage as one person would have been able to gather. And maybe that one person is very powerful and it would have worked, but most of the time that's not the case. A lot of the best ideas come from being able to consider a variety of different perspectives. So there are two last things that I will link on this. One of them is it's not a blog post. It is a tweet thread and it is about critique. So before I was a software engineer, I worked, I studied as a visual artist. And one thing that has always stood out to me is that artists learn how to give critique. We take classes on how to give critique. We practice giving critique and then we get feedback on our critique. Software engineers don't and I think it shows. And so I wrote a thread about that phenomenon and what we might be able to learn as software engineers from artists who are giving critique. And so that's the first final thing that I wanna link and the second final thing that I wanna link is about capitalizing on alternate perspectives in the workplace. And it demonstrates exactly the kind of wins that you can get as an organization when disagreement is welcome enough that perspectives other than the majority perspective in the room can be not only considered but maybe even prioritized. We always talk about how we should make diverse workplaces because it's the right thing to do. And that's fine, I don't, yeah, okay. I also happen to think that being able to capitalize on the perspectives of people on the margins is maybe the only way to actually create a visionary product because if you're creating a product that's built for the people that the existing products are already built for that's just gonna produce incrementalist work. It's not gonna produce something visionary. And so I wrote about that here and I hope that it can give you a taste of some of the benefits that having a healthy feedback culture and a healthy disagreement culture can provide. Thank you, that's, I didn't know there'd be a lot of homework from this talk but I'm actually looking forward to it. It's optional. Homework is optional. Yeah, so I think we're at the end of our time here. Thank you very much, you really, really appreciated this. Absolutely, thanks for having me. Yeah, and I'm looking forward to the rest of the day and see you on the internet. Yep, see you on the internet, bye folks.