 Hello. Welcome everyone. My name is Fernando. I'm a senior software engineer at Red Hat. I work on the networking services team and today we are going to talk about pay programming, a technique that we have been trying in our team. All right, some context. So I work on Red and especially in network management tools, tools like Network Manager, NM State, Network Role, NISPO, and all that. We are a group of six engineers, one PO and one manager. As I said, we work in four different projects with different backgrounds, different environments, different CI, packaging process, everything is different on each project. Then we are a full remote team. So we have like four or five time zones in total and the time difference is too much because we have engineers in the US, Europe and China and it's really, really hard to get them together. And then we have been starting to do our practice agile methodology. So we started with Scrum. We have a two-weeks sprint. It's quite great because we have been able to agileize everything. And well, that's some context. So we noticed several problems and we decided to address them. The first one is that we needed to extend collaboration. Well, we noticed that most of our engineers were used to work alone. That is good, but at the same time, it could be a problem. For example, when they get a stack, usually they do not ask for help or when they need something from another project or someone else. They are not used to say, hey, could you help me out? Hey, could you let me know how this works? And that's simply not efficient. So we wanted to solve that. Then we wanted to avoid bottleneck situations. Right, we have six engineers, four projects. You can imagine that there are one engineer that knows about one of the projects. Two engineers that know about one of the other projects, etc, etc. So when we have this kind of situation and then suddenly one of the engineers comes to a meeting and says, hey, I'm going to be a father. Right, so I'm going to be on leave for four months. Great, how is it going to do your work? No one knows about your project. If some issue came around, what are we going to do? So we started to learn all the projects really quickly and that, believe me, doesn't go well. And we wanted to avoid that. Also, not because of long leave. It could be also because of PTO, holiday or whatever. And believe me, when someone takes a PTO, that's the day when the customer reports the most important bag ever. Then we wanted to extend the sense of ownership. So, as I say, we are doing springs, we create a spring, we assign the task and we start working on that. So we notice something. The engineers usually solve their own task. And after that, instead of looking at the ball and say, oh, something is not complete. I'm going to help or I'm going to take it or whatever, they usually say, all right, let's take something else. So we wanted to extend the sense of ownership. So all the engineers should feel like they are owner of all the projects. So when they finish everything on the spring, they should be able to go to all the tasks that are on the screen and say, hey, I can't help you with this. I noticed that you didn't start with this. So let me take it and we can complete the whole spring instead of adding more stuff to the spring and then some other engineers not be able to finish everything. And therefore, the scrambles a little bit messed because you are not completing any spring at all. And then the other problem was the bonds between colleagues. So since COVID, we have not met. And that is around three years ago. And that's a big problem because in the end, when you are working with people, you need to trust them. You need to be confident about them. So you need to have a bond with your teammates. This bond is really important, especially when you are asking questions, especially when you are asking for help, when you need to discuss a specific technical topic. Imagine that you have an opinion about technical topic. The other person have an opinion about technical topic. If you don't have a bond with this person, probably there can be a lot of misunderstanding and you can taste things personal. It's hard to know each other well if you never see each other faces. And especially if you are working all remote and then you just have one call a week and in which you say, hey, I'm doing this, this, this and this. Okay, see you. Right? It's quite hard. And also, it's good to feel support at work. If your cat dies, it's great to have someone that say, hey, it's okay. I can take your tax for today. No need to worry. I heard about how your cat tried to feel good. So how we decided to improve this with paper whammy. So one day I was talking with my manager and he said, hey, how have you heard about paper whammy? I was like, no. And he said, take a look. So I volunteered to start implementing paper whammy in our team. So I decided to look over a lot of documents, talks, et cetera about paper whammy and try to understand how it works, what are the advantages, disadvantages, how we could implement it, how to implement it, what are the different styles, et cetera, et cetera. And after doing that, we are here doing this talk. So I hope it's going to be, my research is going to be useful for you. So paper whammy, the basics. Basically this always a driver and a navigator. So it's just that simple as two person, two programmers usually get together and say, let's work together on this specific issue. They sit and they start working on it. There's one condition, just one keyboard or one computer. So now it's not two people programming at the same time. That's something that we used to do even remote, but it's two people on a call or in person looking at the same screen saying, oh, okay. The driver is the one that manage the keyboard, do the coding, et cetera, et cetera. The navigator needs to think on the bigger picture. So for example, if the driver is designing a feature, the navigator should say, oh, but if we do this, we are going to break another customer case because we know that there is a customer using this other option, which is a conflict with this one. This kind of thinking is the responsibility of the navigator and also is reviewing the code all the time. So if you are programming in C and you do a malo, then you need to do a free. So that is part of the navigator responsibility to stay looking at the code. And if the driver doesn't do it, you need to say, hey, you forgot to do that. So, right, this is the basics. Then the switch position constantly or frequently. The idea is that they can, after five minutes, 10 minutes, 15 minutes, it depends a lot on your style and your preference. They switch position and the driver becomes navigator and the navigator becomes the driver. I must say that this can be done with experts between experts, between someone new or more junior and an expert, between two junior engineers, it's beneficial in all the situations. All right. So there are different styles. There are models then, but I think these are the more important ones. So the first one is unstructured code. Besides the usual driver navigator, which is the one that I describe, there is answer to answer for failing, which is very similar to what people do in an office. You have a problem, you are working so in it, you go just, you bring your teammate and say, hey, I want to fix this with you. Sure, let me help you. And you start working together. In this case, there is no structure, there are no rules, maybe you decide to use two computers. Okay, that's fine. The idea is that you work together. This is the simplest way to start doing it. And this is the simplest way to start implementing it in your team. So we started doing this, unstructured coding. We just get an issue. I think our teammate, hey, let's work on this. Perfect. Let's get into it. So like I said, I'm working on this way. Would you like to work with me? This is quite simple. Then we have ping-pong-pairing. So if you are a fan of testing, and you're a fan of test-driven development, if you don't know about this test-driven development, it consists on first, you develop the test that should pass, and then you develop the backfix or the feature that needed so that test passes. So basically one person goes to the other and decides to do the pay programming, and they decide to do test-driven development. So if you don't like test-driven development or in your projects, you don't have tests, you should, but if you don't have tests, you cannot do this. Then the first person creates a test that is failing after that as a driver, and then the other person is the navigator, and after that the second person, the navigator, changes position and creates a code or the implementation that makes the test to pass. So then they switch position again, and this time in this case, the second person creates now the test failing, and the first person creates the implementation so the test passes, and they repeat. This is quite good because at the end you have the feature or the backfix and tests for them. So you are doing two work in one. I must say that this, from my point of view, it's more adequate when you are working between two experts, because it could be quite hard for a newbie to create a test for a project that they don't know, and the test must fail, so it's going to be quite confusing for them to create something that won't work, having on mind the idea that in the future it will work. But between two experts, this is a really, really good scenario. And then we have the backseat navigator, which is perfect for a junior or a newbie and an expert. The idea of this is quite similar to a driver navigator, but in this case, navigator is not thinking only on the bigger picture. It's thinking on what is the driver going to do. So it's like when you are driving and you have someone by your side saying, oh, now to the right, to the left. Now go straight. So it's basically the same, but programming. So the first person to the other seat together and do a paid programming session. But this time, one of the persons, for example, in this skateboard is going to be the driver and highlights the navigator. And the navigator is going to give direct instructions to the driver. So this is really, really good for newbies, because if the newbie is the driver, it's going to get used to the workflow. It's going to get used to the files, to the structure of the architecture of the project, to, I don't know, the CI, to the tools that you are using, et cetera, et cetera. And they also found that this could be quite good for an interview. For example, imagine that you are interviewing someone and you say, okay, so you are the navigator now. Please tell me what I need to do. And that could be great. These help a lot newbies on the team. So these are the three main ways of doing paid programming. But now we have some very, very important considerations. The first one is that both programmers must be active. I mean, they must participate actively, because if not, you are basically in a mentorship session, telling them or explaining them something, and the other person is just in silence. That's not paid programming. Or if the other person is you're a programmer, you're the driver, and the navigator is looking at the email, believe me, that is not paid programming. This is like you are programming alone, and there is another person in the Google called listening you programming. That's all. Both must keep a running commentary. So it's similar when you say paid programming, it's also programming out loud. So when you're at the driver, it's quite hard to, for the navigator, it's quite hard to understand what are you thinking about if you don't talk. So the driver must always describe what are they doing. So yeah, so I want to implement this. I want to create a function that is going to be added to this class, and this class is going to be used in this place, and we are going to add these events to the loop, and this way, the navigator is able to think about your ID and say, oh, but then if we add this event to the loop, it's going to fail because blah, blah, blah, blah. And then the other thing is that relationship between the programs are important. And managers, leaders, or whatever person is implementing paid programming on the team, must not force them because if you have two engineers that they don't get along, believe me, it's not a good idea to force them to work together in a paid programming session because they are going to have conflicts and they are not going to be good at resolving them. So it's really important to first solve them. All right, so the result of our experiment. We have been paying programming for a couple of months, three months or something like that, or even more. We have been doing regularizations, and initially it was quite hard to get people used to it. We proposed it and it was like, yeah, we could try it out. The people were not very convinced, but then we kind of forced it like, yes, let's do it. And I volunteered to that, so I started to propose sessions to several engineers, and I assumed as they are kind and polite, they didn't reject them. I think they were considering it, but they didn't do it. But after a couple of sessions, we got positive feedback and they say, hey, I like this. I really enjoy it. It was good, and I would like to continue doing it. So now the engineers are starting to propose them to all the engineers without my help or without any other people help. So that's a great thing. And we make sure that everyone in the team had a meaningful paid programming experience. So that means that almost everyone were able to have at least a session or a couple of sessions to understand that. We are also flexible, so that means that we are not going to force anyone. So if one engineer says, oh, that's not for me. I don't like it. That's great. You don't need to do it. This is like using Beam, Emacs, or whatever. It's just a tool, and you use it just if it's good for you. And also, all right, so some advantages. So first of them, the knowledge sharing. It was incredible. I mean, doing this, I get a lot of experience in all the projects we already manage, which is quite great. And all the people, all the engineers from the team doing this are getting that knowledge. It's really interesting to see the different scripts that people use, how they build the system, how they build the package, the CI. It's really, really good. And it's really nice to have someone explaining to you while they are working on it. Because, yeah, it's really easy to say, yes, read these docs. And the docs, you see, last time modified, 20 turn. Right. Okay, so then we have focus mind. Other night, it happens to you, but at least it happens to me. When I work alone at home, I'm on the computer and say, hmm, what's I'm going to eat today? Oh, in the afternoon, after I need to go to the grocery. Yeah. Oh, look at the dog. Hey dog. So it's quite hard to keep focus for a long time. But when you are on a meeting and you are working actively with one person, that's changed because you have a person. And so you're not going to say, hmm, what am I going to eat? No, you are focused because you are wasting also all the people's time. And you think, oh, but this, then I need to work more because I'm going to be more focused on the job. So you're going to get more tired. Yes, but you are going to do the work in less time. I mean, if you really focus on the task, maybe you can do it in one hour. But if you are thinking in all the stuff, changing context all the time, maybe it takes three hours from you or two hours. Then we have resilience to interruptions. I know that your email probably looks like this. My one looks like this at least. So when I work on something, I get a slack message. I immediately go to it. I read it and I start engage a conversation. Then I go back to my to my work and then I get another slack message. It's terrible. And when you are on a meeting, working with someone you're talking all the time, you get this as a slack message. And as you need to keep talking with this person and focus on the job, you are not going to read it. And then you learn to ignore them. And this is one of the best things that I learned doing paper. I mean, yet you just learn to ignore all these messages. And one extra thing, it's the code quality because at the end, there are two people looking at the code and four eyeballs, see more than two eyeballs. And we use the coordination efforts. So when working together on a paper organization, everyone were aware of the context. So, all right, so this issue is blocked because the last time we work on it, we noticed this problem. And then if someone has, oh, what is the status? You don't need the only best of working on the issue to reply. You can because you already know the context of the issue. So we use the coordination efforts. But we notice some disadvantages. So the long time, right, it's good to work with peers. And it's really great, but you also need a long time. You need time to reply to emails, phone calls, whatever need you have on your position, or even work on a simple coding issue by your own. That's great. So I don't recommend you to do 24-hour pay programming sessions. So translate all your work to pay programming. I think that's a terrible idea. That's not going to go well. You're going to get banners immediately. So try to do not do that. Try to set a amount of time that works for you and make you feel comfortable. Then Fadi, so it's easier to take last time when you are working alone because you don't feel like a pressure to continue working. So you say, okay, I'm going to drink some water. You go to the kitchen, you relax a little bit, drink some water, think about the problem. That's great. But when you are with someone that you might feel uncomfortable saying, oh, wait for me. I'm going out for some water over there. So in order to avoid this, I recommend you to create like if you are going to say we are going to have two-hour session for one-hour session. Okay, so let's schedule one five-minute break after 30 minutes or 10 minutes break after one hour. So this way you have time to relax and then come back with more energy. And skill levels. A skill level could be opponent. If you put one junior engineer with a really, really experienced engineer and for someone that is new on the team, the new person is not going to be able to speak up. So they have a great idea and they are not, they don't feel confident enough to say, hey, I'm the navigator. I don't think what you are doing is wrong because the other person have been working on the program for 15 years. How could I know more than this person? Sometimes fresh ideas are good and you could have a different mindset that this person and you can bring a lot of value to the project. So in order to avoid this, it's really important for the expert on the session to promote the behavior of speaking and make sure that the newbie or the junior feel comfortable enough to ask questions and also challenge the idea. Like if you say, for example, all right, so we are going to do this function and add events to the queue. And the junior is the navigator saying, let's say, why do you think this is correct? Or what do you think about it? And force them to reply, force them to provide their opinion. Maybe it could be, I think it's correct because blah, blah, blah, blah, blah, blah, and then it's accurate as well. Okay. But maybe they say, no, I don't think it's correct because this is going to have a conflict with all the part of the code. And you think about it and say, oh, that's right. And the first time that that happens, the junior feel more and more confident. And the next time you are not going to force them to reply, they are going to reply by their own. And that's really great. Conclusion. So we were able to extend collaboration. People feel now more comfortable collaborating with each other, which is quite good. And we avoid bottleneck situations. We were able to understand the context of the whole project. So people now can contribute to different projects. And if someone goes on PTO leave or whatever, someone else can take their position. And one thing that went well is that then the sense of ownership, it didn't, it wasn't fixed at all. We improved a little bit, but we are still, we are still struggling with this. So we are trying to fix that. Maybe next year, there's another talk of how to fix that. I don't know. And then one thing that improved a lot is the bonds between the colleagues. So thanks to pay programming, I learned a lot of things about my teammates. And I mean, personal things like, oh, what, do you like to play an instrument? Oh, I do this. Do you like video games? Do you like working? Do you like, and then you notice that you share hobbies and you share information about these hobbies. And I found myself talking with them about random stuff, hobbies, or whatever. After work, like, hey, I saw this movie and I really like it. And you like the other one. So maybe you like this one. So that's great because now you have a better connection. And it's easier to, when there's a problem, it's easier to challenge the idea and to share different opinions and get an agreement because you understand the other people context. And also because you have personal bonds with that person. And usually you care more for other people's feelings and other people's ideas when you have a bond with them. Maybe the solution for a lot of companies have a bond with Casino. I don't know. So all right, what I recommend is to try it out. If you want to try it out, we have a workshop of pay programming today. So please feel free to join. We are going to, we are going to have some hands-on experience. And all of them, that's if you are on our team and you are a manager, a software engineer, team lead, code owner, whatever position, and you want to try this out, feel free to reach me out or follow these advices. Don't force people if they are, if they say that it's not, they are not wanting to do this. Try out first unstructured paving because it's going to be much easier to implement. If they like it, then you can change to ping-pong, you can change to whatever other method that you would like more. And yeah, the last one is that make, if you are a manager and you want to implement this, talk with one engineer that's have some interest on this. And say, okay, you are going to promote this on the team. And this person needs to see some talks, read documents, et cetera, et cetera, gets a formal opinion and try, try it out then on the team. All right, so I think that's all. Thank you very much. So any questions, suggestions or comments? How does it fit with current review? With what, sorry? But with current reviews, merge requests and so on. Right, you counted it and see if it works as your report. Oh, okay, perfect. Yeah, so the question was how it works with code reviews. So with code reviews or merge requests, portal request reviews, it works well. I mean, you can, in the end, you can apply pay programming to almost everything. You can apply pay programming for a feature, backfix, or even debugging, which is quite useful as well. But also for code review, I mean, imagine that someone, there are three persons working on a project and two of them decide to review the other person code. It's quite good because you are both of them at the same time. So it's parallel review. So that's great. And at the same time, you can do it with a person that did the code and ask questions. So you get immediate feedback when you write down a comment in GitHub, like, why did you do this? Then the next day, the person reply, I do this because blah, blah, blah. Oh, I don't understand exactly why, what do you mean here? So it's a really slow process. But if you go to this person and say, hey, let's review your code. Hmm, you read it? Okay, so why did you do this? Oh, I do this because whatever. Oh, really? And why? Then you get immediate feedback. So that's completely Yes, I meant a slightly different question. Oh, sorry. Do you regard the code which is produced through pet programming as being already reviewed? No, because you've got to navigate to review and get one of the driver rights. No, when you review the code, someone else out of the pay needs to review it. So because also when you are working on the issue, you can get biased, like you're working on that. So it's always great to have many reviews as possible. You can get 10 reviews. If that's possible, please do it because usually someone is going to spot a bug or a problem in the code or whatever or something that could be improved. All right, thank you. I have more questions. Yeah, please. Is that problem if you can't finish your task within the peer programming session? All right. As I mentioned, it is only one hour. You do it and the next time you're going to start in the middle of a task and then probably going to be confusing. Yeah. So the question is that if it is a problem, if the people cannot finish the task in the paper running time box. So yes and no. I mean, for example, you're going to start doing paper running for one hour. You didn't finish it. You can continue by your own. That's completely fine. Then you get more reviews. That's just that. Or maybe you can stop, take some notes. Like if you were coding alone, when you're coding, at least on my case, there are some times in which I sit down, start working and I don't finish the task. So then I need to start again in one day. That's also happens when you work alone. So it's the same situation. It's both solutions are fine. You can just continue with the pair in another day or in another time and just think what we did. Oh, we did this. We implement this. And sometimes that is also useful because you read the code that you wrote together and say, oh, but this is wrong. So then you just call that problem because you have a fresh mind and you're not tired of coding. And also you can just continue on your own. Right. Please. So you mentioned that the navigator and the driver would switch positions every 15 minutes. How do you handle like like different maybe navigator and the driver has different IDEs? How do you, the stuff that you have been working on is local on the driver's computer. So now we have to like say you have like a draft branch or something where then the other person pulls and, you know, continue working on it. All right. So the question is, as I mentioned that driver and navigator switch, how do you do it if they have different workflows or they call this local or different tools? So the thing is that it's, you must be flexed. So depending on the situation or what you are working on, you can switch every 15 minutes, maybe 30 minutes, maybe you switch each task. So you have three tasks scheduled and you say, okay, so the first one you're going to be the driver and I'm going to be the navigator. Then we are going to switch for the next one. That's also an option. The thing is that you need to adapt the method to your project, to your team, to your situation. It depends a lot. As some considerations, I didn't say and it's really important to have a stable internet connection because if not, that, if you're remote, that won't work. If you're embarrassing the office, that's fine. So as I say, it's just try to adapt it to yourself and to your team. And as I say, maybe I describe three styles. Maybe you think, oh, but I like to do it different. That's perfect. I mean, that's completely fine. Do it as you prefer. All right, so I think we are out of time. So thank you very much and I hope you enjoy it.