 Hi, everyone. My name is Surya, and I work as a Cloud infrastructure developer at CERN. So thank you, everyone, for coming to the presentation today. I'm actually doing a presentation in the field of open development. So it's actually a non-technical one, as opposed to most of the technical ones that all of you love to attend. And the crowd for these sessions are generally thin. So I'm really grateful that all of you are here today. And I promise to keep it light, since it's almost the end of the working day. So moving on to what I'm going to be presenting about, it's going to be my journey in OpenStack about what I've been doing in the past year over here. So you're going to be seeing the experiences of how a very naïve developer entered the world of OpenStack. And you'll be seeing it through her eyes of what she experienced. And hopefully, I'll be able to give some tips to the newcomers in OpenStack. So yeah, so what can you expect from today's presentation? The first part of the presentation will be about my experience and what I've done in OpenStack so far. While the second one, hopefully, is more of the do's and don'ts in OpenStack about what you should be careful about, what you shouldn't do versus what is good to do, and how you can probably feel better in the whole community as such. So where I work, so for those of you who do not know what CERN is, it stands for European Center for Nuclear Research. It's located in Geneva. And what we specialize in is particle physics research. If you attended the keynotes yesterday, you must have met one of the particle physicists. He described what it does briefly at CERN. It's much more than that. There are a lot of experiments that go on in CERN, the main one being the 27 kilometer ring that you can very much see over there in the slide. It's called the Large Hadron Collider. There are four main experiments located along the ring, which record data. And they are the Atlas, ICMS, Alice, and LRCB. These are just synonyms. You don't have to worry what they are about. If you want, you can go and read the website for more details about this. But it's just to give an overview of what happens. So this is our use case. This is what we do at CERN. And in CERN, of course, we have an IT department. That is the aim of the IT department is to mainly assist the physicists in their research. So that's not the mainstream. But in the IT department, there's the cloud team. And that's where I work, where I've been working for the past two years in this amazing team. And I actually have my colleagues sitting here right up in the front to support me today. But yeah, so we run a private cloud using OpenStack. It went into production in the year 2013. And we have deployed all those services, actually. They are the current ones in our production. The team consists of approximately 12 to 15 people, out of which 12 of them work on OpenStack. Six of them are service managers who are sort of permanent and long-term people. While the rest of them are floaters, I'm a floater. So I come, I do a project for which I'm hired. I try to complete it and then move on. And also, I have put a picture of the dashboard to roughly show the size of our cloud. We have around 9,000 hypervisors, 35,000 VMs. And a lot of cores, around 300K. So that's roughly the size of our cloud. So with this comes a lot of scaling challenges, performance challenges. That's what makes it more exciting to work at CERN, at least for me, to work in this team. So the beginning, the beginning of my journey, it was, so I was hired in September 2017 by this team. And when I applied for the job, I did not know what I would be working on. Usually, there are X number of candidates, and they are matched to Y number of projects. And when my supervisor called me, that's when he actually discussed OpenStack. And that was the first time I was here in the name of the project. So yeah. And this was my first job outside university. I didn't have any real experience outside my university education and the small projects that we do there. Initial challenges for me, at least, was all the terminologies associated with OpenStack. For instance, my first day, I still remember the stand-up meeting, where I just went. And I heard people say the words Neutron and Newton. And I knew Newton. Well, the physicist. But yeah, it was really hard to associate what these releases were and the terminologies and architecture. I didn't know what NOVA was, the CINDA was, or any of the co-projects. And I felt pretty stupid every day. The first week was almost out of blamshur. All of you had the same experience. How many of you here work on OpenStack? Like, yeah, that's quite a few. So yeah, diving into the code base again. And for me, I was hired as an upstream developer. But it was also good to get associated with a bit of downstream work. So now and then I would also assist with downstream tickets and work. So it was, for me, it was actually trying to adapt with both upstream and downstream. So I had those challenges. And the community, oh man, it was so huge. I mean, the mailing lists, the IRC, it took me a whole month to just introduce myself in the IRC. I know this sounds lame, but yeah. And my vocabulary was pretty limited. I was told that, oh, let's check out, please do check out what DevStack is and get started from there. So I sort of installed DevStack, went, and I actually forgot to describe what my project was. So after I got hired in the first week, that's when I was told that I would be working with this concept called cells. For those of you who have never heard what cells are, it's just a very simple scaling mechanism used in Nova. And CERN was actually using an older version of cells. And there was a huge upgrade that was about to happen to cells we do. This is all I was told. I didn't know what cells were either. I just thought they were the biological cells that, you know, that's the first thing that came to my mind, at least when I heard. So the learning curve for me, so as you can see in the previous slide, those were the few words that I picked up in the first week. And gradually, they actually learned, I actually made my way and now I know all those terms, at least. So this was my learning curve. Basically, I was hired for cells, so I focused on that first. There's this management, it's called a Nova Manage Interface. That's what I actually started working on. My first bug fix was on that. How many of you remember your first patch in OpenStack? Yeah, so for me, I still remember mine as well. So from cells, Nova Manage is a way of managing or talking to your cells we to set up. That's where I started and I slowly also worked on some specs which helped make it more resilient, more manageable, more performance efficient, helped in assisting the upgrade from cells we want to cells we to for stone. That was my main project, but outside that, I started with a few bug fixes and then when I submitted patches, I obviously got some minus ones. It was a bit depressing and upsetting first because I sort of felt I was wasting the time of the reviewers that I should put up better code, things like this, but gradually in the learning curve as I went up, I also had my first spec which is a very, very easy spec. It was disabling cells and it touched only one component, probably only the scheduler and Nova. But for me, it was a big deal at that time and then went along doing bigger bugs, more complicated features, and I even managed to give a few minus ones, although I'm not a very active reviewer, but it felt good to be able to give back something that I always get, usually. And yeah, so, and then from cells, I also, for the features that I ended up doing, I ended up touching the other components of Nova, which is the API, scheduler, placement, conductor, driver, compute. So it felt good to finally get a good picture of what Nova is about and it took me a year. So that was, and also one of the things that was important for me was because I was hired by Sun, my main job was to actually find out the downstream requirements and as soon as they came, try to communicate with upstream and push them. So this was also pretty challenging. It's not as easy as it looks like, especially if you don't know, if the whole community and the team is new to you and it's pretty overwhelming because people in the channel, especially the Nova channel is always active. People are always talking all the time about things that you've never heard of. So it's always difficult to chip in or contribute at least for a new person. It's really hard. So that was a big challenge. So of course the current status. After the gradual learning curve, this is where I am. So we have a huge deployment at Sun with 70 cells, almost 70 cells now split in several regions. And these are the big, the huge parts of Nova. I'm not going to go into the technical details because this is not a technical presentation. I just wanted to show that this is where I reached in the end. So working parallely both upstream and downstream, this is one of the challenges that I had in my journey because while the downstream deployment of OpenStack in CERN is quite close to the upstream one in the sense that we are always in the latest releases. We're not as old. So we are actually running Stein and some of our services are on Rocky. So it's pretty recent. The challenge is that it's the viewpoint actually. When you're an upstream developer, you are a developer. So you actually look at the project from that point of view. Whereas in downstream, there are more operators and they are running the services. So you have the challenges in terms of scale and performance and resiliency and efficiency. While when you work on your DevStack and when you're working on a feature that you're trying to fix, you often don't think of these things. So you just end up fixing a bug or contributing a feature. In the end, after you do it when it comes out, and when you run it in production, it becomes a pain point. That's when you realize that this small world of yours, this DevStack, so you're called in which everything works. You have tests and you have your own beautiful unit test cases and functional tests. But these, sometimes the real world scenario don't work very well. So this is my most favorite line that I use in my day-to-day life, but it works on my DevStack. There's something that I tell all my downstream colleagues that but it should work. It's working on my DevStack. So why doesn't it work in production? But this is always the case. Most of the production bugs are not reproducible at all. So yeah, and there's the up there that you can see the manager face. Well, that's something that the downstream colleagues might not understand. So I really feel that we should, for at least the tickets part of it, for tracking upstream tickets in the downstream world, we need to have a different system. Because often the specs take cycles to complete. So you just bump the sprint versions, like micro versions in the new API, for instance. It just goes from five, six, seven, all the way. And the downstream colleagues are just looking at you, like, what are you doing? Are you even completing this ticket? What's happening? So yeah, that's just one of the fun facts that I just put up there. Typical work day downstream. So my day definitely does not begin at 8. It begins a bit later, but I just put up the 8 to 5 schedule here. So usually it starts off with me planning my day, saying I'm going to work on this ticket or this task. And I start off with my coffee, and I start working. And then I see someone. Because I don't work remotely. I know a lot of you work remotely, but I work on site. So there's someone that just comes by the office and says, ah, but I'm having this issue. I think it's an upstream bug. And then you're like, OK, let me have a look. And then of course, you try to reproduce the bug. And where do you reproduce it on your dev stack, in which it works, obviously. There's no bug, and everything's fine. But then you tell them, oh, well, it seems to be working. I'm not so sure if it's an upstream bug. But then again, you have to prove it's not an upstream bug. So then you go on. And then most of my days, I just spend it going in loops around the code in Nova from one component to the other. And you're just trying to find out which point it is, which is the point that's failing. And finally, you reach the conclusion as if it's a downstream-specific error, then you discuss with the operator of that particular service. And then you come up with a viable solution. And if it's upstream, then you end up filing an upstream bug. Then you work on the patch. And then, well, the whole day is gone. But the aim of the day was to actually finish something which you just postponed for tomorrow. I just want to illustrate that this is how usually a downstream upstream parallel working person spends their day. Whereas a day working upstream is, I think, mixed emotions depending on what bugs you find or where you find the bug. If it's in your own work, then you don't consider it a bug. It's often a feature for you. But if it's someone else's feature that you find the bug in, then you're super happy because you give a minus 1. And so the methodology of fixing it, usually there's a bug. If it's a simple bug, then you just write a few lines. Probably the fix is five minutes. But then you spend a lot of time with the functional tests and unit tests, especially if you're a first-timer in NOVA to understand the testing suite and the framework and all the fixtures and all the integrated helpers and all the things that you have there to help you, they become more complicated. So you have to actually learn that framework before you can actually write a test. And in the end, you just end up replicating a test that someone else has written on a similar feature and you just modify that test. But yeah, so it takes double the time for the test than the code, in my opinion, and then you push, you review, you go through these iteration cycles upstream and you merge, and finally you release it. If the bug is not a simple one, then usually test-driven development is the recommended way. But in my case, even that, I write the tests after the code, which is not a... But I know a lot of people in NOVA who actually do test-driven development, so it's a good practice in general, of course. And yeah, so when you finish that bug, you fast-forward a few several months and then you come up against another bug and then you realize that this is a result of the bug that you fixed several months ago, which you don't even probably remember. So I guess this is us creating more work for us all the time, so yeah. So this is the experience part of my presentation, now coming to the do's and don'ts. This is mostly for the newcomers, but I think people have been around OpenStack for a long time. We'll also at least have a nostalgic feeling of how it is done. So the first and foremost important point is approach the team. They're usually very welcoming. And this is one of the snippets of the first conversation that I actually had on IRC. So the one month that it took me to introduce myself, this was during one of the NOVA meetings. Actually, NOVA has several components. So each component also used to have meetings, now we don't. So there was a sales meeting that I went to and I waited in the channel for half an hour. Nobody turned up. And then the next day there was the NOVA meeting and I went there and I introduced myself and then they said, oh, but we have a sales meeting. And I said, I was there, but nobody turned up. And they said, oh yeah, that's because we are not many people, so we decided not to hold a meeting. We just sat at our hangouts or something. But yeah, they were really friendly and they said they introduced me to this world of NOVA. So it was really easy to mix in because I had the support of the upstream developers. And even so to the new people, this is a very important thing. Just come to the channel, introduce yourself, say what you're gonna be working for. And after that, even when you come for the PTGs and summits, make it a point to talk because my first PTG that was in Dublin the early part of last year, I went to the NOVA room on the PTG day, the first day. It was a massive room and it was just filled. I had no seats, there were no free seats. So I was actually standing in the first half of it because it was so crowded. But it can be overwhelming, but I think the important point is to get in touch with the people who are active in NOVA or whichever project that you're aiming for. It's not just applicable to NOVA, to any project that you come to. You can always go and talk to them and you can usually lay out your use cases or say a bit about yourself also because they're always looking for new developers. It's always less resources and more work in every project in OpenStack. So feel free to come and actually talk. Community tools. So after you approach the team, it's also important that you get integrated with these. Even today, I'm not a very active person on the mailing list, for instance, but I think it's really important that you convey what problems you're facing. Today, I actually attended the presentation. They were talking about scheduling performance issues in NOVA and they were using their own scheduler. I think it's called a locking scheduler, something like this. And I actually went up to the person and I said you should actually write to the mailing list because I'm pretty sure others are also facing this similar use case that you have. So the more you talk, the more you can actually spread your knowledge also. Same with IRC, OpenDev, Launchpad, Storyboard. It's good to get acquainted with these tools. So that even in the IRC, ask the right kind of questions. I remember one of the course telling me, please ask the right kind of questions, read the documentation, don't come unprepared, of course. And submitting patch sets, fixing bugs, features. This can be frustrating at times, especially you might have to wait for a long time, cycles and cycles, your patch might get shelved. But once you get someone to review your patches and if they have reviewed, please make it a point to do continuous iterations. If you are on holidays or something, make it a point to tell them. Because if you don't respond immediately, it's going to get shelved and nobody will get, look at it for a couple of months maybe. And also, this is actually a funny picture that I have. So it was one of the micro version changes that you've done in Nova. It's a pretty painful change usually. It's a long change and especially if you have notification change, then it adds on top. So it's, after all that, when it's almost ready to merge, you find out that someone else has taken your micro version and you're gonna have to rebase your patch. This often happens. But even then, I think being patient is the only key to get things merged, especially in the big projects. So also, while doing reviews especially, or contributing patches, it's always not just work. Sometimes we also have fun and we exchange comments. So it's actually fun to do both, to also contribute and write code as well as review. And coming to the reviews part. So some of the don'ts I have specifically for the reviews, please do not minus one if you don't have a reason for giving a minus one because it's obviously not useful. I don't know how many of you have faced this. I have at least. So please specify a reason and the other important thing is if you haven't specified a reason, the person at the patch said, we'll ask you why are you minus oneing? And if they ask you that, it doesn't mean you have to state. So just because you have to state a reason, don't steal the reason from some other person. So for instance, you can see here clearly that Mr. X has actually said other commands should be prefixed with the dollar or whatever. And you just come back and say, ah, yeah, you should do this, dollar, Nova Quota, don't copy the reason from others. If you're doing this, at least don't do this always. This is a clear, I mean, that someone was reviewed it, you're just copying the same statement again and giving a minus one. This is not helping anybody in the community. It's pretty clear. You're trying to boost your statistics. Don't do that. It's annoying for the people who are writing code. Also for the silly reasons, again, don't give minus one for this. Just say NITNIT and just specify that please you can fix this thing. It's a comment message. I mean, this is nothing. So it also, you know, you lose respect of the people in the team if you do this. And especially if you're aiming to be a core contributor to the team, it's very important that you do quality reviews. Again, when you're asking questions, don't give minus one. You can just go and ask the question. Just clarify what you want to ask them. And you can also do this with a zero vote. And if you feel what you've said is correct, you can always go back and change your vote to a minus one. Don't go and say, I have a doubt. I don't understand what you've done in this part, minus one. It's going to piss off the person who wrote the code. So as opposed to the minus ones or the minus twos, the plus ones and plus twos, you can give them without a reason. It's okay. It's not as bad as giving them, giving a minus one without a reason. However, it's always better to specify a reason because one is it stays in the history. You can go back and check why you thought dispatch was okay. Or you can also, this also shows that you have read and verified what's happening in the code, especially if you want to be a core reviewer. Core reviewer, sorry. It's a good practice to actually state why you think it's good or why you think it's bad, both for the plus one and for the minus ones. Team specifics and working. This is very, again, specific to teams. It's good, like I said, the more you hang out on IRC, you'll know how teams work. There's usually weekly meetings that you can join. There's bug triaging and bug smashing that you can participate in. And one of the advices that I was given is, when I joined the team, is don't try to push your patches, instead try to solve the problem. And then the patches and the code work will get merged eventually. The first thing is to actually state why you feel your problem is important enough for the reviewers to care to review your work. If you can get them to do that, then the code will get merged eventually. Also, don't go to the IRC channel and ping people privately or also ping them using the IRCNICs if you don't know them well. First try to get to know the people and then ping them saying, hey, could you please help me with this or that? Or just ask a general question and someone will reply and then you can start off a conversation with that person. Proposing complicated features. Start off with the easy bugs and specs. Before you say, just don't go to the project and say I'm gonna work on this because they don't know if you will complete it or not so they will not invest enough time in you. So you have to form a bond with the community before you go and propose a huge feature because they're not sure if you will stick around to maintain it or they're not sure if you'll stick around to complete the feature. So if you leave a half-hanging feature in the community, they will really not even bother to start with the review. So yeah, so understand how the team works. We have something called Runways in Nova but this is because it's a huge project and things are usually busy. So you have them in queues. And for this, you can also know whom to ping for what specific thing in a project. There's always specialists. Like for instance, if a person has contributed a software rate feature in a project, go ping that person so you'll have more chance of getting replies. Know what time zone that person is in. This also helps in a way. So tips to be a core reviewer. This is one of the things that I mentioned the abstract of my presentation. The one thing that I've noticed in OpenStack is if you do reviews and if you do quality reviews, you can easily become a core reviewer. Even if you haven't, well, easy is relative depending on the project. But even if you haven't contributed a code as much, if you review a lot of patches, you can become a core. And quantity doesn't matter as much. For Nova, I think it does but for the other projects I'm not really sure. But yeah, so quality and then quantity. Attend meetings, be active, participate more. And yeah, try not to piss the team members at least not regularly. It's okay to do it once in a while. And being a valuable non-core member as opposed to a core. So I am not a core and I didn't take enough time to do reviews. I enjoyed the development side more but I think it's equally important to go and read and contribute more to what others have done. So you can still plus and minus one. So your plus one and minus one still has values. Especially if you're a downstream upstream person, there's a lot that you can bring to the plate. I had this conversation with the person where I said, well, because I'm not a core, I'm not so sure if I can actually be of any value. And I remember the person saying that but the skill that you have and the problems that you know, especially with the performance and scaling that we face downstream. These are not things that the developers usually face upstream or they get to know this quite often. So it's pretty, the value that you can bring is also huge. Depending on what you actually, yeah. Depending on what you work, this usually differs, of course. So yeah, open bugs. It's not necessary that you need to come up and take time to fix them. You can also just report bugs and there are people who are ready to do this for you. So in your own way, I think all of them can be a part of the community and contribute. And in turn this makes it more healthier. Not everybody needs to be a core is what I mean. And so this is how my journey has been in OpenStack. It's been a lot of ups, a lot of downs. Frustrations at times in patches don't get merged or minus ones. But it's also been a good journey. So far I've learned a lot and I think as a first job, it has been really good to work at CERN and to also work in OpenStack. I think it's been a plus, plus, plus, plus workflow I think so. Thank you so much for attending the presentation. If you have any questions, both on my technical work and also in general, feel free to ask. I'll be also Wilbur in IRC and yeah. Thank you.