 Right. My name is Ashik Khan. I'm from Entity Docomo and I have Ryota Mibu from NEC and Julian Dhanju from Red Hat. And three of us from three different segments of the industry will share our experience with you on how to work upstream with OpenStack. So, well, from the presenters, again, the segment of the industry point of view, I'll give you the operator's view. After that, Ryota will give you the telecom vendor view, and Julian will share you the developer's view in working upstream with OpenStack. For my part, one thing is I do not imply anything or do not trying to give you any advice. Through my work with OPN and the doctor project and OpenStack, I'll just share with you a few things. I thought was helpful towards me in being successful in OpenStack. I think in the keynote, you saw the demo of OPN and the doctor today where Mark cut the cables. So, that implemented telco grade fault management feature in OpenStack. We started that project two years ago. I'll basically share you my experience of that two years. So, from operator's position point of view, my role basically is to convince the community about the use case that what I'm trying to develop, what I'm bringing on the table is quite useful for not only for DoCoMo, my company, but for many other companies. Let's put it that way. And then I need to find out developers who will help me writing the codes and implement it in OpenStack. These two are my primary roles. So, the use case. This is somehow somewhat a complicated picture. But I come from a mobile operator. Obviously, I'll bring a requirement that is useful to mobile operators. I'll not explain you the figure. This is 4G mobile core network. Each of these nodes, the rectangles you see, they basically host few hundred thousand mobile phone sessions. So, the point I want to make as soon as you show this figure to a mobile operator, someone from mobile operator, and you explain them the problem, they instantly understand. They understand the severity of the problem. They understand the necessity of solving this. So, each of these nodes, as they're hosting few hundred thousand subscriber sessions, if one goes down, few hundred thousand mobile phones is their connection, they try to reconnect. Going down is still okay, by the way. They try to reconnect instantly, creating a congestion no one can handle. You cannot recover the node, actually. So, the problem, this problem as soon as you explain to someone from the mobile operator, they instantly understand. And I use this slide a lot in different presentation. Along the process, the opnv doctor project, we have five largest, well, some of the largest mobile operators as commuter and contributor in the project. So, you've got to have a decent amount of user support for your project. So, you explained the use case very well. You should do that. I did that. Now, that's what the mobile operator understand. And that's a very large scenario. That's about high availability of a mobile telecom system. But what exactly the problem I'm trying to solve? I cannot solve everything at the same time. There are many other problems. The feature I want to develop is basically this. So, you have, let's say, two green telecom nodes. And the five-nine story you heard in the keynote session today is in order to, well, let's say, ensure five-nine, you will have a hot standby. There is no other way. So, just follow the animation. If there is a fault in the infrastructure, the cloud management system, Veeam OpenStack, needs to collect that fault information instantly. It needs to find out who we should inform about the fault. And it needs to send out the notification in real-time, possibly as soon as the fault was found. Now, the VNF manager, application manager, it switches to standby. And the standby becomes active. So, that's how you actually ensure five-nine. This is the whole feature set I need to develop. Now, that's, again, still quite a large scenario. I need to scope it. You have to scope your work to, well, preferably in a quite small problem space. And I'm using the HCNAV end-to-end architecture, because for an operator, the end-to-end view is quite important. And the flow I showed in my previous slide, if you fix the scope to the cloud manager, Veeam, virtualized infrastructure manager, it basically boils down to one node-bound interface, which I have shown as ORVI, which will send the notification to upper management system. And there is the south-bound or, you can say, westbound interface, which will collect the fault. Basically, basically, in principle, only these two features. So, it is important to scope your work, and it is also important to use a reference architecture or system in order to socialize with a wider audience. Using a reference system is very helpful. Now, from here, I try to reach the open-source community. In order to do that, on telecom, I speak a different language. My specification, everything I've written in different terminologies. And this is, I actually did, and I did it because of the feedback I got from open-source community. So, it's talk open-source. I have, again, used the same HCNAV architecture that we use, and we already commercialized that in our network. But I do not use, let's say, NFVI or, let's say, cloud manager, virtual infrastructure manager. I have changed them to something more comprehensible to the OpenStack community. So, where exactly the positioning of OpenStack instead of using VNF, VMs, what people will easily understand. This translation work, I would say, is necessary and was very helpful. Now, you have to find out resources. You have to bring resources, you have to find out resources. And different skill sets are available from the different segments of the industry. Operators can give you a practical requirement. And the requirement I brought in, the OpenNV Doctor project, it came from our everyday life operational experience. We used OpenStack in our network, and we found out that this feature was missing. And then, if it was missing, it has to be custom built, expensive. Well, let's try to do it in open source. So, operators, network architect can give you an end-to-end view. And when you are developing a microfeature, it might have different kinds of, let's say, impacts on its neighboring functional blocks. It's better if you have a network architect who can actually solve these things on an end-to-end basis. You need developers who will actually write the codes and develop it. After that, you'll need the system integrator vendors who will put the solution in, again, a complete end-to-end commercial system. And then operator will, again, deploy it. My view is, looking at the contributors and commuters in the OpenNV Doctor project, for a successful feature development, I would say it's better to have all of them. It's not, well, let's say it's not necessary, but it's better if you have all of them. It will make the process from requirement development to actually commercialization quite fast. Engage your upstream community. It is very important. What I'm doing over here, actually, participating in events, you need to find out who is interesting. It is developing a decent crowd behind you who support the use case and who thinks it's necessary. Reaching out to the developers, find out key contacts when you are working through the corridors, invite, let's say, Nova Core Developer or a Neutron Core Developer to your breakout session. We did it. We just went to them and said, can you come and join one of our meetings? It would take 30 minutes. They gave us very valuable advice. And actually, some of the time, they just straightforwardly say that it's not possible. So we changed direction at an early stage, actually. And, well, it is also, again, finding out resources who will do the work. A good start point is to find out in your own ecosystem. So I'm a telco. I have a telco vendor relationship. Vendor has a vendor-to-developer relationship. From there, your first contacts, I'd say, will come from there. And I actually found out many interested people through my presentations in different summits. So a multiple representation in OpenStack Summit actually matters. This is my summary from an operator's perspective. To summarize the whole thing, socialization is the key. You need to talk to a lot of people, find out the right people who can help you, will write the code, who will also influence the open source community, let's say OpenStack community as well. Find out a decent crowd behind you. Find out support from your own community. If you are mobile, I'm mobile telecom, so find out more telecom supporters. And this is important. Availability is a very large topic. Keep it small one step at a time. So developing a southbound interface for one particular feature and a northbound interface for one particular feature, basically sending one message out, is not that difficult. When you present that, take it to the open source community, people will be reluctant to send out. So go step by step and do care about the existing code base. Don't develop anything, well, if you develop anything that has a negative impact on the existing code base, it will not be accepted anyway, but better not take it the first place. It creates a good impression about you. And lastly, one point I want to emphasize, there are ways you can bring your requirements to the open source community. You have different working group, you have Etherpad, you can write your requirements there, but the point I want to make, it's better if you do more. Just writing the requirement there and then disappearing and hoping that someone else will develop it for you. That will work, but it will work faster if you yourself participate in the actual work. That's all from me. I'll hand over to a telecom vendor, Ryota Miibu from NEC. Thank you. Thank you. So I'm Ryota Miibu, working in NEC and also OP NFE doctor project. And also I'm core of the OpenStack alarming project, AODH, or A. And so I'm standing between the telecom operators and developers who are working in the OpenStack. And telecom operator has a requirement, a use case, and it's very huge and sometimes bad, but we bring those requirements to the developers and this is not the one-way communication. We have to have some discussions back and forth and checking what is the problem and what could be the solution. So I hear these three messages when I run during the past two years. So first you can find a requirement use case while you are supporting operators to develop their desired features. Then after that you can find out solutions by interpreting operator needs and developer ideas back and forth repeatedly. And then you can do some integration. Then integrate software component into the huge telecom systems and that time you might find some bugs or some other things. So then you should have still hanging on the OpenStack or open source community, upstream communities and improve the code or OpenStack or softwares, whatever. And then at that time you still have to remind that you should bring the problem and solutions or code comes later. So that's the generic my messages. And then I'd like to share some of my experience that I found during the OpenIP Doctor project. So first we learn the problem from the operators and they have very clear use cases but not dictated as solutions. As you can see, Ashik mentioned one orange block that was we called BIM and it can be covered by OpenStack. But as you may know, OpenStack is very huge. There are many components in it. So we have to identify what feature are they going to use. And then we try to show the problems very obviously by having some tests. So you can see some slide up there. So the upper one is taking a longer time to get the service back to the original one when the failure occurs. And the bottom one is improved one. And we still have time to wait for new VMs coming up but it really shows what they think about the problem. Because when I showed this slide to the operators, they said it should be fine. They still have active standby node. So when the failure found immediately, then they can switch their status of the VM or the application. And then the service can continue. So yes, so we have a lot of conversation with them and asking operators if the problem analysis is correct or not. And then we have a lot of discussion with the developers and during that we really think about the gap between Tereco and Cloud. So Tereco has more like a suit style and developers like jeans and t-shirts. And the mine is so different. But they still have the same thing. So for example, Tereco is saying we have to find out failure and we have to switch over to the backup VM and the main service running. And Debra also says, so what we need is to translate their terminology and figure out what they are talking about and make sure they are on the same page. So as you can see on the right bottom, we have many dark blue boxes and that are covering one big orange box that you saw in the ASIC slide. And then we talk about how we can make the better solution. So for instance, asking some salameter cores, developers, hey, how about this feature? Then we can find failures very quickly. Maybe we can implement this way. Then they said, no, this is not good and not the right direction of the project. And then we have many conversations and finally we got the solutions and implemented it. And we also have more discussion with the Nova folks and neutrons and also we get great support from Congress. So that was awesome. And then we can figure out what is the solution can be across various OpenStack projects. And at that moment, we already jumped into the upstream committee, but after we implemented missing features in the upstream committee, we integrated them and having a system test or functional test in our own environment or we also have our own CI pipeline in OP NFA. And then we have integrated those features and run test databases and find out some bugs. When we find out such bugs, we put it immediately to the upstream committee and try to fix with them. That's very nice to having the same direction with the upstream community. And yeah, so that was awesome. Sorry. So then next idea is that joining code review or some other project activity, then you can learn the software itself and also the project direction. And then you can also find the key developer who should talk about the specific features and then you can send a patch with some discussions with the folks and skip this one. And this is my final slide and I also want to ask you to think about the upstream. So I think upstream has two meanings. So one is something you are relying on. So you should report a bug or you can ask developer to improve the features something like that. And that is one of the main motivation that we have and we did many upstream communications and development. The second one, if we think about some libraries, so you might know many libraries, Python library or GNMC, whatever. So good developer knows good library and sometimes develop their own library. So it's really nice to have more better modular software architecture and I saw that the person for making Python libraries was so quick and they also reduce many code in the open stack branches and first time I saw his patch I was very excited that he did many lines of code and just inserting one line and at that moment he already implemented some new Python packages and the person is Julian. So I would like to pass to Julian this presentation and let's run more. Thanks. So I'm Julian from Red Hat. I happen to be the PTA for telemetry with Cycle. So first thing I want to talk about is the usual things we see in our projects from people coming, I would say, from the outside and talking with us about what they want to do in the project, what they want to achieve in telemetry. But it applies a lot to the other open source project in open stack or inside of open stack. So there's a big gap between the culture of the enterprise and cooperation and the open source one. So you see that less in open stack in particular because there's a lot of companies engaged in open stack, but that's even more true in the rest of the open source culture. So for example, meetings, it's very natural to trigger a meeting, like Ashik said, we just triggered a meeting and people came. That works pretty well in some open stack projects. It works well less in other open stack and other open source projects when you don't have meetings because you can maybe see people. So it's a different mindset to have, you may not be able to regroup people in the same room and talk to them. For example, a thing that I always find weird even nowadays is when people want to do a meeting with me because I'm the PTA over Skype, I'm like, what? That's weird. That's not the open source way of doing things like send me an email or talk to a friend on my list or something, but doing Skype meetings doesn't really something that is open source. You have to take that into account, for example. There isn't less notion of customers in the open source. We have users, we don't have customers. So if you start talking about your customers, that's your problems in general. You have people and developers working in companies a lot in the open stack community, that's true. So that helps a lot, but in general, we assume that all projects have customers. So that's something we in hierarchy is very different. I think it's pretty obvious, but there is no boss in open stack. And especially not the PTA, so don't always go to the PTA because it's the boss and it's going to solve your problem if you're not happy with a developer or for a feature or whatever. And that's one of the main problems that I have as a PTA is that people always come to me with their problem and I'm not a problem solver. I'm just the guy doing the liaison for the project, the vessel, and that's not a good job even. It's like always people coming and winning, winning about their issues or a problem. It's not helping anything, so don't do that. It's a terrible thing. We see less nowadays, fortunately, but it's still true sometimes. It's cool to go and see and talk to the PTA as a liaison and it will redirect you to over guys in the community. But then you need to engage and to put efforts on your side. Catalan versus Bada, that's an old, known concept from the open source, which is a different way of thinking of things and how you build them, where open source communities tends to be very less organized on a corporation. There's not a lot of bureaucracy in general in open source, but it might be less true nowadays in open stack, but still in general there's not a lot of processes or things that either are key, basically. But it does not work. Same thing about individuals and corporations. People tend to think they're in an organization where there's a lot of people, even in open stack, who see themselves as individuals working on things that they're passionate about or interested in at least. So that's some things that you really need to understand for jumping into open stack and talking to people. Otherwise it's a bit weird and you might end up being rejected just for that without understanding why. So understanding the model of open source, that seems weird too, but we also have people not always understanding the problem, for example, the licenses or back then when we had Stackforce. So, for example, an anecdote, we have in telemetry as that's a few years ago, somebody wrote some background code and they sent us the code and they said, well, so I wrote this code and we said, oh, that's very useful for telemetry, thanks. But I work at a company and I don't know which license I can use, so I'm going to go and see my lawyers. So the guy goes, he has lawyers and he says, okay, my lawyers, okay, I can put my code on Stackforce with the Apache license, that's cool. We say, okay, that's what we want to do, so that's okay. And then we start to put the code there and a few weeks later, we change our mind and we say, well, we don't think Stackforce is a good place for you, I think OpenStack directly is a better place. The engineer says, oh, I don't think we can do that because the lawyers only authorize Stackforce and not OpenStack. And we say, well, it's Apache license so we can do whatever we want with it. You don't have to ask your lawyers, you can ask them but you don't have to. I mean, even if the engineer understood that, the company itself was not really to understand this kind of problem with licenses and what you really allowed to do, et cetera. So it's something pretty important to know. Sometimes you understand that your company behind does not understand everything about licenses and the model behind OpenSource. Like Ashik pointed out, if you have only IDs and you write that on a TAPAD and then disappear or don't have any resources, being most of the time developers, sometimes hardware to offer is not very worth talking often to people. We had people in telemetry for months talking to us with IDs, which were sometimes good IDs but no resources to offer. So you can say, it's a good ID every month but at the end of the day, if you don't have anybody to write the code, you're just wasting every runtime. So if you have a lot of IDs and you see that you count up and they are well received, it's cool but the next thing for you is to find people to add write codes or whatever is needed to move on. As Ashik said, to explain your ID so when you are in an NAV context, for example, I don't know a lot of things about NAV and telco. So I think somebody like Ryota coming in and explaining to us what the use case is, what was the business context. Not everyone. Opposite is very large. So there's a lot of different businesses, a lot of different use cases that might be covered so you can't know about everything. So you have to change your term sometimes or split a problem in smaller problems to be able to explain that to people and not bring with this is my problem deal with it. So what you really want to do because I started with everything that we saw was bad is to observe so that's one of the key and that's always what I did for the last 20 years doing open source. The first thing you need to know is to observe the guys you want to work with. That means subscribe to the mailing list, read the topics, try to find every resources there is on the project because going to a project and saying, oh, I have this idea, it's awesome. We should do that and then the developer says we already had that for two years. You look stupid now because you did not do your own work first and that doesn't really help anyone. So do your own work first and observe along from your community, along who's doing what to such an extent that you can because sometimes it's a bit blurry or the community is too big but a lot of the community, even an open site, are very small, it's very easy to find out what's happening. So observe and learn from the project if there are documentation with them. Sometimes there's not, so that's not your fault. Next thing before trying to push new features is to try to help. So that if you have resources and developer that's the best thing that you can do is to send a few people to try to help. If you learn and observe a project most of the time you're going to install it and you will find bugs. So the first thing you can do is to try to report bugs and work on that level, et cetera, et cetera. It's very easy but it's way better to build trust this way than to jump into a conversation or into a session and say, oh, there's some feature ID of this problem I want to solve and this is the way we're going to do it. This won't work. So it takes time to build that but it's the best way to enter a community and be part of it. And another problem we always see especially for new features is that people tend to really want to solve their problems which is a very narrow minded view of the general problem and that's often be rejected by the community because you have your view of the problem that the developers, they're here with you and they're also here with another group of users with a different set of problems and sometimes these problems overlaps. So they need to generalize the problem so if you can help with that to generalize the concept behind your problem and solution it helps a lot of people, developers, to be able to write code that will solve your problem. Training engineers So most of the time when you send engineers especially to an open stack they are not trained to do open stack. We still have this problem today in telemetry in a lot of other open stack projects where we have newbies coming and trying to help us but not in a good way because they didn't start by observing or asking how they can help the community trust that we need and they're just flooding us with things that are useless and so the worst thing you can do is hire a developer and send them to a project and say, okay, do open stack now and we see that a lot these last cycles and it doesn't have anyone so don't do that either it's pretty bad. You need to train them and to explain exactly what I just said what is open source, how it works how the community works what is the culture behind it's not your company culture it's open source, open stack culture which is different, like I said Do not try to recognize if you find somebody, some group some developer, some PTA, whatever but you don't like or whatever it's okay, it's humans after all just switch take another route go to another project somebody different don't build conflicts so never do that humility is a very important thing in open source in general so terrible thing we saw too is that you saw the developers and you start building trust and then you say everything you did guys so far it sucks I can do everything you did in a better way with a better language, technology, whatever that might be true but that doesn't work either so being humble because in your ideas, concept solution, whatever, it helps a lot to build respect and trust from the other folks working on the open source project that's something I try to apply for the last years doing open source if you don't try to be a jerk rewriting everybody's code for whatever reasons, it helps a lot like I said not only the PTA I wrote a blog post a few weeks ago and that's not for people wanting to contribute to open site and open source that's for people doing open source and I saw a lot of bad practices on the other side too it's not always people trying to contribute terrible things sometimes you can also help to contribute to the project by having a different behavior and to be more open minded or things like that so I won't talk about that now but if you want to take a look at that you might learn a few things and that's it we take question if you have time yes we have question we have 5 minutes if you have any question guys shoot, I don't know if you have a mic or please use mic otherwise we cannot record I'm from Red Hat also and I work with the upstream guys also but I understand the concept get an idea, get a business logic for it get some motivation to do the idea but how do you communicate with those guys with the upstream developers IRC mails, how long does it take when you get an idea and you get someone interested in it it's very depend on the community first so you have community that are going some are running IRC meeting for example some are not I said need to observe and to see first of the communication there are some communities that like to meet face to face very often and so that might be a good idea to join at this point there are some like telemetry, we don't do that a lot we don't run IRC meetings anymore we are on IRC a lot of projects, I think all of the open site projects are on IRC we use a lot the mailing list because it's asynchronous it's easy, you can reply when you want you can think about what you are going to reply which is different on IRC it's pretty easy to see that you just google it you run through the open site where you have a page about our project that might be different from Nova but that's always documented somewhere worst thing you can just find a developer worst thing what I do outside of open site you just do git log and you check the git log and who contributed and you sort of mail I'm sorry, where are you guys hanging out and how can I reach you is there a mailing list or something and they would say yes or no we only use GitHub or whatever and it's as long as you stay humble and just asking nice questions and doing your homework everything is cool and you're just building relationship and trust sorry what I can suggest it was easy for me because we are part of OPNFB which what I want to say that it gave us a recognizable entity so means we were having meetings at the beginning maybe not all of us had a very good idea how open stack works but as it gave us an identifiable entity I mean an address or an identity we could invite people to our meetings and that was a very direct communication so invite people means that's a novel core developer or congress core developers and they could like we explained our use cases straightforward speaking and when they agreed they actually helped us they themselves became our kind of ambassador they wrote blogs that so far the way we are doing it it seemed that there are other compelling use cases when you explain the use case very well because if it's a rice use case the people generally will not deny it so that's something we did we directly invite people to our meetings I also have to comment over that so if you propose some new features it might take more than one year but sometimes less than one month or even one week it really depends on what feature is or how many supporters are around their features and it's really hard to say but one thing I want to say is just be honest or be good to communicate with the developers and try to explain clearly and help their work it's really nice to make a good connection for earning trust from him or her that's it Any more questions? Thank you