 So, what I mean by that is actually, you know, we wanted to enable all of our, you know, engineers who are working in our operations. We want them to go and innovate more and more. And agile development model. So, basically, you know, what I mean by that is actually what happens when you use some of the vendor products. So, we have to wait for their longer release cycles. Basically, our application needs some specific feature. But of course, you know, they have their own, you know, constraints they cannot, you know, release it. Of course, their quality and all those things. But some of the things that we wanted to calculate ourselves, then we said actually we wanted to, you know, deliver every two weeks. If the application needs a functionality, we should be infrastructure. We should be able to be able to deliver within two weeks. So, that's what actually we got some guiding principles. Right? So, what happened actually, you know, by comparing with all this, actually, you know, OpenStack, you know, we chose. Even though actually we have our, you know, cloud that we are building in knows everything. And we chose to have, you know, OpenStack. It's a, you know, big decision as a corporate actually we took. But, you know, I know actually people are talking about, okay, how we are going to make it happen. You know, if you keep talking about it, how we are going to go and do that. So, that's the philosophy that we wanted to add up. But of course, in the past layer, actually, we are still evaluating some of the technologies. We have past. It's not that we don't have it. But we wanted to, you know, go and build ourselves or go and adapt some of the open sources or build, you know, along with open source community. That's the debate still going on. But actually we already made the decision for IAS. It's not only for Paypal. It's for entire reviewing companies, right? And, okay, again, so why OpenStack? Okay, we chose OpenStack. But of course, you know, I have to go and justify our, maybe my colleagues in architecture. We have to go and justify management. Okay, what is it going to make difference? Because I'm already, I'm running my business, right? And, you know, we have some closely known success stories that we can, you know, take it to the management. Extract commerce is one of our, you know, other business unit. Then they're very much comfortable with OpenStack. And also, you know, I'm coming from market places. Actually, we have DevCloud running in OpenStack, right? And, you know, lots of customization capability. Actually, you know, you could do, you know, if you buy a vendor product or whatever, right? So you have to wait for a longer cycle. And if you need a feature again, you know, here you have a sharp weapon. Actually, you know, it's up to you how do you want to use it. Basically, you know, there are a lot of customization opportunities that you could go and, you know, make it, maybe, hack or whatever, it's meeting your business needs. Then go and make it better, right? But if you're for the vendor product, out-of-box product, right? They have to, you know, satisfy all the customers because they have to, you know, make the adjustments in their architecture or whatever very carefully. But here, you know, you develop internally and then, you know, use it and then, you know, contribute back to the community. But actually, you know, we have concentration. At least within, you know, anywhere between two to four weeks, we should be able to, you know, contribute back to the community that we promise to our legal as well as to our eBay software foundation as well. So basically, you know, whenever you fix a bug or whatever, you can't contribute immediately. Or maybe whenever you implement a new feature or blueprint, you know, you can't contribute back to the community, right? And also, you know, it's supported by all the vendors. So whoever actually, you know, we have it in our data center today, you know, they're all being convinced that they're going to be supporting OpenStack, right? And, you know, it's very, very active open source community. In fact, actually, I could code in a couple of examples, actually, that we ran into. I'm being a software engineer for a long time. You know, I'm being in C, C++, Java, and then, you know, Python recently. And before that, I should pull and then, you know, C sharp also for some time, right? So whenever I actually have some questions, actually, I post it in the forum, right? Most of it may be Stackflow or whatever that you, you know, all the developers are using it. So most of the time, you may get the response, you may not get the response, or if you have to keep waiting, or maybe you don't have choice sometimes, right? You can't get into, you know, JVM counter and see actually what is going on. But here, you know, I was, you know, before I just, you know, go into CloudExpo for the same kind of presentation, I was, you know, my team members actually got into some issues. I asked these guys, you know, go and post it into the mailing list, right? I was just following up, actually, I was going to, you know, Santa Clara Convention Center. It's not even, you know, an hour and a half, actually, that problem was solved. That's a very critical problem for us, actually, you know. We are able to go for production, but within two weeks. But we are into your cycle and regression cycle and all those things, but within one and a half hours. But I'm guaranteeing you, if you are using a product, you are going to, you know, be using, going through your customer support and the customer support guys will try out, and it will reach to, you know, the developer who knows actually sometimes, you know, they will know actually from the log file or whatever, they will be able to fix it faster. But this one was fixed one and a half hours. That's what I'm saying. It's very, very active open source community. I've been, you know, using open source for a long time, but this one is, you know, exceeding all of their expectations, right? And also, you know, we are using, it's not that, actually, we are using only the open track at the open source. We are using already Linux and then Java and Kadoop. We are very successful in that. And of course, you know, there is no, you know, hidden secret in that, okay, we are opening up many doors to open, you know, high top talents in the industry, right? And of course, the foundation is standard stuff, right? We were evaluating in Diablo itself, but we were waiting. We were holding on to our decision. Even my colleague, actually, Jesse Martin also, you know, talked about that in the design summit. You are holding on to that because we don't want to get locked into that because some gentlemen actually, you know, they were raising the concerns still privacy, but I have some insight about that. Actually, I've met with Jonathan and Lauren and all this works actually recently. So I'll give as much as possible information actually so that you guys will be comfortable. We are going to address the privacy issues, maybe all the openness, whatever we wanted to come up as open stack. I have some data for you guys, okay? And the foundation, of course, we were waiting for foundation to be formed. And then we decided, okay, this is open stack. Okay, so this is how actually it looks now, what we have in the pilot that we roll into our production. So if you look at, you know, the compute, we'll, you know, go from our hardware infrastructure all the way, you know, upright. So we have our compute nodes actually, you know, using, you know, cell 230 is the HP hardware. It doesn't mean that actually, you know, we just need to use only that hardware, but actually we can use the HP or maybe, you know, anyone else actually use this. We can use any hardware, right? So right now, actually, you know, we chose this, but as long as, you know, any hardware it performs, actually, it's not, but it's a bundle solution that we are going to buy in. So it's going to be open compute. And at the same time, actually, storage actually, it's local storage. We don't have storage now, but some of the applications that we are onboarding into this environment, they don't need storage. And before I go there, actually, you know, what happened? Then I said, okay, this is the open stack that we are going to use. Okay, management said, okay, how are you going to implement it, right? Of course, the executive saws me and then I said, okay, okay, I told, okay, whenever you buy a new hardware, we are not going to buy, you know, bundle solution anymore, right? We wanted to buy open hardware, and I'm going to use, you know, open stack. Then I asked, okay, what's your next purchase that is coming in one or two months or maybe in one quarter, all right? They told me, okay, we are buying for the Q4 capacity add that is coming in Q4, 2012, because that's a holiday season for us. That's a lot of, you know, translation will be coming into our data center. You want to upgrade our, you know, capacity, right? They said immediately, okay, it's going to be in production domain. So not in QAA or maybe staging or whatever, right? Okay, I don't have a choice because I made the commitment, okay? But only I have in a conference that actually, you know, I have the source code. So I have my engineers, they can fix it if something goes wrong, right? So we put it into the lab and we were, you know, we can throw, you know, two and a half months actually, regression cycle, we tested, you know, so many things and finally we were able to make it. Not that actually, you know, we can, you know, have some 10 compute nodes and then go and plug it in at work because we have to put it into the existing infrastructure. That's a huge thing for us, right? So we chose to have, you know, network, actually, we have, you know, two different switches that we use today. It's a top-up rack switches, Cisco 4948 and Arisa 7050 and load balancer is AFI, but our load balancer management service could be using, you know, Citix NetSkiller also. It manages both. But the API is same API, right? And the software stack, actually, we use Cobbler, Salt and Bind and Red Hat Linux 6.3 right now, but actually we are not using Ubuntu because that's our existing, you know, supported version within our infrastructure because not that actually we can't go and create another image and then go and spin it. It's not that we have to manage it, right? Because it's adding, you know, complexity to our systems team. And also the KVM is our hypervisor and as used to monitor our, you know, open stack services. And also, of course, we are using VMC also to monitor on top of this, right? And the foundational services, compute, of course, we use Nova, Swift, and then Glance, Keystone and stuff like that. And for the load balancer management, actually we use load balancers as a service that is built within eBay. And we open sourced our design and also the implementation details recently to the Grizzly Summit. And also, I was running the, you know, the entire load balancers as a service design, design forum also for, it came out really well. So there were, you know, so many different load balancers as a service implementation in the open stack today, even in the Atlas LB and then our implementation and also, you know, you have to get one and I think HP also had one. And we got, and of course, Mirantis started implementing their own. Then actually I was able to convince or maybe I'll be fortunate, I was fortunate to bring all the vendors and also the players in the market to bring into the one room actually. We all need to agree upon one. That should be only one load balancer. It's not eBay's or maybe Seaverse as mine. And finally, we ended up, you know, creating a new load balancers as a service that is going to be part of Grizzly. And, you know, engineers are working on that actually. So it came out well and also it opened up, you know, a lot of opportunity to, you know, enhance quantum framework itself to insert advanced services. Salvatary and one of my other team members is working on that actually, well and within other community members. And that is really, really coming out well. I'm so glad on that, because, and also, you know, DNS as a service actually be built internally. And also my colleague, Jesse Martin actually is taking that to, you know, actually he's working with a lot of other folks to make it happen, make it happening as part of OpenStack itself, right? And orchestration engine. So we still have our own implementation and we plan to use, you know, Heat is another subproject within OpenStack and it is run by Redat. And we met with Redat guys and instead of, you know, we go and build our own, then we wanted to, you know, grow with the community itself, right? And, you know, we'll be putting the resources along with whatever actually Redat guys or maybe any other community member they could join that. So this was actually the overall stack. And also, you know, when you talk about entry for our cloud, only two different, only two entries, not many entries at all. Only through the PD deployment service portal where you have multiple stages for the PD guy and he comes in, he develops the code in each stages and then he rolls in the code himself. But our release engineering team and then system engineering team, we are going to be building the automation to make sure, okay, it's not going to break the life side because we need to have that six, nine availability goals. So without that, our business is not going to be anywhere. So that's very important for us. And of course, you know, we are going to have the workflow and then monitoring on top of that. And when you come to, you know, cloud is not coming for free. You have to manage it in the back end, right? You have to undergo new hardware, new load balances, new networking gear. All of them are going to be still managed, you know, centrally across all the data centers. So this is what actually it looked like today and it might change because that's what opens up all the port. But right now actually we chose to, you know, going with this. But even if we change, the changes will be minimal. All right, so what we have today, the current deployment, okay? One open stack deployment per data center because we don't have the global orchestration layer like, you know, compute cells or whatever that, you know, we are going to be implementing the forthcoming phases, right? They allowed me to put numbers also, that's good. Okay, the 96 compute node in a rack, which is a fully dense rack. So we could create anywhere between, you know, 1000, 2500 VMs within one rack, right? And we have four top-of-the-art switches for production and two for management. Actually, I want to, you know, talk about, talk a little bit more about, you know, these two things. So open stack, right? You have a switch and then you have your compute node, then you can, you know, form the cluster, it will work. But for us, actually, we have a lot of limitations within our infrastructure and data center to meet a lot of compliance issues, right? So you can't run your management and then production in the same interface at the same time. Actually, you know, we got only 223 subnets because we have to put, you know, the new racks in the same, you know, pod or whatever, right? But still, you need to use open stack, right? We actually customized, you know, a lot of things, actually including the schedule as well as the wheel and manager to meet that. Actually, you know, it came out pretty good, actually. You know what? So I was a little bit worried, but within 3.3 and half weeks, actually, we were able to do that because we got the code. The only thing is we got the code, right? And we had two slash 23, so two for production, two for, you know, management. And we are supporting two flavors as of now. Actually, so actually, that's another thing, actually. So we don't want to support too many things. Maybe, you know, one application team will be coming in and say, okay, I'm running my VM or maybe a compute, you know, physical cost on 48 gigram, right? And you might be running on that, but you got to go and optimize your applications. Not that actually, you know, cloud will solve all the problems. The application also should solve some problems to make their applications cloud enabled, right? And, you know, we wanted to have, we wanted to have about only two or three, you know, profiles, not many, and utilizing existing physical as three and firewall. So we are not changing anything in that, but we just kept as it is. And actually, we are going to be, you know, playing in that as well. We'll come back to that later. And then Nova Clanswift, and we are using SS Table 3, and we are in the process of moving to Folsom, yeah. And integrated with load balancers and DNS, and then change management, of course, that's very important for us. It's not that actually you like, you know, some code, actually, you know, you just go through it and it's very important, right? So there are a lot of millions of users, 120 million users are living in that, right? It's very important for us actually to go through some of the existing business process also. We integrated with that, and also the monitoring as well. Okay, what we learned, that's very important, right? Okay, fitting into OpenStack into the existing infrastructure, it's a bit difficult. It's a bit difficult. It's not, you can't build a new data center, a new pod. If you are lucky enough to care that, it is easy, yeah. I really mean that actually. It's easy, but for us actually we have to meet with, you know, all the existing management capabilities that we already in place. It's not that actually, you know, we can change the, you know, enter infrastructure overnight, right? We have to, you know, move step by step, step by step. There are a lot of startup challenges, especially, you know, the meeting that, you know, production interface separately, and also within this last 23, you have to onboard your physical as well as your, you know, VMs also on the same outer space. So we have to do a lot of hacks, and then a lot of tools to make it happen when you create the network or whatever you want, you know, assign your IPs or reserve the IPs in the database table. It's a database change, but still, you know, you have to make it happen. It's not that straightforward, right? And availability zone customization. When you look at even our rack profile, right? So we have four different switches, two per production and two per network. All this four are different, connected to all together in a different L3. So even, you know, one L3 domain is completely gone. We have other off-of-the-rack is available for you. But OpenStack doesn't support, as everyone knows, OpenStack doesn't support out-of-the-box, right? We customize that. Actually, we change the cellular whenever you have, you know, minimum is actually four VMs for any application for us. And make sure that four of them are going to be distributed across, you know, four compute nodes and then two different availability zones in the rack. Right? We have customized all of that. And plug-in options for DNS and load balancer, right? So your compute node is coming in, then you have to go and update your DNS entries. So I don't know if I'll take it for that, right? We integrated that automatically using the DNS as a service. The DNS as a service is not going to be using for our OpenStack. Basically, you know, we have a lot of other, you know, management activities also in the backend operations side. Basically, you know, taking out, you know, some of the bits from the traffic, if something goes wrong or whatever, right? So that's why actually a lot of people ask, okay, why can't you use, you know, DNS update itself? Right? But still we need a service. Say, you know, your infrastructure is already breaking, right? And you want to stop all of that, right? People might, the DNS may be, you know, technology might be simple, but it is very, very important for your availability. Okay? By mistake, if you reconfigure your, you know, I know actually there were in 2003 or 2004 there was an outage in Microsoft actually, okay? They took two and a half days to fix that. What was the DNS problem? What is the DNS problem? Yeah. So you have to be very, very careful actually how you are integrating with DNS. Actually, we put a lot of effort in making that and also the load bands are also, so load bands are the service. So you got your compute node, then you have to go and add it to the existing pools or L7 rows, modification and all those things, right? So we have integrated that. And the lab infrastructure is exactly, so we don't want to put anything directly into the production. I'm going to test, it's like, you know, PD life cycle, right? Product development life cycle. You know, develop something, you know, you're changing your code and we have our CI setup. It is running, you know, every 30 minutes by default and you know, whenever you change your code, we exactly have whatever open stack has today in our lab, right? We have our CI running, our, you know, Tempest running, everything is running actually, you know, whatever open stack has today, even including our code review as well, right? So that's what we have, you know, we have a strong, you know, lab infrastructure and then open stack development life cycle. Actually, I want to, you know, talk a little bit more about that. So basically, we have, in fact, actually being a developer for a long time, I learnt so many things from open stack development life cycle. It is one of the cleanest way of, you know, developing the code. Otherwise, you can't manage those many, developer or car support, you know, there is, you know, job is really, really tough, actually, but we have really, really good, you know, good process around that. So we learnt that. And in fact, we are, you know, some of our engineers also from the product development team actually, they learnt from that. Yeah, and continuous pipeline also, actually we built to automatically deploy in each environment, right? And what is coming up for, what is coming next for us? Sophisticated networking. So I got my racks and then I put it into, you know, infrastructure and I'm still using the VLAN manager, of course, you know, I'm not able to, you know, take full advantage of the investment that I'm making. Think about that, right? So you have a part still, it's not really cloud, right? And, you know, we can open, see, OpenStack is not, you know, going to solve all your cloud problems, right? You still need to, you know, do a lot of stuff to make it happen for you. Maybe it might be good if you are completely, you know, have a different data center, you know, new data center, you know, new business, it's all good. But with existing business, if you want to, you know, have migration from your existing cloud to new OpenStack way, you have to migrate. It's not that you can, you could buy one billion dollar worth of, you know, data center investment and then you put OpenStack on it, right? So you have to gradually migrate into that, right? So we have to solve that and then we are, you know, evaluating a lot of SDN products actually. So we already have, Nisira working in our dev cloud marketplace aside and we are evaluating, you know, a lot of other products also as part of this exercise and we will be, you know, deciding so, right? So that is coming up. And bare metal provisioning, our Hadoop cluster is being, you know, custom built provisioning today. Our entire eBay market places search infrastructure is built using our internal cloud, right? So we, and also, but actually, we wanted to use OpenStack also for that as much as possible. And compute cells, of course, everyone knows why we need it. So we will be focusing on that too. And open hardware spec, yeah. So as I said, actually, we are not going to be buying the bundle solution anymore. We wanted to open the gates for all the vendors. Okay, this is the spec I want for processing my payments or maybe processing my checkout or processing my, you know, mails, patch process or whatever, right? Then it opens up the doors. Actually, you know, whoever actually went to come in, if the hardware is performing, maybe Intel is coming up with, you know, something, you know, cool for the chipset itself for payment processing, why not? So we'll just go on by it, right? So that's what I, we wanted to openly announce, you know, all the spec, whatever actually, not many players in the industry are, you know, have guts to, you know, go and announce it, but we decided to do it because they feel that it is, you know, bread and butter for them. But for us, we strongly believe that our application is the, you know, bread and butter for us. Yeah. And extend that, so we have it in the production today, but we wanted to extend the same infrastructure across the board, actually, QA development, QA staging and load and performance environment and also the pre-production. And self-service and then security. Okay, so today, actually, whatever we have today, it's not self-service because still I want to give the highs and portal to the developer, right? I know for sure they will mess up the cloud. Right? So we wanted to, you know, build the product, the PD deployment portal. We wanted to open whatever actually we wanted to open the rest of it. Actually, we are not going to give it to the developers at all. So it's not self-service. It is being used still by our system team. And security, of course, you know, that's very important for us. That too, you know, for people, you know, it's a big thing. So if we screw up, if I screw up actually in security, then, you know, the business is in trouble. So we are really, really careful about actually, you know, what we could do, what we can do and all those things. And also, I'm glad that actually we formed an open-stack security group in the design summit. And that is, you know, actively being involved in a lot of use cases. I saw, you know, different players actually presented, actually, how we are going to address that. And in fact, actually, we already have a process within, you know, within PayPal or maybe eBay company, right? So before, you know, moving any open-source project, actually, we scan through, you know, different policies, different code. Actually, we have the process. But since, actually, open-stack is already, you know, talking about that, actually, it saves some of my time. And that's good for us. And good for the community too. And we are going to be contributing to that, actually, what, you know, public cloud needs. At least for me, it's just private cloud. But actually, you know, for the public cloud, it does it's still tough because you're using the messages everywhere. And if you hack the message, okay, it's not, see, I'm up here. I don't want to have, you know, I don't want to say that, but sometimes, you know, people mess up things for, you know, fun or whatever. Maybe accidentally, you know, when I was playing with open-stack, actually, I've drawn down our dev cloud. Right? And one of our, you know, pilot applications, actually, it was running completely gone. Good that, actually, it was a pilot. But think about, actually, this customer-impacting application. Right? You lose your business. If you lose your, the 6-9 availability that we have today, then we will be in trouble. I don't want to put our management into trouble or my company into trouble. So we got to be really careful in terms of security. And migration to Falsum and beyond. So we are, you know, setting up, you know, our internal lab to, you know, try out your things. And, you know, we are bringing in SDN also as part of this. And we will be migrating soon in one or two months. And the design, develop, and fix, you know, contribute back to the community. So that we, we committed to our OpenState software, sorry, eBay software foundation, as well as our management, our legal, whatever we develop. And we are going to contribute back to the community. So we are not going to keep ourselves, but we feel that it is liable for us. We don't want to do that. It's not good for us and good for the community. Right? And of course, we are going to, you know, set up the lab. We already have the lab. So we wanted to, you know, package whatever we are using internally. Right? Whatever SD Cloud is doing for, you know, your Hadoop today. So we wanted to, you know, fill the packages, whatever we use, whatever we think it will be useful for the community. We are going to be distributing from eBay software foundation. Yeah. And also, you know, yeah, it is through the eBay software foundation that we plan to do. Okay. So what, where community want to go? Right? Yeah. I met with Lauren and, you know, Jonathan and we talked about, actually, you know, where communities going, a lot of concerns from, you know, previous use, my view upon us. Yeah. We talked about a lot. Right? And, you know, a lot of feedback sessions in the executive board. And we discussed about, actually, how we are going to address, you know, some of the issues coming up. So today, actually the major problem was everyone is worried about, okay, the blueprints that is coming out of, coming out. Basically, you know, company X likes one, Y likes one, they think actually that is the only one standard. And, you know, right? That is the problem. Right? Everyone is concerned about what about my company then? I am also a gold member. I am also a platinum member. Right? So what we did, okay, we brainstormed, actually, we met three hours, I think. We met for dinner, actually, dinner was two hours, but actually extended for four hours. That is another story. Right? We keep talking and then finally, you know, we ended up in creating a stronger user community. So, we are still brainstorming, actually, you know, how we are going to address the blueprint in the creation process. So, it is going to be driven mostly from the user community. So, if user is, you know, I think, you know, somebody, I forgot his name, actually, he was talking about, okay, eBay and PayPal is coming into this. It's a big thing. It's not an easy decision for us, because, you know, we came out and then said, actually, we are going to be the ones that we have. Right? We are going to help the community as much as possible to be open. Right? So, where we wanted to go in the, you know, the cloud, actually, we know, you know, OpenStack is playing in the cloud and all those things, right? So, one thing that actually, you know, we have talked about one more thing also. So, to take our, you know, OpenStack to the next level, we believe that actually, take the kids coming out of the college or university, they should not talk about, okay, because a lot of people, you know, you know, Tristan was telling everyone that actually not many people know the cloud or maybe the OpenStack. Everyone knows cloud, but not many people know the OpenStack. Right? Why can't we take it to the universities and colleges? We are planning to, you know, support universities and colleges with the hardware infrastructure or whatever they want to, you know, put in their lab, try out cloud or whatever, right? We are decommissioning a lot of assets every two years but we will give it to the universities and colleges. You go and innovate, right? Good for us, actually, we could hire and you know, it's good for the universities, good for the students and you know, we are ready to do that, to do that actually. So, and if you do all this and the cloud world will go away. So, India, okay, basically I'm from India, right? So basically, you know, when we start, you know, a photography shop, we never say, okay, I want to do photography. We say, always, it's a product. Yeah, right? But we don't say photography, right? Yeah, the same thing. We are not saying, okay, I'm, I'm, you know, searching in the internet. We use something else. How many, how many of us, you know, talk about search engine now? We talk, you know, I don't know what to tell you. I was able to find a logo without the cloud software in the bottom. Why can't we make this one as a logo for us? Actually, you know, the cloud world, you know, what will go away? Right? That's where we wanted to go. Right? Definitely, you know, it's an event organization. Yeah, so, okay, some selfishness, right? Yeah, yeah. So, actually, you see, and of course, you know, we are hiring actively and there's an open secret and, you know, it's my contact information, my cell phone box in India also. I've verified already. You guys, you know, reach out to me and also you could check over, you know, eBay careers website also and open. Yeah. Questions? Hello. You mentioned open hardware specifications, right? So, did you mean something that was, that is similar to what Facebook has done with the open, right? Yeah. So, do you, do you already use something that was available outside or why do you reinvent the wheel? No. So, I'll tell you actually what happened, okay? So, the rack that we bought for this particular exercise or maybe, you know, our market places that we already use, this, you know, based out of the open compute, just our system engineers and then our infrastructure team, actively looking at open compute, right? But also, you know, I have, it's a different group, actually, they're responsible to, you know, come up with, you know, hardware profile for your data center based on the workload that we run in each environment. It's not that actually, you know, we could buy a compute, you know, rack, it will work for your big data as well as your performance queue. We got to come up with you know, different, you know, but if you want to get out of the SLA that you want to get, maybe, few nanoseconds per transactions, right? It's not going to scale. So you need to find right hardware. Maybe for Facebook, you know, it might be working for them as it is, whatever actually they have it in the open compute.org. But for us, actually, you know, some of the reference, we took it from there. But the one that we have, that's what actually, when I say, it's the open compute, maybe, we could even go ahead and say, okay, this is the hardware profile. It is working for payments. It is working for Violet. Okay, this is working for mobile payments. We should be able to, you know, go to that level and then publish our, you know, specification outside. Does it answer your question? Yes. Yeah. Go ahead. So, the way I understand PayPal has a lot of other companies or organizations which integrate to your system to work. So, when you are moved to OpenStack in production, how easy it was or how tough it was to convince people to come on your cloud and where little more deeper insight on the security cuts and things like privacy or how the transaction is secured when you are putting on the cloud. Can you put some highlight on that? It's not public cloud, it's private cloud. It's a private cloud. But people who join to you, do you expose the capabilities to outside or are you not? No. Okay. Thank you. Adoption to public, you know, private cloud, do you have to change your applications and can you share how much of that was a hassle Yes, exactly. in the IAS model and what's the things that you went around? Yes, exactly. So, eBay and PayPal and then very old companies, right? Not that we, you know, started our company. Every company they come up with new frameworks or maybe, you know, new layer in different areas, you know, to meet a different technology style that is available in the market, different resources that, you know, human resources that we have that engineers available in the market. Not that actually you run all your applications in C or maybe, you know, some of the global language or whatever, right? You cannot find resources, maybe you can find only in the you know, we change our framework and there is an application migration path also. So when you say actually the application, see, every application team they have their own, you know, business charter because every application running as a, you know, different business unit within, you know, we get company, otherwise actually, you know, if it's a simple site, actually, you know, why we have 29,000 employees within even PayPal, right? Every, you know, in the olden days, a computer is given for as much as you can use. But in this model, you're actually trying to balance your infrastructure for optimal provisioning. And there is this DevOps movement, which is primarily looking at a SLO based model of DevOps collaborating. How did that work for you? And basically, the application writers were always assuming that I have everything on that machine. So if I go back to my days of computing, I was always assuming that the whole machine is mine. But that's the world which has gone away. And with cloud, there is a model that you actually get what you get, and you will have to live within that what you get. Yeah. So how does that phenomenon of DevOps work for you? Yeah. So basically, you know, what happens is the cloud world, actually, it's most of our applications, you know, it's self-managed most of the time, at least the capacity that is self-managed. But we have, you know, automation built around that. That's where actually our DevOps folks are helping us out all the way from your release engineering to your business process, right? So we are building the workflow around it, right? So say suppose, you know, I have an application, you know, it needs maybe 50 compute nodes with, you know, load balancer and then firewall or whatever, right? Maybe it might be okay for until you are Q3. That is not, you know, Q4 is, of course, you know, we beat that recently. Because not that it's only long week, and it's, you know, sky-trafficking for us. Actually, you know, there are a lot of other occasions we met today and people actually we exited our, you know, transactions, right? So we... Hi, I am Hains and I work for Accenture. Yeah. So I have a question regarding the technical stack you have shown. So for this orchestration as well as capacity management and even dashboard, development. Yeah. So we have our orchestration engine that is in us built but we are going to be using heat. So that's a template engine that we'll be using for the cloud formation. And what are the normal orchestration activities you do? And why didn't you go for puppet or chef? I'm sorry? Why didn't you go for puppet or chef? We use puppet, actually. We use puppet for the configuration management. Okay. Yeah. This is for the template. Think about it. This is for the application. I need 10 compute nodes 2 for MySQL or maybe 5 or 6 for your application server 2 for MQ or whatever, right? So to manage that template actually in each environment that's the template you will be defining to move your pipeline. So basically what happens today, right? You put everything into one box. That's where the most complexity arises in PD world. It will work. It will work all the way down to your pre-production. But it will not work in the production because it is being distributed. The simple use case, right? It won't work because you need to have the similar environment in every stage. Maybe pre-production or maybe load and performance may need only maybe 50 compute nodes, but your production needs 500 compute nodes, right? That's where actually we want to use the same template across the board so that actually it solves many of our problem running out of time. We'll take the next question during lunch probably. A short question. Which all OSU provision in VMs? I'm sorry? Which all operating systems do you provision? Support. Guest OS. Guest OS. Oh, guest OS. Okay. So it's Retard Linux 6.3 and then 5.7. Yeah, sure in 6.3. But actually we support 5.7 also some of the applications you know they need 5.7 as well. But we'll be moving into Ubuntu. Okay. Yeah. And I'm sure and I'm sure all those who have some more questions going to trouble him during lunch. We are kind of running out of time. That's why we are kind of. Next talk will be from Ritesh. Ritesh works for Ericsson and he'll be talking about what they're doing with OpenStack in Ericsson. Yep. I work with Ericsson as a cloud architect and we belong to a innovations team. Thank you. We belong to innovation team where we carry out innovation across Ericsson in 178 and around 480,000 employees. So even I have my colleague here so why we are choosing OpenStack for Ericsson and what we are doing with OpenStack in Ericsson. So I would invite also my colleague who is the lead architect for this. Chandra, please. Then even I would join telling about the technologies what we are doing with OpenStack. Good morning. I'll just quickly go through some of the part of the presentation and then this is just a brief agenda that we are looking at. There are some enterprise challenges and opportunities that we are trying to address with OpenStack and then I'll just discuss about some of our interests and goals that we have and how OpenStack is a solution for all these challenges. And then what we are trying to do are implementations and building a cloud that we face and of course a research area that we are trying to get into. As we all know that IT costs are always up and there's always a hardware cost involved and data centers, space, green energy preservation. So these are some of the very common challenges that are faced by the industry today. So obviously we have the opportunity for architecture and we also are trying to come up with the pay as you go and come model which will be explained by him and then there's of course a platform deployment on top of that and then software service and automating all the IT operations. So these are some of the broad opportunities we are trying with OpenStack using the latest version Falsum and building on top of that platform that can be used by a wide variety of enterprises. We also productizing some applications on cloud that can be used for different enterprise verticals like retail, travel and especially the telecom sector since we are working in that and then our focus is on actually some of the research things and then coming up with products that can be used by the small and medium enterprise and of course there's a lot more for the telecom operators that we keep building on. Yeah and yeah so this is a broad architecture of our current cloud and then so I'll just hand over to Ritesh after this you can explain thank you. I've been looking how we can further integrate the OpenStack into our existing environment like when we talk about doing with the customers and all those things there has been always a debate on why we should move on a cloud and all those things so at present in Erikson we have around 1500 course which are running on OpenStack and we target about creating 2000 virtual machines within Erikson we are at present doing this within Erikson like we support our customers on top of cloud they can provision machines do the DevOps environments and all those developments can be done on this top of OpenStack so we have the infrastructure and then we are at present we are not using any storage technology like what OpenStack released this new version which is this for the computation then we use networking which is no network and we have the security identity and SS management keystone all this in place then on top of that we have our own service cell portal which is like an orchestration layer which provides me start from the order management to the order fulfillment and the billing parts and all those things so this was the OpenStack what we are doing we have a Folsom in place in our data centers but then we have announcements in storage like at present we are using network file systems not the any proprietary solution in place so we have like Moose FS we use Moose FS for providing high quality of a storage and all those things and announcements in networking we at present use no one networks if I talk about the Erikson internal for the employees then we make on the basis of the VLANs where we provision different departments on the basis of VLAN and they can use the machines on their doing in their departments and about the customers part we use this no one network to use the different technologies available in the market like the MPLS and all those things to be connected to the OpenStack clouds who are small and medium enterprise companies who are on the MPLS and all those things directly then we have launched IIS within Erikson India at present it is supporting around 20,000 people within Erikson India and we target about the 1 lakh people all over across Erikson then we have a self-holding in place I told you before which provides all the functionality what OpenStack provides and even top of that we provide the orchestration layer we can do all the things then we even have a product which is helpful for some of the we have productized some of the apps which can be used on top of this OpenStack then in this we provide effective ordering, reporting, provisioning, authentication then even the past part it is still under future expansion but I did mention we are evaluating number of past vendors like cloud foundry open shaped and stack at all those things and below is the IIS there when we started working on the OpenStack part if we talk about the high availability of the services because there was always an issue we talk about giving any of the services then it should be highly available across then we face that issue and even we had a solution for that in place then we have the networking where we can effectively use load balancing mechanisms and all those can be integrated and they even don't have the time to address these issues so it comes in the new releases and all those things so at present we use we do have a solution to do the load balancing and routing mechanism features in the networking part then scheduler it is the privatization of the services how we can effectively use the scheduler and all those things so we have in place our own algorithm how scheduling works within OpenStack then storage how can my data be more highly available whether my data would be available what backup and recovery solutions I would be taking in care and the security and assist management then I talk about security then this is the first thing when we talk about offering this services to our customers or even internally we talk about when we say anything about the cloud like we are selling our cloud everyone who says it's like what about the security and all those features so we do take care of the security then all those things then we did we are as I told we are evaluating past solutions and even billing of the solutions we have a small solution of billing in place and even we have our small meters and all those things now we are moving trying to even evaluating the metering part which OpenStack has now in place so this is a small solution introduced what we did with the high availability part like we wanted to make the services using the software keep alive and it depends on the topology we created a lot of topology reading the rack space architectures give me a lot of knowledge about the OpenStack architecture how it can be deployed in different data centers how architectures I can make if I talk about providing availability solution between two different nodes solutions and all those things so we covered all these topologies and we came out with the topology to keep the services highly available then we have even highly available services for RabbitMQ and MySQL we use the high availability for RabbitMQ as proposed by RabbitMQ only where we talk about mirroring queues then providing it highly available load balancing those queues and all those things then if you talk about the monitoring part this is the most thing in which we if you talk about the deployment about the alarms and all those things so in place we have monitoring solution introduced for Keystone, RabbitMQ for the DSCP servers NOAA services GLAN are now effectively monitored within our solution now other research areas like we have big data analysis what we are targeting to do further in this cloud area and machine learning data mining scalable how can we make more scalable solution and trusted computing is the most hot topic now in security what we say about how we can make trusted computing on my loads if I talk about scalability it's not just of adding on nodes we need some more security like trusted computing and all those features within this I think so I've connected with this it was a very small slide I didn't wanted to bore you all in place which we offer to customers like the operators and all those and we enable them to provide the services to their enterprise customers sorry so we have a whole cloud platform which we offer to our operators and the solutions like the services like the chat and collaborations mail and messaging solutions even some of the telecom applications like if you might have heard about the telecom applications IMS we offer the solution to the enterprise customers who are operators and there was when Ericsson started working on this was in the year starting when we demonstrated this whole solution in Mobile World Congress you might have heard about the Mobile World Congress Barcelona this was the place where we hosted this solution and demonstrated and around if I say there was seven cloud solutions within Ericsson they were demonstrating those key leads I have a question how reliable is the high availability with RabbitMQ with Q mirroring and stuff like that and also does it scale so does it scale under load about the RabbitMQ MirrorQ solution is a quite good solution if I say it says about mirroring all your Qs to the slave servers and you keep on adding nodes keep on adding the RabbitMQ nodes and keep making updates and if somehow your master goes down then one of your slaves is promoted as the master so only the master is active and everyone else is passive if I say it's like just making your Qs available in all the nodes you need because the Qs basically in RabbitMQ are perishable they are not persistent when we talk about mirroring the Qs we have to make the messages and Qs all those persistent we can implement the reverse proxy or some sort of to provide the load balancing features and I'm curious like a few slides earlier you had shown challenges on various access and there was storage access on which you had said like backup replication and things like could you possibly expand upon it a little bit like the storage where you're talking about replication and data backup could you possibly expand a little bit on that to appreciate what's going on and is there some NetApp storage in front of there or not there is already Sarge is there and laughing we already had a talk about this it's like an outside let's talk about it outside probably we'll talk about it offline I will tell you like as storage for the storage part at present we didn't did much we wanted to make our data highly available that's it because there was no as provision instance and all those which are hosted on this open stack so at present we use a network file system but we focus as the senders comes in and we are on the for some part we focus to make storage solutions to be part of this so we are evaluating a number of solutions I have a question on the networking part so what challenges did you guys really face on networking for the networking at present I see a lot of protocols like VXLine VGRE blah blah blah so what really is the problem you face for the networking part at present we are not using quantum we use simple VLAN manager where we talk about providing these VLAN connectivity to enterprise customers directly using the whichever technology they use the MPV list so we did a lot of tweaks in the code also to provide the services because what Noah does is makes his own gateway and everything in the case about the when providing this MPList connectivity to our customers we need to make tweaks in the gateways firewalls then provide the edge routers and all those but is it answers your question? it's right yeah so I mean you got this would be this would be integrated to our core networks and further we offered it to our edge routers okay that is the thing like if I say in my every VM would be given a different gateway which would be outside gateway and further they redirect to the edge routers and all then further so you you may say your whole cloud networking is based on MPList based stuff or yeah for the customers part we do it on the MPList part I present MPList and even the open SSL VPN and all those open stack or any other your own applications and all the required things is there any hurdle that that you face while updating the firmware on the physical server like example you may need to update the bias on the physical server or some red controller firmware on the physical server so any such kind of hurdles which you faced to address the physical firmware related to the physical components physical servers which are flexible and even a normal server which is available in the market can be it can be any x86 64-bit architecture can run this open stack so in that case we already had in place are all servers for in x86 64-bit architecture so it didn't face as such problems in this but we address some issues like this is basically the cloud definition by NIST which says what's a good cloud you know the cloud paradigm basically changes the model where you have everything for yourself to actually everything is shared and when everything is shared what are the changes that you need and what are the layers that come you have different deployment models which is public, private and hybrid clouds and you also have the various models which is actually the platform as a server infrastructure as a server and software as a server and on the characteristics which you want in a cloud is primarily on-demand self-service elastic which is primarily elastic resource pooling broad network access measured services and elasticity so these are the things that you need that form of cloud mostly when people say what's cloud we have different answers to it and this is one consistent definition which actually shows that and OpenStack seems to be working on that model where this is the Folsom OpenStack architecture which kind of has various services I think this is not very clear you have actually OpenStack Horizon portal you also have Cinder which is actually the storage part which is primarily doing the network network management you have Nova which is talking of compute management you have Keystone which is primarily the identification and security part of it and there are other services which are coming in and if you look at OpenStack everything started with Nova and if you look at Nova Networks is bifurcating and going as Quantum Cinder was part of Nova volume and now it's separating it out and things are evolving OpenStack people say what a score you also have Glance and the Swift repository which is also not which is there so these are things which are evolving and as what you see is projects start evolving out of Nova and it's starting to go out like that the other initiatives which are happening inside that is Celiometer which is by and large the metering and solution you are also seeing things like heat which is actually cloud formation kind of APIs and there are other things you are seeing interest with HPC trying to adapt to cloud you are seeing interest from hyperscale which is trying to adapt to cloud everybody is trying to get cloud in that form so why this topic of what I will try and talk about there's Nova and specific and the physicalization plus virtualization and what are the forms which are the main topic of interest so if you look at Nova we have been talking that you know this is the Nova drill down and Nova basically has a Nova API it has a Nova compute and the compute controller the Nova API and the compute controller talks through a Nova scheduler and the database and you have drivers which given any form of compute whether it's a Unix or a Linux or any kind of system or any kind of compute you could actually go and plug in that compute underneath Nova and the biggest change what it does is the way you do it is actually by writing a driver and that driver manages the last mind so in lot of ways OpenStack is a pluggable architecture is what we said there are multiple points on the API level where you can actually extend the second one is the message bus where you can start attaching things to the message bus and last but least is the various control points and the drivers that you can do so if you look at quantum you have you can write different L2 drivers you can write various drivers so same way Nova actually has a compute and that's how the driver gets self-configured and each of these can connect to the message bus okay so that's primarily the architecture and if you want to keep yourself isolated the plugin model and the driver model gives you that much of flexibility to plug in any kind of compute that you want okay so let me look at compute models you have physical servers you have and I've listed few which is actually virtualized servers as ESX Hyper-V KVM and Zen server and you have other models which is either partitions which is like L-parts from some companies and V-parts from some companies and also container-based virtualization which is primarily alignment terminology which is primarily LXC or OpenVZ okay so these are models which you see in servers being fully consumed for the app so it used to be said that that's a database server that's the you know that's the SAP server or whatever it is that is how we use to address the servers and that's where it started and each of these servers were fully dedicated and they were big monoliths if you take a mainframe or any of those models it was a monolith which was dissected into small partitions and so on so that's primarily the physical nature of the computing and if you look at workloads there are workloads which are primarily very you know savvy for it needs high low latency it has it needs response time it needs high computing and for example if you look at Hadoop clusters if you look at HPC clusters these are things which still need physical hardware okay they don't go into any model which is primarily virtualized the world when what is the drawback of virtualization virtualization gives you lot of optimization it puts lot of VMs on to the same machine and you can use the flavors that you want but it's not near physical okay your I.O. is not near physical your storage bandwidth is not near physical your compute is not near physical because somebody is interpreting it for you and it's having said that these are the problems which are there with virtualization however applications which don't require a very high which is non tolerant which can be tolerant to some of these models can actually adapt to virtualization and they really work together very well the advent of virtualization has seen optimizations in the industry and you can actually have abstractions of true cloud you can actually have a resource pool which is collected together so that you can have you can start playing instead of calling it a single host you can build a cluster and then start placing VMs on it and then you get a lot of models around it and what virtualization has also pushed us is we always have where does your OS reside where does your things reside everything is structured and such a way that everything is tied the virtualization has pushed into a model something like it's a shared nothing model where nothing is shared and you could do a migration of a VM across any host to any cluster or you know you can do storage live migration you can do various other migration and that gets you a shared nothing model okay and a shared nothing model kind of gets you portability of the particular VM across anywhere to anywhere and your whole model of HA is completely dynamic and it has changed the paradigm out there involving the physical model is not something that is stagnant it's not dead so even today we want our database servers on physical we want other things on physical but how is physical shaping up physical is shaping up in such a way that you can have with that went of mobile phones with that went of various other things processors are getting more power savvy and one of the challenges of any data center is building so you will see lot of servers which are coming for a specific type of workload okay and this when you say specific type of workload you might even see a calcid a processor for a web type of workload or a search type of workload and a lot of process which are tuned for the type of workload and it actually consumes one fifth or one tenth and you can't say that given a one VM running or 20 VMs running you can't control the power to the granularity that you want to and this is what is shaping physicalization which is a anti-paradigm to virtualization which is actually starting to get lot of physical servers which are made for the type of workload that it runs and this is a paradigm shift which you are seeing and physicalization to get in there and if you really want to play around with something I think TriStack offers these physical servers which are actually small and nimble which you could use for the type of workload that you want okay there are a lot of scale out computers which are coming out which is actually trying to give you a a processor which is made for what your workload is rather than buying a huge server I will try to say how much can I use that capacity okay so these are paradigms which you are seeing as changing and while everybody is bidding on virtualization there seems to be a reinvigated interest in physicalization and the topic of choice is primarily to look at these two paradigms and how it is happening and you have a third paradigm which is not virtualized which is for LXC or container-based virtualization which can give you pass through IOs which can give you pass through certain things of it and even virtualization there are some vendors who give you pass through in that model so that you can get pass through IO you can get through pass through disk access and stuff like that but the competition to go near is an area which is of great interest to see which one wins and I believe there is a market for all but that's where the physicalization and virtualization and these paradigms will actually shape how compute will shape up in the future so that's primarily the introduction of what's happening in the market and how things are shaping and my next model is to show how a NOVA is fully architected and how NOVA can be looked at in this model the NOVA actually has different drivers and if you want to support different hypervisors you actually have a KVM host which actually has a LibWord driver and you could write a LibWord driver and that gets you a KVM adopted into the OpenStack community while your API remains the same and then you actually write a LibWord driver to adopt it to support KVM the same way when you want a VMware driver you actually write the APIs and then write the last mile which is actually the driver which helps you make the ESX adapt to OpenStack and same way if you want a Hyper-V host you do a WMI driver and that WMI driver kind of gets you these technologies enabled and then you have something like this which is actually an OpenStack model which actually integrated into a single cloud so here I'm trying to show a ZEN a Hyper-V or a KVM Hyper-V and ESX in this model and this is something which we have working and it's available in Grisly the only thing which was not there before Falsum was ESX and that's also available as we speak so the only change between the ESXI because of which you need to have a VM which is running on top of the ESXI and that will be your compute node which is the NOVA in all other cases the compute node can be installed within the Hyper-Viser so that it runs within that and you can see CloudBase is the company which offers the Hyper-Viser model for Hyper-V and KVM is ready to go and you also have ZEN server in the same model which has support for LXE OpenVZ is something which is a blueprint which is available in Grisly so that covers the entire gamut of various things okay now outside of this there is also bare metal deployment which is an area which OpenStack is pretty much interested and there are various blueprints and there are various tools which actually help you do bare metal deployment like Chef you have Puppet you have Metal as a Service you have Juju and all of these are solutions which are actually trying to give you bare metal deployment however there is a blueprint for in OpenStack for basic bare metal deployment which is trying to say what are the technologies that they will do and how no one will actually evolve to that I will talk about it in a few seconds but there is a puppet master to actually build a technology which is for building Hadoop or HPC clusters at one shot so those are models which you are seeing and all of this are primarily to enable beyond IAS and actually look at certain levels of pass layers so that you can actually start stitching a cluster in whole rather than infrastructure as a service which across each of these hypervisors which are supported and if you look at it it will cover all the way from creating a VM deleting a VM making it pass making it run and all of this is available in the community so you can do a snapshot you could do a cold migration you could do a live migration you could do NOAA networks you could do ISKC attach and so on so all of these features are as we speak what is different okay so why do you believe that open stack should will survive is primarily because the driver model and the plugin model where any person who wants to write the driver for the last mile of the type of hardware he wants that will actually help him okay so this is the list of features which it is supporting and last but least a physical bare metal provisioning this is a base level architecture which primarily and most of the tools this basic technology and there is a blueprint which is available which is being worked on and there are various tools as I said which is actually available which uses this basic construct so what do you have you have actually a bare metal driver you have a power management tool which is primarily IPMI based which can actually power on and power off the node you have a pre boot environment which you can actually which has some network services enabled and you pre boot the image that you want to pull down and then install on it so this is by and large bare metal what most of the whether you take it crowbar whether you take you know the the mass or any of these they use this basic construction but by and large they have their own implementations to it and that's how they do it so the one of the aspects which is really why is this very important because they want to give physical servers as infrastructure on the longer run and especially with workload aware you know smaller computes or compute nodes which is made for a type of workload this model will actually evolve and there is this terminology called metal as a service and you will see metal as a service emerging out and it's over time we'll see whether virtualization wins or what type of market wins what and metal as a service and other forms of services which is primarily for physical models will stay is another way of saying it and there is this various paradigms which you are seeing one is virtualized environments one is semi virtualized or you know container virtualized environments and then this physical environments where hyper scale needs are primarily towards physical environments and how this will evolve is what we'll see so this is pretty much what I had for talking and this is all about virtual we can take questions or look at it question about this database you said that database will lie on the physical infrastructure just wanted to know will this infrastructure be outside the cloud or be a part of the cloud it can be part of the cloud and that's where if you look at what you are trying to do is you can have a mix of virtual and physical