 So, good afternoon everybody. My name is Lothar Wieske and I will talk about something I did during the summer, setting up an open stack cloud based on technology of the open compute project. And I will try to figure out with you what these architectural considerations about using open compute project are. In a bit, we had to slow down because we had the customer who was still in a very traditional mode of enterprise systems buying the more expensive stuff on the network and on the storage side. So, going really big steps was not a way for him to go. So we tried with a difficult strategy going for the network approaches in the open compute project first. For me it would be interesting to see who is familiar with the open compute. Because when I talked with other peoples about that, this was not so familiar to people. I got in contact with the open compute project very long ago. I think it was 2010, 2011 and there happened a lot of things. What I think is interesting that we have this open source community understanding that the whole approach of open source is not tied to software alone but can change the hardware market as well. And it cannot in the future but it already has done now. If you look at what happens to traditional hardware companies, you see that these companies mostly lose revenues at about 10% year to year. This is something huge and we see the workforce changing in that. So going with open source concepts into the hardware market changed the whole way how hardware concepts worked in the data center from the procedural mode, very process oriented, very rule oriented, sorry, to an approach where a lot more self-organization and community building are the major success factors. So open compute projects I do not want to be that detailed on what it is about. They are essentially disrupting traditional infrastructure. This is another way to say what I just explained. It's not a second type of hardware they are building. They are building new economies, new ecosystems around another thinking of infrastructure especially for the so-called hyperscalers. So Facebook was the driving force behind the open compute project. They stepped in in 2009 and decided to do a very modern and very cost saving and from an economical perspective a preferable type of infrastructure and they looked out for a certain data center and they did it for a data center in Oregon. Once more they did not decide only to build this new data center but in addition they decided to make the whole design open source and available to the public. So not only is it available to everybody but people have to face that this new design was almost 40% more energy efficient and almost 25% cheaper than what was available before. This is not really stuff that can be compared Apple to peaches because the architectures you have on such a kind of infrastructure are other ones then you would build it on traditional infrastructure. So this does not only save cost but it can in some sense drive cost on the side of architecture or as others have put can open up a strand where you have two types of development and architecture, one for the more hyperscaler stuff and one for the more traditional stuff with a major shift in the future going to this hyperscaler stuff. So we now have an open compute project. This was built in 2011. They built an open compute project foundation so that this is not only bound to interests of Facebook alone and Facebook is really a front runner in this project because there are many citations who tell that all Facebook data centers now are 100% open compute project compliant. OCP is the usual acronym for open compute project. If you look in detail in Wikipedia, there is some doubt whether this is really the case because the citation given there does not really contain this sentence but I think even if it goes in such a direction that 80% of their data centers are OCP compliant, this is a very, very good success rate. So if you look at what they announced very early on and what surprised me, they put into news fora and communities that they have a very standardized way to build their data centers. So the whole Facebook has only five types of servers, five types. If you look at your classical IT department and some of them have something like shopping cards where you can buy things and have a lot of variations on that. So going down to having really five types of servers and nothing more. You decide to not build anything special even for a price above what you usually have to pay. So this was the first sign in the direction what might come there. I already told that we focused first on network stuff. So not that much on what is available as server designs and storage designs in the open compute project as well because this can be a bit too much of a change. Because in the open compute project you have this decision where you go into different rack form factors where you have broader racks than usual. Usual racks are 19 inches. What the open compute project does has 21 inches. This makes it even more difficult to bring this into a traditional data center and traditional managers in IT operations. So we decided to go with network stuff alone. And something like the flagship in the network part of the open compute project is called Facebook Vetch. Vetch 100 because it's a 100 gigabit Ethernet technology and they built a design for a 32 port top of rack switch. It uses Broadcom's Tomahawk ASIC. This is a very usual switching circuit, special hardware. But you find it in many, many switches. The other things are not that important for our stuff so I will skip them. So what we now have is Facebook has a design called Vetch 100 and what can we do with that? The open compute project has had success in the hardware market as well. That means you do not have to take the designs from Github and try to figure out which manufacturer builds you a chassis, builds you a board and stuff like that. You can simply go to the market and buy these stuff. Edgecore is a company which builds the Vetch 100, a certain variation for that. And this has a quite low price point. And this is a way you can make yourself in an easy way comfortable with the open compute project. And without building special competencies in building hardware, you can buy this stuff. You can buy this stuff for very cheap. Does someone have an idea what such a switch might cost? 5000. Because what we now have is the design is open and many, many manufacturers can build this at a scale factor that is tremendous. Because what Amazon, Google not really building this hardware on their own. They have manufacturers in the back. We do not often know who these manufacturers are. But this is something which is really a matter of mass production today. And that means that the price points for that are very low. And if, for example, you look at Edgecore, they are the mother company of Edgecore called Acton. And Acton is a company who by now produces hardware, for example, Euler Packard Enterprise. So there are connections from the mostly Asian-based companies doing this builds to other vendors. So the next slide tells us a bit what this market change is about. What does it mean to disrupt the traditional hardware market? It means going from the proprietary stack in the left column to a new design that is quite common in cloud infrastructures. You mostly have those cloth fabrics. Ross was an engineer, I think it was an American engineer, thinking about telephone switching networks. And he found out that in a certain way, if you build those networks, you can have non-blocking switching networks. That means you have enough connections so that another connection fits on. And this is something which is closely related to how we want to build networks and is the style we have usually today. Because the traditional network infrastructure is, very roughly speaking, more optimized to North-East traffic. And you have now to have designs which support East-West traffic and even greater amounts of East-West traffic very well. So you come to these cloth networks. And what we have to do as a transfer right here, because the graphics have very different sources, a cloth fabric has input, the switching part in the middle and output, ingress, middle output. And what you do if you talk about a cloth fabric in the data center networking world is, you fold the cloth network. That means input layer and output layer becomes the same. And this is what we usually have. So the blue boxes on the bottom are input and output. And on the top is what's in the middle. And one other type of name you find right here very often is a leaf-spine network. Where spine is a name for the layer on top and leaf is a name for the boxes on the bottom. And there are a lot of variations how you can build up these leaf-spine networks and we come to that in a minute. On the right is what really happened in the market. And why these effects were quite tremendous. We did not have those components with very high price points because network companies who were very experienced let customers pay for that experience. That means they build their own hardware, they build their own software and want to have a high margin on top of that. What the Open Compute project really did was breaking up this bundle. That means there is a certain manufacturer building the switching chips. This is Broadcom and there seems to be no company who can go into a real competition with Broadcom. But on the hardware side, you can now take the OCP designs and buy them or similar ones from different hardware companies and you have as a third part of this disaggregation different companies where you can buy your software driving not only a single switch but the whole leaf-spine network as a layer on specialized software coming for example from Big Switch or from Cumulus. I did not find Big Switch here. I think they are not here but Cumulus does something similar but we are able to focus here on Big Switch because I did not work that much with Cumulus right now but I can recommend to you, you can take those ideas and do that with Cumulus networks in an almost identical way or let's say similar way. So both sides of approaching this building a leaf-spine layer on top of an OpenStack reference architecture are very similar. So going a bit deeper into that. What Big Switch did is as part of the Open Compute project it brought something up with the called Open Network Linux, ONL. This is a traditional Linux distribution but enriched with respect to components for networking. So this is the network operating system. They put as a standard on all those switches coming from OCP. Now you have a guarantee that those bare-metal bright box hardware works with an operating system on top of that. And as network devices are different and have their variations, there is not one single network operations driving all those network switches around but you have certain additions that are proprietary. Because finally what Cumulus does, what Big Switch does, they want to make money, they are companies and want to get a certain revenue and drive their wins in. So what ONL looks like if we look at it from a perspective of cooperation architecture, we have this hardware, then we have a traditional CPU which is in 2-day switches most times Intel based and we have this special switch silicon, this ASIC on the right. Then the stacks are essentially integrated but they do different things. So the Linux kernel on top of the CPU is more for controlling the whole thing and the part on the switch silicon is more for making the packets fly as quick as they can. This at some point is a very special kind of programming because you are programming this special Broadcom ASICs. This is not the way general programming usually works and this has brought some success. So much success that for example the data center networking magic program from Gardner from July this year, so very actual, very still hot, has Big Switch networks, the blue dots a bit lower and a bit more to the left and cumulus networks on top in this visionary quadrant. And what you see as well is that those classical networking companies that come into mind when people talk about data center networking like Cisco, like Juniper, like Arista who sell you those components with a high price point. They are in the quadrant to the right and on the top. But this software only thing has been quite successful over the years and all those companies really know what they do. They are in the market for quite some years now and they have a lot of customers where this software really has to show that it works in a cloud in a hyperscaler scenario. So it was our decision to go in this direction as well. We had a first small prototype built on Big Switch networks and work perfect. So we have a layer 3 leaf spine network, layer 3 because you can build leaf spine networks based on layer 2 or based on layer 3. If you build them on layer 3 and make these choices like here is from my point of view the most scalable way you can think of right now. You go away from, for example, spending tree in traditional network architectures because you have now as a protocol ECMP, equal cost multiparthing. That means you do not have a protocol eliminating links but you have a protocol which enables you to use any available link and with EBGP you tell those components in the leaf spine network how they can reach each other and how packets can really fly. So what happens to the whole open stack thing is that we have two kinds of software defined network. We have one physical software defined network made up of big cloud fabric that is the software driving the leaf spine layer and those white box switches cabled as a leaf spine layer. This is the underlay network and on top of that why a network virtualization is what we classically built in open stack. That means this is built on VXLan encapsulation L2 over L3 and the connectivity with NSXT. We decided to go in that way but it has a bit to do that the customer had VMware deployments and we had to connect between the two. The big cloud fabric BCF and NSXT can be quite easily integrated so these were the design choices and we decided to go that way as a minimal viable product. That means going with this network components first and not doing the complicated stuff going into server designs and storage designs from the open compute project as well. This can come in the future but we decided to not enforce that right now. The essential open stack design is a very usual one. I put up a graphic with the more modern icons for the components. This is again to be understood as a minimal viable product because there are other components which might come on top of that in the near future. For example, analytics stuff built on Sahara or cluster stuff built on Magnum but these are easily integrated. The more important point of view is that we have defined something like a hardware abstraction layer. A hardware abstraction layer where we have the open stack distribution in the middle, multiple hardware vendors with different controller nodes and compute node designs where we can choose from or where the customer can choose from because this is interesting from our point of view as an integrator as well if we do not have to make an offer where we decide which vendor to go with but if we can leave this decision to the customer they feel much more comfortable because usually in customer environments you find something like for example buying history conflicts with certain vendors stuff like that. This is not technically related but if you make a design who fixes one vendor then it might be a reason for the customer to not go with your offer and so we made it a lot more general. On the storage side we decided to go with SEV. This is nothing surprising in the open stack world as well. It was a very good decision in that respect because they had very traditional storage vendors. Do not want to mention the name but everybody can imagine and if for example you look at how do they depart the image protocol of open stack, the block protocol of open stack and the file protocol especially Manila was a problem for them. When you find this support is not existing or unclear or left move from the vendor to the community all of that stuff and the only stable thing we could recommend was to go with SEV. And we have on the left side this hardware obstruction working through big switch cloud fabric and multiple hardware vendors for the switching hardware as well. So if you look at the side of big switch networks you find that you can choose from quite some technologies. The fastest being 100G, 32 ports. I think there are hardware devices available right now supporting 64 ports of 100G. So this whole world is changing quite fast in that way. And if you look at the open networking suppliers, the manufacturers of those bright boxes and white boxes, we find quite some icons. We find very familiar ones like Dell, like Eulah Packard and this work that way that big switch networks does certification for these network products. So you do not really read in a brochure. This might support open network Linux but this is a matter of certification and we decided to only recommend to our customer the ones in the yellow box right there because these are the gold certified devices. So we can be very sure that open network is running on that. So our essential design was like that. We had three sides, each with three fire protection areas. The three fire protection areas come from the CEPH part because we wanted with a replication factor of one plus two having a design where even if a fire protection area goes down we have the whole side surviving and recoverable. We have the network layer on top. We have the compute layer in between and have the storage component right below and I left those boxes in the middle out because I think if you want to build something similar then this is something like a breathing design. In the one situation you have to provide more storage. In the other situation you have to provide more compute. But what I have at the final slides in this talk is that from our purchase list I brought the network device specification, the compute device specification and the storage specification and if you would like more details we can mail about that. So this is open material. I will not talk about vendors in another detail level than I have here and I cannot tell who the customer is. This is clear of course. But I went to our purchase design department with exactly this list and they were very happy because usually they get a list where people are told you have to buy for example from HP server XYZ, exactly 16 cores and 250 GB of RAM and this usually leads to not that many offers and not really having a competition between those different vendors. Though they really like the style that I on the open network switch level only told about recommended candidates because of what the certification status is and on the other hand only talking about quantitative measures and compute nodes is a bit higher one, not the one-rec unit form factor but the compute node is again a one-rec unit factor connected with two times 25 gigabit ethernet. We have those 100 gigabit ethernet switches on top. That means each port is divided into four so we have a very high speed interconnect for the compute nodes and we have an even greater speed at the storage nodes because I decided to make the nick for storage nodes having even four of these ports so saturating our storage nodes is something which we were not successful with because this is a bleeding edge so to speak. This would be my talk. Maybe we have time for two or three questions. I just talked about how the price points are and if you look at a price point for the hardware at about 50,000 then the price point of the software per switch is almost the same so this is another price point than what you get if you look into the pricing sheets of the bigger windows in the top right quadrant of the Gardner magic quadrant. So thank you very much. If you would have questions or want to have the slides or whatsoever just mail me. I cannot give you every detail of this customer engagement but a lot of that and I tried my best to abstract it in a way that I can talk about it to a greater audience. Thank you.