 I hope you all enjoy watching the glorious task, which is connecting laptop to the thing projector. I'm going to apologize. Probably everything that comes out of my mouth is going to be ridiculous, because it's been one of those really full days today. And I don't even know what my own name is right now, which is not entirely true. So thanks for coming and to hear us talk. We're going to chat about an idea called Open Cloud Exchange from Massachusetts Open Cloud. I'm going to talk a bit at the top and then pass it over to Christie. But first, just to introduce myself, hi. My name is Monty. I work in the office of the CTO at Red Hat. And people tell me that I am responsible for various things in OpenStack, which should terrify everybody. But I guess there's life. Do you want to introduce me? I'm Christie. I'm a software engineer with MOC. Yeah. And something, something Keystone, something, something. Oh, yeah. I also work upstream on Keystone, yeah. Cool. So as we're going to chat about, and also for those of you who know my tendency to run over time on presentations, given that we're starting a little bit late, that's even more so. And what's best is when I spend half of the time of the talk talking about how I'm going to go over time. So it's really a self-fulfilling prophecy. But I wouldn't want to leave anybody disappointed who's looking forward to seeing me try and squeeze in too much content in a thing. So anyway, we're going to talk about Cloud Landscape. It's out there, which drives use cases for this idea of an Open Cloud Exchange. And then we're going to talk about the, can we still call it the Massachusetts Open Cloud or are we just supposed to call it the MOC now? The MOC or the Mass Open Cloud? The Mass Open Cloud, the Mass Open Cloud. That definitely makes it better. It's too long. Yeah. Something, something, something. Yeah. So the thing that formerly known as the Massachusetts Open Cloud, which apparently has been rebranded because that's too much. And then Christie's going to talk about what they've done, lessons learned, and where we might be able to go next with that. So I'm sure we're all aware that there are these things called clouds. If you are at this point in the day at this conference, not yet aware of the concept of I as clouds, I actually really like to talk to you because I'd like to figure out how you've made it here without that. By and large, there's an interesting, you know, Mark talked this morning in the keynote about the sort of power of open collaboration. And we get all these different companies to get together and work on the software, which is fantastic and awesome. The clouds themselves that are out there, especially in the public cloud space, are typically run by one company, which is sort of potentially unfortunate if you think that a single company might be a bottleneck or a dangerous part of your chain. Those companies are typically pretty secretive about operational factors. We get really excited when somebody gets up on the stage and they talk about the specifics of their deployment. Like we ran these mini cores, and we deployed it on this specific hardware profile. And remember that the San Diego summit, Troy Toman from Rackspace got up and gave a bunch of specifics. And it was like, oh, my God. He told us exactly what they're doing. And people freaked out. And so it's tougher. We've gotten really good with open source. Open Operations is a completely new idea. We do it a bit in the OpenStack infrastructure team, which is the team that runs the developer tooling for OpenStack. Lots of people think that we run OpenStack, which we do not do. We have no idea what we're doing there. But it's a new thing. People aren't used to sharing and collaborating once you get to the operations layer. Like it's not a thing that they do. And that means that our exposure to information about how to optimize things, how to make things better at that layer is less. We know how to optimize software. We know how to collaborate on optimizing software. But we don't know as much other than anecdotally, if you happen to go get beer with the right person, you might learn something about how to optimize the operations of things. But that's not really a scalable process for life, unless you really like beer, which I guess I'm in Germany, so I should assume that we all do. And ultimately, even when people are running OpenStack, you can still wind up with vendor lock-in. There's things that OpenStack doesn't do. We'll mention a couple of those. But you get locked into various aspects of those extra things unless you very explicitly tie yourself only to that, to the core OpenStack pieces. And that can be hard to do, depending on where you're at. And even with the votes for the best of intention, there's holes they're having to fill in that just don't exist in the upstream thing. And so you can't blame them. Like it's not a, they're not trying to put you in a bad position, but it's still kind of there. So this idea came up. And I, golly, I think it was the first Vancouver summit, is the first place that Orrin, also the first Vancouver summit, the first Boston summit, we're repeating ourselves too much. I got to like, indicate which of the times we were in a city. And again, I'm just talking about myself. Came up with this idea of, we're missing some words on some of those clouds, aren't we? Whoops. I don't know where the words went. I was pretty sure there was words there. Sorry about that. But this idea that rather than just have one company run one cloud and operate that cloud and have the, we all know how different people are providing software services on top of a particular cloud. There's lots of people deliver services on top of Amazon and Google and those things. Can you have a collaborative environment where different companies can provide the underlying IaaS services into what you call an open cloud exchange? So it's in the realm of like a stock exchange, so that there's a marketplace for those services. So you might have two or three or four different storage vendors, for instance, all having storage service implementations underneath the underside of that cloud and that a user of that cloud can then select, I want to get some storage from this vendor or that vendor or this other vendor, is that even possible? So that was the sort of impetus for trying to work on this and trying to dig in. So the open cloud exchange is about sort of solving this, that you can have these different participants show up, stand up different services, and make them available to a user base. This allows us to have people collaborate much in the same way that we've been trying to pioneer in OpenSack development land. Pioneer is a strong word, I apologize. But we've been focusing in the OpenSack on people collaborating across organization on the development of things to collaborate on service offerings to actually be able to run that potentially reducing cost, access to data, especially this is sort of come out of university engagements. You've got lots of people doing research. You've got those sorts of activities that aren't necessarily trying to monetize directly the services. They just want to work with each other on these things and be able to facilitate that. And also, and we've got a little bit more in-depth in a later slide on this, that we can actually get a better feedback loop. So that Morgan and I are working on software upstream, neither one of us happened to be running that software in production. And so we're filtering feedback from users through maybe a vendor, and then through another person, and then there's a bug report. And you're like, I'd like to know some more information about how that's working. Can I see your operational logs? And they're like, yeah, that's going to happen. So that's kind of tricky. And this can be important, even if this isn't like a global scale cloud. A lot of people that I talk to when we talk about cloud things get really caught up in the, are you competing directly with Amazon, with this cloud? I'm like, well, OK, we don't have to all have. It's not necessary for every cloud we spin up to be at that size. We can collaborate on the things that are hard and do that. So I think I actually said all everything on that slide in the previous one. Sorry about that. So as, like I was saying with the Morgan thing, as we move forward in these things, as we look into this, having collaborative community access to data about real users, access to scale, most of the, a shocking number of the people who are developing OpenSack don't have access to bare metal resources. And in fact, the 8 gig VMs that we give them to run integration testing for OpenSack in the gate is often free. Those are actually sized because we know that we can count on somebody having a laptop with 8 gigs of RAM. And we want people to be able to debug the OpenSack software on their laptop. Most people run clouds that are larger than a single laptop. That might be shocking. But access to scale, I think, is a really important piece in this. And it is a real missing piece of the puzzle in terms of completing the loop. Those of you who are familiar, it strikes me that that's actually a really old book right now. Were you born when that book came out? When did it come out? Say 99? Yeah, it was. OK, thank God. No worries. I'm just making sure. He's young. You have those moments where you're like, oh, God, that book's been around a long time. But the final reason of why we care about this is that we don't even necessarily know about all of the things that can happen as we open up collaboration across communities on new topics. I don't think that any of us, when we started OpenSack, would have imagined the size and scale. And I certainly wouldn't have imagined that AT&T would be making 5G calls from outside of a ferrity cage on stage on top of an OpenSack cloud when we were doing that. But those are those things you can't predict them. Like, they're what comes out of there. So Cathedral and Bazaar style, we've got these big giant proprietary public clouds. There's more of them than the US tech press likes to mention. But they're still there. So we're starting to see some of these things over here on the right-hand side. We've got MassOpenCloud, which Christy's going to talk more about. We've got other services that people are starting to provide. The Open Data Hub project, there's a third person who's supposed to be up on stage here bailed on us who would have talked more about that. So I'm hoping he's watching the livestream of this video and just know we're thinking about you. We're sorry you're not here. That's why we care. And I think I've already covered a bit of this. But we run, as I talked about this morning with this also, we run a crap load of CI inside of OpenSack, probably an unprecedentedly large amount of it. It's kind of an insane amount of that. But the thing we don't do is we don't have the delivery deployment part of that down. There isn't a cloud that all of those patches are flowing into that the OpenSack developers have access to. Some of them are running clouds. Some of them are not running clouds. So there's pieces. We do lots of synthetic testing. We try and do our best to set these things up. But we're missing a feedback loop. And so if there is a community-oriented cloud, a few years ago, HP donated a couple racks worth of hardware to the OpenSack project. And it was we actually, so the OpenSack infrastructure team actually did run an OpenSack cloud. It was really weird because we told people we don't run OpenSack clouds and then we were running an OpenSack cloud. But it wasn't for the purposes, it wasn't for this. It wasn't for the purpose of having an open and transparent operational loop so that as things are flowing through the community, they can flow into a running cloud and we can get feedback from them. It was so that we could have more testing resources. It was a special purpose cloud that we just used to get more VM capacity because that was all that was able to be given to us at that point in time. So there isn't a equivalent place where the community can collaborate on those sorts of issues other than just talking to each other over email because of their actual things. And as I'm sure we all know, there are issues that don't show up at eight gigs of RAM of size. And that also might be shocking, but things like that database index is not good enough. If you've only got 1,000 records in a test database, you're not gonna notice that. Might be an important thing. In fact, that's actually several years ago, Rackspace had a third-party CI that was running all of the Nova migrations on their actual production data, a copy of their production database on every change because of this very reason. Like we kept landing database migrations so when they went to roll them out of their public cloud, like took the cloud down for like a period of time and they got, I don't know, they found that frustrating. So anyways, this is a thing. Just having access to a quantity, having access to a larger thing is a potentially great thing. So that said, if you run such a cloud, it's a really great collaborative effort. There's use cases that are gonna be good for running on top of that cloud and there's use cases that are not gonna be good for running on top of that cloud. I probably wouldn't run a Tinty's 5G mobile network on top of the community-operated cloud, not because it's not gonna be good for software but just because, let's be honest, this is gonna be in a more experimental state. Being able to use that cloud to sort of like the one that we ran in in for a few years ago for a minute until the hardware got put into a data center in Houston and then it was flooded in Houston and then we didn't have hardware anymore because of the hurricanes. Apparently they were also just not even on rack. They were just sitting on top of each other in a stack. It was very strange. So being able to provide that to the community for CICD is a really great thing because we already have eight, nine different clouds. So if one of them goes away for a minute, it's okay. It doesn't affect us. It doesn't kill us in production and it's great that we've got more capacity. Doing scale testing on things. Again, if you've got a thing where it's okay for you to spin up 1,000 VMs in the cloud because it's not gonna kill somebody's profit margins, that's kind of a really cool thing. Data analytics and AI and L. These same things. These intensive, cool sort of cutting edge things. These are great workloads to try out on such a cloud. These are things where we can collaborate on the cloud but then having load on that cloud is useful to the people collaborating on the running of the cloud because a big cloud that doesn't have any users, well, you're not really getting any feedback on how it works at scale, then, are you? So you've gotta have this virtual circle but this gives us an opportunity to both give resources to potentially under-resourced research projects as well as learn from the operation of it itself. We're into some things we've learned so I think it's about time for me to, is that, is that your slide? You can do this as well if you want. We're getting close to his turn and I will just keep running my mouth. I'm trying to be cognizant of that. But yeah, so we've got, there's a lot of alternatives but learning and operating abroad, operators don't want to sit out there and learn about running my open stack on top of MySQL and Postgres. Nobody does that. That's a terrible idea as an operator of an actual service. You run the service the way you're gonna run it and doing things like running some of the newer open stack services maybe even before they're ready, well that's not a thing you're gonna do in a commercial public cloud because you would all of a sudden you've got customers on the hook for that. So it's a thing that you can learn. Individually it's hard for a lot of the smaller clouds to get the types of scale that we're talking about for doing massive scale testing and there's just other things. So the highlight of the thing there is the sort of key thing, without having real users and real workloads without collaborating on the running of that there's a real informational missed opportunity and then it makes it harder for us as a community to address real world, real customer usage without sort of an intermediary on our behalf. With that I'm gonna let him talk and I'm gonna try and stop talking. It's so cool. Not really gonna work. I'm not gonna let him really talk. I'm gonna try, I promise. Thanks Monty for doing pretty good motivation while you should have an OCX. I will now try to speak about how we're trying to turn this vision of an open cloud exchange and how we're trying to make it real. So we're a nonprofit and we have a very small team so it's three developers, one operator and a bunch of student minions who come and go all the time and even so we are trying to build this as a nonprofit. We still want it to be sustainable so we wanna charge and have actual customers on top of it. This all means that we should have charge back, show back, building reporting, all of these features which a public cloud generally has are required in our cloud as well. And OpenStack was not really built with these things from the get go and in mind and that's one thing where the collaboration that we're having with OpenStack and the upstream and everyone else is so important. So what is the MOC? It's a collaboration between five universities of the Boston area. So it's UMass, Harvard, MIT, Boston University and Northeastern but not only that it's also a lot of industry collaborators. And one of the things that makes us special is the MGHPCC which is a huge data center hosted out of Olioke. It has hundreds of thousands, space for hundreds of thousands of cores with room to grow. And another cool thing about the Boston area is that if you take all the Pacific Research Platform and you shrink it to the size of a building you basically have the MGHPCC which is where we're hosted from. So what are the capabilities which we have to build to have an OCX, like an MVP of an OCX? So first we need an elastic secure infrastructure for on-demand hardware use, a production OpenStack-SEF and Kubernetes services for both end users and high level service offerings, single sign-on, resource federation with multiple OpenStack services, a pricing guide and building system and user management. So, and once all of that is done participants will be able to deploy their own hardware, deploy services on top of hardware allocated dynamically for the purpose and charge for the services. So build a sustainable marketplace. So let's go one by one. So an elastic secure infrastructure for on-demand hardware use. We actually have built that. It's called M2. It's built on top of Hill, which is a service we have developed which does network isolation at the switch level. So each nodes gets its own VLAN and users communicate with an API to reserve nodes and give them back to a pool of shared nodes. We also have M2 which is network booting and image management. And one of the cool things about network booting is that it allows you to have VM style image management so you can suspend a VM, suspend a bare metal machine, give it back to the pool of machines then get it back and use it for something else. And assume the job you were doing after you get it back without even noticing. And we're also working on having a snapshotting of the memory so you could just pick up exactly where you left off. But this wouldn't really be useful if it wasn't safe. Like say you gave a machine up and then you got it back and someone messed up with a firmware. So we're working on Bolted which is a project which combines Heal M2 and Kilimata Station from MIT Lincoln Labs to make sure that bare metal machines were intempered with the time that you gave them back. And you can do this verification. You don't have to trust someone else. So that was the first one. Then you need a production open stock, self and Kubernetes which is harder than it sounds. Trust me. So we started. It's easy. Of course. So we started with 500 cores but 1,600 cores will soon be available. We started at Kilo and we have Pyke now. Upgrading was a rough process because we were using puppet manifest from the Kilo era and we just forked them and slowly started patching them up to support Pyke. And this grew very, very hard. And now we're planning a redeployment with Pyke with all supported bits. So we won't have to mess anymore with changing puppet manifest to that extent that we have. We also have a pure research open stock cloud which we only mostly reserve for researchers and it has GPU nodes and stuff like that. But we do not support it as much as the pure, as the production cloud. We also have an open shift running on top of open stock and upgrades with it have been pretty hit or miss going from one version to another. It's almost like you're saying that upgrades are hard. Upgrades are always hard. So lessons learned. Cep is hard. We got bitten by the Velosev Raptors and user support is very time consuming especially when you have a team of one ops. So, SSO. So yeah, we want users to come in and have one single account and with that account be able to use our multiple production, our open stock clouds, open shift and the other services which we deploy like dataverse, open data hub, et cetera. So, we have federated with Incommon which is a federation of universities and we use key cloak as an IDP proxies. So users go into open stock, they get redirected to key cloak and then the key cloak sends them back to Incommon and their key cloak session is valid for all the open stock or the MOC services that we have deployed on the cloud. This was of course more complicated than it looks because key cloak doesn't really support federations. So we had to build another IDP proxy layer which is composed of shibboleth and simple SAML PHP. Hopefully we will be able to roll this into Keystone. We are talking with other Keystone folks about this and some specs will be pushed up for review and you can take a look at them and contribute to the development, open source development process. There is in fact a talk about that. Do you keep going in one second? So, it's all good. Now a user is able to log in and come to each of the services but what if they wanna combine the resources they have in one open stock cloud with another open stock cloud or with OpenShift or use their data sets from dataverse into open stock. For that we have built resource federation. So say a user has, so the MOC isn't really like a single cloud but think of it as a federation of smaller clouds. So do you run their own infrastructure? Northeastern runs their own infrastructure, MIT runs their own infrastructure and the user has a machine which is running on BU and wants to attach a volume running from Northeastern. Through resource federation they should be able to accomplish this and for this purpose we have worked, we have built Mix and Match which is an open source API proxy. It does Keystone to Keystone federation between open stock clouds and forward API requests to the appropriate deployment. So the Nova service will be able to use Cinder volumes from a different service in a different cloud. And wow, we went through four, now there's five and six. So probably the harder ones. Pricing guide and billing system and I guess you're all curious to know how the pricing would work and we actually expect to be able to charge half or one third of commercial big clouds and we also need to figure out other models because research is different. So sometimes researchers have grants for purchasing hardware but do not get grants for subscribing to clouds. So we should have a model where we allow them to set up their hardware into our cloud and then they can use that for credits or we're still working on this but there's a lot of cool things to think about. And yeah, of course, users want to have something that works and lets them solve the problem and that's especially true for researchers which have very strict deadlines. So in terms of billing, it should be per project, the organization which is sponsoring the project. Well, this is boring stuff but it's very hard to get all this information. Projects then should be aggregated by institution because the institution should receive the bill for all the projects that they're using and we have succeeded in proving that those 10,000 ways will not work. It actually was so much harder than we expected, especially gathering telemetry data from OpenStack. When we started about three and a half years ago, Syllometer was very slow and it made our network crash. We investigated Monoska, then we switched back to Syllometer because a tool we tried to use to solve that issue required it but it didn't fit our use case and now we're back to reinventing the wheel yet again and it would be so much better if we could work with OpenStack community to have this upstream. Yes, this is the hardware we'll need the most help. Then user management system. You as a user or I as a user want to go into the open mass open cloud, sign up for multiple services and when I leave I want my resources to be automatically returned for other users. This is one of the harder problems in OpenStack especially reclaiming the resources and this is another area where we should collaborate with the OpenStack community and we actually have Adrienne Terjack here who developed Adjutant which should help us optimize or build these workflows into an actual official OpenStack project rather than yet again reinventing the wheel. I think there's a session this week on that too, really auto deleting or deleting cleanup. Yeah, it's next. It's next. Excellent, good. So yeah. Yeah, the identity federation proxy thing is tomorrow at 11.40 by the way. Yeah, so one important missing feature in OpenStack is that users of projects and organizations should be able to sign up for resources without requiring human intervention. Everyone has gone through that stage of their OpenStack career where they had to write this. Maybe we should do something about that. Yay! So other stuff we've been working on we have a bunch of FPGAs and GPUs into OpenStack and OpenShift. We are reaching out to startups in the area so our audience is not just researchers and students but it's also startups and manufacturers and every user which has an interest in using our OpenCloud to research IT organizations which want to offload their infrastructure or help us in collaborative running our infrastructure. More things we've learned. Yeah, every time someone says a feature will work make sure that someone has tried it before actually pick flowing into the production. It's a very important question. And if your only senior infrastructure engineer is out of town, run. The community has always been incredibly supportive and giving so thanks to everyone of you who is here and who has helped us. Thanks a lot and we still want your help so we're not fully through the hardest parts yet. What sorts of things do you need? User, developers, money, everything you can throw at us. I'll take it. Do you need more requirements? You go with use cases? You just throw some use cases over the wall. Would that be helpful? Probably not. We're already swimming in use cases. There's a link there to a very horribly designed website that's something we're malisonian helping. Yeah, well thanks everyone. And we are at time. So we actually, I think have, we have zero minutes for questions. So if anybody would like to ask or we could probably squeeze one in before they kill us for going over time. Yes, go for it. Good day. I was just wondering around a platform you look at, you know, AWS and you know, even I like tunes and all that sort of thing. It's very much about the content, the applications. Is there any thinking in the community around creating like a application repository sort of for this open state community? Like a juju charms or a doing, but connecting these community clouds into that? So... Interesting question. I know that there was, so there was a thing that doesn't exist anymore called, there was the Murano thing, but there was a public Murano repository that Olaf was PTL of for a second. I can't remember the name of it, but yeah. So that actually at the, that was spun up around the first Vancouver summit. So somebody sort of started going down that road. And unfortunately it didn't get anywhere. So it was one of those sort of things where it's a really good idea and nobody showed up with content essentially. So it would probably be really worthwhile to, I don't want to hand Christy another use case, but I do think that's definitely a thing that would be really helpful. It's an important use case because if we build a marketplace of services, like the service doesn't have to be open stock, it can be an application. Like if you were in yesterday's keynotes, you saw Red Hat talk about Chris. It's an application for radiologists to consult and run processing jobs in our cloud. So that's one type of application that can be exposed to users. Also if the pricing stuff works, then people offering their services in our cloud also will get money back. So there's more of an incentive to provide services. Whereas in Murano, probably you get nothing except photos, so. She's pulling on as great. Yes, if people get money out of it, people will provide their services, same as the App Store and other kind of stuff. Yeah, cool. I think they're gonna kick us out of here. So thank you very much.