 And the first project lead I'm going to pass it over to is for Savannah, Sergei, and have his slides here. Please let me know for some reason you can't see them or having difficulties, but otherwise everyone please mute your lines and send questions in through the chat function on Meeting Birdr. Sergei, are you there and ready? Yeah, thank you. As far as slides, it looks okay. Okay, first of all, thanks for the OpenState Foundation for Make Us Able to share a project status update. So let's start. Could you please open the next slide? Okay, my name is Sergei Okanov. I am the project technical lead for the data processing program in OpenStack. It consists mainly of Savannah project and some sub-projects related to it, like disk image builder elements, OpenStack dashboard, plug-in and something else. Let's start from the HiveN release overview. We had about 70 bullet points implemented in HiveN, about 150 bucks fixed. I am very excited to see that we have about 900 code commits from 31 people and more than 3,000 code reviews. I was the most important achievement for the HiveN release is that we were officially incubated for the Ice House release cycle. You can find more details on our LaunchPad page about the HiveN release. I'd like to add some notes. For the HiveN release, we have no adjusted release cycle to OpenStack and we had two surprise reviews for this time range, all the two and all the three. You can find release notes on the wiki page. So let's move on to the next slide about the HiveN highlight. So I'd like to mention that we grown from the proof of concept to the really good Continuity-backed project in HiveN release. It was very, very excited to see how we achieved new contributors and the features for the last nine months. As for the contributors in core, we have folks from Durantys, HoughtonVolks, and Red Hat. And in addition, we had a lot of contribution from HP, IBM, and I just taken Red Space and it looks very, very pleasure for us. In the HiveN, we had two plugins, vanilla plugins, and installs vanilla Apache Hadoop with related data processing tools. And we had HoughtonVolks data platform plugins that installs Apache Unbarry management platform that could install Hadoop and a lot of related tools. So it's a plugin installs management system that helps to install a lot of stuff on cluster nodes. So as I mentioned, we have cluster team, cluster API, and supporting HiveN release. We have elastic data processing, and of course we have integration with OpenStack dashboards. Each country lives in a separate repository and could be enabled to the plugin for the OpenStack dashboard. Okay, let's take a look on the cluster feature in Savannah. First of all, it provides an ability for users to provision cluster for different sizes and topologies with a lot of configurations that could be passed to Hadoop and related tools. So we have a configuration management system. It works on template mechanism, including two plates for both clusters and node groups. It makes users able to define different configurations and store them in Savannah, and provision different clusters in one click manner. We support cluster scaling for both adding and removing nodes and the cluster for the data nodes we support and the condition that makes users able to remove data nodes from the Hadoop cluster too. As a very interesting feature, we support data nodes into affinity. It works like a schedule hinge in Nova. In fact, we pass some schedule hinge to Nova when creating instances. It deploys cluster in Savannah to have only one data node per computer host, per physical computer host I mean. It makes Hadoop cluster reliable and consistent. It makes data consistent. So we have some performance testing that it shows that Savannah could deploy about 200 of nodes in just five minutes. So it's a good performance for Hadoop cluster provision, especially in comparison with manual Hadoop cluster provision. So the next big feature in Savannah release and I think that's the main feature of Savannah that's an elastic data processing. In fact, it's an API to execute or produce jobs without exposing any details of underlying infrastructure. It means that you can know exactly anything about the operation system, start on nodes, about how which works, about how OpenStack host configures, maybe Neutron could be used or Nova network, Cinder could be used for volumes and etc. And after that we can provision using cluster in some cluster. That's an option for EDP too because we can just push our jobs and EDP will set up the cluster only for these job ends and terminate it after we get the results. It works like Amazon's EMR service that provides the same ability but it's not open source and it works only on Amazon and provides a very short list of different options and types of Hadoop tools. So let's take a look at some internals of EDP. We're providing a concept of data sources. Currently we support Swift and external HDFS. For the heavy memory you can see we support on the Swift. It's a data source. It could be used for both input and output data. As for the different job types, we're working with pure Java and jobs, Pigeon-hype jobs. Both Pigeon-hype are SQL-like scripts and languages for writing map-reduced jobs in much more short manner than custom Java or some other languages. And of course we have user-friendly UI for ad hoc and latex based on hypo peak. So you can just go to the UI and upload your hypes script and it will work on the data that already exists on Swift, for example. Okay, that's one group is open the next slide. Thank you. Let's take a look at some stuff for Icehouse. Currently we're working on Icehouse 2 Development milestone. It will be released next week, I hope. And so far we have about 24 Blueprints implemented, 43 bugs fixed, and about 200 of code commits from 26 people. And of course we have some code reviews, about 1,500. Next slide, please. For other Icehouse highlights, I'd like to mention that we have a bunch of new contributors, saving the old one too. It's rather some of them, innovating some soon into E-Palm systems at Zurich University. I'd like to thank all our contributors for making efforts for making Solana better and making new features and new plugins. As for the plugins, we have an intuition of how to plug in Icehouse. It was the first basic version that was already committed to Solana. And we're working now on making it better and support EDP feature and other stuff. So currently we have two sets of huge Hadoop vendors with Savannah plugins. So it's a good achievement too. And we're looking for the last one, for the Code Era distribution too. And the guys have Blueprints and some initial discussions about making a plugin for Savannah that will use and install Code Era. So in Icehouse we already have implemented heat integration for orchestration engine in Savannah. And we have complete integration with the stack and running some gating jobs in Savannah currently in a synchronous manner. But we hope to make the synchronous gating later in the Icehouse or early in the release. We currently have not basic integration with Tempest and the basic API test and review. And I hope that it will be more soon. And we have CLI proof of concept. It was included to the latest Solana client release that was ordered for that one several days ago. As for the Icehouse plans, the main plan is to make heat the default orchestration engine for Savannah. We currently have mostly all features with the lack of maybe one or two of them. I mean the heat engine. I think that it's possible to make heat the default orchestration early in Icehouse, probably in Icehouse 3, developing my milestone. And of course the very important parts of Icehouse release should be Tempest integration who are planning to move our currently implemented and used integration test out of the Savannah. And our integration test framework to the Tempest scenario stats and we'd like to add some CLI tests too. As for the API, we're looking for the implementation and releasing the V2 API that would be based on Tikan and Wisni. It should be some kind of polished and more consistent API with probably some new features but mostly it should be just an update consistent and user-friendly with our current API. And of course we'd like to complete our CLI implementation. And the last but not the least point is the guest-agents proof-of-concept who are now working hard on implementing some kind of proof-of-concept for agents and we are proposing an idea of having unified guest-agents in OpenStack for different services to be able to not duplicate efforts on making agents between different projects. It was already discussed, usually discussed in the mailing list and we'd like to return back with some proof-of-concepts later. So I think that's all for the product status implementation date. Thank you. Any questions? Great. We'll switch it back to the chat to see if anything's come up in chat. I don't see any immediate questions. So we will move on to Michael and then see if we have any questions at the end if that sounds good. Yep. You ready, Michael? Can everyone hear me? Yeah, can everyone hear me? I can hear you perfectly. Thank you. Okay, then it sounds like we're good to go. All right. Howdy everybody. My name is Michael Bassknight. I am the PTL for Trove, previously named Dredd Dwarf. We've been around for a few years now at Rackspace and we've been working on this product for a while. Now we're working. All right. Now we're having some talk. Please talk. No. Okay, so I guess we are now officially integrated, which is really awesome. We were incubated in Havana watching Sergey's project update. I recognize a lot of the things that they're working on as things we're working on right now and things that we had to do during our incubation cycle as well. So I wish him the best of luck in getting that stuff done. It's a fun six months going from seeing very little traction on your project to having 10 random companies and people hopping in your channels every day asking how that can help. So right now we are officially integrated and what I'll first do is talk about some of the things that we did in Havana. I'm going to focus a little less on some of the plumbing and get to the few features we actually got to implement while we were doing all the plumbing work to get integrated. So our friends at Marantis did the first feature and that was really awesome, RPM integration. So previously our guest images only worked on Debian-based OSes. So that made it really hard for people who wanted to test, say our guest spinning up and running MySQL in Santos or Red Hat Fedora, any of those. So the guys at Marantis focused on that early on. That was one of the first features they worked on and knocked it out of the park and now we've got an equal footing on both RPM and apps-based distros for installation. And of course for running it, there's small differences between all the different OSes that we have to kind of keep handled on as well. So JSON Schema Validation was another one. We were hand-rolling a decent amount of validation and this came up as part of something that the OpenSec community was looking into so we decided to jump on the bandwagon. And it actually provided a really clean, like single place to put all your validation, a single place to look at your validation and it's removed from the actual code that does the work, the business logic so to speak. So it's a little less bothersome when people are doing business-logically kind of work. So Notifications is another one. At Rackspace we had kind of done some external stuff with notifications for our billing system and so we worked with the guys at HP to kind of bridge that together into an internal modification system that could send those messages on the message bus to whatever was interested in consuming them. And from what I understand, the HP guys are using that as well in their deployed stack. Modified Users is another one. This is one of the good forward-facing features we got to work on. And that was effectively being able to be modified by a user's password or the host that they can log in from in MySQL and granting users access to databases, removing them from... sorry, revoking them from databases, listing their access and stuff like that. We still don't have fine-grained user access, but given that it's an extension for MySQL, it's not something that we necessarily are going to put on the top of our list right now in the community. I mean, if a company wants to come along and do it, then that's great. And the last one that we really worked on, sorry if you guys can hear my dog, she just decided to get up because I'm talking now, is Volume Extend. So our friends at United Stack helped us get this working at Rackspace. We had built this in before Cinder had actually had the ability to do it. So once Cinder had the ability to extend or grow volumes, we then built that into the API proper. For a little bit of history, it was actually in the CLI, but it wasn't in the public code base itself because it didn't exist. So those are the big features we had for Havana. And this is interesting because it's supposed to... I don't think you're on Presentation View because those are supposed to pop in one at a time. Oh, I'm sorry. When I was on Presentation View, it actually just shows up as black. Oh, it's working now. Sorry, here we go. It's okay. Yeah. So a good amount of Havana was spent on integrating with OpenStack. There's still a lot more to come in terms of integration, and I'll discuss some of those things in Icehouse features. But some of the other things we did in Havana that were plumbing-related were heat integration. It's being built out better now, and we're going to use heat integration as the default installation for clusters. So it's actually going to be the only implementation for clustering in an application. And integration with DevStack and all of those other things that end-users don't care about, but the developer community does care about. So on to Icehouse. We've implemented some of these, and of course we still have, like Sergey said, we're just about to release I2. So then after I2 you get the I3 milestone and then go into bug fix and release candidate mode to hammer out the release. So some of the features that we worked on, and you know, Miranda's also helped with this one. They've been a really big help in the project, so that's really awesome. So multi-data store, we now have the ability to provision multiple data store types. Now this is something we've talked about for a while, and we went back to the TC to amend our mission statement, so to speak, to include more data stores. This is probably the biggest feature, I think, that is already implemented. Of course clustering and replication is going to be a bigger feature I'd like to talk about. But multi-data store is the stepping stones for being able to spin up Mongo and spin up Cassandra and spin up some interesting clustering technology for MySQL. There's a ton of them. It gives us the ability to, as a deployer, also say, hey, you can spin up a MySQL or a Redis instance or a Cassandra cluster, all in one installation. And of course on the back end, you can do some flavor tweaking and host aggregates to deploy the instances in different, you know, availability zones or whatever you call them in your Nova installation. The good thing about Trove, just like the other project that builds on top of Heat and Nova, is that the user can care less about where these things are and the operators can say, hey, all of MyRedis instances are going to go on these servers. So this is really an interesting feature for us because it gives us the ability. And another thing that it did was it actually brought more people to Trove because it wasn't just MySQL as a service anymore. It was data stores as a service. So, you know, we've got a lot of companies that have come to us and are interested in building out, you know, a Rockstar Cassandra or a Mongo or Redis installation. So the next thing, the next feature that I'm going to talk about is Trove Conductor. This is another really fun one that we did. So our guest sits on the instance that is running, say, MySQL. And that guest has a few functions, one is to make sure that the product that you're purchasing is online and happy. And when it does that, it needs to write a status somewhere. So previously, we were writing status directly to the, as I like to call it, the infrastructure database, which is the database that holds all of our information or the metadata database, but that's an overloaded term. The database that holds all the instance ID information, not the customer's database. So it's writing to the infrastructure database to say, hey, MySQL instance is happy. And then, of course, it's also getting messages from the system to say, hey, maybe create a user, create a database. But the main thing the conductor did was we noticed that there was a lot of chatter to the database from all of our instances and there's security concerns there. So we decided to move the communication over the message bus to a conductor very similar to the Nova conductor. So right now, the conductor is small and it's in charge of taking the information the guest gives it and distilling it to the database, to the infrastructure database. But the future of conductor could actually involve a different messaging layer to not maybe use RabbitMQ to use a different communication layer for these tens upon thousands upon hundreds of thousands of guests to store a in-memory version of the data so that it's a lot quicker to access so that you don't actually have to go to the database every time. So the conductor for us is the stepping stone for a much smarter communication platform from the guest to the rest of the infrastructure. And that is already, it's in DevStack. It spins up and runs and all of the data store writing information is sent to the guest right now. So that's phase one. So there's a blueprint that if anyone's interested then I can link later that talks about some of the other thoughts we've had about where we can go with the conductor. None of that obviously is set in stone yet because there's no code for it other than what I said it concretely does today. So moving on, this is a fun one. This is the exact same thing that the Savannah guys are going through right now. We need to be able to spin up instances and clusters. We need to be able to provide custom images built by the disk image builder, which is the, I don't know the name of the program, but triple O had built the disk image builder to build images so that you can install them onto your VMs. So it's a really easy command line tool that gives you the ability to write some scripts and put them in a specific hierarchy and then give it to disk image builder and it will then do those extra things on top of building like a base Ubuntu or base Fedora VM. And so all of these things that go into the gating procedure in order to run the Tempest test because you have to have a fully instrumented trove environment to run an accurate test scenario to actually spin up an instance and see what's going on. So that is a piece of what we're working on as well as the Tempest test themselves to actually spin up those instances. And the desktop work in order to say, you know, execute disk image builder and cache those images because they don't change nearly as often as the trove code base. So that's kind of a big pipeline that we're working on right now in order to get testing working a little bit better. Excuse me. For all of the projects that need images and need Tempest tests. And the next thing, so heat integration. This is something that we already have but we don't have a really amazing implementation of. So we have a basic implementation but it doesn't have a lot of features or it didn't in Havana have a lot of features like security groups, DNS, things like that. So we've had a group of individuals comprised of Mirantis and HP and a little bit of Rackspace really focused on making heat the default implementation for spinning up instances and spinning up clusters. I've already said that I don't want to see any old school, I guess the original way we install old school installations of clusters. It doesn't make any sense because heat is a perfect example of a resource to group, as they put it, a stack or a grouping of resources that are comprised of a logical unit like a cluster together. So they already do a lot of this work and they do all the installation too so that's something that we're going to build upon for clustering. And that's going to kind of fall into I believe it's the next one. Oh, just kidding. I guess I said the best to last. So configuration management, this is another I don't know, this is a headache for customers, right? You want to be able to really tweak out your data store and you don't want to, like for my SQL if it for some reason needs to be power cycled because the physical machine, something happened to it, you lose anything you saved that was just, I mean anything, any commands that you did that were dynamic changes to memory or to variables you lose. So the ability for a customer to say yes, you know, change some parameter to my particular data store and keep that saved. And every time I spin up a new instance, you know, use a group of beats that I have to find as, you know, this is what I want my WordPress blog to look like. So I'm spinning up my 8th WordPress blog, attach my WordPress template to it and edit those templates and be able to edit them across multiple instances and be able to delete things from those templates as well as to be able to see a list of the minimum and maximum values for those and a way to remove any values that operators would define as being, as something that could harm the user. For instance, if you're doing point in time recovery and the user has the ability to say turn bin logs off from my SQL, you don't want to do that. So we could, you know, as the operator you could remove specific things that you thought would cause harm. So all of those things go into configuration management. It's under review right now. It's been worked on by a few people and it's in a really good state right now. Our newest core member, Austin, has spent the last few days tirelessly pulling it down and testing it as much as he possibly could and giving Craig some really good feedback there. So that feature is going to be really awesome and it's going to be available in Icehouse. And so that brings us to the last big feature, clustering and replication. So I'm really hopeful that something good comes out of this in Icehouse. We're pretty far along and we've done a lot of talking about it and a lot of back and forth about clustering because we're trying to come up with a good API for clustering and replication for multiple data stores. And that's not easy. And we don't want to rev the API four times if we can not do that. So we haven't really spent a lot of time on it. We've got some really dedicated guys from Rackspace that are doing what they can. And an awesome company just came up and is very interested in helping us with this. So their name is Paralastic and they have really taken a lot of time to come up on the project and understand what's going on and they've been reading the blueprints and asking good questions. And they, as a group, I believe, want to really tackle how does clustering work for technologies that need it in Trove and building something and then saying, yeah, this work, this didn't work and maybe iterating on that a few times. So we may have something experimental in Icehouse for clustering and replication. I don't think we're going to have anything that's going to be a hard and fast API, though, just because I want to stress that it's something that we're going to have to iterate on a little bit before we say this is the well-defined API. Everyone go, deploy this, and get customers running on it. So I do think there will just be a little bit of give and take with the API for the first few revisions of it. So experimental, I mean, experimental support is to be determined with clustering and replication. I know that there are so often dudes they're going to really knock it out as fast as they can. So we'll see how that goes. It'll be really cool to have that in Icehouse as well. I mean, that's kind of, you know, if you think about it, that's kind of a jumping point onto really complex, you know, topologies with Trove. You know, we can then start thinking about in-tiered slaves on MySQL or, you know, multi-availability zone standard deploys. Some of the stuff that Savannah's actually already tackled. So we're going to, you know, talk to them and use their subject matter experts as well in the process of actually building out the API. And so I'll be trying to do as much of that kind of networking of groups of people, excuse me, as possible to really make this an awesome, an awesome product that we can then build, you know, really intelligent clusters from and have people go install them in their data centers and have fun. So I think that's the end. I think I have a cue and I, oh yeah, you're right. That did just totally turn black there. Yeah, it worked out though. Well, I have a quick question for you. You are now unmuted. I don't know if there's any other questions. You mentioned a few times, I think during the presentation, but as you guys are heading to be integrated with the Ice House release, could you talk a little bit about who is using Trove now? Obviously, you are using Trove now. Yeah, absolutely. So Rackspace is and has been running Trove in production for, oh gosh, I don't even know. I would have to ask someone about that. Yeah, it's been a really long time since it was red dwarf, since we were the only group really hacking on it. HP has it deployed and are running it in their environment. And I'm not sure if it's for pay or for play right now, as in beta or full charge. I can't speak to that, but I know they have it running and they're running trunk. And the guys over at eBay are running it as well in a former fashion. I'm not too, I don't know too much about it, but in terms of fully deployed pay for environments, Rackspace and HP, and then I believe there's some people that are doing smaller deployments, internal deployments, but no one's really, you know, no one stood up to say that they're doing it internally, like they're running it internally. So we still don't have a lot. I mean in the last, yeah, it is in the wild and it's been, you know, the last six, nine months, there's been a lot. It went from no one asking how to install it to a decent number of people asking questions about installation and making the documentation better over the course of the last few months. So even though it's fully deployed and pay for, in a lot of places I do think that it's gaining traction and I mean I've definitely seen that and I'm sure that the Savannah guys would say the exact same thing. I mean it goes from no one talks about your project ever except, you know, you may hear it mentioned twice at an OpenStack Summit by a random person to, you know, you have completely random people mentioning it at OpenStack Meetups that you're just sitting in, you know, on a given day. So, and then lots of people coming in and asking questions. So it's really, it's a really awesome process and it helps a lot for us, for traction. Well you guys must have been extremely thorough because we do not have any questions in the chat for you right now. But I wanted to say thank you to both Sergei and Michael for taking the time to put together these materials and coming to speak today. It's really valuable content for us to have and we'll make this recording available on their slides available after the fact as well. So thank you both very much and thanks everyone for attending and we'll keep doing more of these. Thank you. It was really fun to do. Great. Thanks Sergei. Thank you.