 Thanks, Patrick. Thanks for coming. Yes, I'm David Disadorp from SUSE, a work in the storage team there. And I'll be talking about just a project I worked on for a Hack Week at SUSE, which was basically a USB set gateway. So a quick look at the agenda. I'll start off just with an introduction to the project itself. And we'll take a look at or just have an overview of CEP, hopefully caught the talks earlier about the CEP architecture and how it solves all your storage problems. A look at USB storage, so the USB storage stack in Linux. Then move on to a demonstration. So I'll do a live demo with the board I have with me. I have a CEP cluster running on my laptop, so I'll use that for the test demo. And then look at how this could also be used for public cloud storage, where we have, say, Amazon or Azure behind the storage gateway. And finally, look at some future challenges. So what else could be done with a device like this or this sort of project and finish with a conclusion? Can everyone hear me OK? So the project itself was sort of conceived with Hack Week. So as an engineer at SUSE, we get normally once or twice a year an event, or we have an event called Hack Week. And within Hack Week, we can work on whatever we feel like working on. So it's really great time to sort of innovate within the company. We can work on something which we're sort of passionate about. So in my case, it was, I think, a year or so ago now. I had an onboard, so a QV truck sort of gathering dust in the corner. Wanted to do something with it. I wanted to learn something new. And I work on storage normally, so I thought, yeah, look, I'll combine these things and create a USB storage gateway. So in this case, we have basically our USB host. So any stupid device with a USB port, which nowadays is pretty much everything, I can then connect this USB gateway to my device and basically access the Ceph cluster or the Ceph storage through USD. So the goals of the project, yeah, initially, my main goal was just to get something usable. So I thought, OK, this could then be something at home. I could then potentially, say, run a Ceph cluster in my basement, then connect my stereo, my television, whatever else has a USB port, connect that and plumb it all into Ceph for storage. Another possibility would be booting from the Ceph cluster. So most laptops nowadays can boot from USB. So you plug in the gateway, you boot straight from the Ceph cluster or the cloud storage. Encryption on the device itself. So in cases where I don't trust the cloud storage, especially for public cloud, where I have no trust that they're going to keep my data safe. So I want to do encryption on my side as close to the client as possible. I can then keep the keys on the gateway, carry that with me, and, yeah, don't share anything with the public cloud or with the cloud. And finally, simple configurations. So something like this, I didn't want to need to SSH into the board itself every time I wanted to change the config there. That was also a goal. So now look at Ceph. Hopefully you caught the talks earlier. It's basically a project, an amazing project which provides the ability to aggregate a pool of hardware or the storage across that hardware and have a logical storage pool, which can then be carved up for block storage, for file system, for object storage. Yeah, it's scalable. There's no sort of single point of failure. In this case, for this project, I'm mostly just focused on the Rados block device interface for Ceph. So in this case, we have a block device image on the Ceph cluster, which is backed by individual objects on the Ceph object store for Rados at the back end. So the Ceph RBD feature or interface has some quite cool features. It offers, yeah, it's in provision. So basically, you're not consuming space within your Ceph cluster until you're writing to the image. Yeah, resizable, so you can grow and shrink your images there. You can also do things like snapshots and clones. And there's also support within the Linux kernel. So with Linux, you can basically map a Rados block device image. It appears as a local block device, and you could use that as you would any other disk. On the user space side, there's for QEMU's integration so you can run virtual machines with QEMU, and that's then backed by Rados block device images on the Ceph cluster. So now look at the hardware that I had for this USB gateway project. So I started with the top left there, which is a QB truck. Yeah, that's so a dual core Cortex, yeah, dual core A20, so all-in-A20 board with gigabit ethernet. Yeah, what else does it have? So it's got a USB on the Go port, which is obviously what's needed for this project. It has a bunch of other things on there which just really aren't required and make the board a whole lot bigger and less portable. So yeah, after that, I moved on to the Nanopio Neo, which is this board here. This is sort of, I think it's four centimeters by four centimeters, so it's very much something you could or I could imagine carrying with me and plugging it in and using it on the Go. Both of them are relatively inexpensive, so the Nanopio Neo is under $10. So certainly doable price-wise. Yeah, performance-wise, they're not great in that at least the Nanopio Neo has 100 megabits network and USB 2, so yeah, it would be nice to try something with more powerful hardware, but at this stage, it wasn't a priority, so. Yeah, the big benefit of using these boards is that they have, thanks to the Sunsea community, they have excellent mainline kernel support. There's also an open-soothed tumbleweed port for both boards. Which is something, obviously, working at SUSER, I wanted to run open SUSER on a board like that. So USB storage, I won't go into huge detail. Yeah, I'd recommend a talk by Christoph Parziak. So he spoke yesterday on USB, it was a great talk sort of going through the details of USB on Linux. This is just sort of listing what I used or what I needed to configure for this project. So USB can be used as a SCSI transport, which is how I'm using it in this case. So the two options there, bulk only transport and USB attached SCSI. USB attached SCSI is then a more recent addition to USB and that allows for things like command tag queuing and I think also out of order completion on USB3. The USB gadget stack in Linux includes two modules for handling this. So we have the mass storage KO, which is what I ended up using, but there's also FTCM, which sort of plums into the Linux kernel SCSI target. So that's basically a SCSI transport for USB in Linux. So the gateway itself, as you saw, all of the features are already there in Linux. I mean, we have the RudderSpot device kernel client. We have the USB gadget stack, which supports USB mass storage. There's really not that much to do for encryption, there's a DM crypt. In the end, the project itself was or is mostly just about configuring these different components and making it relatively easy to do that. So here's sort of a look at how the board's configured or how I've set up this gateway. So basically what we have is a board that then boots Linux once it's connected and initially we boot into or we expose a configuration file system via USB. So this is just a RAM disk where you can provide your CIF credentials to access the CIF cluster. You can specify which image you want exposed via the board, yeah, Luxkey for DM crypt. And once you're done with that, you can eject. So this is exposed as an ejectable logical unit. So once that is then ejected, we intercept that eject or we detect it and then basically process the user configuration. So basically at that stage we can remount that image and take the CIF credentials and look at what needs to be exposed and go ahead and do that and expose it via USB. So now on to the demo. So as I already said, I have my CIF cluster running on my laptop. So it's, yeah, just a very simple VSTAR cluster. Just bring up console. Is that okay? Yeah, not really. So there you can see I have my CIF cluster running with three OSDs, one monitor. These are all just local processes on the laptop. And now what I'll do is go ahead and connect to my gateway. So let's plug that in. So one thing I should say I haven't really worked on optimizing is the boot time of the gateway itself. So normally you would expect that once you connect to your USB key, you can immediately access the storage there. So at this stage it's still sort of in the tens of seconds until it boots and then eventually shows the configuration file system. So just wait a little there. It's just coming up now. There we go. So I have my device notification saying that there's a new USB device. So you can see this config file system here. So what this then has, this is exposed as I said earlier as a rant or expect by a round disk. We have our RBD USB conf. So in here basically I'm saying, okay, I want to expose the USB image on my RBD pool within the Ceph cluster. I have my Ceph user that I want to use there, close that. In this case I have a run conf flag. So what this basically says is that when the gateway boots, it should first expose the configuration files system, which is what we're seeing here. So if I want it to boot immediately and expose Ceph, then what I can do is go ahead and delete that file. DM crypt. So that's where we can figure our DM crypt lux keys. And within Ceph, I just have the standard Ceph conf and the key ring there. So one thing I want to do is just copy the Ceph conf and key ring that I have for my Ceph cluster. Just do that from the command line. Good, now back up. So now the other thing I mentioned was that once we eject this configuration file system, that configuration is processed or committed. And we should then see, the Rados lock device in Egypt here. And there it is as Ceph down there. So basically the gateway has committed the configuration, connects to the Ceph cluster on my laptop and then maps it and exposes it via USB. So here we can see, so I have a one gigabyte image there, which is now exposed and connected via USB. Yes? Yes, I did. So in this case, I had a FAT file system on the device. And I'll use, so the reason why I have it as FAT rather than XT or XFAS or but RFS is because I now wanted to demonstrate a stupid device accessing the Ceph storage. So what I have here is just a stock Android mobile phone, which obviously has no knowledge of Ceph. But what I can also do is connect the USB gateway now to the Android phone. So in this case, Android supports FAT32 as USB stick. So what I wanna do is then wait for the board to boot. And just so we've got some data to write to the Ceph cluster, click photo. Okay, and should come up anytime now, hopefully. Yes, there it is. So I'll just copy over that photo to the gateway. So I basically see that as I would any other USB and a storage device and I'll copy that. And now I can go ahead and eject it. Done, so that's ejected. I can power down the gateway now. But what I'll do is use now just the Ruttus Bok device client on my laptop, just to map that image just so we can confirm that what we wrote. So this photo is actually there on the Ceph cluster. So I'm mapping, I've mapped the USB image there. Just need to mount it as well. There I can see that I've, so this is just, as I said, using the Ruttus Bok device client on my laptop, the gateway isn't involved. But there we can see the photo that I, it's on this. So there's a very blurry photo. Well, that's the demo, need to talk. So any questions about the demonstration at this stage or should I go on? Yes, please. So what I have is basically once it can't access, so if the gateway boots and can't access the Ceph cluster, so if you boot it without network access, then it basically exposes the config because it knows something has to be done. So next off, I also wanted to use the same thing for public cloud storage. So in this case, if I didn't have a Ceph cluster at home, yeah, I could then use say a public cloud storage provider like Azure or Amazon S3. I'm gonna use exactly the same technology or components to, yeah, have a gateway for that. With Microsoft Azure, there's basically a protocol or a RESTful protocol for accessing the virtual machine images in the Azure cloud. So I wrote, this was in a prior hack week, I wrote a client which speaks that protocol. So the idea was then to, yeah, basically map between the SCSI requests coming in from the USB host into this Azure RESTful protocol for their blob IO. So with this, I use the LIO, so in kernel SCSI target. With LIO, there's a user space backend, which is a TCMU runner. I think that was added by Andy Grover from Red Hat worked on that. So basically with this backend, we have, yeah, something running in user space which then receives the SCSI request from the USB host. With that backend, I just map that to then a public cloud request. So in Azure, block, block, request, that goes into the cloud. So this uses the Elasto Cloud client which is the project I worked on for that. This is basically just a POSIX-like library for performing RESTful cloud IO. As I said earlier, I have then the LIO backend which handles mapping of the SCSI request to the Elasto library calls. So it looks something like this for the public cloud case. Here we have the USB host on the left connected to the USB gateway. That goes for the SCSI request to then process by the Linux SCSI target, or sorry, SCSI target. There's TCMU above or below that, basically as the user space backend. And finally there's Elasto which handles speaking with the Microsoft Azure cloud. Okay. Next up, testing. Yeah, so with this project, I didn't really want to or at least initially I was testing everything on the hardware itself which is a little fiddly when you're plugging in pulling out cables all day. So I found later on there was this dummy HCD module within the Linux kernel. This is great for testing exactly something like this where basically you can use that as a loopback in a single system. So in my case, I just had a VM with everything on there and used the USB loopback functionality to test it. So now on to future challenges. So in the hack week that I had or the one week I had, I think I proved that it is perfectly possible to have something like this. I think it is a reasonable device or something which is usable potentially to others. But I think there are still plenty of things which could be done or it could be improved in many different ways. Yeah, the first on the list I have is concurrent image access. So imagine where you have say, you know, if they cost under $10 then you could imagine maybe buying 10 of these and sort of exposing maybe the same image via these gateways. In that case, you would need a way to manage exposing a consistent image to all of those hosts or connected hosts. In that case, I think using something, so yeah, avoiding a cluster file system and using something like just snapshots where the first gateway to map sees or has right access to the parent and then any subsequent connects just get a snapshot which was taken as the first host connected. I think that may be something which would, yeah, make having concurrent access usable anyway. Some other challenges. So power is definitely a challenge. I have had problems powering the board from portable devices like mobile phones. It's really a little unsure whether the board will come up or not. So one option would be to add a battery to the board so you're not so reliant on power from the USB host. FTCM, so this is the LIO USB transport that plums straight into the Linux kernel SCSI targets. I didn't actually get that working on the board. So I wanted to sort of have another play around with that and see whether it was a hardware issue or whether I just didn't set it upright. Caching, so many of these embedded boards have, some onboard storage or in this case an SD card. And when the root file system is under one gigabytes, you could use the rest of that SD card or onboard storage for caching as a read cache or a write back cache. That would certainly be something to look at. There's caching again, but performance wise. So I think at this stage, most important would be the boot time. So you saw it takes tens of seconds to come up and expose the storage. One thing I did have, or at least with the QV truck, I did try running everything from the in-at-RAM FS which actually worked quite well. So in that case, I had a booting in, I think it was three or four seconds. It was much quicker, but it's a little ugly implementation wise. Also in terms of raw storage performance, I think having USB 3 and, yeah, Gigabit Ethernet or fast Wi-Fi would be, yeah, great. The only problem is that these boards are quite expensive and they're also not as portable as, or at least the ones I've seen aren't as portable as the Nanopitaneo. In conclusion, yeah, Cep is fantastic. It solves all of your storage problems. I would just say recommend using it. Anyone that hasn't tried it out. In terms of utilizing Cep from many devices or opening up Cep to many more devices, I think something like a USB gateway is useful for that. I think it's a viable option for something like that. I think in particular using encryption on the board itself is particularly beneficial for a public cloud or where it's backed by a public cloud. Yeah, cheap hardware, really under $10 that makes something like this or a project like this quite possible, quite viable. And having open source or Linux kernel mainline kernel support for something like this is hugely beneficial. But otherwise, looks like I'm a little early, but yeah, I guess I'd like to thank beforehand the upstream SunSea community for kernel hacking and getting these boards working. They do really an amazing job. And also, sorry, Andreas Verber, Dirk Muller and Alex Graf from SUSE, because they've also worked on the open SUSE Tumbleweed port for these boards. Otherwise, any questions? Yes. What bandwidth were you able to achieve on USB 2.0? So the question was, what sort of bandwidth or performance could I get with USB 2.0? So I mostly did benchmarking on the QB truck board, which has a Gigabit Ethernet port. In that case, it was bottlenecked by the USB 2.0 port and I think it was, yeah, it was sort of 10 to 15 megabytes per second throughput by USB to the CEP cluster. In this case, my CEP cluster is memory-backed anyway, so yeah, the bandwidth, or sorry, the bottleneck was very much the USB 2.0 port on the board. Other questions? Yeah. I haven't done benchmarks with encryption yet. So the all-winner chips include basically offload support for certain encryption types. It's not fully implemented on the H3 chip. So the A20, I think, was done. The H3 is still work in progress from the upstream Sunsea guys. But yeah, I think once that's done, it shouldn't be too much of a performance penalty. Where your code of time is significant when you go from a Gigabyte root system to a few megabytes of byte-reads that you actually have to load? So the question was, have I considered using a much smaller Linux distribution? Yeah, I work for SUSE, so obviously, playing with Open SUSE is something I like doing. But that said, I could also, so I mentioned I had it running from Inet RAM FS on Open SUSE, so I could also just build this Inet RAM FS and do everything from that. In that case, it was also like 15 megabytes with everything on there to run this project. So it certainly is possible to use your minimal setup and run everything very quickly. Yes? Yeah, sure. Are you sure consistency of the birthplace would be good? Because if you eject the device from USB fast, will you not have time to see the... Yeah, so this is a problem with any USB storage, right? Normally, I think most hosts are configured to do synchronous I.O. to the USB storage. You should always eject the device before you unplug. Same goes for any other USB keys, so I don't think it differs too much in that regard from a regular USB key. I mean... Yes? A network outage is... Yeah. But this is completely synchronous on the gateway side, so... Yeah, it's not buffering anything, so if you basically... If you lose network access, then you won't get successful I.Os on the USB side. So any USB request won't be acknowledged as successful until it's basically reached the Ceph storage, and then Ceph obviously does its magic for replication at the back end. Once the gateway has a completion, then it will finally acknowledge to the USB host that the I.O. is successfully complete. So... Yeah, it's really not doing much in that regard. There's no buffering on the gateway at this stage. One more question? Are you interested in exploring the block? Yes. William? Yeah. Ceph access is set to Western. Can you access that later on again or once you've configured and believe that it's accessible? So this... So the question was, how do you manage accessing the configuration again? This is something Lance asked earlier. So basically if the board comes up or the gateway comes up and can't access the Ceph cluster using the configuration that was provided, then it returns to exposing a configuration file system. And actually it also copies the log onto the config file system, which is, for debugging, quite cool. So you can see, okay, it processed the config, couldn't connect to the Ceph cluster or the image isn't there. This is the error I get. And then, yeah, you can reconfigure your board at that stage. Yeah, exactly. Yes. Exactly. You can boot it with that network and then you'll see your config again. Yeah, someone? Is it possible to... So the question was, whether more than one image could be exposed? Yes. At this stage, you saw the config file. It's not really set up to handle apps, but, yeah, it would be easily doable. So you could expose multiple images. The question is then whether the host supports multiple LUNs or whether, say, your stereo or television, whether you can then access, say, LUN 1 or something aside from the default or the first. Any other questions? Otherwise... Oh, yes, one last. Can we get to the question? I'm guessing it's mostly Q-code and pretty bad. Yeah, it's actually a pretty ugly shell script. Yeah, the reason why I wrote it in shell was because I was initially playing around with doing it from in-it-ID, in which case there's no Python, there's no Perl, there's no Go. I think at this stage, I prefer to rewrite it in something a little nicer. That's considering maybe a Rust project just so I can play with Rust. Yeah. So the link for the code itself is just that top link for Elasto, so this Cloud Storage client. It's also on GitHub. TCMU Runner, so this was Andy Grover's project. There's the open SUSE link for Tumbleweed on a lot of these ARM boards and, of course, the Sunsea community. Okay, well, thanks for coming. That's it, yeah. Thank you very much for that. Live cyber. Hi. Feel free to... Yes? You could have... Yeah. You could have anything behind it. Yeah, absolutely. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Hi. Thank you very much. First of all, although I have a mic, I don't think it's really amplified well. I have a very weak voice. If I need to speak up, just wave to me already. Okay. Boy, that'll be a challenge. Can you hear me? So there's no amplification through a mic. I hope at least the video stream can hear me. Yeah. Hi there. I'd like to give a small talk about a small project I'm involved in since July 2015 called OpenAddict. Let me dive right in. I have probably way too much slides for the time I have. Let's see how far we can come. So basically, what does OpenAddict do? What was the vision behind it? It started about six years ago by now. It started as so many other open source projects with somebody had to scratch his own itch because there was a problem that they needed a solution for. So they thought, well, we can do this by ourselves. They're going with it. The situation was that the company, IT Novel, where OpenAddict evolved from, they were a spin-off of another company. We're doing data center operations for them, and they needed storage. So as you probably are aware nowadays, storage exceeds the boundaries of hardware much faster than people can shove hard disks in. Data growth everywhere. So they needed to replace a number of proprietary storage systems. They were quite surprised by the price tax that they received. So they thought, why can't we do this differently? And if you look at it, a Linux distribution nowadays gives you everything you need to set up a fully fledged storage system. You just buy cheap commodity hardware shop in lots of hard disks and you have a server that fulfills most of the common needs. So the idea was, Linux by itself is good. It has everything, but you need something on top that's a bit more approachable, easier to manage and unified because in many cases you have administrators that might be familiar with using a UI, but they are not that familiar on the command line. So OpenAddict's vision really was to give a more friendly user interface and a unified experience to managing all kinds of storage. Storage here meaning both what is usually called NAS storage, file-based like Samba or NFS, but also block-based storage protocols, particularly iSCSI would be an example here. And later on during the lifecycle of OpenAddict, they also realized that single-server instances or even multi-node configurations can't keep up with the storage requirements. And the developers looked around and figured out that SEF might be quite a nice alternative here, which is a distributed storage system in which you just have a single server where you add more disk, but you simply throw in more servers or even complete racks if you need more storage. And SEF pretty much organizes itself to make use of the storage to ensure the redundancy and make sure it scales along with the hardware that you give it to them. So it started as an in-house project and later became an open-source product, I would call it. The idea behind it was that there was an enterprise version and a community version, and that the company would then sell licenses with added support and other value on top to monetize on the software. Interestingly, that didn't really work out. So when I joined the company in July 2015, we made a number of drastic changes to how OpenAddict was governed and managed and run as a project. Before that, basically the developers all worked in-house at the company, and the development took place like with many proprietary products very internally faced, and every once in a while they released their community version, but there wasn't really a community around it, so there wasn't an infrastructure that was inviting for users to come and work with the project. So that's something that we've changed drastically. Also we got rid of the dual licensing that was in place. Back then the enterprise edition had a few additional bits on top that you would have to pay for. All of this was folded into a single code we released under the GPL and we now had, since then there's no distinction between enterprise and communities, just OpenAddict going forward. We also got rid of the requirement for contributors signing a contributor license agreement, similar to CEP, basically if you contribute to OpenAddict all we require is that you add a sign offline to your commit message similar to how the Linux kernel and many other open source projects do it nowadays. So the bar for contributing code is much lower nowadays and that was really noticeable by just the amount and the growth of the community that we've seen since then. We also opened up a lot of other things that used to be internal, most popular of course, the bug tracker we are based on at Leshen Jira and we now have a publicly hosted Jira instant that's fully open so you can really see all the issues, all roadmap planning, everything is transmitted open. You can leave comments, you can vote on it, you can submit bug reports like you would expect from any other open source project. We also changed the way of how we work on the code. We now make much more use of different code branches. We have established a process for performing pull requests and doing commenting on them. These were all things that were quite new to the OpenAddict developer so we learned as we go along and it's a process that now basically, there's no difference if you are paid for working in OpenAddict or if you are a community contributor it's all going through the same procedures, same requirements and same expectations. We also switched the release model nowadays as we try to come up with a new OpenAddict release at least once per month, roughly every four to five weeks. And we have nightly builds if you're curious so if you are looking for testing a new feature that has just been committed and you don't want to wait for the next release, just take a nightly build. With regards to feature developments we have kind of like a train model so basically people work in parallel on features and once they are ready and they've passed the review and have passed all the tests then they will emerge into the development branch that will eventually become the next release. But if a developer doesn't make it in time since we are on a monthly cycle there's not just a really long period before he has another opportunity to get his stuff merged in so that really helped accelerating the whole development process and making changes to the project. Also in the beginning many different components were managed in separate code repositories so like the documentation was in one repo tests were in another one and integrating them and getting them aligned was always a bit of a challenge so we simply lumped all of these repos together into one single repo which now means that you could basically now can work on a feature, write the documentation create the tests and have them all in a single branch and commit and merge them at the same time so it's much more easy to keep track and keep the stuff synchronized. A few key aspects of OpenATIC we are well aware that we are not alone especially when it comes to storage management there are quite a number of projects out there that do similar things that we do so we try to come up with a few cornerstones of what we would like to focus on primarily the goal is storage management and storage management only you see many projects that start also doing things like managing containers or plug-ins so they are sometimes more aimed at home users that want to have an appliance somewhere in the corner that isn't just a file server but maybe also an own cloud instance or provides a bit torrent server or what have not this is currently fully out of scope so we really focus just on managing your storage and exposing it through various protocols and that's it. Self support is something that we've added recently that's quite noticeable of course we have fully GPL v2 no arbitrary functional restrictions so there are a lot of free storage management systems that you can download and use but they apply some form of limitation on you for example for the amount of data that you can store on it or the amount of concurrent users or what have not and if you reach that limit here all of a sudden you need to buy a license or pay for getting over that barrier that's not the case with OpenATIC you are free to do with it whatever you want in what sizes you want to use it we're based on standard Linux tools as I said most distributions provide all the frameworks and tools that you need to set up such a system by default it's just a matter of orchestrating them and making them more accessible to the user and that's the part that we're taking on we try hard to support multiple Linux distributions originally OpenATIC came from the Debian corner so we started with Debian, added Ubuntu later on since about two years ago we started adding RPMs for CentOS and Enterprise Linux we added SUSE as well and this gives us an opportunity compared to some other storage management systems sometimes are based on non-Linux operating systems one key concern that sometimes comes up here is hardware support that most vendors have pretty solid support when it comes to providing Linux drivers in the service space but if you're getting into non-Linux but unixy operating systems the driver situation can sometimes be a bit more challenging so that usually helps us to get adoption we don't enforce a choice of Linux distribution on you you can basically use what you feel familiar with as the base platform and can put OpenATIC on top what can we do so far? what's the functionality of OpenATIC like? so basically the technology consists of two separate components the most noticeable one is the web UI that is what you see with OpenATIC version 2.0 that started about two and a half years ago we switched from an XJS based to an AngularJS based from web front end so we use JavaScript libraries to make the UI visually appealing and easy to use the back end is the other component which has a RESTful API that's also a new addition in version 2 that we're working on the former version 1.x was using XMLRPC so the RESTful API makes it a bit more easier to talk to with the back end and the web end front end only uses the REST API so everything that you can accomplish by the web interface can also be accomplished by calling REST API calls and with regards to storage we provide the usual suspects in its simplest form and where OpenATIC comes from that you have to group these hard disks with the logical volume manager LVM into a storage pool and we also support the ZFS file system or the ButtFFS file system if you prefer so you have a basic storage unit which is the storage pool and OpenATIC can then be used to carve out volumes out of that pool based on your requirements we support a number of file systems as I said ZFS is one of the file systems and we support ButtFFS for other use cases so you can really choose how to configure storage for the workload at hand that you want to serve we are in the process of adding support for DRVD the distributed replicated block device so in a multi-node setup where you have let's say two OpenATIC instances you can configure that volume that you create on the one node will be replicated synchronously to the second node for redundancy purposes the backend support has been in place for quite a while already and we are now in the final stretch of finishing the UI part of that as well so that support request is really getting close to review now we also do storage monitoring in the backend so one of the things, of course as I said you can just use Linux and set up a share and create a small file server by yourself but something that usually gets forgotten during that process is making sure that the storage is properly monitored and then your users become your monitoring system because they will scream once their disk runs full OpenATIC basically automates this process so each time you create a new volume we also reconfigure the monitoring framework in the background to make sure that it's being tracked and you see the utilization and then as I said local storage is where OpenATIC comes from with the addition of SEF we are now starting to make we want to add functionality that makes it easy to manage the SEF cluster to create new storage objects like block devices or new SEF pools also start doing monitoring so you get an insight of how your SEF cluster is doing this is the functionality that we are now most actively working on at the moment and this combined with the recent changes that I've just talked about with opening the project was something that Sousa got curious and we had a development partnership with Sousa for the entire last year basically so we worked together with Sousa developers on advancing the SEF functionality and in November Sousa agreed on acquiring the team and the project from IT November we are now part of Sousa since then but this doesn't really mean that we will now ditch support for the other distributions there are no intentions to change how the project is being run and covered so components as I said we have on the one hand back end as you can see we are using pretty boring technology here, bread and butter stuff this is by intention because since we need to support multiple distributions we need to figure out what's the common toolset that we can use if you start making choices that are not available on any of the distributions it will be difficult to support it over there so the open attic back end is written in jungle it's a python application usually jungle is used as an application server for let's say web shops or something like that but it turns out that the whole way how jungle organizes data and how it's structured with jungle models makes it a very suitable framework for something like a storage management system as well and basically jungle is the abstraction layer and underneath we are calling the regular Linux tools that an administrator would also use so for example if you create a new volume we are calling VG create or LV create MKFS all the steps that you as an administrator would have to perform step by step to come to the same goal and are automated by open attic for the monitoring we currently are based on Nagis and using PNP for Nagis for the graphs which we are storing in RID I have a picture about that when it comes to CEP the current functionality is using libradar so basically the common API that is used to talk with the CEP cluster to obtain information or to issue administrative commands and we know in the process of doing more than just talking to an existing CEP cluster we would like to be able to also set up and configure and manage a cluster and this is where Solve comes into place Solve is a deployment and automation framework and SUSE is also working on CEP specific management functionality based on Solve that is a project called Deepsea and there is a talk by Jan later on in this room at 3 p.m. if you want to learn more about it the web front as I said AngularJS bootstrap also pretty well in web development terms pretty boring stuff by now but it gets the job done and we are working on improving the functionality and adding more every day basically we also put a strong emphasis on testing so each commit or each new functionality is supposed to be accompanied by a number of tests we test on three different layers basically we have python unit tests where we use the jungle unit test framework the entire application is tested through a test suit that is named Gatling that we developed ourselves in which it calls the REST API directly and automates the testing on that level and we also have automated tests for the full web UI based on Protractor Jasmine where you basically are remote controlling a web browser to simulate clicks on the UI and to check if the web UI gives you the expected results that's the architecture from an individual point of view so you have the jungle application in the middle some data is persisted in a Postgres database if you want to set up a multi-node open attic system the only thing that needs to be shared is the Postgres database so if you have a second node you connect them both to the same Postgres database and then you can use one web UI of open attic to manage your two nodes together since the jungle application doesn't have root privileges we have a separate process which is called the open attic system D which should not be confused with Leonard Petring system D that's a coincidence but this is a root process that communicates with the jungle application through D-Bus and performs the actual shell commands that will get you to the required results like creating a volume, creating a file system, setting up a share and you can basically take a look at the command log of system D to check all the commands that we're issuing to get the job done with regards to communicating with a Cef cluster as I said currently this is mostly based on Librar or LibRBD this is a quick overview of how monitoring takes place again, the system D doesn't only configure the storage itself but it also uses Ginger and creates Nagi's configuration files based on templates and then restarts Nagios to make sure that the new storage objects are being properly monitored PMP for Nagi stores this information in round drawing databases and then we use the backend to take out that information to visualize it right now this is used with RID tool which creates PNG graphs and for Cef we are also using RID tool to export JSON data and the rendering takes place on the web UI instead of just displaying static PNGs this is how it looks like for Cef it's a bit more complicated here since we are using the jungle application to talk to the Cef cluster we have a Nagios plugin that sends its check queries through the jungle application but then again it writes the data to RID we use the JSON export for the visualization so what are we working on at the moment what's going, what's cooking particularly as I said the DRBD stuff needs to get finished this is something that we've been working on for quite a while and one of the things we're currently missing is that we depend on the storage pools that we managed to be existing before so if you want to use ZFS you have to manually create the Z pool on the command line first before we can make use of it similar for LVM once you have that storage pool configured you can tell Open Attic to register it and then creating the actual volumes on top of it can be done through the UI but that's something of course that we would like to change so that's work in progress iSCSI fiber channel functionality needs to be expanded there's quite a lot of things that we haven't looked at yet we track all the things that are still open in the JIRA so we're not just tracking bugs there but all the ideas that we have and we try to group them into bigger stories to have useful chunks of work that somebody can take a look at when it comes to ZEF we've just defined a few goals beforehand we want to be able to both manage and monitor a ZEF cluster through the UI and give a tool that you as a ZEF administrator actually want to use right now there are a few tools out there that give you sometimes a little bit of monitoring sometimes a bit of management but we try to come up with a solution to a more rounded experience here especially considering that a ZEF cluster can become quite large with lots of objects that we make it or that we visualize it in a way that you're not getting overwhelmed and you only see the information that's really relevant for you at this point in time because ideally ZEF is supposed to be kind of managing itself and healing itself but you still maybe want to know about what's going on and very importantly you should still be able to use the command line tools to make changes to your cluster without OpenATIC getting confused by it that's one of the big challenges that we had to face for the local storage systems that we manage we basically assume that OpenATIC is in charge of the configuration and once you have started using OpenATIC for the storage management well you can make changes manually but OpenATIC will simply overwrite them the next time if you haven't made sure that OpenATIC is aware of them and for ZEF we are trying harder to make sure that this is possible so if you're using the ZEF command line tools to create let's say another ZEF pool or an RBD OpenATIC needs to be aware of that and that was a bit of a challenge by the way of how Django works and how it persists data and information I wish I had more time to talk about that but if we have time in the end maybe if you're interested I can share some of the ideas that we have there so what works when it comes to ZEF we have a cluster status dashboard so you basically can see the overall cluster health some of the performance indicators with graphs and everything you can manage ZEF pools you can monitor them including erasure coded profiles for the pools you are able to create rather block devices through the web UI you can delete them again they are monitored we also start looking into the infrastructure so you have the OSD well it's not management yet but you can at least see all the OSDs that are in your cluster in what state they are in when you're using DPCS the backend tool configure a ZEF cluster you also get an inventory list that also knows that your cluster consists of which role they have you can take a look at the ZEF crush map which is basically the algorithm that determines of how data is distributed in your cluster what kind of redundancy you have configured and how the data should be distributed among the various availability levels so to say and we also want to make it possible that you can manage multiple ZEF clusters within one open attic instance let's say you have a production ZEF cluster and a staging or a testing ZEF cluster you can use one tool to manage them both road map well, that's just a small glimpse we have quite a long, long list of stuff that we still want to accomplish the dashboard needs some more love and we would like to make much more information about the ZEF cluster visible from the dashboard we also notice that based on the nature of ZEF that some tasks take some time so you issue a command to trigger an action in the ZEF cluster and it works and it may take some time and you have no way of knowing how much time it takes but as a web application your browser can't just stand still and wait because you would run into a timeout so one of the things that we had to come up with is an MQing mechanism where you can simply MQ these jobs that take longer and then make sure that you get notified once it's finished so the web application doesn't hang or you run into timeouts the whole part about deploying and remote configuration of nodes with salt is something that we are very closely working on with the deep sea developers so as a next step you should not only be able to see all the existing nodes but we would like to make it possible for you to simply boot up a new node that registers with salt and you will see that a new node has joined and used a salt minion basically and you could then use open attic to assign a role to that node let's say this is going to be a new OSD click and then deep sea will do its job to configure the node accordingly more monitoring iscasi target management is also something that we are looking into so basically you define one node in your cluster as an iscasi target host in which RBD images from the self cluster will be made available as iscasi targets open attic already supports that but only on the local node where open attic is running on so usually if you consider the open attic node as a management node it's usually not having the performance parameters that you would need for a full-fledged iscasi target server usually that should be a bit more powerful machine and to avoid having to install open attic on that node as well we are now looking into using deep sea and salt for that instead Rados Gateway is another big construction site the thing is that a self cluster consists of several components and they have their own way of how they are being managed they have their own APIs of how you need to talk to them in the case of Rados Gateway for example there is a Rados Gateway admin ops API which you need to use to talk with the gateway for creating managing the users and the buckets and so on so we need to develop the interface on our end to establish that communication path and the existing functionality like the RBD management or the pool management still needs a lot of more features that we are working on and also monitoring is one of the things that we need to expand right now the expectation is that open attic and the Nagas instance runs on that node in a distributed cluster like self this is not going to scale so we are looking for a more lightweight approach the current plan is that we will be using collectd for that so each self node also runs collectd configured in a way that it just forwards the monitoring data to a central collectd instance so you have a way to consolidate the monitoring data on one node which will make it much easier to monitor and visualize the whole cluster status in its individual nodes alright I didn't dare challenging the demo gods at Foster because network is usually something that you can't rely on you have to live with a few screenshots but we have a live demo that you can toy around with if you like the links will be later in the stage this is our traditional storage management dashboard so to say though so this is what you see when you are using open attic for managing traditional storage like Zambar and FS and so on you can create and define the volumes that are listed over here and for each volume we also create monitoring data and performance data that you can take a look at it's a bit hard to see if you click on the demo you can toy around with this and see it in more details one of the things that is quite interesting and is pretty unique I haven't seen it in the other applications first what we call our API recorder so as I said the web UI uses the rest API exclusively the web UI uses the rest API exclusively to talk with the open attic backend and sometimes you don't want to use the UI but you want to automate a certain task in a script or something through the open attic rest API so instead of having to look up the documentation for the API you basically enable the API recorder in the UI and you click through the task that you want to accomplish once and then you stop the API recorder and it will automatically create a small Python script snippet that basically includes all the rest API calls that you have performed so you can use these as a snippet or a template to embed in your application to get the same or to repeat this particular task this is the Chef cluster dashboard as you can see we're using a different graphing engine here this way we are extracting the data from the run Robin database through JSON and then use JavaScript libraries to visualize it which makes it much easier and much more dynamic to work with the data on the UI the dashboard is fully configurable so you can resize and rearrange those widgets you can have multiple dashboards and they are stored with your user profile so if another administrator logs in he can set up a dashboard by his means and doesn't have to take over what you have configured basically you can also mix UI elements from both the traditional side or the Chef cluster side or if you have multiple Chef clusters you could create one dashboard that shows you the overall view for both clusters in one page so you can really tweak it to your liking Chef pool list, as you see we are always using the same UI elements with the data table on top and then the graphs underneath one thing that I have on my wish list is that I would like to make it possible that these graphs that currently belong to a certain Chef pool could also be taken and pinned onto the front dashboard so if you have a certain pool that you want to monitor more closely it should be possible to drag it on the front dashboard and make it visible there Chef pool creation, some of the features that we support here, boring I'll skip all that, RBD, these are the block device lists now I think the pool request is almost done that you will also see the utilization of the RBDs here it's repeating, as I said, screen shots are not as exciting as a live demo but my past experiences at foster was that the network usually works by the time you're about to head home so that's the crush map editor as I said basically you see visualization of the topology and you are able to drag notes around it you can add new notes, change the properties here and with that I'm already at my link list these are some of the resources that you can take a look on we have a Google group for discussion that serves as our mailing list slash forum if you want to get in touch we are on hash open attic on free note as well so come over there if you have questions and suggestions most of the discussion really happens on Bitbucket in the form of the pool request there's a lot of communication between the developers working on the code and then of course on our bug tracker so yeah, these are the key resources to get in touch with us first and with that I'm a bit ahead of my time, amazing so if you have questions we still have time for that I know it's after lunch okay, here's a question when is software ever ready? when is software ever ready? no, open attic 2.0 is out and based on all the testing that we do we are pretty confident that each release that we publish is safe to use the good thing about open attic, especially if you use it for traditional storage management even though if open attic crashes the actual serving of data is performed by other subsystems of the operating system like the samba server, like kernel nfs we are not really in the path of serving the data to the users even if open attic has a problem and crashes which doesn't really happen that often we are not messing with your data directly unless you really accidentally delete something like that but we are still of course in the process of adding more functionality with each release as I said we have the train model so what we have out right now is ready to use and can be used with confidence but as I said, we still have a lot of gaps to fill and of course we would like you to encourage to give it a try and help us gathering guidance of where we should focus on next so we think that we have now come to a point where we provide a good set of useful functionality we are aware we are not fully there yet compared to other projects but we would like to get your feedback on what your use cases are and what you are looking for, what we should be focusing on basically there was another question here so the question was if we have any plans to support Kaboros for authentication the thing is are you talking about using it for authenticating users to the web front end and the answer is that should work I haven't tested it yet