 Okay. Sure. My name is John Snow. I work for Red Hat. I am a developer on the QEMU project. I've been working on backup solutions for the past two years alongside my other project, which is making sure that ATA drives work well, but you guys are here for the backup part. So before I jump into it, can I ask how many people are familiar with QEMU or even know what it is? Just raise your hand if you... Okay, good. Good. Tons of people. Good. Sometimes I worry that anybody only knows what KVM means, and then I'm like, oh no, there's this other part that does like all the work, but anyway. So before I jump into it, I would like to give my acknowledgments. This is open source, of course, so this is not just my work. This is the work of dozens of individuals. So the feature proposal for incremental backups in QEMU started with Jagan Sundar, who gave a talk at KVM Forum 2011 on wanting to get this rolling and here we are in 2017, so it's been a work in progress for a while. From Zheng, a co-worker from Beijing submitted basically the drafts that became the current version in 2014. Stefan Hoenoetzi and Max Reitz are block layer maintainers, and they've been doing a lot of the review work and they've been very patient with everybody. We've also had a lot of involvement from Virtuoso, which used to be OpenVV, or what was their company name, Parallels, and a few others, and recently they've split off, but Virtuoso has been doing an incredible amount of work for us, so it would be upsetting if I did not give them a shout out here for all of their incredible work. So just as an overview, just so we know where we're going, I want to talk about kind of the problems with QEMU hacking and incremental backup, which is probably obvious to everybody in here. I want to talk about some of the primitives we use to implement incremental backup in QEMU and some of the interfaces and how that works. I'm going to go through some examples showing kind of the life cycle and management of incremental backups. I'll get into some more advanced applications of this, things that we've been working on the last two years, and then to close out I'll talk about some of the features that we're currently working on, some of the features that we're hoping to push upstream soon, some status, and you can grill me with questions if I stammer too much. So to start, the problem, of course, I mean the problem with daily backups, especially if you have dozens and dozens of virtual machines, this is gross. It's a waste of space, it's way too much. The storage efficiency is disgusting, it is clunky, it is slow, it doesn't tie up your network, but it is simple and convenient. If you are doing full backups, you know exactly what you have. It is a full backup, it is stored on another desk, it is easy to restore. It's not too hard to think about. So what we've done, of course, as you might expect, is we've added incremental backups to QEMU. The storage efficiency, of course, is much better. It's only going to copy the modified data. It's a lot fast, but maybe it's more complicated, but I would like to convince you it's worth the extra complication, of course. QEMU added preliminary support for incremental backup in 2.4, so about two years ago now. Good luck with that. Development is ongoing as of 2.8. We're working on 2.9 right now. It's not included as supported in a Red Hat product yet, not that that matters for the open stream version, but if you were to come to the upstream list and ask us, we'd say, well, we're still working on it, we don't recommend this for data center usage yet, but it's mostly for the brave as of yet, but we are nearing feature completion. So the approach to incremental backup in QEMU, originally it was Jagan Sundar's patches, where a whole series of separate command line tools that wasn't baked into the main QEMU binary. It used an entirely new network protocol, so all the existing management protocols we had, this tapped another one on top of it. It ran as an independent thread, so it didn't respect the QEMU state machine at all, or the main loop, it added its own complete separate thing, which made it a little dangerous for making sure that the snapshots were safe, because we had race conditions with this. It utilized temporary snapshots to make sure that the backups were perfectly atomic, and it implemented the tracking with an in-memory dirty bitmap. And it was ultimately not merged due to some of the complexity problems and how it didn't play nicely with QEMU as a whole. So from Zhang took another shot at this in 2014. This was also a dirty sector bitmap-based approach, but it used a lot of our existing primitives that we already had in QEMU, so it was not a separate thread. It behaved in the main loop. It used all of our AIO scheduling. It didn't use any external tooling, no new protocols. It was managed via our existing protocol, which Libvert uses to talk to QEMU and make sure we're behaving. We implemented this simply as just a new backup mode. We have an existing backup feature, and we just added a new argument to it. We can use this with any image format, so QCAL2 is my personal favorite. Everybody else likes raw, that's okay. And we're hoping that because this is going to use all of the existing tools that QEMU has had for a while, that we're going to maximize compatibility with people's existing scripts, and hopefully it won't be too hard for people to add incremental support to their management solutions. So yeah, design goals. Already touching on this a little bit, but we wanted to make sure that we were using as much as possible. QEMU is a frighteningly large project at times, and every time we reimplement the wheel, it grows and grows and grows. So for this task, we wanted to make sure that we were using everything we already had. So we have this typo, good. Glad that made it in. So key structure is the block drive dirty bitmap, and we were already using this to track dirty sectors for our block migration and drive mirroring operations. So when you do a live migration in QEMU and you want to migrate from one physical host to another, QEMU already needs to know which parts of the disk are being dirty so that it can make sure that it migrates those as it pivots to the new host. So we already had something that was kind of doing what we want. It had a configurable granularity to depend on the workload. Some smaller or larger granularities may help the migration pivot sooner. And we could already hook up any number of bitmaps we wanted per drive. So it looks like this was going to be pretty much exactly what we wanted. We wanted to reuse the drive backup interface. I call the QMP protocol well-known. It's well-known to us, hopefully it's well-known to people using QEMU. We already have this command creating full backups for us. It was already capable of point-in-time live backups. We could already export data to arbitrary destinations via the NBD protocol. So, like I said, we just simply added a new argument and, well, a couple of bitmap management commands as well. Another goal for us, coherency, we wanted to make sure that if you were backing up multiple drives, if you have 10, 20 drives attached to your virtual machine, we wanted to make sure that the incremental backup across all 20 was point-in-time coherent across all of them. And we wanted to use an existing QMP transaction feature to accomplish this. We want to make sure that the bitmaps are persistent. So even if the virtual machine is shut down, shuffled around, rebooted what-have-you, we want to make sure that this incremental data, which is kind of precious because it allows us to make very small backups in the future, we want to make sure that stuff survives. We didn't want to depend on the drive data format because everybody has their pet favorites on exactly what kind of data format they're going to be using. And we didn't want to depend on the backup target format. So we wanted to be able to go from whatever format to whatever format, trying to make this as kind of a data storage agnostic solution as possible. Again, we want to make sure that this is migration-safe. And we wanted to make sure that on error, which hopefully it's never going to happen, but of course we know that it always will. So we want to make sure that we have the ability to resume incremental backups without necessarily having to rely on a brand to do a full backup chain if we can help it. And for the persistence, we have to make sure that we're not relying on outdated persistence data, which can lead to lots of backups. So those are the design goals for what we're going for. And for people who are familiar with snapshots, this question comes up a lot, why aren't you just using snapshots for this? Snapshots and backups have kind of different properties. Backups are, you know, we'd like those to be off-site and inert if possible. We would like the leading of backups to be free, for instance. We have snapshot mechanisms. We often use them for format specifics, so we wanted to avoid using that. Some people do use various kinds of snapshots, and then they kind of parse the snapshot data to pull out kind of a point in time backup. But we wanted to skip all that mess and just go straight to an incremental backup. So the incremental backups we have, they're inert, they don't grow, they're not attached to the running system. You can do whatever you want with them, unlike snapshots where you're using a copy-and-write kind of solution, deleting a copy-and-write snapshot may incur an IO penalty, and maybe you don't know how much, depending on how much you're using or using the feature. So this is kind of why we don't use snapshots, but it's going to depend on everybody's use case, but this is another option that we'll offer a better choice of ways to off-site. So building blocks. This is one of the bitmaps and the management list a bit. So this is the existing structure we have to track writes to our drives. Again, I already used the drives mirror and my block migration, and it's implemented using a higher-article bitmap, which is just a multi-level data structure to track writes to sectors. And any number of them can be attached to a drive, which is important because you can have multiple independent backup regimes occurring simultaneously if you wish. So if you want to do both dailies and weekends and monthly, it's just three bitmaps, so you have one to each backup regime. So the higher-article bitmap, I'm sure you can visualize this in your mind, but just because I like making graphs, it's actually a seven-layer data structure, but two is good enough. But this structure allows us to iterate over the data very quickly, especially if it's sparse. We're able to skip large sections of the disk, so incremental backup should proceed quite fast. And as for multiple per bitmap, you can create an arbitrary number. I think we do cap it at some very large number, like 65,000. Some such thing, but you can attach any number of these bitmaps to any of your virtual drives. As you saw in the previous slide, they have names, so the existing usages are anonymous. We call them anonymous bitmaps, but if you are intending to use these for backups, they're names so that you can interface with them from the external API, and the name becomes the unique ID. But each drive can have the same name of a bitmap, so you need the pair of the drive name and the bitmap name to address this backup sequence. So just to illustrate, you can absolutely have something named Bitmap 0 on both drives. I don't necessarily recommend it, it may drive you insane, but you can. And then, of course, we have internally used anonymous bitmaps for mirroring and migration techniques. So I mentioned that we have configurable granularity on this. The smaller the granularity, this is a block-based backup, not files. So we have no idea where files begin and end necessarily. Maybe we could look in and check, but we're not gonna. So we have to go by sector at some point, and so you can configure the granularity. So one byte being dirty may incur you a backup size of 64 kilobytes, but you can configure it larger or smaller depending on your requirements or your expected usages, or you could even, like people never do, try to profile it and then use the one that actually works best for them. But I won't tell you guys what to do. So we default to 64 kilobytes, but it will attempt to match the cluster size of the actual disk you provide. And 64K is the default for QCAP2. Yeah, I was kind of getting ahead of myself here. So we're tracking per sector, but the granularity is configured in bytes because we're trying to get rid of the concept of a sector in a VM. So sometimes we run into some interesting cases like this. So the backup engine itself is going to copy out things per cluster. It's 64K at the moment. It's non-configurable at the second, but it's almost for sure going to be something that you can tweak in the future. So given that QCAP2 defaults to 64K and the backup engine defaults to 64K, 64K is probably the best right now. So the management for these things, everything is done via QMP, which is really good news if you're a computer. Not so great news if you like doing everything yourself by hand. So QMP, for people who aren't maybe super familiar with it, it's a JSON-based protocol. It's like JSON-ish. I think we changed maybe the type of quotes you can use or something like that. It's almost quite nearly JSON, and it is what Libvert and everything else will use to be talking to QV. So it can be a little cumbersome by hand, but we do have little Python tools and things you can use if you want to play with it manually. But there are four commands here. So for the management of dirty bitmaps, there's add, remove, clear, and then there's the more generic query block command, which will show you status in general about all block devices, but it includes some interesting information about the bitmaps attached to each drive. So creation, creation's pretty simple. Bitmaps can be created at any time, on any node, and for any reason, because we're not watching you. Bitmaps begin recording writes immediately as soon as you add them, and the granularity is optional. If you omit it, it's going back to 64K, but here you can see this kind of JSON-ish command, and we execute the block dirty bitmap add command on drive zero. We name it bitmap zero, and we've decided that we want a granularity of 128K this time around. Deletion, similarly simple. The bitmaps can only be deleted when they're not in use, and what counts as in use in this case for us is not actually being used to make a backup when you type it, but otherwise you can delete them whenever you want. Bitmaps are addressed by the node name here, and if you delete the bitmap, it has absolutely no effect on any backups you've already made, it has no effect on any backups you're still hoping to make from other bitmaps. So this is a fairly independent time-on. If you wish to clear a bitmap for convenience instead of deleting the bitmap and then re-adding it, we do have a clear command. Oh, no. I lost my thing. Thank you. As for querying the bitmaps, this is the query block command, which I've truncated a lot of the output here just to show what's relevant to us, but when you query the block, you'll see device drive 0 and it will give you a list of dirty bitmaps. You can see this one is active. The count will show you how many sectors are dirty, so you can get some idea of how big the backup's going to be based on how many bits are dirty in the bitmap. You'll get the name and the granularity, some things as well. The status can also indicate frozen, and that's... Ooh. Sorry about this. The name status indicates that the bitmap is not allowed to be interacted with because it is currently in use for a backup operation. Yes, again, this is the sector count and the granularity, so it can be hard to convert in your head exactly all these different formats between bytes, sectors, clusters, and so on, but the computer will do it for you because the computer is good at math. So the QMP commands are not super useful alone. They're not atomic necessarily. They're only safe when the VM is offline, and there's no cross-drive coherence guarantee, and that's something that I specifically mentioned we didn't want to have happen. So we do have these transaction commands that are going to allow us to create full backups alongside a new bitmap, like immediately, so that every single write is going to be tracked accordingly. We're going to be able to reset bitmaps simultaneously across any number of drives, and we're going to be able to issue any number of backups across as many drives as we have. The problem with the individual commands, for instance, is if you try to reset or add a bitmap and then, no matter how fast you are, and you try to send the other command, oh, by the way, I'd like to start a new backup now. You have no idea. We have gotten a write that happened in the interim, and we like everything to be live and fast and QMU, so pausing the VM to do these operations is not really a great solution for us. So we do have these transaction commands that allow us to batch the commands together to make sure that the operations are data-safe. So the transaction actions that we have to interface with bitmaps, we have the version of dirty bitmap add and dirty bitmap clear. Don't really need a transaction from a move because we're getting rid of it. This works in conjunction with the type drive backup for the transactions, and this can be used for everything I said. The multi-drive coherency, full backups, new incremental chain sync points, it's kind of the major interface you'll be using to do anything with more than one drive, really. So that's a lot of talking. Let's show some pictures. So this is the life cycle for an incremental backup chain in QMU. Hopefully it's exactly as simple as you would hope it to be. You're going to, in general, either create a new backup chain or you're going to take an existing one and synchronize it with a new full backup. And then from then on, you can make as many incremental backups as often as you want according to whatever policy you may have. So for an example, showing the QMP behind this, this is the transaction. It's going to batch the add command and the drive backup command together such that the bitmap will be cleared exactly in time with the drive backup. Here I've also highlighted it doesn't come across so well on the projector, but this is sync. Full is the argument we're giving. We'll start at this point in time and it will clear the bitmap in this point in time, but the backup will still take some time to complete and it will finish later. And this bitmap is going to be kind of in use until that finishes. So to show you what will happen, we have this ID drive zero and we're going to run this transaction and then in one shot, we'll get a full backup and a new bitmap with no dirty sectors in it. So from this point forward, we can use it for incrementals. For a different example, if you didn't want to add a bitmap, if you didn't want to clear it, just kind of like reset, you can use the clear and the drive backup command at the same time together. And that's how we can make it clear. We can add a bitmap to the first backup and similarly, the bitmap's dirty sector count has been reset for us. And then finally, we get to creating an incremental backup. So unlike the other commands with the full backup, we're going to create an image separately here. The image that the backing file is the last full backup we made so that when we create the sparse image of just the sectors that have changed since the last usage, if you were to mount this incremental backup, you would be able to see a full and complete image of the hard drive. It's just each is stored in a separate layer. So we're going to use an external QMI image tool to create an empty QCOW2 for us. We're going to set the backing file the dash B command and the format is QCOW2. And then at any point we wish we can issue this drive backup command and the only difference here you can see it's highlighted is sync is incremental and we name the bitmap and the device and the effect is this. So if we have this many dirty sectors we issue the backup command and now we have a new QCOW2 file using the backup as a backing file. And then the count is reset. And then from here on out as many times as you wish you can keep creating incremental backups just keep creating new new top layers. So the problem here perhaps in the management aspect is of course if you decide to have a backup every single hour or something like this you're quickly going to have a chain of several thousand of these images but hopefully if you've been copying them and asking nicely you can use tools at any time to condense these images so the management layer on the other side of this can take these these long backup chains and periodically condense them or copy them or move them into whatever other dedu situation that you have. So hopefully that is not too complicated for the management side of things. So brief break No, I'm kidding. No, I'm kidding. I'm going to talk about more stuff anyway. So I'd like to talk just a bit about you know, what are jobs so QMP commands are synchronous and the QMP socket is going to block each time you send one so for long running commands we don't really have the ability to block the socket for that entire time so we have this asynchronous task API that allows us to submit a QMP command and get some kind of a long running job that we can manage we can cancel and all sorts of things. For more information on this you can see literally any talk from KVM Form 2016 because three of the block maintainers all gave talks on exactly how block management works and figuratively only three it was Kashyap Chamarthi, Max writes and myself all gave talks on this and there's a link in the end if you want to know how to manage these things so transactions in detail we're going to batch some of the QMP commands each individual item is an action the transaction succeeds only if all of the actions do but the problem is that some actions are going to launch jobs synchronously and some don't and I hope that doesn't cause any problems but of course it did the problem was that if you have a transaction and half of the items are launching these long running jobs and half aren't the concept of a transaction failure became really muddied because normally transactions fail if any component action fails but the problem here is that as long as the job launched we considered that a success so then later on some of the jobs could fail so say if you start a two terabyte backup job because you really hate your network infrastructure and then it failed at 1.8 terabytes or something the transaction still succeeded so the management client now is really confused about what exactly just happened because the transaction oh so what's good and what is correlated at what point of time so this was a bit of a problem for us so yes before the action succeeds if the job is started jobs failing later have no effect on other jobs so some backup succeeded and some would fail and this wasn't quite so great so we added a new parameter to the transaction command as a whole where there was no change from the original behavior where the action succeeds if the job is started but any jobs launched by a transaction now will wait for completion from all other jobs before they finish so for instance if you try to do a backup across a two terabyte drive and like a one gig drive clearly that one gig drive is going to finish way sooner than that two terabyte behemoth but it's going to copy all of its data first but then it's just going to wait in a completion mode waiting for the other job to finish and if the two terabyte job should fail the one gigabyte job knows that it's not safe to clear out the bits in the bitmap and it will roll back to make sure that nothing is lost this isn't necessarily what you want but it's up to the management client some management clients assume a transaction will be fully all or nothing and this will give them that behavior so you can avoid keeping partial completion state and memory or having to worry about that sort of thing but it's up to you if one job fails and another doesn't you don't have to cancel or delete the other backup you could just retry the incremental on both to get a point in time across both but this was a hotly requested feature for people to avoid having to think about partial completion and failure and so on so this is an example of a multi-drive backup doing incremental across two drives there's really no difference other than this thing is all wrapped in the transaction command and here I have actually admitted the group completion mode that I was talking about but it would be an argument below actions so in addition to the actions array there would be like a completion mode group parameter here but this is just to illustrate what happens when we want backup across two drives at the same time so you may have something like this and then upon issuing the transaction point in time we get two backups at exactly the same time yeah just pointing out that those two are absolutely correlated and should be safe to use together so for partial failures again if one failed and the other does not we're going to have to delete one but then the other backup is still good and the way to recover from this would be to add another incremental backup so you'd have two incremental backups for drive zero and just one incremental backup for drive one at the end of that to recover from like a partial failure but if that is untenable for you the other solution is if one fails we're actually going to delete both of them so it is a bit of a waste but it does save you the trouble of having to think about exactly how to manage the recovery situation from there let's see so we have a few more features these are kind of things that were a few more features that we're trying to work on so bitmap migration at the moment the initial version of this did not actually support migrating across live hosts so the initial version of migration for us was to split this data the dirty bitmap into one kilobyte chunks and then serialize it piece by piece and then using the bitmaps themselves we could kind of copy as the drive got dirtied and for things that were small enough we could just kind of skip the live phase and send it wholesale but maybe this doesn't scale so well depending on how many bitmaps or how many drives you have this could really add some time to the migration stream the bitmaps weren't transferred alongside the data so the data would go in its own loop and the bitmaps would go in its own loop and then we had this dirty bitmap kind of solution that was kind of really confusing to name all the variables for but this captures changes during live migration and we could resend the pieces if we needed to and it didn't use a lot of memory but it kind of added some complexity to the migration itself so we have a second approach now using a post copy technique if you're familiar with post copy in general post copy migrations are the concept that you could migrate a VM from one machine to another and then immediately start running the VM on the new machine and then as it needs the memory it would false and fetch the memory from the previous host and the reason this is useful is because it allows you to guarantee when the machine will pivot because if you are not worrying about the rate that the memory is being dirty it's possible that the VM will be so active, so busy, so loaded that it will dirty the memory faster than we can migrate it across the stream and the same kind of problem happens for disk but in post copy you just migrate the VM first with an initial set and then as it needs memory it fetches it from the old host so we figured that we could probably do something like that for bitmaps so we're just going to send the whole bitmap post pivot and then before the data has fully converged we're going to record the new writes on the target with a new bitmap, we're going to disallow backups until all of the bitmaps have come back to us and then we're just going to merge the bitmaps on the target so what this gets us is the ability to guarantee that the bitmaps aren't going to be stalling the migration and then the price we pay for that is you may not be able to issue a new incremental backup for the next, you know, a minute or something after you migrate as everything catches up this does add a new failure point so this is the unfortunate part we do consider this kind of a non-critical loss because you've just lost metadata but the bitmap chains can be restarted using the clear commands and so on but we are looking at ways perhaps to mitigate this kind of a data loss event or I shouldn't say data loss I should say metadata loss there are other options if this really spooks you out you can always use a shared storage migration and flush the bitmaps to the disk migrate and then load the bitmaps from the disk again and that's probably a lot safer so for the persistence like I'm hinting at we were going to do a format agnostic solution but we were going to use QCAP2 as a generic container to store bitmaps for arbitrary other drives but the problem we had with that is that QEMU is not in the business of managing file names or hierarchies for where things are stored we leave that to libvert and other management tools so when we were trying to add a spec that allowed us to use QCAP2 to describe the dirty state of other files and formats it quickly became obvious that we were trying to replicate too much from the layer above us so now we're just targeting the QCAP2 format which some people might be sad to hear but I'll get to that in a bit so for right now it's a QCAP2 format multiple bitmaps can be stored per file the bitmaps have types there's a full spec that was written kind of independent of this feature that allowed us to store these dirty bitmaps in a QCAP2 file so this is extensible and we may use it to restart migrations or things but for now it's incremental backup related the bitmaps can autoload in QEMU like if there's a persistent bitmap associated with a storage file once you give that to QEMU the next time it boots it sees ah yes good and it will start up for you the spec amendment is already merged and there are patches ready on list from Virtuovo and hopefully we're going to get these into 2.9 so for non QCAP2 files there are some options some formats are going to add their own primary support for it I know Virtuoso wants to make sure that they have support in their parallels format but we may extend QCAP2 into a right forwarding like header node if you will where instead of a backing file which is usually read only a great backing file where QCAP2 will store the metadata for other images for you and then that way the managing layers can worry about the backing file relationships and all we need to worry about is that we now have a QCAP2 that doesn't actually take rights it just forwards them to a raw file so we're hoping this is going to let us do persistent bitmap storage for raw disks for people who want that kind of performance would still like incremental backups so I think I heard in the previous talk we were talking about VMware CBT a bit so this was a problem that came up for us because everything I've described so far is what we call a push model where QEMU knows exactly what's dirty and where it's dirty and you ask QEMU to author the backup so this is a largely QEMU driven process which isn't necessarily a problem for us or for people who are already using our models but we do have a lot of requests for making something similar to VMware CBT where you have the ability to query for which blocks are dirty and then manually yourself copy those blocks out so at the moment push model is already in QEMU we are working on the pull model and the way the pull model works is there are a lightweight temporary snapshot so you will connect to QEMU you will request a temporary like a view of the image using the NBD protocol and then through the NBD protocol you're going to have the ability to ask give me a list of the dirty sectors I want to see what's dirty and then the client can at its own pace copy out the sectors it thinks are dirty and perhaps if this third party client knows more than we do about exactly dirty it can make its own decisions about what to back up where why for instance but this is a feature that was requested by Virtuozo and they've been working very hard on this and we are hoping that this is going to add a lot of flexibility to existing clients both for people who use VMware or for people who already use their own scripts for QEMU we're hoping that one of the two push or pull models is going to fit nicely into the workflow so the little bit more about the snapshot it is a point in time snapshot we call it image fleecing you can open up an image and get a still view of the image at that point in time but because of that it's going to require and on disk cache because as the VM continues to run we need somewhere to flush those writes if we run out of memory it has some downsides perhaps compared to the QEMU authored backups but not that different but like I said it's going to offer kind of full control and it's probably the most QEMU agnostic method where you're kind of cutting QEMU out of having to manage these a bit and it is at the moment the only way to query exactly which blocks are dirty so it may be useful for antivirus research or things like that if you want to see what programs are dirty and where and so on so we do have some to-dos we still have to work on the interface for some of like the more advanced features the pull model managing persistence you know if you say I want to delete this out of the drive I want to put a new bitmap in or I don't want this bitmap to be persistent so there's a lot of interface questions that we're still working on we don't have like an integrity checker just yet but we're working on one we would like to add the ability to do offline incremental backups so we would like a reference tool to make sure that if you've shut the VM down you can still parse this dirty bitmap and complete a backup offline if you would like to so and I would also like to give a quick shout out we do have a GSOC a project coming up if anybody was actually interested we want to write a reference implementation just a simple CLI tool probably using Python to take advantage of our existing test infrastructure if it sounds like something that you'd be interested in we're definitely looking for people to help us with this and so just at the end here so the project status the interface is merged the incremental mode is merged transaction support was merged in 2.5 the persistence spec got merged in 2.6 grouped transactions got added just in 2.8 migration support is in review it's on the list right now we're hoping to get it in for 2.9 persistence it's the same story and pull model is still kind of an early design some specs written for NBD to allow us to use NBD for this purpose and we are hoping to get a lot of that in 2.10 and any questions you're not going to like this so that's the management layer's job I don't care so no we tried to keep that complexity out of QME because we didn't want to replicate what Libvert was already doing so Libvert has whatever it has to manage like its own storage pools and things like that so we tried to not care exactly about how many incrementals were already created we say that it's up to the management layer to remember the last incremental it made and to properly hook up the chain so it's a very unix philosophy if you do the wrong thing you will get the wrong thing so in that way we kept our part simple and we just kind of it's a Libvert's problem now so the question was if we were collaborating with Overt or not so downstream at Red Hat yes I've been talking to some project managers to help them explain exactly what we've made and I think this info is getting back to them but I haven't been personally working with someone in Overt but I did write a white paper for it that I distributed to some people and if there are any requests or anything like that if you know somebody directly who would definitely want to talk to me send me an email yes? could you repeat that last part for me? probably more than you would hope for or I should repeat the question which was as storage gets cheaper and deduplication proliferates how much life is left in this kind of a technique more than you would hope because we still use floppy drives for some things so it's always too early to say oh this is dead people do care about this a lot other companies besides Red Hat are pouring a lot of time and effort into making sure that this works so as for how much sense it makes I'm not able to give you an incredibly great answer for that but I can tell you that businesses are extremely concerned with it and are spending a lot of time and money to make sure that it works well so it's probably going to be around for however long you think it should and then another decade past that I guess I'm quite a bit early but oh yeah? maybe I missed understanding that they can limit the bitmaps but can you describe when the number of the bitmaps per drive is more than one? oh sure sure okay I see what you're asking so the question is when would you want more than one bitmap per drive so if you wanted to implement separate backup paradigms at the same time so say you want a daily backup and you could use one bitmap to do every day the second bitmap to do every week you could in theory just do the daily backup and then the back end can manage the different chains for you to give you different views but it was almost already there for us it would have taken more effort to remove that feature so the flexibility is there for us so some further reading if anybody's interested in this stuff the project main page wiki has status and updates for us there's a project page for this in particular the bitmaps documentation for the QMP aspect of managing all the push model backups is in the source tree under the docs folder bitmaps.md for the closest thing we have to a reference implementation right now we have an IO test which is written in python through several backup scenarios like failure on one node failure on two nodes failure on individual mode group mode and so on so that's actually the closest we have to seeing exactly how a program would request these backups the status white paper that I was talking about which is written for the benefit of libvert and other layers above QEMU is currently at this little google link so if that's something that you are interested in reading take a picture or write it down or grab the slides or whatever you will and then for people who are interested in writing tools to interface with QEMU the jobs talk from the last KVM forum is up through the linux foundation if you search KVM forum 2016 find my name and that'll be up there so I think that's actually a little bit early but it'll give you time to scoot out