 Hello everyone. First things first, I'd like to thank you for coming because I know it's late, I know it's the last day essentially of LinuxCon and you still came, so thank you very much for that. My name is Tom. I work at Intel in Gdańsk. We are a team which is devoted to persistent memory programming basically. As you can hear, I'm not really in full health. I did everything I could so that I could stand up here and give you this speech. I've literally taken all the drugs I could take. And by drugs, I mean ibuprofen and something from a sinuses and not crack. That actually was a multi-layer joke. If you're interested in the bottom layer of the joke, come to me later. So the ideal way I see this presentation panning out is that I show you a couple of slides, not too many. I'll give you some specifics. Then you start arguing, talking and then 45 minutes later, without me actually doing anything, we can go grab a beer and then we can talk about persistent memory again. That's probably not going to happen, unfortunately. So how many actually of you have heard of persistent memory programming before LinuxCon? I'm not talking about you guys because you work with me. So three of you, four, okay. Not bad, not bad. So the follow-up question is how many of you have actually tried persistent memory programming? You know, written code that actually works with persistent memory? Yeah, that's still not counting you guys. So persistent memory is hard and I'm going to try and explain some of the things to you as we go. If you have any questions, don't wait until the end of the presentation. Just shout it out. If I don't know the answer, I'll just ignore it. Pretend I didn't hear it. And if I do, I'll try to answer it. So feel free to ask. We're going to start off with a mandatory slide. I'm sorry. I know if you have been to the previous talks by Machio or by Piotr, you've already seen this slide twice, but I have to do this. These are the rules. I have to show it to you. The next slide isn't going to be about C or C++ as the presentation would, the topic would suggest. It's going to be the model devised by SNEA, which is the organization for storage and networking. So they do with standards, basically. If you want to know more about SNEA and what they do, go to their website. Easy to find. So this is the slide. And it presents a couple of persistent memory programming models. We're basically interested only in this one, in the memory type of programming. What this model says is, basically, if you have a persistent memory aware file system on top of a driver, on top of a persistent memory device, and you map a file to your virtual memory inside of your process, you can basically do loads and stores and flashes onto the memory device without really involving the kernel or the file system. Now, that's in theory. I don't know how many of you followed the Linux scholar mailing list. I reckon, all of you. There's been a discussion there that maybe this is not the best way to do this. And my answer to that is, this thing, the whole programming model, the changes that the nonvolatile memory brings, it's supposed to be a revolution, not an evolution. So maybe we need to go the extra mile to make this work. As I mentioned before, I'm part of the NVML team. We have a set of libraries which were initially designed to work on Linux. They tackle the problem of persistent memory programming, which is not the trivial one. And we tend to make it easier for the end user, for the applications to use persistent memory correctly. This is another one of the mandatory slides I have to go through. This is what comprises of the NVML library. We've got Leap PMEM, which is basically the bottom building block. It has all the low-level memory manipulation primitives. These two, Leap VMEM and Leap VMEM malloc, they are designed for volatile use of persistent memory, because obviously that's a thing. Leap PMEM block, it's a very specific usage. It gives you block access to a memory pool. Easy as that. It's atomic with respect to persistence. PMEM log is an append-only file implementation, and this would raise a flag, because it's written append mostly. The mostly thing is because you can rewind the log. But other than that, it's just append-only. And the most interesting library, the most versatile one, is Leap PMEM which is an object store, as my colleague pointed out earlier during one of the presentations. It's not only transactional, because we also have an atomic API, but it's mostly transaction. This is the last slide, without diving into specifics. This is the last general slide, I promise. This is how Leap PMEM looks like. We have a transactional API and an atomic API, and the transactional API is infinitely more versatile than the atomic API. So we're not going to talk about atomic. As I mentioned, Leap PMEM sits on the bottom side of almost all of the libraries. We have internally lists, which we make available for the user if you want to use lists. There's an allocator, which is at the heart of Leap PMEM option, and it's a very interesting topic in itself, and I'm not going to talk about it at all. If you have questions about the persistent memory allocator, because it's a fun piece of code, ask Piotr. Exactly, that's the allocator guy in NVML. If you have questions to the allocator, ask him. And that's basically it. So I'm going to tell you a story, because I think that presentations should be a story. It's late, and I don't want you sleeping here, so I'm going to tell you a story from C to C++, because this is a C++ presentation. But without telling you the pain points of the C implementation, you won't really understand why we bothered with the C++ version. So before we start, there's one thing. Sorry, I'll be drinking a lot. There's one thing I have to mention beforehand. There's this concept in programming that we always take for granted. It's pointers. And as I said, the models says that you have to mmap a file. When you mmap a file, you don't really know which virtual address it will be. You can kind of enforce it with mmapping with a fixed flag. It's not really the most elegant thing to do. And I advise you don't do it at all. So this is a pointer in NVML. This is called a persistent pointer. Actually, it's a persistent memory object identifier, because it's PMMA object. We do it with objects. Basically, this is a pointer. You have a UID of the pool, which is basically a unique identifier of the pool. So it could be a set of files. It doesn't have to be one file. It can be many files. And then you have an offset within that pool. So it's a base pointer. It's an offset. Cool. But you see that there is actually no type information in here, right? There's nothing saying what type of an object this points to. So basically, this is a void pointer. Not very useful, is it? Because you have to cast it to the appropriate type, and it doesn't give you any runtime type safety. So void pointers are cool, because there can be anything who want to be anything. You would like to. But in programming, we would like to have type pointers. And this is one of the challenges we faced during the development of the PMMA object. How do we make a type pointer? And this is what we came up with. It's basically T-O-I-D. And you give it a type. It's a macro. Because it's C. In C, we love macros. So this is a typed object identifier. The question we ask ourselves is if we cannot embed the type within the information stored within the pool, how do we provide type safety? Preferably compile time type safety. So errors. And we did it with a lot of magic, basically. We took some unicorns and rainbows and stuffed it inside the library. Truth be told, it's more of a witchcraft, actually. Because you have to prepare for it. You have to get your chalk and light your candles, draw the circles on the ground. Basically what you do is you name all your types. Not a fun thing, to be honest. Because you have to name all your types. Basically this is a route. I'll talk about that later. Don't worry. You start the layout. You name all your types. Assign it to a layout. Then you can start using typed object identifiers. One thing to note is that while developing NVML, we didn't really want to change the language at all. We didn't want to provide custom compilers and add new pointer types and stuff like that. So we work with what we're given in C. And there's a lot of macros involved. I'm really sorry. We're a C programmer bunch. We love macros. So this is part of the implementation. Some of you might have noticed that we maybe took the same safety numbers a little bit too seriously here because we do store numbers. This here, this is basically a union. So you get the PMO. So this is what's stored on the medium. So a UID and an offset. And we embed the type here. And then we can use the type. We provide you with type safety. All in all, this probably isn't the best looking code I have ever seen in my life. Yeah. Thank you for agreeing with me. Yeah. But it gets the job done. So when you compile and you mess the types, you get a compile time error that you're probably not using the right type for this. And it would be fine if that were all, but it's not, unfortunately. As I mentioned before, underneath, you still have an offset and a UID. You still have a void pointer. So to get the actual type, we had to do something more. Guess what we did? Let's get us what we did. Seriously, I'm asking. I even give you a hint. Macros. We added more macros because we could. Let's see. It's not something unusual to do. This is what actual use of our library looks like. This is an example from our tree. So you declare a type pointer, and then you have these little DRO statements. What they mean is give me a direct, so a volatile type pointer of the correct type from this thing here. And RO means read-only. Kind of, yes. There's also an RW type, micro, which stands for... Everybody's asleep, really? Everyone? Read, write. Basically, this is a cost pointer. Easy. Still, I think that this code is fine. It's readable, with little to no experience in persistent memory programming. You know what's going on, right? You've got checking if the node is null. You have the reference. Everything, it looks normal. Now, the question is, why the macros? Because, obviously, we do have an API behind it, right? This is what the same example looks like using the API. Now, some of you might argue that we could make it this example more compact. And then it wouldn't look that nasty, but it wouldn't be an example, then would it? One thing to notice, I cut it off at 80 characters. I didn't really bother. It goes on and on, probably somewhere there, I don't know, a long way, either way. So it begs the question, is it at all usable? In my opinion, yes. But please do not try to nest too many of those, because if you nest two or three, and you have to reference the inner one, it gets really nasty, really fast. Since this is a C++ presentation, and I haven't mentioned C++ at all yet, let's get on to it. So we have a couple of problems, right? I define a couple of problems. And the question is, can we use C++ to somehow address those issues? And I say we can. And we did. The first one is the explicit type declarations. And frankly, I think that there is some feature in C++, which we could use templates, I think, because they're inherently type defined. So let's use templates. We're still using the C API underneath, so we do have to have a number for the type to provide. But C++ comes to help here as well, right? We have the type ID hash code, which gives us a number, a nickname number for a given type. It's what we wanted, right? C++. Good work. And the other thing is, do we really need macros? Macros is C++, right? We hate macros in C++. We love them in C. In C, they're wonderful. In C++, they're not really... There's one thing in C++ that could help us to get rid of the macros, and it's operators and operator overloading. There's a bunch of operators like member access and stuff like that. Why not use them? And so we did. This is what persistent pointer usage looks like in C++. And I dare say it looks more like programming than in C. We have an allocation function for the persistent pointer, which we also call a persistent pointer, obviously. Since we're using C++ 11 and higher, there's this feature called variadic templates. So what this does is actually call the constructor of foo and passes the arguments because we can do that. Just no real issue here. Why we couldn't? We have member access and function calling on the objects. It works fine. Here you can see that we create a bar object and give it to a foo pointer. So implicit type conversions also work. You can also do an array. Arrays are fine. For some types of arrays, we'd give runtime bounds tracking so we get runtime error if you try to go outside of the bounds of the array. There's also the allocation functions. And this is starting to really look like smart pointer land. And it is because it is a smart pointer. We tried to make it so that C++ programmers are familiar with it. So we tried to mimic, for example, the share pointer. Now, having said that, it's not really a share pointer because share pointers were designed for lifecycle management. And persistent pointer, it cannot destroy an object on scope exit, right? Because it wouldn't be really a persistent object if you destroyed it each and every time it went out of scope. So the question now is that you know all of this, right? I went through this. Is it usable? This is the same example I showed you recently. You should remember it. If you don't, go see a doctor because that may be an issue. And let's see what the same example looks like in C++. Better, right? Let's look at it again. This is normal C++ code. Sure, it has hidden this out. But this is C++ 11. Let's use the features that were given. The only hint that this is, in fact, persistent memory programming is the pool parameter, which tells you that maybe this is sort of persistent memory stuff. But other than that, it's just normal code, right? We had this nasty DOE, this null macro here. And here's just a comparison to a null pointer. You don't really need to reference it. It's just a member access operator. It does everything for you. Basically, it does more. But we'll get on to that once we get two transactions. So, question to me is, to you, sorry, from me. Is this better in C++ than it was in C? At least one of you raised a hand. Thank you. So, kind of agree. I understand the rest of you are C fans. Don't really like C++, fair enough. I say this is a win for C++, but not all that gritters is gold, so there has to be a downside. Sure there is. Basically, the main issue is we don't allow polymorphic types inside the pool. That is because V tables are runtime data. That's one thing. And second of all, there is no well-defined layout of V tables in the standard. It's implementation-specific. It could be anywhere. It could be anything. And since we don't add anything to the language and we support any compiler that does C and C++, we have no way of rebuilding the V tables on each reboot of the pool or the program. So, sorry, no polymorphic type. The thing is, it probably cannot be really because the question was, sorry, I'll repeat it. Can it be used to share the pool between Windows and Linux? And given that the alignment of structures is different in Windows and in Linux, it cannot be. Yeah. So, if you're not sharing with different ABI's, then the second issue that you pointed out at the V table is not specified in the standard, is not an issue because it is specified in the ABI. In the ABI of which compiler? So, the point is, if you have the same ABI that allows you to have same alignments of the structures, that means your V table is also specified in that ABI. I'm not going to argue about that. That might well be true, but I think it's compiler specific. I believe it's compiler specific in the implementation. So, we're talking, this is LinuxCon. All the compilers on Linux use the same ABI for C++. Yeah, right now. It would only be a problem if we stepped out of Linux. So, if you went, for example, for Solaris or... And we're not saying that cannot happen because we are doing a port on Windows. And that's one thing. And since it's not in a standard, you can't really rely on it because that might change. True, but you can't rely. The C standard doesn't specify alignment either. But we do check alignment on... My point is, you're depending on an ABI. I understand your problem and it wouldn't solve the first problem of the address anyway. So... It's far safer to just forbid polymorphic types than... Yeah, you would have to do it anyway. I understand. I'm just criticizing the second point. Okay, sure. Fair enough. There's also another issue, and this is actually taken from the standard. So, I think we're not going to argue on this one. Is that the point is that the hash code is guaranteed to be consistent within one execution of the program. And that really doesn't help us in any way. I know that right now it works this way. So, we're safe, kind of, right now. But if you wanted a permanent solution, there's some fix in the standard saying that some way of generating the hash code should be name and, I don't know, member mangled or something like that. So, there's downsides to all the stuff I just told you. Normal. There's upsides and downsides. The next thing I'm going to go through are transactions in LeapPMM Alpge. I'm going to introduce a couple of concepts and then, as always, go with C++ and see if C++ can do better. So, transactions in LeapPMM Alpge have ACID-like properties. The key point is the like here because they are atomic in respect to persistence. We provide consistency of our metadata. The consistency of your data is your problem if you mess it up. For isolation, we do provide you with primitives and an entry point where you can synchronize your access. And durability, well, this is called persistent memory, right? If we didn't have durability, then this wouldn't really make much sense. So, underneath, transactions, you see exceptions. And by see exceptions, I mean long jumps to handle the error cases. So, this is what it looks like using the API directly. You first set jumps or save your environment. Normally, this is the way you do it. You then handle the error case. You have to end the transaction in case of an error. Kind of counterintuitive. Then you can start the transaction. This is the synchronization entry point I told you about. You can provide locks here which are held throughout the whole transaction. Then you do work. It's just like about 15 lines of code of work you can do here. And then you commit the transaction and you end it. And this looks bad, if I may say so myself. Because you handle the error case, then you begin the transaction, then you end it. Just it isn't the flow of the application, the flow of the logic of the program isn't nice, isn't top to bottom. It's randomish. So, you already know that we have a C implementation. We are C people. So, guess what we did? Macros. Yeah, we did macros. Because we love macros. So, yeah, it's macros. Sure. We hid the whole set job stuff in the txbegin macro. You can still provide the locks. It's fine. You can ask, what about the error cases? This is how we handle error cases. We give you an on-a-board section. And it's optional. You don't really have to have the on-a-board section. There's also an on-commit section. And finally, if you need those. The only... This looks cool. Because you hid everything and it's clean and nice. The only thing that isn't nice is the txend macro. It has to be there because this is basically a glorified switch. That's all it is. So, we have to have it here, unfortunately. We have an ugly-looking API. We fixed it with macros. Everything's fine, right? Wrong. Because even though we fixed it, it's still error-prone. I told you that underneath the transactions you see exceptions, long jumps. And this tiny thing with long jumps is that they mess up your registers in the CPU. It's most unfortunate, but it does. So, let's look at an example. Because everything is best explained. An example. You have a pointer. It's called bad example. If you expect bad example to be well-defined and usable, for example, in the on-a-board section, it's not. Because we're going to long jump. And if I were a compiler, this goes into a register. You're going to use it frequently in the function. I'll put it in a register. And afterwards, you're, like, undefined behavior land when you try to de-reference it. And the bad thing is you don't see it, right? There's no set jump. There's no long jump. It's here. We did you a favor, right? We hid the set jump in the TX begin macro. Good job, right? So, this is actually from our man pages, as an example, from our man pages. So, we have it in our man pages. We have a blog post on it. I reckon this is recorded, right? So, it's probably going to be YouTube. And frankly, I don't think there's more we could have done. I just told you about it, right? Feel warned. And I still think people are going to find this bug. And on-a-board section, this is, like, you know, 1% case where you fail transaction. Good luck debugging that. It's not an obvious one. So, how do transactions work? Oh, I have to speed up. So, how do transactions work? We basically snapshot stuff. But it doesn't happen automatically in C. So, you have to tell us when to snapshot and what to snapshot. And I think that it's probably the biggest pain point in this whole API, micro. Is this TX ad? Because you have to remember. And remembering is it something we do good. Because we're lazy people. This is a digression for me. Laziness isn't a bad thing, not in my book. You know the saying that if you have to do something more than twice, write a script. Because we're lazy people, right? Laziness is what got us here as a society. See this? This is remote control. I could be standing there and pressing space or something. But I don't. It's because I'm lazy. You know what was the prototype of this? Children. True story. You had a TV remote. Before you had a TV remote, you had children. You told your kid, basically, go change the channel. But we were too lazy to do that. Were we? We invited remotes so that we didn't have to talk to our children. Lazy bunch. And it's a good thing. I say it's a good thing. We would be, you know, using sticks and stones to get our food if we weren't lazy. End of the digression. I don't have time for anymore. So basically, if you forget this and the transaction fails, this won't get reverted. You either have the change or don't have the change. It depends on cash pressure. Once again, not a really fun thing to debug. There's going to be a talk later on after this by Krzysztof. We did some work into dynamic instrumentation. So you don't know Valgrind, right? Yeah. So we wrote a tool for Valgrind to detect this stuff to help you debug these kind of errors. I think it's cool. Helps a lot. So I've been pacing a lot, haven't I? Back and forth. That's a habit. I picked up in jail. Kidding mental hospital. Cool. So I showed you the problems with C++. We'll see. Sorry. C++ doesn't have those. It has other problems. Let's see what C++ does to alleviate problems. We have three types of transactions, which are actually two types of transactions. Two of them are scope-based. One of them is function. I like to call them lambda transactions because that's how I prefer to use them with lambdas. This is an example of a manual transaction. It's called manual because you have to do the transaction commit by hand. You might ask why commit and not abort by hand. It's far easier to make the programmer handle the normal case than the exceptional case. Because if you forget this, everything fails, and you notice right off the bat that something's wrong. But in the odd-error case, if you forget to manually abort, then basically a week or two debugging or stuff. This looks nice, right? This starts a transaction. You can put locks here as well if you want. This starts the transaction, and once you're here, don't put anything after the transaction commit because I don't even know what will happen. Once you're out of here, the transaction ends. The main issue is that you don't really know how it ended because we can't throw exceptions from a destructor. That's bad behavior in any book I've read. I haven't written any book yet. We have this static function. It basically tells you whether you had an error in the transaction or not. It's ugly, but I can't do anything about it. Sorry. You can nest transactions as well. This is also available in C. And if you explicitly abort an error transaction, the outer transaction gets aborted as a whole. One of the features. So you still have to check whether it aborted or not. Now, you might ask, why do we have to commit manually? Why can't we just decide when we go out of scope whether we can commit or not automatically? Well, the issue is in the standard, actually. It's well-known. It's that STD uncalled exception. It's badly designed. They fixed it in C++17. They added an S here. It's now STD uncalled exceptions, and it's fixed. You add a letter and everything is fixed. If you want to know more about this, there's a nice talk by Aleksandr Esku. There's also a standard amendment draft by Herb Sutter. Both of them very nice videos and articles, if you want to know more. So, we fixed this in C++17. If you have a C++17 compliant compiler, which has the appropriate uncalled exceptions defined, you don't need to explicitly commit anymore. But we still don't throw an exception. So, the getTXLastError call. Sorry, it's there to stay. Now, on to the Lambda version, because we got like 10 minutes. Have you got any questions because you're a silent audience, really? No? Okay, let's go, because we got 10 minutes. The Lambda version, which I prefer, way nicer. This is how it looks like. I added this line because you can't really allocate outside of the transaction because it's a multi step thing. We have a different API for allocating outside of transactions. This is what it looks like. You start the transaction. This is your function. It can be normal function object. We don't really care much. Then, inside of the transaction, you make changes. Remember what I told you? That we're lazy people and we don't want to remember to TX add everything. It's going to get here done automatically. There's not only the persistent pointer. We have another template class. There's too much to talk about in 50 minutes. Basically, this is going to automatically add to the transaction, so you don't have to worry about it. All deletions can also be rolled back on a transaction abort. Now, the only thing that I hate with the Lambda transactions is this thing. This is a lock. Now, why this has to be done this way is because we're using variadic templates and they have to be the last parameter. So, you start the transaction, you do work, and then you mention the locks. Sorry. I couldn't do better. If you can, give me a... You can issue a pull request. It's on GitHub. If you can fix this, please do. If not, sorry. One more cool feature of the Lambda version is that you never really lose an exception. If in an inner transaction, something happens, this could be a library colliding. Something happens, and you get an arbitrary exception. For example, STDBedAllock. It gets waterfall outside, and you can catch it here. I think it's a cool feature, necessarily one. Nine minutes. So, to summarize, if you really want to have scope transactions, please go with C++17. It's easier. Let's have a server prompt for you. But I say go with the Lambda version. It's nicer. It's self-contained. It looks better. The only problem is with the locks. Now, this is the most interesting part. Sorry. I had to put it in the end. If I put it at the beginning, it wouldn't make much sense. We had this idea of using C++ standard library containers with persistent memory. And to do that, we need to implement the standard allocator concept, which was designed not to do allocations, but to switch pointers. Obviously. It's called the allocator to switch pointers. This needed a little bit more work. Right now, we're targeting C++ and STD C++ implementations of standard library containers. And I think I've got most of the code on our site done. And this is what it looks like to actually use this. So we have an STD vector. We provide it with our custom allocator. And then, basically, it's your regular vector. Right? You have the vector here. It's my vector in the root. This is actually working code. That's why it looks so nasty. You open a pool. You take the root object. And then you can place back, push back. You can even iterate through the whole thing and do stuff with your objects. And it works. And it works because this is a vector. Vectors are simple beasts. It's like in the GCC implementation, in the STD C++, it's like three pointers, I think, inside of the implementation. So this actually is atomic if you use it in a transaction and it survives power failure. As far as I know, this works. For other containers, not so much. Sorry. So for Leap C++, I have basically all of the containers compiling. They're not Power Failsafe, but they do work in the optimistic case. Leap STD C++, however, not so much. I don't know if you can see this. There's a little asterisk here, which means that someone somewhere is expecting a normal pointer. And normal pointers, they don't understand persistence. They don't survive power failures. So Leap STD C++ needs a little bit more work. But that's nothing we can do, right? We send people to the moon and back. It's like 60 years ago, 50. This is software, right? We can do this. I believe in us. Okay, so the main pros for this approach, containers are free. They're there. It would be a shame not to use them. So we decided we use them. It's not really free because as I showed you, it doesn't really compile all the time. And we had to modify the implementation a bit for this to work. But it can. And I think it should. You got a familiar interface. Everybody knows containers for C++. Everybody who's written more than hello world in C++ basically use the container of some sort. The implementation is well-tested, not for persistence. Obviously, we have to add some tests for persistent usage. But other than that, containers have been used widely by almost everyone. And they work. And there are issues. This is unfortunate. I'm going to get more water for this one. So there are containers which have internal data structures, like colors of the red-black tree for the nodes. And those metadata, they don't understand persistence. They don't know that they have to snapshot themselves before they change. So we have to introduce this knowledge of persistence into other containers somehow, preferably in a standard way. By standard, I mean the standard. As I showed you, there's also cases when there's a simple pointer used instead of the allocator trades pointer, which I think needs to be fixed either way, even if this doesn't make any standard or isn't widely adopted by the community. Yes, I'm sorry. I made a test in my tree that used almost all of the containers from C++ 11. The vector works both for C++ and C++. Thank you for noticing. I didn't say that. No, it doesn't. It could do a rebind to a different type. It could use the allocator rebind. That's what lib C++ does. But thank you for noticing. Those are good questions. Thank you very much. The other thing is we need some sort of layout versioning because we're going to keep it in persistent memory. There was no notion of persistent memory. And the standard doesn't say anything about layout versioning. So if you change the internal implementation of one of the containers and you open up the pool, it's not going to work. It's undefined behavior land, as far as I'm concerned. So that has to also be addressed in some standard way that we can check the layout version of the implementation. And lastly, we are currently working only on x8664 because that's our target. You see this? Intel? What can I do, right? But this is open source. If you want this to be ported to a different architecture, please do. You're more than welcome. We will look at the code. We will merge it. If everything's fine, it doesn't break anything. So if you need this on PowerPC, sure. Maybe you do. And I've got like two minutes left. God, thank you. This is the last slide. Thank you very much for listening, for coming this late on the last day of LinuxCon. This is pretty much all I have right now. No, I have this one. These are some links. If you want to play with the implementation, this is my private branch because obviously I cannot upstream this into NVML because it doesn't work yet. This documentation and there are some blog posts on some of the stuff I mentioned. We also have a Google group. If you want to ask a question, participate in any way, please do. So thank you again. One minute on spot on. Thank you.