 Carnegie Mellon vaccination database talks are made possible by OttoTune. Learn how to automatically optimize your MySeq Core and PostgreSQL configurations at ottotune.com. And by the Steven Moy Foundation for Keeping It Real, find out how best to keep it real at stevenmoyfoundation.org. Welcome to another Vaccination Davis seminar talk. Today we're excited to have Andre Goreny. He is the CTO and co-founder of McObject, which is the company that develops StreamDB. So I've known that about StreamDB for a long time because when I was working on it, it was a commercial example of a system. So we're excited for Andre to come talk to us about what they're actually doing because there isn't a lot of papers about this. So Andre is the co-founder and CTO. He also holds an undergrad degree and a master's degree from the University of Moscow. So if you have any questions for as Andre gives the talk, please unmute yourself, say who you are and where you're coming from, and feel free to interrupt at any time. We want this to be a conversation for Andre and not him talking to the empty space that is void. Andre, with that, the floor is yours. Thank you so much for being here. We appreciate it. All right. Thank you, Andy. And I'm glad I can share our new development with all of you. And we'll talk about predictable database management in embedded applications. Before I do that, what I'd like to do, I'd like to set the stage from the industrial standpoint. So I'd like to share our motivation with you. So commercial database management has been used in embedded systems for 20 years now. And these days, developers don't even debate whether to buy or build their own database management for embedded systems. It's those components are almost always bought. However, what surprised me in the last few years, those commercial database management software started appearing in systems that long been off limits for any commercial vendors. Systems like certain type of medical devices, radiation devices, or they call it cyber life, aircraft navigation systems, and avionics, and pilot assistant programs. Recently, most recently, it's autonomous driving that people all heart and body about driver assistance systems of various kinds. So altogether, they can be called mission critical or safety critical applications. So the failure of the system is likely to cause harm. And of course, the real time operating systems are widely used. But again, surprisingly, other middleware components like file systems or network stacks or database management systems are always are in demand. Now, the requirements of those of those applications are frequently defined as real time constraints on system temporal behavior. So we see the need for a commercial database management system that preserves the temporal validity of data and implements predictable execution of critical transactions. So with that, I'd like to introduce our latest design, latest addition to XtremeDB, and that is predictable transaction management. Now, I would like to structure this talk in the following way. First, I'll talk about XtremeDB as a system, what it is and what it is not. Then I will focus on a few internal components of XtremeDB that are relevant to the real time design. Now, after that, I will go into the real time design. And this is not, it hasn't been a product yet. It is what it is. It's a design. And I explain how we implement real time constraints on the database transactions. And then if we're all awake, I'll give you a few examples of the actual implementations for different real time operating systems. So what XtremeDB is not? It's not an enterprise client-server type of database. It's not oracle. It's not MySQL. On the other hand, it's not a simple key value pair, such as a Berkeley DB. It's not entirely no SQL database. It's not a simple cache system. So let's see what it is. First and foremost, it's a professional developer tools to integrate very high-performance databases into various native languages, C and C++, Java, Python, Lua, et cetera, et cetera. So XtremeDB is implemented as a set of libraries that collectively are called database runtime. We provide a SQL-level API. We also provide other modules. And next slide will show you the entire ecosystem of XtremeDB capabilities. But essentially, it's an API for different languages. Your applications can be written in Java, Python, .NET, many, many different languages. So once XtremeDB libraries are linked with your application or used with your application, your application is capable of creating fully transaction-compliant database. And we provide a rich set of different index algorithms. We support all C-language data types and some of the C++ data types like variable length strings, vectors. We also support vertical layout, as well as horizontal layout. And also XtremeDB includes various distributed options. So the database itself can be either useful high availability like master-slave and master-slave setup. It could be used as a cluster where each node is equal. We support sharding. We support different multi-layer communications between different nodes in the architecture. So this is the big picture. The big red circle in the middle, it's an XtremeDB kernel. And as you can see, there's a rich set of indexes. We support multiple data layouts. Dynamic data layout is when your objects, when your records are variable lengths. There's a different data layout that supports small records that are fixed lengths. We support blobs, vertical layouts, we support multiple transaction managers. Transaction managers based on MVCC, MVCC, multi-version in concurrency control and pessimistic transaction managers. XtremeDB supports both transient and persistent storage. We support in the kernel. We also support multiple synchronization, multiple notification events, which is quite important for embedded systems. So that's in the kernel. And outside the kernel, you can see so much different models, so much different capabilities. I talked about, a little bit mentioned, the distributed components, high availability cluster. We also have a full blown SQL engine. We've got various memory compression, various backup capabilities, multi-language support. Anything that you can think of is there. And to that extent, XtremeDB has been around for 20 years. So there's no surprise that we have so many components that are integrated with the product. Now, going back to the API. So XtremeDB provides two types of API. There is an API that we call Static API, which is basically implements database control, transaction control, courses, error handling, type of that nature. Now, there's also a so-called dynamic API. Dynamic API means that you, as a developer, describe your data in a very simple language, our own, as a proprietary language. And then, based on that description, we generate database access API. So this is how it goes. So your data is first described in your, with the data definition language. It is similar in semantics in C to C++, and very easy to use and learn. And then we have a tool that takes the data description and generates essentially C API. So the good thing about it is that you don't have to learn, you, as a developer, don't have to learn any cryptic API, any cryptic functions to access the database. If you know the description of your data, you intuitively know your API as well. So it's easier, it's quite easy to actually use XtremeDB to that extent. Now, this is the sample of the database schema. It's at the top. So we use the terminology class. So class is a table. In this particular sample, there's three different data fields, an integer field, a daytime, another double, another integer field, and there's a bit tree index over two of those fields. And down below is the generated API. You can see that there's a function temperature new, for example, that let you add data to allocate rather a new record. Then you can write and read data out of that record. And since there's an index, there's a search API, the lookup function, that you can use, pass an inner key, and search for data. Now, this is your typical sample application in C. I am oversimplifying it here. But what essentially happens when you program with XtremeDB, you create a database and it's the call to open database at top. And then a task connects to the database through the connect API. Those are part of those control static API. From that point on, you can start using transactions and create, either create new records or read data back. So essentially what happens to you, the application connects to the database and start the transaction. Transaction can be either read-only or read-write transaction. Manipulate with data and then close the transaction. Transactions can be, of course, either committed or rolled back. So in this example, I mean, you'll see that it's pretty trivial. So this record is created. A value is written into the allocated space. And then the next transaction is a read-only transaction that looks up, looks up, creates a dataset, looks up some data and reads it back. So that's essentially the flow, the code flow for application that uses XtremeDB. Now, a few words about physical layout. How we store data. So XtremeDB supports the notion of logical database devices. Logical devices are simply an abstraction of your physical media. So there could be two different media that we support. We support memory devices. Again, it could be conventional or shared memory. And persistent devices. Persistent devices could be files, multi-files, it could be repetitions for that matter. So regardless of the nature of the physical media, all memory devices, all physical devices are logically divided into pages. Pages are simply fixed-size segments. So the top picture illustrates we have two in-memory devices there. And pages are simply free pages, are simply organized in a link list. So it's a very simple page manager. Everything in a database, whether it's an object, a variable length object, variable size object, or a fixed-size record, or an index of any kind, everything is built upon these pages. So the page manager simply takes the page out of the memory device and makes use of it, building either a database record or an index for that record. Now, the persistent, the picture down there, it's illustrated the persistent database. Again, the persistent storage is divided into pages. Pages on the persistent storage are larger than in-memory pages, and that is to minimize IO. So pages are put up in a cache, and cache is again, is a memory device. And then those pages internally are treated as a collection of smaller pages that the database runtime, the database kernel, builds objects and indexes and everything that the runtime works with. So let me say it again. We have a number of page managers. So every object in a database is built upon those pages. Now, transaction managers. Sorry, go back to the previous slide. When you write to raw partitions, so that basically means you have to, like, you're given some device where there doesn't have a file system, and you have to manage the data. You manage the partition yourself. That's correct. It's basic. You have to build your own sort of pseudo file system for the data system. Can you talk about what is the performance benefit or improvement you see over what the custom file system you built for, TVB versus, like, the same hardware just with the device having the file system, the OS providing the file system. The rough conventional wisdom is that, like, you see, like, a 10% to 15% improvement in performance by having the database and manage the files themselves. Can you see that in your deployments? Well, it's not necessarily related to performance. File systems take space. File system takes overhead. So that's one thing. Another thing is that there's some devices that just don't have a file system at all. And when we use a raw partition, we don't actually try to implement our own file system. We don't need all that functionality. All we need is a simple page manager. So we take that physical device or rather a driver that normally implements some sort of a block device interface. And we use that block device to build our page manager. Performance-wise, well, it depends. Modern file systems are very optimized. And quite honestly, when there is a file system, we'd use one. But XtremeDB is often put in the environment where file system is a drag. They're either badly written or they just take space. So I can't really quantify whether it's 15% faster. But it could be. Okay. So out of curiosity, then if you say if it offers you a file system, you'll use it. Are you ever given the option of whether for the particular deployment, like you can use a file system and say, hey, you need a raw blocks or a file system and you guys always choose the file system? Well, people normally, I mean, it depends on the application. When the application runs on Linux or some other non-real-time environment. Yes. Obviously, people use file system. Not just because of the database, but they use file systems for a variety of reasons. Okay. So internally, we have we call it file system wrappers. So we have our own very thin layer that looks like we access the file system as functions like your normal POSIX file system calls. Okay. But the way we implement those, they can be based on a real file system or they can be based on a block device in API. Okay. So that's yes. And there's always a choice. Okay. So let's go back to two different transactions. So we have generally speaking, we have two different ways to implement transactions. And transactions are implemented through models called transaction managers. So one is called Mursif, multiple read single write. That's the abbreviation. And it's essentially implement specific concurrency model. So it uses locks. And it is implemented at a very simple queue. So we allow multiple read transactions at a time. And we allow only single write transaction at a time. So the write transaction blocks everything else. So, and the other option is an optimistic concurrency model, which is based on MVCC. And of course, it allows multiple simultaneous reads and writes. And we support all normal SQL isolation levels, read committed, repeatable read. MVCC is not when you have, so people use MVCC based manager when they have multiple tasks that access the database at the same time. In embedded environment, there's no so many tasks that simultaneously access the database. It is in fact normally just one. And the Mursif transaction manager is very light. It doesn't use many locks. It doesn't use operating system primitives to lock the database. Those are very expensive in performance terms. So I would say that generally speaking, half of our embedded applications use the Mursif-based transaction manager. Now with persistent, the next few slides, I'll talk about rollback, rollback recovery mechanisms. So for persistent databases, there are generally two transaction logging policies. This is normal undo immediate modification transaction policy. And there's also right ahead logging. And most databases that are on the market support both of those transaction policies. I don't think that we need to go further in discussing the persistent rollback recovery simply because our real time design doesn't support persistent databases at the time. But what I'd like to say is that it's there. What you expect from the normal full blown SQL database, we do support various transaction logging policies on persistent databases. Now let's talk about transient in-memory databases and their recovery mechanisms. So essentially we have two different ways to rollback transient transactions on transient databases. So one is quote, unquote, logical recovery mechanism. So I'll explain what's on the picture on the right just in a second, but I'd like to say this first. Essentially, this recovery mechanism algorithm is aware of the logical structure of the objects that are being rolled back. That has its benefits. Sometimes that has performance benefits. And sometimes it's beneficial from different points to know what is it that you modify to then recover it back into its original state. So this logical recovery mechanism, it has been there forever. So that was our original recovery algorithm. And it's generally oriented towards commits. So commit with our logical recovery mechanism is faster, is better. So the other recovery algorithm is so-called page-based as physical recovery mechanism. This algorithm is not aware of any logical structure that is being recovered. So we simply take the pages that are modified in one way or another, put them away. We call that in a rollback buffer. And when the recovery is necessary, those pages are simply copied back. So this recovery mechanism is rollback oriented in a way that when you have more rollbacks than commits, it's beneficial from performance standpoint, but also, which is what is very important, is deterministic in time. The reason it is deterministic in time is that because when you copy a page, that time can be very easily measured. And when you copy the same page back, it's the same time. So we can prove, and I'll talk about this in a bit, that this page-based recovery mechanism is in fact deterministic in time. So what you see on the picture on the right-hand side here is a sample of what happens at the top. At the top of this picture is the logical algorithm. At the bottom is the physical recovery algorithm. And this is simply an example of what happens with the database objects when we update a key value, a field which is indexed. I'm not sure that I need to go through this picture. It's on the slide, it's documented pretty well here. I mean, it says what it does. But again, what I'd like you to take out of this slide is that the two algorithms, two mechanisms, one is page-based, it's physical, it's not aware of any logical organization of the database of what is being recovered, and it's deterministic in time. Now, this is some statistical information that we collected to compare those two recovery algorithms. What you see here, the blue line is the old, the logical the blue plots is the logical recovery algorithms. And the red is the physical, the new method. So on each of those slides, the different operations. There's delete operation, insert operation. The first one is insert, in fact. The second one is absurd is when the key is modified. And the last one in the left-right corner is when the key is updated. What you see here is that we're on every transaction first. We first run the update, and this is the body. Then we run the commit. The third plot here is just a summary of those two operations. And then the exact same transaction is run again, only it doesn't commit, but it is rolled back. So what you see here, regardless of the operation itself, regardless of the transaction, it shows that the physical recovery is not so bad. It's actually in par with the logical recovery. On this slide, though, we see what happens with the rolled back. And this graph is normalized. The logical recovery is taken 100%, right? And you see that the physical recovery, the copy-on-write-based recovery, normally takes just a fraction of time of what the logical recovery would take. Now, this is probably the last concept that I'd like to go over. And this is just your normal transaction flow, extreme DB transaction flow. Again, I'm talking about the Mursi, if not MVCC transaction manager. So you see a number of different operations on the left-hand side, insert, update, delete, delete all. And then there's a workload. So the first thing that happens, the transaction is entered into a queue. It waits until it is allowed to run. Now, it runs its workload. So there's some transaction body here. And then at some page, at some point, the application calls commit. And those modifications are committed. In extreme DB, commit is divided into phases. The first phase of the commit, essential updates, indexes. And the last phase of the commit, it essentially, it's uninterruptible. It clears up various locks, various latches, rather. It's normally a very short in duration, and it's atomic. There's no way that operation can fail. So the second commit phase is very short, and it's always, always succeeds. Now, the rollback, if and when the rollback happens. Again, this graph shows what operations happen when they're all back, when they're all backgrounds. So it just, this picture gives you a sense of just generally how transaction is structured in extreme DB. To phase commit, there's a workload, there's a queue interval, there's a wait interval, there's a workload interval. Well, the transaction does it's, does it work? And then, again, to phase commit, and the rollback is simply, if the rollback happens, it simply puts everything back to, restores it to the original state. Okay? Now, let's talk about, at this point, let's talk about the real-time design and what happens with how we use extreme DB and what changes we made to extreme DB to enable predictable transactions. But first of all, and that's probably kind of marketing statement. Don't confuse fast and high throughput with real-time. With real-time, any real-time processing, there's a line in the sand. So there's bad transactions and good transactions. With high throughput, speed is just, it's a figure of merit. So with real-time, speed is not so important as predictability. So here's our goals. What we'd like to do, we'd like to impose and enforce transaction deadlines while preserving the consistency through asset properties. Essentially, what we do, we introduce a deadline value into the transaction call. So why is it difficult, why is it difficult algorithmically and why is it difficult technically? But first of all, enforcing temporal consistency of data and transactions. It's never been, I'll take it back. So with real-time transactions, transactions have their deadlines. So what can happen, they can meet their deadlines. So they can commit within the time frame that they're given. They can miss their deadlines, which means that they are all back again within the time frame that they're given. But what should never happen is that never should run past their deadlines. So that's something that real-time operating systems do. They guarantee predictable intervals. Now, database operations are very difficult. So the complexity evaluation of any database operations, it's hard to know the upper boundaries in time of various operations. And it is also very difficult to make an accurate estimate of the time of execution on any level. It could be either application or the database kernel. And when you calculate the worst case scenario, you can tell, you can say that, yeah, we can say that three operations, search operations are logarithmic. However, those worst case scenarios could be too pessimistic. They can be so pessimistic that they are almost useless. Now, real-time scheduling goals are also different from application to application. Some applications want to schedule transactions to minimize the number of transactions that miss their deadlines that are all back. They may or may not allow preemption. It could be high-priority transactions that run in a context of high-priority tasks. And the database scheduler may or may not allow preemption. Now, there could be other different criteria that real-time applications impose. For example, maximize the total weight of successful transactions. You can assign a number, like a deadline plus priority to the transaction, and your goal could be to maximize the total weight of successful transactions. So, essentially, scheduling goals are very different for real-time applications. Now, the success and performance criteria, again, they're not the same as for non-real-time applications. Performance criteria for your normal database, non-real-time database, is normally to optimize the average number of transactions per second. With real-time applications, it's not the average time. It's the worst case scenario. Now, why is it difficult technically? Well, first of all, real-time applications, they run in a context of a real-time operating system. And if they don't, they have to be able to access hardware, interrupts, memory management, timers, things of that nature. Now, real-time operating systems, they have different scheduling policies and different ways of measuring time. The database that runs in the context of real-time operating system and the scheduler and recovery algorithms should be able to adjust to those services, operating system services. Now, real-time applications, not always, are considered mission-critical or safety-critical. So, their underlying operating systems is one of the certified operating systems. And generally speaking, manufacturers approach how operating system vendors certify their products. What they do, they cut system services that they cannot fit into the certification boundaries. Essentially, we have much less functionality from the operating system than you would have on running on the normal Linux or Windows or non-real-time, non-real-time OS. Now, the same certification rules are normally applied to development tools, compilers, profilers, etc., etc. And again, those tools are generally speaking safety-oriented as opposed to anything else. So, well, that's a challenge. That's a technical challenge. Can you give an example from something that the real-time OS is going to not have that full-fledged server-class Linux is going to have that you care about for a database? Sure. For example, semaphores. In VxWorks and real-time version of VxWorks, certified version of VxWorks, you can take a semaphore, but you can never release it. Releasing a resource is considered a security risk because you actually can pull the rug out of your operation. That's what the thought is. Allocating memory is a big no-no. You never can call alloc or malloc in a certified environment. Again, the thought is that if you do, then you can forget to free it and you create a resource link. So, to that extent, XtremeDB doesn't use any system services that allocate memory. All our memory managers are our own memory managers. And the same goes for synchronization primitives. If we are able not to use a semaphore, we don't. We create our own ledge instead that we can fully control. Did I answer the question? Yeah, that was perfect for you. Okay. Now, this slide shows why the real-time operating system is in fact essential. So, what you see here is the two cases. Let's see. So, the top picture, you see two threads running on macOS, doing something with a database, updating the database, running on the macOS and on the right-hand side under the operating system called deals. It's a real-time OS. So, what you can tell from this slide that under macOS, only 12 percent of transactions miss their deadlines. So, all those transactions, they have the same priority. They run in the context of the same priority task. And they essentially do the same thing. Under the real-time operating system, all of those transactions run successfully. None of them miss their deadlines. Now, the second example, the bottom picture, is when we have one transaction, a writer. Essentially, it's a sensor that populates data and writes the database. And it's a high priority transaction. The low priority transaction is a transaction that looks up the values and read data back. Again, if you look at this picture under general-purpose operating system, 12 percent of the transactions are missed deadlines. And in real-time operating system, due to their predictable scheduling policies, essentially, all of them run through. So, that's the difference. What I'd like to emphasize here is that real-time environment is essential for real-time database. Real-time operating system is essential for real-time database. How good is, like, you can run Linux in RTS party mode, right? Like, you can set the schedule, maybe in real-time scheduler, how close is that to be, like, how good is that for the database, for the stuff that you're doing? Honestly, we don't use RTS Linux. Okay. So, I can't... Wow. Because none of your people that are using it are using it, or just because it's out of the box, it doesn't do what you need? It's market-oriented. Our customers run, when they run real-time processing, they run real-time operating systems. They run VxWorks, they run Integrity, they run Zeus. None of them run Linux. Another point here is that, yes, real-time RTS Linux and RTS Windows, for that matter, can be quote-unquote real-time. But as I mentioned earlier, there's this certification requirement, requirements in the applications that we deal with. And Linux, by definition, can never be certified. So, we deal with, I'll say, 30% of the applications that this design is intended to, would run on this certified operating system, which is never Linux. Okay. Thank you. Now, how we deal with complexity evaluation for the real-time project? Well, we follow the footsteps of the operating system vendors, and we cut off those services that we can't or don't want to put into real-time out. As you can see here, all the modules that are outside the kernel, we've gotten rid of it. We don't support that. So, all we've left is a low-level backup capabilities. So, and inside the kernel, we will have three types of trees, of indexes rather, a bit tree, a hash, and an R tree. And we also don't support persistent storage at this point. We only leave the transient storage, and we only support more safe transaction manager. We don't support MVCC transaction manager at all. So, that's the extent of the current design. We would have to, at some point, to add persistent storage support. And that is because many modern devices, they do have persistent storage. It hasn't been done. It will be. Now, let's talk about real-time transaction schedule. So, real-time database should guarantee the logical consistency, ought to guarantee the logical consistency of the database, and also make sure that transactions meet their deadlines. So, we implement a so-called high priority, earliest deadline, first scheduling algorithms. Transactions are scheduled for execution based on their priority and deadline. Essentially, when you have two transactions, the first thing that happens is we see are they the same priority or their priorities are different. If they're the same priority, we allow to run, we schedule to run transactions that deadlines are closer, shorter. So, if the priority of transactions are different, we first schedule the transaction with a higher priority. Now, we also allow preemption. So, if you have a running transaction, there's certain rules, and they're written here, and that allow us to kick that transaction out. It doesn't happen always, but if there's a higher priority transaction, chances are that the currently running transaction will be kicked out. So, again, is the high priority, earliest deadline, first EDF scheduling, and this is a pretty common scheduling mechanism, but with a kick, we allow preemption. Now, so, how we enforce deadlines through rollback. So, the database kernel, again, firm transaction deadlines are deadlines are enforced through EDF scheduling mechanism. So, we must guarantee that transactions are finished, whether they commit or rollback within their defined deadlines. So, we are able to identify late transactions, we're able to interrupt those transactions, and we're able to initiate the rollback in time to satisfy the transactions deadline. And the key problem here is that we must ensure that when we interrupt the transaction, it's still in a recoverable condition. So, we actually can run a rollback in time and keep the database consistent, logically consistent. Now, the way we do that, the database kernel, the database runtime, reserves enough time out of the given transaction deadline to initiate the rollback. And we can prove, we can mathematically prove that the transaction's rollback time is taking less, is less than the total time of the transaction has spent modifying the database. So, that's the key statement, the key assertion here. So, again, we can guarantee that we reserve enough time out of the given deadline to be able to rollback all the modifications. So, may I ask a question? Sure. Yeah, so, if you say you can prove that rollback time is definitely less than the time spent modifying or searching, does that mean that, say, if the deadline is one second, then at half a second, you have to rollback if you haven't finished everything? Because, I mean, you can only guarantee that the rollback time is, at most, to be the same time you have spent, right? Does that make sense? Right. Yes. Like, for example, if the deadline is a second or an hour for that matter, we can guarantee that we can rollback in half an hour. Yes. So, essentially, if the transaction has not finished all the work within the first half an hour, then, yes. Okay. Thank you. Now, how much time do you have left? How much time? How much about an hour? We are, let's see. You probably need to talk about that more. Okay. Well, give me another... Maybe five minutes. Five minutes, okay. When you want me to start rolling you back, though. Right. So, give me another five minutes. So, here's my key assertion then. So, the time to reverse any modifications to the database made by a transaction up to any point in the transaction, it doesn't exceed the time required to apply those transactions. The rollback time is always less than the time that the transaction took to modify the data. And, as I said, we can prove that. We can prove it mathematically. What I'd like to do, I'd like to illustrate that. It's a pretty obvious statement for read-only transactions because read-only transactions don't actually modify any data. They modify internal structures, right? So, for the read-write transactions, the mechanism that I've described earlier, the page-based recovery, is the crux here. So, since we need to copy pages, but the time that takes to copy a page depends on the size of the page. But this is all it depends on. So, based on that simple fact, again, we can prove that putting pages back during a rollback wouldn't take more time than the original modification. And these two slides, they actually illustrate the process and there's code snippets that show what happens when the transaction runs, when the real-time transaction runs. And this is some statistics that we've collected to demonstrate our key assertion. And, again, it shows the blue line is the workload, the red lines are the rollback, and we've done a lot of research and a lot of experiment and testing, in effect, and that verifies the assertion. I mean, it's not a proof, but it's an illustration rather. Five minutes, did you say? Yeah, exactly. You should wrap up because you have my questions doing it. Okay, yeah, it's not me, it's the baby. Yeah, I understand. So, let me just then emphasize one other thing. So, it is important. And so, we need to be able to verify time. We need to be able to see, okay, do we have time left? So, that means that inside this, inside our kernel, we need to put verification points, points in the code, where we must check the time, whether we have enough time to run, or it's time to roll back the transaction. So, this is a very delicate thing, and this is essentially why we limited our services to the amount of services that were limited. So, and this is probably the most time-consuming and important part of that design. So, obviously, every runtime call needs to have this verification point. Obviously, every loop needs to have a verification point for, on every loop iteration, right? But what's more important, that those time measurements, they depend, they heavily depend on the operating system environment and on the hardware that that operating system is running on. So, what we've done, we've taken several setups that we claim to support, and we've measured the maximum interval between control points on every supported architecture. And then, we made sure that we added control points when necessary, so that we guarantee that the transaction never run past its deadline. So, then there's certain techniques to estimate control point value, to estimate the time, when it's time to interrupt transaction. And again, we assert that half of the deadline is generally safe. Then it goes to methods to enforce deadline, and essentially that too, the database runtime doesn't have a way to measure time itself, it's just a library. So, we either use an application created timer, or we use an application created callback, in the absence of the timer. So, and our API, our real-time API supports both of those methods. Again, we either measure, we either use a timer created by the application, or we use a callback, and the application handles the time. All right, so, Andres, I've got to cut you off, I'm sorry. Right, okay, well, again, so I will applaud on behalf of everyone else, thank you for being here. So, we have time for one question, and you go to the audience. Okay, so I'll ask my question. So, in the very beginning, you showed the example of, especially if you define, it's like an ORM, you define the schema in a speech drop, or speech class, and then you convert that into, you know, C code that you write your application against. Then you also showed a bunch of other sort of APIs that you supported, like you had JNI, you had, you mentioned Lua and which are the things, and you had one where you had embed SQL. So, I guess my question would be what percentage of your customers are using XtremeDB through like the standard C API versus something like SQL? Well, it depends on the field. Embedded customers, I would say 50, half. Financial, people in financial arena, they of course use SQL, because there's no way not to use it. Our customers who use our distributed designs, all of them, 90% of them use SQL. But in, again, in embedded settings, it's mostly either C or Java, but it's a language called API. Okay. All right. Well, thank you so much for bringing this. We appreciate you spending time with us and explaining that. Happy database system we know we don't see about, so it's super interesting.