 Okay, so today's lecture is going to be about database storage part one, this is the first of two-part lecture, just before we get going, a few housekeeping things. So homework one is going to be due on Sunday, September 12th at 11.59 p.m. Project number one will be released the following day, Monday, September 13th. A few other things, unfortunately, you may have noticed DJ Mushu couldn't be here with us today. He's actually out on the west coast in LA, I don't want to get too much into details about it. He has this long-standing beef with a crew out there, so he's trying to get that wrap up and he should be back in class on Monday. The other thing, the office hours should be finalized now on the website. A few of them have switched to Zoom only, so you have more remote only options if you'd prefer. And we also created a Google calendar, which is now posted on Piazza that shows office hours, deadlines, those sorts of things. Okay, so that's all the housekeeping stuff. Are there any questions about anything related to that before we get started with the material? Great. Okay, so just kind of a high-level recap, an overview of what we've talked about. So far, mostly we've been focused on the high-level or application-level side of things, topics like the relational model, SQL query languages, how an application or an end user would interact with the DBMS. So kind of now, our focus is going to shift for the rest of the semester about how to build the DBMS. How to build the software that actually manages the database, how to build a database management system, and kind of go through all the different layers of that. So I showed a slide that was similar to this in the very first lecture, and kind of it breaks down the different high-level pieces that we're going to cover in the course. So we've already kind of talked about the high-level idea of relational databases in query languages, and now we're going to dive down into each layer of the system. So starting with storage, working up to query execution, concurrency control. So if you have concurrent transactions or queries that are executing, how do you manage that? Recovery, if your system crashes, how do you recover from a failure? And then some more advanced topics at the end. So that's kind of the high-level outline. The other way that you can think about it is sort of this software stack diagram that's shown on the other side of the slide, where kind of the base layer is the disk manager, and we build additional layers on top of it. So each one of these blue boxes corresponds roughly to a self-contained part of the system. So each one of these boxes represents kind of an abstraction that each upper level in the stack can kind of call on and interact with. It doesn't need to worry about the implementation details of the layers below it, but it can still have some kind of API where it gets what it needs from those layers. So not everything is perfectly clean cut. Sometimes certain layers will need to interact with each other and know what's going on at a finer grain level, but in general the system is designed to be modular such that you can build these layers one on top of the other and you don't have to know necessarily low-level details of the underlying layers. So the topic of this lecture and the next lecture is going to be specifically the disk manager component that manages how the system is going to interact with the underlying file system. So kind of the whole premise of this course is going to focus on what we refer to as a disk-based DVMS architecture. And what that means is that the DVMS is going to assume that the primary storage location for all of the data in the database is on some non-volatile secondary storage. So you have some persistent storage medium like disk or SSD or something like that. And the DVMS assumes that all of your data lives on that storage medium. So the components are going to manage specifically the movement of the database files to and from whatever your secondary persistent storage is. So the DVMS is going to handle reading from and writing to the underlying disk. So it's helpful to think about kind of at a high level how the storage hierarchy in a system works. So kind of starting at the top you can get all the way up to like the CPU and at the very bottom is the kind of lowest form of storage. And it's helpful to think about kind of the trade-offs that you get with each one of these layers in the hierarchy. So at the top again on the CPU side it's much faster to access data if it's in CPU registers or CPU caches. But the available space that you have to store data is a lot smaller and it's much more expensive per byte of data that you store. And as you go all the way down to the lowest layers it becomes much slower to access data but you have a lot more space and it's cheaper per byte. So the important distinction that we're going to make in the course is between volatile and non-volatile storage. So volatile storage you typically can access data randomly and it's byte addressable. But the problem is that if you remove the power source, so if you trip on the plug and it gets unplugged or you know the power goes out or for whatever reason the system crashes you're going to lose all of the data that you have sitting in your volatile storage. On the other side of the line is non-volatile storage which is typically block addressable so you can't necessarily access an individual byte. You can get rather a block of data, a range of bytes. But the benefit is that if the power goes out all your data is safe. So power can go away fully, plug it back in and all the data is there. So this split here is the important part we're going to care about for correctness guarantees in our system. So anytime that we talk about DRAM we're just going to refer to it as memory in the course anytime we talk about any kind of disk or persistent storage it's anything that's non-volatile. So it could be SSD, it could be a traditional hard disk, it could be some kind of network storage whether that's like a distributed file system like HDFS or if you're on the cloud maybe like Amazon S3 just something where you have network accessible storage. So kind of just memory is volatile and disk is a non-volatile storage. And then there's also some, as I mentioned, higher level CPU storage. But for the purposes of this course we're not really going to be concerned with that. So this is kind of the traditional hierarchy. There are other storage modalities. For example, if you have fast network storage we're not really going to be too concerned with that. There's this other layer that kind of sits in between DRAM and what you would consider persistent storage called non-volatile memory. The marketing people decided it would be better to be called persistent memory so you might see that now. It's kind of interesting because it gives you a lot of the same properties as traditional DRAM but if you remove the power it retains the data. So there are some trade-offs, it's more expensive than some other forms of persistent storage and it's slower than just regular DRAM but it gives you the byte accessibility so you don't have to worry about losing your data. So Andy actually, in his first PhD student, Joy actually read a book about non-volatile memory database management systems, kind of how to redesign the stack to benefit or best leverage from non-volatile memory. And there are increasing numbers of products being released. This one's from Intel. For a long time there was this sort of constant promise of non-volatile memory out there. The marketing people always said it was coming soon but soon never really came and it's kind of been in this perpetual limbo but more and more products are starting to come out and maybe I don't know within the next ten years or something it might become more commonplace but for now we're not going to worry about it because it's not common in typical deployments. So another way to think about kind of the storage hierarchy is to consider the access times for each layer in the stack. So at the top you have the CPU caches and those take on the order of nanoseconds and on the very, very bottom is tape archives which is like a billion nanoseconds. So these numbers are kind of difficult for humans to wrap their heads around so if we convert it to a little bit easier timescale, if you think about it, normalize it to like seconds. On the top reading from the CPU caches is like half a second and on the very bottom if you want to read from tape that's going to take you like 31, 32 years. So this kind of storage hierarchy is going to be important because as designers of the DBMS it's important for us to consider the access latencies to the different levels of the storage hierarchy where data can live in the system. So maybe another helpful way to think about it from a famous database researcher Jim Gray, his analogy was in terms of like if you want to read a page from a book right in front of you that's like accessing the highest layers in the hierarchy. Going to disk is equivalent to going to like Pluto and then if you want to read from tape that's like going to a different galaxy altogether. So again this is going to be really important just to keep in the back of your mind when thinking about the design decisions in the system. So I also mentioned that there's this distinction between sequential versus random access. So typically in memory random access is okay. You can access a random offset in an array and it's not too bad. Sequential access might be a little more performant but there's virtually no difference between them. Whereas if you're accessing data stored on a traditional hard disk because of the way the disk read head has to move to access data on different platters it's more beneficial to do sequential access. So the DBMS kind of design we'll be focusing on is really going to need to consider these access patterns sequential versus random. So the algorithms, a lot of the algorithms we'll look at are going to try and maximize sequential data access for contiguous data blocks at all costs. Kind of the allocating multiple pages at the same time in like a contiguous range is referred to as an extent. So kind of the design goal that we're looking at here is to allow the DBMS to manage a database that exceeds the amount of memory available. So this was especially important a long time ago when memory was scarcer than it is now but if you have a really large database you're still running. Now for the rest of the commercial, I'll pass the mic on to my fellow. No need for a mic, check, plus it. The bees will set to grab a 40. To put them in the yoke and snap the snacks and I'll take a sip then wipe your lips, cue my 40s getting warm. I'm out, he got split tipped. Drink it, drink it, drink it, then I burp. After I slurp, I scoop, I put in much work. With the BMT and the eDrum, get us a Saint-Eye Brew on the gum.