 So how's the conference going for everybody? I feel like I should get up here and whoop, cheer, right? Yeah, a couple of good talks I've seen so far, so that's good. Hopefully this will be one of the decent ones. So I'm going to talk to you today about Squirrel and Apache Accumulo. So I am CTO of Squirrel. Squirrel is a startup focused around big data with an emphasis on security. Accumulo is an Apache Software Foundation project. And we're going to go into details on how they work. We're going to talk about some of the motivation, particularly in the real-time perspective. I'm going to try to derive some of the elements of what's needed in a real-time database to support this stuff. We won't go too deep on that part, but we will go pretty deep in Accumulo technology. And hopefully you guys will be able to go code up a big database after this talk. All right, so again, two halves of real-time. We're going to talk about data-driven and query-driven real-time analysis. We're going to talk about Accumulo and Squirrel. And then in particular, since we have a big emphasis on security, I'm going to talk about data-centric security, how that fits into big data analysis in general. We'll talk about some table designs to support a variety of applications, given this context of real-time and security and all that. And then I'll share some performance benchmarks so you guys can get an idea of how this actually works, how well it works. OK, so just to derive this definition of real-time, I'm breaking things up into two different halves. So on the left, you see the data-driven side. In data-driven real-time, real-time generally means you're trying to reduce the time between an event happening and the time that you react to that event happening. So this is typically the realm of stream processing engines. Things like Storm are in that category. I've got Twitter showing up there. IBM has InfoSphere Streams. There are a whole bunch of vendors that are in that space. Then there's this other half, which is query-driven. And in query-driven, the real-time typically means reduce the amount of time between ingesting data and querying that. So an event happening and the ability to see that event affecting the results of the query. And there's sort of a connection between the two. So on the data-driven side, many of the challenges have to do with increasing the state space available. So if I have to make a decision on the fly and I can only look at what fits in cache on the chip, I can't bring in a whole lot of context. But then if I can query out to a database and get an idea of when was the last time I saw a similar event, what's the background model behind it, is it anomalous, there are a number of those types of queries, which I would want to be able to enrich the data as it streams by. And then on the query-driven side, typically there's an ETL process. So query-driven is often complemented by data-driven capabilities of processing data, transforming data, and contributing to a database. And the way I see it is there's no SQL database layer. That's really what kind of combines these two has. So people moving to denormalize data models to bring data together so that you can ask very quick low-latency questions of a very rich data set. That's tying these two halves together. So here's my basic pseudo architecture for query-driven and data-driven real-time analysis. So here we have data flowing in from the left. The data at the same time gets put through a stream processing engine and gets stored in some type of database. I've just got that labeled as no SQL plus for today. And then there are a whole bunch of interactions there. So on the right side we can take actions, we can display dashboards, we can support operational and exploratory analytics on it, like through interactive analysis tools. And each of those things is gonna use a different set of interactions here. So if we just go through that, effectively we're talking about as data streams through the stream processing engine, we're gonna enrich it using no SQL queries, again, denormalize model. Results of stream processing engine often get written to the no SQL database to give you that longer-term storage of some transformed version of the data. It might be kind of a map into it or a graph view of it, link extraction into de-extraction, these things fit into that model. And then all of these things flow into the human interaction. Could be human, could actually be automated analytic on the right-hand side. In this particular talk, we're not gonna focus on the whole space, we're really just gonna focus in on that no SQL part and in particular, a cumulo and squirrel as an implementation that supports really all of these types of interaction. All right, so, a cumulo and squirrel technology. So I'm just gonna kind of dive deep into what a cumulo is. For the most part, a cumulo is a sorted key value store that's derived from Google's big table paper. So there are a bunch of big table clones out there or things that are similar to big table clones. So it's in the category of HBase and similar to Cassandra in some ways. But really what we're storing is sorted key value pairs. A key in this case is a structured element. So we're trying to take Hadoop and provide some structured interaction with it. That's kind of the general space of no SQL. So let's just go over a couple of examples here. Keys in a cumulo are sorted lexicographically and hierarchically. So I can group data by rows and in these examples, everything shares the same row identifier. So this is a row about John Doe, even though in this chart we're displaying four rows. We've considered this one row inside of a cumulo. A bit of a nomenclature issue there, but yeah, anyway. Things are then grouped by column family. So I've got notes and test results in here. Column qualifiers show up as basically our ability to de-normalize any data we want together and extend rows, support flexible schemas and support an arbitrary number of columns in a given row. One of the things that makes a cumulo unique is the addition of a visibility label. And I'll talk about the overall security architecture and where this comes to play. But that's inside that core data model, that core key value store, that visibility label makes a cumulo unique. And then effectively we map these things into values and the value is just an arbitrary byte array. We can provide higher level semantics on top of that in many different ways. A lot of people work on that. I'm not gonna dive too deep into that in this talk. But there you have it. We're storing key value pairs. Any questions on that? Okay. All right, so one of the things we're trying to do with that is scale up. So basic computer science approach to scaling is divide and conquer. In this case, we're dividing by partitioning our key space. So we wanna store sorted key value pairs. We can break those up into contiguous chunks of keys. Those things are called tablets. And that's what I'm displaying here in the white rectangles. So here's a table at the bottom. This is my table, Adam's table. And I've got two tablets. So I can break up my data into anything from the negative infinity key to things. And then anything greater than things goes in and out. So I split that up. I've created two, what we could call units of work that can then be distributed throughout the cluster. And I can do that with all of my tables, sort of keep a set of tablets. Those tablets, I need to keep information about them, metadata about the tablets. Specifically, what are the files in HDFS that are associated with those tablets? Where are they being hosted? A little bit of life cycle stuff and dealing with timestamps. There's a little bit of extra metadata around that. Those things are stored in a metadata table, which itself is a table in a cumulo and it's broken into tablets. And the metadata about those tablets are stored in a root tablet, giving us a three-level hierarchy of units of work that are distributed throughout the cluster. And then I can just keep a well-known pointer to that. It's a very nice scalable model, gives us that root of scalability of a cumulo. And it's all really derived from that big table model that Google published. So from a process perspective, we can take those tablets and we can distribute them across tablet servers. So a given tablet server will host a subset of the tablets. Clients on the right-hand side, our applications here, read and write directly to the tablet servers. It's sort of a shared nothing architecture in that you don't have to go through a single point to do all of the interaction there. Instead, you can go kind of directly to the machine that is responsible for hosting the data. And then there are a series of background processes here that manage data, manage availability, balance load, assign authority, delegate authority through the acquisition of locks, distributed locks in ZooKeeper. And this is the basic architecture here. So effectively what we have is a series of tablet servers. We can distribute through a large cluster. We've scaled these up to thousands of nodes and it scales up nicely and gives good parallel performance across them. So zooming in just to one of these tablet servers hosting one tablet, we have a situation here where we're really looking to support rights coming in at a very fast rate and reads coming out also at a very fast rate. And we're trying to minimize the time between rights showing up and reads coming out. So this is where we're tying back into that real-time behavior. Really, if we can write data and within a millisecond have it show up as the result of a query, that's real-time. So this is how we do it. Rights come in, we write them to an in-memory map. At the same time, we write them to a write-ahead log which is on disk. And once it's in the in-memory map, reads will see that data. The in-memory map is a sorted binary tree, a balanced binary tree. Again, since that's in memory, it supports very fast random IO. So if key value pairs are coming in in random order, I can very quickly get them into a sorted order to satisfy queries. The other thing that we're trying to optimize here is our impact on disk resources. So if every time a key value pair came in, I had to write it to a particular place on disk. I'd be seeking disk all over the place. I'd already be increasing my latency to something like the eight millisecond standard seek time of hard drives, which is, that's big. So instead of doing that, basically I'm writing sequentially to a write-ahead log. And then when my in-memory map fills up, I write sequentially to a sorted index file, something we call an R file inside of a chemulo. But the sorting happens here, this is a sequential write. Eventually we write a bunch of files as single sequential writes. If I have too many of these files, I basically have to do a merge at query time to bring that data out. That becomes more expensive, so I want a smaller number of files. So I have a compaction operation that basically compacts those together. So this overall flow that we're looking at, that I just took you through, is the log structure merge tree technology. Goes back to the mid-90s. Google adopted it for a big table. We've coded our own version of it for a chemulo. But it's, again, that ability that allows us to do random IO with low latency between ingest and query. And minimize that impact on disk resources. So along with that, we have these operations, minor compaction, merging or major compaction, and scan. All of these things are places where we are streaming key value pairs in sorted order through the processor, effectively. And they're very natural places for extending our analytical capabilities to do real-time analysis. So this is another thing that makes a chemulo unique is this mechanism here called the iterator tree. So it writes flush to disk. We take them through the iterator. That iterator can do operations like filtering, aggregation. On the scan time iterator, we can also do some types of joins even that are localized there. So I'll take you through a few examples of that. Here's one of them. So assume I'm doing word count. I have a corpus of documents. I want to basically generate key value pairs where the word is the key, the value is the number of times I've seen it. Effectively, what I'm doing is if I have the same word repeated multiple times, I can calculate a sum across the values and then my aggregate view of that after it passes through the iterator is that the word count, effectively. So this type of operation, this type of aggregation operation, since we're operating in these compaction time scopes and in the query scope, we can provide both consistency, always give the same answer, always give the most correct answer at the same time as we're piggybacking on this efficient use of hard drive. So effectively what we're doing here is we're avoiding read, modify, write while still performing analytics that are traditionally performed using read, modify, write technology. So one way to do word count, read the previous value, increment it, write it back out. We're not doing that. We're batching things up together and sort of aggregating them on the fly through the log structure merge tree. So later on, I'll show you a performance number of why that's orders of magnitude better performance than read, modify, write. So effectively the iterator framework here allows us to do a huge number of operations and this is in fact where most of the logic for just dealing with key value pairs in general and accumulo resides. So all of these things are actually implemented as iterators inside of the tablet server. So it's not just analytical capabilities, it's also just kind of core management of key value pairs. And in fact, that's where a lot of the security is implemented inside of accumulo as well, incidentally. Okay, so let's look at latencies here. The way I write data to accumulo is through this batch writer mechanism. I send a bunch of key value pairs grouped as mutations to rows. Those hit the tablet server, go into the in memory map. At some point they get scanned. The scan also brings in data from files that exist there and then on the client side, on the read client side I have another scanner over here. And the time to ingest into the in memory map, there we're talking about a millisecond on the order of the latency to scan that data from the in memory map into scan iterators and the R file into scan iterators on the order of a millisecond. If we have a lot of seeks, it can be larger. If we have caching, it can be smaller. But again, on the order of millisecond. And then querying it out of that, again on the order of millisecond. So, end to end our latency is very much real time. Not necessarily real time from the perspective of making trading decisions, but it is real time from the perspective of analyzing the most up to date view of the world from a human analysis perspective. So, a big thing that we're trying to do here is really scale this up and get as much throughput as possible through this type of channel as well. So let's just look at the throughput from this perspective. And this is what I'm trying to do here is build the case for avoiding read, modify, write. So from that perspective, a cumulo, as we've written it, since we do these kind of batching things up together and efficient pipelining of all of this, we can actually support on a per node basis, upwards of 500,000 entries per second in just into a node, right? Across a cluster, multiply by the number of nodes in the cluster and that's the kind of performance you get. A query time, it can actually be a little bit more efficient and get about a million entries per second. Somewhere on that order of magnitude. If we were doing a client side read, modify, write operation and we have to sort of read the previous value, compute some new version of that and then write it back out, already we're introducing millisecond order latencies into a single write operation. So that actually becomes quite a challenge to provide greater than a single thousand entries per second, much less, 500,000 entries per second per node. So with that read, modify, write, already you're greatly reducing the rate at which you can ingest and query data in the system. So avoiding this, great, if you don't have to do read, modify, write, if you can just do inserts and sort of fire and forget, batch them up, that's great. With the iterators and with what I'm talking about there, what we're trying to do is increase the types of analytics that you can perform without going to a read, modify, write, type solution. So that's where this aggregation on the fly or aggregation through the log structure merge tree becomes a very useful tool supporting high throughput real-time analytics. Yeah, so we haven't really tied this to a particular use case yet, right at this point at this point in the talk, we're just kind of talking about abstractly what's the technology for that. But all of this actually ties into higher level concepts like search, log analysis, cybersecurity applications, graph analysis, all that. And I'm gonna go through a lot of that. Yeah, like millisecond latency type things are good. Microsecond latency type things, we can't support that with this type of architecture. We're optimized for disk-based access, right? Storing data that doesn't fit in memory, that type of thing. Okay, all right, so effectively what we've talked about is this accumulo layer here. My company is Squirrel, and where Squirrel sits is in this stack, right? So we've got HDFS at the bottom, accumulo stores its data in HDFS, provides that sort of key value IO, provides that low latency key value access with high throughput. But again, like you mentioned, this is still a pretty abstract concept. We're dealing with key value pairs, sorted key value pairs. How does that translate into the application space? What we're doing with Squirrel is to take a series of design patterns and table structures for organizing those key value pairs and bringing that into things like JSON document, ingest, and output, right? Graph analysis, did anybody go to the Neo4j talk? It was a good one, you shouldn't have missed it, but you did, anyway. We do things like aggregation, like I mentioned, very efficiently using that compaction time aggregation and this is actually exposed through a subset of SQL. So group-by type queries in SQL were very efficient at that. And then search, right? Search in general, this is talking about building indexes. So if you recall, we have the stream processing engine and the NoSQL database against the NoSQL database. We have the operational and exploratory applications. That type of search capability, and what we're exposing here is Lucene style search. That's really critical, having the ability with low latency to query with rich semantics into that very large data set and get the most up-to-date answer. That's really what we're talking about doing here with SQL. Okay. All right, so let's just, we'll take a little side tour here into data-centric security and then we'll talk about the table designs that really enable SQL. Okay, so data-centric security, simple definition of this is you have a system in which data carries around with it metadata or really elements of provenance information that is required to make decisions about that policy at query, sorry, policy decisions about the releasability of that data. All right, so I have a set of data, I have a set of users. Some of those users can see a subset of the data according to policy, but I need to be able to make that decision very quickly, supporting these high throughput streams of data, scanning key value pairs out of the database and filtering it based off of those security requirements. So effectively you can think about this as I have a database of key value pairs, I've got two different users, each of which can run the exact same query and see a different view of the data. All right, so this one gets an anonymous view of the name while this one sees the name as Jones. All right, this one sees the age of row one while this one just doesn't see that data. Those types of things, fine-grained access control that are all back-ended by squirrel and Acumula. It is cell-based, it's kind of the intersection of those, but it's a little bit deeper, so you can have two different rows that would have the same visibility and two different columns that have the same visibility, but the points of intersection there can just have arbitrary different visibility. So it's more expressive than the combination of row-level security and column-level security. Does it do what? Versions of it. Yeah, there is some versioning of it. There are ways of expressing that in it, yes. Okay, so let's just focus in on this particular example here. Whoops, okay, anyway, push the wrong button. All right, here again, we're looking at a row, which is John Doe. John Doe has some information about him. Primary care physician notes. Maybe the visibility of those is only the primary care physician of John Doe can see those notes. So in this particular case, we're looking at a healthcare example and just search over healthcare data. In the US and in many other countries, there are laws that protect that type of healthcare information. In the US, we have HIPAA law, we've got the Affordable Care Act. All these things create requirements like if I wanna see somebody's electronic health record, I have to have a relationship with them. I have to be their care provider and I have to, you know, there are sensitivities associated with this as well. So things like the fact that you have AIDS, right? Maybe even your primary care physician can't see that because it's protected at a higher level. There are certain classes of information there that provide kind of this rich set of policy and rich set of interactions there. And that's the type of thing that we can express inside of that visibility label, right? We can support Boolean expressions there as well. So maybe John Doe can see his own cholesterol test results as well as his primary care physician. And his psychological data can only be seen by his psychologists. These are the types of things that are directly derived from legal restrictions in many industries like banking and consumer internet, there's privacy policies around the data. All of those things really, if you want to share this data amongst a large group that have diverse access authorities into that data, that's where you need that fine-grained access control. In this model we have, you can think of this as we've organized our data into some key value pairs and then we stick labels on those key value pairs according to who can see them. So it's not necessarily a row per roll, but the role actually plays a part of defining the key value pair itself, right? That's right. And in fact, getting away from encryption, barring any kind of like homomorphic encryption which will allow you to do some basic analysis on it. This is a way where you can do a much richer set of analytics while preserving that privacy. And we can tailor that towards bringing in large groups of people with diverse access authorizations into a single database rather than having a whole bunch of separate databases for those separate information sets. Yes, and a lot of what we've done is, let me jump to the next slide here, provide an ecosystem in which this is a manageable system of rules instead. So, right, so how do you populate the visibility column? Effectively, this is what we're talking about in this view here. Here's our data-centric security ecosystem. Typically we have data and policies that show up that they're already there. The data shows up, the policies describe how that data can be used. The end-users exist, there are authentication mechanisms and attribute databases set up around those. These could be LDAP, Active Directory, something like that. There's a diverse and growing set of applications. So we don't have a single application that we're really designing for, but it's a growing set. At our core, we've talked about the technology to create a filtering mechanism and a labeling storage mechanism that stores those labels alongside the data, but there are a couple of key elements here, the labeler and the policy engine, and this is a lot of what we're building in Squirrel to really manage the label sets around this. So the job of the labeler is really to take some provenance information and take some perhaps schema information around the data, maybe even derive from the content of the data, things like sensitive diagnoses in healthcare, and stick the appropriate label on the data according to that. But it has to do that in a way that's relatively immutable. So if I take data and if I label it with, if I label my electronic health record with my doctor's name and then I change doctors, do I have to go back and relabel my data? If we're storing petabytes of information, that relabeling, that's hard to manage. So effectively what we're doing here is we're separating this policy into immutable elements of policy and mutable elements of the policy. And that allows us to create a more manageable structure, which still has a smaller set of labels, but allows for that user attribute to change and the mapping of that policy and user attribute to the entitlements of that user to change dynamically at query time. So effectively that's what we have here. We have tools inside of Squirrel to do both of those elements of it. And really I think this is a critical part of big data and no SQL, anytime you have those types of diverse security requirements. Yeah, so can you bypass this by going to those lower layers of the stack? Very good question. We have encryption implemented inside of Accumulo so that even if you can read the underlying data from HDFS, you can't get access to it. And all of these mechanisms for restricting access and authentication, those are shared between Squirrel and Accumulo. So you cannot bypass these mechanisms. It's mandatory access control. Yes, and that's where actually this key management comes in. So we actually do things like separating the key from the cluster even. So we can do remote unwrapping of session keys and really provide very strong guarantees of the security. A big part of that is to push those security requirements, that security enforcement mechanism into the infrastructure itself. So you can trust that, you can build your accreditation just on that infrastructure so that even though this set of applications is continually growing, the mechanism where you verify that that application is performing the right filtering and performing the right restriction of data, that's a lot easier to verify in this type of setup. That's really the promise of data-centric security in general. It can be either. Typically what we do is a symmetric key that's wrapped by an asymmetric key and then that's stored for remote unwrapping. But there are other ways to set it up that makes sense. So we can change keys over time but that key has to be accessible to the entire key minlow cluster in order to get us recovery where if a tablet server goes down I have to migrate its tablets throughout the cluster, all of them need access to essentially that key. But we can change those keys over time and cycle them and it ties into the external key management and remote unwrapping of keys. That's effectively the keys that we use on a profile basis with kind of a one master key at a time. Okay, so let me jump off this security stuff and we'll try to get through some more of this table design elements here. All right, so table design. So we have key value pairs, we have to take those and sort of organize them into a way that makes sense in an application. So this is a diagram that's a hierarchical decomposition. And in this case I've just taken a particular set of data which is focused around people and attributes, purchases and returns of those people. And I've kind of created this hierarchy in order to show a denormalized view of that data. Here's how you would denormalize that. And as we do that we can take different layers in this hierarchy and we can assign them to the different elements of our key value pair. So here's a, as a row we'll throw a person in there and then the column family will have this set of specific elements. And if I take that and if I instantiate this, what's in fact a schema here, I plug in who the actual person is, I plug in the values associated with it and I get a forest here where if I take any traversal of a tree in this forest from root into leaf, I get a key value pair. So those are my key value pairs, that's what I stick into, a cumulo. By doing scans over it, I can get the entirety of everything under bill by a range scan and it's a very quick mechanism, all that stuff is stored locally, supports high concurrency, supports low latency, ties into that interfacing with the data-driven real-time system. As well as follow on types of bulk processing that are also supported by the denormalized data model. All right, so this is a very basic model. We've created several other design patterns on top of this and here's another one. This is a forward and inverted index model. So in this case, I take a UUID, just some identifier for a document and underneath it I can group things by type and field within the type and then term within the field. So effectively, I have a collection of field terms that are grouped under a single UUID, right? That's my forward index. If I wanted to query this, basically I can take anything from the prefix of the key and create a range that just gives me the information, just gives me the key value pairs associated with that prefix, right? So effectively, if I already know the UUID, I can find all of the field term combinations underneath that. If I don't know that, if in fact I want to query for anything where field foo is term bar, in that case I need an index in order to be able to do real-time lookups, low-latency lookups. So that's the inverted index. I take the term from here, I stick it up here, I put the UUID under that and then throw the type and field underneath that. The UID is essentially a remote reference to that forward index and now I have a basic indexing technique. And this is the core indexing technique that we use for a number of customized indexes on top of it inside of Accumula. Yeah, so term in this case could be just text. In fact, this could be a string where we've extracted a bunch of terms from a value that supports full text indexing. Absolutely, right? So in fact it's not a single design pattern that we're gonna use and Squirrel is actually a collection of these design patterns where if you were building an application you might create a bunch of design patterns and stick those in a bunch of Accumula tables and use those to satisfy the needs of your application. We've done that with a bunch and we've exposed them in that higher level API inside of Squirrel. So that's what you get out of it. Yep, so storm is a stream processing engine, again on the data driven side. Storm would be used to maybe reach back into the query mechanisms that we provide here. So it's a compliment to Squirrel. All right, so let me move on here. I wanna get to the interesting stuff which is the performance results. That's what everybody's waiting for, right? Okay, all right, so here's what that looks like from another perspective. I have an event, I map that to a single place in the forward index or the event table as it's called here and then I generate a bunch of index entries. In order to support the real time, effectively, I'm just generating a bunch of key value pairs as I ingest this and I'm throwing those key value pairs at the database and letting it manage them. On the query side, then I can take a particular term, look up the one place in the inverted index that has all of the references to that term and potentially go back to fetch the documents associated with those UUIDs as I get them out. And this is actually one of the places where storing a digest of the event itself inside of the index, there we're denormalizing a subset of the information associated with that event so that instead of having to do that secondary lookup all over the place in the event table, we can just get all of the information we need from that one place in the inverted index. So it's a pretty powerful design pattern there. We can extend that, right? We can do a bunch of custom indexing on top of it. So here's a case where we'll take a geohash. So maybe we have latitude, longitude, and depth. We interleave those coordinates to create a geohash. That thing becomes a one-dimensional coordinate inside of this three-dimensional space. Let me stick that in there. Do range searches across it. Again, that's a very efficient indexing technique. One of the other things that's pretty natural in this space is graphs. So the next few slides are gonna be discussing a thing called this D4M 2.0 schema. So this is some work done by MIT Lincoln Labs. And effectively, it's taking a chemo and treating it as a graph. So we have something like a t-edge table here. This one is applied to Twitter data. So we're gonna have edges that take the idea of a tweet, gets stat information, time information, user information, word information, and link these together. So you can think of these as RDF triples. And in fact, we store the transpose of that as well. The Lincoln Lab guys like to talk about this as matrices, sparse matrices. Those and graphs, very similar concepts. One of the things that they've done inside of the D4M 2.0 schema is to create an edge degree table. And in this case, this is one of those tables that has the aggregation built into it. So if we take a particular value inside of a stat and store the degree, there we can see how many times did we see that stat across, that particular stat across all records in there. So that gives us some statistical capabilities. It's a count table. It's basically used for query optimization. And then storing the text of the tweet itself. That's also a mechanism that we use here. So another view of that just from a matrix perspective. We've got that edge table up here as a sparse matrix. We store that, we store the transform of that table. That gives us forward and backward edge traversal. And then those other two tables I mentioned there too. Okay, so that's a small collection of table designs. We've gone through a huge number of them in the past. But that's the type of technique you use when you're building an application on top of a cumulative. So let me just jump into some performance benchmarks here. This one is one done with that D4M 2.0 schema as applied to the graph 500 benchmark. Have you guys ever heard of the graph 500? So it's effectively a benchmarking suite around graph analysis where you create a large graph, insert that and then do queries on top of it. So this is just an ingest performance benchmark that they did on an eight node cluster with 192 cores. So what they did was they varied the number of processes, basically the ingest processes that were ingesting into that fixed accumulo size and saw that they were able to effectively get up to about four million entries per second on an eight node cluster. So that type of performance is immensely better than published results of other databases like Cassandra and HBase. This is into accumulo hitting disk. It's not just limited to memory. That's sustainable over long periods of time. Okay, another benchmark that's done in terms of scalability, NSA recently gave a talk at CMU where they discussed using accumulo as a large graph store. So in this case, they stored, again using the graph 500 benchmark, a 70 trillion edge graph, which is about a petabyte in size on a 1200 node cluster. And you can see this is actually just a scalability of traversed edges per hour in this case. But 70 trillion edges scales up linearly from these smaller graphs. And they were able to do things like componentization of just enormous graphs using this type of technique, storing graphs inside of accumulo. Okay, and then I just have one more benchmark here, which is this is the difference between read modify write strategy and iterators or combiners using accumulo. So let me just talk about this read modify write strategy in HBase for a second here. So this particular benchmark is just looking at a single node instance of HBase versus accumulo. HBase has a highly optimized read modify write strategy, which is what they call an upsert, where if an element is already in memory, it'll look at that, change it and write it back to the exact same spot in memory while holding a lock. So doing that, they're able to get thousands of upserts per second on a single node, which isn't bad, but when you compare that to the increment rate that you can get without doing that read modify write strategy, batching things up, sorting them together, that's, you know, it's orders of magnitude difference. So avoiding read modify write, do it at all cost if you're trying to get real time. Another thing that you see here is the curve falling off here. This is where things fall out of memory, no longer fits in the right cache inside of HBase. Acumulo had the same size right cache, but regardless of whether we're hitting disk or in memory, that's essentially, you get constant performance across that, so. So there you have it. I'll take any questions from there, and I think we're a little over time, so we'll just take a couple, go ahead. Yep, so effectively we're able to introduce encryption with only kind of single percentage hits to throughput in there, and we align it with all of the organization of data that's already there with the file formats that are already there, so it's pretty small. With Squirrel, there's additional organization ripping apart documents into key value pairs, that type of stuff. There is some overhead in that, but again, it's fairly linear, and we preserve really the scalability, we preserve a very large portion of the performance with that as well. Yeah, so since we've done the encryption, the question is basically, since we're encrypting, does that limit how much we can do in the analysis space? And this is a key differentiator between us and other solutions. We rely on that fine-grained access control mechanism to really provide the restriction, and encryption essentially for us, it narrows down the trust boundary, so that a cumulo can still read all the key value pairs. It's not encryption outside of a cumulo, excuse me. So what that gives us is, as long as we are using a cumulo key value pairs to store our data and to store our indexes, we can distribute those types of index lookups and a lot of the analysis to the cumulo cluster and allow it to do those types of analytical operations while still preserving quite a bit of security. All right, well, thanks a lot, folks. I'll be sticking around if you wanna ask more questions.