 Okay. Okay, let's start. My name is Thomas. I work at SAP like, like my two predecessors and I'm going to talk about MetaSpace. I've been working with GBMs for about 15 years at SAP and I've worked on and off with MetaSpace in the last maybe four years. I've contributed a number of patches and I thought it's actually a nice topic for a talk. Some legalese. Okay. Why do we even care? The thing is MetaSpace is kind of a bit of a dusty and forgotten corner of the VM. So there are more exciting topics usually, but the thing is MetaSpace can actually contribute quite a bit to memory consumption, to non-Yavaheeb memory consumption within the JVM. So it pays to actually look from time to time at it and to look whether we can improve things and the answer is yes, we can improve things within a reason. Some basics first. MetaSpace is a region within the, it's a memory region within the hotspot VM which contains class metadata. Very shortly, class metadata is basically everything contained within the class file sucked up and digested by the VM and rolled out into runtime for easy consumption by the VM and things like constant pooling methods and notations but also purely runtime stuff like JIT profile account and so on. Really enough not for instance a code cache but which would be kind of a candidate for MetaSpace but it isn't that. So all this stuff used to live in the Java heap like in ancient times in something called the permanent generation and this had a number of disadvantages. One thing is that it made, it was kind of difficult for users to configure correctly. So it was difficult to size and because it has an inbuilt limit I mean it couldn't grow beyond the size of the permanent generation. And another thing is that it was difficult to work with and hotspot engineers had problems with it because it used to, it used to move around and because GC gets compacted, it's not at a stable so and then it was decided at some point in time to move all that stuff to native memory and this was inspired by J rocket VM which had been acquired some time before by Oracle and so this was the JEP 122 and remove permanent generation and so that was the birth of the MetaSpace and we have been involved since quite a while in MetaSpace development mainly because SAP has customers which tend to have large installation large more lethic installation which do a lot of class loading and unloading and so we had all the sweet spots of MetaSpace and had a lot of issues in the beginning especially with JDK8 and so we contributed quite a bit of work to the MetaSpace and improved Metas especially in JDK11 and also provide a number of analysis tools and all those were culminated in a way with JDK15 in a rewrite we did which has not been published yet but I will talk about this presently okay to understand MetaSpace it's difficult to understand the life cycle metadata are located when class are loaded which is not really that surprising and exciting more important metadata are tight lives from a lifetime are tied to the loading class loader so all metadata are get released when the class loader is collected which means that we actually have a byte delete scenario which makes things a lot simpler there are some exceptions to those rules for instance class redefinition and you have to throw away all bytecode and stuff but that's the uncommon path that's not what MetaSpace is designed for so which brings us to the question why actually both are writing a custom that allocator I mean can we cut use like a general purpose allocator like Maloch or so and the basic answer is because we think we can do better and because we have byte delete scenario we can get away with arena style allocation which is actually pretty fast and very memory efficient and we do not have to track individual allocations for the purpose of freeing them so a general purpose allocator is actually not the bad fit because it does a lot of work which we don't have to do and especially in the case of Maloch and there are different cases against Maloch and one thing is that we couldn't do compressed class space I will come shortly to this and we also have platform specific limitations with Maloch so compressed class space I see this in block entries and so on I see this often confused with MetaSpace it's not the same compressed class space is actually only a part of MetaSpace and not even a very large one at that and it's an optimization we do for 64 bit platforms and only exists there and can also be switched off the story is like this so we have a Java object which lives in heap and it has a header part of this header is a pointer which points to a class in MetaSpace which in a structure in MetaSpace a native structure which is called class really written with k and contains the central metadata for this class and and stuff like V table I table so it's also um it's also variable sized and on 64 bit this pointer is 8 byte which is kind of annoying because it uses up space so we reduce the size to 32 bit which says pair Java object 8 4 bytes which is actually good and can save Java heap in about 5 to 8 percent depending on how many Java objects and and what small granularity you have so the 32 bit index is actually an index into a pre-arranged memory range which contains which of course has to contain now all the class structures and also may contain the shared classes and the if cds is enabled so um this the way the class the little class point has calculated this would give us in theory a range of 32 gigabyte but the compressed class space size is actually limited to 3 gigabyte max artificially this brings us to the fact that MetaSpace consists of two parts the one thing is the compressed class space which has to be of course reserved at any random VM comes up and into the compressed class space go all the class structures and the compressed class space is committed on demand and because we have to reserve it when the VM come up we need of course to know its size so so and this is the compressed class space size switch which by default is one gigabyte um this is only reserved space so so I will not explain the difference between reserved and committed because um because I assume you guys know this I mean on most modern platforms reserved size doesn't really cost that much or anything at all actually so depending on how the page table is implemented so and and so with one gigabyte and in general each each class structure is an average 1k size and this is a this is a good size so okay and so in general with one gigabyte you have clear you have space for about one million classes which is enough usually so everything else so everything which is not a class structure goes into the non-class MetaSpace which is a chain of memory mappings which grows on demand is is committed on demand and so on and note that this is a bit of a misnomer the compressed class space is neither compressed nor is it really the sole container for class data and non-class MetaSpace is also not really non-class but it is what it is current implementation and I will speed a bit through this um very much simplified as we have said we have a byte delete scenario every loader has a chunk of memory which can vary in size between 1k 64k and so on and usually also much larger and we allocate from this with very cheap variation pointer bump and pointer bump so one someone wants to have a constant pool from us for 15 bytes we point 15 bytes is not much so let's say 100 bytes so we we bump the top pointer by 100 bytes so and then we accumulate a series of chunks which are full or almost full and they are can get a chain to so-called retained chunks retire chunks and this means that we have when we have several loaders each loader accumulates a series of chunks and these chunks are carved out of the MetaSpace in the order they are allocated usually and which means that the MetaSpace fills this chunks which are which belong to different loaders which potentially different lifetimes which is what I call actually MetaSpace documentation and note also that that if you think you only have one loader and this is usually not correct because the VM also creates like tiny loaders for reflection glue code or for things like in the classes for lambdas and they get all over the way and so when a loader dies and is collected its chunks are marked as free and added to a free list and and they may be reused in the future if if if you resume class loading if you never do this is wasted memory basically basically wasted memory this is very much simplified the the whole design is way more intricate but this is like the gist of it the problems the current implementation is is the following and free list can get really huge we have seen a customer slide used to free ratios of one two three which means like we have an off of I don't know an only a quarter of the of the memory allocated actually for for MetaSpace is actually used for MetaSpace this is about actually worse and JDK8 it has to be somewhat in a mitigator with JDK11 which is by the way a good reason to upgrade but it's not it's far from perfect so another thing is intra chunk waste which is can make up between two and actually 15 percent which is also kind of annoying basically if you have a loader a typically every loader at some point in time stops loading classes and the remainder of its current trunk is wasted and the the the problem gets worse the smaller grain the the loaders are actually on so this is an example for the first problem the the we are not really elastic at all so I mean I mean we we if we have a spike in a MetaSpace usage we do not really recover from it so um as we can see we load a bunch of classes and then we release them in batches of four there's an example and for batches and as you can see the the first the the committed space which is the blue line doesn't really budge at all until maybe four fifths of all the loaders are really released it's really bad like in this case we have like retained in three quarter of of of the memory is actually in free list and and idles around and you can see this for instance with our command we added and and Retta also had was j command via MetaSpace which um weirdly enough is not really that well known as they are even not at Oracle which is kind of weird and we contributed this command for JK11 which shows a lot of fine-grained details about the MetaSpace if you want to look at MetaSpace details this is the thing to do um shows basically class loader um class loader's classes the whole occupation your trunk geometry and everything and it has a nice waste section as you can see here you can see at one glance where waste of course so as a Java programmer you usually kind of do something and it's basically you can stare helplessly at these numbers but for hotspot engineers it's good because we know where to optimize okay so um last year we were thinking about this and I was kind of unhappy with the state of affairs I was kind of thinking about about re-implementing this I yeah okay I was thinking about re-implementing this and so I did this as a kind of exercise coding exercise and the basic idea is that we try to make MetaSpace way more elastic by uncommitting the trunks in the free list I mean the thing is we cannot do anything about fragmentation we cannot we cannot move this stuff it has to be at a stable so we cannot compact however what we can do is when the trunks are on free list have a certain size we can basically decommit the memory below the trunks which is especially good when you have large continuous free ranges and a spin-off of this idea is actually that we can delay committing trunks until they are really used I mean I mean that's that has a lot of that helps with intra-trunk waste actually and so I won't go into details because time is running away the thing is one concern one main concern with decommitting memory is you do not want to do this in a too fine granular fashion because this increases the number of virtual memory areas which for instance in the case of linux means that we performance of kernel memory management may suffer because VMA lookup gets slower and so on and we also may certain hit certain process limits so this is something to keep an eye on so we introduce something called commit granules so where today we have basically a contiguous area of committed stuff we divide the memory range into into commit granules and to so-called commit granules of adjustable sizes it was amazing mainly to to play around the sizes yes so and to to be open for for whatever comes we also had to we had to improve the chunk allocation scheme it it's currently really unused and unsuitable for for for decommitting so first thing I did was replacing the current body allocation the current allocating allocation scheme with with with 1k 4k 64k chunk sizes with power 2 based body allocation scheme which is basically a very old venerable algorithm like 70 years old and and the cool thing is that it's really very easy and very cheap to split and merge chunks it's it's an old algorithm everyone knows this and it has a very low externality for a fragmentation so so you can really you arrive at at you have a high chance of at arriving at contiguous free areas which then are good targets for uncommitting and it also doesn't hurt that it's a it's a good standard algorithm everyone knows it so it's good for for for maintenance and the allocation and works like this you someone gives back a chunk and releases a chunk and through the allocator and we merge it with their bodies and with their bodies recursively until you either arrive at the largest possible chunk size or you cannot merge anymore and this works actually quite well so one thing was that chunks had headers and headers needed to go the reason for that is if you have a header you cannot decommit the page where the headers in which it does not only cost memory but also even more importantly increases fragmentation because you have like a fence page like at every chunk which is kind of annoying so we remove the trunk headers and put them somewhere else and now the whole chunk is payload can completely decommit it in one go and even more importantly you have like swathes of chunks if you have like swathes of free chunks you can decommit them all and basically at the operating system level you only have like one contiguous free virtual memory area struct so okay so these concepts work very well together and the funny thing is they are kind of independent so there are not many chunks and granules there are not many dependencies so we can actually change one implementation or the other and do not affect the whole thing at all so I think the design is quite robust so as you can see like a one chunk can spend multiple granules and multiple granules, multiple chunks can be contained within one granule the only condition we have is that every all the sizes are power two sizes and it works like this so we have if someone frees a chunk and we fuse it with the neighboring chunks and reach a size which is one commit granule or more we can actually decommit the underlying memory and this is good so we actually return this memory to the operating system similarly when we have a large chunk which covers multiple commit granules we can actually give this chunk completely uncommitted or only partly committed over to a loader and then he can commit it as he goes so if he does power to bump allocation and if the pointer the top pointer moves into a new commit granule that new commit granule gets committed only when needed I think let's speak through this so we have seen this before this was bad and this is you know the result basically this is really really elastic yeah same same example patched VM as we can see we do not retain any memory at all this is also a bit tunable if we want to retain memory for any way for any reason we could do but I do not see actually a reason why we would want to and the chance to the it's it's way better to give this memory back to the operating system to be reused somewhere else or maybe somewhere else in within our own process and instead of only ear marking it for this one case maybe someone does class loading there's a difference between the stock VM patched VM jdk 14 dark low stock like to patched as we can see we save about 200 megabyte 250 megabytes here nice is also that the spike that the spike is not as high so so we also save even if with no class unloading involved we save a little bit this is because we have done some optimizations that the code is just better yes I mean I mean we have we have tighter memory packing and stuff rss yeah I think it's the battery okay you guys hear me yeah okay and this is rss just to just to show that I don't lie and I don't have hidden somewhere memory okay so this is outside bottom line the result of our work at the moment the work is not finished but basically meta space is now elastic which is a good thing for large monolithic applications which do a lot of class unloading and we also see a modest decrease in consumption as I had shown before even if without mass class unloading between one and maybe 10 percent and these are all examples which are very very simple and which don't do much at all and this is still ongoing work we still have a lot of ideas to to to follow through let's see how it goes I also think that the coding is overall way better I mean I mean it's way better maintain a long term I think so um how do we go patch is stable I think it it needs a little bit more tdc but I think it's okay um it lives in a jdk sandbox directory um repository and everyone can build it and and play around with it and I did made a jeff with it currently in draft state and called elastic meta space I'm not really sure but this is actually the right we hike it for this kind of work but but I did it it's there so I would really love very much to get this into jdk 15 and the problem is that it's quite difficult to bring such a large patch upstream into the open gdk if you don't work at oratory but but let's see I will try I will try and finally yeah finally it's a good candidate actually for backpotting I think because because um honestly I mean meta space role is somewhat diminished in in modern modern VMs because we have cds and stuff like that but like in older VMs like jdk 8 and jdk 11 which are actually mostly used like in large legacy scenarios where you don't have micro kernels microservices and stuff but but large installations I think that would be very good and especially in jdk 8 it would um make a lot of sense so of course I'm at the mercy of the maintainers so let's see okay thank you and any follow-up questions I'm reachable and in various means and yeah we've got a couple of minutes it's going to be a bit awkward with only one microphone you'll have to repeat the questions but does anybody have anything to ask well you've won oh that's a question will there be like a pause time cost for doing this uncommitting I know we've seen that when we're uncommitted memory I sorry I don't understand I guess the uncommit is done in the pause and I don't think that much we didn't see any differences I did like like very micro