 All right, so I think we can get started. So welcome everyone to the DEF CONF and to the workshop room in case you just joined us. And let's hear from Yergy about inspecting and modifying Java classes in running JVM. And in case you'd like to join with your audio and video, please click the Share Audio and Video button at the top right corner or post your questions into the chat. All right, Yergy, take it away. Yep, thank you very much. Hello, good people. My name is Yergy Vaniek and I am a member of the Red Hat Open GDK team. And I have a presentation for you about looking into the running JVM, checking what it's running inside for bytecode and even how to modify it in that running VM. Now, all this workshop would be about this Java Runtime the compiler tool, which you can download from the links. You have the slides in the presentation. They are all the links. You will need them because if you want to participate on the workshop, you have to download the JRD in that 6.1, which is the latest stable version released on the GitHub. Or you can install it from the federal repositories. So please, while I am jibbering, feel free to download the tarball and unpack somewhere you will need it. So JRD, Java Runtime, the compiler is tool which started to be developed about five years ago. In that time, I was working on the runtime for the Java applications. And very often, I got to produce a proprietary blob. Very often, I found that my runtime is good, but the blob is wrong. And usually, what I needed was to change one or two lines in their code, but I didn't have the sources. And I usually didn't have even the complete binary blob. So that was quite a hacking. And in that moment, the idea of Java Runtime, the compiler, actually rized up. And it grown a bit over the years to really tool which can do hotpatching for you if we are running VM, which is a little bit broken and you have a hotpatch but you do not want to shut it down. You can use that. It's here to investigate what other tools have actually touched your code because you, yes, you developed source codes, yes, you built it, but sometimes there are other tools later in the chain, which you maybe do not even know they are here. So instrumentation can be here. And if your code misbehaves, again, the JRD can look inside and see what's at the end. And of course, it's also for the curious people who just want to see what happened and for other tools like, for example, biteman does to date code base. And for testers, that is a useful use case that you can use JRD to reach unreachable code, for example, to inject some exception somewhere or only to modify some field to misbehave also. JRD have two parts. Agent, which is injected into running VM and the client, which is both graphical user and the command line. All this talk is, let's say, about the agent because the agent is the heartbeat of any inspection and modification of the classes in the running JVM. Now, agents, there are two types of them. Native, which are based on the JVMTI API and Java once. This presentation would be purely about the Java agents only because in the same way as JVMTI agents are much more powerful, they are much more hard to use and you usually end with the broken JVM if you are using them too visually. So agents can be injected into the fresh JVM. That means that when you are starting your application, you added a Java agent parameter and then your application is started with the Java agent where the Java agent transfers. That's mean that this is very useful, for example, for the code coverage. And the agent is so simple thing that it have just agent-mine method, which is called first. Well, and then you need to do your dark magic. In addition, you can inject the agents to the running JVM. No matter how dark magic is, beanie is all that, the API is super simple. You will just, your application, please give me all running JVMs and please inject the agent into that one. That's all. For example, integrate development environments are using that to allow you to quickly replace classes in the program you are debugging. And so the fun parts of the instrumentation API are these two files. Yup, okay, wrong window. Let's check them because they are very, very simple. Here we go. The instrumentation API from the point of view which I'm interested in can give you all loaded classes in the JVM and can allow you to inject the transformer where the transformer is where all the fun is. The transformer API is really super simple. You receive the bytecode of the loaded class and you return a new bytecode. So it's super simple with the only question mark how to get the new bytecode. It's important to understand that agents can chain. You can have unlimited number of the agents in the running VMs. So sometimes you are pretty wondering what is the resulting bytecode at the end. Then you have your original bytecode which comes from the class pass. So the static class is written in the disk. It's loaded by the JVM. Then all the incapable agents upload and the incapable agents actually are not allowed to change the bytecode. So this is tricky but they have their part. And after this, you have the saved bytecode. The saved bytecode is the origin of the bytecode. You will be running when all the agents are processed. So you take your saved bytecode and first capable agent, up to all these transformation. Then another capable agent, again up to all these transformation and so on. And at the end of this process, you have the final bytecode you are running. It's really important to understand that there is no pernament change. If you change the code by the agent and your agent somehow stop providing the change, sooner or later the bytecode will reset back to the saved bytecode or to the, your agent will not be applied. Now I have to confess in the JRD6, we will be using in the examples. There is very severe bug, which is actually exactly making this wrong. So if you will, because it was originally the compiler and we didn't change it soon enough to become a compiler. And the agent, every time the agent is making anything, it's actually injected again. I will be warning you about that in the runtime and this is fixed in the JRD7 but JRD7 is far from being released. Okay, hotswap of the classes. If you are, if you receive the original bytecode and your agent is transforming it, good. You return your new bytecode, but the changes may not necessarily need to propagate. The JVM needs, let's say some motivation to take that effect. Usually calling the, creating the new instance or even the calling the change method is enough even for just in time compiler to drop all his native work and get it again. The hotswap have severe limitation in the scope of the JVM you are running in. Generally saying only the method bodies can be changed and other changes will be revoked by the JVM. There is fork of open JDK called DCEVM, which is, oh, I've just forgot what that's mean. Code evolution, dynamic code evolution of VM. It have quite a few patches on the top of the open JDK which are allowing nearly unlimited hotswap. It's super cool for development. You will really make quick rotations of your code but it's unstable and usually very delayed behind all other releases. The patches are not somehow super hard but have severe performance impact and are very, very large. So they are very hard to adapt when open JDK is evolving. As for the tools which are touching the bytecode in somehow, okay, I have JVM and I want to change the bytecode, such tools are pretty rare. I am aware only about the byteman and JRD. Bitman is pretty old, was developed at the part of the JVOS project and it's based on rules. Let's check that, yeah. This is an example of the byteman rule. So you are saying a rule which has some name in class, date printer, in the method print date, do something in this case, add another trace line. So byteman is pretty safe but you need to know the byteman language and it's not simple. This was really simple example. If you would look much deeper, then the byteman rules are pretty complex but it's safe and stable and safe in the meaning that it will not allow you to break JVM at least not easily. JRD on the other hand have no rules but you must compile. Depends what your source code is if you will be really using the freshly the compiled source code from the byte code obtained from the running JVM. No matter how hard we tried, the compiler sucks. We tried a lot, we make quite a lot of fixes for them but usually the code is not compilable back out of the way. JRD also must resolve the dependencies but that is done under the hood and you can, for example, change the hash of the Java object class and in that case you will simply kill the JVM. The class pass less compiler was created in the last year as the part of the JRD is the heartbeat of the thing which allows us to take the byte code from running VM and use it for the compilation. That library is pretty interesting and if your project need compilation that this is pretty nice layer over the Java C API where our API for the class pass less compiler is pretty simple. It really have just compile me the class where you have some class provider, the second API and then you simply provide the bytes, the byte arrays and the class provider is again simple. It is providing just the list of the byte code. Okay, hands on will be since now forever. I hope you will be able to download the tar ball to extract it and to run that. And for the later examples, we will need also this tar ball to unpack somewhere because there are other examples which we will running and we will be modifying them. As for the warm up, I will show you how JRD actually can somehow affect itself in the runtime. So let's start the JRD. Okay, this is our global Java in the compiler. There is the list of the running JVM in this, in my case, there is only the JRD running. Just a question. Is there somebody who is already running JRD right now and trying what I'm trying? Anyway, if yes, and if you need to stop and return, don't hesitate to ask, don't hesitate to ask for the audio. So we can really workshop and discussion rather than me just speaking about that. Okay, back to JRD. So list of running JVMs and list of all the classes in the JVMs. For example, just class JSON, which is pretty intense. So the compiler took while and yeah, this is what freshly decompiled source code of the JSON dependency of JRD. There is several of the compilers. We are using CFR is far best. Fan flower is from the JEDA and it's probably the only one maintained. Procurement is pretty experimental, but can do wild things. And we have also this assembler, that's just some in jcoder, where the JSON is surprisingly readable and useful and really the annotation in that works and back compilation, the back assembly of that is working. Okay, so the task in the JRD would be this class, JRD renderer, which is one of the few classes which is not completely dummy and can be decompiled and compiled. Now first, JSON. So there is some magic field pit somewhere in that. Okay, which is actually here in the left column in the local processes. So if I change it to some number and I will try to back compile and upload, compile succeeded and the new class correctly uploaded to the VM. So now you will see that there is no longer pit in the left column, but there is already some number. So this assembly of the code and back assembly worked perfectly. But yes, you usually do not want to, but yeah, usually you do not want to work with the assembly language. So let's check the CFR. And this is now the same byte code, the JRD looked back to the JVM and took the byte code. So you can see that there is already some number instead of the original PID and now it will be some word, word, strange number. So how the compilation will go. Then compiling, compilation, have some issues. And that's bad. It seems that the assembler did something enough. And now the decompiler class can't actually be decompiled correctly. But I can easily remove my overrides. I remove what I have done. Okay. Now you see that there is again the PID in the left column and there is PID also in the source code. So let's see that now. Again some number, let's try to compile that. Compilation finished correctly, but the remote JVM refused to accept the byte code. This is, as I was saying, the limitations of the hotswap are there. In this case, the definition failed because I have added nest host and nest member attributes. I'm not sure if it's readable in the shared screen, hopefully as, or maybe you have encountered it on your own. Okay. Does anybody know what is nest host and nest member attribute? I guess it's very hard to reply through the chat without the audio and video. Anyway, the nest host and nest members where new byte code constructs are that in the JDK, nine. So what I need to do, the only thing is to change the compiler arguments to compile for eight, as you do on the command line. Now the compilation and approach should work. Compiling, compiled, and new classes including the inner class because it was also the output of this compilation are correctly replaced. And if we make some motivation, some number is here. So you can see that really JRD is affecting the byte code of the underlying classes. Well, let's say correctly. Yes, there is a question in the chat. Where to find the log console? Oh, if you click the compile, it will pop up immediately. If not, then there is the help and it's the log. And why I do not see the question here because I'm in some wrong window. Oh, Anna, sorry, I was in the wrong chat window, sorry for that. So yeah, hopefully it's working for you. Okay, so that was tried, so it's in target eight, yeah. Okay, any questions until now about the agents, about the hotswap, about the JRD? Okay, assuming no. Let's try some more fun examples of what you can use JRD for and how it works. I have some example here. And, oh, that was that, sorry. If you can download that CD-STAR-XZ and unpack it, then you will be able to run all these examples immediately. I swear there is no malicious code in that. It comes, I think I should have it unpacked somewhere on my machine, yeah, the CD-STAR. Okay, now, I have started some process. It's running, it's doing something and it's not working correctly. First thing which you may notice in such thing, in such output of some blob is that there is unknown source actually in the places where you would like to see the source lines of the code. So again, this is the complete start of debugging of any blob. The JRD can help here. There is a new process appeared and there is egnetworkalk on the top which is exactly what we can see here, egnetworkalk, yeah. And, yeah, so we can decompile the source, it's look readable, we can try to compile that. It failed, unreachable statement, okay, that's a bug in CFIA, the compiler, that this break need to be removed and should work. Now compilation passed and only by the fact that I will compile it back freshly, I will add the debug info because the debug info is by default on by the Java C and that jar was compiled by somebody who didn't want us to have any debug info and he was evil, he didn't want us to allow us to debug his blob. So simply by compiling and uploading, I will suddenly have all the debug info here. Okay, let's see that, okay now. And now there is, there are the, there is the most necessary calc73 and even better is that it's well aligned to our current form of code. So 73, that will be going here and I guess most of the Java guys around had guessed that there is no such method exception than usually the get declared methods and score is here. Okay, now we know what is causing that, we know that it's calling some in mouth method and I guess, mood method is really not here but there is move method. So the super simple thing, super naive is if the parameter, this is the first here which is used to bear the name of the method, mood. Then we can set it to the correct form and be happy with whatever the malicious program left for us attempt to compile works good but upload it, upload it in version. And now it should multiply. Let's wait, oh here it is, multiplying works but obviously there is some other error in that code base. Now, if there'd be workshop in a room, I would be walking between each of you and was watching if you are doing something if you can fix this second failure on your own. But as I do not see your monitors and I cannot walk between you, I'm not sure if you are and I cannot know if you are working on that or not. So if there's somebody online who really is trying that and who wish to try to fix the second exception, the division by zero or should I just proceed? I guess that everybody's really busy fixing the division by zero, nevermind. Yeah, the fixing of the division by zero would be completely the same as we've done in this super dummy way because if you look to the division zero, that is really just simple division by zero, yeah. So just adding if here would do all the magic. Now going to more funny things anyway, I will kill that and see what my slides I suggest things. Okay, we fixed that issue, we fixed the multiplication and anybody could fix the division by zero. Now, another way how we can interfere with the code would be another agent. This would really fight, let me off topic example because I will use biteman to do the fixes. And I will be copying my old biteman rules because I'm definitely not fluent in the biteman. So hopefully everything around will work. Okay, let's start the math again and it's broken, yep. Now I can use JRD to list all the JVMs. So that is process of our math and I can install the biteman agent into the, oh, I need to kill the JRD because I will need to attach the JRD agent after the biteman agent. Okay, so I will install the agent into the process 267. Now biteman have an agent here. Now I will need to inject the rules which are fixing that and I will really copy my old, this is a date from the rules because really to be fluent in biteman is not simple. So this is the rules which I wrote long ago to actually fix these methods and I will simply copy that all but well that should be the most fun. I already have some rule what I have here. Do, do, do, do, do, okay. Oh, that's it. Okay, seems like I was practicing yesterday. So yes, this I have the file with the rules and let's now biteman submit the rules. To that agent we just installed and we can see that biteman is saying that it really installed all these rules correctly. Now, what will JRD say about that? Again, the JRD process and the eternal crashes process let's investigate what biteman does because that's one of the use cases of JRD to investigate what's happening around. Okay, so you see that there are some new imports already, some biteman stuff and if you will look into that we will find some nasty code around that would know human road and yes, there is a biteman rule exception, biteman rule execution and another biteman rule execution and another biteman rule execution. So JRD helped us to investigate what other agent did. Let's check if the code was really thick but I believe yes, yeah, instead of division of zero there is something nice and it's multiplying correctly. Okay, this was an example of how JRD can inspect the code which other agent modified. Now, I have one more use case like this. Again, I missed walking between you and showing you but maybe we will leave it later to somewhere. Second example, again, now I will be using for the back info so there should be no issue with the missing exceptions but we will see another issue here, let's start that. Oh, I forgot an important thing. Since JDK9, you have to allow JVN to attach the agents otherwise any attacker on localhost will be able to do all the dark stuff I do. So it really depends on the admin if they consider agents as friendly or as an enemy tools. Okay, now this program is saying that it should be printing the current time and date every second. However, a freshly fired programmer was obviously doing this as his last job and something went wild and he eaten that. What JRD is saying about that? Ex-date printer, it's here, good. And we see the method that is pretty simple. There is again that back from the CFR and we can really see that there is consume exception. Cool. So instead of that, we can add the print stack trace. Will it compile? It will compile. Will it upload? It have uploaded. Good. Have it fixed anything? No. Anybody, any idea why? As I was saying, JVN needs motivation to change the byte code it's running. It's because of the JIT. And if we will look to the code, then we have changed something in the eternal loop. And no, JVN will not change JIT loop just because somebody say so. So to do that, we need to go a little bit more properly. Let's say this method is very suspicious. That is obviously intentional bug. So let's try it here. Let's try to compile that and upload it. Oh, I have reached end of file. Something, some exceptions, some bracket. Okay, that should be better. Compiled, uploaded, good. Now it really should fix something and we have that exception in the place we were expecting that. So again, the fix would be super simple and super obvious simply using the correct D1 here. Uploading, fixing, done, outpatching, done. Date is printed. Okay, any questions on that? Anybody is able to try that or able to trying that? Okay, click our example, this one example. Now, there will be one more case in the same date. Okay, we already seen biteman in action that's not going to repeat, but I will show you another processor of the code and what they can do. So I will run the obfuscated build of the same example. It should be somewhere, no debug obfuscated. That usually that people like to publish. Okay, again, same exception, same issue. Let's look to the JRD. What's wrong here? Let's check that turn crushes. So first you can see that instead of the nice name, it's a rename. That's the first thing which any obfuscator does. Unluckily for us, we can immediately see that the obfuscator did some kind of optimization to our code when he was obfuscating that. And it removed all the methods and it inline it here. Where it may seem good in our case, it is prohibiting us to fix it because if you would try to add the method, then it will just kick us out. I may just try that to move it somewhere to the method to turn and already this would go wrong. Unreachable statement, sorry, that's the break. Again, up compiled, upload failed. And upload failed because of that, we have again tried to unsupported operation exception, class, the division failed, added a method. So no way here, an obfuscator actually prohibits us hot-patching. Where it would be even more terrible that if you would have really proper source and you would be hot-patching from the proper source, you would suddenly see that you are running something completely different. So yeah, it's sometimes tricky to release proper binaries but okay, the last example I have, okay, any questions on this? Any questions or any topic to discussion who is already working for you is are examples working for you? I sometimes somehow feel all that looks like which are saying no. Okay, the last example, that should be a memory leak. A memory leak on the binary blob is probably the most terrible thing you might encounter. And there will be a little bit more fun in that. Okay, so I'm running it from the series from the through the bug build and running the example row which will misbehave. This is printing just a really huge amount of the numbers and the print of these numbers is really wrongly implemented. I should probably shoot over the XMX issues but still alive. Oh no, Java heap space is already running out but JVM can survive that. So let's check if I can see something with JVMs. Let's see what is causing that and that would be the JMAP command on the process that then should print as the usage cool. Now, from this view, it should be pretty obvious what is wrong, all that it is. The JMAP histogram is writing as the usage of the classes and we can see that three map of size of 32 megabytes is wrong. Definitely, you don't be there. And already seeing three map here is usually triggering a bell because three map is using weak pointers. And that means that they can really be easily, memory leaked. So looking to the, let's look to the JRD what they can see. And oh, it's no longer here because the process crashed. It's correct. That's what this example should do, okay? Out of memory heap space. Now I have two way to fix that. First is quickly because I already know that it's a three map. I can fix it here but yeah, all that compilation fails and so let's try to be quick. Dun, dun, dun, division road. And yeah, the simple if changing all the three maps to the hash maps is pretty thing to do when it work. Error to upload it, no, no, correctly compilation passed and upload it, good, I was quick enough. And now the program should work forever because we have already fixed it. Now that will be interesting. Watch into that via the jmap again. That would be this PID. Okay, we can see that our hash map is still pretty big but it's not so enormously big. And also we can see that the old three map is still there but that's at least the number is not growing anymore. Okay, you could see that hunting crashing VM in the running VM is a little bit nonsense. Luckily for us, the JRD have one more solution for that and that is that it can work also with the local VMs and the local class paths in the completely same way as with the running VMs. So that would be, I can kill that. What I was running, I was running this, this jar. Okay, it's there, that's correct one. So we can add the static class path thing here. Added it here, it's here. And now this is the content of that. We were modifying some division row if I remember correctly. Yeah, that's one of the thing. And this code is obtained from the static class path from the file. So again, I can fix it by the hash maps. Oh, and see the nice thing. Luckily the guy who's voting that is returning interface if he would be returning the three map, we would be doomed. Okay, let's try to compile that. Oh, it finished. No, looks like yes. And right now the GRD really changed the classes in that jar. So GRD is satisfied forward with that and it is capable to handle also the nested jars. Let's try to run that thing. If I'm not lying, yeah, hopefully that's the same thing. Again, let's check the running VMs. It's again here. And okay, and this is the running code. So I'm not lying. There is correctly hash map used to gain the hash map. So yes, the jar was really changed. Beanie's everything. And this program again, we can check it by the jmap. Get the pit, PID, get the histogram of the classes. Oh, but one is one. Yep, no three map nowhere. So garbage collector is correctly eating every run done hash map we are having and it's more over alive. So yeah, that was it. Jmap for hunting that, working perfectly well. Okay, that's all from me intentionally shorter because 20 minutes should be there for you to actually try all that stuff. I am done right now. I will be online and I hope that at least somebody will be able to unpack the JRD, unpack the issues and try to then that I hope I will be stormed by the questions in the upcoming 20 minutes both in the chat and in the audio videos. Really feel free to request the audio and ask whatever you have on this topic. Okay, that's interesting question. Not sure if you ask the compiler in the case of the GVM you have at least two compilers in the way. What the static compiler did, the optimization, you can see them in the final byte code that usually the compilers will not observe it but sometimes says, for example, unused field which will be dropped by optimizing compiler. You will see in the decompiled code that it's missing here, for example. So yes, that will happen. And of course, another optimization is died just in time compiler. And no, you will never see that by the pure Java word because this is usually going to the native code. If it had not answered your question, go on. An optimized byte code, you can see it through this assembler as we have in JRD but that's pretty hard to find here. You need to know what you are reading. So it depends on the optimizations. Some of them can be seen, some of them know. Okay, I know that Anna is definitely running JRD at least she is trying. I'm really wondering how much other people like that was around. Happy to hear that, Marcin. Good luck with that. Don't hesitate to write me. Don't hesitate to fill the issues to the GitHub. Yeah, the JRD is still pretty young project. And as you can see, there is a lot of fun with that. So really good luck with that. And I am looking forward for any kind of feedback. Phew, the JRD should be buildable fine with the JDK 17 from the sources. It really should, with 18, no. Because we are building in the eight compatible way and that compatibility ends with the JDK 18. Phew. Anna, feel free to fill the back or whatever from simply build the JDK 17. I cannot guess much more. But generally it should be really working fine. If you have already running JRD and you are running in the 17, then it really depends on what JVM you are injecting to. The compatibility is more overworking, but there are cases. Like, for example, the other nested fields, yeah. Yeah, the JDK 11, we can call it mainstream right now. It will take some time before the 17 will be completely accepted. But I was really ensuring the JRD 6.1 is working fine with 17. So I'm really sad to see that, that it failed for you. Oh, I forgot to show you an interesting thing which you may use. Good, okay, I lost JRD in between. Now, this is JRD and these are the running processes. It's after running the crashes. That is a nice thing. It's called the detailed info, which is pretty super useful for the hunting of hard cases because it is writing the origin of the class from where it bailed, it from which jar or directory it was loaded and the class loader which picked it up. This is super interesting info and I am not aware of any other tool which is printing it like in this global way. And still, because not much more questions last few minutes, the command line interface is pretty good. Oh, okay, except it's unreadable in this site. So the terminal. And for example, list classes and list class details is quite good here. So the done list classes, list class details from the PID again. That would be still the same, which is running. Oh, okay, class loader unknown. That would be lambdas. Okay, listing classes, the details again. You can see what loaded it from where and so on. Yeah, the command interface can do everything what can do GUI. Okay, I think the time is getting up. If you need me later, I am usually on the Fedora Libera chat and in all cases, the GRD issues, feel free to fill. Even the feature request, because there is always, it's a project for me and usually for one or two students, which work around. Okay, thank you everybody for attention. Thank you for watching and enjoy the rest of the conference. And I think I am done and over now. Bye-bye. Yeah, thank you very much for the workshop. And thank you everyone for attending. Have a break and come back to the workshop room. You can also reach out, I believe usually they can also reach out to you in the work adventure. Yeah, that's a cool thing. I was trying that before the session. Yeah, try out session room number five if you go there right away. Okay. If you go there right away. Yeah, and yes, and everyone don't forget that at 1.30 at the stage, there will be a dance lesson, introduction to the swing, following by the lighting talks. So see you all around and thank you very much.