 Christmas with Lock4J. The vulnerability problem using the example of Lock4J. Why Christmas with Lock4J? That's what we could ask ourselves somewhat desperately when the Apache Foundation reported the serious security issue on December 12, 2021, which was immediately rated at the highest risk level. Around Christmas time the company, where our work is already very busy anyway, went then, on the top of that, on the weekend, an impact like Lock4J comes along, it makes especially much noise. In the modern world where complex systems and system components are essential to the function of society and business, it is increasingly important to understand and manage the risk that these systems and components can pose to the task they support. What's certain is that nothing is certain, so I'm suspicious just in case. Any person who has been involved in information technology long enough knows, however, that nothing is really secure. Many problems surrounding the topic of security are unsolved and will remain so for the foreseeable future. Nevertheless, we want and need security. That sounds a bit paradoxical. Therefore, the correct term should be, we must try to create more security until it is secure enough. The paradox is, often we only know if something is secure enough, if it wasn't, if it somehow broke, or in the case of an attack, was correct. Or when somebody finds the bug and publishes the result, as with Lock4J. That's why this much maligned incident is also partly a success story on how helpful it is to deal openly with vulnerabilities. There is no such thing as perfect security. Perfect IT security would only be possible if every single component was error free, and in addition, the system assembled from the components would be perfect. Lock4J is a good example how difficult this is. Lock4J did not contain an actual error in the sense of a bug. The aggregation of different functions and their risky application made Lock4J toxic. Even though there are many ways to better manage such risks, we have to admit to ourselves something like this can happen again and again. We have to learn to live with this uncertainty. Talking about security therefore always has something dystopian about it. It's like the end is near. But there are certain things we can do. Therefore I would like to introduce an obviously useful pattern of action for dealing with given uncertainties. It helped the teams in our company to deal with the Lock4J incident, and the incident in turn helped us to improve. Creating security is a circular process that is carried out uninterruptedly, serving continuous improvement as well as a consistent attention by preventing habituation effects. Possible weaknesses and your priorities reduce your attack surface, be capable of action, practice crisis situations, learn through retros, and again be prepared for short. It is a generic pattern of action that always helps. Of course, we can and often must improvise. But in a crisis situation, this is often more risk than opportunity. In short, security can be increased in a planned manner and processes can be practiced. Conclusion? The Lock4J Lock4Shell incident was something special but not that special. We can learn from the incident and be better prepared for the next one. During the hot phase of Lock4J, we at Greenworn kept a kind of diary that, in retrospect, offers a lot of useful information and insights into how more security can be created with and through open source and a few good processes. Here we have summarized the topics from this diary and supplemented them by aspects that make the content more understandable. We and presumably many others who have had the opportunity to deal with the subject were primarily concerned with three questions. Are our infrastructures protected? Are our products or projects affected? How can we help our clients? We and hopefully many others were well and successfully protected. But nevertheless, the topic occupied us intensively and ultimately even for several weeks. Here is the sequence of events after the vulnerability was discovered and initially reported. First, a developer from Alibaba, the internet business giant from China, discovers the vulnerability and reports it to the project's maintainer at the Apache Software Foundation. After about a week, the Apache Lock4J vulnerability is largely fixed. Later, smaller, less critical bugs will follow which also will be fixed in a timely manner. After two weeks after the initial disclosure, the vulnerability was added to the National Vulnerability Database, the NVD by the vendor. The night of this release, we at Greenbone developed a vulnerability test and released it just one day after the public announcement. The total latency from the first message to the fix was thus about three weeks. This can be considered as good or bad, fast or slow. I think overall, this is not a bad value. So here's the text of the CVE 2021 44228. Apache Lock4J GNDi capabilities do not protect against attacker-controlled LDAP and other GNDi-related endpoints. Those who know about vulnerable direct-reservices like LDAP and interfaces like GNDi quickly saw the problem. The vulnerability was given the highest available score of 10.0 on the National Vulnerability Database. This was the short form of the message in the Vulnerability Database. It's about remote code execution, the severity is critical, and all versions from 2.0 beta-9 were affected. The topic of vulnerability assessment is actually worth more in-depth consideration, but it leads too far here. For now, it should be sufficient that the Lock4J vulnerability had to be classified as particularly severe. Remote code execution exploitable via the Internet, very widespread, this is very dangerous. Unfortunately, it is unknown whether the vulnerability was exploited unnoticed for a long time. However, we know how long it existed and what we learned was nasty because it existed for quite some time. On September 14, 2013, with the version of 2.0 beta-9, the GNDi lookup plug-in was added to the Lock4J package in the Lock4J 2313 release. Since then, Lock4J was unfortunately broken. That was about 3,100 days or about 101 months or 8 years and 5 months ago. When the vulnerability became known, the attackers reacted quickly. Although the vulnerability Lock4J was not disclosed until the last three weeks of 2021, Lock4Shell attacks were the fifth most common external attack vector in 2021. Conclusion, we do not know how often Lock4J could be exploited. Those who learned about the vulnerability and were able to patch it quickly were at an advantage. Those who had sufficient strong security policies in place in the first place were not affected. Precaution pays off. Let us now turn to the aspect that made the actual in-specialist Lock4J a particularly suitable attack candidate, Lock4Shell. Lock4Shell is not a vulnerability like any other. Lock4J is a very simple, convenient and useful tool in software development. It is considered as a standard in Java to lock states and activities when using software. This is called logging, hence the name Lock4J, where the J stands for Java. Malicious attackers can exploit a vulnerability through Lock4J to execute code on affected systems. Because this vulnerability can be exploited over the Internet without any authentication, this can be very critically for affected systems and companies. Because software is also included in a large number of software and services that are accessible via the Internet, a very large number of companies and services were probably also affected. A logging function is needed in virtually every piece of software. Moreover, it is not only used as development time but also carried out in the application usage context and thus massively spread. The area of use can be very diverse. Enterprise are Internet application servers, desktop systems, Internet routers, firewall devices, and many more could be affected by Lock4J. Lock4J is very easy to use and an awareness of the possible risk has not been very pronounced, certainly often along the lines of what can be dangerous about logging. Moreover, a person using Lock4J in program does not always have a clear picture on how the component behaves in use and which configuration parameters have which influence. In other words, Lock4J made it very easy to achieve very powerful functionality. This also made it particularly dangerous. Lock4J has been around for quite a few years. Lock4J won since 2001, Lock4J 2 since 2012, without any major problems becoming known so that its use has not been evaluated or questioned in concrete application scenarios. Vulnerabilities can vary greatly in terms of how easy they are to exploit. When the crucial vulnerability in Lock4J was found, it immediately became apparent that it was very easy to exploit with potentially dramatic consequences. The foreign general conditions like extremely widespread use in a wider variety of application systems, high possible risk when using the component, little activity and awareness of risk-limiting measures, easy exploitation of the vulnerability together led to great potential damage. We will now take a look on how an attack with Lock4J works. A simple function, logging, is used to inject content into a system. Wherever user input or other interactions or locked attackers can use them to inject content, like from form inputs, search fields, api calls, etc. Unfortunately, the logging function does not only support writing of strings, but also the evaluation of expressions called a lookup. These expressions can be called to Java functions, including those from a library called GNDI, Java Naming and Directory Interface. GNDI also allows to access to remote directory services and reading of environment variables, among other things. This allows code from a remote source to be executed in the context of the attack application, also called RCE Remote Code Execution. That is why the vulnerability was also named Lock4J because it enables a remote shell in which arbitrary code can be executed. Furthermore, the values from the attack systems can be transferred to an attack system and vice versa. How could this possibly be so easy? In short, through a chain of unfortunate circumstances and a well-intentioned idea. Lock4J supports the mentioned feature called lookup by default. This makes Lock4J easy to use in logging. Lookups evaluate strings at the time a message is logged. For example, logging, hello world, dollar, curly braces, Java version would result in the following being logged. The environment variable, dollar, Java version is resolved to its actual value. This is not yet bad, but practical. It becomes bad when combined with other capabilities of Lock4J. These capabilities include the aforementioned GNDI, which supports LAP as a protocol, for example. A used LAP server can be local or remote. LAP is a protocol for reading data from or writing data to so-called directory services. In principle, such a directory service is simply a hierarchical database in which user data, device data and the lack of network and other things are managed. A very well-known directory service is Microsoft Active Directory. Basically, any data can be inserted and retrieved there through GNDI. And I can also set up my own malicious directory service and use it to download malware by getting Lock4J to call my server and download my malware. Sounds spooky? It is. If I get Lock4J to lock a string like the following, GNDI, LAP, example.com, hello world.class, Lock4J will make a remote request to example.com, get hello world.class and execute it, which is a remote code execution. GNDI supports not only LAP, but actually any kinds of name and directory services, such as domain name system, network information services, godness, file system, we'll name it. Depending on the configuration of the attack system, I can do almost any terrible thing. Upload Trojans, download data, everything that is bad. In case of doubt, I nest myself in the target system and do nothing for the time being until the appropriate time for the attack seems to have come. Common exploit strings. I want to show you a few examples of common exploit strings. They can be categorized into three groups, spy on environment, steal access data, load and execute code. In our first example, we call an external LAP server and send the Java version to it. You could send anything else, like in the second example, we will transmit our AWS secret access key, a bit more hurting than the Java version, I guess. And in the third example, as already mentioned, we can load and execute code. From this remote server, we load a class and this class is executed in the context of Lock4J. Even in places where web applications perform input control to prevent things like these examples, I can still get lucky as an attacker by discussing my attacks. In order to do so, initial lookups can be used and with a bit of luck or misfortune for the attack person or system, web application firewalls can be tricked. As an attacker, I have to get my attacks to possible victims. I can do this by manually using input fields and web solutions, like forms. But it can also be done more efficiently as attack attempts are relatively easy to automate. Since most attacks are not targeted but work on the principle of try and error, automation is extremely useful for attackers. For example, I can crawl websites and then insert the malicious string into multiple HTTP request headers. URL parameters and form fields that I sent to the URLs I find. Here's a simple code example that does this. Of course, there are much more sophisticated frameworks that are used for attacks. I have shown this example to show that all of this is not great art. If you want to make more difficult attacks or make it more comfortable, as an attacker, you can also buy appropriate solutions. There are also marketplaces where attackers can directly buy the loot from such rates. It is an industry of its own in which hardly someone builds their own weapons. Either complete frameworks are built like in the sophisticated attack from NSO or directly results are acquired. We can reasonably prepare for vulnerabilities by implementing three measures. Secure configurations, in case of lock for day, e.g. disabled lookups, disabled GNI, frequent updates and patches, regular check or scan for possible vulnerabilities and insecure configurations. In addition, the process view of threat situations is also important, especially in critical situations, responsibility and processes must be well defined. This requires a few self-evident facts in advance and precautionary planning, like definition of areas of responsibility, including substitution arrangements, availability, etc. Knowledge and sharing this knowledge in documentation and acting as a team. Processes that ensure interaction with as little friction as possible. Of course, the best process cannot replace knowledge and certainly no commitment and a sense of responsibility. But without processes, it is much more difficult to be able to act and be successful in preventing attacks or dealing with them, especially in critical situations. After all, you don't start building lifeboats and do an evacuation exercise when the ship is already sinking. We are therefore looking at the most important areas of protection, our infrastructure, our projects and our products, our customers and users and our supply chain. Infrastructure. Vulnerabilities are reported every day, but not all of them are relevant to your own infrastructure. For example, if you don't run Microsoft Exchange Service or SAP systems, you don't have to worry about your own infrastructure if a vulnerability becomes known there. With Lock for Shell, we were not quite sure because every product announces whether it uses Lock for J or not. That's why we triggered an escalation process in our company when the Lock for J incident happened. We formed a task force in our technical operations department. This task force immediately reviewed the status of any affected system with the highest priority. Luckily, none of our products and none of our internally or externally provided services were affected. What we did find, however, were Wi-Fi routers and individual switches. We patched these at a very short time and we were able to continue our operations without major interruptions. Throughout the process, it helped. We created and audited an ISO-certified ISMS information security management system last year. As a result, we already have systematically dealt in advance with the cataloging of our systems and with any escalation process that may be required and have documented many things in this regard. Projects and products. Our teams reviewed our source codes and dependencies and determined that our own open source products and open source projects were not affected. Our feed team, providing the vulnerability tests. Our production unit, packaging and delivering our systems to customers and to the community. Our marketing unit, which coordinated and executed communications. And our information security officer, the manager of duty, who ensured that all stake-out holders were coordinated and security objectives were met. Due to the serious nature of the incident, our top management was also involved. We have distinguished three phases in the process. Red, urgent activities available of key people outside regular working hours and on weekends, constant monitoring of sources. Yellow, processing of operations with priority, regular monitoring, partially increased availability. Green, wrap-up clean-up priority in the context of normal working hours. The red alert ran from Friday, 5.00, o'clock in the morning until Monday, close of business. It was concluded with the provision of vulnerability tests to customers and community users. So, roughly speaking, we had a week of escalation mode and then the usual diligence. Now let's talk how you can protect against vulnerabilities. There are currently hundreds of thousands of known vulnerabilities, varying in severity impact on other components and distribution. Of course, not every vulnerability affects every system and not every exploit of the vulnerability is equally relevant for the operators of systems. For example, if availability is particularly critical for some services, then availability that enables a denial of service attack is particularly relevant, even though it might not have access to data on the service. In other scenarios, availability failures are more tolerable, but access to data is not. However, there are a few fundamentally equally dangerous effects that apply to all services, such as the assumption of administrative rights, the infiltration of ransomware and the like, and especially dangerous, of course, vulnerabilities that allows malware like Trojans to infiltrate the system. Ultimately, vulnerabilities can and are used for economic damage, sabotage, manipulation, espionage and surveillance, and I avoid the term cyber wall. What can I do preventively? How quickly can I react to threats? And how quickly am I able to restore safe operation capability in case of doubt? So why there are vulnerabilities at all? There are four reasons for this. Vulnerability arise due to configuration, unplanned interactions, programming errors and faulty processes. Moreover, when vulnerabilities arise, they usually ignite further discussions that tend to be in the realm of programming language, operating system and technology, even licensing. Unfortunately, another reason for vulnerabilities is bad faith. In a proprietary software, the vendor can put in all sorts of things I don't know what. In open source this is much more difficult to do, but it's not impossible. Nevertheless, the greater security of open source due to its possible auditability is a strong argument in favor of open source. The risk of detection is dramatically greater with open source. With highly secure applications, I can audit the entire tool chain if I want to and have the resources. Especially, where very common products are considered as standard, there is an implicit expectation that these are safe enough. After all, so many people use them. This expectation is not justified on the contrary. Because particularly popular products are naturally also particularly popular targets for attacks. And very often we only know the functions of a software that are useful to us and know little about how it is created. Another implicit connection between popularity and vulnerability is products are particularly popular when they are particularly convenient. However, convenience and security must be weighted up carefully. Not every vulnerability enables convenience, but too much convenience also means greater vulnerability. Simple examples are convenient password or not using password managers or too factor authentication. A more complex example is a high degree of integration with other systems which is one of the reasons of the Log4J problem. Second, legacy systems. A not insignificant problem is the sometimes surprising lifetime of software systems. Despite significant advancement in software technology and programming languages over the past decades, many safety critical areas still rely on legacy systems that were developed using insecure programming techniques or must be executed on an insecure basis. Third is innovation is faster than safety. Many systems are used in complex and critical infrastructures and are not easily replaceable due to the many players in the field of operation. At the same time, more and more of these systems are being networked for obvious advantages, so they are gaining new attack surfaces. This applies here only by a way of example to fields of application as diverse as healthcare, industrial plans or global trade. To give you an example, to make software for ships more secure, we have to talk to port authorities, ship owners, cruise lines, public administrations, custom authorities, importers, industry, insurance companies and more. They all need to be involved and they all have systems running. At the same time, new additions are making legacy systems more interconnected and thus more vulnerable. This is only meant to be an example for many networked scenarios and means for us. We will have to live with legacy systems and their vulnerabilities for a very long time. And networking with internet technologies and the internet of things doesn't make it easier, it makes it more challenging. Now let's talk about how do vulnerabilities become known. Vulnerabilities become public in a number of ways. Through an entry in a public vulnerability database, for example at NIST or another cert, by publishing a manufacturer's advertisement, through direct publication of the finding people, through entities in a public bug tracker, about different media, combinations of their birth, or worst case not at all. Of course it makes sense to make the vulnerability public when it is also fixed or there are effective countermeasures. Otherwise it would be an invitation to cyber criminals to spread exploit faster than the gaps can be closed. This kind of reporting process is also called responsible disclosure. In case of a vulnerability management at NIST, for example, the following process is basically takes place. First, a vulnerability is reported. Second, it is given a unique identifier and entered into the database. Third, effective products are assigned. Fourth, a vulnerability is evaluated. Since new information usually keeps coming in, this is a cyclical process with constant change. Especially with new vulnerabilities, this can be really dynamic. In addition, there is a communication level, warning, support and dealing with the respective vulnerability, etc. The most important information is the threat posed by the vulnerability. How do I know how serious a vulnerability is? The common vulnerability scoring system and a corresponding calculator have been available since 2005. This is the leading standard for uniformly evaluating security vulnerabilities. One of the reasons why evaluation is difficult is that vulnerabilities have a relevance that cannot be standardized depending on the context in which they may occur. Nevertheless, the CVSS is the best attempt to provide an orientation here. So here is an example to help us understand the rating. Lock for J, as already mentioned, had the top score of 10 and it is not every day that that vulnerability is rated like this. Note, the full evaluation scheme is even more complicated but we see the most important part here. Below we look shortly into in this detail. The attack vector is network. This means vulnerabilities with the rating can be exploited remotely. Addison, the attack must all originate from the same or physical, logical network. Local attackers must access the system locally or remotely via SSH or RDP. Or they must use social engineering or other techniques to trick unsuspecting users into helping with the exploit. Or physical, this type of attack requires attackers to physically interact with the target system. Second is complexity. Low, no specific preconditions for utilization are required. Or high, there are conditions beyond the attacker's control to carry out a successful attack. Next is required privileges. So none, no privileges or special access rights are required to perform the attack. Low attackers need basic user level privileges or high administrative or similar access rights are required for successful attack. We've seen that the log4j attack didn't require any privileges which improve the rating. User interaction, none user interaction is required or user action is required like clicking on a link or do anything to make an exploit successful. A bit complicated is the attribute of area which means that the exploited vulnerability can affect other systems too. This is the case with log4j, for example. Because it doesn't only affect log4j but the many other components which uses log4j. So to get the highest score you have to have a vulnerability which affect other systems. What is leaked in the end or what is caused by the vulnerability? Like confidentially, so the attackers have access to resources of the affected system, integrity, they can change data, availability, like DOS attacks so the systems might be no longer available. All factors together lead to a rating. In this rating I can determine that everything above the value of 7 is high and everything above the value of 9 is critical. The urgency with which I have to take care of a weak point depends on this rating but of course also on my own specific circumstances. In the context of log4j a whole series of applications appeared relatively soon that claimed to detect the vulnerabilities in users systems and these two usually worked but they have fundamental advantages and disadvantages. Open source tools certainly have an advantage as the source codes can be checked in case of doubt. Ultimately, however, it's always a risk to use such tools if they do not come from a trustworthy source. Another thing is it works with one vulnerability but it's not very practical to chase every vulnerability with a dedicated tool because there are too many vulnerabilities out there so this is an advantage of vulnerability management products that cover many vulnerabilities and take care of new vulnerabilities that have become known. Again, open source products have a big advantage here because you let these tools carry your systems and so you should think about whether you want these from a proprietary software. All vulnerability scanners work in the same way they make queries to systems and services and read from the responses what kind of systems and services these are and also what products are behind them. They include information such as versions or even settings and other properties. In many cases this makes it possible to determine whether a vulnerability exists and whether it already has been eliminated, maybe. In practice these are sometimes very complicated and nested queries but above all of them these are all very, very many queries. Entire networks are scanned for thousands of different vulnerabilities with hundreds of thousands of tests. What they do is first you find physical and virtual hosts to find the machines where the software runs and second you find out which software runs on these hosts then you can scan this software for vulnerabilities you can review the results and then create reports. You have to do this frequently to be on the safe side it's always a cycle of scanning, patching, securing your configurations and this besides the good old backup thing is the best you can do to make and run your system secure. There is no such thing then absolute protection there will always be vulnerabilities and there will always be people to exploit them for therefore this helps be prepared, know your risks, your possible weaknesses and your priorities reduce your attack surface be capable of action practice crisis situations and learn through retrials. Thank you for your attention we are at the end of this talk I hope that you could take something out otherwise just mainly I was happy to talk to you have a great time, stay safe, bye bye.