One year ago, the Log4j vulnerability was discovered. The finding immediately stunned the world due to the exploit’s severity and the pervasiveness of the exposure — it’s estimated that upwards of three billion devices that use Java were affected. The Log4j vulnerability can lead to remote code execution (RCE), the most deadly cyber threat. For this reason, it has been called “the single biggest, most critical vulnerability of the last decade.”
Maintainers were quick to issue a fix to the Log4j vulnerability. Yet, it still took months for many software providers to update their applications to the new version. And, a year later, one in four Log4j downloads still possess the vulnerability according to data from the Maven Central repository. Although this number is steadily decreasing, it still means that millions of applications are prone to the vulnerability. As a result, Tenable Research found that 72% of organizations remain susceptible to the Log4j vulnerability.
Given the exploit’s severity, why haven’t all applications upgraded to the latest version? I recently reconnected with Brian Fox, co-founder of software supply chain security company Sonatype, to understand why it’s taken so long for organizations to respond to Log4j. Below, we’ll explore why Log4j has gone unmitigated for so many deployments. We’ll also suggest solutions to rectify the unsustainable condition we find ourselves in when it comes to open-source software dependency awareness (or lack thereof).
Unpatched Log4j Vulnerabilties Persist
Sonatype’s research center has tracked over 130 million Log4j downloads since December 2021, 34% of which are vulnerable. And at the time of this writing, within the last 24 hours, 25% of downloads, were vulnerable. This means that one in four downloads of Log4j are still of an outdated, vulnerable version.
The staying power of the Log4j RCE exploit has severe ramifications on a global scale. It’s a backdoor that hackers attempt to exploit ten million times per hour, according to a 2021 Akamai study. It’s also being leveraged in state-sponsored cyberwarfare. In October 2022, the National Security Agency, Cybersecurity and Infrastructure Security Agency (CISA), and FBI jointly revealed the top CVEs exploited by Chinese state-sponsored actors, and the report lists Apache Log4j CVE-2021-44228 Remote Code Execution as the #1 top threat.
Reasons Behind Enduring Log4j
So, why do so many vulnerable downloads persist? Well, the most simple reason is a lack of awareness. Most consumers don’t have the wherewithal to understand the intricacies of the software they consume, described Fox. This awareness could be significantly increased through the use of software bill of materials (SBOMs), yet sharing SBOMs has yet to become a standard practice. Even if it had, most organizations don’t yet possess the tooling and processes in place to audit SBOMs and handle them at scale.
Furthermore, nearly all these vulnerable downloads come from automated software build processes, says Fox. Machines simply download whatever version is specified in an application’s Project Object Model file, which likely hasn’t been updated for some time. (Arguably, open-source consumers are responsible for the majority of risk if they’re not updating dependencies in a timely manner.)
And when human engineers do make upgrade decisions, they don’t always make the most effective choices. Sonatype’s State of the Software Supply Chain report found that 69% of all upgrade decisions were suboptimal. The optimal zone is living close to the edge, but not on the bleeding edge since unknown malicious components might live there, too, says Fox.
Root Cause Solutions
Plugging the hole that is Log4j will require a continued effort to educate developers. “We’ll have to continue to talk about it, write about it, and continue to get the policy to make [fixing] it a standard,” said Fox. But, simply patching one vulnerable package doesn’t get at the root cause of the greater software supply chain dilemma.
“People don’t know this is happening because they don’t know what’s in their software,” said Fox. In order to make open-source safer, we must raise the bar so that both the people selling software and the people consuming it know what’s inside it. This means improving awareness through greater SBOM use and establishing a culture around dependency management.
Making smarter, more frequent upgrade decisions can help avoid many of the known vulnerabilities that are so commonly inserted into packages. Of course, each update costs time and money, and it might not be practical to update with each minor release. Yet, you don’t want your dependency patches to pile up to where it takes a “Herculean effort” to update them all. Fox recommends balancing the two extremes — just as a pilot follows uses their instrument landing systems, you want to stay in the sweet spot between the indicator lights.
Lastly, maturing dependency management will undoubtedly require some dedicated tooling. Because when you have hundreds or thousands of dependencies in an application, all changing ten times a year, it becomes a headache to manage without automation. “You can’t manage that via a spreadsheet — that would be insane,” said Fox.
Final Thoughts: Raise The Profile
In late 2021, Wired announced that the Log4j vulnerability would “haunt the Internet for years.” And unfortunately, a year later, it appears that this statement rings true. Since the Log4j logging system is integrated into so many applications, the exploit has major staying power. “One-fourth being vulnerable is not a great place to be,” said Fox.
Bringing awareness to Log4j and similar exploits may also require fixing deeper issues, like a lack of communication between departments or convincing developers to prioritize software supply chain security. “Once more companies pay attention, they start answering questions. That’s how we avoid this situation going forward.”
Consider the standards and regulations around recalls in the [manufacturing industry]. If a car’s component is faulty or presents a safety hazard, a recall is issued for all affected parts. Imagine if software providers were required to respond to vulnerabilities in their code with the same degree of tenacity. “We need to raise that profile even more,” said Fox.
Want more cybersecurity insights? Subscribe to the Cybersecurity as a Business Enabler channel:
