ExpressVPN chief architect Peter Membery vividly remembers seeing the news of the Log4j vulnerability online.

“As soon as I saw how it could be exploited, it was horrible,” says Membery. “Like in one of those disaster films where there is a nuclear power plant, they find out that it is going to melt, but they cannot stop it. You know what’s coming, but there is very little you can do.

Since the vulnerability was discovered last week, the cybersecurity world has shifted into high gear to identify vulnerable applications, detect potential attacks, and mitigate exploits as much as possible. Nonetheless, serious hacks using the exploit are almost certain.

So far, researchers have observed attackers using the Log4j vulnerability to install ransomware on honeypot servers, machines deliberately made vulnerable in an attempt to track new threats. A cybersecurity company reported that nearly half of the corporate networks it monitored saw attempts to exploit the vulnerability. The CEO of Cloudflare, a website and network security provider, announced early that the threat was so severe that the company would deploy firewall protection for all customers, including those who had not paid for it. But concrete information about exploitation in the wild remains scarce, possibly because victims do not yet know or are unwilling to publicly acknowledge that their systems have been violated.

What is We know for sure that the scope of the vulnerability is enormous. A list of affected software compiled by the Cybersecurity and Infrastructure Security Agency (CISA) – and limited to enterprise software platforms only – spans more than 500 items at the time of going to press. A list of all affected applications would no doubt extend to several thousand more.

Some of the names on the list will be familiar to the public (Amazon, IBM, Microsoft), but some of the more alarming issues have arisen with software that remains behind the scenes. Manufacturers such as Broadcom, Red Hat, and VMware build software on which corporate customers build businesses, effectively spreading vulnerability across the backbone infrastructure of many businesses. This makes the process of finding and removing vulnerabilities that much more difficult, even after a patch has been released for the affected library.

Even by high-profile vulnerability standards, Log4Shell is hitting an unusually large part of the internet. This reflects the fact that the Java programming language is widely used in enterprise software, and for Java software the Log4j library is extremely common.

“I queried our database to see every customer who was using Log4j in one of their applications,” says Jeremy Katz, co-founder of Tidelift, a company that helps other organizations manage dependencies. open source software. “And the answer was: each of them that has applications written in Java.”

The discovery of an easily exploitable bug found in predominantly business-oriented language is part of what analysts have called a “near-perfect storm” around the Log4j vulnerability. Any company can use many programs that contain the vulnerable library – in some cases with multiple versions in one app.

“Java has been around for many years and is used a lot in enterprises, especially large ones,” said John Graham-Cumming, CTO of Cloudflare. “This is a great time for people who manage software within organizations, and they will be performing updates and mitigations as quickly as possible. “

Given the circumstances, “as fast as they can” is a very subjective term. Software updates for organizations such as banks, hospitals, or government agencies are typically performed on a weekly and monthly basis, not a day; Typically, updates require many levels of development, authorization, and testing before they can be integrated into a live application.

In the meantime, mitigation measures that can be postponed quickly provide a critical interim step, saving valuable time as organizations large and small scramble to identify vulnerabilities and deploy updates. This is where network layer level patches have a key role to play: Since malware communicates with its operators over the Internet, measures that restrict inbound and outbound web traffic can be a stopgap. limit the effects of the exploit.

Cloudflare was a rapidly growing organization, Graham-Cumming explained, adding new rules for its firewall that blocked HTTP requests containing strings characteristic of the Log4j attack code. ExpressVPN also modified its product to protect against Log4Shell, updating VPN rules to automatically block all outgoing traffic on ports used by LDAP – a protocol the exploit uses to retrieve resources from remote URLs and download them to a vulnerable machine.

“If a customer is infected, we’ve already seen scanners as malicious payload, so they can start scanning the Internet and infect other people,” says Membery. “We wanted to put a cap on it, not just for the sake of our customers but for the sake of everyone else – much like with Covid and vaccines. “

These changes usually happen faster because they take place on servers owned by firewall or VPN companies and require little (if any) action from the end user. In other words, an outdated software application could still get a decent level of protection from an updated VPN, even if it doesn’t replace a proper patch.

Unfortunately, given the severity of the vulnerability, some systems will be compromised, even with quick fixes deployed. And it may take a long time, even years, for the full effects to be felt.

“Sophisticated attackers will exploit the vulnerability, establish a persistence mechanism, and then disappear,” said Daniel Clayton, vice president of global cybersecurity services at Bitdefender. “In two years we will hear about significant violations, and then we will learn that they were violated two years ago. “

The Log4j bug once again highlights the need and the challenge of adequately funding open source projects. (A huge amount of tech infrastructure might as well depend on a “project that some random person in Nebraska has been maintaining tirelessly since 2003,” as an always-relevant XKCD comic explains.) Bloomberg reported earlier this week that many of the developers involved in the race to develop a patch for the Log4j library were unpaid volunteers, despite the software’s global use in enterprise applications.

One of the latest vulnerabilities to rock the internet, Heartbleed, was also caused by a bug in a widely used open source library, OpenSSL. As a result of this bug, tech companies like Google, Microsoft, and Facebook pledged to invest more money in open source projects critical to the Internet infrastructure. But in the wake of the fallout from Log4j, it’s clear that dependency management remains a serious security issue – and one that we’re not close to fixing.

“When you look at most of the big hacks that have happened over the years, it’s normally not something really fancy that is breaking big business,” says Clayton. “This is something that has not been fixed.”