Executive Summary
- Our analysis has concluded that there are now around 28,910 hosts that show signs of compromise as of October 25, 2023.
- The attackers have adapted, and our previous methods of identification are no longer effective
- Researchers have determined a new (albeit not as precise) way of determining whether the backdoors on these devices are still active
Last week, we shared information about an ongoing event with Cisco devices and a backdoor installed on tens of thousands of Cisco hosts. And in our last update, we noted a significant drop in the number of infections. And would you believe that number has dropped to zero? Well, sort of…
Since then, the group (or individual) behind this mass compromise has seen the error and attempted to cover its public visibility by removing the configuration that enabled researchers to determine whether the device was compromised. In doing so, the methods to find these compromises instantly became deprecated.
At its core, these Cisco devices are running the Nginx web server, and (one of) the configurations being modified by the attackers is the hosts’ Nginx configuration file. An astute reader over at Fox-IT noticed in the screenshot Talos provided the attacker had another added location configuration directive, which can still give us some insight into whether the backdoors are installed on the device.
A location directive in Nginx allows an administrator to act differently at an HTTP URL path level, and this added one was a case-insensitive regular expression match for any path that included the percent sign (%), and if that percent sign is found in the path, it simply sets a few headers and returns a 404.
Unfortunately, due to how Nginx handles 404 error returns, the headers added in this directive are never output to the client, but since these Cisco devices seem to have a custom 404 handler, and this new configuration directive overrides that handler, we get a different 404 status message (the default Openresty/Nginx version) when this configuration is in place.
We’re getting into shaky territory when it comes to determining the status of this backdoor, and we’re not 100% sure if this is the best way to conduct these types of tests, but it’s the only way we have right now.
So, what is the significance of this percent sign? We can only assume our answers because, in most cases, a percent sign in a URL can translate non-printable characters into something a server can see (for example, %41 == ‘A’). So, the attackers are attempting to hide something, but we have no way to tell for sure. Unfortunately, most security organizations like Talos will not divulge anything more than what they post, so this is all we know.
On October 25th, we conducted another secondary scan against every IP seen with Cisco-like characteristics over the past seven days (this may include checking devices that aren’t specifically Cisco XE, as we are taking a more general look into all Cisco-like devices, just in case), looking for any device that will answer with the default Nginx/Openresty 404 message when the URI ‘/%25’ is requested. Our input list contained over 135,000 potential devices, and only 28,910 hosts responded with what looked like a default Openresty/Nginx 404 handler when requesting ‘/%25’, which indicates that the backdoor is installed.
Once again, we cannot say 100% sure if this is an actual indicator of compromise, but these Cisco devices do respond differently when requesting a resource that shouldn’t exist against these hosts. For example:
Requesting /%25 on a previously compromised host.
Requesting some random file that shouldn’t exist on a previously compromised host.
What’s intriguing about this incident is how specific and targeted everything was. Cisco uses Openresty, which is an Nginx-based web server with the ability to develop and embed Lua right in the configuration files – this is basically like moving the application development into the web server itself; this isn’t anything out of the ordinary since there are currently around 800,000 Openresty servers online right, which isn’t a lot, but can be considered moderately popular when compared against other server technologies.
While at first glance, it seems like these attackers may have burned their zero-day, it’s evident that this was a coordinated, well-planned, and well-executed attack. Not only did they find the vulnerability in these devices (if the vulnerability was not purchased), but they also utilized the underlying technologies running on these devices to implement a backdoor. This means there was a reasonable amount of effort put into learning the tech and implementing the system-specific backdoors.
And while it wasn’t a perfect execution, they did learn from their mistakes and quickly fixed them. We’ve seen this pattern a few times before (Deadbolt, ESXiArgs):
- An incident happens, information is publicized
- News and research spreads
- Attackers monitor and adapt their techniques to subvert the identification and analysis
- Repeat
In this case, we were given enough information to find the initial compromises but were quickly locked out when attackers changed their methods and techniques. If this new method of finding compromises is accurate, then the number of devices has drastically decreased.
