Executive Summary
- Open Directories are listings of files and data, often used for malicious purposes. While their existence is often not meant to be public, we find many instances that are.
- 50% of open directories are online for a day or less, indicating highly ephemeral behavior. Further, 50% of these open directories change the content listed in three days or less.
- Real-time visibility is critical for investigating such highly ephemeral adversary infrastructure.
Previously, we investigated the time to live, or how long a piece of infrastructure remains online, for two popular command and control (C2) families, Cobalt Strike Team Server and Viper. In this installment, we examine the lifespans of a different type of infrastructure that is often malicious: open web directories, specifically from a network liveliness and content similarity view.
Open directories are another interesting avenue for us to perform investigations as they are literally open directories or filesystems, often containing payloads and files used for nefarious purposes. These open directories are hosted on the public internet, which means not only can attackers find them, but so can we.
To do so, we need to approach the problem differently. Where previously we had the benefit of being able to track Cobalt Strike Team Server via a distinctive watermark, open web directories do not have a specialized field that we can key on. However, we consider the defining characteristic of an open web directory to be the HTTP body, as that is where all the interesting content of the open directory is typically found.
So we examine the lifespans of open web directories based on their HTTP bodies. We narrowed down our scope to look at only open web directories that are located on an HTTP service for the month of April. To compare the HTTP body of the open directory, we calculate the hash of the HTTP body using TLSH, a rolling hash algorithm. In short, TLSH allows us to compare HTTP content more easily. We calculate the score between the service’s current TLSH hash and the service’s previous TLSH hash, where 0 is not similar at all and 100 is the same, and we compare the service lifespans of open web directories based on their service liveliness and their TLSH comparisons.
In the above graph, we show the percentiles of network lifespans, or pure observability, vs lifespans based on content similarity. We find that open web directories have shorter network lifespans, a median of 1 day, meaning that 50% of open directories generally come online only to disappear a short while later. However, when we examine open directories through a content lens, we find their median lifespan is closer to 3 days. This means that even if a web directory blips in and out in terms of network visibility, the content doesn’t necessarily change. This distinction is important, as it allows us to understand how much an open directory has changed, a critical component for actor investigations.
What does this mean for defenders? Having up-to-date network visibility is crucial because it provides an accurate representation of what hosts and pieces of malicious infrastructure are available and what are not. However, accuracy is not only the only piece of the picture. Knowing the content of the infrastructure allows us to track how open directories are changing and fluctuating in terms of content, regardless of their network availability.
Explore Open Directories on Censys
If you don’t have a Censys account, you can sign up for a free account here. If you have a Censys account, you can input the query below to start exploring open directories.
host.services.labels.value: "OPEN_DIRECTORY" or web.labels.value: "OPEN_DIRECTORY"The Censys 2025 State of the Internet Report concludes with our wrap up blog.
