Databases. EXPOSED! (Redis)

Published on 09.19.2022

Part 1: Redis

Takeaways

• There are 39,405 unauthenticated Redis services out of 350,675 total Redis services on the public internet.
• Almost 50% of unauthenticated Redis services on the internet show signs of an attempted compromise.

Preface

In this new series of posts, we decided to answer the question: “What is the state of databases on the Internet?”. We can answer this question in extreme detail using our dataset.

This report is the first of several. Over the coming months, we will release a detailed analysis of several different database technologies, and we will begin our journey into “Databases. EXPOSED!” with the popular in-memory database: Redis

But before we go much further, let’s talk about what it means for a database to be “exposed” on the Internet. Our scanner will attempt to speak the native language of whatever service we are trying to enumerate. For example, our scanner will construct a packet that only a MySQL server knows how to handle, and in return, we get a response that gives us more information about that running service.

A MySQL server’s handshake response.

Censys will never attempt to authenticate with any of the database services we find. We establish a handshake with the remote server using the native protocol and parse the responses into a set of fields, making searching easier.

At the time of writing, there were 220,010,967 hosts with one or more exposed Internet services. Of those, 5,889,954 hosts (2.6% of the total) are running one or more of the twelve database technologies we will discuss throughout this series. Below is a graph showing our top databases ordered by the number of services.

(Database services found by Censys)

Notes:

• When we say “host” or “unique host”, we are referring to a single IP address.
• When we say “service” or “services,” we refer to one or more services on a host.

Introduction.

Redis is the fourth most used database engine we consider in this series. Unlike traditional relational databases, Redis was not designed with security in mind, with the expectation that it should always be for internal and private communications only (i.e., not directly connected to the Internet and sitting behind a firewall). The following quote is from Redis’s own documentation on this matter:

Redis is designed to be accessed by trusted clients inside trusted environments. This means that usually it is not a good idea to expose the Redis instance directly to the Internet or, in general, to an environment where untrusted clients can directly access the Redis TCP port or UNIX socket.

In more recent versions (starting in version 3.0.0), Redis has addressed the growing problem of passwordless servers exposed to the internet by running in a “protected mode” if it finds itself using the default configuration. This protected mode will only respond to requests on the loopback interface and block requests sourced from the Internet. Although as we will see further in this post, the problem still persists.

Redis services do not enable authentication by default, and it is because of this lack of security, Censys can see tens of thousands of unauthenticated Redis deployments on the Internet.

Exposure.

(Geographic heatmap of Redis hosts on the Internet)

At the time of writing, Censys observed 350,675 Internet-accessible Redis services across 260,534 unique hosts. While most of these services require authentication, 11% (39,405) do not.

“11% of Redis services on the Internet do not require authentication.”

Below are the top ten countries with Internet-exposed Redis servers ordered by the total number of services. China was number one with 130,839 Internet-accessible Redis services, 15% (20,011 services) of which do not require authentication. While the United States holds the number two slot with 96,904 total Redis services, only 5% (5,108 services) are left open without authentication.

By looking at countries with more than 100 Redis services, we can visualize what regions of the world have the highest percentage of misconfigured Redis installs. In the graph below, we see that while the country of Israel only has 187 Redis services, but over 72% of them lack authentication. Israel is one of the only regions where the number of misconfigured Redis servers outnumber the properly configured ones.

By default, the Redis service will run on TCP port 6379. Still, Censys has observed it running on over two thousand other ports, ranging from TCP port 6380 with 10,143 unique hosts to TCP 24491 with only a single host. Below are the top five ports we have found Redis to be running on.

Redis holds all of its data entirely in memory, and when our scanner issues a non-intrusive `INFO` command to the service (which gives us an overview of the current operating status) we can see how much memory is in use, and in turn, how much data is being exposed to the public internet. While we never request or view the contents of the data from an exposed Redis service, a malicious user could easily dump all of the stored data from the service.

By aggregating the Censys search field “services.redis.used_memory“, we can calculate that out of the total 39,405 unauthenticated Redis servers we observed, the potential data exposure is over 300 gigabytes.

Unauthenticated Redis Services by AS	Exposed Redis Data by AS

TENCENT-NET-AP (AS45090) has the highest number of unauthenticated Redis deployments, with 13,359 services and about 48 gigabytes of exposed data. But ALIBABA-CN-NET (AS37963), while only having 2,377 unauthenticated Redis services,  has the most data exposed to the Internet with just over 60 gigabytes (62,341,142,583 bytes) of data.

From an attacker’s perspective, a smaller number of misconfigured services with a higher data exposure is more valuable than a higher number of misconfigured services with little to no exposed data.

Interestingly, we found that the host with the most data exposure ran eight Redis servers on eight different ports, revealing more than 6 gigabytes of data in total. What was interesting about this host was that two of the eight running Redis services required authentication. This means that the host admins were both aware and knowledgeable enough to enable authentication on some Redis services but overlooked six of the eight. The reasoning could be attributed to a bad configuration, a deployment mistake, or a permissive firewall rule. But no matter the underlying reason, one thing is highly probable: the host owners do not understand their external attack surface.

And this isn’t a unique circumstance; we found that 406 hosts running multiple instances of Redis had a mix of both authenticated and unauthenticated services on different ports on the same host. This gives credence to the fact that some things will always slip through the cracks, and understanding your Internet footprint is critical to an organization’s continued security.

As for the amount of data exposed per country, the graph above depicts the top ten countries ordered by the total amount of data exposed by misconfigured Redis services. China dominates with 146 gigabytes of exposed data, and the United States trails behind with around 40 gigabytes.

Versions.

In the introduction of this post, we stated that Redis developers have attempted to quell the problem of passwordless servers on the Internet by introducing a “protected mode,” starting with version 3.0.0. When Redis listens on all interfaces (0.0.0.0), and a password hasn’t been configured, this protected mode will only respond to requests from the loopback interface (127.0.0.1) and rejects external requests. An administrator can manually disable this mode by running the following Redis command: config set protected-mode no.

One small thing to note is that the official Redis docker image doesn’t appear to have the protected mode setting enabled by default. Below is an example of starting the official Docker Redis service and fetching the value of the protected-mode configuration. As we can see, this protected mode is disabled when running under Docker.

Unfortunately, our findings show that this protected mode is not a silver-bullet solution, as most unauthenticated Redis services are running versions above 3.0.0. For example, China has 3,989 unauthenticated Redis services running version 6.2.6 and 3,730 services running 3.0.504. The chart below shows the top twenty Redis versions we found without authentication, broken down by country, all of which use protected-mode capable versions.

Below is a table of the top 10 (total across all regions) Redis versions found online without authentication enabled. We can see a fairly broad number of versions used in the wild, but a majority of these unauthenticated servers are running v6.2.6 (with 5,791) and v3.0.504 (with 5,781).

Finding Abused Redis

After reviewing known issues for the Redis server on GitHub, issue #4791 caught our attention. This user reported that their Internet-accessible Redis server had several keys which contained a crontab-formatted value that attempts to run a shell script fetched from a remote server that they did not set themselves. Worried that there may be an unknown vulnerability, the user requested assistance from the developers. A second user chimed in and confirmed that their server had been hit by something similar.

As it turns out, attackers have been using a lesser-known technique to trick Redis servers into writing data to arbitrary files for years. This technique, known only as the “Redis Unauthorized Access Vulnerability,” turns Redis’s runtime configuration system against itself.  This attack is quite simple.

First, we must understand that Redis has a mechanism to store the in-memory data on disk to survive a restart or failure. The default settings for these can be found in the Redis configuration:

What might come as a surprise is that these configuration values can also be set remotely at runtime using the Redis messaging protocol. The general idea behind this exploitation technique is to configure Redis to write its file-based database to a directory containing some method to authorize a user (like adding a key to “.ssh/authorized_keys”), or start a process (like adding a script to /etc/cron.d), for example,

The above would tell the Redis service to write the contents of memory to the file “/root/.ssh/authorized_keys”. The goal is to write the value of the “backup1” key into this file, hoping that the SSH service ignores any binary data that Redis may add to the file, and accepts clients using this SSH key.

Note: The authorized_keys file in SSH allows a user to use their SSH public key in place of local password-based authentication. The SSH server will accept a login without checking the local password if you own the private side of any public key defined in $USER/.ssh/authorized_keys.

It wasn’t just the two users in the aforementioned GitHub issue that had been hit with such a thing. We found indicators that someone attempted this attack against tens of thousands of unauthenticated Redis servers on the Internet. We do not have the ability to say whether the attacks were successful, but we can report on the leftover artifacts that these attempts leave behind.

We found that this specific attempt used several Redis keys prefixed with the string “backup”  to store malicious crontab entries into the file “/var/spool/cron/root” using the following Redis commands:

This command deletes all data from the Redis server. This is done in order for the contents of the following key/values to be written as close to the beginning of the database file as possible.

This sets the key “backup1” to contain a crontab job that will fetch and run this init.sh script from a remote server every four minutes.

This sets Redis’s data directory to the system’s “/var/spool/cron/” directory;  the default directory where individual user crontabs are found and executed by the “crond” process.

This sets Redis’s data filename to “root,”; meaning that the contents of the database will be stored in “/var/spool/cron/root,” i.e., the root user’s individual crontab file.

Finally, this tells the Redis server to sync the memory to disk, and if successful, the contents of  “/var/spool/cron/root” will be read and executed by the crontab process.

This “init.sh” script (as seen in the “SET backup1” command) was still accessible and could be downloaded and viewed at the time of writing. This script, when executed, includes many nefarious actions, including

• Stops and disables any running security-related process
• Stops and disables any running system monitoring processes
• Removes and purges all system and security-related log files, including shell histories (e.g., .bash_history).
• Adds a new SSH key to the root user’s authorized_keys file
• Disables the iptables firewall
• Installs several hacking and scanning tools such as “masscan”
• Installs and runs the cryptocurrency mining application XMRig

Note: In the case that this script has been purged from the Internet, we have created a gist that is a copy of the original which the reader can find here.

Using the most recent list of unauthenticated Redis services running on TCP port 6379, we ran a one-time scan that simply looked for the existence of the key “backup1” (note: we did not fetch the value) on every host. We found that out of the 31,239 unauthenticated Redis servers in this list, 15,526 hosts had this key set. This means that someone attempted the attack described in this section on over 49% of known unauthenticated Redis servers on the Internet.

Still, this does not mean that there are over 15k compromised hosts. It is improbable that the conditions needed for this vulnerability to be successful are in place for every one of these hosts. The primary reason many of these attempts will fail is that the Redis service needs to be running as a user with the proper permissions to write to the directory “/var/spool/cron” (i.e., root). Although, this can be the case when running Redis inside a container (like docker), where the process might see itself running as root and allow the attacker to write these files. But in this case, only the container is affected, not the physical host.

Once again, we can only determine whether this attack was attempted, not if it was successful.

What can be done?

Redis is a fantastic product. But it’s also meant to be run on internal infrastructure, not public-facing servers. Administrators must ensure that external entities can not access your Redis instances in any way. A good first step is reading Redis’s documentation on Redis security. But in summary:

• Enable client authentication in your Redis configuration file.
• Configure Redis to only run on internal-facing network interfaces.
• Disable the “CONFIG” command by running ‘rename-command CONFIG “”’ to avoid configuration abuse.
• Configure your firewall only to accept Redis connections from trusted hosts.

Network and host administrators should also constantly monitor what services their hosts expose to the Internet. The best way to do this is to use the free Censys search tool to see how the public views your Internet-connected hosts. If you have a large network or you are unsure of your asset inventory, Censys Attack Surface Management is a product that can automatically find hosts owned by you and alert you on any unauthenticated or misconfigured Redis servers.