Pulse Connect Secure: A View from the Internet

Introduction

Pulse Connect Secure is a low-cost and widely-deployed SSL VPN solution for remote and mobile users. Over the years, researchers have found several significant vulnerabilities in the server software, some even resulting in the active exploitation of critical infrastructure by malicious threat actors. In April of 2021, CISA released a report detailing some of these activities, which included exploiting several unknown (at the time) vulnerabilities and resulted in swift action from Ivanti, the Pulse Connect Secure software developer.

In this post, we will attempt to paint a picture of the current state of vulnerable Pulse Connect Secure devices that are still running on the Internet. In the first section, we will cover seven different security advisories released by Pulse Secure, while the second half of the article will go over the techniques we used to fingerprint, identify, and version these services.

Pulse Connect Security Advisories: An Internet View

In total, Censys has found 30,266 Pulse Connect Secure hosts running on the internet. One of the easiest ways to find these running using Censys is to search for a specific URI that can be found in the HTTP response body of a Pulse Connect Secure web service.

services.http.response.body:  `/dana-na/`

Of those exposed, 4,460 hosts have been identified as running a software version vulnerable to one or more of the seven security advisories we reviewed. This post will detail each advisory, its hosts, and associated services.

Breakdown by Security Advisory

The following is a breakdown of each security advisory we studied and the number of hosts that are vulnerable to that particular advisory. Note that these numbers will differ from the total number of vulnerable hosts since a single host can be susceptible to multiple security advisories.

While SA44858 wins out for the highest number of hosts on the Internet (3,528 hosts), it is by far not the worst of the vulnerabilities, given that the CVEs associated with SA44858 require a valid account to leverage the exploit. The bigger problem comes with SA44784, which includes CVE-2021-22893, an authentication-bypass vulnerability that allows an unauthenticated user to perform remote code execution on Pulse Connect Secure devices. Censys has observed vulnerable versions on 1,841 Internet-facing hosts. To give a little more context, in April of 2021, Rapid7 and Mandiant reported that this CVE was used to actively install malware that harvested credentials from vulnerable Pulse Connect hosts.

Breakdown by Country (Top 20)

In the following diagram, we can see the ratio of versions we identified that were vulnerable to a security advisory (teal), versus ones that are not vulnerable (blue). The United States has the most significant total number of Pulse Connect installations with 8,575 hosts, but only 12% have a version vulnerable to one or more of the seven advisories we analyzed. On the other hand, France only has 1,422 Pulse Connect devices on the Internet, but a little over 30% of them are running a version susceptible to one of the seven advisories we analyzed.

Breakdown by Autonomous System (Top 20)

Breakdown by Version (Top 20)

Most Pulse Connect versions deployed are not vulnerable to any of the security advisories we looked into. The most popular is 9.1.14.18105 (9.1R14), with 3,177 hosts running this version, closely followed by 9.1.15.18393 (9.1R15), with 2,319 hosts; both of which are not vulnerable.

Fingerprinting Pulse Connect Secure

Introduction

Historically, when software is upgraded and released,  we found that the new versions have visible differences from the previous versions. With enough time and enough data, those (sometimes nuanced) differences can be found and (hopefully) used to confidently say what version of the software is running on a server.

For example, these slight differences can frequently be seen in the HTTP response body when the software runs over HTTP, although it is sometimes ambiguous at first sight. And more often than not, these differences are introduced by external tools that aid in the build and deployment process (such as continuous integration (CI) systems or webpack for Javascript). These systems will auto-generate filenames in formats that can change from one build to the next when the content has been changed, ultimately creating visible artifacts that uniquely identify the software.

In many cases, we are limited to using the raw host data we collect during our scans (banners and protocol responses). Because of this, we frequently resort to devising more hands-off techniques and methods to identify the software and services we’re touching. More importantly, we develop these techniques to keep our actual long-term interaction with a running service down to a minimum.

Identification

In the Censys search screenshot for a Pulse Connect service above, several segments of the HTTP response has been highlighted in yellow. Indicators like these are usually a combination of a filename suffixed with a bunch of random characters, probably generated via a program like “sha256sum,” which computes a SHA256 message digest of the file’s contents. Having this information for only one host isn’t very useful, but when you take a step back and look at the aggregate of all hosts on the Internet, patterns begin to materialize. As we can see, Pulse contains several examples of these uniquely rendered artifacts.

Our primary goal is to associate these random-looking filenames with a specific software version. But first, we must determine whether we can even use the above data to identify specific versions by seeking out all other instances of this service running on the Internet to compare the data. In the case of Pulse Connect, a common element found in the HTTP response of the service is the string “/dana-na/” (found both in the body and located within the HTTP redirect chain). So the following Censys search query should work for identifying most Pulse Connect services:

services.http.response.body: `/dana-na/`

Once this core characteristic is found, we need to understand how many unique filename arrangements of this data exist on the Internet. If every single host includes an entirely different filename pattern, or if every host has the same filename pattern, then it is unlikely this artificat can be associated with a specific version. We are looking for a somewhat even distribution of unique filename “hashes” across several hosts (the more hosts, the better).

Censys Enterprise Data customers may be familiar with our BigQuery interface, which allows users to write SQL to investigate hosts and services. To do this, using BigQuery is likely one of the easiest ways to answer the question of this filename question. The following query will parse out these (seemingly) random filenames (file_hash) and count the number of hosts associated with each (host_count).

SELECT
  REGEXP_EXTRACT(SAFE_CONVERT_BYTES_TO_STRING(service.http.response.body),
    ".*/dana-na/css/ds_([a-f0-9]{64}.js).*") AS file_hash,
  COUNT(DISTINCT host_identifier.ipv4) AS host_count
FROM
  `censys-io.universal_Internet_dataset.current` main,
  UNNEST(services) service
WHERE
  SAFE_CONVERT_BYTES_TO_STRING(service.http.response.body) LIKE '%/dana-na/%'
GROUP BY
  file_hash
ORDER BY
  host_count DESC

At the time of writing, This query found 159 unique instances of this “/dana-na/css/ds_” prefixed filename. We can then export the results of this query and get a visual representation of the distribution of filenames:

In the above chart, the x-axis represents the filename “hash” found in the HTTP response body, and the y-axis represents the number of hosts that contained that string. This means that multiple devices include the same filename, which indicates that there may be a pattern that could result in specific versions.

The next step is to determine if there is a way to get the actual software version to associate them with these filenames. When we were fingerprinting GitLab, we wrote scripts to pull down every tagged image in GitLab’s public Docker repository to make this association. In the case of Pulse, this task wasn’t as straightforward.

Since VPN services (commonly) require administrative-level access on a client host to function, more often than not, the VPN client software will be available to download from the server. Pulse Connect is no different, and after some quick googling, users have discovered that the client can be downloaded by requesting the path “/dana-cached/hc/HostCheckerInstaller.osx”.

This “HostCheckerInstaller.osx” file is an executable that includes the running software version embedded inside a preferences (PLIST) configuration file. We need to download this executable and search the contents to extract this information. For example, the following command will fetch the binary from a Pulse Connect server and print out the embedded version string.

~$ curl -sk https://HOST:PORT/dana-cached/hc/HostCheckerInstaller.osx   \
   -o HostCheckerInstaller.osx && strings HostCheckerInstaller.osx | \
   grep -A1 -m1 'version'

We want to download this binary from three or more Pulse Connect servers for every unique hash we found with our SQL query. The idea here is to verify from more than one source if there is a direct correlation between these filenames and the specific version of Pulse Connect. If multiple hosts have the same filename pattern, but the binary we downloaded from each host gives us a different version string, this would be considered a failed correlation.

To accomplish this, we need to combine the data we already have (a list of the unique filename hashes), the Censys search command-line utility, and a little shell scripting. So as an example, if we wanted to find out what version is associated with the filename: “/dana-na/css/ds_d0208775377750cb7eeb6341e70188f7cd46d4fbd2c3e99aa4549d0890d28af8.js”>/font>

We grab a random host in Censys using the following search term:

services.http.response.body: "/dana-na/css/ds_d0208775377750cb7eeb6341e70188f7cd46d4fbd2c3e99aa4549d0890d28af8.js"

For those results, find the port and service where this pattern matched, and download the VPN client from the server like so:

~$ curl -sk https:///dana-cached/hc/HostCheckerInstaller.osx \
  -o HostCheckerInstaller.osx

Finally, search through the downloaded binary by looking for the string “<key>version</key>”, the value of which will be the software version.

~$ strings HostCheckerInstaller.osx | grep -A1 '<key>version</key>'
<key>version</key>
<string>22.2.16.21349</string>

After running this over a few hosts with the same filename pattern, we can say with some confidence that “ds_d0208775377750cb7eeb6341e70188f7cd46d4fbd2c3e99aa4549d0890d28af8” is Pulse Connect version 22.2.16 build 21349.

We created this shell script that will iterate through each unique filename from our SQL query response, issue a Censys Search request using the command-line utility to look for the filename pattern in an HTTP response body, then choose four random results and download the “HostCheckerInstaller.osx” binary from the hosts. The trick is to format the output file so it is easy to map the filename hash from the response body to the host from which we downloaded it.

Once this script has finished running, we should have a directory of files prefixed with the filename pattern we are looking for and suffixed with the IP address from where the script downloaded the file. We can then iterate over each file and grab the embedded version string:

~$ for file in bins/*; do echo $file; strings $file | grep -A1 -m1 '<key>version</key>'; echo "" ;done
bins/00c5cc1ea977e9c13feb1f8853d034de26b4d079f85005ae990114345940095b__192.168.3.64
<key>version</key>
<string>9.1.5.5460</string>

bins/00c5cc1ea977e9c13feb1f8853d034de26b4d079f85005ae990114345940095b__192.168.100.4
<key>version</key>
<string>9.1.5.5460</string>

We can then use this information to generate a report of all the found versions and validate that each unique filename pattern we were looking for has a unique corresponding version string in the downloaded file. First, we create a small script to generate a CSV containing the version data for each host:

#!/usr/bin/env bash
HASHFILE=$1
BINDIR=$2

echo "host,path,version"
while read line; do
    INPATH_STRIPPED=$(echo $line | sed 's/\/dana-na\/css\/ds_//g'| cut -d'.' -f1)
    files=$(find $BINDIR -type f -name "*${INPATH_STRIPPED}*")
    for file in $files; do
        ip=$(echo $file | awk -F'__' '{print $2}')
        v=$(strings $file | grep -A1 -m1 "<key>version</key>" | tail -n 1 | sed 's/<string>//g' | sed 's/<\/string>//g')
        echo "$ip,$line,$v"
    done
done < $HASHFILE

An example run of the above script using the data we pulled down looks like the following:

~$ bash make_report.sh hashes.txt bins | tee versions.csv
host,path,version
10.1.2.3,/dana-na/css/ds_d0208775377750cb7eeb6341e70188f7cd46d4fbd2c3e99aa4549d0890d28af8.js,9.1.14.18105
10.6.1.1,/dana-na/css/ds_d0208775377750cb7eeb6341e70188f7cd46d4fbd2c3e99aa4549d0890d28af8.js,9.1.14.18105
10.1.2.1,/dana-na/css/ds_d0208775377750cb7eeb6341e70188f7cd46d4fbd2c3e99aa4549d0890d28af8.js,9.1.14.18105
10.1.1.5,/dana-na/css/ds_d0208775377750cb7eeb6341e70188f7cd46d4fbd2c3e99aa4549d0890d28af8.js,9.1.14.18105
10.4.9.2,/dana-na/css/ds_d0208775377750cb7eeb6341e70188f7cd46d4fbd2c3e99aa4549d0890d28af8.js,9.1.14.18105

Then we can create the aggregate report that tells us how many hosts we checked for a single path. As stated above, we download the client from four hosts for each unique path, so we expect each unique path+version combination to have four matching lines.

~$ cat versions.csv | awk -F, '{print $2”,”$3}' | sort | uniq -c | sort -nr 
      4 /dana-na/css/ds_fd9c25fc621eb26eb49c16c7028d1bd90cf5f7c1aab102ffc8b7659912a8b20a.js,9.1.9.9189
      4 /dana-na/css/ds_f92149ac6a768f91e748db511ab310d84e7f84512bb3f272c69c94d0d68757db.js,9.1.4.5035
      4 /dana-na/css/ds_f2aacd8bd5408b7c3c1d4633e178ad10de68367f03b366fd194c70a51fbd8621.js,8.1.14.59737

Once we manually verify that each unique path is directly associated with a version (which it has), we can create a vulnerability matrix for the security advisories we are most interested in.

To do this part of the job, we must track each vulnerability we want to report on, for this post, we analyzed seven prominent Pulse Connect security advisories (SA), which each have one or more CVEs associated:

But before we continue, one may notice that the version numbers found in Pulse security advisories don’t exactly match up with the version numbers found on the devices themselves. For example, here is an excerpt from a Pulse Connect Security advisory:

Here, we see that “8.3R1” is affected by a particular vulnerability; this version format must first be converted to “8.3.1” to match the version format we find on the hosts. But if we look closer, we are missing the final set of digits found in our data, which is the “build number”. To obtain the build number for a particular Pulse Connect version, we use our Google skills to find the release notes. For example, if we wanted to find the build number for version “8.3R1”, we would Google search for “Pulse 8.3R1”, and scroll around for release notes.

Google results for “Pulse 8.3R1”	Release Notes

In this example, we can say the vulnerable version “8.3R1” has the build number “55339”, and can be formatted as “8.3.1.55339” to match what is found on the hosts. This build number is most important for determining the patched versions of the Pulse Connect software.

To generate a vulnerability matrix, we created a small python script using the “semver” API to match versions for each of the seven security advisories we listed above. This script reads in our CSV of unique file paths and discovered version numbers, and adds new columns representing which security advisories that version is vulnerable to.

• You can view the contents of this small Python script here.

The above script takes two arguments, an input file and an output file. The following is an example of this script running, the input is the CSV file we generated in the last section, and the output is standard-out but also redirected to the file “vulnerability_matrix.csv.”

~$ python3 parse_version.py path_version.csv /dev/stdout | tee vulnerability_matrix.csv
path,version,SA43604,SA43877,SA44101,SA44516,SA44588,SA44784,SA44858
/dana-na/css/ds_00c5cc1ea977e9c13feb1f8853d034de26b4d079f85005ae990114345940095b.js,9.1.5.5460,False,False,False,True,True,True,True
/dana-na/css/ds_065be83c9cf5a8d6da2a66a5b9c8f7d95d2f1a02815c7b2cebdeba0b7feaedea.js,9.1.10.12179,False,False,False,False,False,True,True
/dana-na/css/ds_10636a65bffaf6e64eba54360e47f389bd3c7d398af38ab052099bc167780d61.js,9.1.14.16847,False,False,False,False,False,False,False
/dana-na/css/ds_118434847ad133161d5a347570f89c743e30f402cc6235c24f90ae0110a03098.js,9.1.10.10119,False,False,False,False,False,True,True
/dana-na/css/ds_11954f0b01a1b777450dd7be52f5e31554cc6ef3f078e4bbd1bc992e87aa369f.js,8.1.3.35989,False,True,True,False,False,False,False
/dana-na/css/ds_11aaa782e76aab152312ee9fa1e1fb6292c85162262ec05670517c63842110ba.js,9.1.11.14929,False,False,False,False,False,False,True

Here we see that for each unique filename, we have the associated version and a truth table for each of the seven security advisories we are analyzing. We then take this new CSV file and import it into a new BigQuery table using the BigQuery command-line utility:

~$ bq load --source_format CSV \
  --replace --autodetect \
  <bigquery-project-id>:<database>.pulse-vulnerability-matrix \
  vulnerability_matrix.csv

The above command will automatically generate a BigQuery database schema and load the data. The resulting table should look similar to the following screenshot:

With all of this in place, we can now easily use this newly created table to join the data in our global Internet dataset to report on the hosts and services vulnerable to these security advisories. The following SQL query will give us an overview of each security advisory and the number of services on the Internet they affect.

WITH
  pulse_hosts AS (
  SELECT
    host_identifier.ipv4,
    REGEXP_EXTRACT(SAFE_CONVERT_BYTES_TO_STRING(service.http.response.body), ".*(/dana-na/css/ds_[a-f0-9]{64}.js).*") AS file_hash,
  FROM
    `censys-io.universal_Internet_dataset.current` main,
    UNNEST(services) service
  WHERE
    service.service_name = 'HTTP'
    AND SAFE_CONVERT_BYTES_TO_STRING(service.http.response.body) LIKE '%/dana-na/%')
SELECT
  COUNT(*) TOTAL,
  COUNTIF(pulse.SA43604 = TRUE) SA43604,
  COUNTIF(pulse.SA43877 = TRUE) SA43877,
  COUNTIF(pulse.SA44101 = TRUE) SA44101,
  COUNTIF(pulse.SA44516 = TRUE) SA44516,
  COUNTIF(pulse.SA44588 = TRUE) SA44588,
  COUNTIF(pulse.SA44784 = TRUE) SA44784,
  COUNTIF(pulse.SA44858 = TRUE) SA44858
FROM
  `pulse_hosts`
LEFT JOIN
  `db.table.pulse-vulnerability-matrix` pulse
ON (file_hash=pulse.path)

You may also download the CSV-formatting mappings of different paths mapped to specific Pulse Connect versions.

How can Censys Help?

• Censys ASM customers have access to risks that will alert on any assets that match these criteria.
• The path to version CSV can be downloaded here.