What is the issue?
On July 9th, SolarWinds published an advisory for a critical vulnerability in Serv-U (CVE-2021-35211) that was reported to them by Microsoft. According to Microsoft, in a blog post from July 13th, the issue is being actively exploited in the wild. The vulnerability is present in the SSH component of Serv-U, which is used for secure transfers using SCP.
SolarWinds has released a hotfix for the issue, which is available to licensed customers in their support portal.
An internet analysis
Censys can see over 8,300 assumed SolarWinds SSH services exposed to the internet, based on an SSH banner fingerprint. A good number of them are in China and the US:
As of the time of writing, several services belonged to U.S. based state and local governments as well.
Shared encryption keys for SSH
While researching the issue, Censys discovered another oddity with Serv-U hosts: They frequently present the same SSH host key. Initially this was perplexing, as the hosts are spread over a diverse set of networks (no obvious commonality). When host keys are exchanged with multiple entities, they are effectively public, and allow servers to be impersonated. A realistic attack is difficult to pull off, though someone could theoretically capture and decrypt SSH/SCP traffic if they sit at a privileged location on the network and can man-in-the-middle a connection.
A Censys report from our search platform shows that there are plenty of hosts sharing the same host key that match a Serv-U SSH banner:
At the time of writing, 4,344 hosts on the internet share the same SSH host key 53b8131ea00459671cf6ce1169a7c5bacec88ec76fa9b77fdc84bc26e8a1df2b. This is similar to the Western Digital issue Censys reported on with Dan Goodin of Ars: Hosts sharing the same encryption material are easier to identify and at risk for leaking information over the wire.
Investigating the reason for key sharing
To understand in this case why this was occurring, Censys downloaded and installed a Serv-U trial, and took a look at the contents of the installation. As it would turn out, there is a default certificate and private key that ships with the installer that appears to be used in many installations around the world. We can dump the sha256 sum of the host key from the default certificate and validate this:
# pwd
/usr/local/Serv-U
# ls *.crt *.key
Serv-U-DefaultCertificate.crt Serv-U-DefaultCertificate.key
# ssh-keygen -l -f /dev/stdin <<< $(ssh-keygen -i -m PKCS8 -f /dev/stdin <<< $(openssl x509 -pubkey -noout -in Serv-U-DefaultCertificate.crt)) | awk '{print $2 "="}' | cut -c 8- | base64 -d | xxd -p | tr -d '\n' && echo
e9d8efd43200bf3780c3bff6e45601e69f05089c93a1ff36fc2e552a5da3b935
We can see the key e9d8efd43200bf3780c3bff6e45601e69f05089c93a1ff36fc2e552a5da3b935 appears as the second most popular reused host key fingerprint in the table above. At the time of writing, Censys search shows 928 devices utilizing the current shipping host keys (by fingerprint) in the SolarWinds Serv-U installer. Though this is primarily a user configuration problem, SolarWinds bears some responsibility, and can alleviate this by generating new certificates on startup, or when a fresh install is performed. These certificates are simply self-signed.
We can go a little deeper though, and drop the SSH banner component mentioned above when looking for exposed Serv-U hosts. If we do this, and search simply on the host key fingerprint, we end up finding many more hosts. This means that some hosts have hidden Serv-U in their banner information. These hosts commonly reply with a banner that looks like “SSH-2.0-2.0,” instead of a banner resembling “SSH-2.0-Serv-U_###” (where ### is a version). Comparing this search to the prior search, we found 1,311 hosts utilizing the shipping host keys, or 383 additional hosts than we found prior when we included the Serv-U SSH banner filter.
It’s more than just SSH
While researching the shared host keys, we decided to dig a little deeper into whether there were any other shared services using the certificate (.crt file) as well. If we dump the details of this certificate, we get interesting information:
$ openssl x509 -noout -text -in Serv-U-DefaultCertificate.crt -fingerprint -sha256
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 0 (0x0)
Signature Algorithm: sha1WithRSAEncryption
Issuer: C = US, ST = WI, L = Helenville, O = "Rhino Software, Inc.", OU = Software Development, CN = ftp.Serv-U.com
Validity
Not Before: May 13 17:27:39 2020 GMT
Not After : May 11 17:27:39 2030 GMT
Subject: C = US, ST = WI, L = Helenville, O = "Rhino Software, Inc.", OU = Software Development, CN = ftp.Serv-U.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:b1:71:4f:17:61:bb:11:b2:91:94:24:96:6f:72:
89:8b:f5:eb:af:12:1a:a5:c1:35:d5:18:70:51:bc:
ae:6c:1c:71:8c:8e:3d:fb:71:e3:d2:c2:cc:79:52:
c6:05:2e:8a:aa:c8:a7:27:b9:ef:c6:93:ff:c1:53:
05:00:4a:2a:41:e1:36:8b:de:08:71:7b:aa:86:17:
f2:cd:27:92:a6:ac:69:ea:8c:f2:9c:92:b4:c3:7e:
ea:d3:d2:c7:20:0d:00:76:42:2f:66:8c:ae:ce:88:
51:6b:a4:d3:45:c5:e9:72:e9:d2:43:e7:dc:5e:15:
ad:6e:57:d9:54:ea:9e:df:ea:41:e7:5c:0f:64:c7:
9e:b5:62:48:48:4f:00:d6:bc:f3:93:3c:17:d8:c5:
3e:22:d5:a0:5e:66:74:be:9a:62:40:0a:a0:30:34:
04:c0:48:e8:c9:3d:76:81:87:f1:54:41:a3:d8:ba:
6d:5a:c6:b5:dc:ff:bb:46:04:1c:0d:df:35:d6:f1:
b9:c5:5b:a0:ee:4f:ca:3d:ef:0c:f2:fe:fe:f3:16:
0e:8e:a1:6f:f6:08:27:45:f3:5e:b6:16:52:b7:d7:
c8:0e:8a:d6:ec:48:b6:54:f7:d7:f9:b8:e2:f1:3f:
b5:77:d5:ab:00:04:45:7d:df:79:42:7b:31:7f:02:
3d:73
Exponent: 65537 (0x10001)
Signature Algorithm: sha1WithRSAEncryption
5b:75:e7:49:03:35:f6:03:94:fc:d5:9d:81:a6:f3:12:8f:81:
74:c8:89:7d:1f:f0:d2:ec:50:f2:b1:18:be:53:ea:68:9d:7f:
63:29:56:4f:f3:12:23:15:da:52:ae:0d:e0:c1:e5:5c:c3:c4:
56:df:4c:c0:56:5b:2b:02:b9:a5:93:bc:c3:7d:c9:74:1a:22:
61:c3:42:42:62:ea:91:bc:81:5f:ee:8c:fc:18:bb:1a:d2:ea:
06:9e:a7:e2:06:17:af:61:6e:57:7a:49:fd:a7:2b:38:b1:53:
26:d1:be:9d:e8:f5:26:3c:fa:18:c6:2b:35:ee:b8:f5:a5:d0:
97:86:cb:ff:17:40:38:9e:40:34:18:73:ff:7b:80:f9:69:64:
03:f6:07:b9:4d:76:48:a6:aa:07:f5:ce:fb:cd:0d:e5:d5:7a:
2c:94:8a:cf:b7:bc:0a:20:4c:da:72:db:a6:5b:ed:26:a0:c0:
d7:0c:21:1c:0f:3e:80:96:e2:30:80:dd:9f:2b:b2:e2:63:a1:
f4:83:02:b2:c6:4c:bd:58:a2:c0:55:68:99:26:c9:18:ab:79:
bf:10:1b:06:37:9f:eb:e2:1d:9e:3c:5c:67:18:8f:56:65:6d:
a3:fe:cd:fa:0d:7f:0d:69:67:ae:da:db:18:7b:58:49:5d:19:
5f:3c:37:f9
SHA256 Fingerprint=C2:AE:DB:13:CC:A1:37:9A:F5:7C:4E:09:A8:7F:20:ED:34:69:49:20:34:A6:B8:A0:FB:8E:26:D0:44:80:84:79
The unique fingerprint of the certificate looks interesting. Can we find additional services on the internet through Censys search that share this fingerprint? As it would turn out, we can. Searching on the certificate fingerprint, we can find 3,293 hosts that utilize this certificate across FTP, HTTP and SMTP. These are additional services that Serv-U utilizes for transfer, and they’re all similarly at risk.
Why does it matter?
According to Microsoft, “If Serv-U’s SSH is exposed to the internet, successful exploitation would give attackers [the] ability to remotely run arbitrary code with privileges, allowing them to perform actions like install and run malicious payloads, or view and change data.” Microsoft goes on to attribute the active exploitation campaign to a group they’re referring to as DEV-0322, based out of China. This group has been primarily targeting U.S. defense and software companies, according to their blog post.
There are thousands of SolarWinds Serv-U hosts exposed to the internet. These hosts are easily discoverable because they all share common characteristics (host keys, as well as banners). Moreover, the discovery of the thousands of default SSH keys means thousands of Serv-U hosts have fragile encryption. As HD Moore put it, “it’s basically telnet at that point.” It’s clear that the issue reported by Microsoft has a far reaching impact based on the count of public-facing Serv-U hosts. An exploit will inevitably come out as researchers binary diff the patch to see what the SolarWinds researchers have fixed. It’s important that organizations quickly patch this issue to avoid a breach, as Serv-U hosts are easily discoverable.
What do I do about it?
- Apply 15.2.3 HF2 (hotfix 2) from the SolarWinds customer portal (or block internet access) to eliminate CVE-2021-35211.
- Roll your encryption keys for Serv-U if you’re using the default certificate (unrelated to CVE-2021-35211, but a good practice if you are using default keys).
- Perform a forensic analysis on any Serv-U host that has exposed SSH to the internet.
- Censys ASM can help you easily identify any hosts using the default Serv-U FTP/HTTPS/SMTP certificate and close them down. Additionally, view all hosts running SSH in your environment, identify Serv-U, and close any exposed Serv-U SSH/SCP service. The Censys ASM software inventory can also help you easily identify software across your attack surface.
Resources
