Last week, DigiCert disclosed a compliance issue affecting 83,267 certificates due to a Domain Control Verification (DCV) bug, prompting requirements for their revocation. This has significant implications for organizations, who must quickly reissue these certificates or face potential service disruptions and loss of user trust.
At the time of writing, Censys observed 26,373 impacted certificates still in use on public-facing hosts – nearly 99% of which have been revoked. This incident serves as another reminder of the ongoing difficulties in balancing compliance and response time for organizations, following closely on the heels of the recent Entrust removal.
In this blog, we examine the details of the incident, as well as Censys’s perspective on its scope, and identify the top companies and industries affected.
Executive Summary
- On July 28, 2024, DigiCert reported a compliance issue due to a bug in their Domain Control Verification (DCV) process. The Certification Authority Browser Forum (CA/B) requires DigiCert to revoke affected certificates within 24 hours to maintain compliance as a trusted Certificate Authority.
- The incident has caused significant disruption, with organizations scrambling to replace certificates within the stringent deadline. Notably, Alegeus, a financial tech company in the healthcare sector, sought a court order to delay the revocation process, citing severe operational impact and potential disruption to client access.
- DigiCert currently ranks as the fourth most active trusted Certificate Authority (CA) in our dataset, and reported that 83,267 unique certificates were impacted by this issue. Using Censys data, we determined that 33,201 of these were in use on the public web as of July 30. Within a week, the number of affected in-use certificates decreased by nearly 7,000, and now sits at 26,373 as of August 6.
- Our analysis of the registered domains shows that the Technology and Telecommunications industries were most impacted in terms of numbers of affected certificates are affected.
- As of August 6, 2024, 98.82% of unique leaf certificates affected by this incident that we observe in use on public-facing hosts have been revoked. DigiCert stated that all impacted customers are required to re-issue affected certificates by this Friday, August 9 at 20:30 UTC.
- This incident comes not too long after Entrust was removed as a trusted Certificate Authority last month, highlighting the ongoing hurdles that organizations face in effective certificate management. Striking a balance between compliance requirements for Certificate Authorities and giving users sufficient time and resources to respond adequately to such incidents continues to be a tough challenge.
- Detection with Censys:
- Censys ASM customers can identify services that are actively using an impacted certificate within their workspaces by querying for a new low-severity risk named “Certificate Affected by DigiCert July 2024 Revocation Incident”
- Users of our Search feature can find hosts with affected certificates by querying labels=digicert-revoked-dcv. To refine the results for your specific domains, adjust this query to filter on services.tls.certificates.leaf_data.names. Please note that this label detection has just been deployed, so it may take up to 48 hours from now to fully propagate.
The Issue
On July 28th, 2024, DigiCert issued an initial incident report revealing that a subset of their certificates were non-compliant due to a bug in their Domain Control Verification (DCV) process. Consequently, DigiCert had 24 hours to revoke and reissue all affected certificates to maintain compliance with root certificate authorities like Google and Mozilla.
DigiCert operates two primary systems for issuing certificates: a high-volume system for handling requests from CDNs and cloud providers and a low-volume system for users requiring higher configuration management. This issue affected only certificates issued from the low-volume system, while the high-volume system remained unaffected. Given that the low-volume system caters to customers needing precise configurations, these certificates are likely used in highly secure environments.
The root cause of the issue stemmed from DigiCert’s process for verifying domain name ownership, known as Domain Control Validation (DCV). DCV can be conducted through various methods such as email, HTTP, or whois, but the bug in this case was related to DNS-based DCV.
In DNS-based DCV, customers must create a specific (and temporary) DNS record with a value that matches the content requested by the CA (DigiCert). For example, DigiCert’s DCV documentation outlines two methods for this process:
- Option 1
- Go to your DNS provider’s site and create a new CNAME record
- In the hostname field (or equivalent), enter _dnsauth.
- In the record type field (or equivalent), select CNAME.
- In the target host field (or equivalent), enter [random_value].dcv.digicert.com to point the CNAME record to dcv.digicert.com.
- Option 2
- Go to your DNS provider’s site and create a new CNAME record.
- In the hostname field (or equivalent), enter the random value that you copied from your CertCentral account.
- In the record type field (or equivalent), select CNAME.
- In the target host field (or equivalent), enter dcv.digicert.com to point the CNAME record to dcv.digicert.com
Let’s say our DigiCert random challenge is _FOOBAR. For “Option 1,” the resulting DNS record would appear as follows:
_dnsauth IN CNAME _FOOBAR.dcv.digicert.com. ; _dnsauth.example.com.
Whereas “Option 2” would look slightly different:
_FOOBAR IN CNAME dcv.digicert.com. ; FOOBAR.example.com.
DigiCert can then make a DNS request for the given domain name, verify the correct value in the response, and thus confirm that the user requesting the certificate owns that domain name.
This process works well, but it’s important to note the preceding underscore in the random value (challenge) we were given. The reason for this is described in a recommendation from an IETF draft outlining best practices for DCV using DNS:
“The RECOMMENDED format is application-specific underscore prefix labels. Domain Control Validation records are constructed by the provider by prepending the label “_<PROVIDER_RELEVANT_NAME>-challenge” to the domain name being validated (e.g. “_foo-challenge.example.com”). The prefixed “_” is used to avoid collisions with existing hostnames.” – draft-ietf-dnsop-domain-verification-techniques-02 Section 5.1.1
Essentially, this means that using a preceding underscore helps prevent conflicts where the generated random string for DCV might inadvertently match an existing DNS record. While underscores technically violate DNS RFC standards, most DNS resolvers accept them. Underscores are typically not used for general resolution but are instead reserved for automation systems to process.
DigiCert also described this feature in a blog post on July 29th, 2024, in a similar fashion. While the underscore is merely a recommendation in the IETF draft, it is a requirement for certificate authorities. However, DigiCert’s low-volume certificate issuing system did not enforce this requirement.
The idea that this feature is only in place to avoid already-existing DNS records may understate other, potentially larger, security risks when the underscore is not used. A user on Bugzilla highlighted in a reply to DigiCert that this underscore check is not just to prevent potential name collisions. In certain circumstances, a malicious user could exploit this oversight to create a certificate for a domain they do not own.
“The actual reason for the underscore is so that services which allow users to create DNS records at subdomains (e.g. dynamic DNS services) can block users from registering subdomains starting with an underscore and be safe from unwanted certificate issuance.“ – Comment #10
This user pointed out a scenario where a product or service allows users to control a DNS record of a parent domain name they do not own. For instance, consider a product that lets users register accounts, with each account or username being mapped to a controllable DNS record. An example provided involved Dynamic DNS services, where users can automatically update certain public records based on their currently allocated DHCP address, using the username as a sub-domain. For example, if a user gets a DHCP lease of 192.168.1.2, they can update $username.some-dyndns-service.com to point to that IP.
If that were the case, you could technically register a new account on that service with the username being the random challenge string provided by DigiCert. Then, you could update the DNS record $username.some-dyndns-service.com with a CNAME value of dcv.digicert.com. This would allow you to pass the DCV check from DigiCert, leading to the issuance of a valid certificate.
One individual pointed out that noip[.]com may be a good example of a target of this issue, which could potentially allow an attacker to obtain a valid certificate for ddns[.]net.
“No-IP (https://www[.]noip.com) allows the creation of arbitrary labels under their domains like ddns[.]net, enabling users to easily create subdomains such as <some_arbitrary_string>.ddns.net and map them as CNAMEs to any target.” – Comment #15
And this definitely seems to be the case; in the following two screenshots, we show that we can create an arbitrary subdomain of our choosing with the CNAME value of dcv.digicert.com without an error when excluding the preceding underscore, but fails when we do include an underscore.
 |
 |
| Without the preceding underscore. |
With the preceding underscore. |
 |
Technically, an attacker could have created valid certificates for any of the pre-configured domain names listed on the left (ddns[.]net, gotdns[.]ch) |
A service that implements such a feature should automatically deny any new account or records with an underscore in the prefix, effectively treating them like reserved keywords (like noip does above). However, since DigiCert was not enforcing the underscore prefix, any DNS modification access on such systems could be exploited to obtain a valid certificate for the parent domain name.
In short, the underscore is crucial not only for avoiding name collisions but also for providing an extra layer of security for services that allow users to modify protected domains. It enables filtering mechanisms to prevent (un)authorized users from exploiting DNS records to obtain valid certificates.
Real-world implications.
This incident has created quite a stir amongst individuals and businesses, causing chaos in engineering teams around the world all racing to get their hosts updated with new certificates. So much so that one financial tech company specializing in the healthcare industry, Alegeus, filed a court order to obtain a temporary restraining order on DigiCert in an attempt to delay the revocation.
“Recertifying the Alegeus Websites requires Alegeus to coordinate with each of its Clients and cannot be completed for all of the Alegeus Websites by the DigiCert prescribed deadline. Accordingly, Alegeus requested three days to complete recertification of the Alegeus Websites. DigiCert has refused.” – Page 3.8
This is basically asserting that the short deadline was not enough time for Alegeus and figured one of the best ways to extend the deadline was to make it a legal matter. Their reasoning seems legitimate:
“If DigiCert revokes the Security Certificates for the Alegeus Websites, it will cause Alegeus severe and irreparable damages in an amount to be determined at trial.” – Page 9
And if we read the plea from the original email chain sent from Alegeus to DigiCert on July 29th, 2024, we can solidify their reason even more by reading how dire the situation is:
“Revoking the certificates as proposed will result in significant access issues to our clients and their participants, which means that millions of plan participants and consumers will not be able to access funds reserved for and designed to pay for health care benefits until the access issues are resolved.” – Page 3
This basically states that without the proper extension of the deadline, real human lives may be affected. I think a good point to be made here is that while the certificate authorities themselves may have no issue with these tight deadlines, their customers do not always have the same luxury – incidents like this can be quite troubling for the consumer to deal with. Obviously, the best way for this to have gone is to have not happened at all, but here we are.
References
Censys’s Perspective
DigiCert’s Market Share
According to DigiCert’s original incident notice, this affected “approximately 0.4% of the applicable domain validations” – let’s put that number in a real-world context.
DigiCert is a major Certificate Authority (CA). They rank as the fourth most active trusted CA in our dataset, having issued approximately 54.1 million certificates, representing nearly 7% of all currently trusted issued certificates.
On Mozilla’s Bugzilla forum, a DigiCert representative attached CSV files with links to 83,267 leaf certificates reportedly affected by this DCV issue.
On July 30th, a few days after the incident was disclosed, we identified over 455,999 distinct IPv4 IP addresses (spread across both physical and virtual hosts) utilizing 33,201 of these 83,267 affected certificates.
A week later, on August 6, 26,373 affected certificates remained online, a drop of 6,828 certificates within a week.
The table below summarizes these weekly statistics from our perspective.
| Date |
Physical IPv4 Hosts |
Physical and Virtual Hosts |
Active Leaf Certificates |
Unique Domains |
| 7/30/2024 |
456002 |
4085959 |
33201 |
30528 |
| 8/6/2024 |
241343 |
3599368 |
26373 |
24693 |
Industry and Company Impact:
We analyzed the domain names associated with these active certificates observed on July 30, categorizing them by the number of unique hosts and classifying the corresponding companies and industries for certificates with ten or more hosts (totaling 2,262 unique domain names). Note that only registered domains were considered, not subdomains (ex: www.censys.com and community.censys.com are both evaluated as “censys.com”).
| Company |
Domain Count |
Host Count |
Unique Leaf Certificates |
% of Total Affected Leaf Certs |
| Vodafone |
19 |
6165 |
2178 |
16.10% |
| Bloomberg |
6 |
3685 |
630 |
4.66% |
| ABB |
5 |
601 |
250 |
1.85% |
| Yahoo |
6 |
6283 |
223 |
1.65% |
| Hitachi |
1 |
260 |
215 |
1.59% |
| Poste Italiane |
1 |
332 |
200 |
1.48% |
| Atlas World Group |
1 |
315 |
196 |
1.45% |
| Bouygues Telecom |
5 |
425 |
148 |
1.09% |
| Qt Group |
1 |
191 |
146 |
1.08% |
| Engie |
3 |
463 |
142 |
1.05% |
Vodafone stands out with over 2,100 affected unique leaf certificates, indicating they likely heavily rely on DigiCert for their certificate needs. Bloomberg, ABB, Yahoo, and Hitachi are the top five companies impacted of those we could properly attribute to an owner.
| Industry |
Domain Count |
Host Count |
Unique Leaf Certificates |
% of Total Affected Leaf Certs |
| Technology |
916 |
184276 |
3001 |
22.23% |
| Telecommunications |
65 |
81456 |
2652 |
19.64% |
| Financial Services |
192 |
21259 |
1017 |
7.53% |
| Media |
102 |
15308 |
1007 |
7.46% |
| Healthcare & Pharmaceuticals |
156 |
7679 |
708 |
5.24% |
| Government |
80 |
3987 |
687 |
5.09% |
| Utilities |
22 |
1694 |
568 |
4.21% |
| Insurance |
54 |
5817 |
565 |
4.18% |
| Education |
133 |
12997 |
510 |
3.78% |
| Manufacturing |
61 |
7218 |
495 |
3.67% |
| Retail |
80 |
12495 |
421 |
3.12% |
| Transportation & Logistics |
50 |
1712 |
314 |
2.33% |
| Entertainment |
95 |
13127 |
286 |
2.12% |
| Consumer Goods |
51 |
4689 |
139 |
1.03% |
| Advertising & Marketing |
35 |
1922 |
107 |
0.79% |
| Engineering & Construction |
16 |
741 |
106 |
0.79% |
| Recruitment & Employment |
14 |
1202 |
99 |
0.73% |
| Aerospace |
3 |
178 |
96 |
0.71% |
| Legal |
6 |
172 |
88 |
0.65% |
| Automotive |
30 |
783 |
82 |
0.61% |
The data also reveal that the Technology and Telecommunications industries are the most affected by this revocation incident. Technology has the highest number of unique leaf certificates, at 3,001, and Telecommunications follows closely at 2,652. Digital certificates are commonly used in these industries to secure large networks and communication infrastructure.
The Financial Services, Media, and Healthcare & Pharmaceuticals industries are among the most impacted. Our research on the Entrust incident in July, when Entrust was removed as a trusted certificate authority, identified Financial Services as the most affected industry.
How many of these certificates are revoked?
According to DigiCert’s updated incident page, all affected certificates must be replaced by August 9, 2024, at 20:30 UTC. As of now, 98.82% of the affected leaf certificates have been revoked, which is a significant increase from July 30, 2024, when only about 5% had been revoked.
| Date |
Total Active Leaf Certificates |
# Revoked |
% Revoked |
| July 30 |
33201 |
1652 |
4.98% |
| August 6 |
26373 |
26061 |
98.82% |
What can be done?
- Impacted customers should have received a notification from DigiCert, and are recommended to take immediate action to reissue and install new certificates through their DigiCert CertCentral account.
- Censys ASM customers will see a new low-severity risk named “Certificate Affected by DigiCert July 2024 Revocation Incident” related to this issue that can be used to detect affected certificates in their workspaces.
- Search customers can use the query labels=digicert-revoked-dcv to find hosts that are running one of the affected certificates. To narrow results to filter on your domains, modify this query and replace “yourdomain.com”. Note that we have just pushed this label detection out so it may take over 48 hours from the time of writing for it to propagate.
References:
