Introduction
HTTPS and Certificate Authorities
One of the structural pillars of the modern internet is that establishing a connection from your computer to your favorite website is incredibly secure, at least from the perspective of the underlying traffic. No person or organization can pry into your internet habits just by observing the traffic between your computer and the website you’re visiting. All of this is transparent to the end-user. Still, the underlying technologies are advanced but hinge on defining circles of trust and public-key infrastructure.
The Role of SSL/TLS Certificates
If your company, FooBar, Inc., owns the domain foobar.com and wants users to feel secure when interacting with your online widgets, you would obtain an SSL/TLS certificate from a trusted Certificate Authority (CA). After installing it on your web server, your website can provide users with an HTTPS service. This ensures that operating systems and web browsers can verify that the website they are communicating with is indeed foobar.com, not an imposter.
Trust and Verification
Technically, anyone can generate a TLS certificate for any domain and set it up on their infrastructure. However, unless the certificate is issued by a recognized and trusted Certificate Authority (CA), and vetted by those CAs, then browsers and operating systems will flag any attempt to use that certificate with security warnings, deterring users from proceeding.
Becoming a Trusted CA
Operating as a CA means issuing certificates that are trusted by the internet at large. This requires your root certificate to be recognized by major organizations and vendors that maintain a database of trusted root CAs. At a minimum, you must be trusted by Google, Mozilla, Apple, and Microsoft, which takes enormous work.
Compliance and Certification
To be considered by any of these governing organizations, you must go through a virtual gauntlet of compliance audits and certifications. Each governing organization has its minimum requirements:
Meeting these standards is not a one-time effort but a continuous process of maintaining trust with root store maintainers and the internet community. This includes publicly responding to any incidents, whether discovered internally or reported by the community. Both Google and Mozilla require that a trusted CA disclose and respond publicly to incidents using Bugzilla.
“Chrome Root Program Participants MUST publicly disclose and/or respond to incident reports in Bugzilla, regardless of perceived impact.” – Google.
“At a minimum, CA operators MUST promptly report all incidents to Mozilla in the form of an Incident Report” – Mozilla.
Google’s Decision to Remove Entrust
Entrust is a privately held software and hardware company that has been issuing SSL certificates since 1997. While the exact proportion of their business reliant on the SSL certificate market is unclear, it is well-known that a significant portion of the financial industry depends solely on Entrust-issued certificates
In an announcement, Google stated that Entrust was no longer meeting their expectations for maintaining security standards. As a result, Google decided that certificates from several Entrust root CAs would no longer be trusted by default for any certificates issued after October 31, 2024.
“TLS server authentication certificates validating to the following Entrust roots whose earliest Signed Certificate Timestamp (SCT) is dated after October 31, 2024, will no longer be trusted by default.”
This change means that Chrome will not trust new certificates issued by these Entrust root CAs after the specified date. Existing certificates will continue to function until they expire or are revoked.
The primary reason for this seems to be rooted in Google’s view of Entrust’s lack of communication around incident reports; however, not much detail was provided about the specific reasons, but we may be able to gain some insights by reviewing some of Entrust’s past issues. In the next section, we will review the bug report/incident that seems to have been the final straw in Google’s decision to remove Entrust from the Chrome root certificate store.
The Unwinding Trust of Entrust
An engineer from Google reported to Paul van Brouwershaven (Director of Technology at Entrust) that over twenty thousand certificates were missing a required field in their certificates. In turn, these certificates should have been treated as invalid and marked as “mis-issued.” – Entrust stated that this was a mistake due to a misinterpretation of recently amended requirements.
The real drama started after a user commented that Entrust was still mis-issuing certificates [Comment #3]. Had Entrust fixed its mistake? At the moment, it was uncertain.
Entrust then states that there was no plan to stop the current issuance or revoke the already-existing certificates since they viewed the whole thing as a non-problem, and the revocation could have more negative consequences than the alternative. The effect of this was to draw their own conclusions and interpretations. [Comment #10]
“We firmly believe that not revoking, and the continuation of issuance does not harm the security, reliability and compatibility of the ecosystem or the users in some other way.”
A few more comments came in to remind Entrust that they cannot simply interpret and disregard the policies that are put in place [Comment #5, Comment #6, Comment #7] with an action item to stop the current issuing of new certificates and to revoke those that have already been issued, citing the Mozilla CA wiki which states that “In misissuance cases, a CA should almost always immediately cease issuance from the affected part of its PKI.”
Entrust once again responds and doubles down on their reluctance to revoke these mis-issued certificates [Comment #10], this time stating that this would affect thousands of customers, much of which are not managed through any sort of automated process, and that reissuing these certificates would mean an unmanageable amount of work. In a final act of defiance, the Entrust representative had this to say:
“If Mozilla and the community expect Certificate Authorities (CAs) to allocate their valuable time to matters driven solely by principle, we may be moving in the wrong direction and inadvertently hindering ecosystem progress.”
Pushback from the community.
A software engineer responded quite curtly, expounding on the nature of self-regulating ecosystems, such as a certificate authority failing to abide by these self-regulating rules; in lieu of self-regulation, the only option (obviously) is a state-run solution that would hinder progress even more. [Comment #11]. Another user chimed in shortly after to remind Entrust that the “P” in PKI means “Public”, and that Entrust services that “P” in “Public”, and not the other way around. [Comment #13]
And like the phoenix rising, a representative from the Google Chrome Root Program (the original reporter of this issue) wrote an extensive comment with some very specific asks.
They were unhappy with the response from Entrust; one problem was the format of the response itself – it lacked some key details that are required by the Common CA Guidelines, such as defining the number of affected assets, a key fact that was only discovered in the responses, and not the initial report. They also scolded Entrust for demonstrating questionable judgment and disregarding the concerns of the community in an event that was both unnecessary and avoidable. [Comment #19].
For nine hours on March 18th, 2024, the comment section of this issue went silent, only to be broken by a short, pointed response by Entrust:
“We have has stopped issuing miss-issued certificates and fixed the EV certificate profile.
All impacted customers will be advised that their certificates will be revoked.
We will create a delayed revocation bug and will follow up on other questions in the next few days”. – Entrust / [Comment #20]
And so, on March 18th at 3 PM PST, Entrust resigned to their insistence. But little did they know that this was only the beginning of a wave of inquiries.
“So you managed to stop the mis-issuance in less than 10 hours, but only once Google intervene? Before then, you simply didn’t want to stop?” asked JR Moir in Comment #21 – a valid question that was probably on the minds of many at the time. JR continues: “Trust in Entrust should be removed.” The first shot of revolt rang across the grassy field of internet security.
Over the next few weeks, Entrust interacted with the community in an attempt to map out all of the affected certificates. It was not until April 4th that Entrust responded directly to the Chrome Root Program’s initial comments and questions in Comment #35. Entrust stated that while their response was not aligned with the established expectations of a CA, the intent was to act in the best interests of the customers or, as they put it, the web ecosystem. Entrust then answered each question in the comment, insisted that this was just an oversight, and acknowledged their wrongdoing.
“In retrospect, we acknowledge that we should have stopped issuance of EV certificates without the policy qualifier upon confirmation of the issue, and then followed up to pursue what we saw as a possible oversight in the EV Guidelines.” – Comment #35 / Answer to Question 1 in Comment #19
Comment #35 is full of valuable information giving insight into all of the missteps that Entrust made along this journey, including details around their entire issuance process; great amounts of effort were put into the explanations of some of the assumptions that were made that led to these decisions; They did not include the required field in new certificates because they had wrongly thought that it was an error in the CA guidelines text itself:
“…we believed that the missing “certificatePolicies:policyQualifiers:qualifier:cPSuri” in the issued EV certificates was due to an error in the EV Guidelines.” – Comment #35 (Entrust)
Six days later, Entrust posted an updated Incident report for this issue (Comment #41) that included a full timeline of events. A response that only evoked the ire of the community once again. In Comment #44, a user writes that they hope that other Certificate Authorities will view this entire incident as a ‘lesson to learn from’ after noting that even after three weeks, not even half of the “affected customers” have been fixed, and only half of the mis-issued certificates had been revoked. And then another call for the removal of Entrust:
“These repeat of failings show clearly that Entrust cannot be trusted in the WebPKI and needs to be removed.” – JR Moir (Comment #44)
In response, the Entrust representative stated that their subscriber base is different from other (possibly larger) CAs – possibly referring to the class of customer that Entrust caters to. Entrust continues:
“We hope that members of the community who have experience in the field where Entrust operates understand our challenges and that we are doing everything we can to make our subscribers better prepared and more agile for security incidents” – Entrust (Comment #46)
The thread goes silent after this, aside from a spattering of comments here and there with stories and tales of Entrust’s historical failings. The latest comment was by the Mozilla CA program manager, Ben Wilson, requesting additional action items from Entrust, which was issued a new ticket ID #1901270: Entrust: Action Items from June 2024 Report.
This was only the latest drama that Entrust was embroiled in. In March 2024, it was reported that Entrust mis-issued 1,176 certificates because its tooling did not pick up on some non-compliant certificates. While this was not a big deal in itself, the certificates affected by this mistake were not revoked in a timely manner, resulting in another incident being reported to the public.
Some commenters even made reference to the sparse nature of the follow-up and pointed out that Entrust seems to be blaming customers for their inability to revoke these certificates quickly. A month prior to this, it was reported that Entrust was using SHA-1 (an outdated hashing algorithm) to sign responses over OCSP and had a long back and forth with the community at large, answering frustrated questions as to why they had not discovered this information earlier.
Several more examples of the public losing faith in Entrust exist, most of which involve delayed responses and slow certificate revocation times. These compounded issues are why Google has decided to no longer support Entrust-issued certificates in its root certificate store.
With over two decades in the industry and their (unfortunately) poorly received recent history, it looks like it may be the beginning of the end for Entrust’s certificate authority. Today, it’s Google; tomorrow, it may be Mozilla and Microsoft.
This is kind of a big deal; a decision that shouldn’t be taken lightly. If anything, there are lessons to be learned here. The erosion of trust between a public service and its community can have dire consequences.
Signed Certificate Timestamps
SCTs are digital receipts issued by Certificate Transparency (CT) logs when a CA submits a certificate for logging. These timestamps provide proof that the certificate has been recorded in a public log, allowing for public auditing. When a Certificate Authority (CA) issues a new certificate, it sends it to one or more CT logs. Once the CT log receives the certificate and creates a record of it, the CT log generates an SCT, which includes the following:
- A timestamp indicating when the certificate was logged
- The hash of the certificate
- A unique identifier for the log
- A digital signature created by the log’s private key.
This SCT is then most often embedded directly into the certificate. The following is an example of how one could view the SCT information from the domain name “www.censys.com”:
echo | openssl s_client \
-connect www.censys.com:443 2>/dev/null | \
openssl x509 -text -noout | \
grep -A 11 "Signed Certificate Timestamp"
The output of this command will look similar to the following:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 3F:17:4B:4F:D7:22:47:58:94:1D:65:1C:84:BE:0D:12:
ED:90:37:7F:1F:85:6A:EB:C1:BF:28:85:EC:F8:64:6E
Timestamp : May 21 05:16:30.739 2024 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:21:00:EC:76:D9:73:19:C7:94:12:EE:60:28:
D7:0B:71:F6:7F:33:60:ED:47:3F:B1:CD:E4:26:59:83:
21:B0:AD:99:5C:02:20:24:98:7F:9B:DC:E5:FD:E8:A9:
1E:B9:25:F7:5E:0C:15:ED:B7:38:2B:5F:C4:68:FB:26:
A9:E9:03:60:D1:CF:F5
- Version: Indicates the version of the SCT; here it is version 1.
- Timestamp: The date and time when the SCT was issued..
- Signature: The cryptographic signature that validates the SCT.
- Log ID: A unique identifier for the log that issued the SCT, represented as a sequence of hexadecimal bytes
To find the log provider for a given “Log ID”, (where details about this issuance can be found) the following commands can be used:
$ LOGID=$(echo 3F:17:4B:4F:D7:22:47:58:94:1D:65:1C:84:BE:0D:12:ED:90:37:7F:1F:85:6A:EB:C1:BF:28:85:EC:F8:64:6E | \
sed -s 's/://g' | \
xxd -p -r | base64); \
curl -s https://www.gstatic.com/ct/log_list/v3/all_logs_list.json | \
jq ".operators[].logs[]|select(.log_id == \"$LOGID\")|."
{
"description": "Let's Encrypt 'Oak2024H2' log",
"log_id": "PxdLT9ciR1iUHWUchL4NEu2QN38fhWrrwb8ohez4ZG4=",
"key": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE13PWU0fp88nVfBbC1o9wZfryUTapE4Av7fmU01qL6E8zz8PTidRfWmaJuiAfccvKu5+f81wtHqOBWa+Ss20waA==",
"url": "https://oak.ct.letsencrypt.org/2024h2/",
"mmd": 86400,
"state": {
"usable": {
"timestamp": "2022-11-30T17:00:00Z"
}
},
"temporal_interval": {
"start_inclusive": "2024-06-20T00:00:00Z",
"end_exclusive": "2025-01-20T00:00:00Z"
}
}
This output above tells us we can validate this SCT by utilizing the “Let’s Encrypt ‘Oak2024H2’ log” server.
In the case of Google Chrome, an extra step (or constraint) is added to the verification process for any given certificate. This constraint checks to see if this SCT was generated on or after October 31, 2024, and if the constraint has been met, the certificate will be rejected.
The Impact on the Internet
We can examine the Censys host and certificate datasets to understand Entrust’s standing among other Certificate Authorities and the potential impact that this may have if not addressed promptly.
Breakdown of Certificate Authorities
In total, Google has trusted certificates from 464 distinct authorities, amounting to 7,559,114,541 (seven billion) issued certificates. Among these, Entrust, Inc. ranks 22nd, with 3,999,827 (0.05%) Google-trusted certificates, and AffirmTrust ranks 78th, with a whopping 37,646.
The numbers are different if we look at the currently valid certificates, meaning certificates that have not been revoked or expired. As of June 28th, 2024, Google trusts 747,659,643 valid certificates issued by 244 distinct certificate authorities. Entrust, Inc. ranks #20 with 571,469 currently valid certificates, and AffirmTrust ranks #120 with 625.
![](https://censys.com/wp-content/uploads/2024/07/ENTRUST_02.png)
These numbers reflect the total number of certificates issued, not necessarily those currently in use. Determining the number of in-use and active certificates is impossible, as it would require access to every organization or individual computer. However, using current and historical scan data, Censys can provide data on the number of Entrust-issued certificates observed on the public Internet.
Entrust on the Internet
In the original post, Google lists nine different root CA issuing authorities owned by Entrust and links to each corresponding entry on crt.sh; From here, we can note the SHA-256 certificate fingerprint from each of these and search for hosts that have one of these nine fingerprints in their certificate chains:
Issuing CA |
SHA-256 Fingerprint |
CN=Entrust Root Certification Authority – EC1 |
02ed0eb28c14da45165c566791700d6451d7fb56f0b2ab1d3b8eb070e56edff5 |
CN=Entrust Root Certification Authority – G2 |
43df5774b03e7fef5fe40d931a7bedf1bb2e6b42738c4e6d3841103d3aa7f339 |
CN=Entrust.net Certification Authority (2048) |
6dc47172e01cbcb0bf62580d895fe2b8ac9ad4f873801e0c10b9c837d21eb177 |
CN=Entrust Root Certification Authority |
73c176434f1bc6d5adf45b0e76e727287c8de57616c1e6e6141a2b2cbc7d8e4c |
CN=Entrust Root Certification Authority – G4 |
db3517d1f6732a2d5ab97c533ec70779ee3270a62fb4ac4238372460e6f01e88 |
CN=AffirmTrust Commercial |
0376ab1d54c5f9803ce4b2e201a0ee7eef7b57b636e8a93c9b8d4860c96f5fa7 |
CN=AffirmTrust Networking |
0a81ec5a929777f145904af38d5d509f66b5e2c58fcdb531058b0e17f3f0b41b |
CN=AffirmTrust Premium |
70a73f7f376b60074248904534b11482d5bf0e698ecc498df52577ebf2e93b9a |
CN=AffirmTrust Premium ECC |
bd71fdf6da97e4cf62d1647add2581b07d79adf8397eb4ecba9c5e8488821423 |
Or, in Censys search terms, we can look for these hashes in the search field to identify hosts.
services.tls.certificates.chain.fingerprint = {
02ed0eb28c14da45165c566791700d6451d7fb56f0b2ab1d3b8eb070e56edff5,
43df5774b03e7fef5fe40d931a7bedf1bb2e6b42738c4e6d3841103d3aa7f339,
6dc47172e01cbcb0bf62580d895fe2b8ac9ad4f873801e0c10b9c837d21eb177,
73c176434f1bc6d5adf45b0e76e727287c8de57616c1e6e6141a2b2cbc7d8e4c,
db3517d1f6732a2d5ab97c533ec70779ee3270a62fb4ac4238372460e6f01e88,
0376ab1d54c5f9803ce4b2e201a0ee7eef7b57b636e8a93c9b8d4860c96f5fa7,
0a81ec5a929777f145904af38d5d509f66b5e2c58fcdb531058b0e17f3f0b41b,
70a73f7f376b60074248904534b11482d5bf0e698ecc498df52577ebf2e93b9a,
bd71fdf6da97e4cf62d1647add2581b07d79adf8397eb4ecba9c5e8488821423
}
Industries and Organizations Affected
At the time of writing, 876,681 physical and virtual hosts were utilizing a certificate issued by one of the nine Entrust CAs. While this may not be an exciting number, we can look into the certificate leaves to understand what organizations and industries these certificates are in.
We analyzed each registered domain name found in the leaf certificates issued by these nine Entrust CAs. To the best of our ability, we then categorized the domains by the number of unique hosts currently found on the internet. We further classified the associated companies and industries for certificates with ten or more hosts (5,285 domain names). Here, we do not include sub-domains, only the actual, registered domain names (“www.censys.com” becomes “censys.com”)
![](https://censys.com/wp-content/uploads/2024/07/ENTRUST_03.png)
The vast majority of Entrust certificates issued online are in the Financial Services (254,654 hosts and 19,151 unique leaf certificates), Technology (136,408 hosts, 9,609 leaf certificates), Entertainment (63,884 hosts, and 1,121 leaf certificates), and Government industries (58,490 hosts, and 8,044 leaf certificates)
However, the number of leaf certificates per industry paints a slightly different picture. “Professional Services” goes from #9 to #3 when we examine the number of leaf certificates – this indicates that while the “Entertainment” industry is found on many hosts, it covers a limited number of companies or domain names. However, we can also see that the financial services industry stays at #1 in terms of both the number of hosts and unique leaf certificates. In other words, lots of financial companies are relying on Entrust.
Top 20 Industries by number of hosts and leaf certificates.
Organizations
We can also examine the specific organizations that make up these industries to understand better who will be affected by Entrust’s broken relationship with Google. Again, we separate the statistics by the number of hosts and unique leaf certificates found with an Entrust issuer on the Internet.
Here, we can see very different results based on how you order the data. The Disney company has Entrust-issued certificates on over 57k unique real and virtual hosts but only 636 unique leaf certificates. A quick Censys Search will reveal why this is happening.![](https://censys.com/wp-content/uploads/2024/07/image3.png)
Many hosts we see Entrust-issued certificates for “*.disney.com” are found in Akamai, a Content Delivery Network that many medium to large-sized businesses use to cache and serve their data. While Disney only has 636 certificates, they have been found on over 57k “virtual” hosts in our scan data.
Disney ranked #1 for the number of hosts, and had 40 domain names across 636 unique leaf certificates, the majority of which were for disney.com and disney.com-related domains. Below are the top twenty Disney-related domains with an Entrust-issued certificate and corresponding host and leaf statistics.
However, Ernst & Young had only ten domain names that we categorized with 6,200 unique leaf certificates issued by Entrust on 16,063 hosts.
![](https://censys.com/wp-content/uploads/2024/07/ENTRUST_07.png)
Once again, most of these ey.com certificates are found in Akamai. However, a good portion of them are also running in Microsoft’s autonomous system, indicating that Ernst & Young runs a lot of its infrastructure in Azure.
![](https://censys.com/wp-content/uploads/2024/07/image2.png)
But these host counts can be very deceiving because a single certificate can contain many different names it is authoritative over; for example, this single Ernst & Young certificate on the left is for both domain names “ey.com” and “eyclienthub.com”, so if you’re breaking down statistics by the registered domain name, this certificate on a single host would be counted twice; In other words, it is probably better to look at the number of unique leaf certificates rather than the raw number of hosts to estimate the level of effort it would take to fix this problem.
What can be done?
- While currently trusted certificates will not be affected by this change, if an individual or organization is using Entrust now and still needs certificates for the foreseeable future, they must plan to obtain their new certificates from another trusted CA and update the infrastructure accordingly.
- If you think you may be affected by this change, this search query can be modified to find any hosts that Entrust issues for your domain (just replace “yourdomain.com: with your own domain name).
- Censys ASM customers can access a new low-priority risk named “Entrust Issued Certificate,” which will automatically notify customers when an Entrust-issued certificate is found in their environment.