How the removal of Entrust from Chrome’s Root Store will Affect the Internet

Introduction

HTTPS and Certificate Authorities

One of the structural pillars of the modern internet is that establishing a connection from your computer to your favorite website is incredibly secure, at least from the perspective of the underlying traffic. No person or organization can pry into your internet habits just by observing the traffic between your computer and the website you’re visiting. All of this is transparent to the end-user. Still, the underlying technologies are advanced but hinge on defining circles of trust and public-key infrastructure.

The Role of SSL/TLS Certificates

If your company, FooBar, Inc., owns the domain foobar.com and wants users to feel secure when interacting with your online widgets, you would obtain an SSL/TLS certificate from a trusted Certificate Authority (CA). After installing it on your web server, your website can provide users with an HTTPS service. This ensures that operating systems and web browsers can verify that the website they are communicating with is indeed foobar.com, not an imposter.

Trust and Verification

Technically, anyone can generate a TLS certificate for any domain and set it up on their infrastructure. However, unless the certificate is issued by a recognized and trusted Certificate Authority (CA), and vetted by those CAs, then browsers and operating systems will flag any attempt to use that certificate with security warnings, deterring users from proceeding.

Becoming a Trusted CA

Operating as a CA means issuing certificates that are trusted by the internet at large. This requires your root certificate to be recognized by major organizations and vendors that maintain a database of trusted root CAs. At a minimum, you must be trusted by Google, Mozilla, Apple, and Microsoft, which takes enormous work.

Compliance and Certification

To be considered by any of these governing organizations, you must go through a virtual gauntlet of compliance audits and certifications. Each governing organization has its minimum requirements:

• Google Chrome
• Mozilla (Firefox, Thunderbird, etc..)
• Apple (Safari, MacOS, etc…)
• Microsoft (Edge, Windows, etc.…)

Meeting these standards is not a one-time effort but a continuous process of maintaining trust with root store maintainers and the internet community. This includes publicly responding to any incidents, whether discovered internally or reported by the community. Both Google and Mozilla require that a trusted CA disclose and respond publicly to incidents using Bugzilla.

“Chrome Root Program Participants MUST publicly disclose and/or respond to incident reports in Bugzilla, regardless of perceived impact.”  – Google.

“At a minimum, CA operators MUST promptly report all incidents to Mozilla in the form of an Incident Report”  – Mozilla.

Google’s Decision to Remove Entrust

Entrust is a privately held software and hardware company that has been issuing SSL certificates since 1997. While the exact proportion of their business reliant on the SSL certificate market is unclear, it is well-known that a significant portion of the financial industry depends solely on Entrust-issued certificates

In an announcement, Google stated that Entrust was no longer meeting their expectations for maintaining security standards. As a result, Google decided that certificates from several Entrust root CAs would no longer be trusted by default for any certificates issued after October 31, 2024.

“TLS server authentication certificates validating to the following Entrust roots whose earliest Signed Certificate Timestamp (SCT) is dated after October 31, 2024, will no longer be trusted by default.”

• CN=Entrust Root Certification Authority – EC1
• CN=Entrust Root Certification Authority – G2
• CN=Entrust.net Certification Authority (2048)
• CN=Entrust Root Certification Authority
• CN=Entrust Root Certification Authority – G4
• CN=AffirmTrust Commercial
• CN=AffirmTrust Networking
• CN=AffirmTrust Premium
• CN=AffirmTrust Premium ECC

This change means that Chrome will not trust new certificates issued by these Entrust root CAs after the specified date. Existing certificates will continue to function until they expire or are revoked.

The primary reason for this seems to be rooted in Google’s view of Entrust’s lack of communication around incident reports; however, not much detail was provided about the specific reasons, but we may be able to gain some insights by reviewing some of Entrust’s past issues. In the next section, we will review the bug report/incident that seems to have been the final straw in Google’s decision to remove Entrust from the Chrome root certificate store.

The Unwinding Trust of Entrust

The Report: BUGZILLA ISSUE #1883843: EV TLS Certificate cPSuri Missing

An engineer from Google reported to Paul van Brouwershaven (Director of Technology at Entrust) that over twenty thousand certificates were missing a required field in their certificates. In turn, these certificates should have been treated as invalid and marked as “mis-issued.” – Entrust stated that this was a mistake due to a misinterpretation of recently amended requirements.

The real drama started after a user commented that Entrust was still mis-issuing certificates [Comment #3]. Had Entrust fixed its mistake? At the moment, it was uncertain.

Entrust then states that there was no plan to stop the current issuance or revoke the already-existing certificates since they viewed the whole thing as a non-problem, and the revocation could have more negative consequences than the alternative. The effect of this was to draw their own conclusions and interpretations. [Comment #10]

“We firmly believe that not revoking, and the continuation of issuance does not harm the security, reliability and compatibility of the ecosystem or the users in some other way.”

A few more comments came in to remind Entrust that they cannot simply interpret and disregard the policies that are put in place [Comment #5, Comment #6, Comment #7] with an action item to stop the current issuing of new certificates and to revoke those that have already been issued, citing the Mozilla CA wiki which states that “In misissuance cases, a CA should almost always immediately cease issuance from the affected part of its PKI.”

Entrust once again responds and doubles down on their reluctance to revoke these mis-issued certificates [Comment #10], this time stating that this would affect thousands of customers, much of which are not managed through any sort of automated process, and that reissuing these certificates would mean an unmanageable amount of work. In a final act of defiance, the Entrust representative had this to say:

“If Mozilla and the community expect Certificate Authorities (CAs) to allocate their valuable time to matters driven solely by principle, we may be moving in the wrong direction and inadvertently hindering ecosystem progress.”

Pushback from the community.

A software engineer responded quite curtly, expounding on the nature of self-regulating ecosystems, such as a certificate authority failing to abide by these self-regulating rules; in lieu of self-regulation, the only option (obviously) is a state-run solution that would hinder progress even more. [Comment #11]. Another user chimed in shortly after to remind Entrust that the “P” in PKI means “Public”, and that Entrust services that “P” in “Public”, and not the other way around. [Comment #13]

And like the phoenix rising, a representative from the Google Chrome Root Program (the original reporter of this issue) wrote an extensive comment with some very specific asks.

They were unhappy with the response from Entrust; one problem was the format of the response itself –  it lacked some key details that are required by the Common CA Guidelines, such as defining the number of affected assets, a key fact that was only discovered in the responses, and not the initial report. They also scolded Entrust for demonstrating questionable judgment and disregarding the concerns of the community in an event that was both unnecessary and avoidable. [Comment #19].

For nine hours on March 18th, 2024, the comment section of this issue went silent, only to be broken by a short, pointed response by Entrust:

“We have has stopped issuing miss-issued certificates and fixed the EV certificate profile.

All impacted customers will be advised that their certificates will be revoked.

We will create a delayed revocation bug and will follow up on other questions in the next few days”.  – Entrust / [Comment #20]

And so, on March 18th at 3 PM PST, Entrust resigned to their insistence. But little did they know that this was only the beginning of a wave of inquiries.

“So you managed to stop the mis-issuance in less than 10 hours, but only once Google intervene? Before then, you simply didn’t want to stop?” asked JR Moir in Comment #21 – a valid question that was probably on the minds of many at the time. JR continues: “Trust in Entrust should be removed.” The first shot of revolt rang across the grassy field of internet security.

Over the next few weeks, Entrust interacted with the community in an attempt to map out all of the affected certificates. It was not until April 4th that Entrust responded directly to the Chrome Root Program’s initial comments and questions in Comment #35. Entrust stated that while their response was not aligned with the established expectations of a CA, the intent was to act in the best interests of the customers or, as they put it, the web ecosystem. Entrust then answered each question in the comment, insisted that this was just an oversight, and acknowledged their wrongdoing.

“In retrospect, we acknowledge that we should have stopped issuance of EV certificates without the policy qualifier upon confirmation of the issue, and then followed up to pursue what we saw as a possible oversight in the EV Guidelines.” – Comment #35 / Answer to Question 1 in Comment #19

Comment #35 is full of valuable information giving insight into all of the missteps that Entrust made along this journey, including details around their entire issuance process; great amounts of effort were put into the explanations of some of the assumptions that were made that led to these decisions; They did not include the required field in new certificates because they had wrongly thought that it was an error in the CA guidelines text itself:

“…we believed that the missing “certificatePolicies:policyQualifiers:qualifier:cPSuri” in the issued EV certificates was due to an error in the EV Guidelines.” – Comment #35 (Entrust)

Six days later, Entrust posted an updated Incident report for this issue (Comment #41) that included a full timeline of events. A response that only evoked the ire of the community once again. In Comment #44, a user writes that they hope that other Certificate Authorities will view this entire incident as a ‘lesson to learn from’ after noting that even after three weeks, not even half of the “affected customers” have been fixed, and only half of the mis-issued certificates had been revoked. And then another call for the removal of Entrust:

“These repeat of failings show clearly that Entrust cannot be trusted in the WebPKI and needs to be removed.” – JR Moir (Comment #44)

In response, the Entrust representative stated that their subscriber base is different from other (possibly larger) CAs – possibly referring to the class of customer that Entrust caters to. Entrust continues:

“We hope that members of the community who have experience in the field where Entrust operates understand our challenges and that we are doing everything we can to make our subscribers better prepared and more agile for security incidents” – Entrust (Comment #46)

The thread goes silent after this, aside from a spattering of comments here and there with stories and tales of Entrust’s historical failings. The latest comment was by the Mozilla CA program manager, Ben Wilson, requesting additional action items from Entrust, which was issued a new ticket ID #1901270: Entrust: Action Items from June 2024 Report.

This was only the latest drama that Entrust was embroiled in. In March 2024, it was reported that Entrust mis-issued 1,176 certificates because its tooling did not pick up on some non-compliant certificates. While this was not a big deal in itself, the certificates affected by this mistake were not revoked in a timely manner, resulting in another incident being reported to the public.

Some commenters even made reference to the sparse nature of the follow-up and pointed out that Entrust seems to be blaming customers for their inability to revoke these certificates quickly. A month prior to this, it was reported that Entrust was using SHA-1 (an outdated hashing algorithm) to sign responses over OCSP and had a long back and forth with the community at large, answering frustrated questions as to why they had not discovered this information earlier.

Several more examples of the public losing faith in Entrust exist, most of which involve delayed responses and slow certificate revocation times. These compounded issues are why Google has decided to no longer support Entrust-issued certificates in its root certificate store.

With over two decades in the industry and their (unfortunately) poorly received recent history, it looks like it may be the beginning of the end for Entrust’s certificate authority. Today, it’s Google; tomorrow, it may be Mozilla and Microsoft.

This is kind of a big deal; a decision that shouldn’t be taken lightly. If anything, there are lessons to be learned here. The erosion of trust between a public service and its community can have dire consequences.

Signed Certificate Timestamps

SCTs are digital receipts issued by Certificate Transparency (CT) logs when a CA submits a certificate for logging. These timestamps provide proof that the certificate has been recorded in a public log, allowing for public auditing. When a Certificate Authority (CA) issues a new certificate, it sends it to one or more CT logs. Once the CT log receives the certificate and creates a record of it, the CT log generates an SCT, which includes the following:

• A timestamp indicating when the certificate was logged
• The hash of the certificate
• A unique identifier for the log
• A digital signature created by the log’s private key.

This SCT is then most often embedded directly into the certificate. The following is an example of how one could view the SCT information from the domain name “www.censys.com”:

The output of this command will look similar to the following:

• Version: Indicates the version of the SCT; here it is version 1.
• Timestamp: The date and time when the SCT was issued..
• Signature: The cryptographic signature that validates the SCT.
• Log ID: A unique identifier for the log that issued the SCT, represented as a sequence of hexadecimal bytes

To find the log provider for a given “Log ID”, (where details about this issuance can be found) the following commands can be used:

$ LOGID=$(echo 3F:17:4B:4F:D7:22:47:58:94:1D:65:1C:84:BE:0D:12:ED:90:37:7F:1F:85:6A:EB:C1:BF:28:85:EC:F8:64:6E | \
  sed -s 's/://g' | \
  xxd -p -r | base64); \
 curl -s https://www.gstatic.com/ct/log_list/v3/all_logs_list.json | \
 jq ".operators[].logs[]|select(.log_id == \"$LOGID\")|."

{
  "description": "Let's Encrypt 'Oak2024H2' log",
  "log_id": "PxdLT9ciR1iUHWUchL4NEu2QN38fhWrrwb8ohez4ZG4=",
  "key": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE13PWU0fp88nVfBbC1o9wZfryUTapE4Av7fmU01qL6E8zz8PTidRfWmaJuiAfccvKu5+f81wtHqOBWa+Ss20waA==",
  "url": "https://oak.ct.letsencrypt.org/2024h2/",
  "mmd": 86400,
  "state": {
    "usable": {
      "timestamp": "2022-11-30T17:00:00Z"
    }
  },
  "temporal_interval": {
    "start_inclusive": "2024-06-20T00:00:00Z",
    "end_exclusive": "2025-01-20T00:00:00Z"
  }
}

This output above tells us we can validate this SCT by utilizing the “Let’s Encrypt ‘Oak2024H2’ log” server.

In the case of Google Chrome, an extra step (or constraint) is added to the verification process for any given certificate. This constraint checks to see if this SCT was generated on or after October 31, 2024, and if the constraint has been met, the certificate will be rejected.

The Impact on the Internet

We can examine the Censys host and certificate datasets to understand Entrust’s standing among other Certificate Authorities and the potential impact that this may have if not addressed promptly.

Breakdown of Certificate Authorities

In total, Google has trusted certificates from 464 distinct authorities, amounting to 7,559,114,541 (seven billion) issued certificates. Among these, Entrust, Inc. ranks 22nd, with 3,999,827 (0.05%) Google-trusted certificates, and AffirmTrust ranks 78th, with a whopping 37,646.