User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36

Steps to reproduce:

Preliminary Incident Report

Summary

On November 5, 2024, during our internal quality assurance process, we identified the mis-issuance of a single TLS OV certificate. This certificate was issued before the completion of the corresponding domain revalidation/renewal process, violating TLS BR Section 3.2.2.4 by not using an approved domain validation method. After notifying the affected customer, the mis-issued certificate was revoked on November 6, 2024. As of this notification, no further instances of similar mis-issuance have been found. A complete incident report, including root cause analysis and corrective measures, will be provided by Friday, November 15, 2024.

Complete Incident Report

Summary

On November 5, 2024, during our internal quality assurance process, we identified the mis-issuance of a single TLS OV certificate. This certificate was issued before the completion of the corresponding domain revalidation/renewal process, violating TLS BR Section 3.2.2.4 by not using an approved domain validation method. After notifying the affected customer, the mis-issued certificate was revoked on November 6, 2024.

Impact

One customer with one TLS certificate revoked on November 6, 2024

Timeline All times are UTC

2024-11-4 16:00 Received a TLS renewal request for the affected certificate that required revalidation of the associated domain. This revalidation required manual steps to approve the request.

2024-11-4 20:00 Registration Agent assigned to process the request erroneously approved the renewal before completion of domain revalidation.

2024-11-5 15:00 Registration Supervisor reviewing the request discovered the manual processing error which is the domain was approved for renewed certificate issuance prior to completion of domain validation.

2024-11-5 15:00 The error was explained to the Registration Agent and the agent was removed from the processing pool.

2024-11-5 19:20 Registration Supervisor notified the affected customer that request was inadvertently approved and to not request issuance of renewed certificate.

2024-11-5 19:42 The customer had an automated process that requested and received the affected certificate associated with the renewal thereby resulting in a mis-issuance.

2024-11-6 14:29 Registration Supervisor discovered the mis-issuance and revoked the mis-issued certificate, and notified the customer.

2024-11-8 16:00 Registration Agent was retrained on the domain validation approval process.

Root Cause Analysis

We offer several methods, including multiple versions of our Application Programming Interface (API), for requesting TLS certificate renewal. In this case, the customer used an older API version, which lacks the technical controls available in the newer versions.

The Registration Agent approved a pending TLS certificate renewal request without verifying the completion of the domain validation process. In addition to this manual processing error, the mis-issuance occurred because our system lacks a technical control to prevent the approval of certificate renewal requests pending domain validation when such requests are received via an older version of our API. This is the first instance of such a human error, likely due to the limited number of customers using this older API.

Our daily QA process detected the error and confirmed that no other TLS certificates were approved before the completion of domain validation.

To prevent similar mis-issuance in the future, we have thoroughly retrained our Registration Agents. Additionally, to eliminate the risk of recurrence, we will implement relevant technical controls in the older version of our API.

Lessons Learned

A technical control must be added to the older version of the API to prevent the approval of requests that have not completed the domain validation process.

What went well

The issue was promptly detected and the mis-issued TLS certificate was revoked within 24 hours.

Action Items

| Action Item | Kind | Due Date |
| Retrain Registration Agents | Prevent | 2024-11-08 |
| Add technical control to older version of API | Remediate | 2025-02-08 |

Appendix

Details of affected certificates

https://crt.sh/?id=15226683712

We plan to implement a technical control by February 2025 as part of our pending remediation action. We will provide monthly updates until the task is completed, with the next update scheduled for December 31, 2024.

"The Registration Agent approved a pending TLS certificate renewal request without verifying the completion of the domain validation process. "

This suggests your staff can cause issuance of a certificate even if required process like domain verification are not completed.

That seems very very bad.

Waiting 2-3 months for a fix seems worse.

This is a weak root cause analysis.

In this case, the customer used an older API version, which lacks the technical controls available in the newer versions.

So, IdenTrust has knowingly kept an unsafe issuance system around? And the remediation is to wait another 3 months to “fix” it?

This is unacceptable given the amount of trust the community has given IdenTrust.

Disabling this old API until you could add this technical control would be a small step in the right direction.

(In reply to JR Moir from comment #3)
We appreciate your perspective. To address this and prevent any potential recurrence before the implementation of a permanent systemic solution, we have enacted the following measures:

1. Comprehensive retraining of all Registration Agents on correct approval procedures.
2. Deactivation of concurrent instances of approval interfaces.
3. Implementation of a daily 100% audit of all manual domain requests until the systemic fix is in place.
   The delay in implementing the systemic code until February 2025 is required by the year-end code freeze across both our platforms and those of our customers. We assure you that this temporary solution will maintain the highest standards of security and accuracy in our certificate issuance process.

(In reply to amir from comment #4)
We appreciate your concern regarding the issuance system. However, we respectfully disagree with the interpretation that we knowingly maintained an unsafe system. As outlined in our response to comment #3, we have implemented robust additional controls to mitigate any potential risks while we work on a comprehensive, systemic solution. We remain committed to ensuring the highest standards of security and reliability in our systems.

We are on track to deploy a production fix by February 28, 2025.
Our next update will be posted by January 31, 2025

We are on track to deploy the promised fix by February 28, 2025.
Our next update will be posted no later than by February 28, 2025

Report Closure Summary

Incident description:
Issuance of one TLS certificate on November 6, 2024, without proper domain revalidation violating TLS BR Section 3.2.2.4 by not using an approved domain validation method.

Incident Root Cause(s):
An older version of an API did not include a technical mechanism to block the approval of certificate renewal requests that were still awaiting domain validation.

Remediation description:

• Rolled back the API deployment on the same day – November 6, 2024
• Revoked the mis-issued certificate within 24 hours on November 7, 2024
• Retrained Registration personnel acting on requests from this API - November 28, 2024
• Added the missing technical control to the API and successfully deployed on February 8, 2025

Commitment summary:
All Action Items disclosed in this report have been completed as described, and we request its closure.

Unless there are comments or issues to address, I will close this on Wed. 19-Feb-2025.