Postmortem: The July 21 TLS Certificate Incident on api.mailchannels.net

July 21, 2026

On the morning of July 21, 2026, we rotated the TLS certificate on api.mailchannels.net, the endpoint that receives every message sent through our Email API. Within minutes, a subset of customers saw every request fail with an error that looks alarming out of context:

SSL certificate problem: self-signed certificate in certificate chain

Email sending does not tolerate downtime, and we know exactly how disruptive this was for the customers it hit. This post explains what happened, why it only affected some senders, what we changed, and what we are putting in place so a certificate rotation never surprises anyone again. It is also a short tour of how TLS trust actually propagates across the internet, because the root cause lives in a corner of the system most engineers never need to think about until the day it bites them.

What customers saw

Affected clients, most of them using curl or libcurl-based HTTP libraries, received TLS verification failures on every request to the Email API. No connection completed, which means no message was accepted, silently dropped, or exposed in transit. Senders got clean, immediate errors. There was no security impact of any kind.

The certificate we served was genuine. It was issued on July 6, 2026 by SSL.com for *.mailchannels.net, valid through January 20, 2027, with correct hostnames and Certificate Transparency log entries. The certificate was never the problem. The problem was trust.

What the error actually means

When your client connects to our API, our server hands it a chain of certificates: the leaf certificate for our hostname, plus the intermediate authority that signed it. Your client’s job is to walk that chain upward until it reaches a root certificate it already holds in its local trust store. That trust store is a file or database that shipped with your operating system, your language runtime, or your HTTP library, and it is the entire foundation of the exercise. A root certificate is, by definition, self-signed. Nobody vouches for a root except the fact that your system already carries a copy of it.

OpenSSL’s verify error 19, the one behind the message above, means something very specific: the client assembled the chain we sent, reached the self-signed root at the top, and could not find that root in its local trust store. It does not mean we generated a homemade certificate. It means the client had simply never heard of the authority vouching for us.

Why it only broke some customers

The renewed certificate chains up to a root called SSL.com TLS RSA Root CA 2022. As the name suggests, that root was created in 2022, and root certificates propagate slowly. A new root must pass audits and the inclusion reviews of the major root programs run by Mozilla, Microsoft, Apple, and Google, and only then does it start shipping in operating system and browser trust stores. This particular root only started landing in mainstream trust stores in late 2023, and updates roll out to real systems gradually after that.

Any machine whose CA bundle predates that window has no idea this root exists. In practice that means systems like CentOS 7, Amazon Linux 2 instances that have not pulled recent updates, older Debian and Ubuntu releases, long-lived container base images, and applications that ship a pinned cacert.pem file. Every one of those verified our previous certificate chain without complaint, then failed the moment the new chain went live. Meanwhile, any reasonably current machine trusts the 2022 root and saw nothing wrong at all, which is exactly why our own first attempts to reproduce the report came back clean.

One more detail for the protocol enthusiasts. We were also serving the self-signed root itself as the third certificate in the handshake. That is unnecessary, since clients must already hold the root locally for it to count, and it is the reason affected customers saw error 19, self-signed certificate in certificate chain, rather than error 20, unable to get local issuer certificate. The outcome is identical either way: no trusted path, no connection. But it was untidy, and we have stopped doing it.

The fix: cross-signing

The elegant solution to the slow propagation of new roots has been around for decades, and it is called cross-signing. A certificate is fundamentally a public key wrapped in a signature, and nothing prevents the same public key from being wrapped twice. A certificate authority can therefore publish two editions of a new root: the self-signed edition that root programs distribute, and a cross-signed edition in which the new root’s key is signed by one of the authority’s older, long-established roots.

SSL.com publishes exactly such an edition of TLS RSA Root CA 2022, signed by SSL.com EV Root Certification Authority RSA R2, a root that has been ubiquitous in trust stores for many years. The cross-signed certificate is valid from November 2023 through November 2038.

Our API frontends now serve precisely three certificates: our leaf, the SSL.com TLS Issuing RSA CA R1 intermediate, and the cross-signed root. A modern client recognizes TLS RSA Root CA 2022 on sight and stops there. An older client keeps walking one step further and lands on EV Root Certification Authority RSA R2, which it has trusted since long before 2022. Both paths terminate in trust, and both kinds of client verify successfully. This is the chain we should have been serving from the start.

If you are still seeing errors

With the corrected chain deployed, the overwhelming majority of clients need no changes. If your requests are still failing, your trust store is old enough that we would encourage updating it anyway. On Debian and Ubuntu systems, refreshing the ca-certificates package brings in current roots. On Red Hat family systems, update the ca-certificates package and run update-ca-trust. If your application pins its own CA bundle, refresh it from a current source or append the SSL.com TLS RSA Root CA 2022 certificate, available from SSL.com’s official repository.

You can test your environment directly:

openssl s_client -connect api.mailchannels.net:443 -servername api.mailchannels.net </dev/null

Look for Verify return code: 0 (ok) near the end of the output. If you see anything else, send the full output to our support team and we will get you unblocked quickly.

What we are changing

The lesson here is not subtle. A certificate can be perfectly valid and still be the wrong certificate to serve, because validity is a property of the certificate while compatibility is a property of every trust store your customers actually run. Choosing a chain that worked for modern systems but not for the long tail was our failure, and catching it in testing rather than production was our responsibility.

Our certificate rotation runbook now requires verifying the full served chain against legacy trust stores, including bundles of CentOS 7 vintage, before any deployment, not just against current ones. We are adding canary monitors that perform TLS verification against our endpoints using deliberately outdated CA bundles, so a compatibility regression pages us before a customer ever sees it. And we will keep serving minimal, deliberate chains with cross-signed paths for as long as a meaningful population of legacy clients exists.

Thank you

Finally, a genuine thank you to the customer who reported this within minutes and included a complete openssl s_client transcript in their first follow-up. That single artifact contained everything needed to diagnose the problem, and it is the reason the fix moved as fast as it did. To everyone affected: we are sorry for the disruption, and we appreciate your patience while we made it right.