Discussion on Hacker News Discussion on lobste.rs

Rant ahead: I hate TLS “Inspection” software with a burning passion and I wish we collectively as an industry would just knock it the fuck off and stop pretending it’s some great security benefit. Every time I encounter it, in whatever form, it’s a gigantic headache that makes everyone’s life worse off and as far as I am concerned offers next to zero tangible benefits.

For those blissfully unaware, this is a class of “security” software or appliance that is supposed to let organisations monitor all encrypted traffic. It does this by inserting itself in the middle of traffic, stripping the encryption off so it can inspect it and then re-signing it with its own certificate. If that sounds familiar, it’s because it’s a widely known class of attack - the Man In The Middle attack. Great stuff, we’re literally deploying the exact attack vector that TLS was designed to prevent, but slapping a “security” label on it.

Firstly, it undermines one of the most important protocols of the modern Internet as it deliberately breaks all the guarantees that TLS encryption is supposed to offer. If the MITM certificate is installed everywhere, your company can intercept and monitor everything you say and do. Consider the ramifications of that - confidential messages to HR, medical information, insider trading information, personal things like your banking sessions - would you feel happy BCC’ing every single email to your IT department? Would you print out your therapy notes and pin them to the kitchen notice board? ¹

But even ignoring the philosophical arguments about privacy and trust, I argue it actively makes your security worse. Consider this - what is the likelihood of every certificate authority on the Internet having their private keys compromised simultaneously? I’d wager that’s almost at the whatever is the statistics equivalent of the Planck length level of probability. ²

On the other hand, what’s the chance of your company’s MITM private key getting compromised by an attacker? Even if you completely trust your IT team and vendor (and if you do, you clearly haven’t been paying attention to any tech news for oh… the last few decades), you have to admit that chance is a lot higher. And depending on the vendor or tech stack involved, it could be a LOT higher. One disgruntled employee, one unpatched vulnerability, one phishing email to the right admin and choo-choo, it’s all aboard the FAIL train. Now an attacker could have the keys to your entire kingdom. ³

Then there’s the practicalities of it. It’s simply a massive hassle. Different Operating Systems expect certificates in different formats (PEM? DER? PFX? P7B?) installed in different places with different tooling to manage it all. update-ca-certificates vs update-ca-trust is just the tip of the iceberg - and that’s just the OS level. You then have language runtimes (Java keystore anyone?) and the applications themselves that all need to be configured.

And the problem is compounded with modern cloud-native apps. In a Kubernetes cluster, as well as having to handle updating the node VM images and container runtimes, you’ll have dozens if not hundreds of different base images each of which has their own standards. Alpine uses a different certificate path than Ubuntu. Your Node app expects them somewhere else entirely. The various CRDs or Helm charts you are using may or may not support custom CA bundles, and if they do there’s no agreed-on standard.

Now I’m not saying that because a problem is hard we should simply give up, but even if the benefits were worth it the simple fact is even with the best tooling and automation, you are guaranteed to miss something. Whether it’s some obscure tool that has a custom keystore and non-standard tooling, a quick “one off” command in an ephemeral container, some app that uses certificate pinning or an aging switch firmware that doesn’t even support custom certificate bundles, something will slip through the cracks. And when it does, guess what happens?

Which brings me to my biggest peeve: it normalizes bad security practices. Given that you will never have 100% coverage of your CA certificate installation - particularly amongst your technical teams who will be using a multitude of different tools and platforms - you get developers and sysadmins used to TLS errors. Instead of treating each one as an anomaly and something to be investigated, you get used to just running with --insecure or curl -k because you just need to get shit done. Turning off certificate verification becomes a routine troubleshooting step. “Oh, it’s probably just the corporate proxy again” becomes the reflexive response to any TLS error. You’ve just trained your entire technical staff to ignore one of the most important security warnings on the Internet!

And don’t even get me started on the performance and availability implications. All your traffic now has to be decrypted and re-encrypted by your magic box. Hope you sized that appliance correctly! Hope it doesn’t become a single point of failure! Hope it supports all the latest TLS versions and cipher suites!

There are a multitude of ways to protect yourself that are not only less invasive but are often more effective because they’re designed for how modern infrastructure actually works. Anomaly detection, Zero Trust network architecture, EDR, Netflow analysis… You don’t need to create single points of failure, and you can actually work with modern cloud-native infrastructure instead of fighting it. Plus, y’know, there’s this AI thing which as it turns out is actually quite useful at analysing metadata and spotting odd behavioral patterns.

In my experience: TLS Inspection MITM is a gigantic administrative burden, it normalizes bad practice, it creates bottlenecks and availability risks, and actively worsens your security posture.

Just stop it already.

NOTE : This was a particularly cathartic rant, and I didn’t expect it to blow up and hit the front page of Hacker News! While I was venting away, I do appreciate that there is actually some nuance at play here so for a fuller discussion (with some dissenting opinions as well as plenty of similar stories of pain), please do check out the comment thread.

¹ = Let’s be realistic - even if you have a “no personal activity” rule, people are still going to do this. Which means you may have a problem with that intercepted data, even if you didn’t want it in the first place.

² = In particular, I worded this poorly. My point still stands, but what I was thinking of was more along the lines of being able to do this without detection via CT logs etc. A commenter on HN pointed out that you only need one trusted CA to be compromised which is of course true.

³ = And that’s not to mention the question of what some vendors may be doing with your sensitive data - like using it for AI training, for example.