Trivy Scanned My Compromised Server This Morning. It Found Two OpenSSL Bugs.

October 2026 — the scanner was healthy, up to date, and looking straight at it

Last weekend I finally got around to a project that had been on the list for months: proper vulnerability scanning for the cluster. Trivy Operator, a built-in server so every scan doesn’t re-download the vulnerability database, a UI, alert rules, an accepted-risk register with a review date. The works. I was genuinely pleased with myself. The cluster was now observed.

What I did not know — could not have known, and that is the entire point of this post — is that while I was writing alert rules on Saturday, somebody else’s software was unpacking itself into my Plex pod. We were both deploying that weekend. Only one of us got a congratulatory cup of coffee.

The discovery, or: why is the storage node on fire

The irony is that I do run on-call for this cluster, with full Grafana OnCall paging to my phone. And the alerting system did, in fact, do its job — just completely sideways.

An alert fired: the storage node was pinned at 100% CPU. Not elevated. Pinned. All four cores of its little N100, load average fifteen, fans contemplating resignation. My first reaction wasn’t security panic; it was pure homelab sysadmin annoyance: what on earth is Plex doing now? A rogue library scan? An unhinged credit-detection run? A batch intro-analyzer gone feral?

My OCD was duly triggered: nothing is allowed to burn 100% CPU on my storage node without my explicit permission. So I opened the terminal to tell Plex to behave itself.

I looked inside the pod. Plex Media Server was sitting at 0.4% CPU. Blissfully idle. It wasn’t transcoding a thing. Meanwhile a process calling itself postgresql — there is no PostgreSQL in that pod, there has never been a PostgreSQL in that pod — was burning two and a half cores, supervised by two processes calling themselves systemd, which were lying about being systemd.

That is not a Plex enthusiasm problem. That is a stranger.

So in a twisted way, my alerts did catch it — not because an intrusion detection system warned me of a botnet breach, but because an unconstrained miner broke my node’s performance baseline and offended my sense of order.

What had moved in

The short version, because the forensic detail deserves its own write-up and this post is about something else: an XMRig Monero miner, installed by a scripted worm through a chain of Plex vulnerabilities — CVE-2026-96652, -96654, -96655 and -96656, collectively the sort of thing that starts with an unauthenticated server-side request forgery and ends with “arbitrary file write, executed on load.” Fixed in Plex 1.43.3.10861. I was running 1.43.3.10828. Thirty-three builds of difference between “fine” and “free CPU for a stranger.”

The timeline is the part that should worry everyone:

• The CVEs were published and added to CISA’s Known Exploited Vulnerabilities catalog on a Tuesday.
• The worm’s build stamp says it was packaged three days later, on Friday.
• It moved into my pod on Saturday. That Saturday. The Trivy weekend.

Five days from “publicly known” to “mass-exploiting whatever the internet offers.” I have had groceries last longer.

And here is the punchline, the one that made me sit back from the keyboard: Trivy scanned that exact image while the miner was running inside it. The report is timestamped mid-compromise. Zero criticals. Two high-severity findings, both in OpenSSL. The scanner was working perfectly. It was healthy, current, correctly configured, and looking directly at a KEV-listed, actively-exploited vulnerability.

It just couldn’t see it.

Why the scanner was blind, and why that’s not a bug

Trivy — and this is true of essentially every image scanner — works by matching the packages inside an image against vulnerability feeds. OS packages from Debian, Ubuntu, Alpine; language dependencies from npm, PyPI, Go modules. If it’s a package with a name a feed knows, it gets flagged.

Plex Media Server is a proprietary binary. It is not a Debian package with a maintainer and an advisory feed. The CVE was in NVD with a perfectly good CPE identifier, the exploit was in the news, CISA was waving flags — and none of that information had a path into “what packages are in this container image.” The vulnerability lived in a layer of reality my scanner does not model.

There is a second failure, and this one is on me. Trivy Operator has a config-audit scanner — the one that inspects your running workloads and flags unconstrained compute, missing CPU limits, and broad capabilities. It would not have caught the unindexed Plex binary vulnerability either, but it would have flagged that the entire pod was running without CPU limits, ready to let any runaway thread pin the physical cores unchecked. Alongside it sat an rclone sidecar with elevated privileges — needed for FUSE filesystem mounting (/dev/fuse, bidirectional volume propagation so Plex can see the virtual debrid media paths on the fly) — but nonetheless expanding the pod’s attack surface. I had disabled that config scanner in my Helm values, calling the setup “deliberately narrow.” I was very proud of how deliberate it was.

The worm, for its part, never attempted privilege escalation or container escapes. It didn’t need to. It runs the same mass-scanning dropper across thousands of internet-facing pods; hand-tailored container escapes are a waste of an automated botnet operator’s time. My pod was worth about six cents a day to them. They were happily mining it with my electricity.

The uncomfortable math

While we’re being honest: most self-hosters run no scanner at all. I know this because I was one of them until last weekend, and I run a cluster that hosts my family’s email and photos. The baseline in this community is not “Trivy with a partial blind spot.” The baseline is nothing. No image scanning, no config audit, no egress policy, no idea. If you take one thing from this post: even a blind scanner beats the void, because the void finds nothing, ever, by definition.

But the deeper lesson is about what kind of answer a scanner gives. A vulnerability scanner answers the question “is this image vulnerable?” It does not — it cannot — answer the question “is this pod breached?” Those are different questions asked of different systems. The first is a hygiene question, answered against a database. The second is a behavioral question, answered against reality: why is CPU at 300% with Plex idle? Why is a media server holding a TLS connection to a Monero mining pool? Why did a process decide to call itself systemd when systemd is not in this container?

The things that would have caught this in hours instead of days are not scanners:

• A targeted CPU anomaly alert. The node-level CPU alert screamed, but a specific pod-level anomaly alert — noticing a pod burning multiple cores while its actual main application sits at 0.4% — would have named the pod immediately instead of waiting for me to blame Plex. I have added those specific workload rules now.
• Egress restrictions and alerting. A default-deny egress policy would have blocked both the payload download and the pool connection. The miner would have suffocated on arrival.
• Runtime detection. A process executing out of /tmp and renaming itself after kernel threads is not subtle. It is, in fact, screaming. Something like Falco hears that screaming.
• A patch SLA measured in days for internet-facing software. Five days was not enough time for my normal upgrade cadence. It was plenty for theirs.

I have since turned on the config-audit scanner I had disabled, added the alerts I didn’t have, and written up the full forensic trail — indicators, wallet, abuse reports, the lot — in a private incident repo. The miner got about three days and roughly thirty-five CPU-hours. I got a post-mortem, a patch, and this post. I still think I came out ahead, but it’s closer than I’d like.

The parting gift: an abuse sweep

Before wrapping the incident, I took thirty minutes to make sure the operator didn’t enjoy their thirty-five CPU-hours in peace.

Worms run on a fragile financial and staging pipeline. When you pull the binary apart and grab the strings, the pipeline is standing right in front of you:

1. The mining pool (SupportXMR): The worm was happily hashing away to wallet 83YFdNe...6f5SeX with "rig-id": null to blend victim nodes into one stream. A quick query against the pool’s public API showed the whole picture: an aggregate ~8.72 MH/s botnet hashrate across nearly 300 million valid shares, with 26.3 XMR already paid out and another ~0.35 XMR pending. I sent a report to SupportXMR administration with the worm’s CISA KEV exploit chain, binary hashes, and timestamped victim telemetry, requesting an immediate freeze on pending payouts.
2. The Command & Control beacon (AlexHost): The worm’s persistent stage file periodically checked in with http://176.123.7.7/ks/ksoftirqd.vlog on AS200019 to dynamically swap out mining pools, update target wallets, or push secondary stages. A formal abuse report went to their network operations to pull the plug on the C2 endpoint.
3. The payload staging server (FreeINet): Dropper scripts pulled architecture-specific ELF binaries directly from http://80.87.206.86/bins/ (and fallback port 18081). Sent their abuse desk the dropper SHA-256 (a43724fa...), the miner payload (77d764ce...), and the supervisor wrapper (e9c4bc82...) to take down the distribution server and preserve subscriber records.

If you ever find yourself cleaning up after an automated worm, don’t just format the disk and walk away quietly. Take the indicators of compromise, query the pools, and file the abuse tickets. Breaking their infrastructure and freezing their crypto wallets turns an otherwise quiet, profitable campaign into an operational migraine.

The measures: closing the blast radius

To learn is to accept making mistakes — so here is what actually changed in the cluster after the dust settled:

1. Rebuild, clean, and verify: Rebuilt Plex on patched build 1.43.5.11079 (surpassing the 1.43.3.10861 KEV baseline) via GitOps, scrubbed the infected /config volume clean of all dropper and supervisor implants (.irc, .f, the trojanized PlexMediaScanner), and rotated PLEX_TOKEN through Bitwarden. A cluster-wide sweep across all pods and physical host roots confirmed the breach never escaped the container boundary.
2. CPU & Memory limits: Placed explicit resource limits on the entire pod (limits.cpu: 2000m on main, 500m on sidecars). When an unconstrained miner lands, it starves the whole machine; with limits, CPU throttling trips metric alerts instantly instead of silently pinning the node.
3. Behavioral alerts: Added Prometheus/VictoriaMetrics alerting rules for PodRunawayCPU (alerting on sustained uncharacteristic CPU draw), PodCpuThrottledHard, and NodeCpuSaturated. If a rogue workload starts burning compute, it pages within the hour.
4. Runtime intrusion detection with Falco: Deployed a Falco DaemonSet with modern eBPF across all nodes. It monitors system calls directly for the exact worm behavior: processes executing out of /tmp or /dev/shm, binaries masquerading as kernel threads or systemd, shell spawns inside media containers, and outbound connections to mining stratum ports. This catches the entire worm class at execution time, regardless of whether a scanner recognizes the image.
5. Egress kill-switch: Enacted namespace NetworkPolicies restricting outbound traffic and blackholing the known C2 and pool IP ranges. A worm needs outbound network access to breathe; cut off the staging and pool egress, and the payload is inert on contact.

None of this makes a cluster magically invulnerable. What it does is make the next intrusion small, loud, unprofitable, and short-lived.

What I’d tell Saturday-me

You did the right thing deploying the scanner. You just mistook it for a wall. It’s a smoke detector — it tells you about a specific kind of smoke, in the rooms it can see, and it says nothing about whether the house is already on fire in a room it can’t.

The worm wasn’t better than my tooling. It was simply playing a different game: not “find the vulnerability in the image” but “find the image that is vulnerable,” at internet scale, with a five-day head start. You cannot scan your way out of that. You can only make the compromise small, loud, and unprofitable.

Limits make it small. Egress policy makes it unprofitable. Runtime alerts make it loud.

And then, then the scanner earns its keep — telling you, on a quiet weekend, which image to rebuild next. Just don’t ask it whether you’re currently on fire. That was never its job. It was mine.