Skip to main content

Webhook Signature Verification for Git-Push Deploys: What a Self-Hosted PaaS Has to Get Right That GitHub's Own Docs Gloss Over

11 min readDora NodaDora Noda
Share
On this page

In 2022, the Jenkins Git Plugin shipped a webhook endpoint with no authentication at all. The result was CVE-2022-36883: any unauthenticated attacker who knew (or guessed) a repository URL could trigger builds of the jobs using it — and check out an attacker-specified commit. Not a memory-corruption exploit, not a supply-chain compromise of a dependency. Just an HTTP endpoint that believed whatever POST body arrived at it.

Every git-push PaaS — Render, Heroku, Railway, Coolify, Dokploy, or the one you are building yourself — has exactly this endpoint. A forge (GitHub, GitLab, Gitea) POSTs "a push happened" to your ingest URL, and your platform responds by cloning a repo, running a build, and deploying the output with the platform's own credentials. The only thing standing between "a push happened" and "anyone who finds the URL can make my platform build and run code" is HMAC signature verification, done correctly. GitHub's own webhook validation docs contain almost every ingredient — but they present them as implementation notes, not as the load-bearing security boundary they are, and they say nothing about the middleware-ordering trap that breaks most real-world implementations. This post is the audit checklist those docs never assemble.

The Failure Mode, Concretely

Before the checklist, be precise about what a subtly wrong implementation leaves open. A webhook payload is just JSON. A typical push event carries the repository's clone URL, the ref that changed, and the head commit SHA. If your endpoint accepts a forged payload, the attacker controls those fields. Depending on how your build pipeline consumes them, that means:

  • Attacker-specified source: the payload's repo URL or commit SHA points at attacker-controlled code, and your builder clones and executes it — build scripts run arbitrary commands by design. Jenkins' CVE-2022-36883 is precisely this: unauthenticated build triggers with an attacker-specified commit.
  • Deploying a stale or wrong ref: even if the repo URL is pinned server-side, a forged "push to main" can redeploy an old, vulnerable commit, or a replayed payload can roll a service back after you shipped a fix.
  • Build-resource abuse and denial of service: every accepted forgery is a free build job on your workers — CPU minutes, registry pushes, deploy churn.
  • Reconnaissance: verification that responds differently for "unknown repo" vs. "bad signature" tells an attacker which repositories exist on your platform. Security researchers at Cider Security (later Palo Alto) showed webhook surfaces are probed at scale, not just in targeted attacks.

A build trigger is a remote code execution primitive with extra steps. Audit it like one.

The 10-Point Audit Checklist

Run this against your platform's webhook-ingest endpoint. Every item is a concrete check; the sections that follow explain the three subtlest ones.

  1. Verify against the raw request bytes — the body exactly as it arrived on the wire, captured before any JSON body-parsing middleware touches it. Never re-serialize a parsed object.
  2. Use HMAC-SHA256 via X-Hub-Signature-256 (GitHub/Gitea convention), not the SHA-1 X-Hub-Signature header GitHub keeps "for legacy purposes."
  3. Compare signatures in constant timecrypto.timingSafeEqual (Node), hmac.compare_digest (Python), hmac.Equal (Go). Never ===, ==, or a string equality that can exit early.
  4. Length-check before the constant-time comparecrypto.timingSafeEqual throws when buffer lengths differ, so an attacker sending a short garbage signature crashes a naive handler. Check lengths first and return false.
  5. Reject missing or malformed signature headers outright — a request with no signature header must fail closed, before any other processing. "No secret configured" must never mean "skip verification" for an endpoint that triggers builds.
  6. Verify before any side effect — no queueing, no repo prefetch, no logging of payload-derived values, no database writes keyed on payload contents until the signature has passed.
  7. Use per-tenant (or per-repo) secrets with real entropy — a random 32+ byte token per webhook, not one platform-wide secret shared by every tenant. One leaked secret should compromise one integration.
  8. Deduplicate deliveries — record the delivery ID (X-GitHub-Delivery) and drop repeats. Note: that header is not covered by the signature, so it's an idempotency tool, not an authenticity one.
  9. Support two active secrets per endpoint — verification that accepts either of two secrets is what makes zero-downtime secret rotation possible; single-secret designs make rotation an outage.
  10. Return a uniform error — same status code, same (empty) body, for every verification failure. Don't tell the attacker whether the repo exists, the header was missing, or the signature was merely wrong.

Here is the reference shape in Node.js — raw body first, length check, constant-time compare:

javascript
import crypto from "node:crypto";
import express from "express";
 
const app = express();
 
// Capture raw bytes; do NOT let express.json() run first on this route.
app.post(
  "/ingest/github",
  express.raw({ type: "application/json" }),
  (req, res) => {
    const secret = lookupSecretForEndpoint(req); // per-tenant secret
    const header = req.get("X-Hub-Signature-256") ?? "";
 
    const expected =
      "sha256=" +
      crypto.createHmac("sha256", secret).update(req.body).digest("hex");
 
    const a = Buffer.from(header);
    const b = Buffer.from(expected);
 
    // timingSafeEqual throws on length mismatch — check first, fail closed.
    if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) {
      return res.status(401).end(); // uniform error, no details
    }
 
    const event = JSON.parse(req.body); // parse ONLY after verification
    enqueueBuild(event);
    return res.status(202).end();
  }
);

Ten lines of crypto, and every one of them has a wrong version that type-checks, passes the happy-path test, and ships. The next three sections cover the wrong versions.

Gloss No. 1: === Leaks the Signature Byte by Byte

What GitHub's docs say — to their credit, plainly: "Never use a plain == operator. Instead consider using a method like secure_compare or crypto.timingSafeEqual." What they gloss over is why, and the why is what makes engineers take it seriously instead of treating it as lint pedantry.

A standard string comparison returns as soon as it finds the first mismatched byte. That means the time it takes correlates with how many leading bytes were correct. An attacker who can send many requests and measure response times can, in principle, discover the expected HMAC one byte-position at a time — each position needs at most 256 guesses instead of the full keyspace, turning an impossible brute force into a statistics problem. Network jitter makes the measurement noisy, but "noisy" is a rate limit, not a defense — averaging over enough requests recovers the signal, and your webhook endpoint is by definition open to unlimited unauthenticated POSTs.

Constant-time comparison functions check every byte regardless of where the first mismatch occurs, so timing carries no information. The catch — and this the GitHub docs never mention — is Node's sharp edge from checklist item 4: crypto.timingSafeEqual throws a RangeError if the buffers differ in length. An attacker who sends X-Hub-Signature-256: x gets your handler to throw; without a length guard, that's an unhandled exception per request — your one line of defense doubles as a denial-of-service lever. Compare lengths first (length is not secret — the correct signature is always 71 characters: sha256= plus 64 hex digits), then compare contents in constant time.

Gloss No. 2: The Raw-Body Trap Is a Middleware-Ordering Bug

GitHub's docs say the signature is computed from "the payload contents" and remind you to handle UTF-8. What they never spell out — because it's a property of your web framework, not of GitHub — is that "the payload contents" means the raw bytes on the wire, and most frameworks destroy those bytes before your handler runs.

The standard Express setup calls app.use(express.json()) at the top of the file. By the time your webhook route executes, req.body is a parsed JavaScript object; the original bytes are gone. The tempting fix is JSON.stringify(req.body) — and it is wrong in a way that passes most tests. JSON parsing and re-serialization does not round-trip byte-for-byte: whitespace between tokens is dropped, unicode escapes may be normalized (é vs. a literal é), numbers can be reformatted, and nothing guarantees key order. Sign the re-serialized string and your HMAC differs from GitHub's whenever any of those diverge — an intermittent, payload-dependent verification failure.

Here is the insidious part: teams hit those intermittent failures, can't reproduce them, and "fix" the flakiness by weakening verification — logging instead of rejecting, or disabling the check "temporarily." A subtle correctness bug decays into an open endpoint. The fixes are mechanical:

  • Express: use express.raw() on the webhook route (as above), or the verify callback of express.json() to stash req.rawBody before parsing.
  • Next.js / serverless: disable the automatic body parser for the webhook route and read the stream yourself.
  • Any framework: the test is simple — the buffer you HMAC must be the buffer the TCP socket delivered. If you can't prove that, you're signing a reconstruction. The same applies in front of the process: a proxy or gateway that rewrites bodies (compression, re-encoding) breaks verification just as invisibly.

Gloss No. 3: The Header Zoo — SHA-1 Ghosts and Plaintext Tokens

GitHub sends two signature headers with every delivery. The docs recommend X-Hub-Signature-256 (HMAC-SHA256) and describe X-Hub-Signature (HMAC-SHA1) as "only included for legacy purposes" — no removal date, no deprecation warning in the header itself. So tutorial code verifying the SHA-1 header keeps working forever, and copy-paste keeps it alive. SHA-1 has been practically broken for collisions since 2017; HMAC-SHA1 is not immediately exploitable the way raw SHA-1 is, but building a new security boundary on it in 2026 is indefensible when the stronger header is in the same request. Verify the 256 header; ignore its ghost.

A self-hosted PaaS rarely ingests from GitHub alone, and the forges do not agree on a scheme:

ForgeHeaderScheme
GitHubX-Hub-Signature-256HMAC-SHA256 of raw body
Gitea / ForgejoX-Gitea-Signature / X-Forgejo-SignatureHMAC-SHA256 of raw body
Bitbucket (Data Center)X-Hub-SignatureHMAC (SHA-256 default) of raw body
GitLabX-Gitlab-TokenThe secret itself, in plaintext — no HMAC over the body

That last row deserves a stare. GitLab's classic webhook scheme sends your shared secret as a literal header value on every delivery — an approach GitLab's own issue tracker has debated replacing for years and has begun to address with newer signed tokens. Under the classic scheme, anything that can read the request in transit or in logs (a proxy, an APM tool, a request-logging middleware) has captured the credential itself, and nothing proves the body wasn't altered. If your platform accepts GitLab webhooks: compare that token with a constant-time compare too (it's a secret — same timing rules apply), keep the header out of your logs, and treat per-endpoint secret rotation (checklist item 9) as more urgent than it is for HMAC forges. Your ingest layer must implement each forge's scheme per source — while enforcing the checklist's invariants uniformly.

What the Signature Doesn't Cover

Passing signature verification proves one thing: someone holding the secret signed these exact bytes. It does not prove the request is fresh. A captured delivery — from a log, a proxy, a compromised staging box — re-sent tomorrow carries a valid signature, because the signature covers the body, and the body hasn't changed. For a deploy platform, a replayed "push to main" from three weeks ago is a rollback to a vulnerable build, signed and approved.

Two mitigations close the gap. First, dedup on the delivery ID (checklist item 8): X-GitHub-Delivery is a per-delivery GUID; store seen IDs with a unique constraint and drop repeats — remembering it is not part of the signed material, so it's for idempotency, not authentication. Second, cross-check the payload against reality before acting: your builder should resolve the ref via authenticated git fetch from the configured repository and verify the pushed SHA exists on it — never clone from a URL the payload provided. That single rule converts most forgery-and-replay outcomes from "attacker code executes" into "no-op build of your own repo."

And when a secret does leak, rotation must not be an outage: verify against {current, previous} for a bounded window (checklist item 9), point the forge at the new secret, then retire the old one.

The Boundary Is Ten Lines of Code

Strip away the checklist and one fact remains: on a git-push platform, webhook verification is not input validation — it is the authentication boundary for the machinery that builds and runs code. GitHub's docs hand you the ingredients (they really do say "never use a plain =="), but they frame verification as an integration nicety, say nothing about your framework eating the raw bytes, and keep a deprecated SHA-1 header flowing next to the real one. The gap between their sample snippet and a production ingest endpoint — per-tenant secrets, fail-closed missing headers, length-guarded comparison, replay dedup, dual-secret rotation, uniform errors — is exactly the gap this checklist walks.

Audit yours this week. It's one endpoint, ten checks, and the alternative was CVE-2022-36883.


Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Its webhook-ingest surface is exactly the endpoint this checklist audits — and because it's open source, you can read the verification code instead of trusting it. Star the repo on GitHub or deploy your first app today.

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex