CrowdSec 1.8 Adds Bot Detection to Its Open-Source WAF: What Fingerprinting Plus Proof-of-Work Changes at a Self-Hosted Edge

Sometime in 2024, the web quietly flipped: automated traffic outnumbered human traffic for the first time in a decade — 51% of all web traffic, per Imperva's 2025 Bad Bot Report. The 2026 edition puts the 2025 figure above 53%, with human traffic down to 47% and still falling. If your site sits behind a big managed edge, some flavor of bot management is bundled into the bill. If it sits on machines you own, behind an ingress you configured yourself, it has exactly as much bot defense as you installed — and the scrapers do not care which one you picked.

CrowdSec 1.8, released August 31, is the open-source answer catching up to that flip. Its headline feature puts bot detection inside the CrowdSec WAF (the AppSec component): any visitor without a valid challenge cookie must clear a browser-fingerprint check plus a proof-of-work puzzle before reaching your app. Alongside it comes a dedicated Kubernetes datasource that lets one CrowdSec instance read pod logs straight from the API server — no per-node collectors.

The concrete version of the first feature: a headless-Chrome probe against a protected endpoint scored 150 — 100 points for speaking Chrome DevTools Protocol, 50 for a WebGL fingerprint that disagreed between page contexts — against profiles that reject at 75 (Balanced) and 100 (Permissive). That is a 2x margin over Balanced and 1.5x over Permissive, decided before a single byte of application code ran. Here is how that verdict gets computed, what it changes about bot economics, and the checklist that decides whether it survives contact with your ingress.

How the challenge works: fingerprint first, puzzle second​

The flow has six steps, and every one of them happens before your origin responds — this walkthrough follows an independent deep-dive published the week of the release:

1. A request with no valid __crowdsec_challenge cookie reaches a protected route. The bouncer forwards it to AppSec, which answers SendChallenge instead of proxying to the origin.
2. The visitor gets a challenge page loading three JavaScript payloads: the open-source fpscanner fingerprinting library (served as-is, public code), a proof-of-work and crypto bundle, and a per-epoch signing-key module. Only the last two are obfuscated.
3. The browser collects a device fingerprint, solves a proof-of-work puzzle whose cost the operator tunes with SetChallengeDifficulty, and posts both to /crowdsec-internal/challenge/submit.
4. AppSec cryptographically validates the submission, decrypts the fingerprint into a FingerprintData object, and runs the on_challenge_submit scoring rules: automation verdict, or pass.
5. On a pass, the client receives a sealed success cookie carrying the fingerprint and an expiry timestamp — no re-challenge until it expires, with lighter re-checks on later requests. A browser with cookies disabled can never pass, so the page says so up front instead of failing mysteriously.
6. Signing keys rotate on a schedule without invalidating already-issued cookies (they are sealed under a separate long-lived key). One master_secret roots every key — share it across all instances in an HA deployment, or cookies minted by one instance won't verify on another.

The fingerprint is the interesting half. Collected by Antoine Vastel's open-source fpscanner library and posted back encrypted, it spans seven signal families:

Two things fall out of that table. First, the scoring behind the 150 in the intro is unified and configurable: the collection ships scoring profiles (Balanced rejecting at 75, Permissive at 100), and the signal functions are available to your own appsec-configs and scenarios. Fine-grained custom patterns — such as extending a ban for an IP that keeps tripping virtual-patching rules — compose with the challenge verdict instead of living in a separate tool.

Second, legitimate automation is a first-class case, not an afterthought. Dedicated exclude configs cover search engines, AI crawlers, crawler files, feeds, monitoring probes, social previews, static assets, webhooks, and API paths. They are matched by MatchKnownBot() on IP ranges or forward-confirmed reverse DNS, then skipped with ExemptFromChallenge() — no cookie, no puzzle.

What it changes about bot economics​

The rest of the WAF answers "what is this client doing" — payloads, CVE signatures, virtual patching. Bot detection answers the question underneath it: "what is this client" — a real browser, or a script wearing one as a costume. Two baselines put the change in perspective.

Against fail2ban, the inversion is total. Fail2ban is reactive by design: parse logs for abuse patterns, ban IPs after the fact, one host at a time, with no notion of a browser at all. A scraper that stays under thresholds and rotates addresses never trips it. CrowdSec 1.8 decides before the origin responds — patient, low-rate automation still pays the proof-of-work cost on every new client and still fails the fingerprint the moment it looks headless — and the verdict feeds the same decision pipeline that already escalates and shares bans.

Against Anubis, the comparison is closer and more instructive. Anubis proved self-hosted proof-of-work works at scale: a challenge computed in under 3 milliseconds, upwards of 10,000 requests per second on a single core, the whole filter in about 128 MB of RAM. Roughly 98% of real browsers pass versus 95% of headless scrapers stopped — and it now guards Codeberg, FFmpeg's mirrors, GNOME's GitLab, and the United Nations.

CrowdSec 1.8 keeps those economics (the attacker's cost per attempt is real compute, tunable via difficulty) and adds what a standalone challenge proxy can't: a fingerprint verdict with named, auditable signals; allowlists for legitimate bots; and escalation across requests. The shipped behavioral scenarios watch what per-request hooks can't — 5 challenge submissions in 20 seconds becomes a genuine decision enforced at the bouncer, not just another puzzle served. And every verdict lands as CrowdSec signal: bot-detection alerts in cscli alerts list and dedicated bot-detection metrics your own scenarios can build on.

Put as cost curves: proof-of-work taxes volume, fingerprinting taxes stealth (a headless framework must now simultaneously hide CDP, keep WebGL consistent across page, iframe, and worker contexts, and present plausible device values), and decisions tax persistence (repeat probing earns a ban, not infinite retries). Three different prices, one integration, no third-party captcha account.

One instance watching the whole fleet​

The second headline feature is quieter and, for fleet operators, arguably bigger. The dedicated Kubernetes datasource lets CrowdSec fetch logs directly from the API server: select pods by namespace and name, tag the stream with a program so the right parsers match, and analyze cluster logs with no per-node collectors and no separate forwarding layer in front of CrowdSec. The before/after for a Cluster API fleet is stark — before, each node's container logs needed collection, an agent footprint on every machine or a shipper tier to maintain and upgrade; after, a single CrowdSec instance with pod-log read access follows the fleet by selector.

Honest scope, though: the datasource removes the log-shipping layer, not the deployment. You still run LAPI with its decision database, you still need a bouncer at every ingress point, and HA still means sharing master_secret plus the usual LAPI considerations. The rest of 1.8 fills out the operational picture: HTTP helpers in the expression language so parsers and scenarios can enrich from external services, a machine heartbeat metric, optimized decision deduplication, a configurable WAF-to-LAPI authentication timeout, and cscli improvements — plus two DoS fixes and a long bug-fix tail covering duplicate Loki ingestion, syslog acquisition, Docker crashes, and WAF rule tracking.

The adoption checklist: six things to get right​

1. It is default-off. Bot detection arrives as a Hub collection — sudo cscli collections install crowdsecurity/appsec-bot-challenge pulls the appsec-configs, scenarios, and parsers — plus an acquis entry wiring the configs to the AppSec listener and a compatible bouncer pointed at it:

   yaml

   listen_addr: 127.0.0.1:7422
   appsec_configs:
     - crowdsecurity/appsec-default
     - crowdsecurity/appsec-bot-*
   labels:
     type: appsec

   No collection, no challenges.

2. Check your bouncer. Challenge support is per-bouncer: nginx and OpenResty first, with HAProxy SPOA, Traefik, and Envoy carrying the Bot Detection badge in the docs and Traefik's support landing through its plugin (a v1.8.0-alpha was in flight in mid-September). If your ingress isn't on the list yet, the challenge has nowhere to be served from.

3. Mind the version gate. Engine 1.8 or newer with a current bouncer (nginx bouncer 1.2.x or newer): composing bot-challenge config against an older engine FATALs CrowdSec on start — AppSec and every scenario down, fail-open. Upgrade the engine and the bouncer together, and verify versions before composing config.

4. Know what can't pass. The challenge requires SSE4.1 and writable-executable memory, so older silicon and strictly hardened W^X clients can't solve it. Survey that exclusion set in your own traffic before enforcing — it is the one item in this list most likely to surprise.

5. Exclude what isn't a browser. API clients, mobile apps, webhooks, and feed readers can't run JavaScript challenges; the collection ships exclude configs for API paths, webhooks, feeds, static assets, and monitoring for exactly this reason. Challenge your HTML routes and exempt your machine routes, or you will break your own integrations on day one.

6. Plan HA keys early. Rotation never invalidates issued cookies, but every instance must share the same master_secret from the start — a second instance with a different secret rejects the first one's cookies.

Longer term, the industry is converging on cryptographic bot attestation so legitimate automation can prove what it is instead of being fingerprinted; until those standards land, a verified allowlist plus a challenge that taxes everyone else is the pragmatic shape of self-hosted bot defense. CrowdSec 1.8 makes that shape a collection install away — the cheapest serious bot-versus-browser decision your own ingress has ever had.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.