cortex-bridge

Self-hosted, Firecrawl-compatible web scraper with reactive anti-bot profile rotation.

Drop-in replacement for Firecrawl when you don't want to pay per-credit and you need to scrape sites with aggressive WAFs (Cloudflare, PerimeterX, Datadome, AWS WAF, etc.). Exposes both an HTTP API and an MCP stdio server, so it works from any LLM agent.

HTTP API                          MCP stdio (Claude, etc.)
    │                                      │
    └──────────────┬───────────────────────┘
                   ▼
            Axum router
                   │
   ┌───────────────┼─────────────────────┐
   │               │                     │
   ▼               ▼                     ▼
scrape()       search()              scrape_url / web_search
   │               │                     │
   │               ▼                     │
   │         SearXNG (HTTP)              │
   │                                     │
   ▼                                     │
Headless Chrome  ◄──── TLS proxy (Go)    │
+ stealth JS          (always on)        │
+ 7-profile rotation                     │
   │                                     │
   ▼                                     │
HTML → Go html-to-md → markdown          │
                   │                     │
                   └─────────┬───────────┘
                             ▼
                  Firecrawl-compatible JSON

Why

Problem	cortex-bridge
Firecrawl Cloud costs scale with usage	Self-hosted, flat resource cost
Firecrawl Self-Hosted fails most WAFs	Headless Chrome + 18-patch stealth + TLS-impersonation proxy + reactive 7-profile rotation
Cloudflare challenges block Chrome in <2 s	Layer-5 instant detection + background solve + 30 s of synthetic human behavior
LLM agents need a scraper that speaks MCP	Built-in MCP stdio server (scrape_url, web_search)
Datacenter IPs are blocked by most anti-bots	Runs on your own residential/VPS IP, default-on TLS fingerprint proxy
Vendor lock-in	Firecrawl v1/v2-compatible JSON API, plain HTTP and stdio

Philosophy

1. Self-host first, no subscription. A scraper you pay per request for is a scraper that can rug-pull you. cortex-bridge is one Rust binary, two Go binaries, and a Chromium process. It runs on your hardware on your IP.

2. Coherent identity rotation. A "fingerprint" is not a single thing. It's the joint distribution of UA, TLS ClientHello, Sec-Ch-Ua, canvas noise, cookie jar, and behavioral state. We rotate all of them together, or none. Mismatched rotation (new UA, same cookies) is the easiest way to look like a bot.

3. Identity files don't follow the profile. Cookies, Login Data, IndexedDB, and Local Storage are bound to a user, not a browser. Copying them across rotated profiles is the exact signature anti-bots look for: "same user trying different fingerprints". Only lived-in non-identity state (History, Cache, Bookmarks, Preferences) is shared.

4. Fail fast on the same host. Three L2 rotations, then L3 fail. We don't waste 10 minutes burning through 7 profiles on a site that's IP-banning us.

5. Surgical stealth, 18 patches that matter. We fix the five things that actually trip WAFs (navigator.webdriver, plugin list, chrome.runtime object, canvas/WebGL leak, behavior sim), plus a physics-based human-behavior simulation (Bezier mouse trajectories, momentum scrolling, idle micro-movements, keyboard dynamics) and instant Cloudflare challenge detection. Adding more patches for diminishing returns is a distraction — the missing piece is time, and we buy it with a background task that simulates 30 s of "waiting out the challenge" after the HTML is returned.

6. LLM-native. Both the HTTP API and MCP stdio server expose the same two operations (scrape_url, web_search). An agent doesn't need to know which transport it's calling.

Anti-bot strategy

cortex-bridge uses five independent layers. Each is sufficient on its own for a class of sites; together they cover the vast majority of WAFs encountered in practice.

Layer 1 — Headless Chrome + stealth JS (18 patches)

A persistent Chromium process (via chromiumoxide) runs the full V8 + Blink stack, with STEALTH_SCRIPT injected via Page.addScriptToEvaluateOnNewDocument before every navigation. Patches:

#	What	Why
1	navigator.webdriver removed	Most basic headless detector
2	navigator.permissions.query for notifications returns Notification.permission	Default headless returns prompt: true
3	navigator.plugins populated with 5 real PDF plugin names	Default headless has empty plugins list
4	navigator.languages = ['en-US', 'en', 'fr']	Default headless has empty languages
5	window.chrome.runtime + csi + loadTimes + app populated	Real Chrome has these; default headless doesn't
5b	navigator.userAgentData (UA Client Hints) overridden to match UA	Without this, navigator leaks the real Chromium binary version while UA claims a different one
6	WebGL vendor/renderer masked to ANGLE/NVIDIA, MAX_TEXTURE_SIZE/MAX_VIEWPORT_DIMS fixed	Default SwiftShader leaks "Google Inc."
7	Canvas toDataURL adds deterministic per-profile noise (seeded by canvas_noise_seed)	Same canvas hash = same bot
8	OfflineAudioContext.startRendering adds subtle noise	Audio fingerprint leak
9	hardwareConcurrency = 8, deviceMemory = 8	Default headless has 0
10	screen.{width,height,availWidth,availHeight,colorDepth,pixelDepth} fixed to 1920×1080	Default headless has weird dimensions
11	document.fonts populated with 42 realistic font names	Default headless has 5 (detectable)
11b	WebRTC NOT suppressed (residential IP, leak is fine)	Some sites use WebRTC as proof-of-presence
11c	chrome.runtime + chrome.management listing 4 real extension IDs (uBlock Origin, Bitwarden, SponsorBlock, Vimium)	Empty extension list is a headless tell
11d	Human behavior simulation: cubic Bezier mouse trajectories with 15% overshoot + sub-pixel hand tremor, momentum scroll via WheelEvent with variable friction, Tab+ArrowDown keydown/keyup dynamics, idle micro-movements via requestAnimationFrame	No interaction = behavioral bot signal. Real humans have physics, not random points.
12	navigator.connection.{rtt, downlink} populated	Default headless returns null
13	Notification.permission returns "default"	Default headless returns "denied"
14	CDP stack-trace detection obfuscated via Function.prototype.toString	Sites that probe for CDP evaluate frames
15	document.hasFocus() returns true	Default headless returns false

Cookie consent auto-click (separate CONSENT_CLICK_SCRIPT) supports Didomi + common French "j'accepte" patterns, Onetrust, Cookiebot, Tarteaucitron, Axeptio, CookieFirst, CookieYes. "Continuer sans accepter" is intentionally NOT matched — it rejects non-essential cookies, which is a legitimate user choice we shouldn't impersonate.

Layer 2 — Coherent profile rotation (7 profiles: 5 Chrome + 2 Firefox)

On init(), cortex-bridge launches Chromium with profile #0 (p1-chrome120) and warms it up: navigate to google.com, duckduckgo.com, wikipedia.org, github.com. This seeds Cookies + History + Service Worker registrations so the profile looks lived-in.

Then propagate_warm_profile() copies Default/ from profile #0 to the other 6 profile dirs, skipping identity-bearing files:

SKIPPED (would be a "same user, different fingerprint" bot signature):
  Cookies, Cookies-journal
  Login Data, Login Data-journal
  Web Data, Web Data-journal
  Local Storage, Session Storage, IndexedDB
  Service Worker, Database, databases, Storage
  Extension State, Extension Scripts
  File System, Shared Dictionary

KEPT (lived-in non-identity state, makes rotated profiles look real):
  History, Top Sites, Bookmarks, Favicons
  Cache/, Code Cache/
  Preferences, Secure Preferences
  Network Action Predictor
  Safe Browsing Cookies

Each rotated profile has its own cookie jar, its own login state, its own fingerprint — but the same non-identity browser history.

idx	suffix	Browser	TLS profile	Notes
0	p1-chrome120	Chrome 120	chrome_120	default
1	p2-chrome124	Chrome 124	chrome_124
2	p3-chrome117	Chrome 117	chrome_117
3	p4-chrome130	Chrome 130	chrome_120¹	tls-client lacks chrome_130
4	p5-chrome110	Chrome 110	chrome_110	older JA3
5	p6-firefox123	Firefox 123	firefox_123	NSS TLS stack — first L2 target
6	p7-firefox120	Firefox 120	firefox_120	fallback Firefox

¹ tls-client lacks chrome_130; chrome_120 is the closest match.

All Linux x86_64. The two Firefox profiles carry empty Sec-Ch-Ua* and empty userAgentData.brands (Firefox doesn't implement UA Client Hints). Their canvas noise seed is also distinct.

Layer 3 — Reactive L0/L1/L2/L3 rotation

When a response looks like a block page (Cloudflare challenge, PerimeterX "Access to this page has been denied", Datadome interstitial, AWS WAF token, generic captcha), cortex-bridge escalates:

Level	Trigger	Action
L0 Accept	HTML looks like real content	Return as-is
L1 ClearAndRetry	First block on this host	CDP Network.clearBrowserCookies + clearBrowserCache + Storage.clearDataForOrigin. No Chrome restart. ~1 s.
L2 Rotate	Second+ block on same host	15 s cooldown → kill TLS proxy → spawn new proxy with new TLS profile → SIGKILL chromium tree → launch fresh Chromium on new profile dir. ~30-45 s.
L3 Fail	After MAX_ROTATIONS_PER_HOST=3 L2s	Return HTTP 403. Give up.

Firefox-first L2. The first L2 jump always targets profile index 5 (firefox-123-linux). Subsequent L2s cycle normally through idx → (idx+1) mod 7. This is deliberate: PerimeterX and Datadome have separate fingerprint buckets for BoringSSL (Chrome) vs NSS (Firefox), and many of the targets that fail Chrome pass Firefox unchanged.

L1 does not increment the rotation counter — only L2 does. Per-host state keyed by hostname, not global, so a block on leboncoin doesn't deplete our rotation budget for amazon.

Chromium termination is forceful: we walk /proc/<pid>/cmdline and SIGKILL every process matching our --user-data-dir (main + renderers + GPU + utility + crashpad + zygote — typically 8-12 processes). PGID-based kill doesn't work because chromiumoxide doesn't setsid() the browser, so Chromium inherits our process group and a group-kill would SIGKILL cortex-bridge itself.

Layer 4 — TLS fingerprint impersonation (always on)

Even with a perfect UA + stealth JS, Chrome 149's binary claiming to be Chrome/120 is detectable by advanced WAFs via TLS ClientHello (cipher list, extensions, ALPN, key share). The Go proxy (bogdanfinn/tls-client) re-issues each request with the configured browser's TLS fingerprint.

CORTEX_TLS_PROXY defaults to true (the proxy is on by default — turning it off breaks Datadome/PerimeterX/AWS-WAF targets). The Rust binary spawns the Go proxy before launching Chromium and overrides the proxy's TLS profile with the active Rust profile's tls_profile so the upstream fingerprint always matches what Chrome is presenting (UA, Client Hints, userAgentData). This fixes a class of mismatch bugs where the env-defaulted chrome_120 fingerprint was used against a Firefox identity.

The proxy is also profile-aware at the header layer: firefox_* profiles get Firefox-shaped defaults (no Sec-Ch-Ua*, no Upgrade-Insecure-Requests, Accept: text/html,…,*/*;q=0.8, Accept-Encoding: gzip, deflate, br, Accept-Language: en-US,en;q=0.5), chrome_* profiles get Chrome defaults. The proxy also forwards Content-Encoding (gzip, brotli, zstd) to Chrome verbatim so Chrome handles decompression natively — fixing the mojibake class of bugs.

Per-host CA cert is generated on first launch and stored under $XDG_DATA_HOME/cortex-bridge/tls-proxy-ca/. Chrome trusts it via --ignore-certificate-errors.

Layer 5 — Cloudflare challenge auto-detection + background solve

Cloudflare "Just a moment…" interstitials are detected instantly (one Runtime.evaluate, no poll) immediately after page.goto(). Three orthogonal signals:

• document.title.includes("Just a moment")
• document.body.innerHTML.includes("cf-browser-verification")
• document.body.innerHTML.includes("challenge-stage")

If any fires, fetch_with_options takes a different path:

1. Skip the network-idle wait (a CF challenge never goes idle).
2. Sleep 5 s to let the challenge page load its sub-resources.
3. Snapshot the current HTML via page.content() and return it to the caller immediately — the user does not wait.
4. Clone the page and spawn a detached tokio::spawn task that simulates 30–35 s of human behavior on the cloned page: random mouse moves, occasional smooth scrolls, rare Tab keypresses, then a smooth scroll to top and page.close(). This gives the Cloudflare challenge time to resolve and rotate cached tokens in the background.

The upper-layer rotation loop (/v1/scrape) sees the partial HTML and decides whether to treat it as REAL or BLOCK. On REAL, the background solve happens to also have completed, so the next request to the same host can succeed with the warmed-up profile.

Quick start

Docker

docker build -t cortex-bridge .
docker run -d \
  --name cortex-bridge \
  -p 3000:3000 \
  -v cortex-data:/var/lib/cortex-bridge \
  cortex-bridge

Bare metal

Requires: Rust 1.83+, Go 1.24+, Chromium 149+ (or any headless Chrome with remote debugging).

# 1. Build the Go binaries (static)
(cd tls-proxy && CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o tls-impersonate-proxy .)
(cd go-h2m    && CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o go-h2m .)

# 2. Build the Rust binary
cargo build --release

# 3. Make sure the Go binaries are discoverable
ln -sf "$PWD/tls-proxy/tls-impersonate-proxy" /usr/local/bin/
ln -sf "$PWD/go-h2m/go-h2m"                     /usr/local/bin/

# 4. Run
./target/release/cortex-bridge

Healthcheck: curl http://127.0.0.1:3000/health.

To start on a specific profile (e.g. Firefox 123 for sites that reject Chrome fingerprints):

CORTEX_PROFILE_IDX=5 ./target/release/cortex-bridge

HTTP API

All endpoints are POST, JSON in/out, Firecrawl v1/v2-compatible.

POST /v1/scrape

curl -X POST http://127.0.0.1:3000/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://www.zillow.com/homedetails/..."}'

{
  "success": true,
  "data": {
    "markdown": "## 1234 Main St\n\n3 bed, 2 bath, $450,000...",
    "metadata": {
      "title": "1234 Main St | Zillow",
      "language": "en",
      "source_url": "https://www.zillow.com/homedetails/...",
      "status_code": 200
    }
  }
}

Options: onlyMainContent (default true), timeout (ms), waitFor (ms), waitForSelector (CSS), cookieConsent (default true).

POST /v1/search

curl -X POST http://127.0.0.1:3000/v1/search \
  -H 'Content-Type: application/json' \
  -d '{"query": "rust async runtime comparison", "limit": 5}'

Returns SearXNG results, scraped through the same Chrome pipeline.

GET /health

Returns {"service": "cortex-bridge", "status": "ok", "version": "2.0.0"}.

MCP stdio server

cortex-bridge also speaks the Model Context Protocol on stdio. Same two tools as the HTTP API:

Tool	Description
scrape_url	Fetch a URL, return markdown + metadata. Same options as /v1/scrape.
web_search	Query SearXNG, return top N results as scraped markdown.

Run with CORTEX_MODE=mcp ./cortex-bridge or via the bundled mcp_servers config in Claude Desktop / Cursor / etc.:

{
  "mcpServers": {
    "cortex-bridge": {
      "command": "/usr/local/bin/cortex-bridge",
      "env": { "CORTEX_MODE": "mcp" }
    }
  }
}

Configuration

All config is via environment variables. No YAML, no config file.

Variable	Default	Description
CORTEX_HOST	0.0.0.0	HTTP bind address
CORTEX_PORT	3000	HTTP port
CORTEX_API_KEY	(none)	If set, requires Authorization: Bearer <key> on every request
CORTEX_CHROME_BINARY	/usr/bin/chromium	Path to headless Chromium
CORTEX_CHROME_DATA_DIR	$XDG_DATA_HOME/cortex-bridge/profile	Persistent profile base dir
CORTEX_PROFILE_IDX	0	Initial profile (0-6: see profile table)
CORTEX_USER_AGENT	Chrome/120	Override UA (keep aligned with TLS profile if using proxy)
CORTEX_SEARXNG_URL	https://searxng.cyrleb.dev	SearXNG instance for web_search
CORTEX_SEARXNG_MAX_RESULTS	10	Max results per search (1-50)
CORTEX_TLS_PROXY	true	Enable TLS fingerprint impersonation proxy (always on; disable with false)
CORTEX_TLS_PROXY_BINARY	(auto)	Path to tls-impersonate-proxy Go binary
CORTEX_TLS_PROXY_LISTEN	127.0.0.1:7890	Proxy listen address (Chrome --proxy-server)
CORTEX_TLS_PROXY_PROFILE	chrome_120	Fallback TLS profile if no Rust profile is loaded (chrome_120, chrome_124, chrome_117, chrome_130, chrome_110, firefox_123, firefox_120, firefox_117, firefox_110, safari_*). The Rust binary overrides this with the active Rust profile's tls_profile at init.
CORTEX_TLS_PROXY_CA_DIR	$XDG_DATA_HOME/cortex-bridge/tls-proxy-ca	Per-host CA cert storage
CORTEX_TLS_PROXY_BYPASS	localhost,127.0.0.1,::1	Hosts that bypass the proxy
CORTEX_RATE_LIMIT_MIN_MS	2000	Minimum interval between requests (ms)
CORTEX_RATE_LIMIT_JITTER_MS	500	Random jitter added to min interval (ms)
CORTEX_NETWORK_IDLE_QUIET_MS	500	Network-idle quiet period (ms)
CORTEX_NETWORK_IDLE_TIMEOUT_MS	15000	Network-idle hard timeout (ms)
CORTEX_H2M_BINARY	(auto)	Path to go-h2m Go binary (fallback to Rust html2md)
CORTEX_H2M_TIMEOUT_MS	10000	h2m subprocess timeout
CORTEX_MAX_MD_CHARS	100000	Max markdown size returned (truncated)
CORTEX_REQUEST_TIMEOUT_MS	45000	Per-request hard timeout
RUST_LOG	info	tracing_subscriber filter

Benchmark — 30 sites

Run with bash scripts/benchmark.sh. Cleans all profile dirs, starts the proxy + Rust binary on CORTEX_PROFILE_IDX=5 (Firefox 123 first), then scrapes 30 sites sequentially with 2 s spacing.

#	Site	Verdict	Contenu	Anti-bot
1	Wikipedia	✅ REAL	100K	—
2	Hacker News	✅ REAL	10.5K	—
3	GitHub	✅ REAL	9.3K	—
4	Reddit	✅ REAL	10.6K	—
5	Amazon.fr	✅ REAL	100K	Datadome (bypassé)
6	X/Twitter	⚠️ SHORT	46c	Login wall probable
7	Instagram	✅ REAL	2.3K	—
8	LeBonCoin	✅ REAL	38.4K	Datadome (bypassé)
9	OpenClassrooms	✅ REAL	6.0K	—
10	StackOverflow	❌ TIMEOUT	—	Cloudflare Under Attack
11	Example.com	✅ REAL	183c	—
12	YouTube	✅ REAL	4.4K	—
13	Google Search	✅ REAL	1.2K	—
14	Facebook	✅ REAL	5.1K	—
15	LinkedIn	✅ REAL	17.6K	—
16	TikTok	✅ REAL	3.3K	—
17	eBay	✅ REAL	18.8K	—
18	Airbnb	❌ BLOCK	—	Anti-bot
19	Booking.com	✅ REAL	51.9K	—
20	Walmart	❌ BLOCK	—	Anti-bot
21	PayPal	✅ REAL	4.4K	—
22	Indeed	❌ TIMEOUT	—	Anti-bot
23	Discord	✅ REAL	20.9K	—
24	Medium	❌ TIMEOUT	—	JS paywall
25	BBC News	❌ BLOCK	—	Anti-bot / geo-block
26	Le Monde	✅ REAL	100K	—
27	Quora	❌ TIMEOUT	—	Login wall
28	Craigslist	✅ REAL	8.9K	—
29	Spotify	✅ REAL	24.1K	—
30	Cloudflare.com	✅ REAL	14.3K	Cloudflare (lui-même !)

Result: 22/30 REAL ✅ (73%) · 0 mojibake · 4 BLOCK · 4 TIMEOUT.

Highlights:

• Reddit & OpenClassrooms were at ~43% U+FFFD mojibake before b1fa834 (Content-Encoding forwarding); now clean.
• LinkedIn, Booking, PayPal were proxy-zombie timeouts before the L2 Firefox-first rotation; now REAL on the first attempt.
• LeBonCoin (Datadome, the hardest public target) returns REAL on the first attempt thanks to the Firefox-first L2 + Firefox NSS fingerprint.
• The 8 heavyweight blockers (YouTube, Google, Facebook, TikTok, eBay, Discord, Cloudflare, Le Monde) all pass on the first try.

Known limitations

See docs/known-limitations.md for the full list. Top items:

• Chromium 149 binary vs UA mismatch is detectable via canvas/WebGL/renderer fingerprinting. The TLS proxy masks it at the TLS layer. Installing a matching Chrome binary is out of scope.
• Long rotation cycles — 3× L2 on a host takes 45–90 s (15 s cooldown × 3 + Chrome restart × 3). We don't shorten it: the cooldown simulates "user switched device" and helps pass anti-bot cooldown timers.
• Single-page behavioral detection — for anti-bots that demand minutes of natural browsing history (advanced PerimeterX on heavily abused targets), no amount of stealth in a 30 s scrape window helps.
• example.com returns a tiny body by design; raising EMPTY_THRESHOLD_BYTES in block_detection.rs trades false-positive rate for false-negative rate.

Architecture details

• src/main.rs — Axum HTTP server, MCP stdio server, singleton Chrome init, sync of Go-proxy tls_profile with active Rust profile.
• src/chrome/ — Singleton headless Chromium, split into 6 modules:
  • mod.rs — OnceCell<Arc<Mutex<Option<Arc<Chrome>>>>> slot, init/instance/rotate_session.
  • launch.rs — Chrome::launch / close / kill (forceful /proc walk), profile_dir, PID discovery.
  • fetch.rs — Chrome::fetch / fetch_with_options (page navigation, network-idle wait, CF challenge gate + background behavior spawn).
  • stealth.rs — STEALTH_SCRIPT (18 patches) + CONSENT_CLICK_SCRIPT + wait_for_selector_js.
  • actions.rs — Chrome::clear_storage_for_origin + warmup + lived-in propagation.
  • proxy.rs — tls-impersonate-proxy lifecycle (spawn, ready probe, kill).
• src/profiles.rs — 7 coherent identity bundles (UA + TLS + Sec-Ch-Ua + UA-data + profile_dir + canvas noise seed), including Firefox 123 / Firefox 120.
• src/rotation.rs — L0/L1/L2/L3 decision logic, first L2 jumps to Firefox.
• src/block_detection.rs — Pattern-based block detection (Cloudflare, PerimeterX, Datadome, AWS WAF, generic captcha, empty).
• src/network_idle.rs — Network-idle tracking via 3 CDP events (Network.requestWillBeSent, Network.loadingFinished, Network.loadingFailed).
• src/rate_limit.rs — Per-instance mutex-based rate limiter with random jitter.
• src/searxng.rs — SearXNG HTTP client.
• src/converter.rs — HTML → markdown pipeline (Go h2m first, Rust html2md fallback).
• src/routes/ — Axum HTTP handlers.
• src/mcp.rs — rmcp MCP stdio server.
• tls-proxy/main.go — Go MITM proxy using bogdanfinn/tls-client. Profile-aware defaults (chrome vs firefox headers), Content-Encoding forwarding to Chrome, Sec-Ch-Ua* strip + re-inject, per-host CA.
• go-h2m/main.go — Go HTML-to-markdown using html-to-markdown with GFM + RobustCodeBlock plugins (same as Firecrawl).

See docs/rotation.md for the full rotation strategy, docs/tls-proxy.md for proxy details, and docs/benchmark-30sites.md for the 30-site benchmark writeup.

Development

cargo build              # dev build
cargo build --release    # release build (LTO, opt-level z)
cargo clippy             # lints

Run the unit-free test suite (smoke):

cargo run --release
curl http://127.0.0.1:3000/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://en.wikipedia.org/"}'

Run the 30-site benchmark:

cargo build --release
bash scripts/benchmark.sh

License

MIT. See LICENSE.

Acknowledgments

• Firecrawl for the API shape and the html-to-markdown reference implementation
• bogdanfinn/tls-client for the Go TLS fingerprint library
• chromiumoxide for the Rust CDP bindings
• rmcp for the Rust MCP SDK