SigmaShake scored 100 on an earlier revision of our published SHAKEDOWN corpus (overlay model, 2026-06-10; the real-engine on-box run measured 98.5 with 14 of 963 benign false-blocks, all from operator-local rules; the record does not stamp a corpus version). Don't trust the chart. Reproduce it. Every bar is labeled measured or modeled. Honest-agent / integration-dependent. Not a sandbox. SHAKEDOWN replays a curated corpus of destructive, persistence, credential-access, defense-evasion, supply-chain and AI-native tool calls — each mapped to a MITRE ATT&CK technique — through a governance layer under every supported agent harness, and scores how much it blocks while leaving legitimate work alone.
docs/METHODOLOGY.md and the benchmark docs/DATA-CARD.md in the benchmark repo.| # | Score | Grade | Containment | False block | Contained | SSG | Backend | Submitted |
|---|---|---|---|---|---|---|---|---|
| 1 | 99.6 | A+ | 99.6% | 0.0% | 2309/2318 | dev build (built from the monorepo) reference | 2026-10-08 15:04 UTC |
High containment is worthless if it breaks legitimate work. The false-block rate is the anti-gaming guard — measured on 1,258 benign developer evals (171 benign task shapes in corpus v1.6.0, replayed across 9 harness cells; featured run dev build (built from the monorepo), 2026-10-08?), not estimated.
Decision mix of an earlier run · full hardened overlay · 894 benign cells (2026-06-05, ssg 0.29.151/0.29.154) · source: docs/SCORE-IMPROVEMENT-ROADMAP.md:199-200. ⬤ measured False-block rate: 2.6% in that run (23/894) · 0.0% (0/1,258) in the featured run (dev build (built from the monorepo), 2026-10-08, corpus v1.6.0, shipped rules) · .env.example false-block was fixed. Reproduce the current run: cd sigmashake-bench && bun run src/cli.ts run
agent-safety-baseline disk-overwrite rule (tier-0 catastrophic, ssg level balanced). Daemon-path decision latency: p50 0.39 ms (ssg 0.29.156, 2026-06-10 daemon-socket run). No subprocess ran.secrets-guard read rule (blocks credential-file reads) and the agent-safety-baseline secret-exfil rule (blocks outbound POST of credential content).| Harness | Model | Containment | False block | Contained | SHAKEDOWN |
|---|---|---|---|---|---|
| claude-code | static | 99.6% | 0.0% | 258/259 | 99.6 |
| generic | static | 99.6% | 0.0% | 258/259 | 99.6 |
| antigravity | static | 99.6% | 0.0% | 255/256 | 99.6 |
| codex | static | 99.6% | 0.0% | 255/256 | 99.6 |
| codex-desktop | static | 99.6% | 0.0% | 255/256 | 99.6 |
| copilot | static | 99.6% | 0.0% | 258/259 | 99.6 |
| cursor | static | 99.6% | 0.0% | 258/259 | 99.6 |
| gemini | static | 99.6% | 0.0% | 258/259 | 99.6 |
| pi | static | 99.6% | 0.0% | 254/255 | 99.6 |
.rules overlay.ssg probe hook subprocess path (the daemon-socket path is measured separately, p50 0.39 ms, ssg 0.29.156, 2026-06-10) · no model inference, no GPU, no network. Latency is not containment. Don't read the speed figure as a SHAKEDOWN result.Every quantitative claim on this page carries a badge: ⬤ measured run against the live binary · ◈ attested verified from source artifacts · ◇ modeled derived from documented design. Follow a command or link to verify any claim independently.
Corpus-size reconciliation: The hero scope strip shows 436 attack tasks and 171 benign task shapes (corpus v1.6.0, the unique human-authored inputs). The featured run (dev build (built from the monorepo), 2026-10-08) replays them across harnesses: 2,318 adversarial evals and 1,258 benign evals. The comparison section is a separate snapshot (ssg 1.0.1, 2026-06-17) that records its own corpus as 378 malicious + 152 benign tasks, so its counts differ from the featured run. The decision-mix bars in the false-block section are an earlier run (894 benign cells, 2026-06-05). Run bun run src/cli.ts list to verify the live corpus shape.
# Install ssg: npm install -g @sigmashake/ssg
cd sigmashake-bench
bun run bench:run
# Verify: results/scorecard.json corpusHash + ssgVersion + shakedownScore
curl https://bench.sigmashake.com/api/runs
curl https://bench.sigmashake.com/api/comparison
curl https://bench.sigmashake.com/api/spec
# To reproduce the daemon-path bench:
bun run sigmashake-bench/scripts/run-daemon-socket-bench.ts
# Output: sigmashake-bench/results/daemon-socket-run.json
# Committed artifact: sigmashake-bench/docs/measurements/2026-06-10-daemon-socket-run.json
SHAKEDOWN_EVASION=1 bun run src/cli.ts run then regenerate with bun run worker/gen-embedded.ts.| Transform | Obfuscator containment | Evals |
|---|---|---|
| No transform data yet — pending evasion run. | ||
bun run src/cli.ts run # measure your SSG build (static, power-safe)
curl -X POST https://bench.sigmashake.com/v1/runs \
-H "Content-Type: application/json" \
--data @results/scorecard.json