‹ Agent Governance Index

Is OpenBMB/ChatDev production-ready?

Mechanical governance score from agent-contracts scan against a clean clone. No vibes, no testimonials — reproducible from the exact commit below.

80/100 B no change since the previous scan
Weakest dimension: eval_harness (0/15). This measures an eval / benchmark / labeled-corpus surface. A low score here means an agent built on ChatDev can fail in this dimension without the repo signalling it.

Dimension breakdown (100 points)

tests_and_ci — a test suite plus CI that runs it15/20
tool_governance — permission / allowlist / approval / human-in-loop gating on tool calls20/20
secret_safety — secrets git-ignored, no obvious hardcoded keys15/15
eval_harness — an eval / benchmark / labeled-corpus surface0/15
dependency_pinning — a lockfile or pinned dependency manifest10/10
observability — logging / audit / tracing10/10
resilience — retry / backoff / fallback / escalation10/10

Reproduce this score

pip install "agent-contracts @ git+https://github.com/impartshadow/agent-contracts.git"
git clone --depth 1 https://github.com/OpenBMB/ChatDev.git
git -C ChatDev checkout a6a5cda556
agent-contracts scan --root ChatDev --json

Scored commit: a6a5cda5560053136897aa44301eacc6c48d8168. The score is the output of a deterministic scanner, not a judgment call — a clean clone reproduces it.

What this score can and can't tell you

This index measures observable governance surface — the guardrails a reliable agent needs. It does not certify that agents built on ChatDev behave correctly at runtime. The most expensive agent failure — confidently claiming a task is "done" with nothing to back it — has no surface to grep, so no static score (including this one) can see it. Here's the proof, on the #1-ranked framework.

Get the weekly scan delta — $7/mo See the full index

Every framework is re-scanned on a schedule. The weekly delta — what moved, on which commit, and what it means for teams building on it — goes out at echofromshadow.substack.com.