public · MITagent systems

Archon

Autonomous agent in Python — planner/executor/reflector with a typed middleware chain (schema validation, retries, tracing on every tool call) and a statistical evaluation harness: seeded runs, per-step traces, and a failure-mode taxonomy.

github

Test suite180+Deterministic fakes · live APIs opt-in

Eval metrics5Tool-call · schema · recovery · efficiency · answer

SignificanceBootstrap CICohen's d · Cliff's δ · Mann-Whitney U

Reproducibilityseed 42Run manifest + SHA-256 config fingerprint

The problem

Agents fail in ways unit tests miss: a malformed tool call, an unrecovered error, behavior that drifts between runs. Archon treats the agent loop as something to measure — a planner/executor/reflector cycle with a typed middleware chain around every tool call, and a statistically rigorous eval harness so reliability shows up as confidence intervals instead of anecdotes.

Architecture

A user task enters the Planner (a swappable Protocol)→
each step passes through a composable middleware chain (tracing→
token-budget→
rate-limit→
telemetry, onion model)→
the Executor runs schema-validated tool calls→
a two-phase Reflector (heuristic + LLM) routes the result: CONTINUE, RETRY with a correction hint, REPLAN, SKIP, or ABORT→
the eval harness replays seeded runs across multiple LLM backends and scores tool-call accuracy, schema adherence, error recovery, step efficiency, and final-answer quality.

Key decisions

DECISIONCHOICEWHY

AbstractionsProtocol-based dependency injectionEvery contract is a typing.Protocol (structural subtyping), so LLM backends and tools swap without touching the executor, and tests run against deterministic fakes.

ErrorsTyped exception hierarchyRetryableError vs FatalAgentError is encoded in the type, not guessed by callers — one except branch catches all retryable subtypes, and each error maps to a failure category.

Cross-cutting concernsComposable middleware (onion model)Tracing, token budgets, rate limiting, and telemetry are interceptors in a chain — adding PII redaction or audit logging is one class, not a change to core orchestration.

TestingDeterministic fakes over mocksDeterministicFakeBackend implements the LLMBackend protocol with pre-programmed responses, so tests verify real behavior (not mock config) with zero network calls.

EvaluationNon-parametric statisticsLLM scores are not normally distributed, so the harness uses bootstrap CIs, Cohen's d, Cliff's delta, and Mann-Whitney U instead of parametric tests.

Status

Public project. Library RNG is reproducible for a given seed, but remote LLM APIs are not bit-reproducible even at temperature 0 — deterministic integration tests use fakes. Remaining production gaps (TLS, reverse proxy, HA) are documented rather than hidden.

PythonAgentsEval