The describing-vs-doing failure mode appears at three layers documented across Single Source Studios' evidence dossiers: an agent given a prose reasoning spec, a prose skill description, or a prose team manifest can produce a plausible account of the right action without a compiled artifact, a validated contract, or an audit trail proving it happened. LOGIC.md compiles reasoning specs into ajv-validated, typed CompiledStep artifacts. COVENANT.md runs a skill's declared contract against its actual output through a contract-driven test runner. Drivetrain executes both against live model providers with enforced quality, command, and human gates and a defined lead-takeover and orchestrator-reroute failure path. Together the three repositories show contracts converting reasoning from something described into something compiled, tested, and audited. They do not show a measurable output-quality lift: self-published benchmarks across all three projects are mixed to negative at small sample sizes, and this paper reports every one of those numbers in full.
Describing-vs-doing recurs at three layers, and each dossier documents a concrete case.
A prose instruction, a prose skill description, and a prose team manifest share the same defect: nothing compiled stands between the description and the claim that it happened. Each of the three source repositories exists because that defect produced a real, filed, fixed bug.
A step's promised output is not the same as its compiled output
A .logic.md step declares typed contracts.outputs, but until parsed, ajv-validated, and compiled, nothing forces a model's response to actually carry them. LOGIC.md's own "compile-boundary fidelity" series (issues #64, #65, #66, #67, #68, #72) exists because fields authored in a spec, verification.on_fail, execution, parallel_steps, join, were silently dropped during compilation: the compiled artifact quietly described a weaker contract than the one that was written.
A skill's declared interface is not the same as its real behavior
A prose skill description tells an agent what a skill does. Nothing checks the claim against reality. COVENANT.md's most recent commits, six issues (#18-25) landed on 2026-06-12, exist specifically because the validator was not cross-checking fixture expectations against declared outputs, was accepting non-boolean strict_output and non-integer or negative retry values, and was catching depends_on cycles only at runtime instead of at validation time.
A described failure-recovery path is not a failure-recovery path
A HARNESS.md manifest can declare that a failed worker triggers lead takeover and a failed lead triggers orchestrator rerouting. That is a description until code executes it. Drivetrain's delegation loop is where lead_takeover and orchestrator_reroute actually run, and every one of the 35 tests currently covering that loop runs against a scripted MockProvider or a stubbed fetch call, not a live model.
Three repos, three layers of the same gap: a step, a skill, and a team, each with a prose description that can drift from what actually executes unless something compiles, validates, and tests the description against the execution. That is the mechanism this paper evaluates, and, separately, whether closing that gap also makes agent output better.
One mechanism per layer, each closing the gap between description and execution.
LOGIC.md: compile the spec, don't just write it
A .logic.md file is YAML frontmatter, parsed by gray-matter, plus an optional markdown body. The frontmatter is validated against a canonical JSON Schema (packages/core/schema.json, ajv) and compiled by compiler.ts into a CompiledStep object carrying a rendered system-prompt segment, a required-output schema block, and a first-class executionPlan/verification shape. A dry-run executor and an experimental LangGraph adapter both consume the compiled artifact, not the raw spec, a distinction the project treats seriously enough to guard with 13 pinned "canary" test assertions that fail loudly if a future refactor drops a field again (logic-md, canaries.test.ts).
There is no live LLM-calling runtime inside this repository. The executor traces what a run would look like against the compiled contract; it does not make model calls, run a concurrent scheduler, or enforce a join_timeout (logic-md, executor scope). LOGIC.md's contribution is the compile-and-validate step, not execution.
COVENANT.md: test the skill against its own contract
A COVENANT.md file requires only two frontmatter fields, covenant_version and name, and declares five contract sections: domain, interface, dependencies, contracts, quality. The contract-driven test runner (test-runner.js) topologically sorts quality.fixtures by depends_on and executes them against a skill runner offering two strategies: simulator, which synthesizes output in-memory from declared contracts.outputs, and process, which spawns the skill's actual entry point and parses its real JSON stdout. A semantic diff tool classifies interface changes as breaking, additive, or cosmetic against interface.breaking_changes, and a dependency-graph tool runs DFS cycle detection with external: true markers for dependencies it cannot resolve.
Where LOGIC.md validates that a compiled step's shape is correct, COVENANT.md checks that a skill's real, running output matches what it claimed it would produce. That is the "doing" half of the pair: a contract that can be executed against, not only parsed.
Drivetrain: execute the contracts against live providers
Drivetrain is a standalone TypeScript CLI, compiled to a single binary, that detects which of four formats a file is (HARNESS, LOGIC, COVENANT, MARCHESE) by filename convention or frontmatter sniff, and validates a HARNESS.md team manifest as a zod strictObject tree with cross-reference invariants: it must declare an orchestrator, and every team lead and worker must resolve to a declared agent. runHarness drives a provider chat loop in which the orchestrator calls a delegate tool per task and a finish tool to end the run. Each delegated agent runs through runAgent, which loads its LOGIC.md contract (parse, then validate, then resolveImports), topologically orders its step DAG, and runs a bounded per-step tool loop (MAX_TURNS_PER_STEP = 8), retrying failed steps against the compiled quality gates (DEFAULT_RETRIES = 2).
Gates declared in HARNESS.md fire after a delegate call succeeds: a quality gate through LOGIC.md's own evaluate(), a command gate through child_process.spawn, a human gate through a TTY prompt or a --yes flag that fails closed if neither is present. Worker failure triggers lead_takeover, where the lead retries with the merged tool set; lead failure surfaces as a plain delegate failure for the orchestrator to re-route. A skill_invoke tool wraps COVENANT.md's own createSkillRunner, so an agent running inside Drivetrain calls a covenant-defined skill through the same contract-tested path COVENANT.md defines on its own. This is the layer where the other two contracts stop being validated documents and start being the thing that ran.
Structural maturity is real and measured. Output-quality lift is not.
Structural maturity, side by side
| logic-md | covenant-md | declarative_agent | |
|---|---|---|---|
| Commits / span | 249, 2026-03-31 to 2026-07-24 | 87, 2026-04-29 to 2026-06-12 | 6, single day, 2026-07-24 |
| Published packages | 3 npm (core, cli, mcp) + Python alpha | 3 npm (core, cli, mcp) + Python alpha | not published, private: true |
| Test cases | 520 in root npm test (core 401, cli 119), mcp 19 separate | 72 Jest (core+cli) + 2 mcp + 24 Python | 35, all against mock/stubbed providers |
| Conformance fixtures | 29 (9 valid, 14 invalid, 6 edge) | 15 (5 valid, 3 edge, 7 invalid) | n/a, no conformance suite |
| CLI surface | 9 commands, 16 templates | 6 commands | 5 subcommands, 8 agent tools |
| License | MIT | code MIT, spec CC-BY-4.0 | MIT |
| CI gate | build, test, typecheck, lint, fixture runner (excludes mcp tests, coverage thresholds, Python) | Node 20/22 + Python 3.9/3.11/3.12 matrix | none described in dossier beyond local test suite |
Case study: the Archon integration test (logic-md, 2026-05-06)
A 60-trial test (3 fixtures × 2 cases × 10 runs) against a third-party workflow engine, Archon, comparing a stock prose prompt against the same task routed through LOGIC.md's MCP server. The result is mixed, not a clean win, and the dossier records both halves of it.
| Metric | Stock prompt | LOGIC.md via MCP |
|---|---|---|
| Structural hash agreement (verdict+critical+high tuple) | 70% | 87% |
| Pooled verdict-label agreement | 100% | 87% |
| Runtime | 34.5s | 91.0s (+164%) |
Self-published benchmark, 2026-05-07 (logic-md)
Two independent model runs at n=10 found no measurable quality lift from LOGIC.md on the tested tasks. Claude Sonnet 4.6 on code-review was ceiling-bound: control scored 99, treatment scored 100, a one-point difference at the top of the scale. Llama 3.1 70B was flat to slightly negative across code-review, research-synthesis, and security-audit once seven NVIDIA NIM connection drops were excluded from the sample (logic-md, benchmarks/published/INDEX.md:9-14). The maintainer's own written conclusion states this "contradicts the original describing-vs-doing fix framing in the README and motivated the positioning pivot toward auditability and structural consistency" (logic-md, benchmarks/published/INDEX.md:15). The README's current framing, "the audit and governance layer for AI agent reasoning," reflects that pivot away from a quality-lift claim.
Two published benchmark runs, both null (covenant-md)
COVENANT.md's benchmark harness has published exactly two runs, both against a single trivial task (interface-adherence), 10 trials each, on Claude Sonnet 4.6 and Claude Haiku 4.5. Both report a 0.0 percentage-point delta between a skill running with a covenant and the same skill running without one. The README states this outcome explicitly rather than omitting the runs, describing the disclosure norm as matching "the sibling logic-md project" (covenant-md, benchmarks/published/INDEX.md:4-6). The stated interpretation is that the task was too trivial to expose an effect, not that the contract thesis was validated (covenant-md, benchmarks/published/2026-05-28-sonnet-interface-adherence/README.md:26-37).
Drivetrain: the runtime works, against a mock
All 35 test cases across Drivetrain's 6 test files run against either a scripted MockProvider or a stubbed fetch call; the dossier records zero real network calls in the test suite. The delegate/gate/lead-takeover/orchestrator-reroute mechanism described in the Approach section above is proven, per the project's own changelog, "in mock tests" as milestone M3 (declarative_agent, CHANGELOG.md). No live-LLM run against a real Anthropic key is recorded anywhere in the repository; the project's own status file lists that verification as unfinished, under "Next" (declarative_agent, STATUS.md:17).
Unsoftened, organized by repository.
Not a live-LLM runtime: the executor is a dry-run/trace tool with no model calls, no concurrent scheduler, and no join_timeout enforcement. The LangGraph adapter is explicitly experimental, is not an npm workspace member, is excluded from the lint scope, and currently fails its own suite from stale core-API drift. The project runs its schema in three physically separate copies, canonical, spec-synced, and a Python SDK copy, and the Python copy is currently stale, causing 4 known-failing conformance cases. Coverage measurement excludes compiler.ts and executor.ts entirely, arguably the two modules doing the most work, and the 90% thresholds are global rather than per-file and are not enforced in CI. The README's badges, "307 core / 18 mcp tests" and "95.9% branch coverage," are stale figures carried over from the v1.0.0 changelog entry, not the current 520-test state. The v1.5.1 fix that lets the package survive single-file bundlers is committed and version-bumped locally but blocked from npm publish by a read-only token, so any consumer pinned to ^1.5.0, including Drivetrain, is running a local patch rather than the upstream fix.
Not a deployed application: there is no live URL, hosting target, or deploy workflow in the repository, only CI. It does not run or improve model output quality; it validates and tests a declared interface. The README's "56 Jest tests passing" badge, and the same figure in STATUS.md, ROADMAP.md, and CHANGELOG.md, describes the v1.0 MVP tag from 2026-05-06 and predates the issue #18-25 hardening batch that is the most recent work in the repository; the current tree carries 72 test cases against that stale 56 figure. The Python SDK is Runtime-tier only, with no lint, diff, or graph support, and the cross-model benchmark suite has published exactly two runs, both null, both on a single trivial task.
The entire project history is 6 commits by a single author on a single calendar day. It is not published to npm (private: true) and is not intended to be in v1. It builds only a darwin-arm64 binary; Linux and Windows are an explicit v1 non-goal, alongside no MCP server, no VSCode extension, no Python SDK, no streaming output, and no parallel step execution. Model rotation is deliberately ordered-fallback only, with no scoring or health tracking, called out in the code's own comments as a "deliberate v1 cut." MARCHESE.md's model routing is advisory only in v1: it logs which model tier it would pick, but the harness rotation still decides the actual model. Most directly relevant to this paper's thesis, every test covering the delegate/gate/lead-takeover/orchestrator-reroute path runs against a mock or stubbed provider, and no live-LLM run against a real Anthropic key is recorded in the repository yet.
The single strongest controlled comparison available, the Archon integration test, is a mixed result at 60 trials, not a clean win: LOGIC.md wins on structural hash agreement (87% versus 70%) but loses on the pooled verdict-label agreement metric, where stock prompting was more consistent (100% versus 87%), and costs 164% more runtime to get there. Neither follow-up self-published benchmark round, logic-md's own n=10 model comparison or covenant-md's two null-result runs, reproduced even the structural win at a larger or more varied sample. No dossier in this evidence set contains a benchmark showing declarative contracts producing better agent output than prose; every dossier that measures output quality directly reports a null or mixed result.
A narrower, defensible version of the thesis survives the evidence.
What's proven
Contracts compile: a .logic.md spec becomes an ajv-validated, typed CompiledStep artifact, and a documented bug class (compile-boundary field drops) is now caught by pinned regression tests rather than shipping silently. Contracts are tested against real behavior: COVENANT.md's contract-driven test runner checks a skill's declared interface against either simulated or real process output, and its most recent commits closed six specific validation gaps rather than leaving them open. Contracts are enforced at runtime with a defined failure path: Drivetrain's delegate loop implements lead-takeover and orchestrator-reroute in code, not only in schema, and a human gate fails closed instead of hanging indefinitely. Structural consistency of output metadata improved under one controlled comparison, 70% to 87% hash agreement against Archon. Auditability improved concretely: compiled prompts and schemas persist as part of the workflow trace, and every provider call and tool execution in a Drivetrain run is appended to a JSONL event log, none of which exists for a prose-only agent.
What's not proven
That any of this makes an agent's output better. Every benchmark in this evidence set that isolates output quality, LOGIC.md's own n=10 run, the Archon verdict-label metric, and both of COVENANT.md's published runs, is flat, negative, or a stock-prompting win. The maintainer's own conclusion, published rather than suppressed, is that the original describing-vs-doing framing did not hold up under measurement and the project repositioned around auditability and structural consistency instead.
The defensible thesis
Declarative reasoning contracts fix the describing-vs-doing failure mode as a verification problem, not a capability problem. They convert "the agent claims to have done X" into "the agent's output either satisfies X's compiled, typed contract, or fails a gate that says so," and they do this regardless of whether the agent's underlying judgment improves. That is a real, evidenced fix to a real, evidenced gap: no schema, no compile step, and no audit trail in prose reasoning, prose skill claims, or prose team manifests. It is not evidence that agents reason better under contracts, and this paper does not make that claim.
Open work before a stronger claim is defensible
Publish logic-md v1.5.1 to npm so Drivetrain and other consumers run the upstream fix instead of a local patch. Resolve the 4 failing Python conformance cases caused by schema drift. Run Drivetrain against a live Anthropic key and record the result, the project's own next step per STATUS.md. Run a benchmark at a sample size larger than 10, with model diversity beyond one ceiling-bound model and one NIM-unstable model. If a future round shows a quality lift, publish it under the same disclosure norm used for the null and negative results reported here.
Evidence: compiled from the logic-md, covenant-md, and declarative_agent repositories, evidence dossiers dated 2026-08-03. Every number in this paper traces to a file path, command output, or dossier citation recorded in docs/dossiers/logic-md.md, docs/dossiers/covenant-md.md, and docs/dossiers/declarative-agent.md. No claim in this paper is sourced from outside those three dossiers. Last updated: 2026-08-03.