GitHub - Coverage-Cat/consumer-agent-testing-tool: Consumer-agent surrogate testing for MCP servers, apps, browsers and APIs, with service-side outcome checks and sealed evidence

Consumer Agent Testing Tool (CATT) tests how an assistant uses your service. Run a scripted or model-driven agent against an MCP server, an MCP app, a website or an HTTP API. Check what the service did, inspect the evidence, and compare results across builds. Muse and Instinct are supported surrogate emulation targets. Muse remains the default. Both use the shared connector testing contract, including routing, memory, payments, browser/HTTP tasks and scheduling. See the Instinct research for its sources and limits. The model that drives a trial is separate from the assistant being emulated. Contributing | Security | Changelog | MIT license A test host. CATT is a surrogate: it approximates an assistant's host, not its private implementation. A passing test does not certify compatibility with Muse or any hosted assistant. Each Muse-specific behavior has a recorded basis: documented, unofficial, assumed or unknown. This project is not affiliated with or endorsed by Meta, Instinct, OpenAI or Anthropic. What you can test today Lane Command What it checks Status and scope MCP connectors connector-run SCENARIO Use the server's tools, apply account and approval rules, protect private values, and check the service's recorded outcome and the agent's answer. Muse and Instinct surrogates: --emulation-target muse|instinct.

See connector lane.

Connector extensions connector-run SCENARIO with extensions: Routes, memory, custom API clients, browser actions, wallet cards, background jobs, untrusted content, channel limits and simulated users. Supported with the lane. user_simulator is exploratory and cannot run in scripted tests or decision contracts.

See extensions.

Scenario scaffolding scaffold URL --output FILE, scaffold --from-run RUN --output FILE Read a tool catalog or turn an agent run into scripted calls. Calls no service tools. Replace every TODO(scaffold) before running.

Supported.

One Streamable HTTP connector; no OpenAPI input. Muse connector readiness check URL --suite muse-guidelines Review the catalog and optional manifest against the pinned connector guidelines without calling tools. Documented constraints, not admission.

See readiness checks.

Shared surrogate readiness check URL --suite surrogate --emulation-target instinct Check the catalog, allowlist, schema syntax and selected projection without calling tools. CATT's contract for either target; see shared contract. Personal Agent Protocol check ORIGIN --suite poppy Read poppy.json. With a manifest, probe sessions, sign-in and one declared operation on the company's hosts. Draft 0.1; no production Muse evidence. Muse gadget registration check FILE --suite gadget Check a link.register message and command specs against the pinned SDK limits. Opens no connection. Documented limits, not admission. The SDK's terms say it is not a developer platform. MCP protocol contracts check TARGET Check schemas, version negotiation, sessions and operation contracts; run the pinned conformance suite.

Supported.

Generic MCP evidence; see readiness checks. Embedded MCP apps surface-run SCENARIO Render the app in Chromium and check forms, tool calls, resources and service responses. An agent driver chooses from the scenario's scripted steps.

Supported.

Generic MCP Apps host, not Muse. See host scope. Websites browse POLICY, run --surface browser Act on accessibility snapshots under request rules. Takeover pauses the run (exit 3); the CLI has no handback. Native Chromium, without OS isolation. Browser scenarios use only the packaged reservation portal. HTTP APIs run --surface api-tools, inspect, check Bind approval to each proposed call and check backend outcomes. Supported with a mapping. Your API also needs a loopback deployment and private control adapter. See API scenarios and API trials. Evidence report, export, replay, diff Inspect reports, redact exports, verify playback and compare runs.

Supported.