Check an MCP release

Callstead compares two complete tool inventories and checks the saved inputs and outputs you supply. All analysis runs in an isolated browser worker. It does not connect to your MCP server or execute tools.

Capture comparable evidence

Use your existing MCP client to save the complete tools/list result before and after an update. Keep the same account, permission scope, configuration and test environment. Combine all pagination pages; incomplete inventories are rejected.

Record test calls in an environment you control. Review any operations that change state. Copy the resulting structured values into a case file. Callstead does not record or replay calls for you.

Case format

Use either baselineOutput/candidateOutput for structured values or baselineResult/candidateResult for MCP result envelopes. Do not use both for the same side. Context labels must identify equivalent capture conditions; they are your assertion, not an independently verified identity.

{
  "context": {"baseline": "sandbox-read-role", "candidate": "sandbox-read-role"},
  "cases": [{
    "id": "known-record",
    "tool": "lookup",
    "arguments": {"id": "example-42"},
    "baselineResult": {"structuredContent": {"status": "ready"}},
    "candidateResult": {"structuredContent": {"status": "ready"}},
    "assertions": [
      {"kind": "equals", "path": "/status", "value": "ready"},
      {"kind": "maxSideEffects", "value": 0}
    ],
    "observations": {"sideEffects": 0}
  }]
}

Download complete fictional examples: baseline, candidate, cases.

Supported assertions

equals: the JSON value at path matches value. Object key order is ignored; array order matters.
exists: a property exists, including when its value is null or false.
maxDurationMs: the supplied recorded observations.durationMs is within the budget.
maxSideEffects: the supplied recorded observations.sideEffects count is within the budget. This count is not proof of idempotency or permission.

Paths are JSON Pointers. An empty path selects the whole output. / selects an empty-name property. Escape / as ~1 and ~ as ~0.

Read the result

Blocked: a supplied call or assertion fails, or a baseline tool is absent from the candidate inventory.
Review: a schema, description or advertised hint changed, or coverage is incomplete.
Invalid: input, metadata, context or a schema cannot be evaluated reliably.
Checked: supplied assertions were evaluated without a finding. This does not establish general compatibility or live correctness.

Deliberately bounded

Up to 512 KiB per document, 100 tools, 200 cases, 50 assertions per case, 40 levels of nesting and five seconds per check. JSON Schema Draft 2020-12 is supported. Unknown dialects, keywords, formats and non-local references are rejected. Only structured output is checked; text, images and other MCP content are not semantically evaluated. Provider-specific extension keywords can require a different validator.

Large integer identifiers must be strings to preserve precision. Duplicate JSON members are rejected. Files are not saved by Callstead; export reports before closing or clearing the page. An incomplete or timed-out check is never a passing result.