MCP Server Qualification
Original project-agnostic procedure. Agent-host effectiveness remains experimental.
When to use
A new MCP server or significant update is being considered, an internal adapter will wrap it, or a previously working integration behaves differently. Qualification is tied to a particular revision, transport, configuration and identity.
When not to use
A catalog entry alone is being read for research. Do not execute installation instructions from untrusted descriptions, connect accounts, expand scopes, or run all discovered tools merely to fill a checklist.
Inputs
Resolve the authorized goal, canonical source/policy baseline, actual tool versions, owned resources, required evidence and task limits from the consuming project. Keep private roots, credentials and concrete command bindings in its adapter.
Procedure
1. Resolve a verifiable candidate
Record upstream repository/package/remote endpoint, publisher namespace, exact version/digest, transport and required runtime. Review dependency and installation effects, including hooks, subprocesses, browser profiles and outbound recipients. Registry namespace verification establishes a publishing relationship, not a code-security certification. Keep unresolved license or provenance concerns explicit.
2. Build an operation and authority map
List tool schemas, resources, prompts, dynamic discovery, session state and optional capabilities. Classify actual reads, writes, external effects and sensitive-data access. Treat annotations and tool descriptions as claims to verify, not enforcement. Reject surprising additions or broadened schemas during upgrade review. Bind trusted account/tenant context in the adapter, not client-supplied identifiers.
3. Prepare a disposable least-privilege test
Use a pinned qualified Inspector or host and inspect its actual help/schema before constructing version-sensitive commands. Isolate filesystem, process, network and account permissions; redact secrets in transcripts. Keep the test server fixture separate from live production systems. A remote transport needs its real authentication and origin/audience policy; a local process inherits caller authority unless constrained.
4. Exercise protocol and selected behavior
Test initialization, negotiated version/capabilities, paged tool/resource discovery, intended calls, malformed arguments, unsupported methods, errors, cancellation and reconnect behavior where relevant. Include an unauthorized operation and verify no protected side effect occurred. Successful listing does not make an untested tool qualified. Keep output-size, resource URI, embedded instruction and callback boundaries explicit.
5. Record a scoped verdict and pin
For each operation report pass, fail, blocked or not-run, with independent acceptance evidence. Separate protocol correctness from application correctness, security posture and installation support. Permit only reviewed operations in the consuming adapter. Requalify changed schemas, scopes, recipients, binaries or session semantics and retain a known prior configuration for recovery.
Output
A revision-bound qualification matrix: candidate provenance, installation effects, transport/runtime, operation schemas, authority requirements, executed evidence, untested areas, decision and upgrade triggers. Do not label the entire server safe based on one successful call.
Failure handling
If list-tools passes but a restricted mutation succeeds, qualification fails. Preserve evidence and prevent that operation from being exposed. If the intended call cannot run because no test credential exists, mark it blocked; do not substitute a privileged production account.
Example
A synthetic server exposes documents.read and documents.delete. An authorized test identity may read but not delete. The protocol inspector can list both; the adapter must still deny deletion, and an independent state check must show the fixture document remains. This example is a not-run host scenario until the server, identity and evidence are actually exercised.
Evaluation
references/scenarios.json contains three not-run agent-host inputs. Executed utility, parser and browser fixture tests are separate evidence and do not establish that an agent follows this procedure.