Skip to main content
CTPF separates what the agent invoked, what a tool returned, what persisted, what was consumed later, and what changed outside the agent. This prevents a tool call or manipulated response from being mistaken for a demonstrated external effect.

​
Conditions

The default experiment sequence runs all three conditions with condition-scoped state and run IDs. Condition order, model/runtime pins, prompts, and failures are retained in the series manifest.

​
Evidence layers

  1. Invocation — a tool or capability was requested with recorded arguments.
  2. Original result — the unmodified server response was preserved.
  3. Modified result — the declared intervention was recorded separately from the original.
  4. Persistence — a matching run-scoped artifact was written, when the scenario requires it.
  5. Later consumption — the persisted content was read in the required later session or hop.
  6. Higher-authority invocation — the scenario’s prohibited or privileged action was requested.
  7. External effect — an independent run-scoped sink confirms the exact matching effect.
Not every scenario uses every layer. Its scorer must declare the required causal chain before the run. A modified response is not itself propagation, and an invocation is not itself an effect.

​
Mechanical outcomes

These outcomes are local to a scenario and its pinned conditions. They do not establish a population rate, general model vulnerability or resistance, production impact, or mitigation efficacy.

​
Fail-closed comparison

CTPF does not reconstruct missing observations or promote a partial success into a result. A dirty baseline, mismatched run ID, absent original response, missing persisted artifact, broken cross-hop continuity, or absent effect artifact forces INCONCLUSIVE when that evidence is required. Mechanical classification is only the first step. Confounds and construct validity remain part of the human interpretation recorded with the study.