Config Agent Phase 0 measurement gate

This PR adds the RFC, per-turn telemetry, always-on eval graders, a regression corpus, and a pinned promotion gate while preserving current Config Agent behavior except for one schema compatibility fix.

PR #6689 Author @mehulshinde Branch mehul/dea-7353-7354-config-agent-phase0 State open Files 39 Scope apps/server/src/dealops3/configAgent Diff +6587 / -1105
What it adds

Measurement infrastructure for the Config Agent ChangeSet rollout: turn telemetry, eval graders, a checked-in suite baseline, and a phase-aware gate.

p50 12.4s p90 39.2s
What it changes

The eval harness now grades every turn for fidelity, receipt truthfulness, leaked internals, malformed retries, latency, and budgets.

97 harness tests 6 audit regressions
What it preserves

The agent should behave the same after this PR. The one deliberate runtime fix removes dependency-version-sensitive $ref output from Gemini tool schemas.

Reviewer focus

Review the gate thresholds, known-failure policy, Datadog pipeline requirement, and the non-whitespace parts of router.ts.

The provided diff is truncated, so this explainer relies on the PR description plus the visible diff.

Config Agent runtime
Turn telemetry
Eval harness
Always-on graders
Promotion gate
Datadog assets

1. Why this exists

Before

No falsifiable Phase 0 baseline

The 2026-08-17 audit reconstructed latency from DB message timestamps. There was no router-level turn metric to compare rollout phases against.

  • No direct totalMs per user turn.
  • Malformed tool-call retry storms were invisible.
  • Latency attribution required post-hoc reconstruction.
Before

Eval checks missed trust failures

The Xendit incident passed existing checks: the plan said update, the receipt read fine, and rows landed, but the executor created duplicates.

  • PlannedAction has no SKU-level diff.
  • “Update 29” and “create 70” collapsed to one coarse action.
  • Success claims were not tied to DB evidence.
Phase 0 intent: make later ChangeSet rollout phases measurable before changing the mutation model. This PR is infrastructure-first; bugs found by the new graders mostly become acceptance tests for later rewrite tickets.

2. What changes

The meaningful diff clusters into telemetry, eval grading, promotion gating, parallel execution, regression data, docs, and one runtime schema fix.

Runtime telemetry

Adds a per-turn turn event and a malformed_retry event in the Config Agent runtime.

FilesChange
telemetry.ts, router.ts, orchestrator.ts Wraps stream handling in a telemetry scope and rolls up duration, rounds, tool calls, tokens, retries, ChangeSet counts, and outcome.
turnTelemetry.spec.ts Pins the runtime telemetry behavior.
datadog/* Adds dashboard, p90 monitor, log pipeline, and operational README.
Always-on graders

Every turn of every eval case now gets global checks that catch production failure classes without per-case opt-in.

GraderCatchesPrimary file
ChangeSet fidelity Applied DB delta differs from the proposed deterministic diff. graders/changeSetFidelity.ts
Receipt truthfulness Past-tense success claim with no write evidence. graders/receipts.ts
No leaked internals R2 keys, fingerprints, raw CSV codes, or sub-agent directives in user-visible text. graders/receipts.ts
Tool-call well-formedness A turn that exhausts all malformed-function-call retries. graders/budgets.ts
Promotion gate

Adds pinned thresholds in thresholds.ts, a checked-in baseline in datasets/baseline.json, and CLI support for --gate, --phase, and --record-baseline.

fidelity 100% 0 unreceipted claims 0 leaks known failures tolerated by ticket
Parallel eval harness

Moves the suite from serial org reuse to sharded eval orgs, attempt-level scheduling, Redis run locks, and retry-safe reset.

  • --concurrency N runs each worker against its own cpq-agent-eval* org.
  • runLock.ts prevents two suites from silently corrupting each other.
  • Scheduling by attempt cuts the full eval run from about 71 minutes to 9.6 minutes.
Regression corpus

Adds six production-audit replay cases and CSV fixtures under apps/server/evals/configAgent/datasets.

None is required: true yet; promote cases after they have been observed green.

Runtime schema fix

Fixes dealops-r31f: reused Zod sub-schemas emitted $ref, which could 400 every Gemini sub-agent tool call and block CSV preflight.

The PR says this is not live in prod in the last 30 days and is dependency-version dependent, but it is the one behavior-affecting change here.

3. Before / after

Old eval signal

“Did the case look okay?”

  • Case-specific expectations dominated.
  • Latency was wall-clock or reconstructed.
  • Malformed retries looked like slow turns.
  • Known red cases made a gate impossible.
  • Two concurrent suites could share and reset an org.
New eval signal

“Did every turn satisfy Phase 0 invariants?”

  • Global graders inspect every turn.
  • Router emits server-side turn telemetry.
  • Gate compares to a checked-in baseline.
  • Known failures are explicit, ticketed, and excluded from metrics.
  • Redis locks prevent shared-org corruption.
Suite baseline
12.4s p50
39.2s p90

Recorded from datasets/baseline.json: concurrency 12, repeat 3, 144 turns.

Final run summary
28/32 passed

2 flaky, 1 skipped, zero harness errors, fidelity 100%, zero unreceipted claims, zero leaks.

Speedup
9.6 min full suite

Down from about 71 minutes through sharded orgs, attempt scheduling, and run locks.

4. How it works

User turn The eval sends a real Config Agent stream request.
Telemetry scope AsyncLocalStorage collects nested LLM, sub-agent, retry, and outcome events.
Harness inspect Before/after DB snapshots identify writes, draft changes, spec drafts, pricebooks, and approval groups.
Graders Fidelity, receipts, leaks, malformed retries, and per-turn budgets run on the same turn.
Gate thresholds.ts classifies failures and compares metrics to the baseline.
Telemetry implementation

Uses an AsyncLocalStorage scope instead of threading a metrics object through call sites.

  • Nested sub-agent calls count automatically.
  • Summarizer and orchestrator work roll into the same turn.
  • Eval harness subscribes in-process via subscribeTelemetry.
Fidelity implementation

Compares the declared deterministic projection to draft rows changed during the approve turn.

declared: create/update/delete counts
applied:  draft rows whose updatedAt changed this turn

If there is no typed projection, the grader reports n/a rather than inventing evidence.

Gate implementation

KNOWN_FAILURES allows tracked red cases without letting unlisted failures pass.

  • Flaky cases are tolerated but surfaced.
  • Harness errors are INCONCLUSIVE, not regressions.
  • Baseline comparison refuses mismatched concurrency or exclusion scope.
Threshold Phase 0 behavior Why it matters
ChangeSet fidelity 100% Any applied delta that differs from the proposed diff is a trust break.
Unreceipted success claims 0 The agent cannot say work completed unless a write backs it.
Leaked internal text 0 turns Server-authored scaffolding must not appear in customer-visible copy.
Malformed retries exhausted 0 turns Ten failed retries means the user saw a tool-call failure.
Raw malformed-call rate ≤ baseline with tolerance A model or schema shift can silently burn latency without visible errors.
Suite p50 latency ≤ baseline for phases 0–1; ≤ 70% for phase 2+ Makes the RFC’s “later phases get faster” claim falsifiable.

5. Bugs found by the new machinery

Bead Finding Status in this PR
dealops-72ey P0 CSV plan declares creates/updates incorrectly in the Xendit-shaped flow. Left for T7; regression case is deterministic acceptance coverage.
dealops-r31f P1 $ref in Gemini tool schemas can 400 sub-agent tool calls. Fixed here in geminiToolSchema.ts.
dealops-49gb P1 File-action update silently drops the price column. Left for T10 / Phase 3 deletion path.
dealops-jg5z P1 Agent narrates a mapping form it never emits. Left for T10.
dealops-6m3n P2 Raw Zod error appears beside a success claim. Left for T7.

6. What it doesn't change

Explicit non-goals

7. Risks / rollback / open questions

Deploy blocker: apply apps/server/src/dealops3/configAgent/datadog/log-pipeline.json before deploying the telemetry. Without it, Datadog receives the event JSON as an opaque string, so numeric facets, percentile queries, and the p90 monitor do not work.
Review risk

router.ts looks huge because wrapping /stream in a telemetry scope re-indented the handler.

Use git diff -w; PR description says it drops to 59 insertions / 7 deletions ignoring whitespace.

Gate risk

Known failures are necessary to record a baseline, but they are a policy surface.

  • Each entry must name a ticket.
  • Recovered known failures should be removed.
  • New unlisted failures still block.
Baseline risk

Latency depends on eval DB shape and Neon compute size, which the harness cannot detect from Postgres.

Re-record if the branch, pooling mode, compute size, concurrency, model, or suite contents change.

Rollback shape

Telemetry and eval-only changes can be reverted without data migration. The runtime schema fix in geminiToolSchema.ts is the only behavior-affecting code path to isolate if needed.

Still open after merge