Harden AI quote generation: async jobs, validation, guardrails

Moves AI quote generation off the request path into a BullMQ job, wraps the model in a single validation/correction pass, and refuses to create a quote when deterministic guardrails fail.

PR dealops#6174 Author yijunz166 Branch yijun/ai-quoting-job-validation-guardrails Files 42 Δ +8225 / −1562 Area Dealops 2 · AI Quoting 🔴 Pricing touched

What it adds

A generateAiQuote BullMQ job, a deterministic guardrail layer, and a single-shot LLM validation/correction pass that returns a corrected quote (not a patch). Suggested minimum commitment now has one canonical formula.

What it changes

The route /api/ai-quoting/... no longer holds open an SSE while Gemini runs. The client starts a job, polls jobs.getStatus, and replays buffered events. Historical-average fallback quotes are gone.

What it preserves

The step-by-step AI Quote Modal UX, tRPC contracts for PricingQuoteInput (extended, not replaced), and existing pricebook/L1/list-price loaders. Renewal opportunities are explicitly excluded from AI quoting.

Request / route

BullMQ job

LLM (Gemini)

Deterministic guardrails

Quote persistence

1. Why this exists

Problem · Latency & cost

The AI quote endpoint held an HTTP request open for the full Gemini run. Closing the modal didn't actually cancel anything — the server kept billing tokens until the SSE reader noticed.

Problem · Trust

The pipeline was happy to ship a "historical-average fallback quote" when generation failed. That meant a PricingQuote could be created without the model ever producing a defensible recommendation.

Problem · Determinism

Pricing rules (L1 floors, list caps, auto-select add-ons, the suggested-minimum formula) were partially encoded in prompts. The model could agree it had violated a rule and still emit a quote that violated it.

2. The new pipeline

Step through the run. Each frame shows which component is doing work and how data flows through it. Use ←/→ or the dots.

Step 1 of 6

Step 1 · Client starts a job The modal POSTs to /api/ai-quoting/start/:sfdcId/:v2Id with the pricebook + currency terms. The route enqueues a generateAiQuote BullMQ job and returns { jobName, jobId } immediately — no Gemini call on the request path.

3. Before vs after — the request path

Before

The client opened an SSE to /stream/:id. The route ran Stages 2–6 inline. The fetch stayed open for the entire model run (often 60–120s).

GET /api/ai-quoting/stream/:sfdcId/:v2Id
  ┌────────────────────────────────────┐
  │ orchestratorStream runs Gemini     │
  │ → SSE chunks: step:start, ...      │
  │ → SSE chunks: complete             │
  └────────────────────────────────────┘
client closes tab → server keeps going

Stage 4 could fail and the orchestrator would emit ai:fallback_used with a historical-average quote. Banner copy lived in AIQuoteBanner.tsx.

After

The client POSTs to /start/:id, gets a job id back, then polls trpc.jobs.getStatus every 1s. Events are buffered on the job's progress field and replayed in order.

POST /api/ai-quoting/start/:sfdcId/:v2Id
  → { jobName, jobId }
BullMQ worker picks up job:
  emit step events → progress { sequence, events[] }
client polls jobs.getStatus.query(...)
  → drains events newer than lastSequence

No fallback quote. If generation fails, the job fails and the modal shows a structured "no quote created" reason.

4. The validation / correction loop

Stage 4 used to emit one recommendation and call it done. Now it goes through a one-shot validate-and-correct pass, then deterministic guardrails — and deterministic guardrails are the source of truth.

1 · Generate

generateAiQuote.ts calls Gemini with the quote_generation prompt + tool calls. Detects repeated identical tool-call loops and fails fast instead of spinning until a vague timeout.

2 · Validate + Correct (LLM)

A second Gemini call (validateAiRecommendationWithLLM.ts + quote_validate_correct.v1.md) returns a complete corrected JSON, not a patch. LLM-only hard issues are downgraded to guidance.

3 · Deterministic guardrails

validateAiRecommendationAgainstRules.ts applies L1 floors, list caps, auto-select / auto-deselect rules, and the minimum-commitment formula. If hard issues remain, no quote is created.

Key inversion. LLM-only validation complaints are advisory. Deterministic guardrails are the final source of truth before quote creation. This is encoded in generateValidatedAiRecommendation.ts — if the LLM says "PRICE_BELOW_L1" but the deterministic check passes, it becomes a guidance note.

5. Pricing rules made deterministic

5a. Suggested minimum commitment formula

flatMinimum = max(0, usageMonthlyRevenue × 0.6 − supportMonthlyFee)
if flatMinimum == 0 → $0
else → roundUpTo500(max(flatMinimum, 500))

Scenario	Raw calc	Final
Usage $3,372.50 / mo · Support $1,000 / mo	`max(0, 3372.5 × 0.6 − 1000) = 1023.50`	$1,500
Usage $1,300 / mo · Support $1,000 / mo	`max(0, 780 − 1000) = 0`	$0
Usage $1,300 / mo · Support $500 / mo	`max(0, 780 − 500) = 280`	$500 (floor)
Support ramp: 6mo @ $0, then $1,000	Apply per-month → ramp `[1000×6, 0×6]`	monthlyMinimumRamp

5b. L1 / list-price floors

Usage products

Flat price must be ≥ L1 when L1 exists
Each tier price must be ≥ L1
Non-zero quotePriceRamp months must be ≥ L1
Zero ramp months are waivers — not raised to L1
All prices capped at active pricebook list price

Support / platform fixed-fee SKUs

Preserve explicit support fee evidence
Capped at list price when list exists
Not raised to usage L1 floors
Detected via isSupportOrPlatformProduct() in l1PricingBasis.ts

5c. L1 lookup basis

L1 is tier-keyed by effective monthly minimum, not just usage revenue. That changes which tier is selected for the deal:

effectiveMonthlyMinimum = quoteMinimumCommitment + monthlySupportContribution
// if either is ramped, use the average

6. Auto-select & auto-deselect product rules

Pulled out of prompt-land into Stage 5's mapToV2QuotePayload in generateV2Quote.ts. The catalog now exposes autoSelectProductIds and autoDeselectProductIds per ProductSpec.

Auto-select

If product X declares autoSelectProductIds: ['Y'], Y is added even if the model didn't emit it. The mapper deduplicates and records reasoning in appliedRuleReasoning.

Auto-deselect (mutual exclusion)

If product X declares autoDeselectProductIds: ['Z'], then Z is dropped from the quote when X is selected. Conflicting SKUs cannot both ship.

Plaid core products

If the customer asks for "Plaid core", the available core set is included unless evidence narrows it. The "do not add unmentioned products" rule does not block required auto-selected add-ons.

7. The failure UX

Before

Errors surfaced raw model/tool internals. The fallback path silently created a quote based on historical averages with an amber banner.

"Gemini API quota exceeded for project ..."
"Tool call malloc failed in iteration 7"
→ Quote created anyway, fallback banner shown

After

Two distinct terminal states, both clean. Provider/tool details never reach the user.

error — pipeline crashed. Modal shows "Quote generation failed" + a structured Evidence/Benchmark/Strategy block when present.
no_quote — pipeline completed but guardrails refused. Shows "No quote created" with confidence pill + failed-rule list.

The client hook (useAIQuoteStream.ts) gained an isNoQuote branch alongside isCompleted / isError. The modal renders a new NoQuoteBanner that parses the reason into labeled sections.

8. Surfaced reasoning in the quote UI

When the quote does get created, the AI's reasoning is no longer trapped in a comment field. The pricing UI now reads aiMetadata from PricingQuoteInput:

Minimum commitment tooltip

minimumCommitmentTable.tsx renders an AIReasoningTooltip type="commitment" next to the minimum cell, showing the formula derivation.

Inspection drawer

InspectionDrawer.tsx shows applied-rule reasoning, validation-loop attempt count, and pass/guardrailed badge.

In-progress phase rail

The modal's AIRecommendationPhaseRail walks through 9 activity labels — query catalog, resolve IDs, validate, fix violations, pass guardrails — driven by server-emitted phase data.

9. Server-side topology

File	Status	Role
`jobs/definitions/generateAiQuote.ts`	new	BullMQ job definition; calls the orchestrator and writes events to job progress
`pipeline/finalizeAiQuote.ts`	new	Stage 5/6 wrapper — filters products, builds `PricingQuoteInput`, creates `PricingQuote` (or returns `no_quote`)
`pipeline/generateValidatedAiRecommendation.ts`	new	Orchestrates generate → validate+correct → guardrails; never falls back
`pipeline/validateAiRecommendationWithLLM.ts`	new	Calls Gemini with quote_validate_correct.v1 prompt; returns full corrected JSON
`pipeline/validateAiRecommendationAgainstRules.ts`	new	Deterministic guardrails: L1 floors, list caps, auto-select/deselect, commitment formula
`pipeline/l1PricingBasis.ts`	new	`effectiveMonthlyMinimum` calc + `isSupportOrPlatformProduct` classifier
`pipeline/generateAiQuote.ts`	+720/−238	Loop-detection, tool-batching, `Stage4GenerationError` typed errors (no fallback)
`pipeline/generateV2Quote.ts`	+601/−16	Auto-select/deselect mapping, ramped flat-fee support, L1 floor raising
`routes/aiQuoting.ts`	+120/−813	Big shrink — route just enqueues the job, no more inline SSE

10. What it doesn't change

The jobs.getStatus tRPC contract — this PR reuses the existing progress mechanism that powers useExportPricingQuote
The pricebook loaders (loadProductCatalog, loadListPriceMap); only adds loadL1PriceMap
The downstream PricingQuote model — aiMetadata is an additive field on PricingQuoteInput
Manual quote creation paths (the CreateQuoteButton menu still offers both, AI option just hides for renewals)
The product-dropped Stage-5 notice — that path still fires when AI succeeds but a SKU fails catalog/name validation

11. Risks & rollback

Behavioral risk. Removing the historical-average fallback means strict regressions in the model can now visibly produce zero quotes. The "no_quote" UX is the intended landing pad, but volume of those events should be watched in the first week.

Pricing risk. The deterministic minimum-commitment formula is opinionated (0.6 × usage − support, round up to $500, floor at $500). This will produce different numbers than free-form LLM commitments did. Worth reviewing the focused Mocha tests in validateAiRecommendationAgainstRules.test.ts before merge.

Rollback. The job is feature-isolated. Reverting the route + client changes restores SSE; the new orchestrator/validator files are unused if the route doesn't enqueue. No DB migrations.

12. Open questions

Should the validation/correction LLM call be cached on the recommendation hash? Currently fires every run.
The auto-select rule attribution in appliedRuleReasoning currently says "for the active pricebook/currency" — is that the right phrasing for the customer-facing reasoning surface, or should it stay internal?
Renewal opportunities are excluded from AI quoting at the button level. Is there a Stage-2 server-side guard too, or do we rely on the UI?

Diff was truncated in the input — anything after the validation/correction module isn't directly inspectable here. The summary above reflects the PR description plus the visible portion of the diff.