Product2 lookup for the Product Catalog Agent

A local-first CrmProduct search with a queued Salesforce/Ampersand fallback, wired into the config-agent chat through a new SSE event and client-side job polling.

PRdealops#6088 Authoryijunz166 Basemain Files12 Diff+1382 / −15 AreaDealops 2 + 3 / configAgent StatusOpen

What it adds

A reusable product2Lookup service, a BullMQ job that runs Salesforce queries off the request path, and two new agent tools (findProduct2CandidatesForAgent, upsertConfirmedProduct2ForAgent).

What it changes

The Product Catalog Agent now resolves crmProductId via local CrmProduct first, then a queued Salesforce lookup. crmProductId is now nullable on agent writes. The catalog context replaces a 5k-row scan with JSON-field DB filters.

How it streams

A new product2_lookup_job SSE event ends the turn with a jobId. The client polls jobs.getStatus and resumes the same thread with candidates or a failure message — the chat stream stays closed.

Product Catalog Agent
Local CrmProduct (DB)
BullMQ job
Salesforce / Ampersand
Config-agent client

1. Why this exists

The Product Catalog Agent stages catalog products with a crmProductId that should map to the Salesforce Product2.Id for writeback. Before this PR, there was no way for the agent to actually look up a Product2: it would either skip CRM mapping entirely or scan up to 5,000 CrmProduct rows unfiltered, with no path to the live Salesforce object.

Problem 1 — no live lookup

Salesforce/Ampersand queries can take seconds. Doing them inline in /config-agent/chat/stream would either time out the SSE stream or stall the agent's turn.

Problem 2 — wasteful local search

The local CrmProduct search loaded everything for the org and filtered in memory. Fine at 50 rows; not fine at thousands.

Problem 3 — no resume protocol

There was no way for a sub-agent to say "I started a job, come back later" and have the client resume the same conversation with results.

2. End-to-end flow

The agent always tries local first. Local hits resolve immediately. Local misses (with Salesforce configured) queue a job, end the SSE turn, and let the client take over polling. Step through the lifecycle:

3. The new server-side pieces

Service · dealops2

services/crm/product2Lookup.ts

Pure functions used by both the agent context and the BullMQ job:

  • product2LookupCriteria() — trims and normalizes inputs
  • getSalesforceProduct2LookupAvailabilityFromSpec() — checks CRM spec + env vars
  • fetchSalesforceProduct2Candidates() — calls crmObjectService.queryProducts per criterion and merges match reasons

It does not write to CrmProduct — that's reserved for the post-confirmation upsert.

Job · BullMQ

jobs/definitions/fetchSalesforceProduct2Candidates.ts

defineJob({
  name: 'fetchSalesforceProduct2Candidates',
  schema, // org + lookup fields
  defaultJobOptions: {
    attempts: 2,
    backoff: {
      type: 'exponential',
      delay: 5000,
    },
  },
  handler: async (job) =>
    fetchSalesforceProduct2Candidates(job.data),
});

Transient Ampersand errors now throw, so BullMQ retries instead of resolving with a fake "failed" payload. Registered in jobs/registry.ts.

Context · dealops3

configAgent/catalogProductContext.ts

Adds three exported helpers:

  • findLocalProduct2CandidatesForAgent — exact crmId via the organizationId_crmId unique, then JSON-path findMany by productCode and name
  • getSalesforceProduct2LookupAvailabilityForAgent
  • upsertConfirmedProduct2ForAgent — writes only the user-approved candidate
Tools · dealops3

tools/catalogProductTableTools.ts

Two new LangChain tools registered with the agent:

  • findProduct2CandidatesForAgent — local first; on miss, dispatch('fetchSalesforceProduct2Candidates', …) and returns { salesforceLookupQueued: true, jobId, jobName, lookup }
  • upsertConfirmedProduct2ForAgent — explicitly post-approval only

Agent write schema now allows crmProductId: z.string().nullable().optional().

4. The streaming protocol change

The orchestrator inspects each askProductCatalogAgent result for a salesforceLookupQueued marker. If it finds one, it does not continue iterating tools — it emits a structured event and ends the turn.

Before — implicit

A sub-agent that needed slow work had no way to hand off. The orchestrator would either block the stream or lose context after timing out.

// Inline Salesforce call here =
//   stream hangs for ~5–30s,
//   or the model's turn drifts.
After — explicit SSE event
// types.ts — new SSE variant
| {
  type: 'product2_lookup_job';
  jobName: 'fetchSalesforceProduct2Candidates';
  jobId: string;
  conversationId: string;
  lookup: {
    product2Id?: string | null;
    skuId?: string | null;
    productCode?: string | null;
    name?: string | null;
  };
}

The orchestrator emits a short user-facing token ("I started a Salesforce Product2 lookup…"), yields the structured event, then yields { type: 'done' }. The SSE stream closes.

Why subAgentSummarizer.ts was touched

The summarizer strips most fields from sub-agent JSON before passing it back up. Without explicitly whitelisting them, the orchestrator would never see the queue marker.

// subAgentSummarizer.ts
const KEPT_FIELDS = [
  // …
  'salesforceLookupQueued',
  'jobName',
  'jobId',
  'lookup',
] as const;

5. The client polling-resume loop

useConfigAgent.ts grows a polling helper and a resume queue. The big idea: do not keep the chat stream open while waiting on a slow job. Instead, poll trpc.jobs.getStatus, then send a new message back into the same thread.

Poll
pollProduct2LookupJob(jobId, signal)
  // every 2_000 ms
  // up to 120_000 ms
  // active statuses:
  //   waiting | active |
  //   delayed | waiting-children
Resume — success

Builds a markdown message containing the original lookup and the candidates payload, then calls enqueueProduct2Resume(threadId, message). The agent confirms the mapping with the user.

Resume — failure

Reports the failure reason to the agent and explicitly invites it to continue staging with crmProductId=null. The user is never silently dropped.

Queue, version, and busy-thread guards

Three new refs back the resume machinery, all keyed by thread id:

RefPurpose
pendingProduct2ResumeRefsFIFO of resume messages waiting to be sent on a thread that is currently streaming.
product2ResumeVersionRefsMonotonic counter per thread. Incremented on stop/reset. Stale poll completions check their captured version and silently drop.
streamingThreadsRefMirror of streamingThreads state so synchronous code (resume flush, migrate, finally block) can read the latest value without waiting for a render.

The finally block of sendMessage now only clears streaming state if the abort controller it owns is still the current one for the thread — preventing a slow finalize from clobbering a fresh send. After clearing, it calls flushProduct2ResumeQueue(threadId) to drain any queued resume messages.

Thread-id migration edge case

An UNASSIGNED thread becomes a real conversation id mid-stream. migrateThreadKey now also moves any queued resume messages and the resume version counter across, so a job that completes after the rename still posts to the right conversation.

6. The local-search performance fix

Before
// effectively:
const all = await prisma.crmProduct
  .findMany({
    where: { organizationId: orgId },
    take: 5000,
  });
// then in-memory string match
// on productCode / name

Worked on small orgs. Pulled megabytes on large ones.

After
prisma.crmProduct.findMany({
  where: {
    organizationId: orgId,
    crmProductData: {
      path: ['productCode'],
      equals: 'IDV',
    },
  },
  orderBy: { updatedAt: 'desc' },
  take: limit + 1,
});

Push-down filter on the JSON column, bounded result set, sorted for recency.

7. Agent prompt and tool-rule changes

productCatalogAgent.ts — new workflow rules

The orchestrator system prompt gains a matching SALESFORCE PRODUCT2 MAPPING section: include the chosen mapping in the user-facing action_plan, require one approval, and let crmProductId=null through with a warning rather than blocking.

8. What it doesn't change

9. Risks & open questions

Polling timeout. The client gives up after 120s. A genuinely slow Ampersand response (cold connector, large org) will surface as "Salesforce Product2 lookup timed out" and the agent will offer to proceed unmapped. That's the intended graceful degradation, but worth confirming the timeout is generous enough for the slowest real customer.
Result parsing is string-based. The orchestrator detects queued jobs by searching the agent's text output for salesforceLookupQueued and then JSON-parsing either the whole reply or a regex-extracted object. If the model wraps the JSON oddly, the queue marker can be missed and the client will never get the SSE event. The whitelisting in subAgentSummarizer mitigates this but doesn't fully eliminate it.
Rollback. Pure additive surface: new files, new SSE event type, new tools, one schema relaxation (crmProductId nullable on agent input). Reverting the PR removes the tools and the event; existing catalog flows continue to work because the agent's instructions only call the new tools when Salesforce writeback mapping is needed.
Open question — concurrent lookups. If the agent queues multiple Product2 lookups in a single turn (bulk import), the orchestrator yields one SSE event per queued job and ends the turn. The client enqueues each resume message, but they all post to the same conversation sequentially. Verify the agent handles a string of "[Salesforce Product2 lookup completed]" messages without confusing itself.