HubSpot product sync now matches by SKU before create

This Dealops 2 PR makes CRM product sync inspect the full HubSpot product catalog, reuse that list in preview and sync, and enforce the shared 500-product batch cap in both client and server paths.

PR #6597 Ticket DEA-7238 Author @dezbah-duchicela State open Base main Head dezbah/hubspot-product-sync-sku-match Files 15 Diff +850 / -90

What it fixes

HubSpot product sync no longer treats “not found in a narrow search” as proof that a product is missing.

It lists all HubSpot products once, then matches by DealOps SKU or stored CRM id before deciding to create.

What it adds

A paginated HubSpotConnector.listAllObjects path follows HubSpot after cursors and refuses partial lists.

The CRM object service exposes listAllProductsRaw for product sync callers.

What it hardens

AmpersandPassthroughAPI now retries retryable 429 responses using proxy guidance headers.

This covers both reads and writes, including HubSpot product creates.

What users see

The V3 product catalog caps selection at the shared 500 sync limit, shows a tooltip, and avoids raw validation errors after selecting too many SKUs.

HubSpot connector/listing

ProductSyncService

Ampersand passthrough

V3 admin catalog UI

Shared types limit

Dealops 2CRM sync surface

500Shared batch cap

100HubSpot page ceiling

3429 retry budget

1. Why this exists

Problem

HubSpot product sync could create duplicate products when matching depended on targeted searches instead of a complete catalog view.

A missing search hit looked like “product does not exist.”
The sync would then call create instead of update.
HubSpot search rate limits made per-product lookup fragile for batches.

Fix

List HubSpot products once up front with the fields needed for matching, then perform in-memory matching for preview and sync.

Match by dealops_sku first.
Also match by stored crmProductId.
Only create when the full product list has no match.

System boundary: this is a Dealops 2 CRM product sync change. The server code lives under apps/server/src/dealops2/services/crm; the user entry point is the V3 admin product catalog.

2. What changes

Server-side product matching path

Select SKUs

Preview or sync job receives up to CRM_PRODUCT_SYNC_MAX_PRODUCTS.

List HubSpot products

listAllProductsRaw([...HUBSPOT_PRODUCT_FIELDS]) follows pages once.

Match in memory

findHubSpotProductByPreferredKeys checks SKU and CRM id.

Update or create

Existing records are updated; only unmatched records are created.

Retry 429s

Transport retries retryable rate limits before surfacing a failed CRM response.

Before

HubSpot preview queried per selected product.

includeProductIds.map(async (productId) => {
  const { product } = await findHubSpotProductForProduct(
    productSpec,
    crmService,
  );
});

Cost shape: N selected products → N HubSpot search calls.

After

HubSpot preview and sync share a catalog snapshot.

const existingProducts =
  await queryAllHubSpotProductRecords(crmService);

const { product, matchedBy } =
  findHubSpotProductForProductInRecords(
    productSpec,
    existingProducts,
  );

Cost shape: 1 paginated list → N in-memory matches.

Meaningful files by area

Area	Files	Change
HubSpot connector	`hubspotConnector.ts`, `hubspotConnector.test.ts`	Adds `after` cursor support to `listObjects` and introduces bounded `listAllObjects`.
Product sync	`PushProductsToSalesforce.ts`, HubSpot service tests, job tests	Loads HubSpot products once, matches by preferred keys, updates existing products before create, and keeps id-less create fallback targeted.
Ampersand	`ampersand_passthrough.ts`, `ampersand_passthrough.test.ts`	Retries retryable 429 responses using `X-Amp-Retry-After`, `X-Amp-Retryable`, provider `Retry-After`, or jittered backoff.
Admin UI	`V3ProductCatalogContainer.tsx`, `useV3ProductCatalogTable.tsx`	Caps selection at 500, shows an info toast/tooltip, and makes the header checkbox settle when only the capped set is selectable.
Shared limit	`packages/types/v2/crmProductSync.ts`, job schemas, tRPC schema	Exports `CRM_PRODUCT_SYNC_MAX_PRODUCTS = 500` and uses it consistently across client, jobs, and tRPC validation.

Client selection behavior

Before

Users could select more SKUs than the server accepted.
“Select all” on a large catalog hit the hidden cap only after mutation validation.
The table checkbox could stay indeterminate because selected count never equaled every visible SKU.

After

Single-row and bulk selection stop at 500.
Attempts beyond the cap show: “Sync this batch, then select the rest.”
Button label becomes Sync 500 / 500 to CRM at the ceiling.

3. How it works

HubSpot pagination Connector layer

listObjects now accepts an optional after cursor and URL-encodes it into /crm/v3/objects/products?...&after=....

listAllObjects loops until HubSpot stops returning paging.next.after.

default limit: 100 max pages: 100 throws on partial list

CRM service wrapper Abstraction layer

crmObjectService.listAllObjectsRaw delegates to HubSpot pagination when the connector is HubSpot.

For non-HubSpot connectors, it falls back to queryObjectsRaw(objectType, undefined, fields).

Preview + sync reuse ProductSyncService

queryAllHubSpotProductRecords fetches HUBSPOT_PRODUCT_FIELDS once for the run.

findHubSpotProductForProductInRecords calls findHubSpotProductByPreferredKeys with:

{
  dealopsSku: productSpec.id,
  crmProductId: normalizeNullableString(productSpec.crmProductId),
}

Create fallback stays targeted Id-less HubSpot create response

If HubSpot create succeeds but returns no id, the sync still performs a single targeted lookup by DealOps SKU.

That avoids putting a full paginated list inside the per-product create loop.

429 retry handling Transport layer

AmpersandPassthroughAPI.request now treats a non-2xx 429 response as retryable work, not an exception path.

The guidance resolver understands that X-Amp-Retry-After is a UTC timestamp, not a seconds count.

retryable writes included max wait: 10s max retries: 3 stop on x-amp-retryable:false

Test coverage added or updated

ProductSyncService.hubspot.test.ts: existing HubSpot products match by DealOps SKU before create; id-less create fallback does not re-list the whole catalog.
hubspotConnector.test.ts: cursor paging preserves requested properties and throws after the page ceiling.
salesforceProductSync.test.ts: preview/sync jobs now expect listAllProductsRaw([...HUBSPOT_PRODUCT_FIELDS]).
ampersand_passthrough.test.ts: timestamp parsing, header casing, retry budget, non-retryable lockout, writes, and non-429 behavior.

pnpm exec mocha --no-config --node-option import=tsx \
  --file src/__test-setup.ts --timeout 10000 --exit \
  src/dealops2/services/crm/__tests__/ProductSyncService.hubspot.test.ts \
  src/dealops2/services/crm/connectors/__tests__/hubspotConnector.test.ts \
  src/jobs/__tests__/salesforceProductSync.test.ts

4. What it doesn't change

Explicit non-goals

Does not add new database tables, Prisma migrations, or stored CRM product fields.
Does not change the core Salesforce product matching path; the broad-list behavior is for HubSpot.
Does not increase the product sync batch size; it centralizes the existing 500 ceiling.
Does not introduce a new REST or tRPC route; existing V3 admin catalog mutations keep their shape.
Does not make HubSpot product sync fully atomic. A batch can still partially write before a later product fails.
Does not turn rate-limit handling into long-running throttling; retries absorb short bursts only.

5. Risks / rollback / open questions

Risk: large HubSpot catalogs

Listing all products is intentionally bounded at 100 pages, or roughly 10k products at the default page size.

If a customer exceeds that, sync fails loudly instead of creating duplicates from a partial view.

Risk: sync duration

HubSpot sync now performs one full listing before writes and processes products sequentially.

That reduces search-rate pressure but may increase elapsed time for very large batches.

Rollback

Code rollback restores the prior per-product search behavior.

The CRM product sync feature flag can remove the admin UI entry point while server-side behavior is investigated.

Reviewer focus: verify the HubSpot field set is complete for both matching and update previews, and confirm that making HubSpot sync sequential is the intended tradeoff versus controlled concurrency.