Langchain Test Org → catalog_product migration

A bespoke flat-price converter, an org-wide V3 versioning seed helper, and two shared schema fixes that keep renewal-seed config and custom end-date overrides from silently disappearing across the V2↔V3 round trip.

PR dealops#6124 Author @pk675 Branch pk/langchain-catalog-migration Files 20 Diff +1808 / −116 Area Dealops 3 · catalog_product Status Open

What it adds

A per-org converter, migration runner, and V3 versioning-store seeder for Langchain Test Org — registered in the publish-path registry so the real publish seam round-trips this org 1-1.

What it fixes (all orgs)

Three top-level productSpec fields were being silently dropped on V2↔V3 round trip: customEndDateOverrideInWeeks and the three renewal-seed fields. Now structural, carried via systemAttributes.

What it generalizes

Pulls the throwaway "seed attributes" / "seed flow" scripts into one shared _utils/versioningSeed.ts + a shared env/org safety guard. Xendit's initV3.ts is rewritten on top of it.

What it preserves

97/97 products (including the priceless internal commitment) and 96/96 datapoints round-trip identically through the registry seam. --write / publish / flow-type flip remain separate manual steps.

Langchain converter
Shared converter fix
Versioning-store seed
Safety / org pinning
Docs / skill

1. Why this exists

Dealops is migrating each customer org off the legacy V2 pricingSpec + dataSheet pair onto the unified catalog_product data model (Dealops 3). Every org needs its own converter because the way prices and attributes are encoded in V2 differs per org — and the publish seam must reproduce that org's V2 spec 1-1 after a round trip, or it can't be cut over safely.

The roundtrip-catalog-migration skill orchestrates this. This PR is the Langchain Test Org output of that skill, plus three corrections that fell out of running it: two silently-dropped V2 fields, and a refactor that lifts the previously-throwaway "seed the V3 versioning store" step into shared, reusable code.

The Langchain shape. Langchain is a flat org: every product is its own V2 productSpec, identified by a tag combination (deploymentType × supportLevel × segment × tier) and priced by exactly one LIST_PRICE currency datapoint. The default Dealops sku == datapoint planner would (a) rename productIds from the datapoint and (b) drop the internal priceless commitment product entirely. Both are unacceptable for a 1-1 round trip, hence a bespoke converter.

2. What changes — file map

New: Langchain per-org migration
newmigration/orgs/langchain-test/converter.ts+230
newmigration/orgs/langchain-test/migrate.ts+214
newmigration/orgs/langchain-test/verify.ts+251
newmigration/orgs/langchain-test/initV3.ts+176
newmigration/orgs/langchain-test/safety.ts+22
newmigration/orgs/langchain-test/README.md+100
New: shared (org-agnostic) helpers
newmigration/_utils/safety.ts+130
newmigration/_utils/versioningSeed.ts+332
modmigration/_utils/registry.ts+8
Modified: shared converter fixes (touch every org)
modcatalogProduct/systemKeys.ts+20 / −11
modcatalogProduct/mapping.ts+2 / −1
modpricingSpec/types.ts+5
modpricingSpec/converter.ts+11
modpricingSpec/v2Converter.ts+10
mod__tests__/catalogProductMapping.test.ts+15 / −7
Modified: Xendit refactored onto shared helpers
modmigration/orgs/xendit-sandbox/initV3.ts+86 / −88
modmigration/orgs/xendit-sandbox/README.md+17 / −3
mod.claude/skills/roundtrip-catalog-migration/SKILL.md+169 / −5
modapps/server/.gitignore+6 / −1

3. The Langchain converter, in one picture

V2 input (per product)
  • productId — opaque UUID-ish
  • tags: deploymentType × supportLevel × segment × tier
  • categoryId (e.g. langsmithDeployment)
  • 0 or 1 LIST_PRICE currency datapoint
  • Optional includeInRenewalSeed, customEndDateOverrideInWeeks
catalog_product row (per SKU)
  • skuId = the V2 productId, verbatim
  • baseProductId = the productLine (= V2 categoryId)
  • baseProductName = that category's display name
  • customAttributes: the 4 tag dimensions
  • systemAttributes: renewal-seed + end-date fields
  • price.flat_price = the one LIST_PRICE (or {} for commitment)

Base product = the category

The baseProductId is the product's productLine attribute, which is literally V2's categoryId (e.g. langsmithDeployment, agentBuilder, langsmithOE, platformSubscription, professionalServices). baseProductName is that category's display name (from productCategories[].name). The category-less internal commitment product falls back to its own name.

This is catalog-UI grouping only — the reverse converter never reads it, so it cannot break the 1-1 round trip.

The internal commitment product

Langchain's V2 spec contains one priceless internal product used by the engine for minimum-commit calculations. The default sku==datapoint planner drops it (no datapoint → no SKU). The bespoke converter keeps it: price = {}, isActive: false, isNotSelectable: true, no pricebook, no category. It round-trips into V2 unchanged. adminOnly stays false deliberately — flipping it would change the V2 productSpec and break 1-1.

4. The two silent-drop bugs this PR fixes

Both bugs affected every org, not just Langchain. They surfaced because Langchain happens to use these fields in production data, and a 1-1 round trip exposed the loss.

Before · customEndDateOverrideInWeeks

Lived in OUT_OF_SCOPE_FIELD_KEYS alongside startDate/endDate. productSpecToRow dropped it. A reconstructed spec lost the value silently — a product configured with a 4-week override would come back with none.

OUT_OF_SCOPE_FIELD_KEYS = [
  'startDate',
  'endDate',
  'customEndDateOverrideInWeeks',  // dropped
];
After · structural field

Moved into STRUCTURAL_FIELD_KEYS. The V3 schema already has a top-level customEndDateOverrideInWeeks: z.number().optional(), so it now rides in systemAttributes and round-trips top-level.

STRUCTURAL_FIELD_KEYS = [
  ...,
  'customEndDateOverrideInWeeks',
  'includeInRenewalSeed',
  'renewalSeedProductSpecIds',
  'renewalSeedProductSpecId',
];

Renewal-seed fields — never had a V3 home

The renewal-substitution config (23 Langchain SKUs use it) consists of three top-level V2 productSpec fields the engine reads to determine what a product renews into. The V2↔V3 converter pair had no mapping for any of them. This PR:

  1. Adds them to v3ProductSpecSchema in pricingSpec/types.ts.
  2. Copies them V2→V3 in pricingSpec/converter.ts (only when set, to avoid stamping undefined).
  3. Copies them V3→V2 in pricingSpec/v2Converter.ts symmetrically.
  4. Lists them in STRUCTURAL_FIELD_KEYS so productSpecToRow stashes them in systemAttributes.
// pricingSpec/converter.ts — convertProductSpec
...(v2.includeInRenewalSeed !== undefined && {
  includeInRenewalSeed: v2.includeInRenewalSeed,
}),
...(v2.renewalSeedProductSpecIds?.length && {
  renewalSeedProductSpecIds: v2.renewalSeedProductSpecIds,
}),
...(v2.renewalSeedProductSpecId && {
  renewalSeedProductSpecId: v2.renewalSeedProductSpecId,
}),

The matching catalogProductMapping.test.ts assertion is updated so the "out-of-scope drop" test now verifies preservation of customEndDateOverrideInWeeks alongside the still-dropped startDate/endDate.

5. The registry seam — how publish picks this up

The shared _utils/registry.ts is the dispatch table the real publish path (trpc/router/v3Admin/_helpers.ts and publishDrafts.ts) consults. Adding an org is intentionally minimal — one entry, two function references:

'langchain-test-org': {
  pricingSpecToV2: langchainCatalogToV2PricingSpec,
  catalogToV2Datapoints: langchainCatalogToV2Datapoints,
},

The Langchain reverse functions are thin wrappers over the shared converters — the org has no variant families to collapse, and its single flat_priceLIST_PRICE mapping is exactly what convertToV2DataSheet(deriveV3Datasheet(...)) already produces. No bespoke per-org reverse logic, unlike Xendit.

6. The versioning-store seed, lifted into shared code

A migrated org isn't usable on DEALOPS_V3 from catalog_product rows alone. The V3 admin shell, publish, and the Config Agent all read from the versioning store (OrgSpecVersion rows + one OrgGlobalVersion), and all five SpecTypes must exist or those surfaces throw.

Previously this was done by hand with throwaway scripts (seedAttributesFromCatalog.ts, seedAttributesFromPayload.ts, …). This PR moves the work into _utils/versioningSeed.ts and rewrites Xendit's initV3.ts on top of it, so every future org seeds identically.

SpecTypeWhat gets seededSource
pricingSpec The org envelope (currencies, pricebooks, flags) convertPricingSpec(V2). Per-product specs come from catalog_product at publish time.
datasheet Base datasheet body (priced datapoints rebuilt from catalog price at publish) convertDataSheet for Langchain; { datapoints: [] } for Xendit (shared converter throws on Xendit fee types).
attributes Custom-attribute schema Derived from the planned catalog rows — union of customAttributes keys across the migration's own SKUs.
flowSpec Minimal valid body (empty products / calculator / finish options) buildMinimalFlowSpec() — relaxed; authored properly later.
workflowSpec Minimal valid body (no approval edges) buildMinimalWorkflowSpec() — relaxed; authored properly later.

Attribute derivation enforces the reader's rules

deriveCustomAttributeDefs() walks the planned rows and infers each key's valueType (boolean / number / enum ≤ 50 distinct strings / string / multi_enum for array values). It drops anything that would make loadAttributeSchema throw at read time:

Atomic five-spec write

// seedV3VersioningStore — applies in fixed order:
['pricingSpec', 'datasheet', 'attributes', 'flowSpec', 'workflowSpec']
// 1. validate every body against its strict V3 schema
// 2. refuse if org already has a global version (unless --force-reseed)
// 3. saveSpecVersion × 5
// 4. createGlobalVersion pointing at all five

7. Safety guards — env + org pinning

Both migrate.ts and initV3.ts run in dry-run by default. Any DB write goes through two independent guards, lifted into _utils/safety.ts:

Env guard — Render-only for prod

--env production requires both a production DB host and process.env.RENDER set. Pointing it at a local/neon DB, or running off Render, fails fast. Without the flag, the script refuses any non-localhost/non-neon host.

Org guard — name + slug pinned

assertOrgIs(orgId, expectedName, expectedSlug) fetches by id and hard-asserts both fields match exactly. A mistyped UUID can never let a migration act on another org. The per-org safety.ts just supplies the constants.

8. Verification — what the PR proves

Round-trip

97/97 products (incl. internal commitment) and 96/96 datapoints reproduced 1-1 through the registry seam.

Type check

pnpm run typecheck green across the monorepo.

Tests

Full Mocha suite green — 5131 passing. The catalogProductMapping test is updated to assert preservation of customEndDateOverrideInWeeks.

The verify.ts harness diffs through the real publish seam (the registry dispatch), bucketing diffs into acceptable (engine-neutral attributes mirror, isRequired: false → undefined) vs must-resolve. A non-clean run exits non-zero.

9. What this PR does not change

10. Order of operations (when this gets rolled out)

StepCommandEffect
1migrate.ts --org-id=<uuid> --env production --write --user-id=<uuid>Upsert draft catalog rows. Idempotent on (orgId, skuId, state='draft').
2APPLY=1 initV3.ts --org-id=<uuid> --env productionSeed the five OrgSpecVersion rows + one OrgGlobalVersion. Refuses if already seeded (use --force-reseed).
3publish (publishDrafts)Promote drafts → live; the registry seam writes V2 back from the catalog.
4Flip Organization.pricingFlowType = DEALOPS_V3Cutover. Reads start using V3.

11. Risks & open notes

Shared field additions touch every org's V2↔V3 converter. The three renewal-seed fields and customEndDateOverrideInWeeks are now copied verbatim in both directions for all orgs. They're additive and gated on presence (undefined stays absent), but any org that had a stale renewal-seed value in V2 will start round-tripping it through V3 instead of silently dropping it.
Existing test caught the contract change. The "out-of-scope drop" test in catalogProductMapping.test.ts previously asserted customEndDateOverrideInWeeks was dropped; it's now updated to assert preservation. That test is the contract — if a future refactor reverts the move, it fails loudly.
Xendit initV3.ts is rewritten on top of the shared helper. Net diff is +86 / −88, and the seeded shape changes from "pricingSpec + empty datasheet only" to "all five specs (with derived attributes + minimal flow/workflow)". If Xendit's versioning store has already been seeded in some environment with the old two-spec shape, a reseed would advance the global pointer — the guard requires --force-reseed to do this on purpose.