--- title: "Data Importers" description: "Data Importers facilitate the migration of historical link, partner, customer, and commission data from external URL shorteners and affiliate management platforms into Dub. By providing automated i..." last_updated: "2026-10-12T07:00:10.142659+00:00" canonical_url: "https://www.doc0.dev/docs/934e554a-e6a1-476f-bb2f-23e62d86c3fd/technical/integrations-subsystem/data-importers" ---
Relevant source files The following files were used as context for generating this wiki page: - [apps/web/scripts/dev/data.json](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/scripts/dev/data.json) - [apps/web/app/ee/api/cron/import/bitly/route.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/bitly/route.ts) - [apps/web/app/api/workspaces/idOrSlug/import/rebrandly/route.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/workspaces/%5BidOrSlug%5D/import/rebrandly/route.ts) - [apps/web/lib/rewardful/importer.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/importer.ts) - [apps/web/app/ee/api/cron/import/rebrandly/route.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/rebrandly/route.ts) - [apps/web/lib/rewardful/import-commissions.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-commissions.ts) - [apps/web/app/ee/api/cron/import/rewardful/route.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/rewardful/route.ts) - [apps/web/app/ee/api/cron/import/tolt/route.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/tolt/route.ts) - [apps/web/app/ee/api/cron/import/firstpromoter/route.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/firstpromoter/route.ts) - [apps/web/app/api/workspaces/idOrSlug/import/short/route.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/workspaces/%5BidOrSlug%5D/import/short/route.ts) - [apps/web/app/ee/api/cron/import/short/route.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/short/route.ts) - [apps/web/app/api/workspaces/idOrSlug/import/bitly/route.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/workspaces/%5BidOrSlug%5D/import/bitly/route.ts) - [apps/web/lib/tapfiliate/importer.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tapfiliate/importer.ts) - [apps/web/lib/tolt/import-commissions.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tolt/import-commissions.ts) - [apps/web/lib/middleware/link.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/middleware/link.ts) - [apps/web/app/api/callback/bitly/route.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/callback/bitly/route.ts) - [apps/web/app/ee/api/cron/import/lemonsqueezy/route.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/lemonsqueezy/route.ts) - [apps/web/lib/rewardful/import-campaigns.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-campaigns.ts) - [apps/web/ui/guides/integrations.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/ui/guides/integrations.ts) - [apps/web/lib/rewardful/import-partners.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-partners.ts) - [apps/web/lib/firstpromoter/import-commissions.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/firstpromoter/import-commissions.ts) - [apps/web/lib/dub.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/dub.ts) - [apps/web/lib/firstpromoter/import-partners.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/firstpromoter/import-partners.ts) - [apps/web/lib/partnerstack/importer.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/partnerstack/importer.ts) - [packages/utils/src/constants/dub-domains.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/packages/utils/src/constants/dub-domains.ts) - [apps/web/lib/middleware/utils/crawl-bitly.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/middleware/utils/crawl-bitly.ts) - [apps/web/lib/rewardful/import-customers.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-customers.ts) - [apps/web/lib/tolt/import-customers.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tolt/import-customers.ts) - [apps/web/scripts/customers/beehiiv/fix-case-a-complex.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/scripts/customers/beehiiv/fix-case-a-complex.ts) - [apps/web/scripts/customers/beehiiv/fix-case-a-simple.ts](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/scripts/customers/beehiiv/fix-case-a-simple.ts)
## Overview Data Importers facilitate the migration of historical link, partner, customer, and commission data from external URL shorteners and affiliate management platforms into Dub. By providing automated ingestion pathways, the system bridges legacy external infrastructures with Dub's workspace environment and [partner programs](https://www.doc0.dev/docs/934e554a-e6a1-476f-bb2f-23e62d86c3fd/technical/partner-programs/partner-programs). Sources: [apps/web/app/api/workspaces/idOrSlug/import/bitly/route.ts:60-140](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/workspaces/%5BidOrSlug%5D/import/bitly/route.ts#L60-L140), [apps/web/app/ee/api/cron/import/rewardful/route.ts:13-37](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/rewardful/route.ts#L13-L37) ## Workspace link import endpoints ### Provider Route Architecture Dub provides authenticated workspace-scoped HTTP routes to connect, inspect, and initiate link migrations from external URL shorteners including Bitly, Rebrandly, and Short.io. Each provider integration exposes standard HTTP verbs (`GET`, `PUT`, `POST`) guarded by workspace authorization checks and specific permission requirements such as `links.write` and `folders.links.write`. Sources: [apps/web/app/api/workspaces/idOrSlug/import/bitly/route.ts:13-143](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/workspaces/%5BidOrSlug%5D/import/bitly/route.ts#L13-L143), [apps/web/app/api/workspaces/idOrSlug/import/rebrandly/route.ts:13-165](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/workspaces/%5BidOrSlug%5D/import/rebrandly/route.ts#L13-L165), [apps/web/app/api/workspaces/idOrSlug/import/short/route.ts:13-147](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/workspaces/%5BidOrSlug%5D/import/short/route.ts#L13-L147) The migration endpoints perform token validation against secure Redis storage, query external provider APIs to fetch domains, groups, and link counts, and provision missing custom domains into both Prisma and Vercel prior to dispatching background ingestion jobs via QStash. | Provider | Access Token Redis Key | Base API Endpoint | Required Permission | | :--- | :--- | :--- | :--- | | Bitly | `import:bitly:{workspaceId}` | `https://api-ssl.bitly.com/v4` | `links.write` | | Rebrandly | `import:rebrandly:{workspaceId}` | `https://api.rebrandly.com/v1` | `links.write` | | Short.io | `import:short:{workspaceId}` | `https://api.short.io` | `links.write` | Sources: [apps/web/app/api/workspaces/idOrSlug/import/bitly/route.ts:15-22](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/workspaces/%5BidOrSlug%5D/import/bitly/route.ts#L15-L22), [apps/web/app/api/workspaces/idOrSlug/import/rebrandly/route.ts:15-23](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/workspaces/%5BidOrSlug%5D/import/rebrandly/route.ts#L15-L23), [apps/web/app/api/workspaces/idOrSlug/import/short/route.ts:15-23](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/workspaces/%5BidOrSlug%5D/import/short/route.ts#L15-L23) > [!NOTE] > During domain selection in `POST` handlers, if a selected import domain does not yet exist in the workspace, the route automatically registers it with Prisma (`primary: false`), provisions it on Vercel via `addDomainToVercel`, and generates a root placeholder link (`_root`) before publishing background cron tasks. Sources: [apps/web/app/api/workspaces/idOrSlug/import/bitly/route.ts:72-101](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/workspaces/%5BidOrSlug%5D/import/bitly/route.ts#L72-L101), [apps/web/app/api/workspaces/idOrSlug/import/rebrandly/route.ts:116-142](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/workspaces/%5BidOrSlug%5D/import/rebrandly/route.ts#L116-L142), [apps/web/app/api/workspaces/idOrSlug/import/short/route.ts:99-125](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/workspaces/%5BidOrSlug%5D/import/short/route.ts#L99-L125) ## On-demand Bitly edge crawling ### Edge Crawl Execution Flow When an incoming request cannot be resolved through standard database lookups during edge redirect dispatch, the `crawlBitly` utility intercepts requests to evaluate whether the requested key matches Bitly shortlink structures. Keys containing unsupported characters specified by Bitly's character regex are immediately rejected from being crawled. Sources: [apps/web/lib/middleware/utils/crawl-bitly.ts:20-27](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/middleware/utils/crawl-bitly.ts#L20-L27) ```typescript const invalidBitlyKeyRegex = /[`~,.<>;':"/\\[\]^{}()=+!*@&$£?%#|]/; ``` Sources: [apps/web/lib/middleware/utils/crawl-bitly.ts:25-25](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/middleware/utils/crawl-bitly.ts#L25-L25) For valid keys, the function queries the Bitly API endpoint using the configured `BUFFER_BITLY_API_KEY`. If the remote bitlink exists and returns a validated long URL via `getUrlFromStringIfValid`, the system provisions the link directly into the database as an on-demand creation assigned to dedicated buffer workspace identifiers. | Constant Name | Value / Identifier | Description | | :--- | :--- | :--- | | `BUFFER_WORKSPACE_ID` | `cm05wnnpo000711ztj05wwdbu` | Workspace assignment for on-demand crawled links | | `BUFFER_USER_ID` | `cm05wnd49000411ztg2xbup0i` | User ID associated with crawled link creation | | `BUFFER_FOLDER_ID` | `fold_1JNQBVZV8P0NA0YGB11W2HHSQ` | Folder categorization ID for buffer imports | Sources: [apps/web/lib/middleware/utils/crawl-bitly.ts:15-17](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/middleware/utils/crawl-bitly.ts#L15-L17), [apps/web/lib/middleware/utils/crawl-bitly.ts:35-50](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/middleware/utils/crawl-bitly.ts#L35-L50) > [!WARNING] > If the Bitly API encounters a rate limit or returns a non-ok response during `fetchBitlyLink`, the handler logs the rate limit hit and returns a `404` status for the target domain and key combination rather than failing the edge request outright. Sources: [apps/web/lib/middleware/utils/crawl-bitly.ts:100-105](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/middleware/utils/crawl-bitly.ts#L100-L105) ### Background Persistence and Fallback Routing Once a bitlink record is retrieved from the Bitly API and sanitized, `ev.waitUntil` schedules background persistence tasks using Prisma to create the link entry, record analytics via Tinybird through `recordLink`, and publish workspace usage events to Upstash Redis streams via `publishWorkspaceLinksUsageEvent`. Sources: [apps/web/lib/middleware/utils/crawl-bitly.ts:36-61](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/middleware/utils/crawl-bitly.ts#L36-L61) ```typescript .then((data) => Promise.allSettled([ recordLink(data), publishWorkspaceLinksUsageEvent({ workspaceId: BUFFER_WORKSPACE_ID, linksCount: 1, timestamp: new Date().toISOString(), }), ]), ), ``` Sources: [apps/web/lib/middleware/utils/crawl-bitly.ts:51-60](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/middleware/utils/crawl-bitly.ts#L51-L60) If the remote lookup fails or yields no valid link, the middleware issues a fallback redirect to `https://buffer.com` with a `302` status code and attaches static Vercel CDN cache headers with an `s-maxage` of `86400` seconds alongside cache tags generated via `linkCache._createStaticPagesCacheKeys`. Sources: [apps/web/lib/middleware/utils/crawl-bitly.ts:70-81](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/middleware/utils/crawl-bitly.ts#L70-L81) ```typescript return NextResponse.redirect("https://buffer.com", { headers: { ...DUB_HEADERS, "Vercel-CDN-Cache-Control": "public, s-maxage=86400", "Vercel-Cache-Tag": linkCache._createStaticPagesCacheKeys({ domain, key, }), }, status: 302, }); ``` Sources: [apps/web/lib/middleware/utils/crawl-bitly.ts:71-81](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/middleware/utils/crawl-bitly.ts#L71-L81) ## Asynchronous link import workers ### QStash Signature Verification and Cron Routing Background synchronization for external provider migrations relies on POST endpoints configured with `force-dynamic` rendering. Every incoming request body is intercepted, read as text, and validated against QStash cryptographic signatures via `verifyQstashSignature` before JSON parsing occurs. Sources: [apps/web/app/ee/api/cron/import/bitly/route.ts:12-19](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/bitly/route.ts#L12-L19), [apps/web/app/ee/api/cron/import/rebrandly/route.ts:9-16](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/rebrandly/route.ts#L9-L16), [apps/web/app/ee/api/cron/import/short/route.ts:9-16](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/short/route.ts#L9-L16) ```typescript export async function POST(req: Request) { try { const rawBody = await req.text(); await verifyQstashSignature({ req, rawBody }); const body = JSON.parse(rawBody); ``` Sources: [apps/web/app/ee/api/cron/import/bitly/route.ts:14-19](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/bitly/route.ts#L14-L19) Depending on the provider target, requests parse specific parameters from the payload to drive batch processing and handle authentication credentials stored inside Redis keys. | Provider Route | Payload Parameters Extracted | Redis Key Lookup Pattern | | :--- | :--- | :--- | | Bitly | `workspaceId`, `bitlyGroup`, `importTags`, `rateLimited` | `import:bitly:{workspaceId}` | | Rebrandly | `workspaceId`, `importTags`, `createdAfter` | `import:rebrandly:{workspaceId}` | | Short.io | `workspaceId`, `userId`, `domainId`, `domain`, `folderId`, `importTags`, `pageToken`, `count` | `import:short:{workspaceId}` | Sources: [apps/web/app/ee/api/cron/import/bitly/route.ts:19-23](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/bitly/route.ts#L19-L23), [apps/web/app/ee/api/cron/import/rebrandly/route.ts:16-22](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/rebrandly/route.ts#L16-L22), [apps/web/app/ee/api/cron/import/short/route.ts:16-26](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/short/route.ts#L16-L26) > [!CAUTION] > If a Rebrandly API key cannot be retrieved from Redis during background processing, the handler immediately throws a `DubApiError` with a `bad_request` code rather than attempting unauthenticated remote fetches. Sources: [apps/web/app/ee/api/cron/import/rebrandly/route.ts:20-29](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/rebrandly/route.ts#L20-L29) ### Tag Synchronization and Rate Limiting Flow When import routines execute with `importTags` enabled, workers check Redis for previously cached flags (`import:bitly:${workspaceId}:tags` or `import:rebrandly:${workspaceId}:tags`). If absent, tag definitions are fetched from the upstream provider API, inserted into Prisma via `createId` with generated badge colors, and mapped to database identifiers. Sources: [apps/web/app/ee/api/cron/import/bitly/route.ts:35-82](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/bitly/route.ts#L35-L82), [apps/web/app/ee/api/cron/import/rebrandly/route.ts:31-62](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/rebrandly/route.ts#L31-L62) ```typescript await prisma.tag.createMany({ data: tags.map((tag) => ({ id: createId({ prefix: "tag_" }), name: tag, color: randomBadgeColor(), projectId: workspaceId, })), skipDuplicates: true, }); await redis.set(`import:bitly:${workspaceId}:tags`, "true"); } ``` Sources: [apps/web/app/ee/api/cron/import/bitly/route.ts:54-64](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/bitly/route.ts#L54-L64) Bitly background workers also evaluate provider-specific rate limits when the `rateLimited` flag evaluates to true, returning a JSON response indicating `rate_limited` status when thresholds are exceeded. Sources: [apps/web/app/ee/api/cron/import/bitly/route.ts:25-33](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/bitly/route.ts#L25-L33) > [!NOTE] > OAuth callback handoff sequences for Bitly store access tokens directly in Redis under `import:bitly:${workspaceId}` before redirecting users back to their workspace dashboard with query parameters configured for initialization. Sources: [apps/web/app/api/callback/bitly/route.ts:46-53](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/api/callback/bitly/route.ts#L46-L53) ## Affiliate partner ingestion workflow ### Execution Walkthrough and Sequence Affiliate partner migration pipelines ingest programs, campaigns, and active partners from platforms like Rewardful and FirstPromoter via QStash-verified background cron routes. The execution router evaluates the incoming `POST` action payload to dispatch processing across specific handlers for campaigns, partners, affiliate coupons, customers, and commissions. Sources: [apps/web/app/ee/api/cron/import/rewardful/route.ts:13-36](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/rewardful/route.ts#L13-L36) 1. `POST` — The QStash cron endpoint verifies message signatures and parses payload actions, routing execution to `importPartners` for active affiliate ingestion via `apps/web/app/(ee)/api/cron/import/rewardful/route.ts`. Sources: [apps/web/app/ee/api/cron/import/rewardful/route.ts:13-26](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/rewardful/route.ts#L13-L26) 2. `importPartners` — Fetches workspace credentials from Redis, instantiates provider clients, pages through remote affiliate records, filters active partners with existing leads or matching campaign IDs, and delegates creation to `createPartnerAndLinks` via `apps/web/lib/rewardful/import-partners.ts`. Sources: [apps/web/lib/rewardful/import-partners.ts:14-110](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-partners.ts#L14-L110), [apps/web/lib/firstpromoter/import-partners.ts:19-102](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/firstpromoter/import-partners.ts#L19-L102) 3. `createPartnerAndLinks` — Upserts partner database records by email, establishes approved program enrollments with group attributes, invokes application approval handlers, and bulk-creates referral links via `apps/web/lib/rewardful/import-partners.ts`. Sources: [apps/web/lib/rewardful/import-partners.ts:165-256](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-partners.ts#L165-L256), [apps/web/lib/firstpromoter/import-partners.ts:116-254](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/firstpromoter/import-partners.ts#L116-L254) ```mermaid sequenceDiagram participant Route as POST (route.ts) participant Import as importPartners participant Create as createPartnerAndLinks Route->>Import: Dispatches import-partners action payload Import->>Create: Iterates active affiliates and calls helper Create->>Create: Upserts partner, enrollment, and bulk creates links ``` Sources: [apps/web/app/ee/api/cron/import/rewardful/route.ts:13-26](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/app/(ee)/api/cron/import/rewardful/route.ts#L13-L26), [apps/web/lib/rewardful/import-partners.ts:14-256](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-partners.ts#L14-L256) ### Platform Integration Parameters and Limits | Importer Platform | Maximum Batch Limit | Credential Cache Duration | Default Target Entity | | :--- | :--- | :--- | :--- | | Rewardful | 10 batches (`REWARDFUL_MAX_BATCHES`) | 24 hours (86400 seconds) | `partner` | | FirstPromoter | 5 batches (`MAX_BATCHES`) | 24 hours | `partner` | Sources: [apps/web/lib/rewardful/importer.ts:8-13](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/importer.ts#L8-L13), [apps/web/lib/rewardful/import-partners.ts:45-52](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-partners.ts#L45-L52), [apps/web/lib/firstpromoter/importer.ts:8-10](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/firstpromoter/importer.ts#L8-L10), [apps/web/lib/firstpromoter/import-partners.ts:52-56](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/firstpromoter/import-partners.ts#L52-L56) > [!WARNING] > When FirstPromoter link creation fails during partner imports, the database enrollment remains committed; error catch blocks ensure the enrollment identifier is still pushed to page-level partner search synchronization queues. Sources: [apps/web/lib/firstpromoter/import-partners.ts:240-252](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/firstpromoter/import-partners.ts#L240-L252) ## Customer referral ingestion pipeline ### Ingestion Flow and Synthetic Click Generation Customer migrations from Rewardful and Tolt parse paginated referral payloads, verify existing database records by Stripe customer identifier or external ID, and manufacture synthetic click events for historical attribution. Because historical imports lack live HTTP requests, the pipeline constructs a dummy `Request` object with predefined headers (`user-agent`, `x-forwarded-for`, and Vercel geo headers) to simulate visitor traffic against the target link URL. Sources: [apps/web/lib/rewardful/import-customers.ts:31-75](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-customers.ts#L31-L75), [apps/web/lib/rewardful/import-customers.ts:210-236](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-customers.ts#L210-L236), [apps/web/lib/tolt/import-customers.ts:33-121](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tolt/import-customers.ts#L33-L121), [apps/web/lib/tolt/import-customers.ts:205-231](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tolt/import-customers.ts#L205-L231) ```mermaid sequenceDiagram participant API as Rewardful / Tolt API participant Importer as importCustomers participant Click as recordClick & recordLead participant DB as Prisma DB API->>Importer: Returns customer/referral page Importer->>DB: Checks existing stripeCustomerId / externalId Importer->>Click: Generates dummy Request and records click Click->>DB: Creates Customer and increments link leads count ``` Sources: [apps/web/lib/rewardful/import-customers.ts:36-90](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-customers.ts#L36-L90), [apps/web/lib/rewardful/import-customers.ts:210-296](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-customers.ts#L210-L296), [apps/web/lib/tolt/import-customers.ts:34-136](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tolt/import-customers.ts#L34-L136), [apps/web/lib/tolt/import-customers.ts:205-292](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tolt/import-customers.ts#L205-L292) ### Platform-Specific Ingestion Rules | Importer Platform | Batch Limit Constant | Link & Token Resolution | Customer Deduplication Keys | | :--- | :--- | :--- | :--- | | Rewardful | `REWARDFUL_REFERRALS_MAX_BATCHES` | `referral.link?.token` or `referral.coupon?.token` with `-coupon` suffix | `stripeCustomerId`, workspace `externalId` | | Tolt | `MAX_BATCHES` | Partner email mapped to program enrollment links (`links[0]`) | Workspace `externalId` (`customer_id || email || id`) | Sources: [apps/web/lib/rewardful/import-customers.ts:35](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-customers.ts#L35), [apps/web/lib/rewardful/import-customers.ts:153-169](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-customers.ts#L153-L169), [apps/web/lib/rewardful/import-customers.ts:194-208](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-customers.ts#L194-L208), [apps/web/lib/tolt/import-customers.ts:33](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tolt/import-customers.ts#L33), [apps/web/lib/tolt/import-customers.ts:83-94](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tolt/import-customers.ts#L83-L94), [apps/web/lib/tolt/import-customers.ts:111-120](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tolt/import-customers.ts#L111-L120), [apps/web/lib/tolt/import-customers.ts:195-201](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tolt/import-customers.ts#L195-L201) > [!CAUTION] > Rewardful customer records missing a valid Stripe customer ID starting with `cus_` trigger a `STRIPE_CUSTOMER_NOT_FOUND` import error log and halt processing for that specific referral. Sources: [apps/web/lib/rewardful/import-customers.ts:181-192](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-customers.ts#L181-L192) > [!IMPORTANT] > When processing Rewardful referrals, chunks of 10 items are handled concurrently via `Promise.allSettled`, while Tolt batches include a 2-second `sleep(2000)` delay between pages to respect upstream rate limits. Sources: [apps/web/lib/rewardful/import-customers.ts:77-90](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-customers.ts#L77-L90), [apps/web/lib/tolt/import-customers.ts:122-138](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tolt/import-customers.ts#L122-L138) ## Commission sync and attribution updates Historical commissions are backfilled via asynchronous workers triggered through QStash endpoints in cron routes such as `apps/web/app/(ee)/api/cron/import/firstpromoter/route.ts`, `apps/web/app/(ee)/api/cron/import/rewardful/route.ts`, and `apps/web/app/(ee)/api/cron/import/tolt/route.ts`. During commission processing, sale amounts expressed in foreign currencies are converted to USD utilizing real-time exchange rates fetched from Upstash Redis hashes (`fxRates:usd`). ```typescript let saleAmount = Number(commission.original_sale_amount ?? 0); const saleCurrency = commission.original_sale_currency ?? "usd"; if (saleAmount > 0 && saleCurrency.toUpperCase() !== "USD" && fxRates) { const { amount: convertedAmount } = convertCurrencyWithFxRates({ currency: saleCurrency, amount: saleAmount, fxRates, }); saleAmount = convertedAmount; } ``` Sources: [apps/web/lib/firstpromoter/import-commissions.ts:231-242](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/firstpromoter/import-commissions.ts#L231-L242), [apps/web/lib/rewardful/import-commissions.ts:48](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-commissions.ts#L48), [apps/web/lib/tolt/import-commissions.ts:52](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tolt/import-commissions.ts#L52) > [!CAUTION] > Commissions with units other than cash or those flagged as self-referrals are rejected and logged via Tinybird import error tracking with specific error codes like `SELF_REFERRAL` or `NOT_SUPPORTED_UNIT`. Sources: [apps/web/lib/firstpromoter/import-commissions.ts:181-199](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/firstpromoter/import-commissions.ts#L181-L199) Status mappings vary across upstream affiliate providers to unify commission states within Dub databases: | Provider | Upstream Status Value | Mapped Dub Status | | :--- | :--- | :--- | | Rewardful | `pending`, `due` | `pending` | | Rewardful | `paid` | `paid` | | Rewardful | `voided` | `canceled` | | Tolt | `pending`, `approved` | `pending` | | Tolt | `paid` | `paid` | | Tolt | `rejected` | `canceled` | | Tolt | `refunded` | `refunded` | | FirstPromoter | `pending`, `approved` | `pending` | | FirstPromoter | `denied` | `canceled` | Sources: [apps/web/lib/rewardful/import-commissions.ts:28-33](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/rewardful/import-commissions.ts#L28-L33), [apps/web/lib/tolt/import-commissions.ts:28-34](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/tolt/import-commissions.ts#L28-L34), [apps/web/lib/firstpromoter/import-commissions.ts:28-33](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/firstpromoter/import-commissions.ts#L28-L33) > [!TIP] > Downstream analytics records are updated concurrently with Prisma database mutations by calling `recordSaleWithTimestamp`, passing parsed click event schemas, customer identifiers, and payment processor details set to `stripe`. Sources: [apps/web/lib/firstpromoter/import-commissions.ts:320-365](https://github.com/blade47/dub/blob/7c93d97abf0ecd37fb669c6a68a1f7ef9109daea/apps/web/lib/firstpromoter/import-commissions.ts#L320-L365) ## Related - [Link Dispatch](https://www.doc0.dev/docs/934e554a-e6a1-476f-bb2f-23e62d86c3fd/technical/link-attribution/link-dispatch) - [Commission Engine](https://www.doc0.dev/docs/934e554a-e6a1-476f-bb2f-23e62d86c3fd/technical/billing-and-payouts/commission-engine) ## Sitemap See the full [sitemap](https://www.doc0.dev/docs/934e554a-e6a1-476f-bb2f-23e62d86c3fd/llms.txt) for all pages in this wiki.