Data Importers

Overview

Data Importers facilitate the migration of historical link, partner, customer, and commission data from external URL shorteners and affiliate management platforms into Dub. By providing automated ingestion pathways, the system bridges legacy external infrastructures with Dub's workspace environment and partner programs.

Sources: apps/web/app/api/workspaces/idOrSlug/import/bitly/route.ts:60-140, apps/web/app/ee/api/cron/import/rewardful/route.ts:13-37

Workspace link import endpoints

Provider Route Architecture

Dub provides authenticated workspace-scoped HTTP routes to connect, inspect, and initiate link migrations from external URL shorteners including Bitly, Rebrandly, and Short.io. Each provider integration exposes standard HTTP verbs (GET, PUT, POST) guarded by workspace authorization checks and specific permission requirements such as links.write and folders.links.write.

Sources: apps/web/app/api/workspaces/idOrSlug/import/bitly/route.ts:13-143, apps/web/app/api/workspaces/idOrSlug/import/rebrandly/route.ts:13-165, apps/web/app/api/workspaces/idOrSlug/import/short/route.ts:13-147

The migration endpoints perform token validation against secure Redis storage, query external provider APIs to fetch domains, groups, and link counts, and provision missing custom domains into both Prisma and Vercel prior to dispatching background ingestion jobs via QStash.

Sources: apps/web/app/api/workspaces/idOrSlug/import/bitly/route.ts:15-22, apps/web/app/api/workspaces/idOrSlug/import/rebrandly/route.ts:15-23, apps/web/app/api/workspaces/idOrSlug/import/short/route.ts:15-23

Sources: apps/web/app/api/workspaces/idOrSlug/import/bitly/route.ts:72-101, apps/web/app/api/workspaces/idOrSlug/import/rebrandly/route.ts:116-142, apps/web/app/api/workspaces/idOrSlug/import/short/route.ts:99-125

On-demand Bitly edge crawling

Edge Crawl Execution Flow

When an incoming request cannot be resolved through standard database lookups during edge redirect dispatch, the crawlBitly utility intercepts requests to evaluate whether the requested key matches Bitly shortlink structures. Keys containing unsupported characters specified by Bitly's character regex are immediately rejected from being crawled.

Sources: apps/web/lib/middleware/utils/crawl-bitly.ts:20-27

Copyconst invalidBitlyKeyRegex = /[`~,.<>;':"/\\[\]^{}()=+!*@&$£?%#|]/;

Sources: apps/web/lib/middleware/utils/crawl-bitly.ts:25-25

For valid keys, the function queries the Bitly API endpoint using the configured BUFFER_BITLY_API_KEY. If the remote bitlink exists and returns a validated long URL via getUrlFromStringIfValid, the system provisions the link directly into the database as an on-demand creation assigned to dedicated buffer workspace identifiers.

Sources: apps/web/lib/middleware/utils/crawl-bitly.ts:15-17, apps/web/lib/middleware/utils/crawl-bitly.ts:35-50

Sources: apps/web/lib/middleware/utils/crawl-bitly.ts:100-105

Background Persistence and Fallback Routing

Once a bitlink record is retrieved from the Bitly API and sanitized, ev.waitUntil schedules background persistence tasks using Prisma to create the link entry, record analytics via Tinybird through recordLink, and publish workspace usage events to Upstash Redis streams via publishWorkspaceLinksUsageEvent.

Sources: apps/web/lib/middleware/utils/crawl-bitly.ts:36-61

Copy.then((data) =>
  Promise.allSettled([
    recordLink(data),
    publishWorkspaceLinksUsageEvent({
      workspaceId: BUFFER_WORKSPACE_ID,
      linksCount: 1,
      timestamp: new Date().toISOString(),
    }),
  ]),
),

Sources: apps/web/lib/middleware/utils/crawl-bitly.ts:51-60

If the remote lookup fails or yields no valid link, the middleware issues a fallback redirect to https://buffer.com with a 302 status code and attaches static Vercel CDN cache headers with an s-maxage of 86400 seconds alongside cache tags generated via linkCache._createStaticPagesCacheKeys.

Sources: apps/web/lib/middleware/utils/crawl-bitly.ts:70-81

Copyreturn NextResponse.redirect("https://buffer.com", {
  headers: {
    ...DUB_HEADERS,
    "Vercel-CDN-Cache-Control": "public, s-maxage=86400",
    "Vercel-Cache-Tag": linkCache._createStaticPagesCacheKeys({
      domain,
      key,
    }),
  },
  status: 302,
});

Sources: apps/web/lib/middleware/utils/crawl-bitly.ts:71-81

Asynchronous link import workers

QStash Signature Verification and Cron Routing

Background synchronization for external provider migrations relies on POST endpoints configured with force-dynamic rendering. Every incoming request body is intercepted, read as text, and validated against QStash cryptographic signatures via verifyQstashSignature before JSON parsing occurs.

Sources: apps/web/app/ee/api/cron/import/bitly/route.ts:12-19, apps/web/app/ee/api/cron/import/rebrandly/route.ts:9-16, apps/web/app/ee/api/cron/import/short/route.ts:9-16

Copyexport async function POST(req: Request) {
  try {
    const rawBody = await req.text();
    await verifyQstashSignature({ req, rawBody });
 
    const body = JSON.parse(rawBody);

Sources: apps/web/app/ee/api/cron/import/bitly/route.ts:14-19

Depending on the provider target, requests parse specific parameters from the payload to drive batch processing and handle authentication credentials stored inside Redis keys.

Sources: apps/web/app/ee/api/cron/import/bitly/route.ts:19-23, apps/web/app/ee/api/cron/import/rebrandly/route.ts:16-22, apps/web/app/ee/api/cron/import/short/route.ts:16-26

Sources: apps/web/app/ee/api/cron/import/rebrandly/route.ts:20-29

Tag Synchronization and Rate Limiting Flow

When import routines execute with importTags enabled, workers check Redis for previously cached flags (import:bitly:${workspaceId}:tags or import:rebrandly:${workspaceId}:tags). If absent, tag definitions are fetched from the upstream provider API, inserted into Prisma via createId with generated badge colors, and mapped to database identifiers.

Sources: apps/web/app/ee/api/cron/import/bitly/route.ts:35-82, apps/web/app/ee/api/cron/import/rebrandly/route.ts:31-62

Copy  await prisma.tag.createMany({
    data: tags.map((tag) => ({
      id: createId({ prefix: "tag_" }),
      name: tag,
      color: randomBadgeColor(),
      projectId: workspaceId,
    })),
    skipDuplicates: true,
  });
  await redis.set(`import:bitly:${workspaceId}:tags`, "true");
}

Sources: apps/web/app/ee/api/cron/import/bitly/route.ts:54-64

Bitly background workers also evaluate provider-specific rate limits when the rateLimited flag evaluates to true, returning a JSON response indicating rate_limited status when thresholds are exceeded.

Sources: apps/web/app/ee/api/cron/import/bitly/route.ts:25-33

Sources: apps/web/app/api/callback/bitly/route.ts:46-53

Affiliate partner ingestion workflow

Execution Walkthrough and Sequence

Affiliate partner migration pipelines ingest programs, campaigns, and active partners from platforms like Rewardful and FirstPromoter via QStash-verified background cron routes. The execution router evaluates the incoming POST action payload to dispatch processing across specific handlers for campaigns, partners, affiliate coupons, customers, and commissions.

Sources: apps/web/app/ee/api/cron/import/rewardful/route.ts:13-36

1. POST — The QStash cron endpoint verifies message signatures and parses payload actions, routing execution to importPartners for active affiliate ingestion via apps/web/app/(ee)/api/cron/import/rewardful/route.ts. Sources: apps/web/app/ee/api/cron/import/rewardful/route.ts:13-26

2. importPartners — Fetches workspace credentials from Redis, instantiates provider clients, pages through remote affiliate records, filters active partners with existing leads or matching campaign IDs, and delegates creation to createPartnerAndLinks via apps/web/lib/rewardful/import-partners.ts. Sources: apps/web/lib/rewardful/import-partners.ts:14-110, apps/web/lib/firstpromoter/import-partners.ts:19-102

3. createPartnerAndLinks — Upserts partner database records by email, establishes approved program enrollments with group attributes, invokes application approval handlers, and bulk-creates referral links via apps/web/lib/rewardful/import-partners.ts. Sources: apps/web/lib/rewardful/import-partners.ts:165-256, apps/web/lib/firstpromoter/import-partners.ts:116-254

Sources: apps/web/app/ee/api/cron/import/rewardful/route.ts:13-26, apps/web/lib/rewardful/import-partners.ts:14-256

Platform Integration Parameters and Limits

Sources: apps/web/lib/rewardful/importer.ts:8-13, apps/web/lib/rewardful/import-partners.ts:45-52, apps/web/lib/firstpromoter/importer.ts:8-10, apps/web/lib/firstpromoter/import-partners.ts:52-56

Sources: apps/web/lib/firstpromoter/import-partners.ts:240-252

Customer referral ingestion pipeline

Ingestion Flow and Synthetic Click Generation

Customer migrations from Rewardful and Tolt parse paginated referral payloads, verify existing database records by Stripe customer identifier or external ID, and manufacture synthetic click events for historical attribution. Because historical imports lack live HTTP requests, the pipeline constructs a dummy Request object with predefined headers (user-agent, x-forwarded-for, and Vercel geo headers) to simulate visitor traffic against the target link URL.

Sources: apps/web/lib/rewardful/import-customers.ts:31-75, apps/web/lib/rewardful/import-customers.ts:210-236, apps/web/lib/tolt/import-customers.ts:33-121, apps/web/lib/tolt/import-customers.ts:205-231

Sources: apps/web/lib/rewardful/import-customers.ts:36-90, apps/web/lib/rewardful/import-customers.ts:210-296, apps/web/lib/tolt/import-customers.ts:34-136, apps/web/lib/tolt/import-customers.ts:205-292

Platform-Specific Ingestion Rules

Sources: apps/web/lib/rewardful/import-customers.ts:35, apps/web/lib/rewardful/import-customers.ts:153-169, apps/web/lib/rewardful/import-customers.ts:194-208, apps/web/lib/tolt/import-customers.ts:33, apps/web/lib/tolt/import-customers.ts:83-94, apps/web/lib/tolt/import-customers.ts:111-120, apps/web/lib/tolt/import-customers.ts:195-201

Sources: apps/web/lib/rewardful/import-customers.ts:181-192

Sources: apps/web/lib/rewardful/import-customers.ts:77-90, apps/web/lib/tolt/import-customers.ts:122-138

Commission sync and attribution updates

Historical commissions are backfilled via asynchronous workers triggered through QStash endpoints in cron routes such as apps/web/app/(ee)/api/cron/import/firstpromoter/route.ts, apps/web/app/(ee)/api/cron/import/rewardful/route.ts, and apps/web/app/(ee)/api/cron/import/tolt/route.ts. During commission processing, sale amounts expressed in foreign currencies are converted to USD utilizing real-time exchange rates fetched from Upstash Redis hashes (fxRates:usd).

Copylet saleAmount = Number(commission.original_sale_amount ?? 0);
const saleCurrency = commission.original_sale_currency ?? "usd";
 
if (saleAmount > 0 && saleCurrency.toUpperCase() !== "USD" && fxRates) {
  const { amount: convertedAmount } = convertCurrencyWithFxRates({
    currency: saleCurrency,
    amount: saleAmount,
    fxRates,
  });
 
  saleAmount = convertedAmount;
}

Sources: apps/web/lib/firstpromoter/import-commissions.ts:231-242, apps/web/lib/rewardful/import-commissions.ts:48, apps/web/lib/tolt/import-commissions.ts:52

Sources: apps/web/lib/firstpromoter/import-commissions.ts:181-199

Status mappings vary across upstream affiliate providers to unify commission states within Dub databases:

Sources: apps/web/lib/rewardful/import-commissions.ts:28-33, apps/web/lib/tolt/import-commissions.ts:28-34, apps/web/lib/firstpromoter/import-commissions.ts:28-33

Sources: apps/web/lib/firstpromoter/import-commissions.ts:320-365

Related

• Link Dispatch
• Commission Engine