--- title: "Test Runners" description: "Test Runners in the Next.js repository encompass repository-level execution scripts, Jest configurations, CLI integration wrappers, and evaluation harness infrastructure designed to validate Webpac..." last_updated: "2026-09-23T10:52:03.16467+00:00" canonical_url: "https://www.doc0.dev/docs/8f4009b0-65bd-4480-9b00-e201f0914bb3/technical/testing-infrastructure/test-runners" ---
Relevant source files The following files were used as context for generating this wiki page: - [jest.config.turbopack.js](https://github.com/blade47/next.js/blob/main/jest.config.turbopack.js) - [run-tests.js](https://github.com/blade47/next.js/blob/main/run-tests.js) - [run-evals.js](https://github.com/blade47/next.js/blob/main/run-evals.js) - [packages/next/test-runner-jest.config.js](https://github.com/blade47/next.js/blob/main/packages/next/test-runner-jest.config.js) - [jest.config.js](https://github.com/blade47/next.js/blob/main/jest.config.js) - [package.json](https://github.com/blade47/next.js/blob/main/package.json) - [packages/next/jest.js](https://github.com/blade47/next.js/blob/main/packages/next/jest.js) - [packages/next/src/cli/next-test.ts](https://github.com/blade47/next.js/blob/main/packages/next/src/cli/next-test.ts) - [packages/next/taskfile.js](https://github.com/blade47/next.js/blob/main/packages/next/taskfile.js) - [packages/next-routing/jest.config.js](https://github.com/blade47/next-routing/jest.config.js) - [packages/next/src/cli/next-info.ts](https://github.com/blade47/next.js/blob/main/packages/next/src/cli/next-info.ts) - [packages/next/jest.d.ts](https://github.com/blade47/next.js/blob/main/packages/next/jest.d.ts) - [turbo.json](https://github.com/blade47/next.js/blob/main/turbo.json) - [packages/next/experimental/testing/server.js](https://github.com/blade47/next.js/blob/main/packages/next/experimental/testing/server.js) - [packages/next/src/bundles/webpack/packages/webpack.js](https://github.com/blade47/next.js/blob/main/packages/next/src/bundles/webpack/packages/webpack.js) - [packages/next-codemod/bin/__testfixtures__/suggest-turbopack/package.json](https://github.com/blade47/next.js/blob/main/packages/next-codemod/bin/__testfixtures__/suggest-turbopack/package.json) - [packages/next/src/server/config-schema.ts](https://github.com/blade47/next.js/blob/main/packages/next/src/server/config-schema.ts) - [packages/next-codemod/bin/__testfixtures__/change-turbo-to-turbopack/package.json](https://github.com/blade47/next-codemod/bin/__testfixtures__/change-turbo-to-turbopack/package.json) - [tsconfig.json](https://github.com/blade47/next.js/blob/main/tsconfig.json) - [packages/next/experimental/testmode/playwright.js](https://github.com/blade47/next.js/blob/main/packages/next/experimental/testmode/playwright.js) - [packages/next/next-runtime.webpack-config.js](https://github.com/blade47/next.js/blob/main/packages/next/next-runtime.webpack-config.js) - [turbopack/packages/devlow-bench/src/describe.ts](https://github.com/blade47/next.js/blob/main/turbopack/packages/devlow-bench/src/describe.ts) - [packages/next/src/experimental/testing/server/index.ts](https://github.com/blade47/next.js/blob/main/packages/next/src/experimental/testing/server/index.ts) - [turbopack/packages/turbo-tracing-next-plugin/package.json](https://github.com/blade47/next.js/blob/main/turbopack/packages/turbo-tracing-next-plugin/package.json) - [turbopack/packages/devlow-bench/package.json](https://github.com/blade47/devlow-bench/package.json) - [packages/next-rspack/package.json](https://github.com/blade47/next.js/blob/main/packages/next-rspack/package.json) - [turbopack/packages/devlow-bench/src/cli.ts](https://github.com/blade47/next.js/blob/main/turbopack/packages/devlow-bench/src/cli.ts) - [packages/next/experimental/testmode/proxy.js](https://github.com/blade47/next.js/blob/main/packages/next/experimental/testmode/proxy.js) - [packages/next-playwright/package.json](https://github.com/blade47/next-playwright/package.json) - [packages/next-codemod/package.json](https://github.com/blade47/next-codemod/package.json)
## Overview Test Runners in the Next.js repository encompass repository-level execution scripts, Jest configurations, CLI integration wrappers, and evaluation harness infrastructure designed to validate Webpack, Rspack, and Turbopack runtimes. Because Next.js spans multiple complex compilation and bundling pipelines, testing requires isolating behavior across different build modes (`dev`, `start`, `deploy`), experimental react channels, and testing backends. This subsystem addresses the challenge of coordinating massive parallel suites across CI environments without cross-test state pollution or unstable timing anomalies. The architecture centers around layered test profiles, structured environment sanitization, and specialized runner adapters. Repository-level orchestration scripts like `run-tests.js` and `run-evals.js` control result caching, package packing, and worker allocation. Meanwhile, Jest configuration factories (`jest.config.js`, `jest.config.turbopack.js`) integrate Next.js custom transforms with test reports, and the Next.js CLI `next-test` command provisions Playwright configurations dynamically. This modular setup ensures that engineers can execute targeted tests locally or leverage caching and sharding mechanisms seamlessly in automated CI pipelines. Sources: [jest.config.turbopack.js:1-16](https://github.com/blade47/next.js/blob/main/jest.config.turbopack.js#L1-L16), [run-tests.js:1-147](https://github.com/blade47/next.js/blob/main/run-tests.js#L1-L147), [run-evals.js:1-91](https://github.com/blade47/next.js/blob/main/run-evals.js#L1-L91), [jest.config.js:1-85](https://github.com/blade47/next.js/blob/main/jest.config.js#L1-L85) ## Test Profile and Result Caching Mechanism ### Overview and Environment Snapshotting To optimize CI execution on retry attempts, the test runner maintains the `TestProfile` class inside `run-tests.js`. This class isolates environment variables that alter test execution behavior from operational variables that do not affect test outcomes. Sources: [run-tests.js:22-64](https://github.com/blade47/next.js/blob/main/run-tests.js#L22-L64) ### Variable Filtering and Hashing When initialized, `TestProfile` records a snapshot of `process.env`. It preserves designated behavioral variables (such as `IS_WEBPACK_TEST`, `IS_TURBOPACK_TEST`, `TURBOPACK_DEV`, `TURBOPACK_BUILD`, `BROWSER_NAME`, and `DEVICE_NAME`) while stripping known operational variables like `NEXT_TELEMETRY_DISABLED`, `NEXT_TEST_JOB`, and `NEXT_JUNIT_TEST_REPORT` via the `IGNORED_VARS` set. Sources: [run-tests.js:65-106](https://github.com/blade47/next.js/blob/main/run-tests.js#L65-L106) ### Diagnostic Description and Caching Guards It then constructs a sorted array of entries containing platform metadata, branch names, commit hashes, and node versions to form a deterministic diagnostic profile description. ```mermaid flowchart TD A["Initialize TestProfile"] --> B["Snapshot process.env"] B --> C["Filter behavioral vs IGNORED_VARS"] C --> D["Append platform, branch, sha, node version"] D --> E["Sort entries alphabetically by key"] E --> F["Generate description string & caching status"] ``` Sources: [run-tests.js:107-124](https://github.com/blade47/next.js/blob/main/run-tests.js#L107-L124) ### Passed Tests Log Loading > [!NOTE] > Caching is enabled exclusively when running in CI (`process.env.CI`), with a valid `GITHUB_SHA`, and when flake detection (`NEXT_FLAKE_DETECTION`) or cache bypassing (`NEXT_TEST_SKIP_RESULT_CACHE`) are inactive. Sources: [run-tests.js:125-147](https://github.com/blade47/next.js/blob/main/run-tests.js#L125-L147) ## Jest Configuration and Environment Factory ### Overview and Next.js Preset Integration Next.js provides extensible Jest configuration generators that adapt test environments depending on whether Webpack, Rspack, or Turbopack is targeted. The root `jest.config.js` invokes `next/jest()` to load project-specific Next.js settings asynchronously. Sources: [jest.config.js:1-5](https://github.com/blade47/next.js/blob/main/jest.config.js#L1-L5) ### Custom Configuration Properties The configuration explicitly sets `displayName` based on environment flags, defines root directories including packages like `packages/next/src` and `packages/next-codemod`, and applies strict Haste settings (`throwOnModuleCollision: true`) to prevent module warning pollution. ```javascript const nextJest = require('next/jest') const createJestConfig = nextJest() const customJestConfig = { displayName: process.env.IS_WEBPACK_TEST ? 'webpack' : 'Turbopack', testMatch: ['**/*.test.js', '**/*.test.ts', '**/*.test.jsx', '**/*.test.tsx'], globalSetup: '/jest-global-setup.ts', setupFilesAfterEnv: ['/jest-setup-after-env.ts'], verbose: true, rootDir: 'test', roots: [ '', '/../packages/next/src/', '/../packages/next-codemod/', '/../packages/eslint-plugin-internal/', '/../packages/font/src/', '/../packages/next-routing/', ], haste: { throwOnModuleCollision: true, }, modulePathIgnorePatterns: [ '/\\.next/', 'packages/next/src/compiled/', ], } module.exports = createJestConfig(customJestConfig) ``` Sources: [jest.config.js:6-48](https://github.com/blade47/next.js/blob/main/jest.config.js#L6-L48) ### Turbopack Jest Configuration Wrapper For Turbopack-specific test suites, `jest.config.turbopack.js` wraps the default configuration to append `/jest-setup-files.turbopack.js` into `setupFiles`. ```javascript const createJestDefaultConfig = require('./jest.config.js') module.exports = async function createConfig() { const jestDefaultConfig = await createJestDefaultConfig() const customConfig = { ...jestDefaultConfig, displayName: 'Turbopack', setupFiles: [ ...(jestDefaultConfig.setupFiles ?? []), '/jest-setup-files.turbopack.js', ], } return customConfig } ``` Sources: [jest.config.turbopack.js:1-16](https://github.com/blade47/next.js/blob/main/jest.config.turbopack.js#L1-L16) ### JUnit Test Report Integration Test reporting reporters are conditionally injected when `NEXT_JUNIT_TEST_REPORT` is enabled, mapping output directories depending on whether Turbopack or Rspack environment variables are present. Sources: [jest.config.js:50-85](https://github.com/blade47/next.js/blob/main/jest.config.js#L50-L85) ## CLI Integration and Next Test Runner (`next-test.ts`) ### Overview and Argument Resolution The Next.js command-line interface exposes a `next test` command (`packages/next/src/cli/next-test.ts`) that acts as a wrapper around test runners like Playwright. When invoked, `nextTest` resolves the project directory and loads the Next.js configuration for validation. Sources: [packages/next/src/cli/next-test.ts:33-68](https://github.com/blade47/next.js/blob/main/packages/next/src/cli/next-test.ts#L33-L68) ### Experimental Test Proxy Enforcement It enforces that `experimental.testProxy` is enabled in `next.config.js`, throwing a fatal error via `printAndExit` if the setting is missing. ```typescript if (!nextConfig.experimental.testProxy) { return printAndExit( `\`next experimental-test\` requires the \`experimental.testProxy: true\` configuration option.` ) } ``` Sources: [packages/next/src/cli/next-test.ts:69-80](https://github.com/blade47/next.js/blob/main/packages/next/src/cli/next-test.ts#L69-L80) ### Runner Dispatch and Dependency Checks The test runner selection priority follows the rule: CLI option (`--test-runner`) > Next config property (`experimental.defaultTestRunner`) > Default fallback (`'playwright'`). If Playwright is chosen, `runPlaywright` verifies required dependencies via `hasNecessaryDependencies`. Sources: [packages/next/src/cli/next-test.ts:81-122](https://github.com/blade47/next.js/blob/main/packages/next/src/cli/next-test.ts#L81-L122) ### Configuration Generation Fallback If `playwright.config.js` or `playwright.config.ts` is missing from the project directory, it automatically generates a default configuration file using `defaultPlaywrightConfig`. Sources: [packages/next/src/cli/next-test.ts:123-198](https://github.com/blade47/next.js/blob/main/packages/next/src/cli/next-test.ts#L123-L198) ## Evaluation Harness (`run-evals.js`) ### Overview and Tarball Packing The evaluation runner script (`run-evals.js`) coordinates agent evaluations against locally built `next` packages. It isolates execution by packing the `next` package into a tarball inside `.tarballs/next.tgz`. Sources: [run-evals.js:1-62](https://github.com/blade47/next.js/blob/main/run-evals.js#L1-L62) ### Variant Experiment File Generation The script constructs temporary experiment files that compare two distinct variants: a `baseline` variant and an `agents-md` documentation-enhanced variant. ```mermaid sequenceDiagram participant User as CLI / pnpm eval participant Runner as run-evals.js participant Pack as pnpm pack participant Agent as agent-eval Runner User->>Runner: pnpm eval Runner->>Pack: Build and pack packages/next Pack-->>Runner: Produced next.tgz Runner->>Runner: Symlink .env and .env.local into evals/ Runner->>Runner: Generate baseline.ts & agents-md.ts in experiments/ Runner->>Agent: Execute agent-eval with generated experiment config Agent-->>User: Report comparative eval success metrics ``` Sources: [run-evals.js:63-91](https://github.com/blade47/next.js/blob/main/run-evals.js#L63-L91) ### Execution Preconditions > [!WARNING] > If `packages/next/dist` is not present when invoking `run-evals.js`, the script aborts immediately with an error instructing the user to run `pnpm --filter=next build` first. Sources: [run-evals.js:92-180](https://github.com/blade47/next.js/blob/main/run-evals.js#L92-L180) ## Benchmark and Scenario Framework (`devlow-bench`) ### Overview and Subcommand Routing Turbopack benchmarking leverages the `devlow-bench` package located under `turbopack/packages/devlow-bench`. The suite exposes a programmatic scenario description API (`describe`) and command-line execution interfaces (`cli.ts`). Sources: [turbopack/packages/devlow-bench/src/cli.ts:1-30](https://github.com/blade47/next.js/blob/main/turbopack/packages/devlow-bench/src/cli.ts#L1-L30) ### Scenario Definition and Normalization The `describe` function registers scenario configurations, normalizes boolean options into arrays of variant permutations, and coordinates interface injection (`interactive`, `console`, `json`, `snapshot`, `compare`). ```typescript export function describe

( name: string, config: ConfigFor

, fn: (props: P) => Promise ): void { if (currentScenarios === null) { const scenarios = (currentScenarios = []) Promise.resolve().then(async () => { const ifaceNames = process.env.INTERFACE || 'interactive,console' const ifaces = [] for (const ifaceName of ifaceNames.split(',').map((s) => s.trim())) { // Dynamic interface loading logic } runScenarios(scenarios, compose(...ifaces)) }) } const normalizedConfig: Record = Object.fromEntries( Object.entries(config).map(([key, value]) => [ key, typeof value === 'boolean' ? [value, !value] : (value as (string | number | boolean)[]), ]) ) currentScenarios!.push({ name, config: normalizedConfig, only: false, fn: fn as (props: Record) => Promise, }) } ``` Sources: [turbopack/packages/devlow-bench/src/describe.ts:1-64](https://github.com/blade47/next.js/blob/main/turbopack/packages/devlow-bench/src/describe.ts#L1-L64) ### Measurement Reporting Guards Measurement functions validate active scenario contexts and finite numeric bounds before reporting performance metrics. Sources: [turbopack/packages/devlow-bench/src/describe.ts:65-183](https://github.com/blade47/next.js/blob/main/turbopack/packages/devlow-bench/src/describe.ts#L65-L183) ## Test Execution Reference Tables ### NPM Test Script Variants | Script Command | Target Bundler | Execution Mode | Headless | | :--- | :--- | :--- | :--- | | `pnpm test` | Webpack | Default | Headless | | `pnpm test-turbo` | Turbopack | Default | Headless | | `pnpm test-rspack` | Rspack | Default | Headless | | `pnpm test-dev-webpack` | Webpack | `dev` | Headless | | `pnpm test-dev-turbo` | Turbopack | `dev` | Headless | | `pnpm test-start-webpack` | Webpack | `start` | Headless | | `pnpm test-deploy-turbo` | Turbopack | `deploy` | Headless | Sources: [package.json:8-57](https://github.com/blade47/next.js/blob/main/package.json#L8-L57) ### Test Runner Architectural Trade-offs | Design Choice | Benefit | Cost | | :--- | :--- | :--- | | **Separated Test Profiles (`run-tests.js`)** | Enables precise CI cache keys by omitting operational noise vars. | Requires explicit maintenance of behavioral vs. ignored variable sets. | | **Dynamic Playwright Config Generation** | Zero-config initial setup for consumers adopting `next test`. | Requires runtime validation of typescript bindings and config files. | | **Isolated Package Tarball Evals (`run-evals.js`)** | Mirrors real-world consumer installs via sandboxed agent execution. | Incurs file-packing and symlinking overhead prior to execution. | Sources: [run-tests.js:22-147](https://github.com/blade47/next.js/blob/main/run-tests.js#L22-L147), [run-evals.js:50-91](https://github.com/blade47/next.js/blob/main/run-evals.js#L50-L91), [packages/next/src/cli/next-test.ts:123-187](https://github.com/blade47/next.js/blob/main/packages/next/src/cli/next-test.ts#L123-L187) ## Related - [Server Testing Utilities](https://www.doc0.dev/docs/8f4009b0-65bd-4480-9b00-e201f0914bb3/technical/testing-infrastructure/server-testing-utilities) ## Sitemap See the full [sitemap](https://www.doc0.dev/docs/8f4009b0-65bd-4480-9b00-e201f0914bb3/llms.txt) for all pages in this wiki.