---
title: "Benchmark Harness"
description: "The @vercel/devlow-bench benchmark harness is a specialized CLI-driven performance measurement tool designed to evaluate developer workflows and runtime characteristics in Next.js applications pack..."
last_updated: "2026-10-12T06:38:56.853737+00:00"
canonical_url: "https://www.doc0.dev/docs/8f4009b0-65bd-4480-9b00-e201f0914bb3/technical/companion-packages/benchmark-harness"
---

<details>
<summary>Relevant source files</summary>

The following files were used as context for generating this wiki page:

- [packages/devlow-bench/src/cli.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts)
- [packages/next/src/build/index.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/build/index.ts)
- [run-evals.js](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/run-evals.js)
- [run-tests.js](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/run-tests.js)
- [packages/next/src/export/index.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/export/index.ts)
- [packages/next/src/cli/internal/static-routes-info.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/cli/internal/static-routes-info.ts)
- [packages/devlow-bench/src/runner.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts)
- [packages/next/src/build/analyze/index.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/build/analyze/index.ts)
- [packages/devlow-bench/src/snapshot.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/snapshot.ts)
- [packages/next/src/cli/next-analyze.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/cli/next-analyze.ts)
- [packages/devlow-bench/src/interfaces/snapshot.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/interfaces/snapshot.ts)
- [packages/next/src/cli/next-info.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/cli/next-info.ts)
- [packages/next/src/cli/next-request-insights.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/cli/next-request-insights.ts)
- [packages/devlow-bench/src/describe.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts)
- [packages/devlow-bench/src/interfaces/compare.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/interfaces/compare.ts)
- [packages/devlow-bench/src/index.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/index.ts)
- [packages/next/src/build/analyze/snapshot.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/build/analyze/snapshot.ts)
- [packages/devlow-bench/package.json](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/package.json)
- [packages/next/src/server/dev/dev-validation-worker.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/server/dev/dev-validation-worker.ts)
- [packages/devlow-bench/src/browser.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts)
- [packages/next/src/cli/next-test.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/cli/next-test.ts)
- [packages/next/src/export/worker.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/export/worker.ts)
- [packages/next/taskfile.js](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/taskfile.js)
- [packages/next/src/diagnostics/build-diagnostics.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/diagnostics/build-diagnostics.ts)
- [packages/next/src/build/utils.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/build/utils.ts)
- [packages/devlow-bench/src/compare.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/compare.ts)
- [packages/next/src/build/route-bundle-stats.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/build/route-bundle-stats.ts)
- [packages/next/src/bin/next.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/next/src/bin/next.ts)
- [packages/devlow-bench/src/statistics.ts](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/statistics.ts)
</details>

## Overview

The `@vercel/devlow-bench` benchmark harness is a specialized CLI-driven performance measurement tool designed to evaluate developer workflows and runtime characteristics in Next.js applications [packages/devlow-bench/package.json:2-3](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/package.json#L2-L3), [packages/devlow-bench/src/cli.ts:28-30](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L28-L30). It provides structured scenario definitions, robust iterative execution loops with warmup handling, and Playwright-backed browser session instrumentation to track latency, resource consumption, and network activity [packages/devlow-bench/src/cli.ts:33-82](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L33-L82), [packages/devlow-bench/src/runner.ts:29-33](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L29-L33), [packages/devlow-bench/src/browser.ts:237-291](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L237-L291). By combining rigorous statistical sampling, percentile aggregation, and nonparametric significance testing (such as Mann-Whitney U) with persistent CSV snapshots and automated Git metadata tracking, the harness enables reproducible performance regression analysis across code revisions [packages/devlow-bench/src/cli.ts:84-91](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L84-L91), [packages/devlow-bench/src/snapshot.ts:9-20](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/snapshot.ts#L9-L20), [packages/devlow-bench/src/statistics.ts:81-106](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/statistics.ts#L81-L106), [packages/devlow-bench/src/compare.ts:58-73](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/compare.ts#L58-L73).

## Command-Line Interface and Subcommand Dispatch

### Commander Configuration and Command Definitions

The `devlow-bench` command-line interface is initialized using the `commander` package, configuring the program name as `devlow-bench`, setting a description for developer-workflow benchmarks, and enabling `showHelpAfterError()` [packages/devlow-bench/src/cli.ts:28-31](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L28-L31). The interface registers two primary subcommands: `run` (designated as the default subcommand) and `compare` [packages/devlow-bench/src/cli.ts:33-34](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L33-L34), [packages/devlow-bench/src/cli.ts:84-85](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L84-L85). Top-level execution errors are caught at the async IIFE boundary, printing the stack trace to `console.error` and exiting with status code `1` [packages/devlow-bench/src/cli.ts:27-97](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L27-L97).

| Subcommand | Arguments | Description | Action Handler |
| :--- | :--- | :--- | :--- |
| `run` (default) | `[scenarios...]` | Run scenario files and report measurements. | `runRun` |
| `compare` | `<baseline>`, `<current>` | Compare two snapshot CSVs side-by-side with p50/p90/p99 and Mann–Whitney U p-value. | `runCompare` |

Sources: [packages/devlow-bench/src/cli.ts:27-91](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L27-L91)

### Option Parsing and Scenario Orchestration

The `run` subcommand accepts a comprehensive set of options governing filtering, sampling iterations, warmup runs, snapshot paths, and reporting sinks [packages/devlow-bench/src/cli.ts:37-81](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L37-L81). Repeatable options like `-s, --scenario` and `-F, --filter` accumulate values into string arrays using custom accumulator functions [packages/devlow-bench/src/cli.ts:37-48](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L37-L48). The `-F` option parses `key=value` pairs, exiting with an error if the equals sign is missing [packages/devlow-bench/src/cli.ts:118-125](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L118-L125).

```typescript
.option(
  '-s, --scenario <filter>',
  'Only run scenarios whose name matches the filter (repeatable).',
  (v: string, prev: string[] | undefined) => (prev ?? []).concat(v),
  []
)
.option(
  '-F, --filter <pair>',
  'Filter variants by property: -F key=value (repeatable; same key repeated = OR).',
  (v: string, prev: string[] | undefined) => (prev ?? []).concat(v),
  []
)
.option('-i, --interactive', 'Select scenarios and variants interactively.')
.option(
  '--n <number>',
  'Run each variant N times; reports mean/p50/p90 per metric. Default: 1.',
  (v: string) => Number(v)
)
.option(
  '--warmup <number>',
  'Discard the first N runs of each variant before sampling. Default: 0. Do NOT enable when measuring cold-start metrics.',
  (v: string) => Number(v)
)
.option(
  '--snapshot <path>',
  'Override the snapshot CSV path. Default: ./.devlow-bench/snapshots/<ts>.csv. Snapshots are always written.'
)
.option(
  '--compare',
  'Print a comparison table at end of run. Baseline = newest snapshot, unless --baseline overrides.'
)
.option(
  '--baseline <path>',
  'Explicit baseline (file or directory). Implies --compare.'
)
.option('-j, --json <path>', 'Write the results to the given path as JSON.')
.option('--no-console', 'Suppress console output.')
.option(
  '--datadog [host]',
  'Upload the results to Datadog (requires DATADOG_API_KEY).'
)
.option(
  '--snowflake [batchUri]',
  'Upload the results to Snowflake (requires SNOWFLAKE_TOPIC_NAME and SNOWFLAKE_SCHEMA_ID).'
)
.action(runRun)
```
Sources: [packages/devlow-bench/src/cli.ts:37-82](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L37-L82)

> [!WARNING]
> Do not enable `--warmup` when measuring cold-start metrics, as discarding initial runs will omit the exact cold-start execution overhead being targeted [packages/devlow-bench/src/cli.ts:56-59](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L56-L59).

Sources: [packages/devlow-bench/src/cli.ts:56-59](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/cli.ts#L56-L59)

## Scenario Definition and Configuration Model

### Scenario Declarations and Typing Abstractions

Benchmark scenarios are declared using the `describe` function, which accepts a scenario name, a configuration object typed via `ConfigFor`, and an asynchronous benchmark execution function taking typed properties `P` [packages/devlow-bench/src/describe.ts:16-20](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L16-L20), [packages/devlow-bench/src/index.ts:1-9](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/index.ts#L1-L9). The `ConfigFor` utility type maps each property key in `P` to an array of possible values for strings and numbers, or an array or single boolean value for booleans [packages/devlow-bench/src/index.ts:1-9](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/index.ts#L1-L9). When `describe` is invoked for the first time, it initializes the global scenario accumulator array and asynchronously imports reporting interfaces specified by the `INTERFACE` environment variable (defaulting to `'interactive,console'`) before triggering scenario execution via `runScenarios` [packages/devlow-bench/src/describe.ts:21-45](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L21-L45). The configuration object is normalized such that boolean property values are expanded into `[value, !value]` arrays [packages/devlow-bench/src/describe.ts:46-54](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L46-L54). Each defined scenario object stores its name, normalized config dictionary, an `only` boolean flag defaulted to `false`, and the typed execution function [packages/devlow-bench/src/index.ts:11-16](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/index.ts#L11-L16), [packages/devlow-bench/src/describe.ts:55-63](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L55-L63). The `describe.only` method delegates to `describe` and marks the newly pushed scenario with `only: true` [packages/devlow-bench/src/describe.ts:65-72](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L65-L72).

| Interface / Type | Key Properties / Signatures | Purpose |
| :--- | :--- | :--- |
| `Scenario` | `name`, `config`, `only`, `fn` | Represents a registered benchmark scenario with matrix parameters. |
| `ScenarioVariant` | `scenario`, `props` | Couples a base scenario with a concrete parameter combination. |
| `CurrentScenario` | `scenario`, `iface`, `measurements` | Tracks active runtime state and recorded metrics during variant execution. |
| `ConfigFor` | Mapped type over `P` | Constrains scenario configuration inputs to arrays per property type. |

Sources: [packages/devlow-bench/src/describe.ts:16-72](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L16-L72), [packages/devlow-bench/src/index.ts:1-34](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/index.ts#L1-L34)

> [!IMPORTANT]
> The `INTERFACE` environment variable supports comma-separated module names or local paths; each imported module is automatically composed via `compose` after evaluating default exports and unwrapping factory functions [packages/devlow-bench/src/describe.ts:25-43](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L25-L43), [packages/devlow-bench/src/index.ts:86-99](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/index.ts#L86-L99).

Sources: [packages/devlow-bench/src/describe.ts:25-43](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L25-L43), [packages/devlow-bench/src/index.ts:86-99](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/index.ts#L86-L99)

### Execution Scoping and Measurement Reporting

Execution scoping is managed through the `withCurrent` utility, which binds a `CurrentScenario` instance during the invocation of an asynchronous function and restores the previous context in a `finally` block [packages/devlow-bench/src/describe.ts:74-85](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L74-L85). Within this scope, measurements are captured using `measureTime` or `reportMeasurement` [packages/devlow-bench/src/describe.ts:89-182](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L89-L182). The `reportMeasurement` function validates that a current scenario is active, that the name and unit are strings, and that the value is a finite number and not `NaN` [packages/devlow-bench/src/describe.ts:105-142](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L105-L142). Relative measurements can be anchored against the `PREVIOUS` symbol or an explicit measurement name, automatically subtracting the baseline value before dispatching through the interface sink [packages/devlow-bench/src/describe.ts:143-182](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L143-L182).

> [!CAUTION]
> Calling `reportMeasurement` or `measureTime` outside of an active `describe` execution scope throws an immediate `Error` stating that it must be called inside of `describe()` [packages/devlow-bench/src/describe.ts:115-117](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L115-L117).

Sources: [packages/devlow-bench/src/describe.ts:115-117](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/describe.ts#L115-L117)

## Benchmark Runner and Execution Lifecycle

### Iterative Variant Invocation and Sampling Loops

The `runScenarios` runner function coordinates execution across registered scenarios and their configuration parameter combinations. It expands each scenario configuration object into a Cartesian product of variant properties, rejecting any properties named `scenario` or `name` [packages/devlow-bench/src/runner.ts:29-67](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L29-L67). After filtering through interface hooks, the runner iterates through each scenario variant to collect a specified number of warmup runs (`warmup`) and measured samples (`n`) [packages/devlow-bench/src/runner.ts:68-111](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L68-L111). Total attempts are strictly bounded by `MAX_ATTEMPT_MULTIPLIER`, which defaults to a multiplier of `2` applied to the sum of warmup runs and target samples [packages/devlow-bench/src/runner.ts:24-28](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L24-L28), [packages/devlow-bench/src/runner.ts:102-111](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L102-L111).

```typescript
const MAX_ATTEMPT_MULTIPLIER = 2
```
Sources: [packages/devlow-bench/src/runner.ts:27-27](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L27-L27)

> [!WARNING]
> If a benchmark run throws an error during execution, the runner catches the exception, invokes `wrappedIface.error`, and discards all metrics buffered during that specific attempt by resetting the `perRun` map [packages/devlow-bench/src/runner.ts:121-168](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L121-L168).

Sources: [packages/devlow-bench/src/runner.ts:121-168](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L121-L168)]

### Metric Buffering and Statistical Aggregation

During a measured run, a wrapped interface intercepts metric measurements and populates a temporary `perRun` buffer. Warmup iterations execute with `collecting` set to `false`, which bypasses metric aggregation entirely [packages/devlow-bench/src/runner.ts:78-115](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L78-L115). Once a run completes successfully, recorded samples are committed into `samplesByMetric` and `collectedRuns` is incremented [packages/devlow-bench/src/runner.ts:143-160](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L143-L160).

| State Variable / Constant | Initial Value / Source | Purpose |
| :--- | :--- | :--- |
| `MAX_ATTEMPT_MULTIPLIER` | `2` | Caps total retry attempts as a multiple of `warmup + n`. |
| `collecting` | `false` | Gates whether run measurements are committed to aggregated sample sets. |
| `perRun` | `new Map()` | Buffers measurements for the current active run before validation. |
| `samplesByMetric` | `new Map()` | Accumulates successful sample arrays grouped by metric name. |

Sources: [packages/devlow-bench/src/runner.ts:18-22](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L18-L22), [packages/devlow-bench/src/runner.ts:27-27](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L27-L27), [packages/devlow-bench/src/runner.ts:71-79](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L71-L79)]

After finishing all attempts for a variant, summary statistics are calculated using the `summary` helper function and dispatched to `fullIface.variantStatistics` [packages/devlow-bench/src/runner.ts:185-200](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L185-L200). If any variant falls short of collecting the target number of samples `n`, the runner sets `process.exitCode = 1` [packages/devlow-bench/src/runner.ts:180-184](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L180-L184).

```typescript
if (collectedRuns < n) {
  process.exitCode = 1
  variantsShortOfTarget++
}

```
Sources: [packages/devlow-bench/src/runner.ts:180-184](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/runner.ts#L180-L184)]

## Browser Interaction and Navigation Instrumentation

### Simulated Browser Sessions and Launch Options

The benchmarking suite utilizes Playwright Chromium to manage browser instances through the `newBrowserSession` factory function [packages/devlow-bench/src/browser.ts:358-375](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L358-L375). This function accepts configuration options controlling headless execution, developer tools visibility, and the base URL [packages/devlow-bench/src/browser.ts:358-362](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L358-L375). Chromium launches with a default timeout of 60,000 milliseconds, automatically attaching the `--auto-open-devtools-for-tabs` argument when headless mode is disabled [packages/devlow-bench/src/browser.ts:363-367](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L363-L367).

Browser contexts are instantiated with a default viewport size of 1280x720 pixels and a base URL defaulting to `http://localhost:3000` [packages/devlow-bench/src/browser.ts:368-371](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L368-L371). Furthermore, context-level timeouts for general operations and navigation events are explicitly configured to 120,000 milliseconds [packages/devlow-bench/src/browser.ts:372-373](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L372-L373).

| Option / Parameter | Default Value / Behavior | Purpose |
| :--- | :--- | :--- |
| `headless` | `process.env.HEADLESS !== 'false'` | Controls headless browser execution. |
| `devtools` | `false` (implicit via CLI/args) | Toggles developer tools window visibility. |
| `baseURL` | `http://localhost:3000` | Sets the baseline URL for relative navigations. |
| Viewport | `{ width: 1280, height: 720 }` | Enforces a standardized layout dimension. |
| Context Timeout | `120000` ms | Sets default timeout for general browser actions. |
| Navigation Timeout | `120000` ms | Sets default timeout specifically for page navigations. |

Sources: [packages/devlow-bench/src/browser.ts:358-373](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L358-L373)]

### Hard and Soft Navigations

The `BrowserSessionImpl` class coordinates page-level navigation primitives, managing instance reuse across test actions [packages/devlow-bench/src/browser.ts:237-356](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L237-L356). A hard navigation via `hardNavigation(metricName, url)` initializes a new page if none exists, wrapping the operation with request metrics tracking [packages/devlow-bench/src/browser.ts:255-260](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L255-L260). It targets a commit load state, validating that the HTTP response is both present and successful before recording progressive metrics for HTML parsing, DOM content loading, and full page load [packages/devlow-bench/src/browser.ts:261-288](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L261-L288).

```typescript
async hardNavigation(metricName: string, url: string) {
  this.page = this.page ?? (await this.context.newPage())

  const page = this.page
  await withRequestMetrics(metricName, page, async () => {
    await measureTime(`${metricName}/start`)
    const idle = networkIdle(page, 3000)
    const response = await page.goto(url, {
      waitUntil: 'commit',
    })
    if (!response) {
      throw new Error(`Navigation to ${url} produced no response`)
    }
    if (!response.ok()) {
      throw new Error(
        `Navigation to ${url} returned HTTP ${response.status()}`
      )
    }
    await measureTime(`${metricName}/html`, {
      relativeTo: `${metricName}/start`,
    })
    await page.waitForLoadState('domcontentloaded')
    await measureTime(`${metricName}/dom`, {
      relativeTo: `${metricName}/start`,
    })
    await page.waitForLoadState('load')
    await measureTime(`${metricName}/load`, {
      relativeTo: `${metricName}/start`,
    })
    const offset = await idle
    await measureTime(`${metricName}`, {
      offset,
      relativeTo: `${metricName}/start`,
    })
  })
  return page
}
```
Sources: [packages/devlow-bench/src/browser.ts:255-291](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L255-L291)]

> [!CAUTION]
> The `softNavigationByClick(metricName, selector)` and `reload(metricName)` methods require an active page instance and will throw an immediate error if invoked prior to a successful `hardNavigation()` call [packages/devlow-bench/src/browser.ts:293-299](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L293-L299), [packages/devlow-bench/src/browser.ts:321-325](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L321-L325).

Sources: [packages/devlow-bench/src/browser.ts:293-299](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L293-L299), [packages/devlow-bench/src/browser.ts:321-325](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L321-L325)]

### Network Idle Tracking and Request Metrics

Network stability is quantified by monitoring active HTTP requests and response lifecycles on the target page [packages/devlow-bench/src/browser.ts:217-235](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L217-L235). The `networkIdle` function tracks inflight requests using event listeners for `request`, `requestfailed`, and `requestfinished` events [packages/devlow-bench/src/browser.ts:231-233](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L231-L233). When all tracked requests settle and the active request count drops to zero, a debounce timer of 3000 milliseconds is initiated before concluding the idle measurement window [packages/devlow-bench/src/browser.ts:223-228](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L223-L228).

```typescript
      if (currentCount === 1) {
        requests.delete(request.url())
      } else {
        requests.set(request.url(), currentCount - 1)
      }

      if (requests.size === 0) {
        timeout = setTimeout(() => {
          cleanup()
          resolve(Date.now() - lastRequest)
        }, delayMs)
      }
    }

    page.on('request', requestHandler)
    page.on('requestfailed', requestFinishedHandler)
    page.on('requestfinished', requestFinishedHandler)
  })
}
```
Sources: [packages/devlow-bench/src/browser.ts:217-235](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/browser.ts#L217-L235)]

## Statistical Sampling and Significance Analysis

### Aggregation and Quantile Estimation

Repeated-sampling performance metrics are processed into centralized summaries using descriptive statistics [[packages/devlow-bench/src/statistics.ts:1-18]. The `mean` function computes the arithmetic average of numeric sample arrays, returning `NaN` on empty inputs [[packages/devlow-bench/src/statistics.ts:20-25]. Quantile values—specifically the 50th percentile (`p50`) and 90th percentile (`p90`)—are derived using linear interpolation corresponding to the NumPy and R "type 7" default strategy [[packages/devlow-bench/src/statistics.ts:27-37]. The `summary` function wraps these helpers to construct a `Summary` interface containing the mean, median, and 90th percentile [[packages/devlow-bench/src/statistics.ts:14-18, 39-45].

```typescript
export function mean(xs: number[]): number {
  if (xs.length === 0) return NaN
  let s = 0
  for (const x of xs) s += x
  return s / xs.length
}

// Linear-interpolation quantile (numpy/R "type 7" default).
export function quantile(xs: number[], q: number): number {
  if (xs.length === 0) return NaN
  if (xs.length === 1) return xs[0]
  const sorted = [...xs].sort((a, b) => a - b)
  const pos = q * (sorted.length - 1)
  const lo = Math.floor(pos)
  const hi = Math.ceil(pos)
  if (lo === hi) return sorted[lo]
  return sorted[lo] + (pos - lo) * (sorted[hi] - sorted[lo])
}

export function summary(samples: number[]): Summary {
  return {
    mean: mean(samples),
    p50: quantile(samples, 0.5),
    p90: quantile(samples, 0.9),
  }
}
```
Sources: [packages/devlow-bench/src/statistics.ts:20-45](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/statistics.ts#L20-L45)]

### Significance Testing and Distributions

Comparative significance between sample groups is evaluated via Welch's t-test and the Mann-Whitney U test, relying on `jstat` for Student's t and standard normal cumulative distribution functions [[packages/devlow-bench/src/statistics.ts:3-12, 53-106]. The `welchsTTest` function handles unequal variances and computes Welch-Satterthwaite degrees of freedom before calculating a two-sided p-value [[packages/devlow-bench/src/statistics.ts:53-79]. The `mannWhitneyU` function applies an asymptotic normal approximation with continuity correction based on average ranks that correct for ties by assigning the mean rank [[packages/devlow-bench/src/statistics.ts:86-124].

| Statistical Function | Return Interface | Underlying Distribution / Method |
| :--- | :--- | :--- |
| `welchsTTest` | `WelchsTResult` (`t`, `df`, `p`) | Student's t-distribution CDF (`jstat.studentt.cdf`) with Welch-Satterthwaite degrees of freedom |
| `mannWhitneyU` | `MannWhitneyResult` (`u`, `p`) | Standard normal CDF (`jstat.normal.cdf`) with continuity correction on average ranks |

Sources: [packages/devlow-bench/src/statistics.ts:47-51, 75-77, 81-84, 103-104]

> [!CAUTION]
> When all values in both sample arrays passed to `welchsTTest` are identical, the function bypasses standard variance calculations and directly returns extreme t-statistics of `0`, `Infinity`, or `-Infinity` alongside degenerate p-values of `1` or `0` depending on whether sample means match [[packages/devlow-bench/src/statistics.ts:56-64].

Sources: [packages/devlow-bench/src/statistics.ts:56-64](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/statistics.ts#L56-L64)]

## Snapshot Storage and Git Metadata

### Snapshot Persistence and CSV Formatting

Performance metrics from benchmark runs are persisted as comma-separated values using a long-format data structure [[packages/devlow-bench/src/snapshot.ts:6-20]. Each record stores a single sample measurement per scenario, variant, and metric, avoiding sparse columns when mixing units such as milliseconds, megabytes, or request counts [[packages/devlow-bench/src/snapshot.ts:6-20]. The schema comprises ten fixed columns defined in `COLUMNS` [[packages/devlow-bench/src/snapshot.ts:22-33].

| Column Name | Type | Description |
| :--- | :--- | :--- |
| `timestamp` | string | ISO timestamp when the snapshot interface was initialized |
| `scenario` | string | Name of the benchmark scenario |
| `variant` | string | Formatted configuration variant properties |
| `metric` | string | Name of the measured metric |
| `sample_idx` | number | 1-based index of the sample iteration |
| `value` | number | Numeric sample measurement |
| `unit` | string | Unit of measurement (e.g., ms, MB) |
| `relative_to` | string | Optional baseline variant reference |
| `git_sha` | string | Git commit hash associated with the run |
| `git_branch` | string | Git branch name associated with the run |

Sources: [packages/devlow-bench/src/snapshot.ts:9-33]

Snapshots are stored in directory `.devlow-bench/snapshots` by default [[packages/devlow-bench/src/snapshot.ts:4]. File paths are generated via `defaultSnapshotPath`, which creates ISO-based filenames with colons and periods replaced by hyphens for filesystem safety and lexicographic sorting [[packages/devlow-bench/src/snapshot.ts:35-42]. The `writeSnapshot` function ensures parent directories exist recursively before serializing headers and escaped CSV rows to disk [[packages/devlow-bench/src/snapshot.ts:44-54]. The inverse operation, `readSnapshot`, parses CSV text into structured `SnapshotRow` entries, dynamically mapping header positions to object keys [[packages/devlow-bench/src/snapshot.ts:56-88].

```typescript
export async function writeSnapshot(
  path: string,
  rows: SnapshotRow[]
): Promise<void> {
  await mkdir(dirname(path), { recursive: true })
  const lines: string[] = [COLUMNS.join(',')]
  for (const row of rows) {
    lines.push(COLUMNS.map((c) => csvCell(row[c])).join(','))
  }
  await writeFile(path, lines.join('\n') + '\n')
}
```
Sources: [packages/devlow-bench/src/snapshot.ts:44-54](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/snapshot.ts#L44-L54)]

### Git Revision Tracking and Historical Artifact Indexing

Before writing records to disk, the snapshot reporting interface retrieves version control metadata via `readGitInfo` [[packages/devlow-bench/src/interfaces/snapshot.ts:14-20]. It checks environment variables `GITHUB_SHA` and `GITHUB_REF_NAME` first, falling back to executing `git rev-parse HEAD` and `git rev-parse --abbrev-ref HEAD` via `execFile` [[packages/devlow-bench/src/interfaces/snapshot.ts:14-20]. If git execution fails or repositories are absent, `tryGit` catches errors and returns empty strings [[packages/devlow-bench/src/interfaces/snapshot.ts:22-29]. Once gathered, the commit hash and branch name are injected into every pending snapshot row prior to file serialization [[packages/devlow-bench/src/interfaces/snapshot.ts:61-66].

```typescript
async function readGitInfo(): Promise<{ sha: string; branch: string }> {
  const sha = process.env.GITHUB_SHA || (await tryGit(['rev-parse', 'HEAD']))
  const branch =
    process.env.GITHUB_REF_NAME ||
    (await tryGit(['rev-parse', '--abbrev-ref', 'HEAD']))
  return { sha, branch }
}
```
Sources: [packages/devlow-bench/src/interfaces/snapshot.ts:14-20](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/interfaces/snapshot.ts#L14-L20)]

Historical artifacts are indexed and queried through directory scanners [[packages/devlow-bench/src/snapshot.ts:92-144]. The `findMostRecentSnapshot` function reads snapshot directories, filters for `.csv` files, and tracks modification times (`mtimeMs`) to locate the newest artifact while optionally excluding a specific path [[packages/devlow-bench/src/snapshot.ts:92-115]. The companion function `resolveCompareTarget` processes `--compare` command inputs, supporting boolean flags to pick the newest snapshot automatically, directory paths to search within, or direct file paths [[packages/devlow-bench/src/snapshot.ts:119-144].

```typescript
export async function findMostRecentSnapshot(
  dir: string = DEFAULT_SNAPSHOT_DIR,
  excludePath?: string
): Promise<string | null> {
  let entries: string[]
  try {
    entries = await readdir(dir)
  } catch (e: any) {
    if (e.code === 'ENOENT') return null
    throw e
  }
  const candidates = entries.filter((f) => f.endsWith('.csv'))
  const excludeResolved = excludePath ? resolve(excludePath) : null
  let best: { path: string; mtime: number } | null = null
  for (const f of candidates) {
    const full = resolve(join(dir, f))
    if (excludeResolved && full === excludeResolved) continue
    const s = await stat(full)
    if (!best || s.mtimeMs > best.mtime) {
      best = { path: full, mtime: s.mtimeMs }
    }
  }
  return best?.path ?? null
}
```
Sources: [packages/devlow-bench/src/snapshot.ts:92-115](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/snapshot.ts#L92-L115)]

> [!NOTE]
> When `findMostRecentSnapshot` encounters an `ENOENT` error during directory reading, it gracefully catches the error code and returns `null` instead of crashing, allowing fallback routines to initialize clean snapshot targets [[packages/devlow-bench/src/snapshot.ts:96-102].

Sources: [packages/devlow-bench/src/snapshot.ts:96-102](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/snapshot.ts#L96-L102)]

## Regression Comparison and Result Reporting

### Diffing Snapshots and Sample Grouping

The comparison engine ingests benchmark snapshot rows and organizes them into a memory structure for differential analysis [[packages/devlow-bench/src/compare.ts:25-46]. The `groupRows` function iterates over raw `SnapshotRow` entries, filtering out non-finite measurements such as failed CSV parses [[packages/devlow-bench/src/compare.ts:27-30]. Each retained sample is indexed using a composite key generated by `makeKey`, which concatenates the scenario, variant, and metric strings separated by null bytes (`\x00`) to prevent collisions [[packages/devlow-bench/src/compare.ts:31-33, 48-54]. The resulting data structure maps each unique key to a `SampleGroup` containing the scenario identifier, variant properties, metric name, unit, and an array of numeric sample values [[packages/devlow-bench/src/compare.ts:17-23, 34-44].

```typescript
export function groupRows(rows: SnapshotRow[]): Map<string, SampleGroup> {
  const out = new Map<string, SampleGroup>()
  for (const r of rows) {
    if (!Number.isFinite(r.value)) continue
    const key = makeKey(r.scenario, r.variant, r.metric)
    let group = out.get(key)
    if (!group) {
      group = {
        scenario: r.scenario,
        variant: r.variant,
        metric: r.metric,
        unit: r.unit,
        samples: [],
      }
      out.set(key, group)
    }
    group.samples.push(r.value)
  }
  return out
}
```
Sources: [packages/devlow-bench/src/compare.ts:27-46](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/compare.ts#L27-L46)]

To compare a current run against a historical snapshot, the `printComparison` function aligns baseline and current `SampleGroup` maps [[packages/devlow-bench/src/compare.ts:58-99]. It categorizes unmatched series into distinct lists: metrics present only in the current run, metrics present only in the baseline, and series skipped due to incompatible measurement units [[packages/devlow-bench/src/compare.ts:75-98]. For metrics present in both datasets, sample counts are summarized using the `modeN` utility, which identifies the most frequent sample size across all pairs and breaks ties in favor of the larger collection size [[packages/devlow-bench/src/compare.ts:103-104, 147-163].

```typescript
// Picks the most common n in the list. Ties are broken in favor of the
// larger n (treat "the typical fully-collected sample count" as the default).
// Returns null for an empty list.
function modeN(ns: number[]): number | null {
  if (ns.length === 0) return null
  const counts = new Map<number, number>()
  for (const n of ns) counts.set(n, (counts.get(n) ?? 0) + 1)
  let bestN = ns[0]
  let bestCount = 0
  for (const [n, c] of counts) {
    if (c > bestCount || (c === bestCount && n > bestN)) {
      bestN = n
      bestCount = c
    }
  }
  return bestN
}
```
Sources: [packages/devlow-bench/src/compare.ts:147-163](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/compare.ts#L147-L163)]

> [!NOTE]
> The `modeN` function implements tie-breaking by preferring larger sample counts (`n > bestN`), ensuring that fully-collected benchmark sets are chosen as the default sample count over partial runs [[packages/devlow-bench/src/compare.ts:147-163].

Sources: [packages/devlow-bench/src/compare.ts:147-163](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/compare.ts#L147-L163)]

### Statistical Comparison Tables and Delta Formatting

For each matched series, the comparison pipeline computes distribution percentiles (p50, p90, p99) via quantile calculations and evaluates statistical significance using the Mann–Whitney U test [[packages/devlow-bench/src/compare.ts:171-175, 196-206]. A row verdict of `'improved'` or `'regressed'` is assigned only if the Mann–Whitney $p$-value falls below the significance threshold of `0.05`, the absolute difference in p50 values is non-zero, and the shift direction indicates performance gain or loss [[packages/devlow-bench/src/compare.ts:8-10, 178-181].

| Column Name | Content Description | Formatting Rules |
| :--- | :--- | :--- |
| `Scenario · Variant` | Combined scenario and variant name | Plain string |
| `Metric` | Measured metric identifier | Plain string |
| `Baseline p50 / p90 / p99` | Baseline distribution percentiles | Unit-split values with optional sample count suffix |
| `Current p50 / p90 / p99` | Current distribution percentiles | Unit-split values with optional sample count suffix |
| `Δ p50` | Absolute change in p50 median | Signed unit-formatted string (`+`, `-`, or `n/a`) |
| `Δ%` | Percentage change in p50 median | Fixed-point percentage (`10.5%`) or `n/a` |
| `p` | Mann–Whitney U test $p$-value | Exponential notation for $p < 1e-3$, fixed 3 decimals otherwise |

Sources: [packages/devlow-bench/src/compare.ts:114-122, 183-191]

Table cells containing unit measurements utilize `splitFormattedUnit` to parse numeric quantities from unit strings [[packages/devlow-bench/src/compare.ts:228-236]. When all three percentiles share an identical unit suffix, the formatter prints the suffix exclusively on the final value to eliminate redundancy [[packages/devlow-bench/src/compare.ts:213-226].

```typescript
function formatGroupCell(
  group: SampleGroup,
  s: { p50: number; p90: number; p99: number },
  showN: boolean
): string {
  const parts = [s.p50, s.p90, s.p99].map((v) =>
    splitFormattedUnit(v, group.unit)
  )
  const nSuffix = showN ? ` (${group.samples.length})` : ''
  // If all three values render with the same unit suffix (e.g. all "requests"),
  // only print the suffix on the last value to avoid "7 req / 7 req / 7 req".
  if (
    parts[0].suffix === parts[1].suffix &&
    parts[1].suffix === parts[2].suffix
  ) {
    return `${parts[0].num} / ${parts[1].num} / ${parts[2].num}${parts[2].suffix}${nSuffix}`
  }
  return `${parts[0].num}${parts[0].suffix} / ${parts[1].num}${parts[1].suffix} / ${parts[2].num}${parts[2].suffix}${nSuffix}`
}
```
Sources: [packages/devlow-bench/src/compare.ts:208-226](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/compare.ts#L208-L226)]

### Interface Reporting Sinks

The benchmark harness abstracts result reporting through the `Interface` contract, allowing execution loops to delegate output generation [[packages/devlow-bench/src/interfaces/compare.ts:11-37]. The comparison interface implementation reads baseline rows from disk via `readSnapshot`, groups them, and populates a current sample map as variant statistics stream in from benchmark scenarios [[packages/devlow-bench/src/interfaces/compare.ts:11-31]. Upon completion of all test runs, the `finish` hook invokes `printComparison`, rendering the formatted differential table and unmatched series lists to the console using color-coded verdicts from `picocolors` [[packages/devlow-bench/src/compare.ts:1-6, 33-35, 251-272].

```typescript
export default async function createInterface(options: {
  baselinePath: string
}): Promise<Interface> {
  const baselinePath = options.baselinePath
  const baselineRows = await readSnapshot(baselinePath)
  const baseline = groupRows(baselineRows)
  const current = new Map<string, SampleGroup>()

  return {
    variantStatistics: async (scenario, props, stats) => {
      const variant = formatVariantProps(props)
      for (const [metric, s] of Object.entries(stats)) {
        const key = makeKey(scenario, variant, metric)
        current.set(key, {
          scenario,
          variant,
          metric,
          unit: s.unit,
          samples: s.samples.slice(),
        })
      }
    },
    finish: async () => {
      printComparison(baseline, current, { baselineLabel: baselinePath })
    },
  }
```
Sources: [packages/devlow-bench/src/interfaces/compare.ts:11-36](https://github.com/blade47/next.js/blob/d8c1c1b0002db8ccb63b58095f338e8bf423fa99/packages/devlow-bench/src/interfaces/compare.ts:11-36)]

## Related

- [Trace Profiling](https://www.doc0.dev/docs/8f4009b0-65bd-4480-9b00-e201f0914bb3/technical/cli-telemetry/trace-profiling)


## Sitemap

See the full [sitemap](https://www.doc0.dev/docs/8f4009b0-65bd-4480-9b00-e201f0914bb3/llms.txt) for all pages in this wiki.