AI-Native Updated 2026-09 12 min read View as Markdown

Testing JavaScript that calls a language model

Your function is now non-deterministic, slow, costs money per call, and streams. Here is a test strategy that still works — including for the streaming part, which is where most of the real bugs are.

More and more Node code is a thin layer around a model call. That code has properties ordinary JavaScript does not: the same input gives different output, each test costs money and a second or two, and — the part specific to JavaScript — most of it streams, which means half your bug surface is in the plumbing rather than the model.

The instinct is to mock the SDK and test nothing meaningful. There is a better middle, and it looks like a pyramid.

The pyramid#

      few, slow, expensive
   ┌──────────────────────────┐
   │  4. live evals (nightly) │   real model, scored dataset
   ├──────────────────────────┤
   │  3. cassette tests (CI)  │   recorded responses, real shapes
   ├──────────────────────────┤
   │  2. contract tests       │   parsing, streaming, aborts, retries
   ├──────────────────────────┤
   │  1. pure logic tests     │   prompt building, chunking, routing
   └──────────────────────────┘
      many, fast, free

Most codebases have only layer 4, run it rarely because it is expensive, and therefore have no signal at all while developing. Invert that.

Layer 0 — the seam that makes everything else possible#

Before any of this works you need one narrow interface between your code and the provider. Everything above it becomes testable without a network.

/**
 * @typedef {{ system: string, messages: Message[], signal?: AbortSignal }} Request
 * @typedef {{ complete(req: Request): Promise<string>,
 *             stream(req: Request): AsyncIterable<string> }} LLM
 */

/** @returns {LLM} */
export function makeClient({ apiKey, model, fetchImpl = fetch }) { /* … */ }

Two methods. Everything in your application depends on that shape, not on an SDK. Swapping in a fake is then one line, and it does not break when the SDK's internals change.

Layer 1 — most of your code is not the model call#

Prompt assembly, chunking, retrieval ranking, routing, token accounting, output post-processing. All ordinary JavaScript, all testable in milliseconds.

import { expect, test } from "vitest";
import { buildPrompt } from "./prompt.js";

test("includes only the top k chunks", () => {
  const p = buildPrompt({ query: "q", chunks: makeChunks(20), k: 5 });
  expect(p.match(/<chunk>/g)).toHaveLength(5);
});

test("truncates a chunk that would blow the budget", () => {
  const p = buildPrompt({ query: "q", chunks: [huge()], k: 1, maxChars: 1000 });
  expect(p.length).toBeLessThanOrEqual(1000);
});

Unglamorous, and it catches a surprising share of real bugs.

Layer 2 — contract tests against a fake#

Test what your code does with a response, not what the model says. Every case below is a real production failure:

import { describe, expect, test } from "vitest";
import { parseCategory } from "./parse.js";

describe("parseCategory", () => {
  test.each([
    ['{"category":"billing"}',                "billing"],
    ['```json\n{"category":"billing"}\n```',  "billing"],   // models fence constantly
    ['{"category":"Billing"}',                "billing"],   // wrong case
    ['Sure! Here you go:\n{"category":"billing"}', "billing"],
    ['{"categorie":"billing"}',               null],        // misspelled key
    ['{"category":"refunds_and_returns"}',    null],        // not in the enum
    ["I think this is a billing issue.",      null],        // ignored the format
    ["",                                      null],
    ['{"category":"billing"',                 null],        // truncated at max tokens
  ])("handles %j", (raw, expected) => {
    expect(parseCategory(raw)).toBe(expected);
  });
});

The contract is never throw, never invent. A parser that returns a plausible wrong answer on malformed input is worse than one returning null.

The tests JavaScript needs that Python does not#

Streaming is where the JavaScript-specific bugs live. Fake the stream and test the plumbing:

async function* fakeStream(chunks, { failAt = -1 } = {}) {
  for (const [i, c] of chunks.entries()) {
    if (i === failAt) throw new Error("connection reset");
    yield c;
  }
}

test("assembles chunks that split a multi-byte character", async () => {
  const out = await collect(fakeStream(["hello ", "wor", "ld 👋"]));
  expect(out).toBe("hello world 👋");
});

test("a mid-stream failure surfaces, and does not leave a half-written record", async () => {
  const db = makeFakeDb();
  await expect(persistAnswer(db, fakeStream(["a", "b"], { failAt: 1 })))
    .rejects.toThrow("connection reset");
  expect(db.rows).toHaveLength(0);      // no partial write
});

test("aborting stops consumption promptly", async () => {
  const ac = new AbortController();
  const p = streamAnswer({ signal: ac.signal });
  ac.abort();
  await expect(p).rejects.toMatchObject({ name: "AbortError" });
});

test("a client disconnect aborts the upstream call", async () => {
  const upstream = new AbortController();
  const spy = vi.fn();
  upstream.signal.addEventListener("abort", spy);
  simulateClientDisconnect();
  expect(spy).toHaveBeenCalled();       // otherwise you keep paying for tokens
});

That last one is the expensive bug. A user closes the tab, your server keeps streaming from the provider, and you are billed for output nobody will read. It is invisible until the bill arrives, and it is one test.

Also worth covering here: retry with backoff on 429, timeout behaviour, and what happens on a 500 halfway through a stream. None of it needs a real model.

Layer 3 — cassettes in CI#

Record real responses once, replay them forever. Real response shapes, zero cost, zero flakiness.

Because everything goes through fetch, you can intercept at that level rather than at the SDK level — which means the cassette keeps working when you upgrade the SDK.

import { beforeAll, afterAll } from "vitest";
import { setupServer } from "msw/node";
import { http, HttpResponse } from "msw";
import { readFileSync } from "node:fs";

const cassette = JSON.parse(readFileSync("test/cassettes/classify.json", "utf8"));

export const server = setupServer(
  http.post("https://api.example.com/v1/messages", async ({ request }) => {
    const body = await request.json();
    const hit = cassette.find((c) => c.request.messages.at(-1).content === body.messages.at(-1).content);
    if (!hit) throw new Error(`No cassette entry. Re-record: npm run record`);
    return HttpResponse.json(hit.response);
  }),
);

beforeAll(() => server.listen({ onUnhandledRequest: "error" }));
afterAll(() => server.close());

Two rules that stop this rotting:

1. Scrub credentials before committing. Read the first cassette you commit, in full.
2. Re-record on a schedule, monthly or so. A cassette from eighteen months ago tests a model that no longer exists, and the diff when you re-record is genuinely informative.

Layer 4 — evals with a scored dataset#

Accept non-determinism and measure it instead of asserting on it.

{"input":"my card was charged twice","expect":"billing"}
{"input":"how do I export my data?","expect":"support"}
{"input":"cancel and refund please","expect":"refunds"}

import { readFileSync } from "node:fs";
import { classify } from "../src/classify.js";

const cases = readFileSync("evals/dataset.jsonl", "utf8")
  .trim().split("\n").map((l) => JSON.parse(l));

const results = await Promise.all(cases.map((c) => classify(c.input)));
const hits = results.map((r, i) => r.category === cases[i].expect);
const accuracy = hits.filter(Boolean).length / hits.length;

console.log(`accuracy ${(accuracy * 100).toFixed(1)}% on ${cases.length} cases`);
for (const [i, ok] of hits.entries()) {
  if (!ok) console.log(`  MISS ${JSON.stringify(cases[i].input)}: got ${results[i].category}, want ${cases[i].expect}`);
}
process.exit(accuracy >= 0.9 ? 0 : 1);

Nightly, not per-commit. A threshold, not an assertion. Read the misses — the list is worth more than the number, because it is where the next prompt change comes from.

When the output is free text#

No equality to assert. Three approaches, in decreasing order of how much you should trust them:

Assert on properties. Deterministic, cheap, and catches most real regressions.

test("summary does not invent numbers", () => {
  const nums = (s) => new Set(s.match(/\d[\d,.]*/g) ?? []);
  expect([...nums(summary)].every((n) => nums(article).has(n))).toBe(true);
});

That one test catches fabricated figures, which is the failure that matters most in summarisation.

Assert on stability across runs. Same input twice at temperature 0 should be similar. Large divergence is a signal even without ground truth.

LLM-as-judge, carefully. A second model scores against a rubric. It works and has known biases — position, verbosity, self-preference. Use it for relative comparisons (is B better than A?) rather than absolute scores, and calibrate against fifty hand-labelled examples before you let a number gate anything.

Cost control in CI#

on: [push, pull_request]
jobs:
  fast:
    steps:
      - run: pnpm vitest run          # layers 1-3. free, every push.
  evals:
    if: github.event_name == 'schedule'
    steps:
      - run: node evals/run.js        # layer 4. nightly.

Set a hard spend cap on the CI key, not just an alert. Use the cheapest model that discriminates for any judge.

Common questions#

Should I mock the SDK or fetch?#

fetch, or your own narrow client interface. Mocking the SDK's internals ties your tests to a library version and tests nothing you care about — and it breaks on every upgrade. Intercepting at the HTTP layer survives SDK changes and exercises your real request-building code.

How do I test streaming without a real API?#

An async generator is a stream. Everything in layer 2 above uses one, and it lets you simulate the cases a real API rarely gives you on demand: a mid-stream failure, a chunk boundary that splits a multi-byte character, an abort.

Temperature 0 makes it deterministic, so can I assert equality?#

No. Temperature 0 is greedy sampling, not determinism — batching, hardware and provider-side changes all move the output, and a model version change moves it substantially. Assert on properties, or use cassettes.

What does running all this cost?#

Layers 1-3 are free — no tokens leave the process. Layer 4 is the billable one, and the abandoned-stream problem makes Node worse than most at wasting tokens; both are covered in tracking and cutting token costs in Node.

Is msw necessary, or can I just stub global fetch?#

Stubbing globalThis.fetch works fine and has no dependency. msw is worth it once you have several endpoints or want the same mocks in browser tests. The important part is intercepting at the HTTP boundary either way.