AI-Native Updated 2026-09 11 min read View as Markdown

Tracking and cutting token costs in Node

Node's specific cost problem is streaming: the user leaves, the stream keeps running, and you keep paying. That plus three other changes usually halves the bill.

The economics are language-independent — what tokens cost and where the money goes is the model. This page is the Node implementation, and it leads with the failure that is specific to this ecosystem.

The abandoned stream#

Node is where LLM output is streamed to a browser, and streaming introduces a cost bug that does not exist in a request/response system: the consumer can disappear while the producer keeps going.

A user closes the tab. Your handler's response socket dies. If nothing propagates that upstream, your server keeps pulling tokens from the provider until the model stops talking — and you are billed for every one of them, for an answer that reached nobody.

At any real traffic level this is a measurable share of the bill, and it is invisible: no error, no log line, no alert.

src/routes/chat.js

export async function chat(req, res) {
  const ac = new AbortController();

  // the whole fix: when the client goes, cancel upstream
  res.on("close", () => { if (!res.writableEnded) ac.abort(); });

  let usage = null;
  try {
    const stream = await llm.stream({ messages: req.body.messages, signal: ac.signal });
    for await (const event of stream) {
      if (event.type === "delta") res.write(event.text);
      if (event.type === "usage") usage = event.usage;
    }
    res.end();
  } catch (err) {
    if (err.name !== "AbortError") throw err;
  } finally {
    record({ feature: "chat", usage, aborted: ac.signal.aborted });
  }
}

Three details:

res.on("close") fires on client disconnect and on normal completion, hence the writableEnded guard.
finally records usage even on abort. You still pay for the input tokens and whatever output was generated before the cancel, so an unrecorded abort understates your spend.
Track the abort rate. If it climbs, either your answers are too slow or too long. Both cost money.

// the metric worth having on a dashboard
record({ feature, usage, aborted });   // then: aborted_calls / total_calls

Attribution with AsyncLocalStorage#

Node's answer to "tag every call without threading a context object through fifteen functions".

src/llm/context.js

import { AsyncLocalStorage } from "node:async_hooks";

export const als = new AsyncLocalStorage();

export function withRequestContext(handler) {
  return (req, res, next) => {
    als.run({ requestId: req.id, tenant: req.user?.tenantId, route: req.path }, () =>
      handler(req, res, next));
  };
}

src/llm/record.js

import { appendFile } from "node:fs/promises";
import { als } from "./context.js";

// per-1M-token prices. Config, not code — they change.
const PRICES = {
  small: { in: 0.25, cachedIn: 0.03, out: 1.25 },
  large: { in: 3.0,  cachedIn: 0.3,  out: 15.0 },
};

export function costOf(model, u) {
  const p = PRICES[model] ?? PRICES.large;
  const fresh = Math.max((u.input_tokens ?? 0) - (u.cache_read_input_tokens ?? 0), 0);
  return (fresh * p.in
        + (u.cache_read_input_tokens ?? 0) * p.cachedIn
        + (u.output_tokens ?? 0) * p.out) / 1e6;
}

export async function record({ feature, model = "large", usage = {}, ok = true, aborted = false, ms = 0 }) {
  const ctx = als.getStore() ?? {};
  const line = {
    ts: Date.now(), feature, model, ok, aborted, ms,
    inputTokens: usage.input_tokens ?? 0,
    cachedTokens: usage.cache_read_input_tokens ?? 0,
    outputTokens: usage.output_tokens ?? 0,
    usd: costOf(model, usage),
    version: process.env.GIT_SHA ?? "dev",
    ...ctx,
  };
  await appendFile(process.env.LLM_LOG ?? "llm-spend.jsonl", JSON.stringify(line) + "\n");
}

Now every call inside a request is tagged with tenant and route, and nothing in your business logic knows about it.

Make the cache hit#

The largest saving available, and in JavaScript the usual killer is object property order.

// BAD — a fresh timestamp at the top invalidates everything after it
const messages = [
  { role: "system", content: `Today is ${new Date().toISOString()}\n${RULES}` },
  { role: "user", content: docs.join("\n") + question },
];

// GOOD — stable prefix, volatile tail
const messages = [
  { role: "system", content: RULES, cache_control: { type: "ephemeral" } },
  { role: "user", content: [...docs].sort().join("\n"), cache_control: { type: "ephemeral" } },
  { role: "user", content: `Today is ${new Date().toISOString()}\n${question}` },
];

[...docs].sort() matters. If your documents come from a Set, an Object.values(), or a query with no ORDER BY, the order can vary between calls — and a different order is a different prefix, which is a total cache miss on that block. It fails silently and shows up only as a bill.

The same applies to tool definitions assembled from an object. Serialise them in a fixed order, because JSON.stringify follows insertion order and insertion order is easy to change by accident.

// a test that catches the day someone adds a uuid to the system prompt
test("prompt prefix is stable across requests", () => {
  const a = buildMessages("q1", ["b", "a"]);
  const b = buildMessages("q2", ["a", "b"]);
  expect(a.slice(0, 2)).toEqual(b.slice(0, 2));
});

Bound the fan-out#

Promise.all over an array has no concurrency limit. A thousand rows means a thousand simultaneous requests, an immediate 429 storm, retries on top, and a budget check that fires after you have already committed to everything.

async function mapLimit(items, limit, fn) {
  const out = new Array(items.length);
  let i = 0;
  await Promise.all(Array.from({ length: Math.min(limit, items.length) }, async () => {
    while (i < items.length) {
      const idx = i++;
      out[idx] = await fn(items[idx], idx);
    }
  }));
  return out;
}

const results = await mapLimit(rows, 8, classify);

Check the budget inside fn, not before the loop, so it can stop the run partway.

Retries multiply#

const RETRYABLE = new Set([408, 409, 429, 500, 502, 503, 504]);

export async function withRetry(fn, { attempts = 3, base = 500 } = {}) {
  for (let n = 1; ; n++) {
    try {
      return await fn();
    } catch (err) {
      const status = err.status ?? err.response?.status;
      if (n >= attempts || !RETRYABLE.has(status)) throw err;
      const jitter = Math.random() * base;
      await new Promise((r) => setTimeout(r, base * 2 ** (n - 1) + jitter));
    }
  }
}

Never retry a 400 — the request is malformed and will be malformed identically. Always jitter, or every worker in your fleet retries in lockstep and you turn a blip into an outage you also paid for.

Cap the loop#

An agent loop resends the whole conversation each turn, so cost grows roughly quadratically with turn count. A ten-turn loop over 20k tokens of context is closer to 200k tokens than 20k.

export async function runAgent(messages, { maxTurns = 12, budgetUsd = 0.5 } = {}) {
  let spent = 0;
  for (let turn = 0; turn < maxTurns; turn++) {
    const { text, toolCalls, usage } = await llm.complete({ messages });
    spent += costOf("large", usage);
    if (spent > budgetUsd) throw new Error(`budget exceeded after ${turn + 1} turns`);
    if (!toolCalls?.length) return text;
    messages = [...messages, ...(await runTools(toolCalls))];
  }
  throw new Error(`hit turn limit (${maxTurns})`);
}

A turn cap is the single cheapest protection against a runaway loop, and it is the one people add after the incident.

The report#

scripts/spend-report.js

import { readFileSync } from "node:fs";

const rows = readFileSync("llm-spend.jsonl", "utf8").trim().split("\n").map(JSON.parse);

const by = (key) => Object.entries(
  rows.reduce((acc, r) => {
    const k = r[key] ?? "unknown";
    acc[k] ??= { calls: 0, usd: 0, ok: 0, aborted: 0, out: 0 };
    acc[k].calls++; acc[k].usd += r.usd; acc[k].ok += r.ok ? 1 : 0;
    acc[k].aborted += r.aborted ? 1 : 0; acc[k].out += r.outputTokens;
    return acc;
  }, {}),
).sort((a, b) => b[1].usd - a[1].usd);

console.table(Object.fromEntries(by("feature").map(([k, v]) => [k, {
  calls: v.calls,
  usd: v.usd.toFixed(4),
  usdPerSuccess: (v.usd / Math.max(v.ok, 1)).toFixed(5),   // the number that matters
  abortRate: (v.aborted / v.calls).toFixed(3),             // the Node-specific one
}])));

const cacheRate = rows.reduce((s, r) => s + r.cachedTokens, 0)
                / Math.max(rows.reduce((s, r) => s + r.inputTokens, 0), 1);
console.log("cache hit rate:", cacheRate.toFixed(3));

console.table is underused and is exactly right here.

The four that usually do it in Node

Cancel upstream on client disconnect. The Node-specific one, and it is free money.
Reorder prompts so the cache hits. Largest single saving, changes no behaviour.
Cap turns and bound fan-out concurrency. Protects the worst case.
Route classification to a small model, and set max_tokens. Output is the expensive side.

Common questions#

How do I count tokens before sending, in Node?#

There are JavaScript ports of the common tokenisers, and they are fine for a budget check. Treat any local count as an estimate — provider tokenisers differ and none of them account for tool schemas or system scaffolding. Decide with the estimate, account with the usage the API returns.

Does the abandoned-stream problem apply behind a proxy?#

Yes, and it can be worse: a proxy may hold the connection open after the browser is gone, so your handler never sees a close. Add a server-side timeout as well as the disconnect handler, and check that your proxy propagates client aborts.

Should I record spend to a file or a database?#

A file to start with — JSONL, append-only, no schema migration, and the report above reads it in ten lines. Move it to your metrics backend when the volume justifies it. What matters from day one is what you tag, not where it lands, because tags are nearly impossible to backfill.

Is Node a bad choice for a high-volume LLM gateway?#

No, but the failure modes are different: streaming lifecycle bugs and unbounded concurrency rather than CPU. If your gateway is doing heavy fan-out with strict budget enforcement, Go makes the cancellation and concurrency parts easier to get right.

Get the JavaScript agent pack

A battle-tested AGENTS.md, the review checklist, and the failure-mode cheat sheet for JavaScript. One email, then occasional updates when the tooling shifts. No course pitch.

Unsubscribe in one click. We never sell the list. Or just take the AGENTS.md now — no email needed.