# Tracking and cutting token costs in Node

> Source: https://learn-javascript.org/ai/tokenomics/
> Part of Learn JavaScript, free to read.

The economics are language-independent — [what tokens cost and where the money goes](https://codelearningdojo.com/token-economics/) is the model. This page is the Node implementation, and it leads with the failure that is specific to this ecosystem.

## The abandoned stream

Node is where LLM output is streamed to a browser, and streaming introduces a cost bug that does not exist in a request/response system: **the consumer can disappear while the producer keeps going.**

A user closes the tab. Your handler's response socket dies. If nothing propagates that upstream, your server keeps pulling tokens from the provider until the model stops talking — and you are billed for every one of them, for an answer that reached nobody.

At any real traffic level this is a measurable share of the bill, and it is invisible: no error, no log line, no alert.

```js src/routes/chat.js
export async function chat(req, res) {
  const ac = new AbortController();

  // the whole fix: when the client goes, cancel upstream
  res.on("close", () => { if (!res.writableEnded) ac.abort(); });

  let usage = null;
  try {
    const stream = await llm.stream({ messages: req.body.messages, signal: ac.signal });
    for await (const event of stream) {
      if (event.type === "delta") res.write(event.text);
      if (event.type === "usage") usage = event.usage;
    }
    res.end();
  } catch (err) {
    if (err.name !== "AbortError") throw err;
  } finally {
    record({ feature: "chat", usage, aborted: ac.signal.aborted });
  }
}
```

Three details:

- **`res.on("close")`** fires on client disconnect *and* on normal completion, hence the `writableEnded` guard.
- **`finally`** records usage even on abort. You still pay for the input tokens and whatever output was generated before the cancel, so an unrecorded abort understates your spend.
- **Track the abort rate.** If it climbs, either your answers are too slow or too long. Both cost money.

```js
// the metric worth having on a dashboard
record({ feature, usage, aborted });   // then: aborted_calls / total_calls
```

:::warn Test this, because it fails silently
There is a test for it in [testing JavaScript that calls a language model](/ai/evals/): assert that aborting the client's request fires abort on the upstream controller. Without that test you will not find out until you read a bill.
:::

## Attribution with AsyncLocalStorage

Node's answer to "tag every call without threading a context object through fifteen functions".

```js src/llm/context.js
import { AsyncLocalStorage } from "node:async_hooks";

export const als = new AsyncLocalStorage();

export function withRequestContext(handler) {
  return (req, res, next) => {
    als.run({ requestId: req.id, tenant: req.user?.tenantId, route: req.path }, () =>
      handler(req, res, next));
  };
}
```

```js src/llm/record.js
import { appendFile } from "node:fs/promises";
import { als } from "./context.js";

// per-1M-token prices. Config, not code — they change.
const PRICES = {
  small: { in: 0.25, cachedIn: 0.03, out: 1.25 },
  large: { in: 3.0,  cachedIn: 0.3,  out: 15.0 },
};

export function costOf(model, u) {
  const p = PRICES[model] ?? PRICES.large;
  const fresh = Math.max((u.input_tokens ?? 0) - (u.cache_read_input_tokens ?? 0), 0);
  return (fresh * p.in
        + (u.cache_read_input_tokens ?? 0) * p.cachedIn
        + (u.output_tokens ?? 0) * p.out) / 1e6;
}

export async function record({ feature, model = "large", usage = {}, ok = true, aborted = false, ms = 0 }) {
  const ctx = als.getStore() ?? {};
  const line = {
    ts: Date.now(), feature, model, ok, aborted, ms,
    inputTokens: usage.input_tokens ?? 0,
    cachedTokens: usage.cache_read_input_tokens ?? 0,
    outputTokens: usage.output_tokens ?? 0,
    usd: costOf(model, usage),
    version: process.env.GIT_SHA ?? "dev",
    ...ctx,
  };
  await appendFile(process.env.LLM_LOG ?? "llm-spend.jsonl", JSON.stringify(line) + "\n");
}
```

Now every call inside a request is tagged with tenant and route, and nothing in your business logic knows about it.

:::warn Floating-point money
`costOf` returns a float, which is fine for a dashboard and wrong for an invoice. If these numbers ever reach a customer's bill, accumulate in integer micro-dollars — see [the failure-mode catalogue](/review/failure-modes/) on why `0.1 + 0.2` is a problem you do not want in a billing path.
:::

## Make the cache hit

The largest saving available, and in JavaScript the usual killer is object property order.

```js
// BAD — a fresh timestamp at the top invalidates everything after it
const messages = [
  { role: "system", content: `Today is ${new Date().toISOString()}\n${RULES}` },
  { role: "user", content: docs.join("\n") + question },
];

// GOOD — stable prefix, volatile tail
const messages = [
  { role: "system", content: RULES, cache_control: { type: "ephemeral" } },
  { role: "user", content: [...docs].sort().join("\n"), cache_control: { type: "ephemeral" } },
  { role: "user", content: `Today is ${new Date().toISOString()}\n${question}` },
];
```

`[...docs].sort()` matters. If your documents come from a `Set`, an `Object.values()`, or a query with no `ORDER BY`, the order can vary between calls — and a different order is a different prefix, which is a total cache miss on that block. It fails silently and shows up only as a bill.

The same applies to tool definitions assembled from an object. Serialise them in a fixed order, because `JSON.stringify` follows insertion order and insertion order is easy to change by accident.

```js
// a test that catches the day someone adds a uuid to the system prompt
test("prompt prefix is stable across requests", () => {
  const a = buildMessages("q1", ["b", "a"]);
  const b = buildMessages("q2", ["a", "b"]);
  expect(a.slice(0, 2)).toEqual(b.slice(0, 2));
});
```

## Bound the fan-out

`Promise.all` over an array has no concurrency limit. A thousand rows means a thousand simultaneous requests, an immediate 429 storm, retries on top, and a budget check that fires after you have already committed to everything.

```js
async function mapLimit(items, limit, fn) {
  const out = new Array(items.length);
  let i = 0;
  await Promise.all(Array.from({ length: Math.min(limit, items.length) }, async () => {
    while (i < items.length) {
      const idx = i++;
      out[idx] = await fn(items[idx], idx);
    }
  }));
  return out;
}

const results = await mapLimit(rows, 8, classify);
```

Check the budget *inside* `fn`, not before the loop, so it can stop the run partway.

## Retries multiply

```js
const RETRYABLE = new Set([408, 409, 429, 500, 502, 503, 504]);

export async function withRetry(fn, { attempts = 3, base = 500 } = {}) {
  for (let n = 1; ; n++) {
    try {
      return await fn();
    } catch (err) {
      const status = err.status ?? err.response?.status;
      if (n >= attempts || !RETRYABLE.has(status)) throw err;
      const jitter = Math.random() * base;
      await new Promise((r) => setTimeout(r, base * 2 ** (n - 1) + jitter));
    }
  }
}
```

Never retry a 400 — the request is malformed and will be malformed identically. Always jitter, or every worker in your fleet retries in lockstep and you turn a blip into an outage you also paid for.

## Cap the loop

An agent loop resends the whole conversation each turn, so cost grows roughly quadratically with turn count. A ten-turn loop over 20k tokens of context is closer to 200k tokens than 20k.

```js
export async function runAgent(messages, { maxTurns = 12, budgetUsd = 0.5 } = {}) {
  let spent = 0;
  for (let turn = 0; turn < maxTurns; turn++) {
    const { text, toolCalls, usage } = await llm.complete({ messages });
    spent += costOf("large", usage);
    if (spent > budgetUsd) throw new Error(`budget exceeded after ${turn + 1} turns`);
    if (!toolCalls?.length) return text;
    messages = [...messages, ...(await runTools(toolCalls))];
  }
  throw new Error(`hit turn limit (${maxTurns})`);
}
```

A turn cap is the single cheapest protection against a runaway loop, and it is the one people add after the incident.

## The report

```js scripts/spend-report.js
import { readFileSync } from "node:fs";

const rows = readFileSync("llm-spend.jsonl", "utf8").trim().split("\n").map(JSON.parse);

const by = (key) => Object.entries(
  rows.reduce((acc, r) => {
    const k = r[key] ?? "unknown";
    acc[k] ??= { calls: 0, usd: 0, ok: 0, aborted: 0, out: 0 };
    acc[k].calls++; acc[k].usd += r.usd; acc[k].ok += r.ok ? 1 : 0;
    acc[k].aborted += r.aborted ? 1 : 0; acc[k].out += r.outputTokens;
    return acc;
  }, {}),
).sort((a, b) => b[1].usd - a[1].usd);

console.table(Object.fromEntries(by("feature").map(([k, v]) => [k, {
  calls: v.calls,
  usd: v.usd.toFixed(4),
  usdPerSuccess: (v.usd / Math.max(v.ok, 1)).toFixed(5),   // the number that matters
  abortRate: (v.aborted / v.calls).toFixed(3),             // the Node-specific one
}])));

const cacheRate = rows.reduce((s, r) => s + r.cachedTokens, 0)
                / Math.max(rows.reduce((s, r) => s + r.inputTokens, 0), 1);
console.log("cache hit rate:", cacheRate.toFixed(3));
```

`console.table` is underused and is exactly right here.

:::verdict The four that usually do it in Node
1. **Cancel upstream on client disconnect.** The Node-specific one, and it is free money.
2. **Reorder prompts so the cache hits.** Largest single saving, changes no behaviour.
3. **Cap turns and bound fan-out concurrency.** Protects the worst case.
4. **Route classification to a small model, and set `max_tokens`.** Output is the expensive side.
:::

## Common questions

### How do I count tokens before sending, in Node?

There are JavaScript ports of the common tokenisers, and they are fine for a budget check. Treat any local count as an estimate — provider tokenisers differ and none of them account for tool schemas or system scaffolding. Decide with the estimate, account with the `usage` the API returns.

### Does the abandoned-stream problem apply behind a proxy?

Yes, and it can be worse: a proxy may hold the connection open after the browser is gone, so your handler never sees a close. Add a server-side timeout as well as the disconnect handler, and check that your proxy propagates client aborts.

### Should I record spend to a file or a database?

A file to start with — JSONL, append-only, no schema migration, and the report above reads it in ten lines. Move it to your metrics backend when the volume justifies it. What matters from day one is *what* you tag, not where it lands, because tags are nearly impossible to backfill.

### Is Node a bad choice for a high-volume LLM gateway?

No, but the failure modes are different: streaming lifecycle bugs and unbounded concurrency rather than CPU. If your gateway is doing heavy fan-out with strict budget enforcement, [Go](https://learn-go.org/ai/tokenomics/) makes the cancellation and concurrency parts easier to get right.