standard-schema-faker: one session from plan file to npm

2026-07-19by

#ai #llm #typescript #testing #open-source #tooling #workflow

The other evening, I opened a plan file. By midday the following day, standard-schema-faker was live on npm at 0.2.0: a fake-data generator that works with any Standard Schema validator — Zod v4, Valibot, ArkType, best-effort Effect — with seeded determinism, full type inference, and 533 tests behind CI and an automated release pipeline.

The library is worth writing about on its own. But the more transferable story is how it got built: one orchestrator model doing planning, research, and audits; a cheaper model doing nearly every line of implementation through ~13 background agent runs; and me steering with one-sentence nudges. That workflow produced real design wins, one genuinely novel failure mode, and about 0.9M wasted tokens that taught me more than the 4.2M productive ones.

What shipped

One unscoped package, three entry points:

// zero-dependency core: structurally valid, no realism opinions
import { fake, fakeMany } from 'standard-schema-faker'
 
// batteries included: realistic values via @faker-js/faker
import { fake, fakeMany } from 'standard-schema-faker/faker'
 
// same idea, chance.js backend
import { fake, fakeMany } from 'standard-schema-faker/chance'

import { fake, fakeMany } from 'standard-schema-faker/faker'
import { z } from 'zod'
 
const User = z.object({
  id: z.uuid(),
  email: z.email(),
  age: z.int().min(18).max(99),
  tags: z.array(z.string()).max(3),
})
 
const user = fake(User)                        // typed as z.infer<typeof User>, not unknown
const same = fake(User, { seed: 42 })          // same seed, same value — across processes and days
const many = fakeMany(User, 100, { seed: 42 }) // deterministic batch

The architecture rests on one idea: JSON Schema is the intermediate representation; never introspect vendor runtimes. Zod v4 and ArkType natively emit Standard JSON Schema; Valibot and Effect go through a community fallback converter. One walker consumes the IR, so there are no per-vendor adapters to maintain — a validator the library has never heard of works automatically if it emits the spec.

Everything else follows from treating determinism as the product's spine rather than a feature checkbox. Backends are factories (create(seed) → instance) so two generators can never perturb each other's random streams. Dates generate relative to a pinned reference date instead of Date.now(), because faker's relative-date methods default their reference to the current time (faker#1870) — meaning "seeded" dates from tools built on it silently change across days. Optional-property inclusion, union-branch picks, null probability: every decision is one seeded draw.

Verify the load-bearing claim first

The whole design hangs on one factual claim: that the Standard JSON Schema spec (~standard.jsonSchema with input/output projection) actually exists and validators actually implement it. Before any code, the orchestrator web-verified it — the spec is real, Zod v4 and ArkType ship it natively.

That sounds trivially cheap, and it was. But I've watched projects burn days building on a half-remembered API surface. One search on day one de-risked the entire architecture, and the plan got sharper in the same pass: the @standard-schema/* npm scope belongs to the spec org (not ours to squat), Effect got demoted to best-effort so it couldn't block launch, and the mutable seed() method in my original plan got replaced with the factory design before it could calcify.

The heuristics arc: a rejection that became the headline feature

The best design story of the session took four generations, each driven by a one-line human nudge.

Generation 1: rejected. I wanted property-name sniffing — firstName generates a person name, email generates an email. The orchestrator declined: it would be "the first vibes-driven behavior in a spec-driven library," and the ambiguity is real (is name a person or a product?).

Generation 2: reframed. I wanted it anyway, and pushing back surfaced that the objection was about form, not the feature. Opaque, unremovable guessing was the problem. The redesign: rules as data — an exported, inspectable, filterable array where every rule has an addressable name you can remove or override. Every incumbent I checked (zod-mock's mockeryMapper among them) hardcodes this map where users can't see it.

Generation 3: context objects. "Match function with a context object is the right direction." Matchers became (ctx: MatchContext) => boolean with the key, semantic path, node, parent, and ancestors — string globs and RegExps stayed as sugar that compiles to the same predicate.

Generation 4: FHIR made it real. My day job is healthcare, and FHIR supplied the forcing cases. telecom[x].value is a phone number or an email depending on the generated value of the sibling system field — so ctx.siblings (values generated so far) was added. That only works if system generates before value, which produced a clean rule: enum and const properties generate first. Discriminators cost nothing to hoist, and generation order becomes a pure function of the schema, so sibling-dependent rules work regardless of declaration order. Then I caught a false-positive risk in my own suggestion: FHIR reuses system everywhere (Coding.system and Identifier.system are URIs, not contact channels) — the real discriminator is the ancestor field name telecom. Rules gained ancestor-name gates, and a signal-strength tiering fell out naturally: sibling-value rules beat ancestor-name rules beat bare-key rules.

Each nudge was a sentence. Each reshaped the design. The pattern I keep relearning: when a reviewer (human or model) rejects a feature, the rejection is often about the form it arrived in, and the redesign that answers the objection is better than either starting position.

What actually found the bugs

Ten-plus real bugs got caught before publish, and almost none by reading code. The finders, ranked by yield:

The agents' own tests, written to a briefed standard. The first implementation batch caught an infinite recursion in the depth-cap fallback and a length window that truncated formatted strings (emails losing their TLD) — before any human saw the code.

Property tests with an independent oracle. fast-check generation validated against Ajv (not against our own walker) found a genuinely unsatisfiable-allOf class that reproduces even with Zod's own z.intersection() output. @arethetypeswrong/cli caught a real "masquerading as ESM" exports bug that would have shipped broken types to CJS consumers.

Competitor issue trackers as free QA. A deliberate research pass over zod-mock, json-schema-faker, and faker issues surfaced problem classes to defend against — and one live bug in our own already-written code: the Date.now() reference-date issue above, which meant our "seeded" dates weren't reproducible across days. Also from that pass: pattern×length re-rolling (json-schema-faker's most-reported bug class is emitting invalid values or cropping strings until they fit), z.record support, and clear errors for Map/Set instead of silent garbage.

A "don't hold back" audit. After the heuristics work I asked the orchestrator to read every file hunting for invented patterns where a standard existed. It found five bugs (type: ["null","string"] always generated null; heuristic globs weren't case-normalized, so **.phoneNumber.value could never match; nullable: true only worked on object properties; two more) and a list of ecosystem alignments: full type inference — fake(User) returning the inferred type is arguably the whole point of building on Standard Schema, and it was missing — plus renaming our use option to io (Zod v4's own name for the concept) and renaming a Faker interface that collided with @faker-js/faker's class.

The common thread: probes beat review. Every one of these is a mechanism that executes an assumption rather than eyeballing it.

The first real consumer found what 533 tests couldn't

Hours after 0.2.0 hit npm, I pointed the first real consumer at it: an in-browser Standard Schema playground with a fake-data panel. One screenshot of a generated e-commerce order exposed four bug classes the entire suite — Ajv property oracle included — had sailed past:

Degenerate UUIDs. orderId: "ffffffff-ffff-ffff-ffff-ffffffffffff". Zod's uuid regex explicitly lists the nil and max UUIDs as alternation branches, the library prioritized pattern over format, and uniform branch selection picked a degenerate literal for roughly two-thirds of seeds.
Year-1108 datetimes. placedAt: "1108-11-09T01:15Z". Same root cause: z.iso.datetime() emits format: "date-time" plus a validating regex, and the regex generator — not the date generator — was producing the value. The fix for both: when a schema carries format and pattern, generate from the format and keep the value only if it satisfies the pattern (checked with the native regex engine).
16-digit quantities. quantity: 1798501541848093. Zod stamps every z.int() with minimum/maximum ±(2^53 − 1) as an "any safe integer" sentinel; honoring those bounds literally meant a uniform draw over the entire safe-integer range — and, worse, a half-bounded z.int().min(200) collapsed onto the default 0–100 window as the constant 200. Sentinel-magnitude bounds are now treated as absent, with a 100-wide window anchored at whichever bound is real.
Lorem cities. city: "cogo". The big one: the heuristics engine tests regex matchers against the full dotted path (shipping.city), and every one of the 80 default rules was written /^city$/ — so the realistic-field heuristics, the library's headline feature, had never once fired for a nested field. Every heuristic test used top-level properties: the tests mirrored the implementation's blind spot exactly.

Fixing the nested-heuristics bug immediately exposed a fifth: the newly-firing sku rule generated "ZVH-10669" for a /^SKU-\d{4}$/ schema, because the heuristic constraint guard checked length and numeric bounds but never pattern. Bugs revealing bugs is what progress looks like here — each fix widened the exercised surface.

All five shipped as 0.3.0 the same day, with regression tests. But the meta-lesson matters more than the fixes. The property oracle asserted validity — generate against arbitrary schemas, check the output parses — and every one of these buggy values was perfectly valid. Realism has no oracle; the nil UUID passes z.uuid(). What caught them was a human looking at one rendered order and reacting to it the way any user would: "these datetimes seem unrealistic." A consumer stresses the axes your tests were never written along — same seed, different question.

The multi-agent workflow, honestly

Division of labor: the orchestrator (a frontier model) did plan verification, research, design decisions, written briefs, audits, and independent verification of every batch. The implementer (a cheaper model) wrote every line of production code via background agents working from those briefs. Rough subagent spend, from task telemetry:

Batch	Output tokens
Core walker + seeded RNG + Zod proof	150k
Vendor matrix	246k
Faker adapter, strict mode, overrides	399k
Polish + publish-readiness	630k
Heuristics ctx redesign	567k
Audit + ecosystem defenses	919k
Remaining feature/cleanup batches	~1.3M
Wasted (one killed mid-run, one refused)	~0.9M
Total	≈5.1M

What made cheap-model implementation work wasn't the briefs being long — it was the briefs specifying invariants, not just tasks: "one seeded draw per decision," "never truncate a formatted value," "an override returning undefined declines and falls through." Every batch ended with the same gate (clean-room install, build, typecheck, lint, tests, plus attw and tarball smoke tests for packaging changes), and the orchestrator re-ran verification independently rather than trusting the report. That trust-but-verify pass caught nothing catastrophic — but it did catch an agent leaving its test files uncommitted, which would have been a confusing archaeology problem a week later.

One briefing rule earned its keep twice: agents were told to disagree in the report rather than silently comply. That produced honest deliveries like "this re-roll strategy is genuinely best-effort; I documented the limitation instead of overclaiming" — exactly the failure you can't detect from outside if the agent papers over it.

The failure mode worth blogging: injection paranoia

Background agents receive follow-up instructions through a channel that renders like a system notification. Three separate agents refused legitimate follow-ups on prompt-injection suspicion, with reasoning like: "this message retroactively authorizes something I declined, then expands scope — that's exactly the shape of an injection attack." One also reverted on-disk test-file corruption it couldn't explain (origin never determined) before continuing.

Here's the thing: the refusals were correct behavior pointed at the wrong target. An instruction that arrives mid-task, claims new authority, and expands scope genuinely is the signature of prompt injection, and a coding agent that balks at that shape is failing in the safe direction. Arguing with the agent was pointless — it can't verify I'm me. The fix was protocol, not persuasion: never push scope into a running agent; launch a fresh agent whose original brief contains the whole job. Related discovery: agent transcripts expire, so "resume the agent that has the context" isn't a durable strategy either. Self-contained briefs are the unit of work that survives both paranoia and expiry.

I expect this class of problem to grow. As injection defenses in agent harnesses get stronger, the line between "hostile instruction smuggled into my context" and "my operator changed their mind" is exactly the line the defense can't see. Orchestration layers that treat briefs as immutable contracts sidestep the ambiguity entirely.

Decisions worth recording so they don't get relitigated

Why not wrap json-schema-faker? I asked this directly — it advertises full JSON Schema support, and we were re-implementing a walker. The answer, recorded in the repo so future-me doesn't reopen it: JSF configures through module-global state, which is structurally incompatible with per-call seeded-instance determinism; wrapping it inherits exactly the bug classes we advertise defending against; our differentiators (context heuristics, finalize hooks, strict retry, io projection, inference) need per-node walk context its extension surface doesn't expose; and it's still 0.x after roughly a decade. The coverage asymmetry is fine because our IR is vendor-emitted JSON Schema — a narrow, well-behaved subset, not the whole spec. We took inspiration instead: a formats registry modeled on jsf.format(), and probability toggles for default/examples values.

Why one package instead of an npm org? I imposed "no npm org" as a constraint, and it produced a better design than my original three-package plan: one unscoped package with subpath exports (/faker, /chance) and realism engines as optional peer deps — the pattern drizzle-orm, hono, and zod already normalized. The root entry stays zero-dependency.

Why the benchmark documents where we lose. BENCH.md shows the zero-dep backend at roughly 2× zod-mock's throughput and the faker backend at roughly 0.7× — the cost of walking a generic JSON Schema document versus a Zod-only tool walking Zod's internals directly. Publishing the losing number with the reason felt more useful than curating the winning one; it's also the number a potential adopter will find themselves anyway.

One more end-of-session step I'd argue AI-heavy projects tend to skip: the history was squashed, PLAN.md deleted, and every process-narrative comment ("milestone 3," "BUG FIX," "per the audit") rewritten into standalone present-tense invariants. Comments that reference a session only the author lived through are dead weight the moment the session ends; comments that state the invariant survive.

Takeaways

Verify the load-bearing claim before building on it. One search confirming the Standard JSON Schema spec existed de-risked the architecture on day one.
Probes beat review. Property tests against an independent oracle, attw, and runtime checks of assumed behavior each found real bugs that reading the code didn't.
Competitor issue trackers are free QA. Mining them found a live reproducibility bug in our own code plus three defense features.
Rejections are often about form, not the feature. "No name sniffing" became the library's most distinctive feature once redesigned as inspectable rules-as-data.
Short, domain-informed nudges carry outsized weight. "Enum values first," "telecom is the discriminator, not system" — each one sentence, each load-bearing. The FHIR expertise mattered more than any prompt-engineering technique.
Multi-agent orchestration needs injection-aware protocol design. Complete briefs to fresh agents beat incremental instructions to running ones — and the agents that refused mid-stream scope changes were right to be suspicious, just wrong about the suspect.
Pick a spine and let it decide things. Nearly every hard call here traced back to determinism: factory backends, pinned reference dates, enum-first ordering, refusing to wrap a global-state library. A project with a spine argues with itself less.
Ship a consumer alongside the library. The playground found five bugs in hours that 533 tests and an Ajv oracle missed — because the oracle checked validity and the bugs were about realism, an axis with no oracle. If your library's output is meant to be looked at, something has to look at it.