# Vex — Design Document ## Purpose Intent-based data access. A caller describes *what* data they need and the *shape* they want it in. Vex synthesizes a query, executes it, and reshapes the result to match. Because it caches each generation under a fingerprint, replaying that fingerprint costs zero LLM calls. **One sentence:** ask for data in English, get it in any shape, from any database. The thesis: most applications have a small, stable set of data-access patterns. Each pattern has a consistent output shape. Once the DSL (and any result mapping) is generated for a shape, it is reused for every later request with that shape — the LLM cost amortizes to zero, and the hot path is pure compile + execute. --- ## Core principles 1. **The DSL is the source of truth.** The LLM never emits SQL. It fills a constrained JSON DSL that is Zod-validated before anything else touches it. Scope, discovery, analysis, and compilation all operate on the DSL or on a value derived from it. 2. **Deterministic core, injected intelligence.** The engine itself contains no LLM code. Natural-language→DSL and rows→shape are two optional function hooks (`generateDsl`, `mapToShape`). With them, Vex answers free-form requests. Without them, it is a deterministic DSL compiler + executor + cache that still serves any already-cached shape. Cortex (and Signal behind it) is an optional peer dependency, not a hard one; Prism is required, because a stored mapping is Prism IR and the engine executes it. 3. **Shape-only caching.** The positive cache key is a hash of the request's *shape* with all values replaced by type tokens. Two requests with the same structure but different values share a cache entry. Intent is deliberately excluded from this key. 4. **Scope is server-side and LLM-invisible.** Access control is injected into the DSL *after* generation, as ordinary filter clauses bound to server `$scope` values. The model cannot see or forge them, so prompt injection cannot bypass them. 5. **One-level field paths.** Always `entity.field`, never `entity.relation.field`. Every entity appears explicitly in `from`. This keeps the resolver simple and entity discovery a pure string walk. 6. **Postgres first, adapter interface from day one.** SQL generation is the only database-specific step. It lives entirely behind the `DatabaseAdapter` interface so other backends can follow without changing the core. 7. **Schemas teach the model.** Every DSL schema field carries a Zod `.describe()`. The JSON Schema generated from it — not prose — is what the query agent reads to learn the DSL. --- ## Architecture ``` ┌──────────────────────────────────────┐ QueryRequest │ { fingerprint?, intent?, shape?, │ │ context } │ + ExecuteOptions │ options: { scope?, entities?, locked?}│ └────────────────┬─────────────────────┘ ↓ ┌──────────────────────────────────────┐ │ Validate (Zod) + hash │ │ fingerprint (positive key) │ │ requestHash (negative key / single- │ │ flight) │ └────────────────┬─────────────────────┘ ↓ ┌──────────────────────────────────────┐ │ Cache read │ │ positive HIT → cached DSL (+ Prism IR)│ │ negative HIT → throw unsatisfiable │ │ fingerprint alone, unknown → throw │ │ cache_miss │ └────────────────┬─────────────────────┘ miss ↓ ↑ hit (skip generation) ┌──────────────────────────────────────┐ │ generateDsl hook (LLM, optional) │ │ single-flighted; negative-caches a │ │ "cannot satisfy" result │ └────────────────┬─────────────────────┘ ↓ ── deterministic pipeline (runPipeline) ─────────────────────────── ┌──────────────────────────────────────┐ │ clamp limit → discover entities → │ │ apply scope → resolve → analyze → │ │ adapter.compile → CompiledQuery │ └────────────────┬─────────────────────┘ ↓ ┌──────────────────────────────────────┐ │ Missing-context check │ │ → return empty result + meta. │ │ missingContext (valid response) │ └────────────────┬─────────────────────┘ ↓ ┌──────────────────────────────────────┐ │ Bind params (context/scope/embed) → │ │ adapter.execute → raw rows │ └────────────────┬─────────────────────┘ ↓ ┌──────────────────────────────────────┐ │ Map to shape (optional) │ │ cached Prism IR → apply; else │ │ mapToShape hook (LLM) → IR + rows │ └────────────────┬─────────────────────┘ ↓ ┌──────────────────────────────────────┐ │ Cache write (by fingerprint) │ │ DSL + Prism IR + schema fingerprint │ └────────────────┬─────────────────────┘ ↓ ┌──────────────────────────────────────┐ │ { result, meta: { cache, context, │ │ timing, warnings? } } │ └──────────────────────────────────────┘ ``` On a cache hit, generation and mapping are skipped: the pipeline re-runs deterministically against the cached DSL and applies the cached Prism IR. No LLM calls. That re-run (resolve → analyze → compile → bind → execute) is what makes scope and context values current on every request, even cached ones. --- ## Subsystems ### The DSL A constrained JSON structure (`schemas/`) that can only express safe, valid queries. Top level (`QuerySchema`): - `from` — array of entity names or `{ as, query }` subqueries (min 1). - `fields` — array of `entity.field` paths or `{ field, as }` (no `SELECT *`; optional when `aggregate` is present — a query needs at least one of the two). - `filter` — a recursive `FilterSchema` union. - `compute` — record of `alias → ComputeExpression` (arithmetic, `concat`, `coalesce`, `case`). - `aggregate` — record of `alias → AggregateExpression` (`count`/`sum`/`avg`/ `min`/`max`). - `groupBy`, `sort`, `limit`, `distinct`. Filter operators: `eq`/`neq`/`gt`/`gte`/`lt`/`lte`, `in`/`notIn`, `like`/ `ilike`, `isNull`/`isNotNull`, `and`/`or`/`not`, plus `semantic` (vector similarity) and `fuzzy` (string distance). Values are literals, `{ $context }` (caller-provided) or `{ $scope }` (server-injected). Two refinements over a naive design, both for the model's benefit: there is no `offset` (cursor pagination via a `gt` filter is safer and cacheable) and no `between` (it is sugar for `and(gte, lte)`). Fewer operators means a smaller JSON Schema and better model adherence. ### Scope Three functions, in two stages: - `discoverEntities(dsl)` walks every place an entity name can appear — `from`, field paths, filter paths (recursively through `and`/`or`/`not`), compute and aggregate references, `groupBy`, `sort`, and into subqueries — and returns the set of referenced entities. No schema needed; pure string extraction. - `checkScope(entities, policy)` looks up each discovered entity in the policy. `public` rules pass; `deny` (or no rule under a `default: 'deny'` policy) throws `VexScopeError`. It runs before resolution and changes nothing. - `scopeResolved(resolved, policy)` places each table's row rules on the resolved query, as `column = $scope` predicates: in WHERE for a driving or inner-joined table, in the join's ON for a left-joined one, and inside every subquery and `exists`. It runs after resolution, because where a rule goes depends on how its table is joined (see decision 4 under *Key design decisions*). The analyzer and the compiler never know scope exists: they process a resolved query that happens to carry extra predicates. `applyScope(dsl, entities, policy)` is deprecated. It runs `checkScope` and returns the query as it came, with no row rule added; a caller that needs rows filtered goes through the engine, or calls `scopeResolved` on the resolved query. ### Engine (deterministic pipeline) - **resolver** (`resolve`) turns the scoped DSL into a `ResolvedQuery`: field paths → table aliases + columns, FK-based join discovery between entity sources (a nullable FK resolves to a LEFT join so a null FK never drops the referencing row; subquery sources are not FK-joined — the compiler cross-joins them, and the outer query can reference their field/compute/aggregate outputs), subquery recursion. - **analyzer** (`analyze`) validates the resolved query and returns `{ warnings, errors }`. Errors halt the pipeline; warnings ride along in the response. Checks: nesting depth vs `maxNestingDepth`; cartesian products (multiple entity sources with no connecting join); unindexed filter columns (warn or reject); aggregates without `groupBy` (warn); vector field without an HNSW/IVFFlat index for semantic search (warn). - **executor** (`executeQuery`, `buildContextContract`, `findMissingContext`) binds parameters and calls `adapter.execute`. Binding resolves each `ParamSlot` from context (`context`), scope (`scope`), or by embedding text (`semantic`). `findMissingContext` reports which required keys the caller omitted so the runtime can return a valid-but-empty response instead of failing. **The limit, and whose number it is.** Every query compiles with a `LIMIT`. A query that states none is given `defaultLimit`; one that states more than `maxLimit` is clamped to it — unless its author seeded it. `maxLimit` exists for the query nobody reviewed: a generated one must not be able to ask for the table. Applied to a number a developer wrote into a seeded entry it did something else: it replaced that number with a smaller one, and the reply did not say so. So the clamp reads provenance off the stored entry: `protected` with no `requestHash` is what `seedCache` writes, and what a host writes when it stores an entry through the cache backend itself. No request can produce it (a generated entry always carries the hash of the request it was born from, and the one field a wire call can change on a stored entry is `protected`), and an entry that has it runs with its stated limit. It records who put the entry there, not whether a person read it: a host that stores a query it did not write has vouched for it. `config.capAuthored` puts the clamp back for a host that wants one ceiling over everything. What is left is the case nothing can decide for the author: a list that states no limit. Returning every row is not offered: nothing would bound the read, and an unbounded read of a large table is the failure `defaultLimit` is there to stop. So the default stays, and the engine stops being silent about it: when the `LIMIT` was its own number and exactly that many rows came back for a list, the reply's `meta.warnings` says the list may have been cut (and the console, once per named entry). No extra row is fetched to make it certain; the line says "may". ### Adapters `DatabaseAdapter` is the whole database boundary: `introspect`, `compile` (`ResolvedQuery → CompiledQuery`), `execute`, and a `capabilities` flag set. Compilation is adapter-internal — there is no shared compiler in `engine/`; each adapter owns its full SQL dialect (the Postgres adapter has its own filter/compute/aggregate SQL generation under `adapters/postgres/`). An adapter may also implement `limitReads(ms)`. The engine calls it at `introspect()` with `config.statementTimeoutMs` (0 leaves reads unbounded), and it answers where the limit is held: `'connection'` (the connection's own setting already covers it), `'statement'` (set per read) or `'unenforced'`, on which the engine warns. An adapter without it runs reads unbounded. A `CompiledQuery` is `{ sql, paramSlots, contextContract }`. The ordered `paramSlots` array drives positional binding (`$1`, `$2`, …); each slot knows its `kind` (context/scope/semantic), `type`, and (for semantic) `dimensions`. The text→vector `embed` function is an engine-level injected provider (`createQueryEngine({ embed })`), used by the executor during binding for `semantic` slots — the adapter only emits the vector-distance SQL, never embeds. The Postgres adapter advertises vector search, JSON fields, full-text, CTEs, window functions, RETURNING, and statement timeout; it coerces numeric column strings (int8/numeric/float) back to JS numbers on the way out. ### Cache — mechanics Identity is the **fingerprint** (see Cache v2 below). The supporting hashes: - **Request hash** (`computeRequestHash`) hashes intent + normalized shape + *context key names* (not values). It identifies the *request* stored with a named slot (match → hit, differ → regenerate-and-replace), and keys the negative cache and single-flight de-duplication. - **Schema fingerprint** (`computeSchemaFingerprint`) is a stable hash of the structural schema (names, types, nullability, PKs, relations, indexes — excluding volatile rowCount and cosmetic fields). It is stored on each entry; a mismatch means the cached DSL may reference changed columns, so the entry is treated as a miss and evicted. Stable across restarts so warm-up entries aren't wrongly discarded. - **Entry kinds.** A discriminated union: `ok` (DSL + optional Prism IR), `mutation` (a dev-authored write — see Mutations) and `unsatisfiable` (a TTL'd negative result — a request the agent declared impossible, so it isn't re-run until the TTL lapses or the schema changes). - **Backends.** In-memory (`createMemoryCache`), durable Postgres (`createPostgresCache`, jsonb table, server-side TTL, `protected`/ `last_used_at`/`request_hash`/`mapping` columns; the table is a strata sequence, `nisc.vex.cache`, applied once through the ledger by `init()`; validate-on-write and evict-on-read), and tiered (`createTieredCache`, L1 + L2 read-through/write-through with `full`/`lazy`/`partial` warm-up). Every backend implements the same `CacheBackend` contract; `init`/`entries` are optional capabilities used for warm-up and the discovery overview. - **Validation** (`validateEntry`) guards every write and every promotion into memory, so a corrupt or schema-drifted row can never poison the cache. It returns `null` for a valid entry and the reason, as a string, for any other. - **GC** (`sweepCache`) evicts entries idle past `maxIdleMs` (`lastUsedAt ?? createdAt`), skipping protected ones. Hygiene, not necessity. ### Cache v2 — fingerprints Agreed and implemented 2026-07-10; replaced shape-keyed caching everywhere (engine, HTTP layer, and all consumers — relay, fable, mythos, showroom). **Why v1 has to go.** Shape-hash identity collides: two different intents with the same normalized shape share one slot, so the second silently returns the first's data stamped `hit: true`. Entries are anonymous — no way to pin, promote, iterate, or roll back a generation. And running a query writes the cache as a side effect, so *testing* and *publishing* are the same physical act: callers either bypass the cache and lose the artifact, or hit it and can never change it. **The model: one identity, `fingerprint`.** - A request WITHOUT a fingerprint generates and caches; the server mints a fingerprint (`fp_…`) and returns it in `meta.cache.fingerprint`. - A minted fingerprint is an **immutable pin** — embed it and replay the exact generation forever. Changing the query means proving again and embedding the new fingerprint. (git: a commit hash.) - A caller-chosen fingerprint (any string) is a **mutable named slot** — the name is yours, its content follows your request. (git: a branch.) - Intent NEVER keys anything: natural language is fake identity in both directions. It remains generator input + human metadata. **Three request postures** — the body replaces every cache mode: ```jsonc // 1. Explore: generate, cache, mint { "intent": "top 10 open deals by value…", "shape": [{ "id": "", "value_display": "" }] } // → meta.cache: { "fingerprint": "fp_9k27ax", "hit": false } // 2. Replay: fingerprint alone (+ context VALUES welcome) — never generates { "fingerprint": "fp_9k27ax", "context": { "sortBy": "deals.value" } } // → meta.cache: { "fingerprint": "fp_9k27ax", "hit": true }; unknown fp → 404 // 3. Own a slot: fingerprint + request { "fingerprint": "deals/table", "intent": "…", "shape": [ … ] } // stored request matches → hit; differs → regenerate + REPLACE the slot // (meta.cache.replaced: true) — unless protected → 409 ``` Request equality = intent string + normalized shape + context KEY NAMES (`normalizeShape` survives for this comparison; context values are runtime data and never identity — which is what makes replays parameterizable: `sortBy`/search terms may vary per call against one cached DSL). **Lifetime = usage.** Every replay stamps `lastUsedAt`; a GC sweep evicts entries unused for N days. An embedded pin is touched on every mount and lives forever by being used; an abandoned exploration ages out. There are NO provisional/permanent tiers and NO promote API — the distinction is replaced by the convention *alive = used*. (Entries are single-digit KB; the sweep is hygiene, not necessity.) **Protection: one stored bit, two writers, one knob.** - `protected: true` on an entry means a differing request 409s instead of replacing. Written ONLY by (i) the seed path and (ii) an explicit `PATCH` (which is what "save/lock this view" calls). Runtime writes are never protected by default. - One endpoint knob: `locked: true` = replay-only posture (no generation at all — the production stance; v1's `cache: 'only'` promoted from caller flag to endpoint config). - No governance fields ever ride in the query body. **The REST surface.** Vex is an API endpoint over REST and is designed as one. Scoped mounts (`/api/deals/vex`) ride the existing `options.entities` filter — scope constrains *generation* (smaller schema, no out-of-scope joins); identity stays global. - `POST /api//vex` — execute (the three postures above). Errors: unknown fingerprint replay → 404 `cache_miss`; protected slot + differing request → 409 `fingerprint_protected`; locked endpoint asked to generate → 403 `locked`. - `GET /api//vex` — self-description: `{ protection: "all"|"some"|"none", locked, fingerprints: [{ fingerprint, protected, schemaFresh, intent, lastUsedAt }] }` plus the schema/DSL/body docs. Doubles as agent-facing discovery: "does a named query already cover this?" is one call. - `PATCH /…/vex` with `{ fingerprint, protected }` — flip the bit. `DELETE` with `{ fingerprint }` — evict (unprotected only). Fingerprints ride the body, not the path — names contain `/`. **What dies with v1:** - `CacheMode` (`use`/`refresh`/`bypass`/`only`), `ExecuteOptions.cache`, the `?cache=` query param — the request body says everything now. - The shape hash as a cache key (and with it the entire collision class). - The embed-VERBATIM requirement for generated screens: a pin has no wording to drift, and context may vary per replay — sorting and parameterized reloads become legal on cached queries again. **Unchanged:** the negative cache (request-hash keyed, TTL'd), schema- fingerprint freshness/eviction, backends and tiering, `validateEntry`. **Where it landed:** - Engine (`engine/runtime.ts`): `resolveFingerprint` — no fp → mint; fp alone → replay or `cache_miss`; fp + request → hit on matching request hash, regenerate-and-replace on mismatch (409 when protected). Fingerprint-only replays use the entry's STORED shape for the array-vs-single envelope. Replays touch `lastUsedAt`. - Request schema: `fingerprint` in; `intent`/`shape` optional. Meta: `cache: { hit, fingerprint, replaced?, intent? }`. - Backends: keys are fingerprint strings; entries carry `protected`, `lastUsedAt`, `requestHash`; `sweepCache` GC; postgres migrates in place. - HTTP: `handleQuery` (no cache param), async `handleDiscovery` overview, `handleFingerprintPatch`/`handleFingerprintDelete`; express + hono adapters. - Consumers: seeds re-keyed to protected names (`deals/list`, `todos/open`, …); every hand-authored read replays `{ fingerprint, context }`; relay's architect embeds `{ fingerprint: meta.cache.fingerprint, context }`; mythos' `/api/query` is `locked` (replay-only); showroom's demos own `vex-demo/` slots (canned = seed + replay, live = delete + regenerate). ### Mutations (write pipeline) Writes are first-class but deliberately NARROWER than reads. The asymmetry is the design: a bad generated query shows wrong data, a bad generated write destroys it — so mutations have **no generation path at all**. A mutation is a dev-authored artifact that enters through the seed path, lives in the cache as a `kind: 'mutation'` entry (the def in the same jsonb slot a query's DSL uses), and is invoked by fingerprint. The def never travels: an inline `{ mutation }` body is not a request shape, and the wire is ONE shape for both kinds — `POST { fingerprint, context }`, dispatched by the entry's kind. Replay-only holds under every posture (`locked` only ever governed query generation); the query engine refuses a mutation fingerprint explicitly. The pipeline: validate (closed grammar — `$scope` unauthorable, update/delete require a WHERE) → desugar (upsert → insert/update by key presence) → require context (a write never executes with holes; the `missing_context` error carries the FULL derived signature) → scope (the same `ScopePolicy` reads use: `set` writes identity on insert and update, `match` pins rows) → column check against the introspected schema → compile (through the read pipeline's own `compileFilter`/`compileFieldOrValue`/`resolveParams`, so values bind as SQL parameters exactly like reads) → execute. Batches run in one transaction; the client is structural (`MutationClient` — PGlite, a pg wrapper, a test double), so the core imports no driver. **A batch that uses what it wrote (`$returned`).** `{ $returned: 'table.column' }` in a later statement reads the one row an earlier statement of the same batch wrote — the create-a-parent-then-its-children case, where the parent's id is the database's to generate and no natural key exists to `$lookup` it by. Three decisions shape it: - *Named by table, nothing else.* A statement has no name of its own, and giving it one (`as`) would be a second vocabulary inside the batch. A reference names the table an earlier statement writes; if two earlier statements write that table the lint refuses the reference, because nothing could say which was meant. - *Bound between statements, not compiled into one.* The statements stay the separate, sequential statements they were: each compiles to an ordinary parameter where the reference sits, and the engine fills that parameter from the earlier statement's `RETURNING` row before running it. Folding the batch into one statement of CTEs was tried and dropped — every CTE in a statement sees the same snapshot, so a `$lookup` could no longer read a row an earlier statement of the batch had inserted, which a batch can do today. - *Exactly one row, or the batch fails.* A `$lookup` that finds nothing writes NULL, because the row it reads may honestly not exist. A reference reads a row the batch claims to have written; when the scope boundary or a WHERE left that statement with no row, writing NULL (or skipping the child) would commit half of what the author described. So it throws inside the transaction and everything rolls back. What can be known without running — no earlier writer, two of them, an `insertEach`, a do-nothing `onConflict` — is refused at seed by `lintMutation`. Scope is untouched by it: each statement is scoped on its own exactly as before, and the value a reference carries comes from a row that statement was already allowed to write. **Derived context signatures** close the discoverability loop for both kinds: a `$context` ref sits at a position whose column the schema types, so "what do I pass this fingerprint" is computed from the stored def — never authored, never stale. Discovery lists every entry with kind, intent, the typed context contract, and shape (reads) or effect (`{op, table, columns}`, writes). An authoring lint (`lintMutation`) refuses un-keyed update/delete defs at seed time. ### Reactive reads (`refresh`) Agreed and built 2026-09-26. A read can keep answering. **The declaration is on the entry**, because freshness is a property of the question — "the current standings" is live, "the answer as asked" is not: ```ts { fingerprint: 'members/roster', refresh: 'reactive', dsl: { … } } ``` `refresh` is a mode, not a flag: `'snapshot'` (the default, and what every read was before this existed — answered when asked, never again) or `'reactive'`. It will grow; `'clock'` — time as a source of change, for a read that depends on `today` — is the next state, and is not accepted until it is built, because a value the schema takes and the engine ignores would read as live and behave as a snapshot. Only a stored entry can be reactive (the mode is authored, never generated), and only one whose mapping is compiled, because a later answer must never wait on a model. It persists like `reach`: a column in the Postgres cache, validated on write, compared by the seed path. **The caller follows by passing `{ signal, onChange }`** to `execute` — the first answer is the returned promise as always; every later answer that differs goes to `onChange` until `signal` aborts. Over HTTP there is no callback, so `handleQuery` takes an optional in-process `VexLive`, and the hono adapter reads one from the request's env — the third argument of hono's `app.request`, which only an in-process caller can set. A read nobody follows is simply a read answered from the rows cache. **The rows cache** (`engine/live.ts`) holds two things apart: - **rows** — key → the rows a reactive query returned. The only large thing, bounded by age (`ttlMs`, default 60 s), count (`maxEntries`) and text (`maxBytes`, measured as each result's JSON length), least-recently-used first out; a result over `maxEntryBytes` is never stored. Evicting rows never breaks anything; it costs one query. It lives in process memory, per engine. - **follows** — key → the tables the query reads, how to run it again, and who to tell. Bounded by what is followed, not by this module: a follow ends when its last follower's signal aborts. **The key is the compiled SQL and its bound values.** Scope compiles into both — row rules into the SQL, scope values into the parameters — so two callers share rows exactly when they would provably get the same rows. The cache never reasons about policy. The MAPPING, which reads `$.scope` (a reader's language), runs per caller over the shared rows: one query for twenty viewers, twenty cheap mappings. **A write bumps its tables' versions.** Stored rows remember the versions they were read at, and are stale when those moved — invalidation is a counter bump, not a walk over the cache. The same stamp closes the race that would otherwise poison it: rows fetched by a read that started before a write and finished after it carry the versions from before, so they are stale the moment they land, and a new read does not join that fetch. **Who invalidates: the handler, after every committed mutation** — `engine.invalidate(tablesChangedBy(writes, schema))`. It sits there, and not in any host's observer, because this is the one door every vex write passes: an HTTP request, a host's in-process machinery write, a reflex's effect. A read that heard about only some writes would be live only sometimes, which is worse than a snapshot that says what it is. `tablesChangedBy` adds, for a delete, every table that references a deleted one, transitively — a cascade changes rows vex never wrote, and the introspected schema carries the reference but not its rule, so a delete is assumed to reach them all (over-telling costs a refetch; under-telling leaves a screen stale). Writes vex never sees — raw SQL, a database trigger — are not invalidations; a host that makes them calls `engine.invalidate` itself, and the TTL heals the rest. **Followed reads refetch once per burst** (`debounceMs`, trailing) — thirty writes are one query — and are numbered, so a slow refetch landing after a newer one is dropped. Change is decided twice: once per key on the rows (the same rows → nobody is mapped), then per follower on its own mapped answer (canonical JSON, hashed), so a follower hears only when *its* answer changed. Every followed read is also re-checked once per TTL, bypassing the cache — a lost invalidation heals within one TTL. A refetch that fails keeps the last good answer and says so (`rows.error`). **Visibility.** `engine.rows.stats()` — entries, bytes, follows, followers, hits, misses, evictions, oversized, refreshes — and `rows.evict` / `rows.error` events on the engine's `onEvent`. An engine evicting faster than it answers warns once a minute: its `rows` limits are too small for what it holds. **Rules for authors.** A reactive entry should `sort` — an unordered read can come back in another order after a write and count as changed (harmless: one extra answer). It reads time from `$scope`, never context: the values bound when the read was made are the ones it refetches with. ### Who "me" is (`behaviors`, `GenerationCaller.bindings`) Built 2026-09-27. A generated query can mean the caller — "what's my name?", "my orders", "after me" — without the model ever seeing who that is. **The meaning comes from what the host declared, never from a name.** The host's behaviors (the `ScopeBehaviors` its policies are compiled from) bind columns to scope keys: `{ set: 'member_id', to: 'userId' }`, `{ match: 'studio_id', to: 'tenantId' }`. Scope values are always the caller's own, so a column bound to a key the caller carries holds that value on the caller's own rows — whether it is called `user_id`, `guest_id` or `id`. **Why the behaviors and not the policy.** A compiled policy keeps only the phases a caller is granted. A member who may only read `members` has `members: { read: [] }`; the rule that says `member_id` is them lives on an insert they do not hold. So the engine takes the behaviors themselves (`QueryEngineConfig.behaviors`) and reads every reach and every phase. **What a generation is handed**: `GenerationCaller.bindings` — every column bound to a scope key this caller actually carries, on a table their policy lets them read. Keys and columns, never values. A caller carrying none of the keys — machinery, an `executeAs` job — has no self, and is told so; nothing else about it is special. The reference agent turns the bindings into one context line (`describeCaller`): filter such a column with `{ $scope: key }` to mean the caller, and read the caller's row as an aliased subquery source to compare other rows against it. **Safety is unchanged.** The bindings are guidance for the model; access is still the policy, and the value is still bound by the engine. A query written this way is stored like any other and answers for whoever replays it — one "what's my name?" serves everybody. **Checked at `introspect`**: every column a behavior names must exist, or the engine refuses to start. ### LLM integration (decoupled) The engine takes two optional hooks: - `generateDsl(request, schema, caller) → Query` — natural language + shape → DSL. `schema` is only the tables the caller's policy can read; `caller.read` runs a DSL through the engine's pipeline as that caller. A generation sees what the person asking could see — and it is keyed (single-flight, negative cache) by request AND policy, because two callers who see different tables are asking different agents. - `mapToShape(rows, shape, intent?) → { ir, config, transformed }` — raw rows → requested shape, returning a compiled Prism IR that is cached for reuse, and the Prism config it was compiled from, which is kept beside it (`mapping`). `intent` is the request's own words, when it has them, so the mapping can give a value the form they ask for (money as `$1.2M`, a date as `Apr 6`). This is the seam that keeps the core deterministic. Vex ships a reference implementation of both, built on `@niscorp/cortex` and `@niscorp/prism` and exported from the `@niscorp/vex/agent` subpath — `createQueryDsl` (fills `generateDsl`) and `createShapeMapper` (fills `mapToShape`): - The **query agent** (`vexQueryDslAgent`, id `vex.query`) is a structured-output Cortex agent over `QuerySchema` with six tools — `getSchema`, `getSampleRows`, `getDistinctValues`, `describeField`, `testQuery`, `cannotSatisfy` — and three stop conditions (`stepCount(20)`, `outputRetries(3)`, `repeatedCalls(2)`); `createQueryDsl` aborts the run when `cannotSatisfy` is called. Context producers inject the live schema and the DSL specification as compact text, and the caller line, so the static instructions stay stable. Every data-touching tool is a DSL run through `caller.read`; the tool set holds no adapter and no SQL. `createQueryDsl` wraps it into the `generateDsl` hook. - The **mapping agent** is Prism's exported `mappingAgent`; `createShapeMapper` builds the same envelope the runtime replays against — `{ result: rows }` for an array shape, `{ result: rows[0] ?? null }` for an object shape — runs the agent once, with the intent as its `notes`, to get a transform config, compiles it to IR, and applies the IR once over that envelope. A flat shape whose fields the rows carry is mapped by Prism with no model call, its declared fields computed from the stored ones; the agent writes the mapping for any other. > The reference agents are a separate concern from the deterministic core: they > live behind the `@niscorp/vex/agent` subpath, so `@niscorp/cortex` (and > `@niscorp/signal` behind it) is pulled in only when you import it. > The engine itself depends on neither — it needs `@niscorp/prism` only to > execute a stored mapping. Provide your own > `generateDsl`/`mapToShape`, or use the reference factories. ### HTTP layer (optional) Vex is a library; it does not own a server. But it ships two thin framework adapters and a framework-agnostic handler so a resource can be exposed with a few lines: - `handler.ts` — `handleDiscovery` returns a self-describing JSON document (entities, fields, relations, the request contract, and the DSL JSON Schema); `handleQuery` parses a body, runs `engine.execute`, and maps `VexError`s to HTTP status codes. - `@niscorp/vex/hono` and `@niscorp/vex/express` wrap that handler: `GET` returns discovery, `POST` runs a query or replays a write, `PATCH`/`DELETE` manage fingerprints, and scope is resolved per-request via a `getScope` callback. ### Events An optional `onEvent` handler receives a typed `VexEvent` stream across the pipeline — `query.start`, `query.cache`, `query.dsl`, `query.sql`, `query.rows`, `query.mapped`, `query.done`, `query.error`, plus `llm.request`/ `llm.response` for callers that wrap their LLM client. This is how the dev server renders per-stage timing and a full LLM transcript log. --- ## File structure ``` src/ index.ts Public API barrel types.ts QueryEngine, QueryEngineConfig, ExecuteOptions, hook types errors.ts VexError (code + details) events.ts VexEvent union, VexEventHandler handler.ts Framework-agnostic discovery + query handlers guide.ts vexGuide — the agent-facing contract text schemas/ query.schema.ts QuerySchema (top-level DSL) + Source, SortEntry filter.schema.ts FilterSchema (recursive union) compute.schema.ts ComputeExpressionSchema aggregate.schema.ts AggregateExpressionSchema value.schema.ts FieldOrValue, ContextRef ($context), ScopeRef ($scope) identifier.schema.ts the name grammar: field paths and identifiers database.schema.ts DatabaseSchema, EntitySchema, FieldSchema, ... request.schema.ts QueryRequestSchema, QueryResponse, error codes index.ts Barrel scope/ discover.ts discoverEntities(dsl) apply.ts checkScope, scopeResolved, applyScope, VexScopeError grants.ts createScopePolicy, mergeScopePolicies, scopeGrants, scopeProfiles, scopeBindings scope.types.ts ScopePolicy, ScopeEntityRule, ScopeMatch, ScopeSet engine/ runtime.ts createQueryEngine — orchestration + cache + single-flight resolver.ts resolve(dsl, schema) → ResolvedQuery analyzer.ts analyze(resolved, config) → { warnings, errors } executor.ts executeQuery, buildContextContract, findMissingContext optional.ts pruneOptional, presenceOf — optional conditions live.ts createLiveRows — the rows cache and follows behind refresh: 'reactive' engine.types.ts ResolvedQuery and friends, AnalysisConfig, TestResult adapters/ adapter.types.ts DatabaseAdapter, CompiledQuery, ParamSlot, capabilities postgres/ postgres.adapter.ts createPostgresAdapter introspect.ts schema discovery from pg_catalog compile.ts ResolvedQuery → parameterized SQL operators.ts per-operator SQL generation statement-timeout.ts how a read's time limit is enforced date-parsers.ts RAW_DATE_PARSERS, shared by the two pool shims pglite/index.ts createPglitePool (@niscorp/vex/pglite) pg/index.ts createPgPool (@niscorp/vex/pg) hono/index.ts vex() Hono app (@niscorp/vex/hono) express/index.ts vex() Express handler (@niscorp/vex/express) index.ts adapter barrel mutations/ schema.ts MutationSchema (closed grammar), MutationDefinitionSchema engine.ts executeWrites, executeMutation — desugar/scope/validate/compile/execute signature.ts collectMutationContext, collectQueryContext, mutationEffect, lintMutation index.ts Barrel cache/ cache.types.ts CacheBackend, CacheEntry (ok | mutation | unsatisfiable) hash.ts normalizeShape, computeRequestHash, computeSchemaFingerprint, mintFingerprint memory.ts createMemoryCache (L1) postgres.ts createPostgresCache (L2, durable) tiered.ts createTieredCache (L1 + L2, warm-up) validate.ts validateEntry seed.ts seedCache, SeedEntry, SeedMutation util.ts isEntryFresh, fireAndForget, sweepCache index.ts Barrel agent/ Reference Cortex agents (exported via ./agent) query.agent.ts defineAgent: vexQueryDslAgent (vex.query → Query) tools.ts the six query tools index.ts createQueryDsl, createShapeMapper, vexQueryDslAgent utils/ context.ts buildValidationContext, resolveParams canonical.ts canonical JSON text + hash sha256.ts SHA-256 of a text, synchronous, no node built-in scripts/ Dev-only: docker, seed, dev server, fixtures ``` --- ## Dependencies | Package | Relationship | Purpose | |---------|--------------|---------| | `zod` | peer (required) | Schema validation + JSON Schema generation | | `pg` | peer (optional) | A Postgres pool for the adapter and the cache; `@niscorp/vex/pg` imports it | | `hono` | peer (optional) | Hono framework adapter | | `express` | peer (optional) | Express framework adapter | | `@niscorp/cortex` | peer (optional) | Reference query agent (`defineAgent`/`defineTool`, standalone runtime) | | `@niscorp/signal` | via Cortex (its peer) | LLM calls + embeddings | | `@niscorp/prism` | peer (required) | Result mapping (IR compile/execute; `mappingAgent` for the reference mapper) | | `@niscorp/strata` | peer (required) | The Postgres cache table's sequence and ledger; request depth guard | `zod`, `@niscorp/prism` and `@niscorp/strata` are mandatory. Everything else is pulled in only by the path you use. **No Node built-in on the path a page takes.** The main entry and `@niscorp/vex/pglite` import nothing from Node, so a bundler builds them for a browser as they are — vex over PGlite in the page, the client-degrade posture. The one thing that stood in the way was hashing: every identity vex computes (a request hash, a schema fingerprint, a policy key, the row and answer hashes of a reactive read) is a SHA-256, and it came from `node:crypto`. It is now written out in `utils/sha256.ts` — synchronous, because three exported functions answer with a hash directly and a reactive read's follower compares two inside a callback, which rules out WebCrypto's `subtle.digest`. The digests are node's, byte for byte, so nothing a cache already holds under one has moved. --- ## Key design decisions 1. **Constrained DSL, not text-to-SQL.** Security (the model can't emit injection through a validated JSON union), cacheability (a structure hashes more reliably than free text), and safety (the analyzer rejects cartesian products and over-deep nesting *before* SQL exists). A validated JSON union is only injection-proof if every string in it reaches SQL as something the union can vouch for. Authored text reaches a statement exactly three ways: a **column** that resolved against the introspected schema, a **literal** quoted the way Postgres's `quote_literal` does (safe under either `standard_conforming_strings`), or an **output name** held to `ident` and quoted. Values from a request bind as parameters and never reach text at all. The name grammar (`identifier.schema.ts`) is checked three times — at parse, in the resolver for a DSL that skipped parsing, and by a compiler that refuses any field position that did not resolve. ⟲ It was not, until 2026-09-24. Field positions (`isNull`, `in`, `like`, aggregate arguments) were typed `z.string()`; the resolver skipped any string without a dot and the compiler pasted it into the statement, so `{ isNotNull: "(SELECT secret FROM …)" }` was a boolean oracle over a table the policy denied. Output names went into `AS "…"` unescaped, and a literal containing a backslash could close its own string on a server with `standard_conforming_strings` off. `test/security/injection.test.ts` is the attack list, against a real Postgres; it failed 35 of 41 before the fix. 2. **The LLM is injected, not embedded.** Making `generateDsl`/`mapToShape` plain hooks keeps the engine deterministic and testable without a model, lets the heavy Cortex/Signal stack stay optional, and lets a consumer swap in their own generator. The bundled Cortex agents are a reference, not a requirement. 3. **Fingerprint identity, minted or named.** The output shape still determines the DSL and the mapping — but the cache *key* is a fingerprint, not the shape. A first request mints one (`fp_…`); a caller may instead own a named slot. Shape-hashing collided (two intents, one shape, one slot) and left every entry anonymous; a fingerprint makes a generation an addressable, replayable artifact — pin it, replay it, roll it back. (See *Cache v2*.) 4. **Scope injected server-side, after RESOLUTION.** If the model could see scope filters it could be talked out of them. Binding them to `$scope` values after the DSL is fixed makes them unforgeable. The policy decides whether scope applies; the values only fill it. An engine with a policy enforces it on every run, and a `$scope` slot nobody filled refuses (`missing_scope`) rather than binding NULL — which matches no row on a read and stamps NULL on a write. ⟲ Until 2026-09-24 scoping switched on only when a policy AND values were both present. Omitting the values turned the policy off entirely, table access included — a call that forgot to say who it was for read as everybody, and one did (relay's record lookup). Writes had always required both; reads now agree. The generating agent's tools had the same hole by a different route: they ran their own SQL, unscoped, beside the pipeline. They now reach the database only through `caller.read`. Scope answers two questions and they run at different stages, which is the correction to an earlier version that ran both at once: - **May this caller touch this table?** — `checkScope`, over entity names, before resolution. Needs no schema and no aliases. - **Where does the row rule go?** — `scopeResolved`, after resolution, because the answer depends on HOW the table is reached. That second stage exists because merging row rules into the DSL's `filter` compiles them to WHERE, and **a WHERE predicate on a left-joined table is a LEFT JOIN demoted to an INNER one.** A driving row whose optional FK is null gets null-padded columns, `null = $tenant` is null rather than true, and the row disappears. No error, no warning, a shorter list — and invisible until the data actually contains a null FK, which in one consumer took months. The resolver had already gone to the trouble of inferring `left` for exactly this reason; the scope layer was undoing it two steps later. Row rules for left-joined tables now go in that join's `ON` clause. The boundary is unchanged — another tenant's row still cannot contribute columns — it simply no longer annihilates the row that had none. A consequence worth having on its own: scope writes nothing into the authored `Query`. The document integrations ship over the wire carries no engine-trusted field, so there is no key to strip at intake and no forgery surface to remember. 10. **`exists` shadows SQL rather than inventing a vocabulary.** Correlated subqueries were the one thing the filter grammar could not express, so "issues nobody was sent to" and "rooms nobody is in" became two round trips each — a read returning a flat array of ids, then a `notIn` against it. That is fine at three dozen rows, wrong at three thousand, and racy in between. `{ exists: { from, filter } }` is the same `{ from, filter }` a query already is, and nothing more: no fields, no sort, no limit. EXISTS asks whether a row is there, so anything shaping output would be noise — and refusing it is what keeps this an operator rather than an invitation to nest arbitrary SQL. The correlation needs no new syntax at all, because a dotted string on either side of a comparison is already a field path: ```json { "exists": { "from": ["tasks"], "filter": { "eq": ["tasks.issue_id", "issues.id"] } } } ``` `not` composes for NOT EXISTS. Whoever knows SQL writes this on the first try, which was the entire design constraint. Two implementation notes that are load-bearing. It compiles **inline**, into the parent's own parameter counter — the other subquery path compiles independently and renumbers afterwards, which is how two parameterised subqueries came to collide on `$1`; there is nothing to renumber if nothing was numbered apart. And its inner aliases come from the **shared** alias counter, so an `exists` over the same table as the outer query cannot take the same alias and silently correlate a table with itself. Scope recurses into it. Access-checking the inner table is not enough: an EXISTS returns a boolean, but a boolean about somebody else's rows is still an answer about somebody else's rows, and an uncorrelated `exists` over a scoped table would otherwise report whether ANY tenant had one. 5. **Negative caching with a TTL.** An "impossible" request is expensive to re-discover every time, so a `cannotSatisfy` result is cached — but TTL'd and fingerprinted, because a schema change can make yesterday's impossible request possible. 6. **Single-flight on the default path.** A burst of identical cache misses collapses to one generation; the rest await it. (The `refresh`/`bypass` cache modes that opted out died with v1 — see *Cache v2*.) 7. **Adapters own compilation.** The DSL and agent loop are database-agnostic; only SQL generation is not. Keeping the entire compile step inside the adapter means a new backend is a self-contained unit and the core never grows dialect knowledge. 8. **A discovery endpoint, not just a query endpoint.** A self-describing resource (`GET` → entities + contract + DSL JSON Schema) lets a client — or another agent — learn how to query it without out-of-band documentation. 9. **Writes are replay-only, one wire shape with reads.** Mutations are dev-authored cache entries (`kind: 'mutation'`) invoked by fingerprint; the def never travels and no generation path exists. The client-facing API is uniformly `{ fingerprint, context }` — the entry's kind picks the pipeline. Context signatures are DERIVED from the stored def (typed from the schema by position), so the input contract can't drift. --- ## Deferred - Dynamic mutation generation — icebox, on purpose. If it ever returns, it feeds the existing pipeline (author → validate → seed) behind an effect contract and an approval step; nothing shipped today gets replaced. - ~~Mutation transactions with cross-statement refs~~ — built: see **A batch that uses what it wrote (`$returned`)** under Mutations. - A reply shape for writes. A mutation still answers with the rows its statements returned (one row as that row, several as an array); choosing what of a batch the caller is handed — a mapping over the writes — is not built. - Engine-derived paging (a cursor the engine mints from a query's sort). Considered with the limit work and left out: an author can already page with a `$context` bound in the filter, and a second way to do it was not earned. - MySQL / SQLite adapters — the interface is ready; implementation on demand. - Multi-level field paths — addable in the resolver without DSL changes. - ~~Intent-aware positive caching~~ — superseded: collisions did bite, and the answer is identity, not intent-matching. See **Cache v2 — fingerprints**. - Cross-process cache coherence for the tiered backend (an L1 TTL refresh). - `refresh: 'clock'` — time as a source of change: a reactive read whose bound scope includes a clock value (`today`) refetches when that value next moves. - Cross-process invalidation for reactive reads — a host carries `engine.invalidate` between processes (moss's fabric); until then each process's TTL bounds the staleness. - Row-level pruning — for a single-table reactive read, test a written row against the scoped filter and skip the refetch. An optimisation, not a design. - Adapter-owned vector serialization — the pgvector `[...]` literal is currently formed in the generic binding step; a second adapter would move it behind the adapter. ## Reach is the caller's, not the table's A `ScopePolicy` fuses two questions: which phases an entity has, and what a granted phase does. The second used to be answered per TABLE — one `match` rule per entity, applied to every caller that held any grant on it. That is right until two callers need the same grant at different distances. "The desk reads every booking; a member reads their own" cannot be said with one rule per table, and the workaround is always the same: a SECOND TABLE carrying the tighter rule, kept level with the first by a trigger. One fact in two places is a drift bug waiting for the first ordering nobody thought about. So a table may declare NAMED rule sets, and `createScopePolicy` takes the caller's profile: bookings: { default: { read: [tenant] }, personal: { read: [tenant, own] } } createScopePolicy(grants, behaviors) // studio-wide createScopePolicy(grants, behaviors, 'personal') // + their own rows The profile is chosen ONCE per caller rather than per grant, because "acts for themselves" is a property of the caller and not of each thing they read — and saying it once is one place to get it wrong instead of eight. A table declaring no variant under a profile falls back to its `default`, which is what keeps shared reads (a timetable everyone sees the same way) free of per-profile entries. It fails closed in the direction that matters. An unknown profile denies everything rather than falling back: a mistyped name that quietly meant "the default" would widen a caller to every row of every table they hold a grant on, which is the one failure mode a policy layer may not have. A table that declares named variants and no `default` likewise refuses an unprofiled caller instead of guessing that "no rule" is safe. AND A CALLER IS NOT ONE ROLE. Reach belongs to a role, and people hold several — the instructor who trains at the studio they teach at is the ordinary case, not the edge. So the unit of compilation is the ROLE: one policy each, merged (`mergeScopePolicies`), broadest rule set winning per entity and phase. A principal may do anything any of their roles permits. That merge has a consequence worth stating plainly: for somebody holding both, a staff role's studio-wide reach WINS over a member role's personal one on the tables they share. Correct for the roster they are paid to read, wrong for their own membership card. SO REACH IS SAID TWICE, and by two different parties. The CALLER carries a floor — how far this principal may see, from their role. The ENTRY may carry a ceiling — how far this READ may see, whoever asks (`reach`). "Their own card" is a property of the question, not of whoever happens to be asking it, and an app that could only say it about people was an app that got it wrong for anybody holding two roles. The two compose in one direction only: an entry's reach recompiles the caller's OWN grants under a different profile, so it narrows rows and never widens verbs. A principal without the verb is refused either way. And a declared reach that cannot be compiled refuses the read rather than falling back — the fallback would be an answer with too much in it, and nothing would say so.