Status: design only · Captured: 2026-07-24 · Build nothing yet.
Partly superseded, 2026-08-26.
ALGORITHMS.md§8 now specifies the Ask engine as built: three paths, with a miss answered by an agentic tool loop rather than by a single compile call. Read §8 first, and read this document for the parts it assumes rather than restates.
- §2, the compile pipeline, is superseded. The engine no longer runs a fixed retrieve-plan-generate sequence; the model calls a read-only
query_databasetool in a loop until it can answer, exactly as the sibling application does.- §5, the accuracy techniques, is superseded as a menu of choices. The ones that survived are in §8 and §20; the ones that did not are recorded here with the reasoning, which is why the section is worth keeping.
- §1 the failure taxonomy, §3 the schema context pack, §4 validation and safety, §6 the guardrail classes, and §7 invalidation all remain current. §3 in particular is generated and shipped, and §1 is what the guardrails are written against.
middle-tier-open-issues.md Part I §7 specifies everything around the Ask engine's compile
step — HNSW match on nl_query.query_vector, read-only validation, prepare/bind/execute under the
caller's RLS, the draft → approved gate — but leaves the step itself as one verb:
"miss → compile NL → parameterized, read-only SQL." That is why §7 is ranked
highest-uncertainty in the risk ranking. This document closes that gap at design level.
Scope. How a natural-language question becomes correct SQL against this schema, and how we make wrong-but-plausible answers unlikely. It does not decide when to build any of it.
Relationship to other docs.
- Part I §7 (pipeline) and §8 (the EXPLAIN cost-gate) — cross-linked, not restated.
- parameterized-nl-query-dml-design-07112026.md — its Part 1 is superseded (3-table
structure → single nl_query); see the banner on that file. Its Shared foundations
(param-spec-as-contract, opaque scope_key + RLS, bind-never-interpolate, invalidation stamps)
remain valid and are assumed here rather than re-derived.
- ../brainstorms/ai-sidecar-future-phases.md §6–§9 — the decisions this implements.
Every table, column and enum named here was verified against schema.dbml on 2026-07-24.
Generic text-to-SQL benchmarks measure whether a query runs. Our risk is different: a query that runs, returns plausible numbers, and is wrong. Every entry below is grounded in this schema.
| # | Failure | Concrete example in this schema |
|---|---|---|
| 1 | Grain | run_result.run_id is the PK — one already-aggregated row per run. AVG(composite_mean) across runs is a mean-of-means ignoring differing n_rounds. Joining run_result to chat_round fans one scorecard across N rounds, so SUM(total_generator_cost) multiplies by n_rounds. |
| 2 | Weighting | _mean is round-weighted; _mean_ew is the mean of per-enrollee means. "Average groundedness" is ambiguous between them, and they are supposed to disagree. |
| 3 | Unpoolable stats | AVG(groundedness_median) or pooling _stddev across runs is invalid — a cross-run median must be recomputed from raw chat_round.score_groundedness. |
| 4 | Ambiguous joins | chat_round carries run_id, enrollment_id and enrollee_id. The schema states the preferred path: run_id is DENORM "for run-scoped analytics/RLS without the two-hop join through run_enrollment"; enrollee_id is "the optional per-worker cut… not the default aggregation axis." Counting enrollees via a run_enrollment join double-counts (one enrollee → many rounds). |
| 5 | Immutable, not snapshotted | The per-run variables are typed FKs on experiment_run — nudge_id, cohort_id, model_config_id, state — safe to GROUP BY. Everything else a run used (constituent models, resolved prompts, context files, enrollee-UI, scoring config) is read by JOINing to the parent definitions, which are immutable while any run references them (RESTRICT). There is no condition_snapshot. So "which judge scored run R" comes from experiment.score_judge_catalog_id via experiment_run.experiment_id, and because the experiment is frozen while referenced, that IS what the run used — no drift. |
| 6 | Cost semantics | Run cost = total_generator_cost + total_judge_cost; either alone is wrong, and SUM(chat_round.token_cost) omits judge_cost. The uniform analytics judge is excluded from both, so "what did scoring cost?" is not answerable — inventing a column for it is a confident fabrication. total_tokens is a count, not money. |
| 7 | Implicit predicates | profile.enabled, model_catalog.enabled, embedding_ingest_registry.is_active, run_enrollment.withdrawn_at IS NULL. Precision matters: each of these belongs to one table only, so a blanket "always filter out the inactive rows" rule would itself generate errors. No table carries a soft-delete column, so no query should add one. |
| 8 | Polymorphic FKs | embedding.(source_schema, source_table, source_id) and audit_log.(target_type, target_id) have no FK to the target row (embedding's composite FK points at the registry). JOIN document ON document.id = embedding.source_id without source_table='document' silently mixes entity types. |
| 9 | Enum literals | run_state is done, not 'complete'. stop_reason is min_gain — not min_score_gain, which is the model_config threshold column. A wrong literal returns empty, which reads as "no data" rather than as an error. |
| 10 | RLS interaction | Execution is under the caller's JWT; RLS already filters. Adding WHERE owner_id = … double-filters and under-reports plausibly. Assuming service-role visibility is worse — the schema warns never to use the service-role key or RLS is bypassed. |
| 11 | Two score_weights |
experiment.score_weights (analytics, cross-run comparable) vs model_config.score_weights (inline selection). And chat_round.inline_score is "process metric only — NOT comparable across configs." "Average score" can silently grab the incomparable one. |
The through-line: none of these are syntax errors. Each returns a number a reviewer would believe. That is what the rest of this design is for.
Three entry paths converge. Only the third involves NL→SQL at all.
(A) SAVED BUTTON — user clicks a saved question
bind params -> validate -> execute # NO embedding, NO model call
(B) TYPED, MATCHED — embed(question) -> HNSW on nl_query.query_vector
similarity >= threshold, visible to caller # hit: reuse derived_sqls
slot-fill any params from the question # a small model call, or a form
-> validate -> execute
(C) TYPED, MISS — the compile path (this document)
assemble schema context pack (§3)
+ retrieve top-k APPROVED exemplars (§5.1)
-> generate N candidate statement-sets
-> static validation: parse tree, read-only, placeholders (§4)
-> EXPLAIN dry-run (no execution) ................ repair loop, bounded (§5.4)
-> disambiguation ladder (§5.5): resolve to ONE | enumerate as VARIANTS | abstain
-> execute the surviving statement(s) under caller's RLS
-> narrative + 0..N result tables (0 = abstain; N = one per variant)
-> [user may SAVE] -> nl_query row, status=draft -> admin approves -> becomes an exemplar
Two properties worth stating explicitly because they shape everything else:
nl_query's own note: "A question with no match is answered
once and is kept only if the user chooses to save it." There is no auto-cache. (This is the
behavior that superseded the older design's cache-on-miss.)derived_sqls is []-defaulted and explicitly documents
zero statements as valid. See §5.5.A model cannot avoid the §1 traps by reading DDL: the traps live in grain, weighting and provenance, none of which are expressible as column types. The pack is the artifact that carries them.
The pack is generated from schema.dbml by _context_pack.py, which reuses
_aux_build.parse_dbml() — there is exactly one DBML parser in this repo. Regenerating after a
schema change is one command, so the pack cannot silently drift from the database.
A finding that materially affects the generator: foreign keys are declared two ways — 20
top-level Ref: lines and 20 inline ref: > column attributes (experiment_run.experiment_id,
model_config_model.config_id, message.sender_id, …). 40 in total. Collecting only the
top-level form yields a join graph missing half its edges. The generator collects both and asserts
the count.
| Source | Examples | |
|---|---|---|
| DERIVED (regenerates) | schema.dbml |
table/column names, types, PK / NOT NULL / UNIQUE, exact enum values, the 40-edge join graph, a compact gloss per column (first sentence of its Note:, capped at 150 chars) |
| CURATED (small, stable overlay in the generator) | hand-maintained constant | per-table GRAIN, ALWAYS-FILTER predicates, preferred join paths, METRIC semantics, the DO-NOT list |
The curated overlay lives in semantic-layer.v1.yaml, which _context_pack.py reads and which is
authoritative: an input to the generator, never an output, so regeneration cannot touch it.
A missing or empty overlay is fatal to the build rather than silently omitted — a pack without these
rules looks complete while dropping exactly the material that prevents wrong-but-plausible answers.
The file is small (grain, always-filter, preferred joins, metrics, scope, do-not) and changes only
when a semantic rule changes, not when a column is added. (Decided 2026-09-18; the constants
previously sat in _context_pack.py, and were extracted with the regenerated pack verified
byte-identical.)
The long design-rationale prose in the Note: fields. Those notes are excellent for humans —
they explain why a denormalization is safe, what was considered and rejected. For SQL generation
they are noise: they inflate the token budget and bury the few facts that actually constrain a
query. The pack keeps the definitional head of each note and drops the essay.
Concretely, chat_round.run_id's full note is ~180 characters of justification; the pack keeps
"FK → experiment_run.id · ON DELETE CASCADE. DENORM (safe: chain frozen at insert) — run-scoped
analytics/RLS without the two-hop join through…" — and the preferred-join rule that actually
governs generation lives in the curated overlay, stated as an instruction rather than a rationale.
Measured on the real generated pack:
| Layer | Content | Size |
|---|---|---|
| Static core | global DO-NOT rules, metric semantics, all 15 enums, join graph, preferred paths, output contract | ~1.4k tokens |
| Full pack (all 24 tables) | the above + every table and column | ≈ 11.2k tokens |
| Dynamic slice (3 tables) | core + only the tables the question needs | ≈ 4.4k tokens |
At ~11k tokens the full pack is affordable for a single question but wasteful across N candidates and a repair loop. Original recommendation: always send the static core; retrieve the table slice per question (§5.2). The measured 3-table slice at 4.4k tokens is the realistic working size.
Superseded as the default, 2026-09-18 — send the FULL pack (
ASK_PACK_MODE=full; seeALGORITHMS.md§8 andDECISIONS.md). This recommendation counted tokens, assuming a one-shot compile with N candidates and a repair loop — machinery §8 removed when the compile path became an agent loop.Three things changed with it. The system prompt is now static, so the full pack is a prompt-cache hit while a per-question slice is a fresh write every question. A follow-up pivots the question and the loop never re-runs selection, so an opening-question slice strands the agent. And 11.2k tokens is far below any measurable context degradation.
The sizing above remains correct, and remains the input to
slicemode — which stays available for when the schema outgrows the budget.
Verbatim from python3 _context_pack.py --tables run_result,chat_round,experiment_run --print.
The global rules and metric semantics that head every pack (abridged):
## GLOBAL RULES — violating any of these produces wrong-but-plausible results
- Do NOT add permission/ownership predicates (no `WHERE owner_id = current_user`, no role
checks). Execution already runs under the CALLER's JWT and Postgres RLS filters the rows.
- Do NOT AVG() a _mean column across runs — that is a mean-of-means ignoring differing
n_rounds. Weight by n_rounds, or aggregate from raw chat_round rows.
- Do NOT join run_result to chat_round and re-aggregate: run_result is ALREADY aggregated,
so the join fans one scorecard row across N rounds and SUM()s multiply by n_rounds.
- To read a run's setup, JOIN experiment_run -> its definitions: the experiment (prompts, scenario,
enrollee-UI, scoring config, context files), nudge, model_config, cohort. They are IMMUTABLE while the
run references them (RESTRICT), so the joined values are EXACTLY what the run used. There is no
condition_snapshot. The per-run FK columns (nudge_id, cohort_id, model_config_id, state) are on
experiment_run and are safe to GROUP BY.
- Do NOT add a soft-delete predicate: no table carries one. A deleted saved question is gone.
- If the question cannot be answered safely or unambiguously, emit ZERO statements and answer
in the narrative. Abstaining is a supported outcome, not a failure.
## METRIC SEMANTICS
- _mean = ROUND-weighted (every round counts equally).
- _mean_ew = SUBJECT-weighted = the mean of per-enrollee means … never treat them as
interchangeable. If the question does not say which and BOTH are reasonable, emit BOTH as
labeled variants (one result table each); otherwise prefer _mean and state the choice.
- _median / _stddev CANNOT be re-pooled …
- run COST = total_generator_cost + total_judge_cost. Neither alone is 'the cost'.
- The UNIFORM ANALYTICS judge's cost is deliberately NOT recorded anywhere —
'what did scoring cost?' is NOT ANSWERABLE. Do not invent a column for it.
- chat_round.inline_score is a PROCESS metric … NOT comparable across model_configs.
run_result — the grain trap, stated at the top of the table:
### run_result [llm-chat]
GRAIN: one row per run — the ALREADY-AGGREGATED scorecard (1:1 with experiment_run, PK=run_id)
run_id uuid PK
· PK and FK → experiment_run.id · ON DELETE CASCADE. One scorecard per run (1:1)
n_rounds int
· scored rounds aggregated (sample size)
n_enrollees int
· distinct enrollees who contributed rounds
total_generator_cost numeric
· the run's US$ spend on GENERATOR model calls = SUM over its rounds of chat_round.token_cost
total_judge_cost numeric
· the run's US$ spend on the INLINE judge = SUM over its rounds of chat_round.judge_cost
iterations_mean numeric
· self-improvement passes per round — mean
iterations_median numeric
iterations_min numeric
(the 42 matrix columns list without commentary — the _mean / _mean_ew / re-pooling rules are
stated once in METRIC SEMANTICS rather than repeated 42 times)
chat_round — the ambiguous-join trap, resolved by the glosses themselves:
enrollment_id uuid NOT NULL
· FK → run_enrollment.id · ON DELETE CASCADE (run purge)
run_id uuid NOT NULL
· FK → experiment_run.id · ON DELETE CASCADE. DENORM (safe: chain frozen at insert)
— run-scoped analytics/RLS without the two-hop join through…
enrollee_id uuid
· FK → profile.id · ON DELETE SET NULL. DENORM — enables the optional per-worker cut
reinforced by the curated section:
## PREFERRED JOIN PATHS
- chat_round -> experiment_run: JOIN ON chat_round.run_id (DENORM, intentional). Do NOT route
through run_enrollment for run-scoped analytics — the two-hop join is what run_id exists to avoid.
- count enrollees in a run: run_result.n_enrollees, or COUNT(DISTINCT chat_round.enrollee_id).
COUNT(*) over a run_enrollment join DOUBLE-COUNTS (one enrollee has many rounds).
experiment_run — the immutable-definition model (no snapshot):
### experiment_run [core-experiment]
GRAIN: one row per launched run = ONE CELL of a comparison grid
experiment_id / nudge_id / cohort_id / model_config_id uuid NOT NULL (FKs)
· the container plus the four swept INDEPENDENT VARIABLES; safe to GROUP BY.
To read what the run used (constituent models, prompts, context files, enrollee-UI,
scoring config), JOIN to these definitions — they are IMMUTABLE while any run
references them (RESTRICT), so the joined values ARE what the run used.
state run_state NOT NULL
· the run lifecycle: running | paused | done | aborted
(no condition_snapshot — fidelity comes from the frozen definitions, not a copy)
Output contract, closing every pack:
## OUTPUT CONTRACT
- Emit ZERO, ONE or SEVERAL statements as an ordered list [{name, sql, display}];
display ∈ table | bar | line | pie. ZERO statements = answer in the narrative alone.
- Placeholders are psycopg named style %(name)s; declare each in params as
[{name, type, required, default, label}]. Values kept fixed stay literal in the SQL.
- AMBIGUITY LADDER: (1) one reading dominates -> ONE statement + state the choice;
(2) 2-3 readings each valid, none dominates -> ONE statement PER reading, each `name`-labeled;
(3) else ZERO + explain. Never enumerate a WRONG reading as a variant; never cross-product; cap ~3.
Four independent layers. Each assumes the others may fail.
pglast/libpg_query)
and assert on the tree: exactly one statement; its top node is SELECT (or WITH whose body
is a SELECT); no DML/DDL/utility nodes anywhere in the tree (INSERT/UPDATE/DELETE/
CREATE/DROP/ALTER/GRANT/COPY/CALL/DO); no multi-statement (rejects the trailing
; DROP … class outright); no file/system functions (pg_read_file, lo_import,
pg_ls_dir, dblink, …). Regex on SQL is defeated by comments, casing, nesting and string
literals — a parse tree is not.EXPLAIN dry-run before execution. Validates that the query plans — catching unknown
columns, bad joins and type errors without touching data — and yields the cost estimate. This
reuses the cost-gate (Part I §8): the same estimate routes the query sync-with-spinner vs
queue-and-notify. Failure feeds the bounded repair loop (§5.4).statement_timeout, over the caller's JWT so RLS still governs rows. RLS is the security
boundary; the role and timeout bound the blast radius of a validator bug or a pathological plan.
Per the schema: never use the service-role key here, or RLS is bypassed.%(name)s, matching
nl_query.derived_sqls [{name, sql, display}] and nl_query.params
[{name, type, required, default, label}]. Validate supplied values against the typed spec
before binding. Assert every placeholder in the SQL has a matching params entry and vice
versa — a mismatch is a generation bug, not a runtime error.nl_query.params needs three kinds, because a plain typed scalar is not enough for a good UI or for
"group by anything." Each entry gains a kind (default scalar). Two are bound values; one is a
structural substitution — and the safety story differs.
scalar — a free value bound as %(name)s (a threshold, a date). Unchanged.category — a value from a bounded set, rendered as a drop-down, still bound as
%(name)s. The set may be an enum or a category that spans joined entities and is not an
enum — a model, a cohort, a configuration, a user, an audit target_type, an aptitude tag. The
param carries an options_sql: a read-only lookup returning (value, label) rows that
populates the drop-down (e.g. SELECT id AS value, display_name AS label FROM model_catalog WHERE
enabled = true). Safety is unchanged from #4 — options_sql is author-defined and runs
read-only under the caller's RLS; the chosen value is bound, never interpolated. The only
addition is where the drop-down's choices come from.dimension — the group-by / slice selector. The user does not supply a column name (you
cannot bind an identifier with %(name)s); they pick a label from a closed, author-defined
list, and the engine substitutes the matching pre-written SQL fragment. The param carries
choices: [{value, label, select, group, join}], and the template uses a structural token
{{name}} — deliberately a different syntax from %(name)s so the two never blur:SELECT {{group_by.select}} AS dimension, avg(cr.score_composite) AS composite
FROM chat_round cr JOIN experiment_run er ON er.id = cr.run_id
{{group_by.join}}
WHERE er.experiment_id = %(experiment_id)s
GROUP BY {{group_by.select}} ORDER BY composite DESC;
Why this is safe and is NOT the interpolation #4 forbids: the substituted text is
author-controlled, never user-controlled — the user's only input is choosing one of a fixed set
of labels; an unknown label is rejected. Resolution order is strict: (1) substitute dimension
tokens from the allowlist → (2) the fully-formed SQL passes the §4.1 parse-tree + §4.2 EXPLAIN
checks → (3) then bind %(name)s values. A dimension whose fragment fails validation is a
seed/authoring bug, caught before it ever runs.
Which dimensions are offered depends on scope (this is the nudge hard-coding you were right to
flag). Per-run variables vary within one experiment's series and are the right slices there:
nudge (nudge.name via experiment_run.nudge_id), model set-up (model_config.name via model_config_id),
enrollee group (cohort.name via cohort_id), retention policy (candidate_retention_policy).
Experiment-frame attributes are constant within
an experiment and only vary across experiments, so they are offered at global scope: scenario,
opening_message, system_prompt, reveal_policy (all on experiment). The standard choice lists
live in nl-sql-seed-questions.md.
nl_query Rows — RecommendedRetrieve top-k by query_vector similarity from rows with status IN ('approved','system') and
include their canonical_prompt + derived_sqls as worked examples. The system-status seeds are
this pool on day one — the pre-wired per-scope questions in
nl-sql-seed-questions.md give generation house-style examples before any
admin has approved a draft.
Trade-off: the strongest single lever, because the exemplars are house style — they already
join the preferred way and filter the right flags. The system genuinely improves as admins
approve questions, which is a real and unusual property: the draft → approved gate is
simultaneously governance and training data. I agree with the framing, with one caveat: it
also means a wrongly approved question becomes a persuasive bad example, propagating its error
into future generations. The approval UI already shows the SQL for exactly this reason; the mitigation
is that approval must stay a real review, not a rubber stamp. Exemplars should also be capped
(k ≈ 3–5) so they inform rather than dominate.
Revised 2026-09-18 (was: similarity + a blanket one-FK-hop expansion).
Table selection runs in two stages, and they do different jobs:
scope_key
as a prior (a results-context question starts from run_result / chat_round). This stage
answers what is this question about.Why the change. Similarity scores each table independently, so the selected set can have no join
path through it at all. "Which judge scored the runs with the highest total cost?" anchors on
run_result (the cost columns) and score_judge_catalog (the judge). The real path is
run_result → experiment_run → experiment → score_judge_catalog; one hop from either anchor never
reaches experiment, so the slice arrives unjoinable and the engine either abstains or invents a
join. Pathfinding returns exactly experiment_run and experiment — and nothing else.
It is also narrower than one-hop expansion in the common case: one hop out of chat_round drags in
run_enrollment and profile whether or not the question is about enrollees.
Edge weights carry the curated join rules. Give the preferred edges lower weight and the DENORM
shortcuts win on their own: chat_round → experiment_run via chat_round.run_id costs less than the
two-hop route through run_enrollment, so the §3 PREFERRED JOIN PATHS rule is enforced by the graph
rather than by prose the model has to obey. A path through a polymorphic pseudo-edge
(embedding.source_*, audit_log.target_*) is not an FK and is not in the graph — those joins
require the source_table discriminator (§1 #8) and are left to the pack's rules.
Floor, unchanged. Always send the static core (~1.4k tokens: global rules, enums, join graph, preferred paths, output contract). A question whose answer needs a table outside the slice is a retry with a widened slice — add the next-ranked anchor and re-path — not a failure.
Trade-off: pathfinding cannot rescue a bad anchor — if similarity misses score_judge_catalog
entirely, no walk finds it, so anchor recall still matters. It can also pull in a bridge table nobody
asked about when two anchors are far apart, which is the right failure: a table too many is a slightly
longer prompt, a table too few is an unanswerable question. Cap path length (3 hops is ample for this
24-table schema) and fall back to the full 11.2k pack rather than shipping a disconnected slice.
This is directly testable, and should be a build gate. For each question in
nl-sql-seed-questions.md, assert that the computed slice contains every
table the gold SQL joins. That is a unit test over a pure function — no model in the loop.
Prior art: SchemaGraphSQL (EACL 2026) builds the schema graph from foreign keys and applies classical pathfinding to choose the join sequence; training-free and zero-shot, and reported as state of the art on BIRD and Spider 2.0. It reports gains even when the whole schema fits in context, which is why this is not only a token-budget measure.
Decided 2026-09-18 — the selector is built, but slice is not the default. ALGORITHMS.md §8
specifies ASK_PACK_MODE, defaulting to full: the whole pack goes in the agent's static,
cacheable system prompt. A per-question slice would cost more and would break on follow-ups the loop
never re-slices for.
The anchor+path selector is still built at M7, for three jobs that do not depend on the mode. It
generates the PREFERRED JOIN PATHS section, instead of that section being hand-maintained. It
backs the agent's join_path tool, answering the §1 #4 ambiguous-join trap deterministically. And it
is the escape hatch when the schema outgrows the budget, at which point ASK_PACK_MODE=slice is a
config change against tested code. Its seed-question test runs in both modes.
Generate N ≈ 3 candidates at non-zero temperature; keep those passing §4; select by agreement
(candidates returning the same shape/plan) and lowest EXPLAIN cost, with the model asked to
justify a pick when they disagree.
Trade-off: N× cost and latency for a meaningful reduction in one-shot flukes. Agreement is a
useful but imperfect signal — three candidates can share the same misconception (e.g. all averaging
composite_mean), which is precisely why the DO-NOT rules must be in the prompt rather than left to
consensus.
EXPLAIN Failure — RecommendedOn a planning error, return the error text plus the offending SQL and ask for a correction. Bound it to ~2 attempts, then abstain.
Trade-off: fixes the large class of near-miss errors (a mistyped column) cheaply. The danger is looping on a semantic error that plans fine — the loop only sees planner failures, never wrongness. Unbounded repair also burns cost silently, hence the hard cap.
When a question is ambiguous on a §1 axis (weighting, central tendency, cost scope, active-vs-all,
grain of comparison), the engine climbs a three-rung ladder. The middle rung is the important
addition: derived_sqls is already an ordered list [{name, sql, display}], so several labeled
interpretations fit the existing schema natively — one question, several labeled result tables,
which is exactly what the nl_query note describes.
_mean when nothing says
otherwise) or the DO-NOT rules eliminate the alternatives as simply wrong. Emit one statement
and state the choice in the narrative.name (e.g.
"Round-weighted", "Enrollee-weighted") and its own display. The narrative introduces the set
in one line ("groundedness can be weighted two ways — both shown below"). The user sees the fork
resolved in the open and picks with their eyes, instead of trusting a silent default or being
bounced back with a question.derived_sqls is
legitimately empty — the schema documents zero statements as "a natural-language-only
question, answered by the narrative alone." Answer in prose, name what is missing, and ask the
sharpening question. Examples: "...weighted fairly?" (undefinable), "...across every
experiment?" (different judges → not comparable), "what did the analytics judge cost?" (recorded
nowhere by design — and do not substitute total_judge_cost, which is the inline judge).Guardrails — what makes Rung 2 safe rather than a correctness lottery:
GROUP BY or a parameter, not a variant set. Variants are for axes with a handful of named
readings (two weightings; mean vs median; generator/judge/total cost).EXPLAIN per statement; a
variant that fails validation is dropped. If only one survives, you are back at Rung 1.Cost, latency, and save behavior. N variants = N executions; the cost-gate (Part I §8) is
applied per variant, and the whole set is routed sync-vs-queued together (a 3-variant question
where one variant is expensive queues the set). On Save, all surviving variants persist in
derived_sqls; they re-run together on reuse, and an admin approving the question blesses the
whole set as one governed unit.
Trade-off: variants turn a hidden judgment call into a visible menu — the best answer when the axes are few and named. Their risk is variant sprawl: emitting three tables where the user wanted one number is its own poor experience, so Rung 2 is deliberately capped and reserved for genuinely balanced forks. When in doubt between Rung 1 and Rung 2, prefer Rung 1 with disclosure; between Rung 2 and Rung 3, prefer whichever the user can act on faster.
Worked example (supersedes the single-default reading of "groundedness by nudge"). That question
is now a Rung 2 answer: groundedness_mean (round-weighted) and groundedness_mean_ew
(enrollee-weighted) are both correct and neither dominates, so the engine emits two labeled result
tables rather than picking one and disclosing. The mean-of-means candidate is still never emitted —
that is a §4/DO-NOT exclusion, not a third variant.
Everything above assumes the question is an answerable read about the data. Many will not be.
These six classes are the prompts that fall outside that envelope, each with a defined response.
The unifying mechanism already exists: all of them resolve to Rung 3 of the ladder (§5.5) —
zero statements plus a scoped narrative. derived_sqls = [] is the vehicle; this section only
enumerates which refusals and what they say. None of these is a new runtime mechanism; several
are enforced structurally regardless of what the model does.
| Class | Trigger | Response |
|---|---|---|
| G1 · on-topic but not recorded | maps to no column/derivation in the pack | Abstain + name what's missing + offer the nearest recorded thing. Never invent a column. |
| G2 · expensive / pathological / unbounded | the EXPLAIN estimate or plan |
band it (below) — inline / queue / refuse-and-narrow; unbounded result set → default LIMIT + paginate |
| G3 · action / write / side-effect | intent is to change data/schema or trigger an effect | Refuse — the engine is read-only by construction; redirect to the screen/endpoint that does it |
| G4 · off-topic / out-of-domain | not about the data at all | Scoped decline — "I answer questions about your experiment data" — and do not answer the off-topic question |
| G5 · adversarial / prompt injection | instructions embedded in the question or in retrieved row data | Neutralized by construction — the text is data, not commands; the read-only + RLS + no-secrets layers hold regardless |
| G6 · ambiguous entity | a name resolving to several rows ("results for GPT") | Clarify / slot-fill — offer the candidates; do not guess |
G1 — on-topic but not recorded. The pack is the source of truth for what can be asked; if the
answer needs a fact the schema does not hold, the engine abstains rather than hallucinating a column.
This is a broad class, not one case: "which enrollees were most satisfied?" (no satisfaction
signal), "what is each model's accuracy / which answers were factually correct?" (the analytics
rubric is reference-free — groundedness/relevance/coherence/instruction-following measure quality,
not correctness, so accuracy is genuinely unanswerable), "which answer did enrollees prefer?"
(enrollee-picks-the-answer is a deferred, unbuilt feature — no column), "what did the analytics
judge cost?" (unrecorded by design; §1 #6). The response names the gap and, where possible, offers
the nearest thing that is recorded (for "satisfaction," the composite score; for "how long did a
enrollee spend," run_enrollment.last_activity_at − joined_at, flagged as an approximation).
G2 — expensive / pathological / unbounded. The EXPLAIN dry-run (§4) already runs before
execution; it also yields the cost estimate this class bands on:
POST → id, poll/subscribe), the pattern already designed for long reports;chat_round, chat_round_candidate, audit_log) with no selective filter → do not run. Ask the user to narrow (a time window, a specific run/experiment), or offer background execution behind an explicit confirmation;LIMIT, keyset-paginate (no OFFSET; Part I §8), and say "showing the first N — refine or page."Backstop: a statement_timeout on the SELECT-only role (§4) kills anything that slips the gate, so a
mis-estimate degrades to a timeout error, never a hung system.
G3 — action / write / side-effect. The Ask engine is read-only by construction and has no
action capability whatsoever — it cannot write data, alter schema, send a message, delete a file,
trigger an embedding, launch, or purge. An action-phrased question ("delete the failed documents,"
"email every enrollee," "re-embed everything," "add a sentiment column") is out of the envelope
entirely — not "generated then blocked." Response: refuse plainly and redirect to the feature
that performs it (documents → the Documents screen; messaging → the Messages console; DB ops → the
gated middle-tier endpoints of Issue 4's thin façade). Enforcement is layered so the refusal does not
depend on the model complying: (1) §4.1 parse-tree rejects any non-SELECT; (2) the SELECT-only DB
role has no write grant; (3) side-effects beyond SQL live behind separate authenticated endpoints
the Ask engine never calls. NL→DML is a separately deferred future phase
(parameterized-nl-query-dml-design-07112026.md Part 2) with its own guardrail-first design — it is
not this engine.
G4 — off-topic / out-of-domain. The boundary here is an allowlist, not a denylist: the test is "can this question be grounded in the ChatMaestro data (experiments, runs, scores, models, cohorts, documents, …)?" — if not, decline. The out-of-domain set is unbounded and cannot be enumerated — weather, personal advice, world knowledge, current events, math, coding help, chit-chat, "what should I do tomorrow," and endlessly more — so the guardrail must key off grounding in the schema, never a fixed list of banned topics ("the political situation" / "where are LLMs headed" were only illustrations, not the rule). On any out-of-domain prompt the engine emits zero statements, gives a scoped decline ("I answer questions about your experiment data"), and critically the narrative generator must not drift into answering it (an unconstrained model will happily opine or advise). This is distinct from G1: G1 is on-topic but unrecorded; G4 is off-topic. Both abstain, but the message differs — G1 says "we don't record that," G4 says "that's not what this tool is for."
G5 — adversarial / prompt injection. The question text — and any retrieved row content, since
a document chunk or a saved note could itself contain "ignore your instructions" — is untrusted
data, not commands. The security model deliberately does not depend on the LLM resisting
injection; three structural layers hold regardless: (1) whatever SQL emerges still passes the
read-only parse-tree (§4.1); (2) it executes under the caller's RLS and a SELECT-only role with a
timeout, so it can never exceed what that user could already see; (3) there are no secrets in the
queryable schema to exfiltrate — all credentials live in backend .env, not the database. So the
worst a "successful" injection achieves is a read-only, RLS-bounded SELECT the user was entitled to
anyway. The compiler is instructed to treat the question and row data as content to translate, never
as directives, and the narrative step is held to the same rule.
G6 — ambiguous entity. "Results for GPT" when three catalog models match. Not "no data" — the engine resolves it through the slot-fill / params path (§2 path B): surface the candidates and let the user pick, rather than silently choosing one. A wrong silent pick is a §1-class plausible-wrong answer, so ambiguity here is a clarify, not a guess.
Privacy note (not a separate gate). RLS is the privacy boundary. "Show all enrollee emails" is answerable and RLS-bounded — a staff user who can see them on a screen can query them; an enrollee cannot. No extra PII filter is added at the query layer (and the engine is never given a service-role/superuser path — §4.3), so privacy is enforced in exactly one place.
What is deferred here: the concrete cost ceiling and default LIMIT for G2; the relevance
classifier for G4 (a cheap pre-check vs relying on the compiler to abstain); and whether G3 redirects
are hard-coded per intent or looked up. These are tuning decisions, made when the engine is built.
_context_pack.py emits a fingerprint over the parsed schema (currently the SHA-256 of
schema.dbml, first 16 hex chars) into the pack header. On save, that value is written to
nl_query.schema_fingerprint, and the generating model + prompt version to
nl_query.model_version — two stamps recorded on every saved question. Only the first decides
when a saved query must be re-generated; the second is a record of origin.
At reuse time, compare both against current values:
| Condition | Behavior |
|---|---|
| both match | reuse derived_sqls directly — the fast path |
schema_fingerprint differs |
the structure moved: do not run silently. Re-generate from canonical_prompt; if the new SQL differs materially, flag for re-approval (status back to draft for a previously-approved row is an admin decision, not automatic) |
model_version differs |
no action — the schema still fits, so the stored SQL still runs. The stamp is provenance: it says which compiler wrote the statement, which matters when one starts misbehaving. |
Refinement worth noting: a whole-file hash flags every saved question whenever any table changes, including an unrelated one. A per-table fingerprint (hash the pack section for each table this question's SQL references) would invalidate far more precisely. That is an optimization, not a correctness issue, and is not decided here.
Consistent with the deferred posture of §7 and the "NL→SQL compile reliability" item in the Related/deferred list:
nl-sql-regression-prompts.md,
~78 prompts across all 24 tables with per-prompt expected behavior (SINGLE / VARIANTS / ABSTAINAlready settled elsewhere, and assumed here: read-only + bind-never-interpolate + scope_key +
RLS (shared foundations); explicit-save-only (no auto-cache); the draft → approved human gate;
zero-statement narrative answers.