mono.state = done, so its scorecard (run_result) has been built.composite_mean and composite_median) and the per-factor five-number summaries in factor_stats (min, q1, median, q3, max). They compare runs across the series and check cost and effort: the token count (total_tokens) and the US dollar spend, split between the doer models and the judge (total_generator_cost versus total_judge_cost), plus passes and latency. To dig deeper they open the Ask panel, either clicking a saved question (a one-click nl_query button) or typing a new one in plain language. The answer comes back as a narrative plus zero or more result tables, each viewable as a downloadable table or chart. A useful new question can be saved for the team. Anything on screen exports to a file.run_result rows on the two variables held constant, varying the third — the nudge, the model set-up, or the cohort — with readable labels joined straight from the immutable definitions. There is deliberately no stored “series” object. Cost is reported two ways that add up: total_generator_cost for the doer models (the variable under test) and total_judge_cost for the judge overhead, so a fast-but-expensive set-up cannot hide. A plain-language question is matched against saved questions or translated to SQL by an AI model, then run read-only, with the generated SQL always shown next to the answer. Only an explicit Save writes an nl_query row.run_result, chat_round, experiment_run, nl_query writes nl_query (only on Save)| If… | Then… |
|---|---|
| Fewer than two runs of the series have finished | There is nothing to compare; the comparison view waits for a second done run. |
| A typed question would modify data | Refused: the plain-language path compiles and runs only reads, and the SQL is shown beside the answer (Ask a Question in Plain Language). |
composite_mean alongside composite_median: when they diverge, the per-enrollee distribution is skewed, which is itself a finding.total_generator_cost (the doer models under test) is kept separate from total_judge_cost (the one trusted judge's overhead, for both picking winners and grading), so the measurement cost is visible and cannot masquerade as answer cost.model_catalog.context_window in each run's configuration — and the configuration is a swept variable, so one arm can have systematically shallower memory than another. Read alone that looks like a quality difference. run_result.history_stats puts the figures beside the scores: the smallest window in force, the rounds truncated, the rounds where the enrollee was alerted, the rewinds, and the turns actually replayed.chat_round.declined, split into appropriate and unwarranted. An appropriate refusal takes relevance to N/A rather than scoring it badly, so a model that correctly declines is not ranked below one that answers anyway; an unwarranted refusal stays scored. The scorecard freezes the tally in run_result.declined_dist, so the refusal rate still reads after a cleanup has removed the rounds it came from. Requests the sufficiency gate turned back are not in it — those never became rounds.composite_mean, the effort/cost stats, and can export — with no rows written.run_result rows; the plain-language path adds an AI translation step but still runs read-only.This is where the study pays off. Every exchange in every run was graded the same way by a single judge AI, so the numbers are comparable, and the run's scorecard rolls them into one line.
You get two headline averages, and it is worth knowing why. One treats every question-and-answer equally. The other treats every person equally, so somebody who sent fifty prompts does not drown out somebody who sent five. If those two numbers disagree, that tells you something real about who drove the result.
Alongside quality you can see cost and effort: how many improvement passes it took, how slow it was, how many tokens it used, and the actual dollar cost, split two ways. The doer models (the ones under test) are billed separately from the judge, so you can tell whether an expensive set-up is expensive because of the answer or because of the grading. A model that is cheap per token but needs many passes can still end up the pricey one.
If you would rather just ask, open the Ask panel. You can click a saved question, a labeled one-click button someone already built, or type a new one in ordinary English. ChatMaestro answers with a short write-up plus any number of result tables, each of which you can flip to a chart. It also shows you the query it used, so you can check its work rather than trust it blindly. If the question turns out to be worth keeping, you save it as a button for next time. Anything you see, you can download.
run_result (the score matrix and the operational and cost matrix, including total_generator_cost versus total_judge_cost) and chat_round. Saved questions are nl_query rows (see Ask a question in plain language). There are no analytics tables and no run-set table; comparisons are grouped queries.