ChatMaestro
Controlled experiments for large language model (LLM) answer quality.
Data sheetOverview
ChatMaestro runs controlled experiments on how the way you configure or talk to a large language model changes the quality of its answers. An experiment fixes a shared set-up and varies one parameter at a time across its runs — each a session with a cohort of real people, all scored the same way by one trusted judge. Runs are comparable within their experiment, and each stays reproducible for audit.
The problem
You can't compare two LLM set-ups fairly by reading a few chats. Answer quality shifts with the system prompt, the guidance you give, the model or ensemble of models behind it, the opening message, the framing, and even the audience. Change several at once, and the result tells you nothing. Score each set-up with a different judge, and the numbers don't line up. A fair comparison changes one factor at a time, holds everything else constant, and scores every run with the same judge. That is exactly what ChatMaestro is built to do.
How it works
1
Compose an experiment
Fix the shared set-up every run holds constant — prompts, scenario, opening message, context documents, scoring, one judge — and let the nudge, model configuration, and cohort vary.
2
Launch a group of runs
Launch several runs that change just one of those parameters and hold the others fixed, each against its own cohort. Every set-up is frozen at launch, and enrollees chat in real time.
3
Score every run
The one trusted judge scores every answer on the same rubric, and each run rolls up into one scorecard — comparable to the other runs in its group.
4
Compare the run group
Run analytics across the scorecards of a group of runs. Because the runs differ in just one parameter, any difference in score points at it. Comparisons stay within the experiment.
What you get
Controlled comparison
Across an experiment's runs you vary exactly one parameter — the nudge, the model configuration, or the cohort — and hold everything else fixed. Any difference in the scores then points at the one thing you changed; runs stay comparable within their experiment.
Single models, ensembles, or routing
Run a single model, or combine several by synthesizing them, voting among them, finding their consensus, or letting a judge pick the best. Or route: a mixture-of-experts set-up sends each question to the model best suited to it, so you pay for one answer instead of several — and the platform measures whether that pays off. Optional self-improvement passes refine an answer before it's shown, and an optional prompt enhancer can sharpen the user's request before the models answer.
One comparable scorecard
Each run produces one weighted 0–100 score across the rubric's quality factors — four by default — ready to rank the experiment's runs side by side.
Ask your data in plain language
Ask analytical and operational questions and get a short narrative plus downloadable tables and/or charts. Common questions are one-click saved buttons, and run results can always be exported to a third-party analytics tool.
Bring your own documents
Upload or link files; ChatMaestro extracts the text (including scanned PDFs) and retrieves it into a run's chats.
Full provenance
Every run's set-up, cost, and history stay recorded and queryable, and a finished run's results never drift because the set-up it used is locked while any run references it.
Copyright © 2026 Agilis Software LLCPage 1 of 2