Evals for your AI
Prove your AI gets it right. Then put it to work.
BrightEngine tests your AI instructions against real examples with known answers, scores every answer, and gives each version that passes a link your other systems can call.
From instructions to results, in five steps
Example data. Select any step to jump to it.
Works with OpenAI, Anthropic, Google, OpenRouter and endpoints you host.
Templates
Two ways to ask a model.
A template holds your instructions, the model that follows them and the shape of the answer you want back. Each version uses one of two kinds of model.
Template editor
Example template. Pick a model type to bring its editor forward.
- Presets library
- Keep shared system prompts, user prompts and JSON schemas in one place and reuse them across templates.
- Schema help
- Describe the fields you need and BrightEngine drafts the JSON schema. For System One, it proposes questions and options for you to review.
- Variables
- {{content}} carries the input. Add your own, such as {{vendor_hint}}, and fill them from values stored on each case.
Bring a key for OpenAI, Anthropic or Google, use the OpenRouter catalog, or connect an endpoint you host.
Evaluation cases
Decide what counts as right.
A case pairs a real input with the answer you expect. You choose how strictly each answer is checked: an exact match, a number rule, or an LLM judge for free text.
- Any input
- Upload PDFs and images, paste text or link a URL. Documents are converted to text by industry-leading document AI providers.
- Case State · Extra facts about a case
- Store key and value pairs on a case and use them in prompts as {{key}}, such as the supplier an invoice should come from.
- Direct comparison and number rules
- Each field is compared with its expected value. A number can instead pass when it equals a value, clears a minimum, stays under a maximum or falls between two limits.
- LLM judge
- For free text, a judge model scores a rubric or checks required points against a pass threshold you set, and quotes its evidence.
- Categories and groups
- Sort cases into categories and groups, edit many at once, and build an evaluation from any selection.
Edit Case · Expected Answer
Switch the mode, or select the number rule to change how strict it is.
Evaluation results
Which field failed, on which case, and why.
Results open as a grid of cases and fields. Scopes take you straight to the problems, and every verdict opens the expected value, the actual output and the judge's evidence. Try it below.
Invoice Data Extractor v6 · Results
- Manual overrides
- Correct a verdict by hand when the check got it wrong. The automated verdict stays on record beside yours.
- Overwrite expected with actual
- When the model turns out to be right, update the answer key in one click. Finished evaluations stay as they were.
- Tokens and cost
- Input, output and cached tokens for every run and every judge call, with cost reported by the provider or estimated.
- Pass share by field
- See which fields drag the score down across the whole run, then filter, sort, search and rerun just those cases.
Iteration
Every change gets a number.
Edit the prompt, swap the model, add a schema field. Run the same cases again and see what you changed next to what it did: which cases improved, which regressed and which held steady.
Evaluations of Invoice Data Extractor, v1 to v6
Example data. Select an evaluation to see what changed since the one before.
- Compare with baseline
- Pick an earlier evaluation as the baseline. Each case is marked Improved, Regressed or Unchanged.
- Reruns
- Rerun any finished evaluation with the same setup, or narrow it to the cases that failed. Each rerun remembers where it came from.
- Versions and rollback
- Every content change saves a new version, and any two compare line by line. Publish an older one to roll back in a single step.
API
Passed? Call it from production.
Promote the version that passed and it becomes the one your trigger URL runs. Post a document to it, and BrightEngine sends the result to your callback.
Promote · Trigger URL · Request and callback
Example data. Promote, copy the URL, or switch between the request, response and callback.
- The tested version runs
- The trigger always runs the published version. Promote or roll back without touching the code that calls it.
- Signed callbacks
- Results are signed with your team's secret and carry your own request ID back, so every reply matches a record on your side and you can trust where it came from.
- Full REST API
- Create templates, cases, evaluations and reruns from your own code with the v1 API and its OpenAPI reference.
Production
Production requests become your next test.
Every request your systems send is logged. Capture the good ones as evaluation cases, and try a new version on real requests before you switch it on.
API Call Logs · Review queue · Cases
Example data.
- API call logs
- See every production request with its input, result, status, duration and cost.
- Capture as evaluation case
- Capture a request in one click. Or set a capture policy that sends matching requests to a review queue, or captures them automatically once you trust it.
- Comparison experiments
- Run an alternate version on past or live requests. Set success criteria first and get a clear conclusion: criteria met, not met, inconclusive, or evidence only.
Run log
Follow one template from draft to production.
A worked example: one invoice extractor, four weeks, every screen along the way.
Teams
Built for teams that share templates.
- Roles
- Admins run the team and its keys. Members work across templates, cases and evaluations. Editors focus on templates.
- Shared provider keys
- Set default models for the team and for judging, and keep keys at team or personal level.
- Copy and promote across teams
- Copy a template into another team to start fresh, or promote into one to update production in place.
- Locked templates
- Lock a template to prevent unintended changes. Only team admins can unlock it.
Put a number on it before it goes live.
Start with one template and a handful of cases you already trust. The first run shows you where it breaks.