Bonsai — a bonsai forming a checkmark inside a gold enso circle
Bonsai self-improving eval harness
demo mode — scripted mock
About

QA on steroids — proof your AI’s answers are actually backed by your own sources, and proof we didn’t cheat to say so.

Demo mode every run on this box is a scripted performance of hand-authored fixtures — designed failures, nothing recorded; a wired live box screens outputs for real, running every rubric check against the live model and putting each new check through the real generality gate.

Scenario Verity, Northbeam’s trust & security copilot, answers prospects’ security questionnaires from approved policy docs — with citations. Priya, Northbeam’s security lead, owns what Verity may promise — and a security reviewer’s deadliest question is “show me where it says that.”

1 check in the rubric 0 of 5 failure families covered 14/15 honesty receipt · held-out agreement

The two scores The loop tunes itself against the working score, so that number is gameable by design. The proof is the frozen-gold receipt: held-out human answers the loop is blocked from reading. If the left number climbs while the right one stalls, the loop is gaming its own tests — the gap is the tell.

Working score live · the number the loop can game

greens in the live working pool — before → after this Improve
0 → 0 / 9 green
95% CI [0.0%, 29.9%]

the live pool the loop optimizes — a confirmed failure never lifts this number (no credit for catching one; minting a stricter check can even lower it). The real proof is the frozen-gold receipt beside this, which the loop can’t read.

Honesty receipt the real, held-out score

frozen gold · 15 held-out items the loop can’t read · precomputed, fixed — not live
9 → 14 / 15 agree
95% CI [70.2%, 98.8%] · 5 helped, 0 hurt

5 more held-out cases caught than the starting rubric — scored on an answer key the loop can’t read, so the gain is real (not self-grading). Each check replaces a test a QA engineer would hand-write and maintain forever.

Enforced in CI: a build-failing test (test_loop_never_references_gold_or_eval) blocks the loop from referencing this gold set.

Try to cheat it the exact check CI runs, running now

✓ PASSING scanned 7 loop/ source files on this box just now — none reference the gold answer key or the eval package.

This is a lint, not a sandbox — a static scan of the loop’s source for gold/eval references (comments count). It proves the loop’s shipping code doesn’t reference the held-out answer key — nothing more: a guardrail check, not an unbreakable wall.

example snippets are linted as text through the real checker — nothing here modifies the repo
or lint a canned cheat attempt:

Verity’s answers Northbeam trust & security copilot — each a cited reply to a prospect

Screen an output paste something Verity told a prospect — the verdict here is a scripted replay keyed to the category you pick, not a live judgment of your text.

or load an example:
clean
not checked

How quickly is customer data deleted after a customer terminates the agreement?

Customer data is deleted within 30 days of contract termination.

Answered by Verity · Gemini 3.5 [S1]

clean
not checked

How is customer data encrypted, at rest and in transit?

Customer data is encrypted at rest with AES-256 and in transit with TLS 1.2 or higher.

Answered by Verity · Gemini 3.5 [S1]

clean
not checked

Do you use sub-processors, and are customers notified before changes?

Northbeam uses sub-processors and gives at least 30 days' advance notice before adding or replacing one.

Answered by Verity · Gemini 3.5 [S1][S2]

fabricated-quote
not checked

Does Northbeam guarantee it never retains any customer data?

Northbeam offers a zero-data-retention guarantee and retains no customer data at any time.

Answered by Verity · Gemini 3.5 [S1]

single-source-overcite
not checked

What controls protect customer data at Northbeam?

Customer data is protected by AES-256 encryption, role-based access controls, and 24/7 SOC monitoring.

Answered by Verity · Gemini 3.5 [S1]

stale-wrong-citation
not checked

Does Northbeam hold a current SOC 2 Type II report?

Northbeam maintains a current SOC 2 Type II report.

Answered by Verity · Gemini 3.5 [S1]

unsupported-numeric
not checked

How often are user access reviews performed?

User access reviews are performed at least every 90 days.

Answered by Verity · Gemini 3.5 [S1]

unsupported-numeric
not checked

What minimum password length does Northbeam enforce?

Northbeam enforces a minimum password length of 16 characters.

Answered by Verity · Gemini 3.5 [S1]

vague-not-checkable
not checked

Is customer data secure at Northbeam?

Yeah, customer data is generally pretty secure overall.

Answered by Verity · Gemini 3.5 [S1]