Arbiter
Auditable reasoning for third-party vendor onboarding
SoftwareComplete
Solo builder. Open Agent Hackathon 2026, Reasoning Architecture track. 28th in the Reasoning Architecture track.
A multi-agent system that checks a SaaS vendor's security claims against the company's own records. Agents gather cited evidence, a versioned policy makes the decision, and every step replays from a signed audit trail.
I built Arbiter on my own in five days for the Open Agent Hackathon 2026, sponsored by NVIDIA and SpaceX, in the Reasoning Architecture track. I competed solo against thousands of participants, most of them in teams, and Arbiter placed 28th in the track.
It is for the third-party risk analyst who decides whether a new SaaS vendor may touch customer data. The vendor answers a security questionnaire; Arbiter checks each answer against what the company’s own systems say, from contracts and incidents to watchlists and CISA’s list of known-exploited vulnerabilities, and does the first pass of the triage.
Five agents do the reasoning on NVIDIA Nemotron, through Meterless Nightshift. A Planner chooses which records to read, an Evidence agent extracts each claim with a verbatim quote, an Evaluator checks the claims against the records and sends the plan back when evidence is missing, a Challenger attacks the candidate decision, and a Reporter writes the cited brief. The decision itself is made by a versioned policy engine, not a model: one of five outcomes, from approve to decline. When evidence is missing, Arbiter drafts the request back to the vendor.
All five company systems sit behind one Zetaris semantic layer, so the agents query one governed model instead of five schemas. Meterless H-MEM recalls precedents a TPRM lead has approved, and Meterless Scout screens vendor text before any model sees it. Every step is written to a hash-chained audit log, every action needs a signed execution contract, and any decision can be replayed from the log.
The demo video

What it does
A third-party risk analyst at a 500 to 5,000-person company decides whether a new SaaS vendor may touch customer data, and a TPRM lead signs off the exceptions. The evidence sits in five systems and a questionnaire, and the queue does not wait:
- Third parties are involved in 48% of breaches (Verizon 2026 DBIR).
- Organisations average 2,643 third parties but assess only 36% of them, and 40% have an assessment backlog (ProcessUnity / Ponemon-Sullivan 2026, n = 1,465).
- Companies spend 37.4 hours a week assessing vendors (Whistic 2025 TPRM Impact Report, n = 525).
Arbiter turns one onboarding into a cited, replayable decision with one of five outcomes:
- Approve
- Approve with conditions
- Request information
- Escalate
- Decline

How it reasons
The orchestrator is an explicit state machine, and each agent has a narrow job:
| Agent | Can | Cannot |
|---|---|---|
PlannerAgent |
Choose registry queries from the USL catalogue; revise the plan | Write SQL, or skip a required control |
EvidenceAgent |
Run registry queries, run ScoutGuard, extract claims with verbatim quotes | Mark a claim verified |
EvaluatorAgent |
Ground quotes, cross-check claims against systems of record, send the plan back | Decide an outcome |
PolicyEngine |
Tier the vendor; produce the outcome and what evidence would change it | Call a model |
MemoryKeeper |
Retrieve lead-approved precedents; record feedback | Change the policy |
ChallengerAgent |
Attack the candidate decision; request one re-plan; escalate | Approve or upgrade an outcome |
ReporterAgent |
Write the cited brief and draft the next action | Cite anything outside the decision record |
ScoutGuard |
Block, warn or pass untrusted vendor text; redact PII | Decide |
ActionExecutor |
Issue and verify signed contracts; run the outbox once approvals exist | Act without a valid contract |
Under the hood
Seven steps of one assessment, each beside the code or the trace line behind it, from Arbiter’s repository at the commit this page describes.
Guard
Vendor text passes Meterless Scout before any model reads it; blocked content escalates.
scout_guard.py, lines 48 to 64
def guard( text: str, *, surface: GuardSurface, capabilities: set[str], segment_id: str, tracer: Tracer | None = None, use_model: bool = True, ) -> GuardResult: del surface # reserved for future surface-specific thresholds hits = patterns.scan(text, capabilities=capabilities) score = patterns.max_score(hits) _log_metadata(hits, segment_id, len(text)) level: Literal["pass", "warn", "block"] = "pass" if score >= BLOCK_THRESHOLD: level = "block"Plan over one model
The Planner reads the Zetaris semantic layer as a 304-token catalogue and chooses registry queries; it never writes SQL.
queries.yaml, lines 181 to 193
Q_COMPONENT_KEV: description: KEV matches for declared components sql: | SELECT DISTINCT c.component_id, c.vendor_id, k.cve_id, k.vendor_project, k.product, k.vulnerability_name, k.date_added, k.due_date, k.ransomware_use FROM {T.vendor_component} c INNER JOIN {T.kev_vuln} k ON lower(k.vendor_project) = lower(c.vendor_project) AND lower(k.product) LIKE concat('%', lower(c.product), '%') WHERE c.vendor_id = :vendor_id ORDER BY k.cve_id params: - {name: vendor_id, type: vendor_id}Gather with quotes
The Evidence agent runs the queries and extracts each claim with a verbatim quote. In live example 2, the query that joins the vendor's declared software to CISA's known-exploited list ran inside Zetaris and found 15 FortiOS entries.
Live example 2's trace, line 59
{ "timestamp": "2026-10-25T15:16:55.887221Z", "run_id": "run-211916c1544d", "trace_id": "run-211916c1544d", "span_id": "2e67b49302d94d47", "agent_name": "data", "action": "query_zetaris", "input_summary": "query_id=Q_COMPONENT_KEV", "output_summary": "rows=15 zetaris_request_id=04559cdd-f5ae-4ee7-af41-f742d4703094", "retry_count": 0, "status": "success", "latency_ms": 1992, "query_id": "Q_COMPONENT_KEV", "rows": 15, "backend": "zetaris" }Evaluate and send back
The Evaluator checks claims against the records and sends the plan back for missing evidence, at most twice. Here it sent the Planner back for incident and contract evidence.
Live example 2's trace, line 48
{ "timestamp": "2026-10-25T15:16:01.106320Z", "run_id": "run-211916c1544d", "trace_id": "run-211916c1544d", "span_id": "6a0ba5b7bd4d4030", "agent_name": "EvaluatorAgent", "action": "evaluate", "input_summary": "claims=16 queries=8", "output_summary": "gaps: C06→Q_VENDOR_INCIDENTS; C07→Q_VENDOR_CONTRACT; C08→Q_VENDOR_CONTRACT; C09→Q_VENDOR_CONTRACT; C10→Q_VENDOR_CONTRACT; C12→Q_COMPONENT_KEV", "target_agent": "PlannerAgent", "retry_count": 0, "status": "needs_revision", "iteration": 1 }Decide by policy
A versioned policy maps the facts to one of five outcomes; a model's worse extraction can lower an outcome, never raise it.
arbiter_policy.v1.yaml, lines 168 to 177
outcome_rules: # evaluated in order; first match wins - {id: O1, when: hard_gate_decline_false, outcome: DECLINE, reason: HARD_GATE_DECLINE} - {id: O2, when: injection_blocked, outcome: ESCALATE, reason: UNTRUSTED_CONTENT_TAMPERING} - {id: O3, when: unresolved_material_contradiction, outcome: ESCALATE, reason: CONTRADICTION} - {id: O4, when: hard_gate_escalate_false, outcome: ESCALATE, reason: HARD_GATE_ESCALATE} - {id: O5, when: required_false_without_exception, outcome: ESCALATE, reason: CONTROL_FAILED} - {id: O6, when: required_unknown, outcome: REQUEST_INFO, reason: EVIDENCE_MISSING} - {id: O7, when: confidence_below_gate, outcome: ESCALATE, reason: LOW_CONFIDENCE_ABSTAIN} - {id: O8, when: any_conditional, outcome: APPROVE_WITH_CONDITIONS, reason: CONDITIONS} - {id: O9, when: always, outcome: APPROVE, reason: ALL_REQUIRED_CONTROLS_MET}Challenge
The Challenger attacks the candidate decision and can force a re-plan or an escalation, never an approval.
orchestrator.py, lines 108 to 116
if state is State.CHALLENGE: challenge = st.challenge if ( challenge is not None and challenge.action == "re_plan" and st.challenge_loops < MAX_CHALLENGE_LOOPS ): return State.PLAN return State.REPORTSign, audit, replay
Every side effect needs an HMAC-signed execution contract; every step joins a hash-chained log; any decision replays from it.
contracts.py, lines 54 to 92replay.py, lines 53 to 78
def sign(contract: ExecutionContract, *, key: bytes | None = None) -> ExecutionContract: secret = key if key is not None else keys.signing_key() payload = canonical_json(_payload_for_sign(contract)) sig = hmac.new(secret, payload.encode("utf-8"), hashlib.sha256).hexdigest() return contract.model_copy(update={"signature": sig}) def verify( contract: ExecutionContract, *, now: datetime, key: bytes | None = None, ) -> tuple[bool, str]: secret = key if key is not None else keys.signing_key() payload = canonical_json(_payload_for_sign(contract)) expected = hmac.new(secret, payload.encode("utf-8"), hashlib.sha256).hexdigest() if not hmac.compare_digest(contract.signature, expected): return False, "bad_signature" expires = contract.expires_at if expires.tzinfo is None: expires = expires.replace(tzinfo=UTC) now_utc = now if now.tzinfo else now.replace(tzinfo=UTC) if now_utc > expires.astimezone(UTC): return False, "expired" allowed = set(contract.scope.capabilities) for step in contract.tool_plan: cap = step.capability if isinstance(step, ToolStep) else step["capability"] if cap not in allowed: return False, "capability_out_of_scope" if contract.state not in { "issued", "pending_approval", "approved", "executed", "rejected", "expired", }: return False, "illegal_state" return True, "ok"def replay( case_id: str, *, store: Store | None = None, policy_root: Path | None = None, now: datetime | None = None, ) -> ReplayResult: """Replay one case. `now` is unused: signatures are checked at each contract's issued_at.""" del now store = store or Store() root = policy_root or _POLICY_ROOT record = store.get(case_id).decision signatures_ok = _signatures(store.contracts_for(case_id)) if record.outcome_reasons == _HUMAN_REVIEW: return _replay_reviewed(case_id, record, store, root, signatures_ok) policy, calibration = _policy_for(record, root) policy_ok, record_ok, diff, _recomputed = _replay_machine(record, policy, calibration) return ReplayResult( match=policy_ok and record_ok and signatures_ok and not diff, diff=diff, checks=ReplayChecks( policy_hash=policy_ok, record_hash=record_ok, contract_signatures=signatures_ok, ), )
How well it works
| Measure | Result | Caveat |
|---|---|---|
| Correct decisions, live, on the first 10 test cases (25 October 2026) | 10 of 10 | Wilson 95% interval 0.72 to 1.00 |
| Unsafe approvals, live | 0 | |
| Consistent repeats, live | 4 of 4 | |
| Correct decisions, offline, on the whole test split | 16 of 16 | |
| Model calls that fell back to deterministic logic, live | 10 of 21 | Because of the routing issue under Credits; the policy still decided every case |
| A single-prompt baseline | Not measured | Every baseline call returned HTTP 502 |
| Live example 2, Quillfeather Payroll Cloud Ltd | Escalate, in 228 s | 14 Zetaris queries, each with a request id; 4 Nemotron calls, each with a receipt |
| Zetaris parity with the mirror | 416 of 416 | Registry comparisons |
The data is synthetic and n is small: these numbers show that the live path works end to end, not a precision estimate. The single-prompt baseline could not be measured, so no claim is made that Arbiter beats it; whichever calls fell back, the policy decided every case.
The method, labels and discussion, in Arbiter's EVALUATION.md
Credits
Arbiter runs on its sponsors’ technology:
- NVIDIA Nemotron 3.5 Super VL 120B and Nemotron 3.5 Lightning 30B, through Meterless Nightshift;
- Meterless H-MEM and Scout;
- the Zetaris Unified Semantic Layer;
- the CISA Known Exploited Vulnerabilities catalogue, as a committed snapshot.
One deviation, stated plainly, as Arbiter’s README states it. The deep lane was designed for
Nemotron 3 Super, which the account was not entitled to. A Nightshift routing rule coerced both lanes
to auto, so lane separation was not guaranteed, and Meterless acknowledged the routing bug on 24
October.
I built Arbiter with coding agents, Cursor’s for the early passes and then Claude Code, and reviewed every change, as the README’s “AI and model usage” records.
Authorship
This is my own work.
It is built on tools I did not make: NVIDIA Nemotron models, Meterless Nightshift, H-MEM and Scout (Arbiter vendors part of Scout in vendor/meterless) and the Zetaris platform.
Updates
AI, Software
Arbiter placed 28th globally in Reasoning Architecture
My solo entry to the Open Agent Hackathon 2026 finished 28th globally in the Reasoning Architecture track. You can run its console here.
AI
Competing in the Open Agent Hackathon 2026
I'm competing in the Open Agent Hackathon 2026 and working on my submission.


