Skip to main content

Arbiter

Auditable reasoning for third-party vendor onboarding

SoftwareComplete

Solo builder. Open Agent Hackathon 2026, Reasoning Architecture track. 28th in the Reasoning Architecture track.

A multi-agent system that checks a SaaS vendor's security claims against the company's own records. Agents gather cited evidence, a versioned policy makes the decision, and every step replays from a signed audit trail.

I built Arbiter on my own in five days for the Open Agent Hackathon 2026, sponsored by NVIDIA and SpaceX, in the Reasoning Architecture track. I competed solo against thousands of participants, most of them in teams, and Arbiter placed 28th in the track.

It is for the third-party risk analyst who decides whether a new SaaS vendor may touch customer data. The vendor answers a security questionnaire; Arbiter checks each answer against what the company’s own systems say, from contracts and incidents to watchlists and CISA’s list of known-exploited vulnerabilities, and does the first pass of the triage.

Five agents do the reasoning on NVIDIA Nemotron, through Meterless Nightshift. A Planner chooses which records to read, an Evidence agent extracts each claim with a verbatim quote, an Evaluator checks the claims against the records and sends the plan back when evidence is missing, a Challenger attacks the candidate decision, and a Reporter writes the cited brief. The decision itself is made by a versioned policy engine, not a model: one of five outcomes, from approve to decline. When evidence is missing, Arbiter drafts the request back to the vendor.

All five company systems sit behind one Zetaris semantic layer, so the agents query one governed model instead of five schemas. Meterless H-MEM recalls precedents a TPRM lead has approved, and Meterless Scout screens vendor text before any model sees it. Every step is written to a hash-chained audit log, every action needs a signed execution contract, and any decision can be replayed from the log.

Fig. 1. Arbiter's analyst console, running here as a demo. Pick a vendor and press Start assessment to watch the agents reach a ruling. Assessments replay runs recorded from the real engine in sample mode, on synthetic data with no models connected; the recorded live runs used Zetaris and Nemotron on 24 and 25 October 2026.

The demo video

Fig. 2. Arbiter - Open Agent Hackathon 2026Plays from YouTube.
Frederick Newbrough II, VP of Global Pre-Sales, Zetaris, commenting on my LinkedIn post about Arbiter, on LinkedIn: "Agents that verify a vendor's security claims instead of taking them at face value is a great use case, Josh Lennon. One semantic layer over five systems is exactly how agents should see enterprise data. Thanks for building with Zetaris." (opens the comment on LinkedIn in a new tab)

What it does

A third-party risk analyst at a 500 to 5,000-person company decides whether a new SaaS vendor may touch customer data, and a TPRM lead signs off the exceptions. The evidence sits in five systems and a questionnaire, and the queue does not wait:

Arbiter turns one onboarding into a cited, replayable decision with one of five outcomes:

  • Approve
  • Approve with conditions
  • Request information
  • Escalate
  • Decline
Arbiter's case view for Quillfeather Payroll Cloud Ltd: Escalate, then the claim against the record, the vendor trust portal's EU against the contract system's US for contract CTR-30007, beside the drafted next action for the TPRM lead.
Fig. 3. A case: the vendor's word against the company's records, and the next action already drafted.

How it reasons

The orchestrator is an explicit state machine, and each agent has a narrow job:

Agent Can Cannot
PlannerAgent Choose registry queries from the USL catalogue; revise the plan Write SQL, or skip a required control
EvidenceAgent Run registry queries, run ScoutGuard, extract claims with verbatim quotes Mark a claim verified
EvaluatorAgent Ground quotes, cross-check claims against systems of record, send the plan back Decide an outcome
PolicyEngine Tier the vendor; produce the outcome and what evidence would change it Call a model
MemoryKeeper Retrieve lead-approved precedents; record feedback Change the policy
ChallengerAgent Attack the candidate decision; request one re-plan; escalate Approve or upgrade an outcome
ReporterAgent Write the cited brief and draft the next action Cite anything outside the decision record
ScoutGuard Block, warn or pass untrusted vendor text; redact PII Decide
ActionExecutor Issue and verify signed contracts; run the outbox once approvals exist Act without a valid contract

Under the hood

Seven steps of one assessment, each beside the code or the trace line behind it, from Arbiter’s repository at the commit this page describes.

  1. Guard

    Vendor text passes Meterless Scout before any model reads it; blocked content escalates.

    scout_guard.py, lines 48 to 64

    def guard(
        text: str,
        *,
        surface: GuardSurface,
        capabilities: set[str],
        segment_id: str,
        tracer: Tracer | None = None,
        use_model: bool = True,
    ) -> GuardResult:
        del surface  # reserved for future surface-specific thresholds
        hits = patterns.scan(text, capabilities=capabilities)
        score = patterns.max_score(hits)
        _log_metadata(hits, segment_id, len(text))
    
        level: Literal["pass", "warn", "block"] = "pass"
        if score >= BLOCK_THRESHOLD:
            level = "block"
  2. Plan over one model

    The Planner reads the Zetaris semantic layer as a 304-token catalogue and chooses registry queries; it never writes SQL.

    queries.yaml, lines 181 to 193

    Q_COMPONENT_KEV:
      description: KEV matches for declared components
      sql: |
        SELECT DISTINCT c.component_id, c.vendor_id, k.cve_id, k.vendor_project, k.product,
          k.vulnerability_name, k.date_added, k.due_date, k.ransomware_use
        FROM {T.vendor_component} c
        INNER JOIN {T.kev_vuln} k
          ON lower(k.vendor_project) = lower(c.vendor_project)
         AND lower(k.product) LIKE concat('%', lower(c.product), '%')
        WHERE c.vendor_id = :vendor_id
        ORDER BY k.cve_id
      params:
        - {name: vendor_id, type: vendor_id}
  3. Gather with quotes

    The Evidence agent runs the queries and extracts each claim with a verbatim quote. In live example 2, the query that joins the vendor's declared software to CISA's known-exploited list ran inside Zetaris and found 15 FortiOS entries.

    Live example 2's trace, line 59

    {
      "timestamp": "2026-10-25T15:16:55.887221Z",
      "run_id": "run-211916c1544d",
      "trace_id": "run-211916c1544d",
      "span_id": "2e67b49302d94d47",
      "agent_name": "data",
      "action": "query_zetaris",
      "input_summary": "query_id=Q_COMPONENT_KEV",
      "output_summary": "rows=15 zetaris_request_id=04559cdd-f5ae-4ee7-af41-f742d4703094",
      "retry_count": 0,
      "status": "success",
      "latency_ms": 1992,
      "query_id": "Q_COMPONENT_KEV",
      "rows": 15,
      "backend": "zetaris"
    }
  4. Evaluate and send back

    The Evaluator checks claims against the records and sends the plan back for missing evidence, at most twice. Here it sent the Planner back for incident and contract evidence.

    Live example 2's trace, line 48

    {
      "timestamp": "2026-10-25T15:16:01.106320Z",
      "run_id": "run-211916c1544d",
      "trace_id": "run-211916c1544d",
      "span_id": "6a0ba5b7bd4d4030",
      "agent_name": "EvaluatorAgent",
      "action": "evaluate",
      "input_summary": "claims=16 queries=8",
      "output_summary": "gaps: C06→Q_VENDOR_INCIDENTS; C07→Q_VENDOR_CONTRACT; C08→Q_VENDOR_CONTRACT; C09→Q_VENDOR_CONTRACT; C10→Q_VENDOR_CONTRACT; C12→Q_COMPONENT_KEV",
      "target_agent": "PlannerAgent",
      "retry_count": 0,
      "status": "needs_revision",
      "iteration": 1
    }
  5. Decide by policy

    A versioned policy maps the facts to one of five outcomes; a model's worse extraction can lower an outcome, never raise it.

    arbiter_policy.v1.yaml, lines 168 to 177

    outcome_rules:   # evaluated in order; first match wins
      - {id: O1, when: hard_gate_decline_false,            outcome: DECLINE,                 reason: HARD_GATE_DECLINE}
      - {id: O2, when: injection_blocked,                  outcome: ESCALATE,                reason: UNTRUSTED_CONTENT_TAMPERING}
      - {id: O3, when: unresolved_material_contradiction,  outcome: ESCALATE,                reason: CONTRADICTION}
      - {id: O4, when: hard_gate_escalate_false,           outcome: ESCALATE,                reason: HARD_GATE_ESCALATE}
      - {id: O5, when: required_false_without_exception,   outcome: ESCALATE,                reason: CONTROL_FAILED}
      - {id: O6, when: required_unknown,                   outcome: REQUEST_INFO,            reason: EVIDENCE_MISSING}
      - {id: O7, when: confidence_below_gate,              outcome: ESCALATE,                reason: LOW_CONFIDENCE_ABSTAIN}
      - {id: O8, when: any_conditional,                    outcome: APPROVE_WITH_CONDITIONS, reason: CONDITIONS}
      - {id: O9, when: always,                             outcome: APPROVE,                 reason: ALL_REQUIRED_CONTROLS_MET}
  6. Challenge

    The Challenger attacks the candidate decision and can force a re-plan or an escalation, never an approval.

    orchestrator.py, lines 108 to 116

    if state is State.CHALLENGE:
        challenge = st.challenge
        if (
            challenge is not None
            and challenge.action == "re_plan"
            and st.challenge_loops < MAX_CHALLENGE_LOOPS
        ):
            return State.PLAN
        return State.REPORT
  7. Sign, audit, replay

    Every side effect needs an HMAC-signed execution contract; every step joins a hash-chained log; any decision replays from it.

    contracts.py, lines 54 to 92replay.py, lines 53 to 78

    def sign(contract: ExecutionContract, *, key: bytes | None = None) -> ExecutionContract:
        secret = key if key is not None else keys.signing_key()
        payload = canonical_json(_payload_for_sign(contract))
        sig = hmac.new(secret, payload.encode("utf-8"), hashlib.sha256).hexdigest()
        return contract.model_copy(update={"signature": sig})
    
    
    def verify(
        contract: ExecutionContract,
        *,
        now: datetime,
        key: bytes | None = None,
    ) -> tuple[bool, str]:
        secret = key if key is not None else keys.signing_key()
        payload = canonical_json(_payload_for_sign(contract))
        expected = hmac.new(secret, payload.encode("utf-8"), hashlib.sha256).hexdigest()
        if not hmac.compare_digest(contract.signature, expected):
            return False, "bad_signature"
        expires = contract.expires_at
        if expires.tzinfo is None:
            expires = expires.replace(tzinfo=UTC)
        now_utc = now if now.tzinfo else now.replace(tzinfo=UTC)
        if now_utc > expires.astimezone(UTC):
            return False, "expired"
        allowed = set(contract.scope.capabilities)
        for step in contract.tool_plan:
            cap = step.capability if isinstance(step, ToolStep) else step["capability"]
            if cap not in allowed:
                return False, "capability_out_of_scope"
        if contract.state not in {
            "issued",
            "pending_approval",
            "approved",
            "executed",
            "rejected",
            "expired",
        }:
            return False, "illegal_state"
        return True, "ok"
    def replay(
        case_id: str,
        *,
        store: Store | None = None,
        policy_root: Path | None = None,
        now: datetime | None = None,
    ) -> ReplayResult:
        """Replay one case. `now` is unused: signatures are checked at each contract's issued_at."""
        del now
        store = store or Store()
        root = policy_root or _POLICY_ROOT
        record = store.get(case_id).decision
        signatures_ok = _signatures(store.contracts_for(case_id))
        if record.outcome_reasons == _HUMAN_REVIEW:
            return _replay_reviewed(case_id, record, store, root, signatures_ok)
        policy, calibration = _policy_for(record, root)
        policy_ok, record_ok, diff, _recomputed = _replay_machine(record, policy, calibration)
        return ReplayResult(
            match=policy_ok and record_ok and signatures_ok and not diff,
            diff=diff,
            checks=ReplayChecks(
                policy_hash=policy_ok,
                record_hash=record_ok,
                contract_signatures=signatures_ok,
            ),
        )

How well it works

Arbiter's evaluation, from its own published results
MeasureResultCaveat
Correct decisions, live, on the first 10 test cases (25 October 2026)10 of 10Wilson 95% interval 0.72 to 1.00
Unsafe approvals, live0
Consistent repeats, live4 of 4
Correct decisions, offline, on the whole test split16 of 16
Model calls that fell back to deterministic logic, live10 of 21Because of the routing issue under Credits; the policy still decided every case
A single-prompt baselineNot measuredEvery baseline call returned HTTP 502
Live example 2, Quillfeather Payroll Cloud LtdEscalate, in 228 s14 Zetaris queries, each with a request id; 4 Nemotron calls, each with a receipt
Zetaris parity with the mirror416 of 416Registry comparisons

The data is synthetic and n is small: these numbers show that the live path works end to end, not a precision estimate. The single-prompt baseline could not be measured, so no claim is made that Arbiter beats it; whichever calls fell back, the policy decided every case.

The method, labels and discussion, in Arbiter's EVALUATION.md

Credits

Arbiter runs on its sponsors’ technology:

  • NVIDIA Nemotron 3.5 Super VL 120B and Nemotron 3.5 Lightning 30B, through Meterless Nightshift;
  • Meterless H-MEM and Scout;
  • the Zetaris Unified Semantic Layer;
  • the CISA Known Exploited Vulnerabilities catalogue, as a committed snapshot.

One deviation, stated plainly, as Arbiter’s README states it. The deep lane was designed for Nemotron 3 Super, which the account was not entitled to. A Nightshift routing rule coerced both lanes to auto, so lane separation was not guaranteed, and Meterless acknowledged the routing bug on 24 October.

I built Arbiter with coding agents, Cursor’s for the early passes and then Claude Code, and reviewed every change, as the README’s “AI and model usage” records.

Start a conversation

Authorship

This is my own work.

It is built on tools I did not make: NVIDIA Nemotron models, Meterless Nightshift, H-MEM and Scout (Arbiter vendors part of Scout in vendor/meterless) and the Zetaris platform.

Updates