Skip to content
← Back to work

AuditAhead

Multi-agent pre-submission QA for Ontario pharmacy claims, producing a tamper-evident evidence packet an auditor can verify without trusting the host.

Role
Senior AI Engineer
AIM Inc.
Period
2026
Status
In development
4
Specialised agents
P-384 HSM
Signing key
Append-only
Evidence chain
None
Verifier backend

Overview

A pre-submission QA and post-audit evidence layer that runs upstream of a pharmacist's own claim submission. For each billed service it answers the regulator's core question — is there a valid, matching consent on file, and does the claim hold up against the formulary and Limited-Use rules — then seals the answer into an evidence packet signed with an HSM-backed key and anchored to a third-party timestamp.

The problem

On an audit, a pharmacy has to prove a claim was justified at the time it was billed. Reconstructing that months later from a PMS is slow, incomplete, and the pharmacy carries the clawback risk. The hard part is not classifying claims — it is producing evidence an adversarial third party will accept without having to trust the system that generated it.

Approach

  • 01Four specialised LangGraph agents in sequence — consent compliance, claims analysis, forensic review, attestation authoring — each emitting a hash-chained receipt that the next one verifies before proceeding.
  • 02A PHI redaction boundary after the first agent: everything downstream sees a redacted receipt only, so the model never handles patient-identifying data.
  • 03A human-in-the-loop countersign gate. The graph pauses at a briefing card and the pharmacist remains the legal trust anchor; nothing seals without an explicit decision.
  • 04A zero-LLM cryptographic path — ECDSA P-384 signing in Key Vault HSM, RFC 3161 trusted timestamps, and an append-only Azure SQL Ledger chain.
  • 05An anonymous in-browser verifier with no backend and no login, so an auditor re-verifies the packet against a published public key rather than taking the platform's word for it.
  • 06A batch tier that turns a folder of PDFs into a reviewable board — deterministic reconciliation pairs consent to billing, then fans matched claims through the pipeline at bounded concurrency.

Decisions & trade-offs

No LLM anywhere in the crypto path, and no retries

Signing, hashing, redaction and ledger writes are pure Python, physically not bound to the LLM modules. One attempt, raise, quarantine — because a retry on a signing operation risks a double-append to an append-only ledger, and a ledger that can be appended twice is not evidence. The trusted-timestamp service is the single graceful-degradation exception: if it is unreachable the receipt carries a null token rather than blocking the claim, and the signature is computed excluding that field so the chain stays independent of TSA reachability.

Redaction is an inverted allow-list that fails closed

Every field is dropped unless explicitly allow-listed with a purpose-of-use rationale, and an unknown field raises rather than passing through. A deny-list would have been less work and would have silently leaked the first field nobody thought of — which under PHIPA is the failure that matters. The cost is that adding a legitimate new field is a deliberate code change, which is the correct friction.

The adversarial reviewer cannot change the verdict

A challenger builds an inverted flag-set and asks the reasoning model to defend it. That inversion is recorded on the receipt and surfaces as a minority report when it is both confident and disagrees — but the main verdict is untouched, and flag codes come from deterministic comparison, not from the model. The LLM can reshape prose; it cannot mint a flag. Letting it do both would have been simpler and would have made the audit trail unfalsifiable in the wrong direction.

The re-query loop is hard-capped at three iterations

When the reviewer's confidence is low it can route back for another pass. Uncapped, that is an unbounded spend loop on exactly the claims that are hardest to resolve. Three trades some recall for a bounded worst case, and a claim that has not converged by then goes to the human rather than looping — which is where it belonged anyway.

The verifier has no backend

It is a static single-page app that runs the chain check, ECDSA verification via Web Crypto, canonical-JSON byte-parity and the RFC 3161 check entirely in the browser, against a pinned published key. The canonical hashing recipe — NFC normalisation, sorted keys, no whitespace — is mirrored exactly in three codebases so the bytes agree. An auditor should have to trust the mathematics, not our uptime.

What it does not do

  • It deliberately does not submit anything to the Ministry or HNS. It sits upstream of the pharmacist's own submission — scope creep into submission would make it a system of record and change the regulatory posture entirely.
  • Hybrid RAG retrieval is opt-in and not deployed by default; the shipped default is a deterministic keyword ranker over small in-package corpora. That keeps the container lean and the test suite offline, at the cost of recall on paraphrased policy questions.
  • The three-iteration re-query cap means low-confidence edge cases resolve to human review rather than to a better automated answer.
  • Document extraction currently calls Document Intelligence directly because the deployed extraction Logic App is still a stub.

Stack

PythonFastAPILangGraphAzure OpenAIAzure SQL LedgerAzure Key Vault (HSM)ECDSA P-384RFC 3161Document IntelligenceWeb PubSubNext.jsViteBicepContainer Apps