10 vectors, 5 maturity levels, one rule
The approach is control-centric, not actuarial: the system doesn't try to predict how likely you are to have a claim. It measures whether you've implemented and maintained technical defenses against ten specific, well-documented ways AI systems fail in production. Stronger controls score lower risk. That's the whole model — no hidden actuarial layer behind it.
The ten vectors
| Vector | What it looks for |
|---|---|
| Uncontrolled internet access | Can the model reach arbitrary external endpoints without gating? |
| Hallucination / OOD behavior | Confidence and out-of-distribution controls on generated output |
| Model drift | Monitoring for output/behavior change over time post-deployment |
| Data poisoning | Controls on training/fine-tuning data provenance and integrity |
| Prompt injection | Defenses against adversarial or embedded instruction content |
| Model extraction / IP theft | Rate limiting, auth, and output controls against model theft via querying |
| Bias & disparate impact | Testing and monitoring for unequal outcomes across groups |
| Vendor concentration | Dependency risk on a single model provider or API |
| Explainability | Whether decisions can be traced and justified after the fact |
| Human oversight | Whether a human is actually in the loop where it matters |
The failure modes behind these aren't abstract. One of the framing examples from the original design work: an LLM generates fabricated case law citations inside a legal research tool, a lawyer files them in a brief, and the result is sanctions or a malpractice claim. That's a hallucination/OOD failure with a direct, traceable liability path — exactly the kind of thing an exclusion clause is written to avoid paying for, and exactly the kind of thing a real control (confidence thresholds, citation verification, human review before filing) can actually prevent.
A maturity scale, not a pass/fail
Every vector — plus seven governance dimensions mapped against frameworks like NIST AI RMF, ISO 42001, the EU AI Act, MITRE ATLAS, and the OWASP LLM Top 10 — is scored on the same 0–4 scale:
| 0 | No control |
| 1 | Ad-hoc — exists informally, not systematized |
| 2 | Implemented — exists as a defined process or system |
| 3 | Monitored — implemented and actively observed |
| 4 | Automated — enforced without relying on a human remembering to do it |
Scanner-first, clarification-second
Scores start from automated evidence: repository scanning, documentation review, CI/CD
execution logs, targeted tests. But the system doesn't treat weak surface signals as proof.
A file named *hallucination* or *ood* in a test directory is a hint, not a
finding — it only counts once the system can cross-reference that the test actually runs in
CI, not just that it exists in the repo.
A file name, a library import, a prompt instruction, or a configured alert alone does not prove production enforcement, execution frequency, or effectiveness.
When the automated pass misses something real, there's a dispute path. If a company can point to evidence the scanner didn't have access to or didn't recognize, the score gets re-evaluated — not on trust, but on the new evidence. A worked example from the original design: the scanner flags model-extraction maturity at 1 (auth only, no rate limiting or output controls visible). The company points out they run differential privacy on outputs, which the scanner isn't positioned to detect from code alone. Once that's confirmed active, the score moves to a 3. The scan doesn't get overridden by an assertion — it gets corrected by evidence.
Combining scores without breaking the direction
Technical and governance scores blend into one number using a weighted formula — 40% technical risk, 60% governance — but the two inputs run in opposite directions (governance maturity is scored high-is-good; risk is scored high-is-bad), which is an easy place to get the math backwards. It's also exactly where this project caught itself making that mistake mid-build — more on that, and what else got corrected along the way, in the next piece: what we learned building this across three AI coding tools.