Confidence-Aware RAG

An LLM answering a security question sounds exactly as authoritative when it is guessing as when it is right. In most domains that is an annoyance. In vulnerability analysis it is a failure mode with consequences: a confidently wrong answer about a CVE sends an analyst down the wrong remediation path, and nothing in the answer's tone warns them.

The fix is not a better model but a calibrated one — a system whose stated confidence actually tracks whether it is correct. This system retrieves along two paths: dense vector search over CVE, CWE, CAPEC, and ATT&CK text, and Cypher queries against a Neo4j graph linking each vulnerability to its root-cause weakness and the techniques that exploit it.

It then scores its own confidence in three stages: sampling multiple answers and measuring their agreement, checking every factual claim against the graph for contradictions, and passing the answer to a separate GPT-4 judge alongside the evidence.

The Bureau approves of any system prepared to state, in writing, that it does not know.