Staging Review — Quality Dashboard

Enterprise Brain · Operational

Review Summary — PL-I-001 · Compliance Policy Extraction

47 items reviewed · 9 Aug 2026

Recommendation:Proceed — extraction quality is acceptable for continued operation

89.4%

Overall precision

42 of 47 correct

93.6%

Topic classification

44 of 47 correctly classified

0.84

Confidence calibration

Well-calibrated (see curve)

Systematic errors

Patterns identified

Extraction Precision by Topic

Internal Policies

14/15 correct (93%)

Decision Rules

18/21 correct (86%)

SOPs & Runbooks

10/11 correct (91%)

Decision Rules has the lowest precision. Review errors below for adjustment opportunities.

Confidence vs. Actual Accuracy

Points near the diagonal = well-calibrated

Insight: Items in the 0.70–0.75 confidence band are 33% likely to be incorrect. Consider raising the confidence floor to 0.75 for Decision Rules.

Systematic Error Patterns — 3 identified

Missing regulatory references in older documents

3 occurrences · Affects: Decision Rules · Confidence band: 0.70–0.80

Cause: Pre-2020 policy documents reference regulations by description rather than citation number. Extraction prompt expects formal citation format.

Fix: Adjust extraction prompt to accept descriptive regulatory references alongside formal citations

→ Escalate to Knowledge Engineer

Topic misclassification: SOPs classified as Internal Policies

2 occurrences · Affects: Internal Policies, SOPs & Runbooks · Confidence band: 0.75–0.85

Cause: Documents titled "Policy and Procedures for..." contain both policy statements and step-by-step procedures. Extraction classifies entire document as one type.

Fix: Adjust chunking strategy to split policy sections from procedural sections

→ Escalate to Knowledge Engineer

Hallucinated ownership for draft documents

1 occurrence · Affects: Internal Policies · Confidence band: 0.70–0.75

Cause: Draft document (EDD-Procedures-Draft.pdf) has no author metadata. Extraction inferred owner from document path.

Fix: Add extraction rule: if document status = Draft, set hasOwner to null rather than inferring

→ Escalate to Knowledge Engineer

Topic Coverage After This Run

Domain

9/12 ↑

Data

8/12

Technical

6/10

Operational

10/13

GovernanceCross-cutting

4/10 ↑

Review complete · Recommendation: Proceed

Export Report Promote to Production Adjust & Re-extract