Continuous Assurance for a Changing AI System
Connect a model update to affected evidence, a renewed decision and enforceable operating conditions.

The assurance problem after an update
An enterprise knowledge assistant has been accepted for internal policy questions. It retrieves approved documents, cites sources and routes uncertain answers to a human reviewer. A provider model update and a revised retrieval index arrive together. The application still opens and the main benchmark improves, but neither observation establishes that the existing permission to operate remains justified.
This illustrative solution architecture uses a hypothetical system to examine the evidence and decisions required after change. It does not describe a customer deployment or claim measured results.
The team needs to distinguish the previous accepted configuration from the new candidate. The decision record covers a purpose, audience, data boundary and escalation process, not every future version of the application.
Establish the baseline before tracing the change
The baseline includes the provider model identifier, prompt version, retrieval configuration, source-document snapshot, tool permissions and application release. Requirements identify the permitted information sources, citation behavior, access restrictions and conditions for escalation. Evidence records link the evaluated configuration to each requirement.
The change owner records both updates in one change record while retaining their separate dependencies. A revised index can alter retrieval coverage even when the model is unchanged. A provider update can alter answer behavior even when the same documents are retrieved. Combining both into an unexplained “new version” would make later findings harder to investigate.
- System owner: defines the proposed operating scope.
- Technical owner: identifies configuration and dependency changes.
- Evaluator: produces new observations against explicit criteria.
- Reviewer and decision authority: challenge the evidence and decide the permitted use.
Trace impact instead of invalidating everything
Start from the changed objects and follow their links to requirements, tests, evidence and decisions. Citation accuracy, refusal behavior and access-sensitive retrieval may require fresh evaluation. Evidence for an unchanged identity-control component might remain usable if the relevant interface and assumptions are unchanged.
Reuse is a documented judgment. Each reused item retains its original baseline, method and limitations, plus the reason it still supports the new decision. Evidence that cannot be connected to a known configuration stays unresolved. The absence of an incident is not a substitute for the missing record.
The requirement-to-evidence traceability model provides the relationship structure for this review.

Run a focused reassessment campaign
The campaign covers representative questions, inaccessible documents, contradictory sources, unavailable retrieval services and attempts to redirect the assistant outside its role. Repeated runs examine variable behavior. Evaluators retain the actual inputs, relevant context, outputs, evaluator version and interpretation, subject to the organization’s data-handling controls.
Results are compared with criteria agreed before the campaign. A test that fails to execute is recorded as inconclusive, not passed. An improved aggregate score does not cancel an access-control failure. Critical findings remain visible to the reviewer even when most scenarios succeed.
Use the TEVV Plan Template to connect the changed baseline, scenario set, acceptance criteria and evidence references.
Make the decision operational
The review can retain the existing scope, approve a narrower scope, require more evidence, reject the candidate or escalate the decision. A conditional approval needs a named owner, enforceable restriction, review date and evidence that the restriction works. For example, the candidate may remain in a controlled trial while access-sensitive questions continue on the previous baseline.
Operations receives the approved configuration, restrictions and stop conditions. Monitoring then checks those conditions rather than merely displaying model performance. A new dependency change or a breached condition reopens the affected decision.
The records that make the approach reviewable
The useful output is a connected change package: old and new baselines, impact analysis, evidence-reuse decisions, focused test results, unresolved findings, reviewer reasoning and operational instructions. These records explain what the organization permitted and why.
Assurio’s continuous assurance approach connects these records across the lifecycle. A deployment discussion should establish the required integrations, event sources, review responsibilities and control-enforcement points for the environment.
Continue with the next step
Discuss your change-assurance workflow
Bring your system scope, evidence sources and next decision to a focused Assurio discussion.