Assurio · Illustrative case study

Continuous Assurance for a Changing AI System

Connect a model update to affected evidence, a renewed decision and enforceable operating conditions.

Continuous Assurance for a Changing AI System: Accepted baseline  -  Record the permitted use; Affected evidence  -  Trace dependencies; Decision review  -  Set conditions and authority
Connect accepted baseline, affected evidence and decision review. View full-size image
01 / Assurance in practice

The assurance problem after an update

An enterprise knowledge assistant has been accepted for internal policy questions. It retrieves approved documents, cites sources and routes uncertain answers to a human reviewer. A provider model update and a revised retrieval index arrive together. The application still opens and the main benchmark improves, but neither observation establishes that the existing permission to operate remains justified.

This illustrative solution architecture uses a hypothetical system to examine the evidence and decisions required after change. It does not describe a customer deployment or claim measured results.

The team needs to distinguish the previous accepted configuration from the new candidate. The decision record covers a purpose, audience, data boundary and escalation process, not every future version of the application.

02 / Assurance in practice

Establish the baseline before tracing the change

The baseline includes the provider model identifier, prompt version, retrieval configuration, source-document snapshot, tool permissions and application release. Requirements identify the permitted information sources, citation behavior, access restrictions and conditions for escalation. Evidence records link the evaluated configuration to each requirement.

The change owner records both updates in one change record while retaining their separate dependencies. A revised index can alter retrieval coverage even when the model is unchanged. A provider update can alter answer behavior even when the same documents are retrieved. Combining both into an unexplained “new version” would make later findings harder to investigate.

  • System owner: defines the proposed operating scope.
  • Technical owner: identifies configuration and dependency changes.
  • Evaluator: produces new observations against explicit criteria.
  • Reviewer and decision authority: challenge the evidence and decide the permitted use.
03 / Assurance in practice

Trace impact instead of invalidating everything

Start from the changed objects and follow their links to requirements, tests, evidence and decisions. Citation accuracy, refusal behavior and access-sensitive retrieval may require fresh evaluation. Evidence for an unchanged identity-control component might remain usable if the relevant interface and assumptions are unchanged.

Reuse is a documented judgment. Each reused item retains its original baseline, method and limitations, plus the reason it still supports the new decision. Evidence that cannot be connected to a known configuration stays unresolved. The absence of an incident is not a substitute for the missing record.

The requirement-to-evidence traceability model provides the relationship structure for this review.

Connect the evidence to the operating decision: Accepted baseline  -  Record the permitted use; Material change  -  Identify what changed; Affected evidence  -  Trace dependencies; Focused evaluation  -  Challenge the changed behavior; Decision review  -  Set conditions and authority; Operational action  -  Apply and monitor the decision
Connect accepted baseline, material change, affected evidence, focused evaluation, decision review and operational action. View full-size image
Accepted baselineRecord the permitted use
Material changeIdentify what changed
Affected evidenceTrace dependencies
Focused evaluationChallenge the changed behavior
Decision reviewSet conditions and authority
Operational actionApply and monitor the decision
05 / Assurance in practice

Run a focused reassessment campaign

The campaign covers representative questions, inaccessible documents, contradictory sources, unavailable retrieval services and attempts to redirect the assistant outside its role. Repeated runs examine variable behavior. Evaluators retain the actual inputs, relevant context, outputs, evaluator version and interpretation, subject to the organization’s data-handling controls.

Results are compared with criteria agreed before the campaign. A test that fails to execute is recorded as inconclusive, not passed. An improved aggregate score does not cancel an access-control failure. Critical findings remain visible to the reviewer even when most scenarios succeed.

Use the TEVV Plan Template to connect the changed baseline, scenario set, acceptance criteria and evidence references.

06 / Assurance in practice

Make the decision operational

The review can retain the existing scope, approve a narrower scope, require more evidence, reject the candidate or escalate the decision. A conditional approval needs a named owner, enforceable restriction, review date and evidence that the restriction works. For example, the candidate may remain in a controlled trial while access-sensitive questions continue on the previous baseline.

Operations receives the approved configuration, restrictions and stop conditions. Monitoring then checks those conditions rather than merely displaying model performance. A new dependency change or a breached condition reopens the affected decision.

07 / Assurance in practice

The records that make the approach reviewable

The useful output is a connected change package: old and new baselines, impact analysis, evidence-reuse decisions, focused test results, unresolved findings, reviewer reasoning and operational instructions. These records explain what the organization permitted and why.

Assurio’s continuous assurance approach connects these records across the lifecycle. A deployment discussion should establish the required integrations, event sources, review responsibilities and control-enforcement points for the environment.

Discuss your change-assurance workflow

Bring your system scope, evidence sources and next decision to a focused Assurio discussion.

Request an Assurio demo

Record the revalidation review

Contact Us

Share a few details about your project, and we’ll get back to you soon.

Let's Talk About Your Project