Real-Time Data Architecture for Enterprise AI

Batch, Near Real Time And Real Time Are Different Design Choices

These are different service designs, not moral rankings. There is no universal definition for "near real time".

Each programme should set latency SLOs based on the business action, the acceptable delay, and the cost of missing the decision window.

This framing prevents overengineering: many teams adopt streaming before proving the business actually needs it. Where existing batch flows still meet the decision window, legacy pipeline modernization is usually the better investment than replacing them with streaming.

Latency Tier Comparison

These bands are practical starting points, not universal SLAs. Set the final latency target from the business decision window and validate it against the enterprise data integration strategy.

• Start with the business decision window and user expectation.
• Define latency, freshness, and recovery targets explicitly.
• Use streaming only where the decision window justifies the operating cost.
• Treat batch and near real time as valid designs when they fit the workflow better.

Identify Events And Decision Triggers

Identify events and decision triggers in business terms first. Define what happened, which entity it applies to, and which downstream action may follow.

Then model the event schema, key, ordering requirement, and retention strategy around that meaning.

This is where many systems fail. They stream technical updates without clarifying which business event the AI workflow is actually reacting to.

• Define the business event, entity, and downstream action clearly.
• Choose keys and partitions from business identity and access patterns.
• Specify which events must be replayable and for how long.
• Document which consumers depend on ordering and which do not.

Design The Streaming And Integration Layer

Change Data Capture And Stream Processing

Change data capture (CDC) should be an explicit part of the ingestion design when operational database changes must enter the event path without application level dual writes or frequent polling.

Define snapshot behavior, source offsets, schema change handling, transaction metadata, delete semantics, and recovery ownership for every CDC connector.

The Debezium source connector documentation provides the implementation reference for capturing database changes as structured events.

Use a stream processor when events require stateful enrichment, joins, windows, aggregation, event time handling, or late data policies before consumption.

Keep processing state checkpointed and replayable, and test watermark and lateness behavior against the business decision window.

See the Apache Flink streaming analytics documentation for event time, watermarks, windows, and late event handling.

Map Sources To Ingestion Paths

Do not treat every source system as if it should enter the platform through the same ingestion mechanism.

Operational databases, application events, SaaS platforms, files, device telemetry, and external partner feeds have different change patterns, ownership models, and recovery constraints.

A database table may be best captured through CDC, while an application event should usually be published directly from the business workflow that owns the state change.

File based and partner feeds may still require scheduled ingestion even when downstream processing is fast.

For each source, document who owns the producer, how a missed event is recovered, what timestamp is authoritative, and whether the source can reproduce historical changes.

This avoids a common architecture gap where the broker is reliable but the source integration is not.

The streaming layer is only as dependable as the path that creates the event and the process that repairs missing or malformed records.

• Choose the ingestion pattern from the source behavior and business ownership.
• Record the authoritative event time and source identifier for every event type.
• Define how missing records are backfilled without creating duplicate business effects.
• Assign operational ownership for connectors, producer applications, and source recovery.

Partitioning And Ordering Boundaries

Partitioning should follow the smallest business boundary that needs ordered processing.

Customer, account, order, device, or shipment identifiers are often better partition keys than technical source identifiers. They keep related changes together while allowing unrelated entities to process in parallel.

Global ordering may look simpler in a diagram, but it creates unnecessary coordination and limits scale. Most enterprise decisions only require consistent order within a specific entity or workflow.

Test partition choices against both throughput and skew. A key that sends a large percentage of traffic to one partition can create lag even when the cluster has spare capacity elsewhere.

Plan for key growth, hot entities, and reassignment before the system reaches production scale.

Consumers should also document what happens when events for different entities arrive at different speeds, because business logic should not assume a sequence that the transport layer does not guarantee.

• Partition by a business identity that matches the ordering requirement.
• Measure key distribution and identify hot partitions during load testing.
• Document where order is guaranteed and where consumers must tolerate reordering.

Handle Event Time And Late Data Deliberately

Real time systems often process events after the moment when the business activity actually occurred.

Network delays, mobile devices, connector pauses, retries, and regional outages can all make an older event arrive after a newer one. For this reason, distinguish event time from processing time.

Event time describes when the business activity happened, while processing time describes when the platform handled it. The difference matters for windows, alerts, aggregates, and model features.

Define a lateness policy for each flow instead of allowing the stream processor to decide implicitly.

Some late events should update a materialized view, some should trigger a correction event, and others may be routed for reconciliation without changing an already completed decision.

Watermarks and windows are implementation tools, but the business rule must state how much lateness is acceptable and what correction behavior is required.

• Carry event time and processing time as separate metadata.
• Set lateness tolerance from the business consequence of delayed data.
• Define correction behavior for events that arrive after a decision window closes.

Illustrative End To End Latency Budget

Use latency figures as planning examples rather than fixed promises. Consider a well tuned flow within one region under normal load.

Such a flow might target roughly 20 to 150 milliseconds for source capture, 10 to 100 milliseconds for broker transport, and 20 to 300 milliseconds for stream processing.

It might then need 5 to 100 milliseconds for a state or feature lookup, 20 to 250 milliseconds for model or rules evaluation, and 20 to 200 milliseconds for the final action handoff.

In that kind of design, an end to end objective around 100 milliseconds to 2 seconds may be reasonable for selected decision paths.

Those figures are illustrative only. They are not universal benchmarks or guaranteed SLAs.

Actual latency can be materially higher or lower depending on geography, network hops, payload size, partition skew, state access patterns, model complexity, downstream API behavior, traffic bursts, and retry policies.

The amount of resilience built into the path also matters. The useful target is the one that still meets the business decision window under realistic peak and recovery conditions.

Treat these ranges as an architecture sizing aid. Measure your own path at normal load, peak load, and recovery load before committing to a service objective.

Streaming and delivery guarantees

Real time data architecture depends on a streaming path built around explicit event contracts, buffering, replay, and predictable consumer behavior.

Define end to end delivery semantics for each flow and align producer, broker, state layer, and consumer decisions.

Example: Partition by customer_id to ensure per customer event ordering without global sequencing.

• Scope ordering to a clear business key or partition (for example, customer_id).
• Choose delivery semantics (at most once, at least once, exactly once) per flow and document assumptions.
• Document required metadata in producer and consumer contracts (timestamps, version, trace ids).

Capacity, Back Pressure And Burst Planning

Design for burst behavior, not only average throughput.

Enterprise event volumes frequently rise around payroll cycles, promotions, market openings, month end activity, batch releases, or recovery after an upstream outage.

A platform that handles the normal rate comfortably can still fail when producers recover and publish several hours of accumulated changes in a short period.

Capacity planning should therefore include sustained traffic, expected peaks, recovery bursts, and the maximum lag the business can tolerate.

Back pressure must be visible and controlled across the whole path. If consumers slow down, the system should preserve data without exhausting memory, overwhelming downstream APIs, or hiding a growing queue.

Define thresholds for consumer lag, processing duration, storage use, and retry volume. Then connect those thresholds to scaling rules and operational runbooks.

The goal is not to eliminate every queue, but to make queue growth predictable and recoverable.

• Load test sustained demand, peak demand, and recovery bursts separately.
• Set explicit limits for acceptable lag and time to recover.
• Protect downstream services with controlled concurrency and bounded retries.
• Alert on the rate of backlog growth, not only the current queue size.

Schema and contract evolution

Version event contracts and enforce compatibility rules early. Prefer additive changes, automate compatibility checks in CI, and publish a changelog so consumers can upgrade predictably.

Example: Add new optional fields with defaults; avoid renames or removals without a migration plan.

• Run schema compatibility checks in the pipeline and CI.
• Publish contract version and changelog with each change.
• Provide a migration plan for incompatible consumers (dual write, adapters, or versioned topics).

State and feature serving

Keep state building logic reproducible and versioned. Use materialized low latency stores for online scoring and keep the same computations in offline pipelines to ensure parity.

Example: Materialize sliding window aggregates to a cache or key value store for fast online scoring while running identical transforms in batch for offline training.

• Version feature computation and transformation code.
• Ensure online/offline parity where needed; use a feature store when reuse and consistency justify it.
• Choose stores tuned for required read latency and throughput.

Validate Data Quality Before Decisioning

Fast delivery does not make low quality data useful.

Apply validation as events enter the decision path so malformed identifiers, impossible values, missing required context, or stale reference data do not silently influence automated actions.

Validation should distinguish between events that can be corrected automatically, events that can continue with a documented default, and events that must be quarantined for investigation.

Quality rules should also be observable as metrics. Track rejected events, schema violations, null rates, freshness, duplicate rates, and enrichment misses by source and event type.

Sudden changes in these measures often reveal source defects before model or business metrics deteriorate. For AI workloads, this layer also helps separate model quality issues from upstream data quality issues.

• Validate required identifiers, timestamps, schema versions, and critical business fields.
• Quarantine invalid events without blocking healthy traffic where the workflow allows it.
• Measure data quality by source so ownership and remediation are clear.

Retries, DLQs, replay, and back pressure

Design retry policies, DLQ rules, and back pressure handling before production.

Build for safe replay: durable retention, checkpoints, and runbooks minimize risk when reprocessing. Use idempotent consumers or deduplication to avoid repeated business effects.

Example: Persist processing offsets, use an append only log retention window, and re run consumers against a topic partition to rebuild state.

• Define retry policies and DLQ escalation rules.
• Implement idempotency keys or dedupe logic in consumers to make replay safe.
• Document replay runbooks and required checkpoints for state rebuilds.

Define Recovery Objectives And Failure Domains

Resilience improves when teams design for specific failure domains rather than a generic idea of high availability.

A connector can fail while the broker remains healthy. A stream processor can restart while the source continues publishing. A state store can become unavailable even though events are still retained.

Each failure mode needs a recovery path that explains what data is preserved, which components pause, and how processing resumes without producing duplicate actions.

Set recovery objectives for the event path itself. Define how much data loss is acceptable and how quickly a consumer must catch up.

Also define how long retained events remain available for replay, and how state is rebuilt after corruption or deployment failure.

These objectives should be tested with controlled fault exercises. A recovery design that exists only in documentation is not enough for a workload that supports operational decisions.

• Document failure behavior separately for producers, brokers, processors, state stores, and executors.
• Set retention long enough to support the required replay and investigation window.
• Test restart, replay, state rebuild, and catch up procedures before production incidents.
• Confirm that recovery does not repeat irreversible business actions.

Governance, observability and execution separation

Separate decisioning from execution when actions are material. Propagate identity, policy, and lineage metadata end to end and monitor key signals to detect drift or failures.

Example: Emit a decision event with audit metadata to a protected topic; a separate executor service reads it after policy validation.

• Instrument event lag, processing failures, state freshness, and decision quality.
• Propagate identity and lineage metadata through the path.
• Keep execution paths idempotent and reversible where possible.

Secure The Event Path And Its Operational Data

Security controls should follow events from production through storage, processing, and consumption.

Authenticate producers and consumers, authorize access at the smallest practical topic or data domain boundary, and encrypt data in transit and at rest.

Sensitive fields should be minimized in the event itself when downstream services do not need them.

Where personal or regulated data is required, document retention, masking, deletion, and access review responsibilities.

Operational metadata also deserves protection. Trace identifiers, error payloads, dead letter records, and replay archives can contain business context that is as sensitive as the primary event.

Logging and observability pipelines should therefore apply the same classification and access principles as the main data path.

This is especially important when incident tooling is accessible to a broader support group than the production application.

• Authenticate every producer and consumer and limit access by business need.
• Keep sensitive fields out of events when a token or reference is sufficient.
• Apply retention and access controls to logs, dead letter data, and replay storage.
• Audit privileged access to event infrastructure and protected decision topics.

For current delivery semantics, idempotence, transactions, and partition scoped ordering, see the Apache Kafka 4.3 design documentation and current documentation.

Build for replay and resilience

Design on append only logs, durable retention, checkpoints, and snapshots so you can replay events or rebuild state after failures without manual migration.

Durable logs plus checkpoints are the core resilience pattern.

Common Mistakes And Choosing The Right Latency

Common mistakes include choosing streaming without a business need, assuming global ordering, skipping idempotency design, hiding state logic inside opaque services, and underestimating recovery work.

Choose the right latency by balancing value against complexity. Faster is not automatically better if it increases operational fragility without changing the business outcome meaningfully.

• Do not define real time by slogan; define it by business value.
• Avoid global order assumptions unless they are truly required and achievable.
• Design duplicate prevention before launch, not after incident review.
• Use the lightest latency model that still supports the decision window.

Frequently Asked Questions