RetailStreaming dataLakehouse

Yesterday's batch became a live operating signal.

Stores, warehouses and e-commerce all had “inventory,” but each meant something different. We built one event contract and one semantic layer so live decisions did not create a second version of the business.

<1 secDashboard latency
4.8MEvents per hour
99.7%Contract validity
12 hrFreshness recovered
01 · Constraint

Faster data would only accelerate disagreement.

The overnight warehouse served finance well, but live replenishment and pricing could not wait. Source systems emitted incompatible product keys, late events and corrections without a shared contract.

Three inventories

Point-of-sale, warehouse and online stock diverged throughout the day.

Late truth

Returns and cancellations arrived after downstream aggregates had closed.

Schema surprise

Producer changes broke consumers silently until the next morning.

02 · Data path

Streaming and analytics shared contracts, not copies.

ProduceDomain events

Sale, return, transfer and stock adjustment

ProtectSchema registry

Compatibility gates before producer release

ComputeFlink state

Watermarks, dedupe and late-event repair

ServeOperational view

Sub-second inventory and pricing API

LearnLakehouse

Durable bronze, silver and semantic gold

Failure path: invalid events entered a replayable quarantine topic. The last trusted aggregate remained available while the producer owner fixed the contract.
03 · Decisions

The system treated correction as normal.

D-01Event time over arrival time

Watermarks and bounded lateness preserved business order during network disruption.

D-02One semantic layer

dbt metrics served both live APIs and executive reporting.

D-03Replay as a product feature

Every materialized view could be rebuilt deterministically from retained events.

04 · Outcome

Operations acted during the trading day.

MeasureBeforeAfter
Inventory freshness12–24 hoursSub-second
Schema incidents6 per quarterBlocked pre-release
Correction handlingManual SQL patchDeterministic replay
Business definitionPer dashboardShared semantic model
Apache KafkaFlinkSchema RegistryDatabricksdbtRedshiftOpenLineage
Design insight: the real product was not the dashboard. It was a trustworthy inventory contract that every channel could consume.

Turn your batch estate into a live system.

We can identify the first event domain worth moving—and the contracts needed to keep it trustworthy.

Map my real-time path →