Streaming and data integration
Kafka, change data capture and APIs, so events from core systems reach fraud checks, dashboards, the lakehouse and AI within seconds instead of after the night batch.
Move the data as it changes, not once a night
Most institutions still move data in batches: an extract after the end-of-day run, a file transfer in the morning, a report by noon. That was fine when the only consumer was a monthly return. A fraud model, a customer-facing dashboard or an assistant that answers from today's records needs the change the moment it happens.
We build streaming platforms on Apache Kafka, with change data capture that reads changes straight from the core databases, schema management so events can evolve safely, and the connectors, APIs and stream processing that deliver each event to every system that needs it, exactly once where it matters.
When it fits
01
Decisions are a day late
Fraud checks, limits and dashboards work from yesterday's extract, and the business has noticed.
02
Point-to-point feeds have multiplied
Dozens of scripts and file transfers connect systems pair by pair, and nobody knows what breaks when one stops.
03
Core systems are overloaded by extracts
Every team queries the core database for its own copy, and the database spends its night serving reports.
04
AI needs current data
The document index and the model inputs go stale between batch runs, and the assistant answers from last week.
What we do
Kafka platform on OpenShift
Design, build and run Red Hat Streams for Apache Kafka: clusters, topics, security, quotas, monitoring and replication to the second site.
Change data capture
Stream row changes from Oracle, PostgreSQL and other databases with Debezium or GoldenGate, with the outbox pattern where applications publish their own events.
Schema management
Avro, Protobuf and JSON schemas in a registry, with compatibility rules so producers and consumers can change at different speeds.
Connectors and integration
Kafka Connect, Apache NiFi and WSO2 integration for files, SFTP, SWIFT and card networks, legacy systems and SaaS, with every feed monitored.
Stream processing
Filter, join, enrich and aggregate events in flight with Kafka Streams or Flink, for fraud scoring, alerts and real-time views.
Delivery guarantees
Idempotent consumers, dead-letter topics and exactly-once processing where a retry must never post a payment twice.
Event flow
One change, captured once, delivered everywhere
A change in a core system is captured from the database log, written to a topic once, and read by every consumer that needs it at its own pace: the fraud model within milliseconds, the dashboard within a second, the lakehouse and the document index on their own schedule. The source system does the work once.
- Change capture
- A connector reads the database's own change log; the core system is not queried.
- Topics and partitions
- Events stored in order, replicated across brokers, and kept for as long as the policy says.
- Consumers
- A model, a dashboard and an index, each reading independently and able to replay.
- Events in flight
- Each change flows through the lanes to every consumer within seconds.
How a streaming programme runs
01
Map the feeds
Inventory the extracts, file transfers and scripts that move data today, and the consumers that need to be faster.
02
Design
Design topics, schemas, security, retention and delivery guarantees, and the cluster and its replication to the second site.
03
Build the first feed
Build the platform as code, capture one core system's changes, and deliver them to one consumer with a measurable gain.
04
Replace feeds in waves
Move the remaining extracts and transfers onto the platform one by one, retiring each old feed only when the new one is proven.
What you get
- A Kafka platform on OpenShift, built as code, with monitoring and replication to the second site
- Change data capture from the core databases, with the source load measured
- A schema registry with compatibility rules and a change process
- Connectors for the files, networks and systems in scope, each monitored
- Stream processing jobs for the agreed real-time views and alerts
- Runbooks, dashboards and training for your team
What we build with
Open-source streaming with enterprise support, and the connectors your systems need.
Streaming
- Apache Kafka
- Red Hat Streams for Apache Kafka
- Confluent Platform
- Apicurio Registry
- Kafka Streams
- Apache Flink
Capture and integration
- Debezium
- Oracle GoldenGate
- Kafka Connect
- Apache NiFi
- WSO2 Micro Integrator
- WSO2 API Manager
Questions about streaming
Will change data capture slow down our core database?
No. The connector reads the database's own change log, the same mechanism that feeds replication, rather than querying tables. We measure the load during the pilot and show you the figures. In most cases the core database gets lighter, because the extracts that queried it move to the platform.
Can Kafka run on premises and air-gapped?
Yes. Red Hat Streams for Apache Kafka runs on OpenShift in your data centre, with images mirrored inside and no dependency on an outside service. It runs that way in our lab.
How do we avoid duplicate payments or double counting?
By design: idempotent producers and consumers, keys that make duplicates detectable, exactly-once processing where the cost is justified, and dead-letter topics so a bad event is parked and reviewed rather than retried forever.
Do we have to replace every batch job?
No. Batch stays where a nightly or monthly run is enough. Streaming goes where a decision, a customer or a model needs the change now. The platform runs both, and the assessment says which feeds are worth moving.
Lakehouse and analyticsMove your first feed to real time
Tell us which decision is waiting for the night batch. We'll propose a first feed, the platform it needs and how to measure the gain.