Comptech Data Pipeline
An open lakehouse on Apache Iceberg, Polaris, Trino and Spark, with scheduled pipelines, dashboards and lineage. A cost-effective alternative to Cloudera, with no lock-in.
Overview
One governed copy of your data, in open formats
Comptech Data Pipeline is a lakehouse: Apache Iceberg tables on S3-compatible Ceph object storage, an Apache Polaris catalog, Trino for SQL and Spark for batch processing. Scheduled pipelines turn raw data into trusted, reporting-ready tables.
It is a cost-effective alternative to Cloudera that complements Oracle rather than replacing it: your core systems keep running, and the lake is fed from them for reporting, analytics and machine learning.
Use cases
Data that regulators and managers trust
Regulatory reporting
Repeatable, traceable pipelines from source systems to report tables.
Customer 360
Join customer data from many systems into one governed view.
MIS dashboards
Superset dashboards on curated tables, with access by group.
ML feature data
Clean, versioned tables to train and score models.
Features
Store, process and query
Iceberg on object storage
Apache Iceberg tables on S3-compatible Ceph storage, in an open table format any engine can read.
Polaris catalog
Apache Polaris REST catalog, so engines share one definition of every table.
Trino SQL
Federated SQL queries across the lake and other sources, with single sign-on.
Spark processing
Apache Spark for large batch transformations.
Scheduled pipelines
Kubeflow pipelines on a schedule, for example daily raw-to-gold banking tables.
Dashboards
Apache Superset for BI dashboards and self-service exploration.
Governance
Know where every number came from
Catalog and lineage
OpenMetadata for the data catalog, lineage and a business glossary.
Access control
Group-based access through Keycloak SSO, with secrets in an OpenBao vault.
Roadmap
Planned, not yet available: row and column masking, Kafka streaming, and Debezium CDC from Oracle and core banking.
Architecture
Raw to gold, with one copy of the data
Data lands in raw Iceberg tables, is cleaned and modelled by Spark and scheduled pipelines, and is served as gold tables to Trino, Superset and ML. OpenMetadata tracks where every table came from.
- SourcesCore banking, Oracle and other systems.
- LakehouseIceberg tables on Ceph, catalogued in Polaris.
- ProcessingSpark and scheduled Kubeflow pipelines.
- ConsumersTrino SQL, Superset dashboards and ML.
Components
What's in the platform
Storage
- Components
- Apache Iceberg on Ceph object storage
- Status
- Available
Catalog
- Components
- Apache Polaris (REST catalog)
- Status
- Available
Query
- Components
- Trino (federated SQL, SSO)
- Status
- Available
Processing
- Components
- Apache Spark
- Status
- Available
Orchestration
- Components
- Kubeflow Pipelines (scheduled)
- Status
- Available
BI
- Components
- Apache Superset
- Status
- Available
Governance
- Components
- OpenMetadata (catalog, lineage, glossary)
- Status
- Available
Security
- Components
- Keycloak SSO, OpenBao, group-based access
- Status
- Available
Masking
- Components
- Row and column masking
- Status
- Roadmap
Streaming and CDC
- Components
- Apache Kafka, Debezium CDC from Oracle and core banking
- Status
- Roadmap
Included
With every Comptech product
On premises or private cloud
Runs in your own data centre, so data stays in the country. Open formats and standard APIs avoid lock-in.
Single sign-on
Keycloak with AD/LDAP, MFA and group-based roles across every tool.
Secrets vault
Credentials and keys held centrally in an OpenBao vault, not in configuration files.
DC/DR design
Designed for a primary data centre and a disaster recovery site.
Local support and SLAs
Comptech engineers in Bangladesh, with Bronze, Silver or Gold SLAs.
Pilot and training
A pilot in 4 to 6 weeks, plus training for your team.
From pilot to production
01
Discover
Agree the first use case, the systems involved and the success criteria.
02
Pilot
Deploy on your infrastructure and prove the use case in 4 to 6 weeks.
03
Production
Add the DR site, single sign-on and the remaining teams and workloads.
04
Support
Local support under a Bronze, Silver or Gold SLA, with training for your team.
Questions
Does it replace Oracle?
No. It complements Oracle. Core systems keep running, and the lake is fed from them. Change data capture from Oracle with Debezium is on the roadmap.
How does it compare with Cloudera?
Both provide a data lake with SQL and processing engines. Comptech Data Pipeline uses open table formats without per-node subscription licensing, so it is designed to cost less and keep your data portable.
How long does a pilot take?
A typical pilot runs 4 to 6 weeks on your hardware or private cloud, with one agreed use case and success criteria set at the start.
What support do you offer?
Local support from Comptech engineers in Bangladesh, with Bronze, Silver and Gold SLA levels. We also train your team to run the platform day to day.
Are we locked in?
No. The platforms use open formats and standard APIs, so your workloads and data stay portable.
Request a pilot
Tell us which reports or dashboards matter most. We'll plan a 4 to 6 week pilot on your data.
Related services