Routing and reliability
Load balancing, fallbacks, retries, timeouts, cooldowns and caching keep AI applications running, and data-residency routing keeps sensitive data on on-prem models.
Routing and reliability
The right model for every request, even when one fails
The gateway decides where each request goes: across several deployments of the same model, to a fallback when a model is down or a request is too large, and only to on-prem models when the data is sensitive.
Retries, timeouts and cooldowns stop one slow or failing provider from taking applications down with it, and caching cuts cost and latency for repeated questions.
Why it matters
Reliability and residency in one place
Keep applications up
Requests move to a healthy deployment when one fails.
Control data residency
Sensitive data is routed only to models in your data centre.
Lower cost and latency
Repeated questions are answered from cache.
Routing
Load balancing and fallbacks
Load balancing strategies
Spread traffic by weight, least busy, lowest latency, lowest cost or usage limits.
Fallbacks
Move to a backup model or provider when the first choice fails.
Context-window fallbacks
Send requests that are too long for one model to a model with a larger context window.
Content-policy fallbacks
Send a request elsewhere when a provider refuses it on content-policy grounds, where your policy allows.
Retries and timeouts
Retry transient errors and stop waiting on slow responses, per model.
Cooldowns
Take a failing deployment out of rotation for a set time, then bring it back.
Caching and residency
Answer faster, keep data in-country
Response caching
Identical requests are answered from cache.
Semantic caching
Questions that mean the same thing are answered from cache too.
Data-residency routing
Requests tagged or detected as sensitive only go to on-prem models; cloud models are never tried.
Strategies
Routing options at a glance
Weighted
- What it does
- Splits traffic by fixed shares
- When to use it
- Gradual rollout of a new model
Least busy
- What it does
- Sends to the deployment with fewest requests in flight
- When to use it
- Several on-prem GPU servers
Lowest latency
- What it does
- Sends to the fastest recent deployment
- When to use it
- Interactive assistants
Lowest cost
- What it does
- Sends to the cheapest suitable model
- When to use it
- Batch and background jobs
Usage-based
- What it does
- Stays within each deployment's rate limits
- When to use it
- Provider quotas
Data-residency
- What it does
- Keeps sensitive requests on on-prem models
- When to use it
- Customer and classified data
Included
With every Comptech product
On premises or air-gapped
Runs in your own data centre, so prompts and responses stay under your control.
Single sign-on
SAML and OIDC sign-in, with AD/LDAP groups mapped to teams and roles.
Secrets vault
Provider credentials held centrally, never handed to applications.
DC/DR design
Designed for a primary data centre and a disaster recovery site.
Local support
Comptech engineers in Bangladesh for deployment and support.
Pilot and training
A focused pilot on one use case, plus training for platform and application teams.
Explore
Other Comptech products
Comptech Container Platform
Enterprise Kubernetes with GitOps, a private registry and DC/DR.
Comptech AI Platform
Private generative AI and MLOps on your own GPUs.
Comptech Data Pipeline
Open lakehouse with SQL, pipelines, dashboards and lineage.
Comptech DocuVault
Document management for banks: capture, document AI, workflows and records.
Comptech AIOps
Monitoring, service desk, AI root cause and DR assurance for bank infrastructure.
Comptech VisionVault
On-prem AI video intelligence, transcription and evidence management.
Comptech SecOps
Security for every workload across your Comptech platforms.
Request a pilot
Tell us which applications and models you want to bring under control first. We'll plan an AI Gateway pilot around them.
Related services