How the platform works
An engineering overview of how ShaSentra provisions, schedules, scales and meters GPU workloads — and the design decisions behind it.
Overview
ShaSentra is an orchestration layer. It does not try to be a new runtime or a new model format — it sits between workloads and accelerators, and takes responsibility for the parts teams usually end up hand-rolling: provisioning, placement, autoscaling, health, and cost attribution.
The system splits into a stateless control plane, a durable job queue, and workers that reconcile desired state against a cluster. Everything a user does — deploy a model, open a notebook, run a pipeline — becomes a record in Postgres and a job on a queue.
Control plane
The API is a stateless Go service. Statelessness is the point: it lets the control plane scale horizontally behind an autoscaler, and means no request depends on which instance happens to serve it.
- Postgres holds durable state — deployments, pipelines, autoscaling rules, users, quotas.
- Redis holds shared ephemeral state — rate limiting, sessions, and the provisioning queue.
- Next.js serves the dashboard and talks to the same public API a customer would use.
Asynchronous provisioning
Provisioning a GPU workload is slow — pulling images, scheduling onto a node, waiting for readiness. Doing that inside a request handler means the caller blocks for tens of seconds and any restart loses the work.
Instead, creating a deployment writes status=provisioning to Postgres, pushes
the deployment ID onto a queue, and returns immediately. A worker pool consumes the queue
and performs the slow work, updating status as it progresses.
POST /v1/deployments
→ persist record (status: provisioning)
→ enqueue deployment id
→ 202 Accepted (~milliseconds)
worker
→ claim job from stream
→ build / pull image
→ schedule onto accelerator
→ health check
→ status: running
The queue is backed by Redis Streams with consumer groups, so a job is
claimed by exactly one worker and survives a worker dying mid-flight. Workers drain
cleanly on SIGTERM rather than abandoning in-flight provisions.
Scaling and scale-to-zero
Autoscaling rules are stored per deployment with explicit minimum and maximum replica counts. A minimum of zero is a first-class case, not an edge case: idle deployments release their accelerators entirely.
This is the single biggest lever on cost. GPU capacity is expensive and most workloads are bursty — an inference endpoint serving a demo, a notebook left open overnight, a pipeline that runs twice a day. Reserved-capacity pricing punishes exactly those patterns.
| Pattern | Reserved capacity | Scale-to-zero |
|---|---|---|
| Demo endpoint, occasional traffic | Billed 24/7 | Billed on request |
| Notebook left open | Billed 24/7 | Released when idle |
| Scheduled batch job | Billed 24/7 | Billed during the run |
| Steady production inference | Comparable | Comparable |
Observability
The platform exposes a Prometheus-compatible /metrics endpoint covering
deployment counts by status, pipeline and autoscaling activity, and uptime. Per-GPU
utilisation comes from NVIDIA DCGM, federated in from the cluster.
The goal is that a user can answer two questions without leaving the platform: what is running right now, and what is it costing me. Those turn out to be the same question viewed from different ends.
Metering
Usage is measured at the accelerator-second. Every state transition a workload makes is recorded, and billing derives from those events rather than from a periodic poll — so a workload that ran for ninety seconds is billed for ninety seconds.
Design decisions
Redis Streams rather than Kafka
Provisioning is low-volume — tens to hundreds of jobs, not millions of events. Redis Streams gives durable consumer groups and at-least-once delivery while reusing infrastructure already deployed for rate limiting and sessions. Kafka would add an operational component earning nothing at this scale.
The queue sits behind an interface, so the broker can change without touching the handlers. Kafka becomes justified for the high-volume metering firehose — per-GPU-second events from many nodes, multiple independent consumers, replay — not for provisioning.
Stateless control plane
Every instance can serve every request. Scaling is adding instances; recovery is replacing one. Shared state lives in Postgres and Redis where it can be reasoned about, not in process memory where it disappears silently.
Standard formats throughout
Containers, standard model formats, an OpenAI-compatible inference surface. Portability is a deliberate constraint: a platform that is hard to leave is one people hesitate to adopt. Your images, weights and data stay exportable.