Module 03 — Core Architecture
How the system is structured.
Every Hyperion deployment is built across four distinct layers. Each layer has a defined responsibility and a clear interface with the layers above and below it. This isn't an abstract framework — it's the actual structure of the systems we build.
Layer 01
Physical Compute
The hardware layer: GPU nodes, interconnect fabric, storage systems, and power/cooling. For bare-metal builds we design this from the ground up. For cloud deployments we make deliberate choices about instance types, placement groups, and network topology rather than accepting cloud provider defaults.
Components
GPU nodes
H100 SXM5 / A100 SXM4 / custom configs
Interconnect
InfiniBand HDR / RoCE v2 for multi-node
Storage
NVMe local + distributed object (S3-compatible)
Networking
25/100GbE per node, dedicated storage VLAN
Layer 02
Orchestration
The scheduling and resource management layer. This is where jobs are queued, priorities set, and GPU time allocated. Getting this layer wrong is the most common reason organisations waste 40–60% of their compute budget.
Components
Job scheduler
Kubernetes + custom resource operators or Slurm
Queue management
Priority lanes, preemption policies, fairshare
Resource quotas
Per-team and per-project GPU allocation limits
Autoscaling
Node pool expansion based on queue depth signals
Layer 03
ML Runtime
The execution environment for model workloads. Training, fine-tuning, and inference have fundamentally different resource profiles — we configure each one separately rather than forcing them to share the same runtime configuration.
Components
Training runtime
PyTorch DDP / FSDP across nodes, gradient checkpointing
Inference serving
vLLM / TGI with continuous batching and KV cache tuning
Experiment tracking
MLflow with artifact storage and run comparison
Model registry
Versioned model storage with promotion gates
Layer 04
Observability
The monitoring and alerting layer. Every metric we collect is chosen because it maps to a decision someone needs to make — not because it was easy to instrument. We don't produce dashboards full of numbers nobody looks at.
Components
GPU metrics
DCGM: utilisation, memory, thermal, power per GPU
Job metrics
Throughput, queue wait time, failure rate by workload type
Cost attribution
GPU-hours consumed per team, project, and job
Alerting
PagerDuty / Slack routing with runbook links in alerts
System Schematic
┌─────────────────────────────────────────────────────────────────────────┐
│ OBSERVABILITY LAYER │
│ Prometheus · Grafana · DCGM · PagerDuty │
└────────────────────────────┬────────────────────────────────────────────┘
│ metrics / alerts
┌────────────────────────────▼────────────────────────────────────────────┐
│ ML RUNTIME LAYER │
│ Training (DDP/FSDP) │ Inference (vLLM/TGI) │ MLflow │ Model Registry │
└───────────┬──────────────────────────────────────────────────┬──────────┘
│ job submission resource claim │
┌───────────▼──────────────────────────────────────────────────▼──────────┐
│ ORCHESTRATION LAYER │
│ Kubernetes · Job Scheduler · Queue Manager · Autoscaler · Quotas │
└───────────┬──────────────────────────────────────────────────┬──────────┘
│ schedule allocate │
┌───────────▼──────────────────────────────────────────────────▼──────────┐
│ PHYSICAL COMPUTE LAYER │
│ GPU Node 01 │ GPU Node 02 │ GPU Node 03 │ GPU Node N │
│ H100×8 │ H100×8 │ H100×8 │ H100×8 │
│ └──── InfiniBand HDR interconnect ────┘ │
│ NVMe Storage │ Object Storage │
└─────────────────────────────────────────────────────────────────────────┘
Design Decisions
The choices that matter
and why we make them.
Why bare metal over cloud for some clients?
For sustained, high-utilisation GPU workloads running more than ~60% of the time, reserved or owned bare metal is consistently cheaper than on-demand cloud at scale. The crossover point varies by GPU type and workload pattern — we model this during the audit and give you the actual numbers for your situation. Cloud is the right answer for variable workloads, burst capacity, and organisations not ready to manage physical hardware.
Why build a custom orchestration layer instead of using a managed service?
Managed ML platforms (SageMaker, Vertex, AzureML) are good products, but they make scheduling decisions on your behalf using logic you can't inspect or tune. For organisations with complex, mixed workload patterns — simultaneous training, fine-tuning, and inference — that opacity is expensive. A custom orchestration layer takes more time to build but gives you full control over priority, resource allocation, and cost attribution.
How do you handle multi-tenant GPU clusters?
Through a combination of Kubernetes namespace isolation, resource quotas, and network policy. Each team or project gets a defined GPU budget enforced by the scheduler. Jobs in excess of quota queue and wait rather than preempting other teams' work — unless explicitly configured otherwise. We also implement per-job GPU memory isolation where the workload supports it.
What does a production-ready inference setup actually require?
At minimum: a serving framework with continuous batching (vLLM or TGI), a load balancer with health checks, autoscaling triggered by queue depth rather than CPU/memory, a model versioning system with rollback, and latency SLA monitoring with alerting. Most teams we work with are missing at least three of these when we start. The most commonly skipped is autoscaling — teams size for peak and pay for it at all hours.
Want us to assess your current architecture?
The audit starts with a read of what you have. We give you a written assessment of what's working, what isn't, and what we would change — no charge.