Architecture Guide

GPU Resource Management Guide

This guide is the long-form technical companion to /technology. It explains how CNLab partitions GPUs, schedules workloads, and moves them between nodes — including the cross-cloud case. Every section maps to an operational primitive your platform team will see in production.

Audience: platform engineers, SREs, and infrastructure researchers. Reading time: ≈ 22 minutes. The companion PDF is identical content, with appendices and citations — download here.

MIG: Multi-Instance GPU

NVIDIA Multi-Instance GPU is hardware-level partitioning available on H100, H200, and A100. A single physical GPU is split into up to seven instances. Each instance has its own SMs, L2 cache slice, memory bandwidth, and DRAM partition.

Profiles

On H100 80GB, the available profiles are 1g.10gb, 1g.20gb, 2g.20gb, 3g.40gb, 4g.40gb, and 7g.80gb. CNLab automatically picks the smallest profile that satisfies the request.

When CNLab uses MIG

For workloads that need full memory bandwidth, hardware fault isolation, or strict tenant boundaries (e.g. multi-customer inference). CNLab's allocator chooses MIG when the request asks for > 14% of the GPU and crosses a tenant boundary.

1% Block Partitioning

For everything below the MIG floor, CNLab introduces 1% Block partitioning: the GPU's compute and memory are exposed as 100 named slices. The minimum allocation is one block (1% of CUDA cores, 1% of VRAM).

Why blocks

Lots of real workloads — student notebooks, hyperparameter sweeps, dev experiments — use 0.5–10% of a GPU. MIG's smallest profile (1g) wastes 13× more than necessary. Blocks fit the demand.

Implementation

CNLab installs a CUDA shim and a per-node daemon. The shim intercepts memory and kernel launch APIs; the daemon enforces the per-block quota. Tenant isolation is namespace + cgroup + memory pinning; cross-tenant CUDA IPC is denied at the driver level.

Note. Blocks combine with MIG. A 1g MIG slice can host 14 1% blocks safely.

Scheduling

The scheduler is a multi-criteria optimizer. Inputs: queue depth, deadline, priority, fair-share, GPU type, data locality, network distance. Output: a placement decision per pending job.

Policies

Deadline-aware. Reserves slices forward in time to meet user-declared deadlines; pre-empts lower-priority work when needed.
Fair-share. Long-term per-team usage normalised over a rolling 7-day window.
Bin-packing. Where deadlines and shares allow, packs work onto the fewest nodes for power savings.
Pre-emption. Production inference pre-empts development training when capacity tightens; pre-empted work resumes from the latest auto-checkpoint.

Live Migration

Live migration moves a running workload from node A to node B atomically. Memory pages are pre-copied while the source still runs; a brief stop-and-copy phase synchronises the last dirty pages and the CUDA context; the target resumes from the exact instruction.

Cross-cluster, cross-cloud

Same engine, different transport. Within a data centre: RDMA. Between data centres: TLS-accelerated TCP. Between on-prem and cloud: WireGuard or PrivateLink. Latency is dominated by network bandwidth, not the migration logic.

Code Examples

Provision a sliced H100 from CLI

# Provision a sliced H100, 25% block, 50GB workspace
$ cnlab server create \
    --image pytorch:2.4-cuda12.4 \
    --profile h100-25pct \
    --storage 50gb \
    --idle 4h
✓ Server provisioning... ETA 18s
✓ Ready: https://nb.cnlab.ai/srv_01HX...

Inspect quota

$ cnlab quota show
Project: ai-research-ml
  GPU-hours used:    342 / 1000 (34%)
  Storage GB-days:   1,820 / 5,000 (36%)
  Concurrent srvs:   2 / 5

Submit a deadline-bound job

$ cnlab run train.py \
    --gpu h100-50 \
    --deadline 2026-04-30T17:00 \
    --priority high
Job queued. Scheduler reserved 4×H100-50pct on cluster A.
ETA to start: 12 min.

Glossary

1% Block. Smallest GPU allocation unit in CNLab — 1/100 of a physical GPU.

MIG. NVIDIA Multi-Instance GPU. Hardware partitioning on H100/A100.

Live migration. Atomic relocation of a running workload between nodes with zero checkpoint.

Burst-out. Auto-extending an on-prem workload to public cloud when local quota is reached.

My GPU First. Strategy of utilising on-prem 100% before reaching for cloud.

RBAC. Role-based access control.

RDMA. Remote Direct Memory Access — kernel-bypass network transfer.