The GPU Shortage Inside Your Own Infrastructure: Why AI Workloads Queue While Capacity Sits Idle

GPU utilisation shows compute activity, not whether capacity is actually available. A GPU can report low utilisation while remaining fully allocated to one workload. This post explains why workloads queue beside idle-looking GPUs, how allocation, memory, placement, and application bottlenecks contribute, and how GPU sharing methods differ in their tradeoffs. Capsule: AI workloads can wait for GPUs even when monitoring shows idle compute. A GPU with low compute activity is not necessarily available capacity for another workload because allocation rules, memory requirements, hardware compatibility, isolation, and placement constraints determine what can actually run. The post should explain how to distinguish an actual hardware shortage from an allocation, placement, or application bottleneck. Use the 5% average GPU utilisation finding from Cast AI’s 2026 report as a research hook, but do not imply that queued workloads and idle GPUs occurred together in the same measured environments or that 95% of GPU capacity is immediately reclaimable. Key takeaways GPU utilisation metrics measure compute activity, not allocation state. A GPU reporting 5% compute utilisation can still be fully allocated with Zero capacity available for new workloads. Kubernetes assigns whole GPUs to pods by default. One workload holding a device blocks all others regardless of how much of the GPU it actually uses. Queued workloads beside idle-looking GPUs have at least four distinct root causes. Only one of them is resolved by GPU sharing. Time-slicing, MIG, and MPS trade off differently on memory isolation, hardware requirements, observability, and cloud provider support. No single method fits every workload. The diagnostic sequence matters: check allocation state, then memory occupancy, then placement constraints, then application bottlenecks. In that order. Cast AI supports all three sharing methods with automatic bin-packing and no changes required to workload manifests. What GPU utilisation tells you and what it misses GPU utilisation is a phrase that covers at least four different measurements, and conflating them is the fastest route to a misdiagnosis. What most dashboards surface is compute utilisation: the percentage of streaming multiprocessors (SMs) that are active in a given time window. That number tells you how busy the silicon is when it runs. It tells you nothing about whether the device is available for a new workload. A model loaded into VRAM occupies that memory continuously. The inference service may answer one request per minute, keeping compute activity at 5%, but the device is allocated, the memory is occupied, and Kubernetes will not schedule anything else on it. From the scheduler’s perspective, that GPU is unavailable. From your monitoring dashboard, it looks nearly idle. The table below separates four metrics commonly grouped under “GPU utilisation” and clarifies what each one does and does not tell you. Metric What it measures What it does NOT show Compute activity (%SM) Fraction of streaming multiprocessors active during the sample window Whether the GPU is allocated; whether VRAM is available for another workload Memory occupancy (VRAM) How much GPU memory is consumed by loaded models and tensors Compute activity; whether queued workloads could fit in remaining memory Queue wait time (pod scheduling) How long pending pods wait before a GPU device becomes available Whether the delay is caused by allocation, memory, placement, or application bottleneck Throughput / latency Requests served per second; time-to-first-token or end-to-end response time GPU resource utilisation or allocation efficiency The Cast AI 2026 Kubernetes Optimisation Report , which analysed tens of thousands of production clusters from January 2025 through April 2026, recorded average GPU compute utilisation of 5% before any optimisation was applied. The accompanying report blog post notes that one cluster in the dataset sustained 49% GPU utilisation across 136 H200s, a gap described as almost entirely attributable to technique rather than hardware. That 5% average reflects a real fleet-wide pattern of overprovisioning and underuse. It does not mean 95% of capacity is free to accept new workloads. Each cluster, and each GPU within it, needs its own diagnostic pass before that conclusion holds. Four reasons workloads can queue beside idle GPU capacity The Kubernetes scheduler sees a GPU as either available or unavailable. It makes that determination based on the nvidia.com/gpu resource count on each node, not on what percentage of the silicon is active. Four distinct conditions can produce a queue even when compute activity looks low. A workload holds an entire device The NVIDIA Kubernetes Device Plugin allocates GPUs exclusively by default. When a pod requests nvidia.com/gpu: 1, it receives sole ownership of one physical device for the duration of its lifecycle. No other pod can use that device, regardless of how much compute or memory the occupying workload actually consumes. This is the most common cause of the queue-beside-idle pattern in inference clusters. A set of models, each holding a dedicated GPU but serving bursty or low-frequency requests, keeps every device allocated. An incoming workload finds nvidia.com/gpu: 0 available on the node and waits, even though aggregate compute activity across the node might be under 10%. The device plugin focuses on allocation, not reclamation. Kubernetes cluster autoscaler can provision new nodes but will not recover idle capacity on existing ones. The problem lives at the scheduling layer, and the solution requires changing how devices are presented to the scheduler. That is exactly what GPU sharing mechanisms do. Memory is occupied, not just underused Low compute utilisation does not mean memory is available. Two scenarios illustrate the range. A 70B-parameter model loaded in FP16 consumes roughly 140 GB of VRAM (base weights only; at 4K context lengths, KV cache adds 15–20% above base weights, while at 128K context lengths, KV cache requirements can exceed base weights entirely). That model requires two or more A100 80GB GPUs to load at all. A 13B FP16 model occupies roughly 26 GB and fits on a single A100 80GB, but it still holds that VRAM continuously. Compute might report 8%, because most requests complete quickly, but the device cannot accept another workload. Before configuring sharing, check whether quantisation applies. An INT4-quantised 7B model occupies roughly 4 GB versus roughly 14 GB in FP16, which changes the co-location arithmetic for time-slicing and MIG configurations. INT8 cuts requirements roughly in half relative to FP16, often with acceptable quality trade-offs for inference workloads. This matters particularly for teams considering time-slicing as a solution. Time-slicing multiplexes access to compute, but it does not partition memory. All replicas share the same VRAM address space. If two workloads together exceed device memory, they will not coexist safely. Diagnosing a memory constraint first, before choosing a sharing method, prevents a failed deployment and potential workload instability. Check memory occupancy with nvidia-smi before assuming a sharing approach will help. If the free memory column is near zero on the relevant devices, the constraint is VRAM capacity, not compute scheduling. MIG provides hardware-isolated memory partitions that can address certain versions of this problem, but it requires compatible hardware and changes how workloads are scheduled. Placement rules exclude otherwise usable hardware A workload may fail to schedule not because GPUs are fully allocated but because the scheduler cannot find a node that satisfies all placement constraints simultaneously. Node affinity rules specifying a GPU model or generation, topology spread constraints requiring a minimum number of nodes, and taints without matching tolerations can all prevent scheduling even when raw device counts look sufficient