Kubernetes

Kueue for HPC and AI: debug admission, placement and distributed startup separately

Configure GPU batch admission with Kueue queues and hardware flavors, then distinguish quota waits from placement failures and distributed application startup problems.

9 min readKueueSchedulingKubernetesHPCAI

A distributed GPU job can have enough quota and still fail to start. The eligible nodes may be fragmented, a volume may be unavailable, or one worker may be waiting for an RDMA resource that the quota model never included.

Kueue is useful for deciding when supported batch workloads should be admitted against managed quotas. Kubernetes placement and application startup still need their own evidence. Keeping those stages separate makes both setup and troubleshooting much clearer.

This guide uses Kueue v0.10.2 as a fixed baseline, with kueue.x-k8s.io/v1beta1 queue APIs. Later documentation can describe different APIs and capabilities. Use the tagged release’s manifests and documentation when reproducing this design, and qualify a currently supported release before adopting it for a new production service.

Define admission policy before queue objects

List the research workloads the service must support: single-GPU experiments, multi-node training, CPU-heavy analysis and jobs that need a specific accelerator or network. Record expected duration, checkpoint support, minimum worker group and the consequences of interruption.

Decide which namespaces and teams share capacity, which resources are covered by quota and whether borrowing is allowed. Quota describes an allocation policy. It does not create nodes, make an incompatible GPU usable or guarantee that every requested resource fits on the same machines.

Keep this design separate from a Run:ai-controlled workload path unless a supported integration has been explicitly qualified. Installing two resource-management systems does not resolve ownership between them. The same principle applies to Slurm worker pools: each physical resource needs a clear admission and placement model.

Map flavors to real hardware

A ResourceFlavor represents characteristics of resources available to a queue. Define labels and, where applicable, taints that correspond to real eligible nodes. Check those labels against inventory rather than using a friendly name that conceals several incompatible GPU types.

For a multi-node workload, include the constraints that matter to communication and memory. GPU count alone is a weak description when some nodes have different accelerator memory, network connectivity or CPU locality. Do not infer a full topology guarantee from a flavor label.

Document the relationship between each flavor, node pool and physical capacity. A quota of eight GPUs is only useful to an eight-GPU job if the rest of its placement constraints can also be met. Maintenance, existing pods and per-node resource fragmentation can prevent that even when a quota counter looks favorable.

Connect ClusterQueue, LocalQueue and namespace

A ClusterQueue defines the managed quota policy. A namespace-scoped LocalQueue gives workloads a queue entry point that references the intended ClusterQueue. Keep the naming and namespace ownership straightforward so a researcher can identify where a submitted job will wait.

Start with one namespace, one hardware flavor and a small quota before introducing borrowing or multiple flavors. Review the release-tagged quota administration documentation for the actual API and fields.

Review namespace selection and access permissions. A valid LocalQueue reference is not a substitute for an authorization policy. Test submission as an ordinary research user and verify that the intended queue is available without giving broad cluster administration rights.

A minimal illustrative queue chain is below. It assumes an existing namespace research labeled example.com/research-queue=true and qualified GPU nodes labeled example.com/gpu-class=research. The quota numbers are examples, not measured capacity. Review all selectors before applying it.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: research-gpu
spec:
  nodeLabels:
    example.com/gpu-class: research
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: research-gpu
spec:
  namespaceSelector:
    matchLabels:
      example.com/research-queue: "true"
  resourceGroups:
    - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
      flavors:
        - name: research-gpu
          resources:
            - name: cpu
              nominalQuota: 8
            - name: memory
              nominalQuota: 16Gi
            - name: nvidia.com/gpu
              nominalQuota: 2
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: research-gpu
  namespace: research
spec:
  clusterQueue: research-gpu

The namespace selector limits which namespaces can use the ClusterQueue; the flavor’s node label guides eligible hardware selection. Neither label installs a GPU driver or creates capacity. Check the namespace and node labels independently before testing the Job below.

Before starting jobs, inspect the installed controller, queue conditions and Kubernetes events. Confirm that the selected resources are covered by the intended quota configuration. If a NIC resource or storage constraint sits outside that model, include it explicitly in acceptance testing and monitoring.

Use a small suspended GPU Job as the first canary

The v0.10.2 Kubernetes Job guide requires the queue label, a suspended Job and declared resource requests. This illustrative canary assumes a LocalQueue named research-gpu already exists and a qualified private image contains the test program:

apiVersion: batch/v1
kind: Job
metadata:
  name: gpu-canary
  namespace: research
  labels:
    kueue.x-k8s.io/queue-name: research-gpu
spec:
  suspend: true
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: canary
          image: registry.example.com/research/gpu-canary:qualified
          command: ["/opt/research/bin/gpu-canary"]
          resources:
            requests:
              cpu: "2"
              memory: 4Gi
            limits:
              cpu: "2"
              memory: 4Gi
              nvidia.com/gpu: 1

Replace the image and command with a tested artifact, preferably pinned by digest. The NVIDIA GPU extended resource relies on a working device-plugin installation. This single-pod canary proves a narrow admission and execution path; it does not demonstrate gang startup for distributed training.

Submit the approved manifest in the test namespace, then inspect its associated Workload, Job and Pod. Record the transitions rather than only the final success. Repeat when quota is intentionally occupied to prove the waiting behavior.

Follow the first unsatisfied condition

Initial read-only checks are:

Illustrative session
Follow the first unsatisfied condition Illustrative workload rows. A running Pod is only one checkpoint; inspect quota admission, device allocation and useful job progress. Illustrative example; not a captured production run. Open full size · Scroll sideways to inspect the terminal.
Read the illustrated session as text
$ kubectl -n research get pods
NAME                  READY   STATUS    RESTARTS   AGE
example-job-abcde     1/1     Running   0          2m
Read and copy the original commands
kubectl -n research get localqueues
kubectl get clusterqueues
kubectl -n research get workloads
kubectl -n research get jobs,pods
kubectl -n research get events --sort-by=.metadata.creationTimestamp

Use the discovered Workload name for a detailed inspection. The tagged troubleshooting guide explains following workload state and conditions. Do not assume the Workload name is identical to the Job name.

Observation Investigate next
Job has no expected managed Workload Queue label, namespace, supported integration and controller events
Workload is waiting for admission Queue conditions, matching flavors, quota and competing workloads
Workload admitted but Pod Pending Pod scheduling events, physical resources, affinity, taints and volumes
Pod assigned but container not running Image pulls, mounts, runtime and security configuration
Workers run but training has not begun Rendezvous, rank count, dataset access and communication transport

Preserve the state before forcing a job to resume or changing queue policy. Manual intervention can hide the condition being investigated and create a workload outside the intended admission process.

Qualify the distributed integration separately

Choose a distributed-job integration supported by the pinned Kueue release and install its required controller and CRDs at qualified versions. Do not assume several independent Jobs automatically become one coordinated distributed workload.

Define what success means: the required worker group starts within a bounded interval, every rank reaches initialization, the intended network carries communication, and the application completes a short useful step. Test an intentionally unavailable worker resource to observe how the configured integration handles incomplete startup.

Kueue admission is not itself proof that all pods are simultaneously placed or that an application can recover after one rank fails. If using release-specific readiness or timeout features, test their actual behavior and the resulting requeue or eviction path. Keep those policies consistent with the application’s checkpoint and restart model.

For GPU/RDMA jobs, verify CPU, memory, accelerators, NIC resources and storage together. The multi-rail Kubernetes guide addresses the transport checks after placement succeeds.

Monitor the waiting time that researchers experience

Measure submission-to-admission, admission-to-pod-ready and pod-ready-to-first-useful-step separately. A growing admission wait suggests a different investigation from a growing image-pull or rendezvous delay.

Track pending workload reasons, queue condition changes, admission outcomes and the number of admitted workloads whose pods have not become useful. Check monitoring freshness and controller health before interpreting an empty dashboard as an empty queue.

Use metric names from the installed release’s endpoint and documentation. Avoid copying alert rules from a newer release without confirming that the series and labels exist. Keep job identifiers in logs or bounded investigation views where permanent per-job labels would create excessive cardinality.

Compare queue policy with research outcomes. A policy that improves raw GPU occupancy while repeatedly interrupting jobs with expensive checkpoints may lengthen time to results. Measure completed work, wasted restart time and storage pressure alongside allocation.

Test recovery before expanding the policy

Exercise controller replacement, a worker failure and a planned capacity reduction in an isolated test window. Observe pending and admitted workloads independently. Verify that completed jobs release the expected quota and that a restarted distributed workload reads a complete checkpoint.

Retain the qualified release manifests, queue definitions, node-label inventory, image digest and transition timeline. Expand flavors and borrowing only after the simple case is understandable under both normal demand and failure.

Wirewalk’s HPC and AI infrastructure services connect queue policy to the GPUs, storage and fabric that researchers actually use, with acceptance tests that can be repeated after changes.

All articles