Chapter 2: DGX OS Software Stack, CUDA Toolkit & Containerized AI Workflows

DGX OS Base System: Ubuntu LTS with NVIDIA Kernel Modules

DGX OS is built on Ubuntu 24.04 LTS but is far more than a stock installation. NVIDIA ships a custom kernel (6.17.0-1014-nvidia) alongside a Hardware Enablement (HWE) kernel 6.14, both tuned for the Grace Blackwell architecture's unified memory, NVLink interconnects, and GPU scheduling requirements.

The GPU driver package — the nvidia-580-open series — is delivered as a .deb package without DKMS (Dynamic Kernel Module Support). This eliminates the fragile driver rebuild step that plagues standard Linux GPU setups. Think of it this way: if standard Ubuntu with manually installed drivers is like assembling furniture from parts, DGX OS is the factory-assembled version.

GPU Monitoring with nvidia-smi and DCGM

The moment DGX Spark boots, the GPU driver stack is operational. The primary monitoring interface is nvidia-smi, which reports GPU utilization, memory, temperature, power draw, and running processes. For continuous monitoring, nvidia-smi dmon streams metrics at configurable intervals.

Beyond nvidia-smi, DCGM (Data Center GPU Manager) provides programmatic access to GPU telemetry, health checks, and policy-based monitoring suitable for integration with Prometheus and Grafana dashboards.

System Management and Diagnostics

Diagnostics follow a layered approach:

1. Hardware layer: nvidia-smi -q reports ECC memory errors, PCIe link speed, thermal throttling
2. Driver layer: dmesg | grep nvidia surfaces kernel-level driver messages
3. Application layer: CUDA sample programs (deviceQuery, bandwidthTest) verify end-to-end functionality

Multi-User Access and GPU Isolation

DGX Spark supports teams through standard Linux multi-user capabilities enhanced for GPU workload isolation. GPU access is controlled at the container level — each user's Docker containers receive dedicated GPU resources through the --gpus flag. JupyterLab, pre-installed on DGX Spark, provides browser-based development access with per-user sessions.

CUDA Toolkit: Installation, Versioning, and Configuration

The CUDA toolkit comes pre-installed on DGX Spark, verified for Blackwell hardware compatibility. Versions include CUDA 12.8 and CUDA 13.0.2 depending on the release batch. The toolkit includes:

• nvcc — the CUDA compiler for .cu source files
• CUDA runtime libraries — the API layer applications link against
• cuBLAS, cuFFT, cuSPARSE — GPU-accelerated math libraries
• CUDA samples — reference implementations for testing

Environment configuration centers on two variables: PATH must include /usr/local/cuda/bin, and LD_LIBRARY_PATH must include /usr/local/cuda/lib64. On DGX OS, these are set by default.

Important: DGX Spark uses the ARM64 (AArch64) architecture via the Grace CPU. Any natively compiled software must target ARM64, and the NGC CLI must be installed from the ARM64 Linux tab.

cuDNN and TensorRT

cuDNN provides GPU-accelerated deep learning primitives: convolutions, pooling, normalization, and activation functions. When PyTorch or TensorFlow execute a convolution, they call cuDNN, which selects the fastest algorithm for the specific tensor dimensions, data type, and hardware.

TensorRT (v10.2 on DGX Spark) is NVIDIA's inference optimization engine. It takes a trained model and produces an optimized execution plan — fusing layers, selecting precision (FP32, FP16, INT8), and calibrating for the target GPU. The workflow: export to ONNX, optimize with trtexec, deploy the .trt engine.

NCCL and Multi-GPU Communication

NCCL (pronounced "Nickel") handles data transfer between multiple GPUs. It orchestrates operations like AllReduce, Broadcast, and AllGather across GPUs connected via NVLink, exploiting high-bandwidth topology rather than slower PCIe. PyTorch's DistributedDataParallel and TensorFlow's tf.distribute.Strategy call NCCL automatically.

Framework Integration: PyTorch, TensorFlow, JAX

DGX Spark ships with pre-installed versions of major AI frameworks, each compiled against the system's CUDA, cuDNN, and NCCL versions. The pre-installed versions are matched and tested, avoiding version compatibility headaches. For different framework versions, NGC containers provide isolated environments with their own stack.

# Verify GPU availability
# PyTorch
import torch
print(torch.cuda.is_available())       # True
print(torch.cuda.get_device_name(0))   # NVIDIA Blackwell ...

# TensorFlow
import tensorflow as tf
print(tf.config.list_physical_devices('GPU'))

# JAX
import jax
print(jax.devices())

NGC Container Registry

The NGC container registry at nvcr.io hosts hundreds of pre-built, GPU-optimized container images. These include framework containers (PyTorch, TensorFlow, JAX), application containers (Triton, RAPIDS), and model containers (NIM microservices). Each image is tested on NVIDIA hardware.

Setting up NGC access:

1. Generate an API key at ngc.nvidia.com → Setup → API Key
2. Authenticate Docker: docker login nvcr.io (username: $oauthtoken, password: your API key)
3. Install NGC CLI from the ARM64 Linux tab (required for Grace CPU architecture)
4. Pull containers: docker pull nvcr.io/nvidia/pytorch:24.08-py3

NVIDIA Container Toolkit: GPU Passthrough

The NVIDIA Container Toolkit installs OCI runtime hooks that expose GPU drivers, CUDA libraries, and device files inside containers without requiring these to be baked into the image. On DGX Spark, the toolkit is pre-installed and pre-configured.

# Run GPU-enabled container
docker run -it --gpus all nvcr.io/nvidia/pytorch:24.08-py3

# Specify individual GPUs
docker run -it --gpus '"device=0"' nvcr.io/nvidia/pytorch:24.08-py3

# Verify GPU access inside container
docker run --rm --gpus all nvcr.io/nvidia/cuda:13.0.1-devel-ubuntu24.04 nvidia-smi

Building Custom Containers

Start from an NVIDIA base image and layer your requirements:

FROM nvcr.io/nvidia/pytorch:24.08-py3

# Install project-specific dependencies
RUN pip install transformers datasets accelerate wandb

# Copy project code
COPY ./src /workspace/src
WORKDIR /workspace
CMD ["python", "src/train.py"]

Build and run: docker build -t my-training:v1 . then docker run --gpus all -v /data:/data my-training:v1

Container Orchestration Patterns

Two orchestration patterns are common on DGX Spark:

• Docker Compose for multi-container workflows (training + TensorBoard, etc.) with GPU resource reservations via deploy.resources.reservations.devices
• Kubernetes with NVIDIA GPU Operator for larger-scale orchestration with automatic GPU scheduling, resource quotas, and multi-user workload management

NIM Microservices: Models as API Endpoints

NVIDIA NIM provides prebuilt, optimized containers that package foundation models as API endpoints. Each NIM container includes the model weights, an inference engine (typically TensorRT-LLM), and an OpenAI-compatible API server.

# Pull and run a NIM container
docker pull nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
docker run --gpus all -p 8000:8000 nvcr.io/nim/meta/llama-3.1-8b-instruct:latest

# Query the model via OpenAI-compatible API
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta/llama-3.1-8b-instruct",
       "messages": [{"role": "user", "content": "Explain GPU memory hierarchy."}]}'

NIM handles model optimization internally — batch sizes, KV-cache memory management, and TensorRT-LLM optimizations. The key advantage: if you can run a Docker container, you can serve a production-grade LLM.

NVIDIA AI Enterprise Platform

NVIDIA AI Enterprise is the commercial software layer providing enterprise support, security certifications, API stability guarantees, and validated upgrade paths. DGX Spark includes an AI Enterprise license, unlocking NIM microservices, enterprise support, and NVIDIA Blueprints.

Triton Inference Server: Multi-Model Serving

Triton Inference Server serves multiple models simultaneously with fine-grained control over scheduling, batching, and resource allocation. It supports TensorFlow SavedModels, PyTorch TorchScript, ONNX, TensorRT engines, and Python-based models through a unified interface.

• Dynamic batching: Automatically groups requests to maximize GPU throughput
• Model ensembles: Chains models in a pipeline (tokenizer → LLM → post-processor)
• Concurrent execution: Runs different models on the same GPU
• Metrics endpoint: Prometheus-compatible metrics for monitoring integration

Triton serves models via HTTP (port 8000), gRPC (port 8001), and metrics (port 8002).

Monitoring and Observability

Production AI services require continuous monitoring across four layers:

1. GPU metrics: nvidia-smi and DCGM export utilization, memory, temperature, error counts
2. Inference metrics: Triton exposes latency, throughput, queue depth via Prometheus
3. Container metrics: Docker/Kubernetes provide CPU, memory, network I/O data
4. Application logs: Structured logging from NIM and Triton for debugging and auditing

Health check endpoints (/v2/health/ready) enable load balancers and Kubernetes to automatically route traffic away from unhealthy instances.