spark-training-gotchas

SUMMARY

What it does

This skill provides preflight checks and diagnostic guidance for ten known failure modes (G1-G10) when running ML training on NVIDIA DGX Spark (GB10 chip, SM121, 128GB unified memory, aarch64). It covers issues related to CUDA ABI mismatch, flash-attention, UMA OOM, thermal throttling, bandwidth ceiling, global UMA resource contention, NVFP4 vs FP8 performance, stale official playbooks, container vs bare pip environments, and dual-Spark parallelism (DDP/FSDP only, not tensor parallelism). The skill includes a quick reference table, detailed descriptions of each gotcha with symptoms, causes, checks, and fixes, and a fast triage section with commands. It references additional files (references/gotcha-checks.md, assets/preflight.sh) for detailed checks. The skill is intended for use before or during training runs on DGX Spark to avoid or diagnose common issues.