← Back to all products

GPU Training Toolkit

$39

Multi-GPU training configs, mixed precision training, distributed training, and cloud GPU setup guides.

📁 16 files🏷 v1.0.0
JSONMarkdownPython

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 16 files

gpu-training-toolkit/ ├── LICENSE ├── README.md ├── examples/ │ ├── train_amp.py │ └── train_ddp.py ├── guides/ │ ├── distributed-training-guide.md │ └── mixed-precision.md ├── src/ │ └── gpu_training/ │ ├── __init__.py │ ├── amp.py │ ├── checkpoint.py │ ├── config.py │ ├── ddp.py │ ├── launcher.py │ └── profiling.py └── tests/ ├── test_config.py └── test_profiling.py

📖 Documentation Preview README excerpt

GPU Training Toolkit

Battle-tested PyTorch patterns for multi-GPU, mixed-precision, and

distributed training — packaged as a small, importable library you drop into

your project. It encodes the parts people get subtly wrong (effective batch math,

fp16 loss scaling, rank-0 checkpointing, OOM-safe batch sizing) so your scale-out

is correct the first time.

Every PyTorch import is lazy, so the configuration/profiling math and the full

test suite run on the Python standard library alone — no GPU and no torch

required to explore, plan, or test.

Features

  • Distributed Data Parallel (DDP) — process-group init/teardown, per-rank

device binding, DistributedSampler sharding, cross-rank metric reduction, and

a main_process_first helper for race-free one-time work.

  • Mixed precision done right — autocast + GradScaler with the correct

fp16-vs-bf16 behavior (a scaler is enabled only for fp16) and

unscale-before-clip gradient clipping.

  • torchrun launch generation — turn a typed config into the exact

single-node, multi-node, or SLURM srun command.

  • Profiling & OOM-safe batch finder — GPU memory snapshots, a throughput

meter (samples/sec, tokens/sec), and a logarithmic batch-size search that finds

the largest batch that fits before you crash.

  • Distributed-safe checkpoints — rank-0-only, atomic writes with automatic

rotation and DDP-wrapper unwrapping.

  • Typed TrainingConfig — one object that computes effective batch size,

optimizer-step counts, and a linear-warmup + cosine learning-rate schedule.

Requirements

  • Python 3.10+
  • PyTorch 2.x and CUDA 11.8+ to run training (one or more NVIDIA GPUs)
  • Nothing but the standard library to run the config/profiling math and tests

The package is pure-stdlib at import time; torch is imported lazily inside the

functions that actually touch the GPU.

Quick Start

1. Explore the math with no GPU


python3 -m gpu_training.config      # effective batch + step-count demo
python3 -m gpu_training.profiling   # OOM-safe batch finder demo
python3 -m gpu_training.launcher    # generate torchrun commands


import sys; sys.path.insert(0, "src")
from gpu_training.config import TrainingConfig, gradient_accumulation_for_target

# Hit a target effective batch of 512 with 8/GPU across 4 GPUs:
accum = gradient_accumulation_for_target(512, per_device_batch_size=8, world_size=4)
cfg = TrainingConfig(per_device_batch_size=8, gradient_accumulation_steps=accum, world_size=4)
print(cfg.effective_batch_size)          # 512
print(cfg.describe(num_samples=1_000_000))

... continues with setup instructions, usage examples, and more.

📄 Code Sample .py preview

src/gpu_training/amp.py """Mixed-precision training: autocast, GradScaler, and the accumulation loop. Mixed precision is two independent ideas that people conflate: 1. **autocast** runs the forward pass in a low-precision dtype (``float16`` or ``bfloat16``) for speed and memory, while keeping a master copy of weights in ``float32``. 2. **loss scaling** (``GradScaler``) is only needed for ``float16``, whose narrow exponent range underflows small gradients to zero. ``bfloat16`` has the same exponent range as ``float32`` and therefore needs **no scaler at all**. Getting (2) wrong is the most common mixed-precision bug: people wrap a bf16 run in a scaler (harmless but pointless) or, worse, run fp16 *without* a scaler and silently train on zeroed gradients. :class:`MixedPrecisionTrainer` encodes the correct behavior — a scaler is enabled iff ``precision == "fp16"``. The pure-Python helpers (:func:`should_step`, :func:`precision_to_dtype_name`) are unit-tested without torch; the trainer imports torch lazily on first use. """ from __future__ import annotations from contextlib import nullcontext from typing import Any, Iterable, Optional # Maps our precision strings to the torch dtype names they select. _DTYPE_NAMES: dict[str, str] = { "fp32": "float32", "fp16": "float16", "bf16": "bfloat16", } # Precisions that engage autocast (i.e. anything other than full fp32). _AUTOCAST_PRECISIONS: frozenset[str] = frozenset({"fp16", "bf16"}) def precision_to_dtype_name(precision: str) -> str: """Return the torch dtype *name* for a precision string (pure Python).""" try: return _DTYPE_NAMES[precision] except KeyError: raise ValueError( f"Unknown precision {precision!r}; expected one of {sorted(_DTYPE_NAMES)}" ) from None def is_autocast_precision(precision: str) -> bool: """True when ``precision`` should run under autocast (fp16/bf16).""" if precision not in _DTYPE_NAMES: raise ValueError(f"Unknown precision {precision!r}") # ... 248 more lines ...
Buy Now — $39 Back to Products