← Back to all products

GPU Training Toolkit

$39

Multi-GPU training configs, mixed precision training, distributed training, and cloud GPU setup guides.

📁 16 files🏷 v1.0.0
JSONMarkdownPython

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 16 files

gpu-training-toolkit/ ├── LICENSE ├── README.md ├── examples/ │ ├── train_amp.py │ └── train_ddp.py ├── guides/ │ ├── distributed-training-guide.md │ └── mixed-precision.md ├── src/ │ └── gpu_training/ │ ├── __init__.py │ ├── amp.py │ ├── checkpoint.py │ ├── config.py │ ├── ddp.py │ ├── launcher.py │ └── profiling.py └── tests/ ├── test_config.py └── test_profiling.py

📖 Documentation Preview README excerpt

GPU Training Toolkit

Battle-tested PyTorch patterns for multi-GPU, mixed-precision, and

distributed training — packaged as a small, importable library you drop into

your project. It encodes the parts people get subtly wrong (effective batch math,

fp16 loss scaling, rank-0 checkpointing, OOM-safe batch sizing) so your scale-out

is correct the first time.

Every PyTorch import is lazy, so the configuration/profiling math and the full

test suite run on the Python standard library alone — no GPU and no torch

required to explore, plan, or test.

Features

  • Distributed Data Parallel (DDP) — process-group init/teardown, per-rank

device binding, DistributedSampler sharding, cross-rank metric reduction, and

a main_process_first helper for race-free one-time work.

  • Mixed precision done right — autocast + GradScaler with the correct

fp16-vs-bf16 behavior (a scaler is enabled only for fp16) and

unscale-before-clip gradient clipping.

  • torchrun launch generation — turn a typed config into the exact

single-node, multi-node, or SLURM srun command.

  • Profiling & OOM-safe batch finder — GPU memory snapshots, a throughput

meter (samples/sec, tokens/sec), and a logarithmic batch-size search that finds

the largest batch that fits before you crash.

  • Distributed-safe checkpoints — rank-0-only, atomic writes with automatic

rotation and DDP-wrapper unwrapping.

  • Typed TrainingConfig — one object that computes effective batch size,

optimizer-step counts, and a linear-warmup + cosine learning-rate schedule.

Requirements

  • Python 3.10+
  • PyTorch 2.x and CUDA 11.8+ to run training (one or more NVIDIA GPUs)
  • Nothing but the standard library to run the config/profiling math and tests

The package is pure-stdlib at import time; torch is imported lazily inside the

functions that actually touch the GPU.

Quick Start

1. Explore the math with no GPU


python3 -m gpu_training.config      # effective batch + step-count demo
python3 -m gpu_training.profiling   # OOM-safe batch finder demo
python3 -m gpu_training.launcher    # generate torchrun commands


import sys; sys.path.insert(0, "src")
from gpu_training.config import TrainingConfig, gradient_accumulation_for_target

# Hit a target effective batch of 512 with 8/GPU across 4 GPUs:
accum = gradient_accumulation_for_target(512, per_device_batch_size=8, world_size=4)
cfg = TrainingConfig(per_device_batch_size=8, gradient_accumulation_steps=accum, world_size=4)
print(cfg.effective_batch_size)          # 512
print(cfg.describe(num_samples=1_000_000))

... continues with setup instructions, usage examples, and more.

📄 Code Sample .py preview

src/gpu_training/config.py"""Typed training configuration plus effective-batch and schedule math. This module is intentionally **pure Python** — it imports nothing from PyTorch so that batch-size planning, gradient-accumulation math, and learning-rate schedule calculations can run (and be unit-tested) on a machine with no GPU and no ``torch`` installed. The central abstraction is :class:`TrainingConfig`. The most common source of silently-wrong distributed runs is a mismatch between the batch size a person *thinks* they are training with and the **effective** batch size that actually drives the optimizer. The effective batch size is:: effective = per_device_batch_size * gradient_accumulation_steps * world_size Get this number wrong and your learning rate, warmup, and total step count are all calibrated for the wrong scale. Every helper here exists to make that number explicit and reproducible. """ from __future__ import annotations import math from dataclasses import asdict, dataclass, field from typing import Any # Precisions understood by the rest of the toolkit. ``fp32`` is full precision; # ``fp16`` requires loss scaling (see amp.py); ``bf16`` has the dynamic range of # fp32 and needs no scaler. SUPPORTED_PRECISIONS: tuple[str, ...] = ("fp32", "fp16", "bf16") def effective_batch_size( per_device_batch_size: int, gradient_accumulation_steps: int, world_size: int, ) -> int: """Return the optimizer-visible (effective) batch size. Args: per_device_batch_size: Samples per forward/backward on a single GPU.
Buy Now — $39 Back to Products