AI-211 AI & Data

Distributed Training Foundations

Understand what actually changes when training outgrows a single GPU: how data, tensor and pipeline parallelism split the work, how gradients are exchanged, and why a job that ran fine on one card stalls or diverges on eight. Learners scale a real training run across devices, keep the maths correct across accumulation and mixed precision, and diagnose a straggler that is quietly halving cluster throughput.

B4Professional

6modules

42skill atoms

2role journeys

Curriculum

What this course covers.

6 modules, 42 named skill atoms. Expand any module to see them.

1Why One GPU Runs Out7 skill atoms

model weight memory mathoptimizer state and gradient copiesactivation memory vs batch sizeOOM failure signaturesparameter count vs VRAM budgetsingle-device throughput ceilingwhen scaling out beats scaling up

2Parallelism Strategies7 skill atoms

data parallel vs model paralleltensor parallelism inside a layerpipeline parallelism and bubble overheadPyTorch DDP mechanicsdevice mesh planning3D parallelism combinationspicking a strategy from model shape

3Collective Communication7 skill atoms

all-reduce and ring all-reduceall-gather and reduce-scatterNCCL backend basicsNVLink vs PCIe vs InfiniBand pathsgradient bucketing and compute overlapbandwidth-bound vs latency-bound collectivesnccl-tests bandwidth baseline

4Batch Size and Precision7 skill atoms

gradient accumulation stepseffective global batch sizelinear and sqrt learning-rate scalingwarmup for large batchesAMP with fp16 vs bf16loss scaling and overflowsilent divergence between ranks

5Checkpointing and Restart7 skill atoms

sharded vs consolidated checkpointstorch.distributed.checkpointoptimizer and RNG state capturecheckpoint cadence vs job costresume-correctness verificationactivation checkpointing compute trade-offcorrupt or partial checkpoint recovery

6Diagnosing Throughput Loss7 skill atoms

straggler rank identificationdata-loader starvationsamples per second per GPUMFU and utilisation readingPyTorch profiler timeline tracesNCCL timeout and hang triagepadding waste from imbalanced shards

Where it fits

AI-211 in the role journeys.

This course appears in 2 of our 45 role journeys. Here is what a learner takes immediately before and after it in each.

MLOps / LLMOps Engineer

Professional stage

CL-403→AI-211→AI-220

AI Infrastructure Engineer

Professional stage

AI-401→AI-211→AI-219

Roles this course serves

AI Infrastructure Engineer MLOps / LLMOps Engineer

The capability ladder

This course is authored to band B4.

Every course we run is written to one rung of the CASI ladder, so a plan can be assembled to take a team from where they are to where they need to be.

What do B1–B6 mean?The CASI Capability Ladder — click to expand

Every course targets a band on the CASI Capability Ladder — our six-band proficiency scale, anchored to open standards (O*NET, ESCO, NICE, NIST AI RMF, Bloom's). A band tells you how deep a course goes, and what evidence proves it.

What the learner can doTypical evidence

AwareUnderstands concepts and vocabulary; uses tools with guidance

Knowledge checks

FoundationPerforms standard tasks correctly in familiar contexts

Guided labs, autograded exercises

PractitionerDelivers complete pieces of work independently

Scenario labs, proctored hands-on exams

ProfessionalHandles production-grade complexity, trade-offs and failure modes

Break-fix drills, design defenses

AdvancedEngineers systems end-to-end under constraints; leads others

Rubric-scored capstones, vivas

ExpertSets direction; recognised authority across teams

Portfolio + panel evaluation

A note on B6. Courses in this catalog target B1–B5. B6 is not taught — it is recognised, through a portfolio and a panel, once someone is setting direction for others. Every journey here is built to land a learner at B5.

Other AI courses at this level.

AI-112

Run AI-211 for your team.

This course runs at several lengths depending on how deep you need to go and how much of it your people already have. Tell us who is being trained and we will scope it.

Add it to a training plan Talk to our team Check your team’s level free

Distributed Training Foundations

What this course covers.

AI-211 in the role journeys.

MLOps / LLMOps Engineer

AI Infrastructure Engineer

Roles this course serves

This course is authored to band B4.

Other AI courses at this level.

Departmental AI Adoption & Automation Design

Transformers from First Principles

Fine-Tuning with Hugging Face

Synthetic Data Generation & Evaluation

Vector Databases & Hybrid Search

Document AI & Intelligent Document Processing

Run AI-211 for your team.