Scaling Distributed PyTorch Training on EKS Using Ray Train DLCs
Learn how AWS handles complex container stacks to scale PyTorch training across multiple GPU nodes using Ray Train DLCs and Elastic Fabric Adapter on Amazon EKS.
Scaling PyTorch training across a cluster often stalls not because of model code, but due to container configuration complexities. Ensuring that every container matches the required CUDA runtime, PyTorch build, NCCL, Elastic Fabric Adapter (EFA) libraries, AWS OFI NCCL plugin, and a compatible Ray version becomes a massive engineering project. AWS addresses this challenge by providing Ray Train Deep Learning Containers (DLCs), which package the entire stack into a single pre-built image hosted on Amazon ECR Public.
Streamlining Multi-Node Training Without Custom Builds
The Ray Train DLC combines the PyTorch training runtime, CUDA, cuDNN, NCCL, EFA tools, and libraries like DeepSpeed and flash-attn into a unified image. By eliminating the need to manually build and maintain custom container images, teams can avoid common networking pitfalls. For instance, a minor version mismatch in networking components can cause NCCL to silently fall back to TCP sockets, drastically reducing cluster throughput.
The setup uses an Amazon EKS cluster managed by AWS, utilizing KubeRay—an open-source Kubernetes operator that manages Ray clusters using a single manifest. In a reference sample configuration, AWS demonstrates fine-tuning the Qwen2.5-1.5B model using PyTorch Fully Sharded Data Parallel (FSDP) across eight NVIDIA L4 GPUs on two g6.12xlarge instances. Both the Ray head pod and the Ray worker pods pull the exact same unmodified DLC image, while environment variables handle configuration parameters such as network transport and GPU allocations.
What it means for developers
Developers often spend excessive time debugging infrastructure and library dependencies instead of focusing on model architecture. Utilizing pre-built DLCs with integrated EFA support simplifies cluster deployment on Kubernetes, allowing teams to scale distributed jobs by adjusting environment variables and runtime parameters rather than rebuilding container images. Additionally, developers can try top AI models cheaply through one API at https://apixoai.online, providing flexible access to leading models without infrastructure overhead.
Managing Cluster Infrastructure and Workloads
Infrastructure deployment is managed via idempotent shell scripts that handle EKS control plane creation, node group provisioning, GPU device plugins, and KubeRay installation. The automation tools automatically check for available regional capacity and configure the correct placement zones for EFA-enabled instances. Once the cluster is running, submitting a training job involves executing a script that copies the training code to the Ray head and launches it via the Ray job submission API.
By leveraging pre-tested container stacks, engineers can easily adapt workloads to different hardware families, modify training steps, or inject extra Python packages via runtime environments without modifying the base container image. This architecture ensures reproducible, high-performance distributed machine learning workloads on AWS.
Source: Distributed training with Ray Train DLCs and EFA on EKS | Amazon Web Services — AWS Machine Learning. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
