AWS Releases SGLang-Neuron for Serving Large Language Models on Trainium Silicon
Amazon has released SGLang-Neuron, enabling high-performance LLM serving and RL rollouts on AWS Trainium chips using standard PyTorch and SGLang APIs.
Amazon has officially announced the public availability of SGLang-Neuron, bringing the popular SGLang LLM serving framework to AWS Trainium silicon. Delivered as an AWS Deep Learning Container through a collaborative effort between RadixArk and Annapurna Labs, SGLang-Neuron allows developers to serve models and execute reinforcement learning (RL) rollouts using the standard SGLang server interfaces and generation APIs they already rely on.
AWS Trainium chips are engineered to offer favorable economics for large-scale AI training and inference. With this release, teams can leverage Trainium hardware without modifying their existing SGLang workflows or client-side request patterns.
Architecture and Under-the-Hood Performance
SGLang-Neuron relies on an out-of-tree plugin design that links SGLang with the underlying AWS Trainium architecture. By keeping the Trainium integration separate from the SGLang core codebase, both projects can iterate independently on faster development cycles. Installing the plugin hooks into SGLang's native extension entry points, enabling hardware selection simply by setting the SGLANG_PLATFORM=neuron environment variable at startup.
Under the hood, SGLang core continues to oversee primary orchestration tasks, including the API server layer, request lifecycle management, RadixAttention, and prefix-cache bookkeeping. Meanwhile, the SGLang-Neuron plugin handles hardware-specific components such as custom model implementations, specialized attention mechanisms, Mixture of Experts (MoE) operations, and the graph runner.
To execute workloads efficiently, TorchNeuron exposes Trainium as a native PyTorch hardware device. The model path combines standard PyTorch operations with high-performance kernels built using the Neuron Kernel Interface (NKI). These operations and NKI kernels are compiled into executable binaries via torch.compile, which the graph runner manages during live endpoint serving.
Additionally, SGLang-Neuron supports Prefill-Decode (PD) disaggregation to handle massive generation workloads. Under a PD disaggregated setup, a dedicated prefill server computes the initial prompt key-value (KV) cache and streams it to a separate decode server over Elastic Fabric Adapter (EFA) using NIXL. The decode node then produces the target output tokens. Throughout this transfer, each instance continues utilizing SGLang's internal scheduler while the plugin coordinates Trainium execution and network transfers.
Deployment Setup and Model Serving
Deploying an endpoint using SGLang-Neuron requires launching an AWS Deep Learning Container on a Trainium3 Amazon Elastic Compute Cloud (EC2) instance. Amazon provides a pre-built container image (public.ecr.aws/neuron/pytorch-inference-sglang-neuron:v0.1.0-neuron-py314-sdk-2.33.0-ubuntu26.04) equipped with PyTorch and the required Neuron SDK components.
Once the container is running in an interactive session, launching the model server involves executing sglang.launch_server with the SGLANG_PLATFORM=neuron flag enabled. In an official example deployment, AWS demonstrated hosting a 120-billion parameter model (openai/gpt-oss-120b) across an 8-way tensor parallel (--tp-size 8) and 8-way expert parallel (--ep-size 8) configuration. The launch command configures key parameters such as FP8 KV cache data types (fp8_e4m3), a 10,240 token context length, a chunked prefill size of 8,192, and a maximum total token cap of 655,360 across 64 running requests.
Once active, the server accepts standard OpenAI-compatible REST API requests via /v1/chat/completions, making integration into application stacks straightforward.
What it means for developers
The introduction of SGLang-Neuron lowers the entry barrier for engineering teams wanting to run high-throughput LLM workloads on cost-effective AWS Trainium instances. Because SGLang-Neuron implements the exact same API surface as standard SGLang deployments, teams can migrate existing serving infrastructure to Trainium without refactoring API calls, client libraries, or orchestration pipelines.
Furthermore, the out-of-tree plugin design ensures that developers benefit from core SGLang innovations—such as RadixAttention and prefix caching—while receiving independent performance optimizations for Neuron hardware. Features like Prefill-Decode disaggregation over EFA provide clear scaling paths for high-concurrency production setups, particularly when serving large MoE architectures or long-context tasks.
For developers evaluating infrastructure options or comparing custom hardware against hosted model endpoints, platforms like Apixo allow engineers to try top AI models cheaply through one API at https://apixoai.online before committing to full-scale infrastructure deployments.
Ultimately, SGLang-Neuron offers developers a practical path toward reducing inference costs on AWS while retaining the tools, APIs, and frameworks they already know.
Source: Deploy frontier AI models on AWS Trainium with SGLang-Neuron | Amazon Web Services — AWS Machine Learning. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
