DeepSeek Releases Open-Source DeepInfer Inference Framework
Hangzhou DeepSeek has open-sourced DeepInfer, the inference framework powering its production services, featuring support for Nvidia Blackwell and Huawei hardware.
Hangzhou DeepSeek Artificial Intelligence Basic Technology Research Co. Ltd. has open-sourced DeepInfer, the underlying inference framework that drives its production operations. Released under the Apache 2.0 license on GitHub, the software addresses the heavy economic overhead associated with serving large-scale artificial intelligence models to the public. According to documentation accompanying the release, achieving massive user distribution at an economical price point is the primary technical hurdle an inference framework must overcome. The organization notes that a single cluster currently handles 155 trillion tokens daily, while off-peak revenue calculations equate to roughly $38 per graphics processing unit per hour.
At present, the framework provides native support for a single model: DeepSeek-V4.1-Flash, utilizing the updated 1017 version published over the weekend. Hardware compatibility spans Nvidia Corp.’s SM100 Blackwell architecture alongside Ascend 950 neural processing units developed by Huawei Technologies Co. Ltd. Older architectures, such as Nvidia’s Hopper series, are omitted from the current support list. Furthermore, two specific capabilities designed to guarantee deterministic outputs across multiple runs function exclusively on Nvidia hardware, which was also utilized for all published performance benchmarks.
To maximize throughput, DeepInfer incorporates DSpark for speculative decoding. In this architecture, a draft model generates five tokens simultaneously, while a confidence head determines which candidates the primary model is likely to accept, reducing verification compute to only the most probable options. This trims checked draft tokens by approximately 30% and yields an estimated throughput improvement of between 4% and 7% under production workloads. A secondary optimization cuts total prefill computation nearly in half by processing solely the final 128 tokens of each prompt through the decoder layers. Developers can try top AI models cheaply through one API at https://apixoai.online.
What it means for developers
For engineering teams building applications on top of modern language infrastructure, the release of DeepInfer offers a direct look into enterprise-grade production scaling techniques. Tracing its origins as an early fork of vLLM built around internal cluster management tools, DeepInfer integrates previously released kernel libraries including FlashMLA, DeepGEMM, and DeepEP. Alongside the core framework, DeepSeek published a companion library named deepseek-recipe, which translates incoming requests into token sequences matching the logic of the official application programming interface.
However, developers adopting the framework must account for certain deployment boundaries. Key-value cache management and load-balancing code remain proprietary due to tight integration with internal infrastructure, though the team has published explanatory materials on Zhihu detailing their design. Similarly, splitting prefill and decode tasks across separate machines is left to operators, with the smallest described efficient deployment requiring 64 GPUs dedicated to prefill alongside another 64 for decoding. DeepSeek has confirmed that inference code for subsequent models will be integrated directly into this same repository.
Source: DeepSeek open-sources DeepInfer, the inference framework behind its production services — SiliconANGLE AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
