DeepInfer: DeepSeek open-sources the inference engine behind its production service under Apache 2.0
What is it?
DeepInfer is the inference framework DeepSeek runs behind its own API. It started from vLLM and was heavily reworked, adding DeepSeek's own kernels (FlashMLA, DeepGEMM, DeepEP), expert parallelism (spreading an MoE model's experts across GPUs), structured outputs and speculative decoding. For now it supports only DeepSeek-V4.1-Flash and runs on NVIDIA SM100 (Blackwell) and Huawei Ascend 950.
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.pyWhy does it matter?
You get to read, in code, how a large MoE model is served at production scale. DeepSeek says a single cluster handles 155 trillion tokens a dayVendor claim. KV cache management, prefill/decode disaggregation and load balancing across replicas are left out, though, and Hopper and older GPUs aren't supported. Treat it as a design reference for teams running their own inference stack rather than something to drop into production.
Maturity
BetaThe code is public but covers offline inference only; pieces a full service needs are missing, so we're calling it beta.
How to track it
- Missing pieces: watch GitHub for the KV cache system and load-balancing code.
- Coverage: check the README for models beyond V4.1-Flash or Hopper support.
- vLLM and SGLang: see whether community engines pick up DeepInfer's optimizations in their PRs.