Deepseek open-sources DeepInfer, the engine behind V4.1-Flash

What this is about

Chinese AI company Deepseek has released a new open-source tool on GitHub called DeepInfer. It is the inference engine behind the Deepseek-V4.1-Flash model. Inference is the everyday operation of an AI: the model receives new inputs and answers them using what it learned in training. According to Deepseek, DeepInfer is meant to cut compute and energy costs and make AI "more affordable." This isn't about prices for users of the Deepseek app or website, but about anyone who runs the model themselves.

What DeepInfer does technically

The framework steps in at several points in the inference stage. Load is spread more evenly across the available processor cores. "Speculative decoding", a technique where likely next words are proposed quickly and then verified, is meant to speed up answers. Outputs are also better structured before they go to users. According to Latent Space, the engine supports distributed inference, SWA bounded replay and structured outputs on Nvidia and Huawei Ascend platforms.

It also includes deepseek-recipe, a collection of Rust libraries. It converts requests from the Chat Completions, Responses and Messages APIs into token sequences, and parses model outputs including reasoning and tool calls. Deepseek says the logic matches the official API. A third-party provider would still have to supply inference, tool execution and HTTP transport themselves. According to the README, this is "offline inference"; the repository doesn't document an API server.

The architecture separates image encoding, prefill (processing the input text) and decode (generating the answer token by token), so each stage can scale on its own. The recommended decode configuration spreads the model's experts across 64 to 128 GPUs, with a prefill-to-decode ratio of about 1:1. The smallest sensible unit is therefore 64 GPUs for prefill and 64 for decode. Deepseek recommends roughly 400 Gbit/s of interconnect per group of eight.

Hardware: not for home users

Supported are Nvidia GPUs with SM100 architecture, meaning data-center Blackwell, as well as Huawei Ascend 950 (which needs the CANN 9.2 toolbox). According to Bloomberg, an Ascend 950 chip costs slightly more than $37,000. Batch-invariant inference (same prompt, same output, regardless of how the batch is composed) and deterministic sampling are available only on Nvidia. The two performance charts in the README are also from Nvidia; the text gives no figures. As of October 20, there were no independent benchmarks, according to Bloomberg.

Operating figures: handle with care

In an accompanying document, Deepseek states that a single cluster processes 155 trillion tokens per day, with revenue of $38 per GPU hour at off-peak times (half the peak rate). Further figures according to Latent Space: a 96% cache hit rate on inputs, 400 ms on average to the first token, 230 tokens/s generation, and up to 300 tokens/s for coding. That is revenue, not cost or margin. Cluster size, chip, input share, and whether the revenue is actually billed or theoretical all remain unclear. For comparison: in March 2025, Deepseek cited a theoretical revenue for V3/R1 that ActuIA converts to $12.9 per GPU hour. The comparison is indirect, since the model and prices differ. Latent Space calculates a margin of over 80% using its own assumptions. That is the newsletter's estimate, not a Deepseek figure. Important: the 155 trillion are served tokens including cache hits, not newly computed ones.

What stays internal

According to the README, a full service is missing the KV cache system (a store of already computed attention states), the separation of phases onto different machines, and load balancing between replicas. For phase separation, most engine features are already implemented; the other components are "tightly coupled to our internal infrastructure" and not fully open. The cache uses 3FS, the distributed file system released in 2025, with about 100 pebibytes of SSD and retention of three days for agent traffic and seven days for chatbot traffic. Of four load balancers, the one for experts is included in the code. Overall, the distribution reaches about 80% of a perfectly balanced offline benchmark.

Background and reactions

In April 2025, Deepseek declined to open its internal engine: it was based on an old vLLM fork, tightly interwoven with its infrastructure, and the small team couldn't maintain a large project. Now the README still names vLLM as its base, with SGLang and TensorRT-LLM as inspiration. Deepseek gives no explanation for the change of course. What's new: the engine targets multiple chip vendors, and inference code for future models is meant to appear there. So far only V4.1-Flash runs; V4.1-Pro has been announced but has no date. The vLLM team, whose project is licensed under Apache 2.0, reports that on GB200 with the same configuration it comes within 6% of DeepInfer's decode throughput. That undercuts the assumption that Deepseek's efficiency is out of reach for existing open-source engines.

Our take

DeepInfer is mainly interesting for operators of large Blackwell or Ascend clusters, not hobbyists. A look into the practice of a provider with enormous throughput is instructive, but the notion of "fully open" holds only in part, since the cache and orchestration are missing. All performance and revenue figures are self-reported and unaudited, and the vLLM comparison suggests the lead may be smaller than claimed. Cost, margin and how well this carries over to other setups remain open questions.