allworld.mediaNews from all over the world
DeepSeek releases DeepSeek-V4.1-Flash model with MoE architecture, 1 million token context, and KV compression
Read more
Pandaily
pandaily.com

DeepSeek releases DeepSeek-V4.1-Flash model with MoE architecture, 1 million token context, and KV compression

DeepSeek has introduced the DeepSeek-V4.1-Flash model, which is a multimodal Mixture-of-Experts (MoE) model with 552 billion parameters. This model is built on the Causal Encoder–Decoder architecture and supports a context window of one million tokens. The company positions it as the smallest member of a new architectural line aimed at enhancing capabilities, accelerating decoding, and increasing throughput during deployment, making it available via the API named deepseek-flash.

The Causal Encoder–Decoder stack utilizes a 40-layer transformer, divided into a 20-layer causal encoder and a 20-layer decoder. Upon prefix activation, approximately 8 billion parameters are activated per token, and about 16 billion during decoding, allowing input-intensive workloads to consume fewer resources compared to similarly sized symmetric MoE designs.

To optimize performance, methods such as Compressed Sparse Attention 2 (CSA2) are used, which distributes Full, Reindex, or Reuse modes across layers, alongside FP4 main KV cache, which reduces the total KV volume to approximately 890 bytes per token—about a quarter of the volume of DeepSeek-V4-Flash. Furthermore, SWA Bounded Replay reduces the constant KV demand on SSDs by about one eighth.

Training was conducted on approximately 45 trillion multimodal tokens, including sparse attention on 64K sequences, with the context extension to 1 million occurring later in the pre-training process. This was followed by supervised fine-tuning, reinforcement learning, and policy distillation, accompanied by intensive automated agent task synthesis.

In agent instruction benchmarks, under maximum reasoning effort, DeepSeek reports the following results: Terminal-Bench 2.1 achieves 90.6, DeepSWE v1.1 reaches 74.2, and AutomationBench scores 54.8. The model also features native vision support thanks to the custom-built DeepSeek-ViT encoder.

The weights and inference recipes have been published on Hugging Face under the MIT license, and DeepSeek states support for open-source inference adapters for large-scale deployments. Older V4 Flash aliases are temporarily redirected to V4.1-Flash; DeepSeek also plans to phase out V4 Pro routing in favor of the new Flash SKU. The main focus is on the architecture and cost-efficient long-context inference based on KV compression and asymmetric activation, making the servicing of agent sessions with one million tokens cheaper.

Similar stories

OpenBMB releases MiniCPM5-2B as an open SOTA model for on-device operation with less than 4 billion parameters
Read more
pandaily.com

OpenBMB releases MiniCPM5-2B as an open SOTA model for on-device operation with less than 4 billion parameters

OpenBMB, an open project associated with ModelBest, has introduced MiniCPM5-2B—a dense language model designed for direct on-device operation. This model features approximately 2.52 billion parameters and supports a native context length of 131,072 tokens, distributed under the Apache-2.0 license.

MiniCPM5-2B is the second release in the MiniCPM5 series, following MiniCPM5-1B. It utilizes the standard LlamaForCausalLM architecture, which includes 42 layers with grouped-query attention, employing 16 query heads and 2 key/value heads. This design allows core engines to load the model without requiring custom kernels or code forks.

The non-embedding parameters amount to about 1.98 billion, enabling the model to fit within the sub-4-billion parameter class suitable for hosting on phones, laptops, and edge devices.

When compared on the 34-benchmark dataset from OpenBMB, MiniCPM5-2B achieves an average score of 53.9, surpassing several larger open base models. Among these, Qwen3.5-4B scored 51.1, and granite-4.2-3B scored 42.7. Conversely, similarly sized models like LFM2.5-2.6B and Qwen3.5-2B lag behind.

The model's strengths are concentrated in areas such as coding agents, tool usage, and long-context information extraction. For instance, on LiveCodeBench v6, MiniCPM5-2B achieved 69.1 compared to 56.4 for Qwen3.5-4B, and on SWE-bench Verified, it reached 46.4 versus 33.6. High results were also obtained on τ²-Bench Telecom (97.1), BFCL v4 (66.6), and NoLiMa (68.1 versus 43.5).

In tasks requiring deep knowledge, performance remains closer to the upper limit of its size; for example, on MMLU-Pro, the model scored 70.8, while the benchmark 4B Qwen model achieved 78.0. OpenBMB specifically notes rows sourced from Artificial Analysis to allow users to distinguish between vendor data and third-party results.

The post-training process followed the UltraData level management: initially, supervised fine-tuning was conducted based on deep reasoning using approximately 400 billion tokens. Subsequently, specialized reinforcement-learning teachers for mathematics, code, agents, and writing were applied using the critique-based JustRL II algorithm. The final stage involved on-policy distillation, which merged 16 RL experts—five of which were agentic—into a single final student model.

OpenBMB attributes the average gain of about 10.96 points in reasoning and general benchmarks, along with 6.96 points in agentic benchmarks, to the RL and distillation stages. To allow for precise measurement of each component's contribution, the company releases intermediate checkpoints: Base, Midtrain, and SFT-only.

Along with the weights, UltraData publishes datasets covering web, code, mathematics, SFT, and RL samples for agents, enabling result reproducibility. Runtime support includes vLLM, SGLang, Transformers, llama.cpp, Ollama, LM Studio, MLX, LiteRT, and multi-chip FlagOS builds. Packages in GGUF, MLX, and GPTQ formats are provided for local assistants that need to perform tool calls and handle long context without relying on datacenter GPUs.

For developers focused on on-device deployment, MiniCPM5-2B is positioned not so much as a general knowledge giant, but rather as a compact Apache-licensed agent and coding companion that already surpasses some open 4B class base models based on published average scores.

Popular