NaiveAI releases Naive-N0.5-Flash model: 309B MoE with 1M context under MIT license

Baidu has introduced the preliminary version of the ERNIE 5.5 Flash model on November 11. This version is the first post-trained model based on the ERNIE-5.5-Flash-Base model, which Baidu opened on November 6. The model has a total of 280 billion parameters, with 14 billion parameters active per token, and supports a context window of 1 million tokens.

Baidu claims that the model demonstrates high performance in areas such as programming, 3D modeling, knowledge work, and content creation, while also showing competitive results in benchmarks with very fast decoding. These statements are Baidu's own assertions, as the benchmark table was not available in text format in the original material.

Model Access

A key point is availability: the API is already operational on Baidu's Qianfan platform. The list of models on this platform, updated on November 10, includes the ernie-5.5-flash-preview endpoint, which has a 1 million token context window and is limited by default to 60 requests and 1 million tokens per minute. However, the issue of open weights remains separate. Baidu promised to make the weights publicly available by November 25, but this is currently only a promise, not a confirmed fact.

Upon checking on November 11, the Preview repository on Hugging Face returned an access error, and Baidu's public list on Hugging Face only displayed two Base checkpoints. Nevertheless, Baidu's vLLM-ERNIE repository lists Preview in FP4, FP8, and BF16 formats in its supported models table, and the deployment guide from the SGLang project indicates that a hourly version will be available soon.

A third-party catalog describes this model as an open-weight model, but this could not be confirmed through public download. The base model repository describes a sparse mixture-of-experts design, pre-trained on 33.5 trillion tokens, with 512 routed experts per layer, hybrid attention compressing most layers, and built-in speculative decoding draft. The preliminary chat model adds reasoning, tool-calling capabilities, and an included draft.

Baidu provides approximate weight sizes: 155 GB for FP4, 280 GB for FP8, and 560 GB for BF16. The company refers to the Preview as an intermediate result of post-training and warns that insufficient training may still lead to capability gaps, noting that post-training is ongoing and the model plans to be improved in the official release based on real user feedback.

The ERNIE team at Baidu has announced the release of ERNIE-5.5-Flash-Base, which is an unrefined pre-training checkpoint of the ERNIE 5.5 Flash model. This release was posted on the Hugging Face platform on November 5th.

The model, which utilizes a sparse Mixture-of-Experts (MoE) architecture, has a total of 280 billion parameters, with 14 billion parameters activated per token. It was trained from scratch on 33.5 trillion tokens, with a maximum training context size of 128 thousand tokens, although support for context in fine-tuning and inference reaches 1 million tokens.

Baidu intentionally released the model in an unfinished state. The weights are obtained from the completion phase of pre-training and have not yet undergone intermediate or subsequent training stages. The company states that this makes it a general starting point for research in domain adaptation, instruction tuning, and reinforcement learning. The team reported that a post-trained version of ERNIE 5.5 Flash will be released later.

The available weights are provided in FP8 and BF16 formats according to the Baidu ERNIE 5.5 Flash Base Model v1.0 License Agreement. The model can be used via SGLang and vLLM, and is accompanied by a fine-tuning example in the NVIDIA Megatron-Bridge repository.

This 43-layer model is designed with an emphasis on efficiency. Most layers use what is called Baidu Sparse Attention, which merges every 128 tokens into a single cache entry. Only one of the seven layers retains access to the full token-level context through multi-head latent attention with top-k sparse sampling. In total, 37 out of 43 layers operate with compressed sequences, allowing for a small KV cache. Furthermore, the low-rank gating attention design reduces the gate projection parameters by 75%, according to the developers.

In terms of MoE, Baidu replaced the standard single router with multiple independently initialized routers that jointly evaluate experts. The company claims that this improves the alignment between routing scores and actual expert utility without changing the number of active experts, computational power, or communication. Each MoE layer selects 10 out of 512 routed experts plus one shared expert, operating in a compressed latent space, which roughly halves the expert communication and computation. A component called ECHO reuses the last layer of the model to predict future tokens instead of adding a separate multi-token prediction module and serves as a draft model for speculative decoding.

The Muon optimizer with mixed-precision FP8 was used for pre-training across thousands of accelerators, along with the new FlashMask-4 attention kernel. Baidu asserts that this kernel operates up to 175% faster in the forward pass than FlashAttention-4 on SM100 class GPUs at large head sizes. Data mixing focused on repository-level code, including commits, merge requests, code reviews, CI-verified fixes, and long documents.

Baidu tested the model on 30 benchmarks, comparing it with DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base using a unified evaluation system. Reported results include 78.4 points on MMLU-Pro versus 69.2 for DeepSeek-V4-Flash-Base, and 82.3 points on HumanEval+ versus 69.5. However, the compared models outperformed in some tests, such as ChineseSimpleQA, BBH with chain-of-thought prompting, and InfiniteBench.

Xiaomi has announced the release of the Xiaomi-OCR-0 model, a lightweight document understanding model. The weights and inference code for this model are available on Hugging Face and GitHub, the company reported on October 30th.

The model boasts 0.8 billion parameters and is designed to perform three types of tasks within a single system: document parsing, key information extraction, and visual question answering via OCR.

The parsing functionality includes recognizing main text, tables, and mathematical formulas, outputting structured results in reading order with support for native Markdown and LaTeX formats. For key information extraction, the model focuses on invoices, receipts, IDs, and business forms, locating requested fields and providing the result in JSON format for automated workflows.

Visual question answering goes beyond simple transcription, enabling cross-regional searching, numerical calculations, and reasoning based on documents and charts.

Training and Methodology

Xiaomi states that the model is built upon the Qwen3.5-0.8B-Base model and was trained on approximately 170 million OCR and document samples. Data labels were filtered by achieving consensus among several expert OCR models that performed mutual verification and voted on areas of text, tables, and formulas.

When working with complex tables and formulas, the team utilized a 'predict, re-render, and compare' loop: candidate results were re-rendered into images and compared with the original at the pixel and structural level to detect shifted rows or missing characters. Synthetic complex cases included multilingual text, historical handwriting, screen moiré effects, and distorted lighting.

The training process occurred in three stages: first, textual fine-tuning to align visual regions with text tokens; second, continued pre-training of OCR on tens of billions of tokens; and finally, reinforcement learning for mixed tasks with verifiable rewards, such as edit distance and table matching scores.

Testing Results

On the OmniDocBench v1.6 benchmark, the model achieved a score of 96.83, ranking second overall, only 0.04 points behind the leader, and becoming the top performer among models with less than one billion parameters. In the Real5-OmniDocBench test, which simulates scanning, crumpled paper, screenshots, lighting variations, and tilted frames, the model scored 95.24. Its performance on Wild-OmniDocBench was 87.94.

The model reaches 97.84 on the UniMER-Test reference formula test and 92.27 on PubTabNet for table structure evaluation. Averaging results across DocVQA, InfoVQA, ChartQA, OCRBench, and TextVQA, it scores 83.2, surpassing HunyuanOCR-1.5 with 76.5 and 8B MiniCPM-V-4.5 with 82.6, although it trails Qwen3.5-4B with a score of 84.9. It should be noted that all presented figures are proprietary data from Xiaomi.

Xiaomi reports that this technology is already integrated into the Super XiaoAi assistant on its phones, where it can extract boarding passes, queue numbers, bill details, and meeting times directly from the screen. In addition to the model weights, Xiaomi provides developers with a browser demo version, the MCP service, and an Agent Skill toolkit.