The NCP team from the Shanghai Artificial Intelligence Laboratory, in collaboration with LUMIA Lab from Shanghai Jiao Tong University, has released NCP-ArchPreview—an open language model with a latent space containing approximately 8.9 billion parameters.
This model is trained simultaneously on standard next-token prediction and Next Concept Prediction. The technical report is available on arXiv under number 2609.10715, and checkpoints for Hugging Face are in the ArchSpace-Collection. Code paths for evaluation and inference have also been published, including links via the InternLM toolkit and the LUMIA evaluation repository.
The project represents the largest demonstration to date of a latent space language model trained at a scale exceeding trillions of tokens.
Instead of predicting only the next token, NCP-ArchPreview is capable of predicting discrete concepts that span multiple tokens. This was achieved by creating a productively quantized concept vocabulary based on the model's hidden states. A specialized Concept Module predicts future concepts, and these predictions are used to guide token-level generation.
The encoder, Concept Module, and decoder utilize a causal Transformer architecture with a hidden layer size of 4096 and a ratio of 16/8/16. Concepts compress the state of four tokens each, with productive quantization applied to 32 vocabularies of 128 codewords. Generation maintains the familiar next-token prediction interface in an NCPOlmo3ForCausalLM-type architecture, allowing the model to be evaluated and used as a standard autoregressive checkpoint, while the latent objective is performed during pre-training.
Approximately 5.73 trillion tokens from the Dolma-3 family were used for training across Stage 1 and Stage 2 curricula. The primary claimed efficiency advantage is that NCP-ArchPreview achieves a final pre-training loss comparable to OLMo-3-7B while using only about 51.3% of the total training tokens.
After full pre-training, the model outperforms the base 7B model by 2.45 points on the mean macro metric across subsequent tasks, including a gain of 5.99 points on the GSM8K benchmark. Controlled ablation analysis showed that this improvement is attributable to both the latent architecture and the next-concept objective function. At a comparable parameter size, the model approaches the training loss of the 8.9B base next-token model while using about 85% of the standard computational budget.
Even after completing pre-training, the latent space remains usable. An update to the vector quantization module, which contains approximately 17 million parameters, provides an easy interface for domain adaptation. Furthermore, incorporating concept representations into the DFlash2 drafter improved the average accepted length by approximately 4.17% with negligible overhead in the described configuration. Base checkpoints from Stage 1 and several Stage 2 stages are available for completion, evaluation, and further adaptation. For researchers tracking pre-training efficiency, NCP-ArchPreview is more of an open recipe for combining concept-level and token-level objectives at nearly 9B scale with publicly available weights and code, rather than just a chat product release.