88Z AI, 11 November — Baidu has released the ERNIE-5.5-Flash-Base foundation model, making its pre-training endpoint weights available. This sparse MoE model features 280B total parameters with 14B activated per token, pre-trained from scratch on 33.5T tokens. It supports a 128K training context and up to 1M for fine-tuning and inference. The weights are unprocessed by mid or post-training, making them suitable for domain adaptation, instruction tuning, and reinforcement learning.
At a glance
- ERNIE-5.5-Flash-Base is a 280B parameter sparse MoE model pre-trained from scratch on 33.5T tokens.
- Releases unprocessed weights supporting 1M context, suitable for domain adaptation and instruction tuning.
The update
Baidu has officially launched its latest foundation model, ERNIE-5.5-Flash-Base, while simultaneously releasing its pre-training endpoint weights. The model utilizes a sparse Mixture of Experts (MoE) architecture, boasting a total of 280 billion parameters with 14 billion activated per token. It was pre-trained from scratch on a dataset of 33.5 trillion tokens, establishing robust foundational language capabilities. Regarding context handling, ERNIE-5.5-Flash-Base supports a training context length of 128K, extending up to 1 million tokens for fine-tuning and inference. Notably, the released weights have not undergone mid-training or post-training processes, retaining their original pre-trained state. This provides developers with a flexible starting point for subsequent optimizations, including domain-specific adaptation, instruction tuning, and reinforcement learning.
Sources & editorial note
This article is an AI-assisted synthesis by 88Z AI based on the public sources below. Product claims and availability are attributed to their publishers. It does not represent independent testing or first-hand reporting.
- 文心大模型Source published
Source dates above refer to the original announcements; the date under the headline is this article’s publication on 88Z AI.