Models & resources
StableLM
Stability AI's open-source language model series offering efficient pre-trained
What is StableLM?
StableLM is a series of open-source language models developed by Stability AI, designed to provide researchers and developers with high-performance, reproducible natural language processing bases.The repository is continuously updated with the latest model checkpoints, covering various architectures from lightweight to medium-scale parameters, supporting innovation and application under transparent licenses.
The series includes models like StableLM-3B-4E1T, which employs a multi-epoch training strategy, and instruction-tuned versions such as StableLM-Tuned-Alpha.These models are built on an architecture similar to LLaMA but improved with techniques like Rotary Position Embeddings and LayerNorm to optimize inference efficiency and stability, with interactive demos available on platforms like Hugging Face.
All base models are trained on strictly filtered open-source datasets, including Falcon RefinedWeb, RedPajama-Data, and The Pile (with specific subsets excluded), ensuring data quality and diversity.StableLM is committed to advancing research in small, efficient models, exploring paths to achieve optimal performance under limited computational resources.
Key features
- Efficient ParameStableLM-3B-4E1T contains only about 2.8 billion parameters but achieves loss values comparable to larger models by training for 4 epochs on 1 trillion tokens.
- Improved TransfoUses a decoder-only structure similar to LLaMA but introduces Rotary Position Embeddings to improve throughput and replaces RMSNorm with LayerNorm with learned bias terms.
- High-Quality TraTraining data mixes large open-source datasets like Falcon RefinedWeb, RedPajama-Data, and The Pile, excluding specific subsets like Books3 to ensure compliance and quality.
- Open-Source FrieBase models such as StableLM-3B-4E1T and the StableLM-Alpha series are often released under open licenses like CC BY-SA-4.0, facilitating academic research and commercial exploration.
- Zero-Shot EvaluaProvides zero-shot evaluation results on multiple standard benchmarks including ARC, BoolQ, and HellaSwag, allowing users to compare model capabilities in commonsense reasoning and QA tasks.
- Continuous IteraThe repository continuously releases new checkpoints, including patched Alpha v2 models and RLHF fine-tuned versions optimized for specific tasks like StableVicuna.
How to use it
- Visit the official GitHub repository page and browse the README to learn about the currently available model list and their features.
- Choose the appropriate model version based on your needs, such as selecting StableLM-3B-4E1T for lightweight deployment.
- Check the provided YAML configuration files and technical reports to obtain complete hyperparameter settings and training details.
- Experience the chat models online via Hugging Face Spaces, or download weights to perform inference tests in your local environment.
- If optimization for specific downstream tasks is required, it is recommended to fine-tune the base model and refer to the official evaluation scripts.
Where it fits
- Academic ResearcUtilize open-source weights and detailed technical reports to study the impact of multi-epoch training on language model performance and data efficiency.
- Lightweight ApplDeploy 3B parameter models on resource-constrained edge devices or servers to achieve low-latency text generation and dialogue capabilities.
- Custom Task FinePerform instruction tuning on high-quality base pre-trained models for vertical domain tasks such as customer service, writing, or code assistance.
Pricing and free quota
Pricing and any free quota follow the official site.
Who it is for
- AI Researchers
- Open-Source Community Developers
- Edge Computing Engineers
- NLP Enthusiasts
FAQ
What does 4E1T mean in StableLM-3B-4E1T?
It indicates that the model was trained for 4 epochs on a dataset of 1 trillion tokens, aiming to study the impact of repeated data on model performance.
Can I use StableLM for commercial projects?
This depends on the specific model's license.For example, some base models use CC BY-SA-4.0, while certain fine-tuned versions may use different licenses.Please check the specific license description for each model in the official repository.
What is the relationship between StableLM and LLaMA?
StableLM's architecture is similar to LLaMA but includes modifications in positional encoding and normalization layers.Additionally, some early versions like StableVicuna are further fine-tunes based on LLaMA derivatives.