AMD Enters the Open LLM Race with a Fully Open-Source MoE Model

In a significant move for the open-source AI community, AMD has released Instella-MoE, a Mixture-of-Experts language model with 16 billion total parameters. What sets this launch apart is not just the model's performance, but AMD's commitment to full transparency, releasing all model weights, training configurations, and even intermediate checkpoints to the public.

Engineered for Efficiency: Novel Architecture Meets Custom Hardware

Instella-MoE is built around the MoE paradigm, designed to be computationally efficient during inference. While the model boasts 16B parameters, it only activates 2.8B parameters per token. This sparse activation is key to its potential for faster, more resource-friendly deployment compared to dense models of similar capability.

To achieve this, AMD integrated several architectural innovations:

  • Gated Multi-head Latent Attention (Gated MLA): Aims to optimize the attention mechanism, potentially improving efficiency on long sequences.
  • FarSkip-Collective Design: Enhances gradient flow and information transfer across the deep network, improving training stability.

The model was trained from scratch exclusively on AMD Instinct MI300X/MI325X accelerators using the ROCm software stack. This end-to-end development on AMD's own hardware and software platform serves as a powerful demonstration of its full-stack AI capabilities.

Benchmark Performance: Competing Beyond Its Size

In performance benchmarks, the base Instella-MoE model achieved an average score of 76.7. This positions it competitively among fully open-source models of comparable scale, outperforming several other well-known open models.

Notably, by leveraging the efficiency of the MoE design, this model—activating only 2.8B parameters—shows performance that challenges larger, dense models in certain evaluations. It also supports a context window of 64K tokens and has undergone a comprehensive training pipeline including pre-training, supervised fine-tuning, and reinforcement learning from human feedback.

A New Standard for Openness in AI Research

The most impactful aspect of this release is AMD's unprecedented level of openness. By providing not just the final model but the complete blueprint—training recipes, data mixes, intermediate checkpoints, and inference code—AMD is effectively democratizing the research and development process for state-of-the-art LLMs.

This move lowers the barrier for academic and independent researchers to study, replicate, and build upon advanced model training. AMD has stated that Instella-MoE represents a key step in its roadmap, with plans to continue developing larger, more capable, and more efficient open-source language models. Its entry adds a significant new voice and technological alternative to the evolving open AI ecosystem.