The Current Challenges in MoE

To simplify, MoE architectures segment the model into smaller components called experts. During inference, a subset of these experts is selected to contribute to each prediction, while others remain inactive. This structure results in increased speed and reduced costs, making it a preferred design in many leading AI models today. But can we genuinely call these components experts?

Delving into the mechanics of models like ChatGPT reveals the existence of what we refer to as a "Transformer block." For each token in the input sequence, two main operations occur:

1. Attention Mechanism: Each token is updated based on its contextual information, allowing it to grasp the nuances of its meaning. For example, the word "bank" may adjust its understanding based on its association with "river," recognizing it as a riverbank instead of a financial institution.
2. Feedforward Layer (MLP): This layer transforms token embeddings into a higher-dimensional space, capturing more intricate relationships among words and their meanings.

Essentially, Transformers like ChatGPT operate by accumulating updates on word meanings, ensuring that each token considers others to encapsulate the overall meaning of the sequence.

Since Transformer LLMs are autoregressive during inference, the attention mechanism operates in a causal manner. In simpler terms, each word can only reference the preceding words, as the correct subsequent word relies on prior context.

So, where do these so-called "experts" originate? They emerge from partitioning the Feedforward Neural Network (FFN) layer into smaller, identical units. This process begins at the onset of training.

The Cost of Complexity

Despite the benefits, the economic and latency costs of employing large models are significant. Words are not processed in their "word form" but as vectors—dense representations capturing underlying meanings. Each component of the vector conveys attributes of the word, and the length of the vector reflects the granularity of these attributes.

While FFNs introduce non-linearity and project vectors into higher dimensions to uncover subtle meanings, they do so at a high computational expense. Meta reports that FFNs can account for up to 98% of the total computation during a model's forward pass.

Although FFNs are costly, they often exhibit sparsity, meaning many parameters do not activate for each prediction. This leads to a situation where extensive computations occur, yet only a small fraction contributes meaningfully to the next-word prediction.

Herein lies the promise of Mixture-of-Experts: each expert represents a small subset of parameters within an FFN.

An Effective Solution to a Major Issue

The MoE system operates on a straightforward principle. For each prediction, a router selects which experts will be involved while silencing the others. This targeted approach allows for specialization, as experts become adept in specific areas based on their training.

In more technical terms, this process effectively regionalizes the input space. Each expert becomes proficient in distinct topics, offering a diverse range of expertise rather than relying on a single "know-it-all" model. However, this specialization brings forth issues of knowledge redundancy and overlap.

The main advantage of MoE lies in the ability to silence experts selectively, resulting in reduced computational costs and predictable efficiency. Typically, models standardize to using 8 experts, activating only 2 at a time, as seen in architectures like Mixtral 8x7B or GPT-4.

It's crucial to clarify that only the FFN layers are partitioned, while the attention layers remain intact. This distinction is vital, as it misrepresents MoE models to suggest they are merely collections of separate models.

Despite their successes, these architectures face two primary challenges: knowledge hybridity and knowledge redundancy. Knowledge hybridity arises when experts are required to handle a wide array of information due to a limited number of experts. Conversely, knowledge redundancy occurs when experts inadvertently acquire similar knowledge, undermining the purpose of their specialization.

To address these challenges, DeepSeek proposes a novel MoE architecture that incorporates two key enhancements: an increased number of experts—up to 64, which is eight times the norm—and shared experts that engage with every prediction.

The Benefits of Enhanced Specialization

The rationale for increasing the number of experts is simple: greater numbers lead to more focused specialization. This modification results in a wider array of expert combinations. For instance, while a model with 16 experts activating 2 per prediction yields 120 combinations, a model with 64 experts activating 8 can achieve 4,426,165 combinations, allowing for better adaptability to diverse requests.

However, excessive specialization can present challenges, as narrowly focused experts might struggle with broader topics. To counter this, the inclusion of shared experts ensures that foundational knowledge is retained while specialized experts handle specific tasks.

The outcome is promising. The DeepSeekMoE 16B outperforms comparable models, such as LLaMA2 7B, across various benchmarks while utilizing only about 40% of the computational resources.

Video Description: This video surveys various techniques for maximizing the performance of large language models, providing insights into effective strategies for enhancement.