ModuleFormer: Modularity Emerges from Mixture-of-Experts (opens in new tab)

(arxiv.org)

1 pointsdhruvdh2y ago1 comments

1 comments

1 comments · 1 top-level

GitHub Repo: https://github.com/IBM/ModuleFormer

GitHub Description: ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters.

j / k navigate · click thread line to collapse