Mixture of Experts

say it: “MIKS-cher uv EK-sperts” /ˈmɪkstʃər əv ˈɛkspɝts/

A model routing each token to a few specialist parts.

What is Mixture of Experts?

A mixture of experts model has many specialist sub-networks and uses only a few for each token, giving big capacity at lower cost.

Level: Master · More AI Basics terms at this level

Updated