Attention Mechanism
say it: “uh-TEN-shun MEK-uh-niz-um” /əˈtɛnʃən ˈmɛkəˌnɪzəm/
How a model weighs which tokens matter for the next one.
What is Attention Mechanism?
The attention mechanism lets a model weigh which earlier tokens matter most for the next one, like knowing what "it" refers to.
Level: Master · More AI Basics terms at this level
Updated