Attention Mechanism

say it: “uh-TEN-shun MEK-uh-niz-um” /əˈtɛnʃən ˈmɛkəˌnɪzəm/

How a model weighs which tokens matter for the next one.

What is Attention Mechanism?

The attention mechanism lets a model weigh which earlier tokens matter most for the next one, like knowing what "it" refers to.

Level: Master · More AI Basics terms at this level

Updated