undefined | Better HN

0 pointsdcrazy4mo ago0 comments

Isn’t the purpose of self attention exactly to recognize the relevance of some tokens over others?

0 comments

1 comments · 1 top-level

That may help with tokens being "ignored" while still being in the context window, but not context window size costs and limitations in the first place.

j / k navigate · click thread line to collapse