Shortformer: Better Language Modeling Using Shorter Inputs (Paper Explained) (opens in new tab)

(youtube.com)

3 pointsdeeplstm5y ago1 comments

1 comments

Modelling long sequences has always been hard for transformer-based models. This paper proposes a super innovative way for the transformer to cache previously processed tokens. And it makes generation 9X faster. This is truly mind-blowing

Paper https://arxiv.org/abs/2012.15832

Code https://github.com/ofirpress/shortformer

j / k navigate · click thread line to collapse

1 comments

deeplstmOP5y ago

Paper https://arxiv.org/abs/2012.15832

Code https://github.com/ofirpress/shortformer

j / k navigate · click thread line to collapse