Vera: Vector-Based Random Matrix Adaptation (opens in new tab)

(dkopi.github.io)

1 pointsegnehots2y ago3 comments

3 comments

3 comments · 2 top-level

code-master2y ago· 1 in thread

Is the code available anywhere

Highly skeptical paper 1) Authors havent released code 2) They claim its published in ICLR , but i wasnt able to find the paper in openreview https://openreview.net/group?id=ICLR.cc/2024/Conference#tab-... 3) GPT-4 to evaluate performance. It seems to be grading the responses rather arbitrarily and not really considering how well the adapted model has learned the particular behavior it was being tuned for.

This just screams 0% code 0% peer review 100% Hocus Pocus.

egnehotsOP2y ago

VeRA makes LoRA ~10x more parameter efficient while retaining the same performance.

It's somewhat like a recursive LoRA scheme, where the LoRA A and B matrices are also decomposed using two small trainable vector parameters.

j / k navigate · click thread line to collapse