Fast KV Compaction via Attention Matching (opens in new tab)

(arxiv.org)

73 pointscbracketdash1mo ago15 comments

15 comments

Considering the insanity of the AI arms race going on now, and the incredible sums of money be thrown at any slight advantage, is there any reason to believe that any meaningful AI breakthrough would be openly published for anyone to leverage?

5424581mo ago

These folks are MIT, so citations are valuable to them. Citations convert into prestige, academic career progression, or a favorable exit from academia into industry.

Also, I don't see why you couldn't patent this if you wanted to monetize it.

BetaDeltaAlpha1mo ago

> Also, I don't see why you couldn't patent this if you wanted to monetize it.

We all just saw the prior art published for the public. That will preclude patenting this work. Further reduction to practice is required.

(I am not a lawyer).

jph001mo ago

Yes there is. Lots of researchers are more interested in making a contribution to societal flourishing than in making incredible sums of money. That’s why there’s still lots of top AI researchers in academia.

abeppu1mo ago

I do sometimes wonder -- if the transformers paper wasn't published, what would the industry be like? Would the same ideas have been put together in almost the same way weeks or months later somewhere else?

mikodin1mo ago

I would say yes.

The reality is that the money being thrown = the time of humans. I guess compute as well, but in terms of people doing innovation - openly published things are the same thing, minus the money.

cma1mo ago

The inventor's grace period under first to file changes still gives them/their university a year to file if they publish openly.

gdiamos1mo ago

I know the frontier “labs” are holding back publications.

I don’t think it will last among researchers who think beyond production LLMs

cadamsdotcom1mo ago

Superficially it sounds like this could create a bit more of a move toward doing compaction on some continuous basis, or compacting in batches once you hit the context limit, rather than starting fresh with a summary and system prompt..

Feels like high fidelity, fast compaction could be a path to “solving” long context.

cs7021mo ago

This looks promising. I've added it to my reading list.

speedping1mo ago

This is big for long-horizon tasks

esafak1mo ago

None of the compaction accuracies look impressive.

yorwba1mo ago

I think matching or exceeding the original cache at 20% compacted size is fairly impressive.

esafak1mo ago

The original cache had 70% accuracy, and the alternatives were only worse.

1 more reply

j / k navigate · click thread line to collapse

15 comments

WarmWash1mo ago

5424581mo ago

These folks are MIT, so citations are valuable to them. Citations convert into prestige, academic career progression, or a favorable exit from academia into industry.

Also, I don't see why you couldn't patent this if you wanted to monetize it.

BetaDeltaAlpha1mo ago

> Also, I don't see why you couldn't patent this if you wanted to monetize it.

We all just saw the prior art published for the public. That will preclude patenting this work. Further reduction to practice is required.

(I am not a lawyer).

jph001mo ago

abeppu1mo ago

mikodin1mo ago

I would say yes.

The reality is that the money being thrown = the time of humans. I guess compute as well, but in terms of people doing innovation - openly published things are the same thing, minus the money.

cma1mo ago

The inventor's grace period under first to file changes still gives them/their university a year to file if they publish openly.

gdiamos1mo ago

I know the frontier “labs” are holding back publications.

I don’t think it will last among researchers who think beyond production LLMs

cadamsdotcom1mo ago

Feels like high fidelity, fast compaction could be a path to “solving” long context.

cs7021mo ago

This looks promising. I've added it to my reading list.

speedping1mo ago

This is big for long-horizon tasks

esafak1mo ago