undefined | Better HN

0 pointstibbar2mo ago0 comments

Doesn't context cacheing mostly eliminate this problem? (I suppose for enough context the 90% discount is eventually a lot anyway)

0 comments

1 comments · 1 top-level

zozbot2342mo ago

Context caching is really storing the KV-cache for reuse. It saves running prefill for that part of the context, but tokens referencing that KV-cache will still cost more.

j / k navigate · click thread line to collapse