undefined | Better HN

0 pointsrfoo1y ago0 comments

IMO the relevant benchmark for now is a mixed stream of requests with 50 (20%), 500 (50%), 2000 (10%) and 50k (20%) input tokens, ignore EOS and decode until you get around 300 output tokens.

0 comments

SuchAnonMuchWow1y ago

I'm really interested, do you have a source for those percentages ?

I tried to look for some service provider to publish this kind of metrics, but haven't found any.

rfooOP1y ago

Sorry, I can't. My employer doesn't publish this kind of metrics, either. What I posted was definitely just some very rough number off my brain.

j / k navigate · click thread line to collapse