Skip to content
Better HN
Top
Best
Ask
Show
New
Jobs
Search
⌘K
0 points
alew1
3y ago
0 comments
Save
Share
But the model ultimately still has to process the comma, the newline, the "job". Is the main time savings that this can be done in parallel (on a GPU), whereas in typical generation it would be sequential?
0 comments
1 comments · 1 top-level
top
newest
oldest
sebzim4500
3y ago
Yes. If you look at the biggest models on OpenAI and Anthropic apis, the prompt tokens are significantly cheaper than the response tokens.
j
/
k
navigate · click thread line to collapse