Claude vs. Gemini: Testing on 1M Tokens of Context (opens in new tab)

Funny you mention this, I literally just got done loading the context window of AI studio up for an hour doing some prototyping and then was frustrated when I couldn’t see where I was at from billing (knew it couldn’t be that much, but I still like to know).

I assumed because I’m on paid tiers it would still cost behind a certain usage amount, but I guess not.

Can you opt out of them training on your data in that free tier?

Geminis free tier allows maybe 5 messages on average, for 2.5 pro at least and this is not usable.

I’m using Claude Pro for daily driver and Gemini / ChatGPT free tiers.

Summarization ? I'm pretty sure there are benchmarks for this because people used summarization to build search indexes (at least a few years ago when I was working on this they did and there were benchmarks)

Note that (in the first test, the only one where output length is reported), Gemini Pro returned more than 3x the amount of text, at less than 2x the amount of time. From my experience with Gemini, that time was probably mainly spent on thinking, length of which is not reported here. So looking at pure TPS of output, Gemini is faster, but without clear info on the thinking time/length, it's impossible to judge.

if they left them both on defaults, flash is thinking-by-default and sonnet 4 is no-thinking-by-default

> Claude’s overall response was consistently around 500 words—Flash and Pro delivered 3,372 and 1,591 words by contrast.

It isnt clear from the article whether the time they quote is time-to-first-token or time to completion. If it is latter, then it makes sense why gemini* would take longer even with similar token throughput.

Anthropic also uses TPUs for inference.

output tokens must be generated in order (autoregressive decoding), inputs don’t have that constraint, so prefill is parallel, with stronger kernels, KV-cache handling, and batching, Claude can outrun Gemini.

Does anyone else have trouble with the archive rendering of that? It seemed to also have the pop up.

From my experience. Gemini is REALLY bad about context blending. It can't keep track of what I said and what it said in a conversation under 200K tokens. It blends concepts and statements up, then refers to some fabricated hybrid fact or comment.

Gemini has done this in ways that I haven't seen in the recent or current generation models from OpenAI or Anthropic.

It really surprised me that Gemini performs so well in multi-turn benchmarks, given that tendency.

The series is 1,084,170 words. At let's say 1.4 tokens per word, this would not fit, but it is getting close.

The entire HP series is about one million words.