GPT-4o mini: advancing cost-efficient intelligence (opens in new tab)

My excitement is now tempered a bit. I just tried one of the too-big invoices with the new model. After successfully getting a little farther than 4o could do, it just went into an endless loop of repeating the same line item until it ran out of output tokens. So…not really an improvement!

If all that AI could do was to turn less than structured data into structured data, it would still be the biggest deal in computation since the transistor.

>I had to do a heuristic layer on top to break up the PDFs into small chunks so the output didn’t overflow

How do you stitch the outputs of all chunks without losing the overall context?

Have you written about this anywhere? Would love to know more about the process you're using!

I’m a little skeptical of processes that seem to create more information than you had to start with. For a game like chess or Go, it makes sense, because winning strategies are implicit in the rules of the game, but it takes a lot of computation to discover the consequences. Similarly for math where theorems are non-obvious consequences of axioms. And computer code can be similar to math.

But how does that work for an LLM in general? They’re trained on everybody’s opinions all at once, both right and wrong answers. They’re trained to generate text supporting all sides of every argument. What does more training on derived text actually do?

Sounds like you're describing mixture of experts, the architecture being used in openai's gpt-4 and mistral's mixtral series of models.

I have posited a similar idea with some of the people I work with. The issue of having complex, multi-step tasks be completed successfully has already been solved. You don't heavily invest in having one single expert for your business to solve all your problems. You build a team. Multiple specialized experts working in unison to achieve a shared outcome. Some people work on the task simultaneously, others sequentially. All with a specific purpose associated with the goal.

These assets are horizontally and vertically scalable based off skills, quality, or performance required. An efficiently designed AI architecture I believe could do the same. Its not mixture-of-experts as you aren't necessarily asking each model simultaneously but designing and/or having the system intelligently decide when it has completed its task and where the output should travel next.

Think of a platform where you had 'visual design' models, 'coding' models, 'requirements' models, 'testing' models, all wired together. The coding models you incorporate are trained specifically for the languages you use, testing the same. All interchangeable / modularized as your business evolves.

You feed in your required outcome at the front of your 'team' and it funnels through each 'member' before being spit out the other end.

I have yet to see anyone openly discussing this architecture pattern so if anyone could point me in that direction I would thoroughly appreciate it.

At scale you should realise that this is still A LOT of money and the models are considerably reduced in cost so the margin probably works out even better. OpenAI are successful, it's a fact, which means they know what they're doing business wise. (Not bootlicking, just trying to be logical).

Think about it this way: Imagine if every email you sent or every online forum post you commented on provided incentive for the provider.

Take a loss on every sale and make up for it with volume!

They're building a beautiful garden with rich soil and generous watering. In fact it is so wonderful that you'd love to grow your product there. A product with deep roots and symbiotic neighbors.

Just be careful when they start building the walls. And they will build those walls.

I think it's heavily quantized, so it doesn't cost them (too much). But I think it's still at cost...

Yeah, to put these prices in perspective: when tokens get this cheap, $1M buys you more than a trillion output tokens.

To earn appreciable revenue at this price, an LLM company needs to be regularly generating multiple internets worth of text.

On the one hand, generating multiple internets of text seems outlandish.

But on the other hand, we're now approaching the point where you can start building LLMs into software without fretting about cost. Now that you can buy ~30 pages for a penny (instead of a dollar) you can really start to throw it into websites, games, search bars, natural language interfaces etc. without every user costing you much.

But small models are not the endgame for these AI companies, as truly general intelligence is a market worth trillions.

What this ~98% cost drop over 2 years hints at is that when AGI does arrive, it might not be horribly expensive.

I think the place for generating larger total revenue/margins would be in the highest end models. Budget models almost "come with" the effort put towards making those high end models so it's alright they are a race to the bottom (so long as someone actually realizes return on higher end models, which is a problem in itself at this moment).

> There's no way this price-race-to-the-bottom is sustainable.

Why not?

what if they can make money? then the problem is on claude/gemini...

These models are still really expensive to run

It's really clear that hacker news puts its thumb on the scale of pretty much everything in a pointedly opaque way. It's really easy to see this in action if you go down to the bottom of comments section and you'll notice a bunch of examples of comments that have negative total votes and are older sitting above comments that have positive votes and are newer. Makes me wonder, is hacker news applying global weights to users? If I post on a page, is there some metric I don't get to see that just says "this person starts with an effective -2 votes"?

I have completely lost patience with it. I no longer use the hacker news front page. Try using the hacker news search instead: https://hn.algolia.com/?query=*&dateRange=last24h

This is just the top in the last 24 hours, or you can switch it to last week to catch up. Plus the search is pretty nice and very fast so if you're looking for something specific it's convenient. This sort's explicitly in order of votes and nothing else. It's a lot better.

I'd tolerate all this rank fiddling better if it was transparent as to why things were being sorted the way they are. But that's not going to happen. Make the best of it you can.

It almost sounds shady... "it's 30x cheaper per token but you now need 30x more tokens per image"?

Has anyone already validated this based on billed cost? running a batch myself to check

EDIT:

Ok so I captioned 500 images in "low resolution" mode with GPT-4o-mini

Each one took approximately: "completion_tokens=84, prompt_tokens=2989, total_tokens=3073"

Reported GPT-4o-mini cost is $0.25

Using GPT-4o this would cost me $1.33 (also in "low resolution" mode), with this breakdown:

"completion_tokens=98, prompt_tokens=239, total_tokens=337"

Good catch: the calculators here are bizarre. For GPT-4o, a 512x512 image uses 170 tile tokens. For GPT-4o mini, a 512x512 image uses 5,667 tile tokens. How does that even work in the context of a ViT? The patches and its image encoder should be the same size/output.

Since the base token counts increase proportionally (which makes even less sense) I have a hunch there's a JavaScript bug instead.

I might be wrong, but I've inferred from OpenAI's pricing behavior that they use it to encourage people to migrate to more efficient models. The 3.5 Turbo pricing is maintained to encourage you to stop using it. Look at davinci-002's pricing, for example - it's very high for something that's relatively ancient.

Predictability with a particular set of prompts and processes. Over time, you'd migrate to the lower cost, higher performing model, as long as it can be at least as consistent as the higher cost model. People have built really weirdly intricate chains of dependency on things that particular models are good at, and sometimes 3.5 turbo can accomplish a task dependably where other models might refuse, or have too wide a variance to be relied on.

Over time, reliability and predictability will be much less an issue.

4o mini is more efficient so it costs them less than 3.5 turbo to host it.

1. It's not a worse model, it's a better model. Two years ago all we had was text-davinci-003, which is much, much worse than, for example, the current Claude 3.5 Sonnet which costs like 5x less.

regarding 1, they have a strong understanding of the tasks/queries their users are performing and they are pruning the model accordingly. It's like playing jenga but with neurons.

Interesting. They switched to a new tokenizer for 4o and 4o-mini, so this might have the same issue.

I noticed the same problem but on 4, it was super-weird, everything was fine except one character, and it occurred consistently in the second and the next answers, never in the first one.

i saw this with github copilot a few days ago, not sure which model it was. it messed up a single character of markup causing the resulting output to be formatted weirdly

How long before Anthropic releases Claude-3.5-Haiku at the same price with significantly better performance? OpenAI in trouble...

I think this is a strong conclusion to jump to. Maybe it's better at spotting content that needs to be moderated because it stands out more from what it's been trained on?

The original GPT-4 was an upgrade IMO. GPT-4 Turbo and GPT-4o were downgrades. GPT-4o seems especially bad (on text-to-text).