Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system (opens in new tab)

Loading a model takes at least a few seconds, usually more, depending on model size, disk / network speed and a bunch of other factors.

If you're using an efficient inference engine like VLLM, you're adding compilation into the mix, and not all of that is fully cached yet.

If that kind of latency isn't acceptable to you, you have to keep the models loaded.

This (along with batching) is why large local models are a dumb and wasteful idea if you're not serving them at enterprise scale.

Models take a lot of VRAM which is tightly coupled to the GPU so yeah, it's basically sitting there with the model waiting for use. I'm sure they probably do idle out but a few minutes of idle time is a lot of waste--possibly the full 82% mentioned. In this case they optimized by letting the GPUs load multiple models and sharing the load out by token.

> I guess I’d assumed this sort of thing would be allocated dynamically

At the scale of a hyperscaler I think Alibaba is the one that would be doing that. AWS, Azure and I assume Alibaba do lease/rent data centers, but someone has to own the servers / GPU racks. I know there are specialized companies like nscale (and more further down the chain) in the mix, but I always assumed they only lease out fixed capacity.

The paper is about techniques to do that dynamic allocation to maximize utilization without incurring unacceptable latencies. If you let a GPU sit idle for several minutes after serving a single request, you're setting money on fire. So they reuse it for a different model as soon as possible, starting even before the first request is finished, because: If you don't have a dedicated GPU for a model, are you going to wait for a multi-gigabyte transfer before each request? So they have a dedicated GPU (or two, one for prefill, one for decode) for a group of models that are processed in an interleaved fashion, scheduled such that they stay within the latency budget.

>Do the GPUs just sit there with the models on them when the models are not in use

I've assumed that as well. It makes sense to me since loading up a model locally takes a while. I wonder if there is some sort of better way I'm not in the know about. That or too GPU poor to know about.

the models are huge, so not a single (latest gen) one can fit on a single GPU.

It's likely that these are small unpopular (non flagship) models, or that they only pack eg one layer of each model.

I don't think that's what this is saying, isn't it that 100 - ~82 = 17.7% ?

Really:

"A paper presented at SOSP 2025 details how token-level scheduling helped one GPU serve multiple LLMs, reducing demand from 1,192 to 213 H20s."

Which, if you scale it, matches the GPs statement.

If you're one who sees progress as an end goal unto itself, what you describe is a good thing. When one party is attempting novel solutions to outcompete the competition we will be faster to whatever the next change is.

That said, I'm not sure what the US policies specifically have to do with this. Countries are always in competition with one another, and if one industry or technology is considered a national security threat they will guard it.

Feel like I've made it

> most of what western companies have created has had significant contribution by Chinese scientists or manufacturing, without which those companies would have nothing. If you look at the names of AI researchers there's a strong pattern even if some are currently plying their trade in the west.

While I don't disagree with your overall point, it's important to recognize that this is only a phenomenon of the last ~30 years, and to avoid falling into the trapn of Han racial chauvinism. E.g. there were ~no Chinese scientists in Germany in the 70s but they were heavily innovating nevertheless.

Ironically, the best way America could have prevented China’s rise in tech was by stapling green cards to diplomas of Chinese citizens who completed their higher education in the U.S. like the plan in the early 2010s.

The whole “China copies everything” narrative is becoming less and less true.

It’s funny - it’s at the point with Chinese manufacturing for niche electronic goods (e.g rooftop van air conditioner) where some Chinese brands are more trustworthy - more value for your money and sometimes even better overall quality. With American brands you gotta make sure you’re not overpaying for dated tech that is inefficient. Maybe the same will happen with LLMs.

> History has shown that withholding technology from China does not significantly stop them and they'll achieve it (or better) in a small number of years.

It's worked for a very long time for aircraft.

China has been pushing to build its own aircraft for >23 years. It took 14 years for COMAC to get its first regional jet flying commercial flights on a Chinese airline, and 21 years to get a narrow-body plane flying a commercial flight on a Chinese airline.

If for no technical reasons and purely political, COMAC may still be decades away from being able to fly to most of the world.

Likewise, in ~5 years, China may be able to build Chips that are as good as Nvidia after Nvidia's 90% profit margin - i.e. they are 1/10th as good for the price - but since they can buy them for cost - they're they same price for performance and good enough.

If for purely political reasons, China may never be able to export these chips to most of the world - which limits their scale - which makes it harder to make them cost effective compared to Western chips.

Re: Western. A similar thing plays out when the term "international community" is used in news. It refers to the US and its major allies which means US, Canada, Western Europe, Japan, Australia and New Zealand more or less.

It's just straight up low expectations and underestimation derived from racism in the assumption that Americans are smarter and more capable, and Chinese are only good for copying designs and making things we come up with. The idea that they can't do that like we can is pervasive.

I am not sure exactly to what degree, but "I hate the term 'western' because some 'weste[r]ners' use it to separated what they think are 'civilized' from 'uncivilized'" is definitely a bit of an antiquated perspective at this point; almost like a justification to hold on to other older perspectives about "racism". I have started resorting to using terms like European Cultural Block because of it in certain communities that understand contemporary topics and have an advanced understanding of the world.

Your first statement is not likely unique to China though, even though they have demonstrated that in about the last 40 years, which I don't really think qualifies as "history". What it does demonstrate is that societies that have a certain kind of ethnic self-respect and can cast off the detrimental influences of foreign, hostile, and even enemy elements to pursue their own self-interest and survival will succeed, regardless of hurdles placed before them.

It's really just a story of personal development and either escaping, evading, and avoiding detrimental, toxic people and their behaviors. All of humanity that all has to currently still share a single planet with ZERO save spots, would be better off if we all not just allowed each other to be ourselves in our won places without others subverting, subjugating, infiltrating, dominating, poisoning, or polluting any other people on the planet. Then everyone can decide if we want to be friends or not friends with each other, collaborate and be friendly or simply avoid each other. We do not have to like each other to get along if everyone agrees on a base understanding that no people can parasitize and abuse and manipulate any others.

Nobody thinks the Japanese aren’t “civilized.” “Western” is just a euphemism for “rich and orderly.”

>History has shown that withholding technology from China does not significantly stop them and they'll achieve it (or better) in a small number of years.

I don't think you can really produce a definite counterfactual that they would or wouldn't have taken longer or shorter without it, but certainly they were pushing for self sufficiency long before technology restrictions. But we're not going to be handing our technologies to our competitors on a silver platter, and it's also best for businesses to start weaning themselves off the Chiinese market. Virtually every market reliant on them today is in big trouble.

As for hubris, I think that's more a projection of your part if you want to start bringing up race cards with regards to contributions, that kind of argument would be applicable to everyone. And AI research is highly diverse and international, Chinese names don't dominate the list more than Turks, Greeks, Malaysians, etc.

> look at the names

Why would I do that tho? If we look at the names of scientists/researchers/engineers/businessmen, the conclusion would be that the US has contributed nothing to the world. Europeans did all the hard work!

this is true for anyone - create challenges, and you optimize efficiency elsewhere.

Also, isn't this the usual path to better computer science? Reducing computation needs by making better/more efficient algorithms? The whole "trillions of dollars of brute force GPU strength" proposed by Altman, Nadella, Musk et al just seems to reinforce that these are business people at heart, not engineers/computer scientists...

What term would you prefer? When you say "western view of China", do you mean to include or not include Latin American countries as part of the group of countries you claim has a hubristic view of China's accomplishments?

It’s helpful to think of westernism as a platonic ideal. Individually derived reason and virtue, superior to state and sometimes ‘gods’ as a tradition to drive up the total survivability, richness, and stability of the community.

Concepts that enable the individual should empower a chosen configuration of society not the other way around.

Contrast this with non westernism where either education of the individual is not valued or the state is the primary goal over the individual.

I’ve worked with states governments and individuals around the world for 20 years and find this very useful definition. What’s confusing is the nations who have half adopted westernism but don’t fully due to either caste systems or government dominated thinking.

It’s an arrow towards rationalism over tradition, individualism over collectivism, flatness over hierarchy, and future over past. But only the limit of the resources any given society has.

In a way withholding a tech becomes a signal saying "Hey this is important" so the result is China dedicates more resources to researching it lol.

western is a cultural term derived from a geographic one. The US is also not 'western' strictly geographically as it is not in western europe, neither is australia. But they both originated from Britain's empire and share in it's cultural ancestry. It means "western europe and it's cultural derivatives". Spain and Portugal's empire fell away long before britain and france's and they don't have similar geopolitical relations like NATO, so it's hard to consider their former colonies/upstarts part of the same sphere of cultural influence.

China for sure will catch up, the question is what they will do with it. They're not ambitious like the US/West. The US wanted influence all over the world as an extension of the cold war and to keep economic interests safeguarded. But China just doesn't operate that way. They're more hands-off. They could be opening up alibaba cloud datacenters all over the US, offering it as an AWS/Azure alternative, funding tons of startups all over europe, the US,etc... to exert their influence, but they won't. They have a more long-term low-and-slow approach to global domination. The "100 year marathon" as they called it, which they'll win for sure.

China's greatest weakness is not just their lack of ambition,but their command-economy. They're doing capitalism but with central control of the economy. It intertwines government policy with corporate policy, making it harder to do business overseas (like with bytedance/tiktok).

The whole "western" or "the west" always makes me laugh. Half the time it's a dog whistle for "white". Like many right-wing commentators love saying "Western Values" to avoid saying "white, Euro-centric, Christian values".

Mexico is a modern country, an industrialized country, a country that is exactly as "western" as the US or Canada. They have the same religious beliefs, speak a dialect of a European language. They have European style cities, a long history of cultural contributions. Yet they're not white enough to be part of "The West".

I think at this point we should be honest with ourselves in it's usage. 90% of the time it's a racist dog whistle.

> History has shown that withholding technology from China does not significantly stop them and they'll achieve it (or better) in a small number of years.

Really? How long has China been attempting to build their own jet engines? How long have they been attempting to build competitive CPUs?

History has shown withholding tech successfully keeps them at least a generation behind the west.

In some fields like CPUs they “make up for it” by just building larger clusters, but ultimately history does not show what you’re claiming. The only thing it shows is that we need to be even more diligent in protecting IP because a large portion of their catching up is a direct result of stealing the tech they were cut off from.

Name one thing China has invented first in LLMs that the “west” adopted as a standard

I find "western" is often used to disparage "western thought", as in it can't grasp the deep wisdom of those mysterious orientals that transcends normal logic and reason. Declaring such a split is the underpinning of a whole lot of woo-woo beliefs.

> But, China's greatest weakness is their lack of ambition and focus on regional matters like Taiwan and south china sea, instead of winning over western europe and india.

That's a strength. Them not having interest in global domination and regime change other than their backyard is what allows them to easily make partners in Africa and LATAM, the most important regions for raw materials.

I went to a school that was heavy on immigrants and had lots of 1st gen citizens as students and all they did was advocate against people like me for admissions and for preferential admissions for their own group. So in my opinion, skilled immigration is not a transfer of talent but an expansion of the upper classes who go to war with each other over a small number of seats. Ironically this zero sum game keeps overall skill levels the same. For every immigrant, say, one citizen loses a seat somewhere.

China's greatest weakness is that their working-age population has already peaked and is in the process of plummeting, which will continue over the coming decades.

>But now the US is trying to be xenophobic like China, restrict tech import/export like China but compete against 10x population and lack of similar levels of internal strife and fissures.

Do I infer correctly that you believe that China has less internal strife and fissures than the US has?

> But, China's greatest weakness is their lack of ambition and focus on regional matters like Taiwan and south china sea, instead of winning over western europe and india.

How can they have international hegemony before they clear their regional order? China is more interested in aligning Taiwan than invading; though it’ll probably invade if it can’t align it diplomatically.

China is probably not interested in continuing the current Western-style order but to implement their own sino-stuff. At least with the CCP at the helm.

> But now the US is trying to … compete against 10x population and lack of similar levels of internal strife and fissures.

I can’t tell whether you think the anti-immigration stance is a good thing or bad thing.

Nobody is anti-immigrant outside of a small pocket of anti-H1B folks in the tech community. People are, however, anti-illegal-immigrant, which is completely different.

The greatest weakness of the US is its utter lack of self awareness and its ambition to dominate others. Nobody is looking for another "leader", people just want to live well without a bully on their neck. So, many countries that are not part of the US closed club are welcoming China as a new business partner.

> China has an import ban on chips

Only in response to the US banning the export of the high-end GPUs China wanted. The import ban is the Chinese government burning the the landing ships, it clearly communicates to everyone that there is no going back, and total commitment is expected.

The US is certainly slowing down China considerably. China would certainly not have an import ban on Blackwell GPUs if they were made available. And upstream, the ban on EUV and other high end semiconductor production equipment has severely limited china’s capacity to produce logic and DRAM (including HBM).

Would they have done that if the US had been more "reliable" in providing the chips and didn't cut them off in the first place?

The point still stands that the US instigated the split.

I've been in Chile, Peru, Colombia, Panama, and Costa Rica.

The streets are flooded with cheap Chinese cars and I see more BYD than American cars. If the car wasn't made in Japan or Korea which probably account for most of the cars, it was likely made in China. Moreover, I haven't been in countries with the closest ties to China.

    > US cars being completely not competitive in non-US markets

This is definitely untrue in Canada and large parts of LATAM. American cars are all over those places, along with Japanese cars.

The premature optimizer is never the innovator.

Japan eventually stopped that role and their products improved greatly.

But they aren't keeping up

They are lauded for the ability to cost ratio, or their ability to parameter ratio, but virtually everyone using LLMs for productive work are using ChatGPT/Gemini/Claude.

They are kind of like Huffy bicycles. Good value, work well, but if you go to any serious event, no one will be riding one.

Have you tried using those models? qwen for example cant even do something as basic as clustering analysis on a list of integers, hell it goes off the rails when just reading said integers from a file - starts babbling about determining number of digits, indexes, tries concatenating numbers together into one big string, no idea wtf is going on with that model.

way too early to say that

while qwen, deepseek and kimi are opensourced and good, they are preferred because of their insane token ratio, they use a lot less for more, but a by product is that they are less accurate it is amazing progress by the chinese companies, but they definitely can improve a lot more

too early to call a winner, though it is disappointing to see US withdrawal from open source. Still the main outcome of open source is distribution / diffusion of the idea, so it will inevitably mean US open source will come back, hopefully via some grass roots maniac, there will be a Linus-like character emerge at some point

positive take: a bifurcated tech tree might give us (humanity) a better chance of faster advancement, as it would be a persistent A/B test in live environment. Where I would join you in the crossing of fingers is to ensure such A/B testing is competitive but not destructive. We may even evolve to a situation of complementarity, an American Ying vs the Chinese Yang. Lets hope so!

Nowadays, China produces about the same amount of high-quality R&D as the US does.

Your view of China is several decades out of date. Chinese labor isn't even that cheap any more. China is moving up the value chain and outsourcing production that needs cheap labor to poorer countries (or replacing workers with robots altogether).

Every single sentence you wrote is untrue and can be disproven by empirical evidence. You can learn about it here[1][2].

1. https://itif.org/publications/2024/09/16/china-is-rapidly-be...

2. https://www.economist.com/science-and-technology/2024/06/12/...

And the US has never stolen IP?

Qwen's max series had always been closed weight, it's not a policy change like you are alluding.

What exactly is Huawei's flagship series anyway? Because their PanGu line is open-weight, but Huawei is as of yet not in the LLM making business, their models are only meant to signal that it's possible to do training and inference on their hardware, that's all. No one actually uses those models.

Small counterpoint but there are also 2 new players putting out SOTA open source models (Moonshots Kimi and zhipus GLM) so we're still seeing the same number of models overall, just via newer entrants.

"... instead of spending the time and money for research and engineering..."

China has plenty of R&D and science now.

I was doing this in the 70-80s with electronics from Hong Kong and Japan. The nice cheap stuff ( I was very young ) was all sheets in things I basically had to pattern match against notes of others on BBS and meetups.