Minigpt4 Inference on CPU (opens in new tab)

(github.com)

102 pointsmaknee2y ago12 comments

12 comments

I know it's not the main point of this, but... so many multimodal models now that take frozen vision encoders and language decoders and weld them together with a projection layer! I wanna grab the EVA02-CLIP-E image encoder and the Llama-2 33B model and do the same, I bet that'd be fun :D

famouswaffles2y ago

Qformer isn't necessary just to be clear. Llava is just a projection layer

GaggiX2y ago

Not just a projection layer but also Q-former, in this case it was already trained for that specific vision encoder but if you change it you would need to train a Q-former from scratch.

famouswaffles2y ago

Not for mini gpt-4 but it's just a projection layer for many others(like Llava). The Qformer isn't a necessary part of the equation.

pizzafeelsright2y ago

I am not an ML expert. I want to know how to add my own documents without sending them off to a 3rd party.

extasia2y ago

Use a local LLM like LLaMa 2.

quickthrower22y ago

Not heard of minigpt4. Why that name? Is it claiming to be specifically a gpt4 competitor?

sanxiyn2y ago

MiniGPT-4 is a multimodal model. The name (a bad one, IMO) is a reference to GPT-4's multimodal capability.

Zondartul2y ago

They should rename it to ManyGPT.

WantonQuantum2y ago

There's more info here: https://github.com/Vision-CAIR/MiniGPT-4 (Linked in the Readme of the repo.)

Der_Einzige2y ago

Any data on inference speed? I’ve found that the non quantized model was much faster on GPU than the quantized versions due to lower GPU utilization.

api2y ago

It's a RAM tradeoff. If you have enough GPU RAM to load the non-quantized model it may be faster.

j / k navigate · click thread line to collapse

12 comments

heyitsguay2y ago

famouswaffles2y ago

Qformer isn't necessary just to be clear. Llava is just a projection layer

GaggiX2y ago

Not just a projection layer but also Q-former, in this case it was already trained for that specific vision encoder but if you change it you would need to train a Q-former from scratch.

famouswaffles2y ago

Not for mini gpt-4 but it's just a projection layer for many others(like Llava). The Qformer isn't a necessary part of the equation.

pizzafeelsright2y ago

I am not an ML expert. I want to know how to add my own documents without sending them off to a 3rd party.

extasia2y ago

Use a local LLM like LLaMa 2.

quickthrower22y ago

Not heard of minigpt4. Why that name? Is it claiming to be specifically a gpt4 competitor?

sanxiyn2y ago

MiniGPT-4 is a multimodal model. The name (a bad one, IMO) is a reference to GPT-4's multimodal capability.

Zondartul2y ago

They should rename it to ManyGPT.

WantonQuantum2y ago

There's more info here: https://github.com/Vision-CAIR/MiniGPT-4 (Linked in the Readme of the repo.)

Der_Einzige2y ago

Any data on inference speed? I’ve found that the non quantized model was much faster on GPU than the quantized versions due to lower GPU utilization.

api2y ago

It's a RAM tradeoff. If you have enough GPU RAM to load the non-quantized model it may be faster.

j / k navigate · click thread line to collapse