undefined | Better HN

0 points123yawaworht4562y ago0 comments

what purpose does a LLM serve on a killing drone, exactly?

0 comments

Open source models in general. Meta has for instance released DINO which is a self supervised transformer model. LLMs are also going multi modal (see LLaVA for instance). The name "LLM" has stuck but they should really be called Large Transformer Models. LeCun is working on self supervised visual world models (I-JEPA) which if successful and released could form the basis for killer drones. It's still a lot of engineering work to fine tune and put a model like this on embedded hardware on a drone, but at some point it might be easy enough for small groups of determined people to pull it off.

Vetch2y ago

For a drone, an LLM derived solution is far too slow, unreliable, heavy and not fit for purpose. Developments in areas like optical flow, better small CNNs for vision, adaptive control and sensor fusion are what's needed. When neural networks are used, they are small, fast, specialized and cheap to train.

A multimodal or segmentation algorithm is not the solution for bee-level path planning, obstacle avoidance or autonomous navigation. Getting LLMs to power a robot for household tasks with low latency to action and in an energy efficient manner is challenging enough, before talking about high-speed, highly maneuverable drones.

martindbp2y ago

Tesla is running these models on 4 year old hardware to control a car in real time (30 fps). You don't need a full 100B model to control a drone, and it doesn't have to be as good as a car to cause a lot of damage. Reportedly both Ukraine and Russia are putting together on the order of a thousand drones a day at this point, Tesla includes the compute to run this in every car they make already today. Hardware is also moving fast, how come people forget about Moore's law and software improvements? To me there's no question that this tech will be in tens of thousands of drones within a few years.

fatherzine2y ago

a multimodal llm is a general purpose device to churn sensor inputs into a sequence of close to optimal decisions. the 'language' part is there to reduce the friction of the interface with humans, it's not an inherent limitation of the llm. not too farfetched to imagine a scenario where you point to a guy in a crowd and tell a drone to go get him, and the drone figures out a close to optimal sequence of decisions to make it so.

IshKebab2y ago

I think GPT-4V could probably make high level decisions about what actions to take.

Not really practical at the moment of course since you can't put 8 A100s on a drone.

fatherzine2y ago

there are rumors that the latest gen drones in ukraine use crude embedded vision ai to increase terminal accuracy. launch and iterate, this will only get more lethal.

j / k navigate · click thread line to collapse

0 comments

martindbp2y ago

Vetch2y ago

martindbp2y ago

fatherzine2y ago

IshKebab2y ago

I think GPT-4V could probably make high level decisions about what actions to take.

Not really practical at the moment of course since you can't put 8 A100s on a drone.

fatherzine2y ago

there are rumors that the latest gen drones in ukraine use crude embedded vision ai to increase terminal accuracy. launch and iterate, this will only get more lethal.

j / k navigate · click thread line to collapse