Such demos already exist. I watched one last year.
The trick is a (by now) common one: you train the LLM on a simple programming language that's interpreted by the robot's software. So natural language goes in and the LLM then thinks out loud to figure out the needed steps, then emits Logo-like instructions to control where the robot goes. More conventional robotics AI then takes over to handle movement of the machine, the arms, etc.
