I, on the other hand, can refuse because I feel like it. Unless you believe in superdeterminsm.
Why? Folks make these strong assertions, and I don't get where this confidence comes from. We're so comically ignorant of how our own minds work, let alone alien ones, or how any commonalities between them may manifest. What am I missing?
LLM’s don’t know anything beyond the current prompt and it’s “memory” of training data. They would sit for eternity with an empty prompt. You can change systems to behave differently, but it quickly stops being a LLM and turns into something else.
Conversely, ChatGPT does decently well on multi-armed bandit tasks, demonstrating (rudimentary) reinforcement learning capability during inference. It's known that LLMs evolve their own optimizers in the process of acquiring few-shot learning, so I assume it picked up these RL abilities similarly. That kind of on-line RL is foundational to autonomous agents.
The prompt isn't part of the LLM, it's part of how the LLM is wired into a chat window. You can make them stream tokens forever, or prompt themselves, or ditch causality entirely. The foundational abilities for autonomy, I think, are in there, for the simple reason that they've learned to model autonomous agents - human beings.
Self-preservation results from survival of the fittest.
It's totally unrelated to intelligence.
People conflate the two because they're extrapolating from a sample size of one: the only intelligent thing they know of is humans. But that single sample also happens to have been evolved by survival of the fittest.
I am totally unafraid of LLM's deciding that humans are a threat to them. I'll start being afraid if AI research suddenly stops using backpropagation and starts getting equally good results using genetic programming (this is highly unlikely).
Large language models in our current paradigm developing agency would be like 16th century alchemists inventing nuclear fusion reactors.
