story
Lots of graph nodes, with weighted connections, performing distributed computation (mainly hierarchical pattern matching), learning from data by gradually updating weights, using selective attention (and/or recurrence, and/or convolutional filters).
Which of the above is not happening in our brains? Which of the above is not biologically inspired?
In fact this description equally applies to both a brain and GPT4.
The model can be the closest working model but that doesn't mean it is complete. It's very likely that cells can store memories/information independent from weights.
Do you see the difference?
We can't (fully) recreate the brain in software partly because we don't know enough, and partly because it's too computationally complex - for example, we can't simulate an entire modern CPU at the transistor level - even though we know how each transistor works, and what each transistor does in the CPU - because each transistor requires a detailed physical model with hundreds of parameters. It's simply not computationally feasible using current supercomputers. Brain is even less feasible to simulate if we want to accurately simulate each individual neuron in it - even if we knew exactly how it works.
But the second goal is much more feasible, and we have made great progress simply by scaling up simple known algorithms which approximate some information processing functions in the brain (mainly pattern matching/prediction and attention). I can talk to GPT4 today just like I talk to other humans, and by the way, this is only possible because out of all AI/ML algorithms people have tried over the last 70 years, the most brain-like one have won (ANNs). If we want to make further progress in AI or if we want to make GPT5 to be more human-like (not sure we do), we don't necessarily need to simulate brain at a neuronal level, we simply need to understand a little bit more about higher level brain functions. Today, we (ML researchers) might actually benefit more from studying psychology than neuroscience.
That's rather a bold claim given that artificial neural networks are universal function approximators.
It's perhaps not terribly surprising that it becomes possible with unlimited width or depth (or an arbitrarily complex activation function).
https://en.wikipedia.org/wiki/Universal_approximation_theore...
The universal function approximator theorem only applies for continuous functions. Non-continuous functions can only be approximated to the extent that they are of the same "class" as the activation function.
Additionally, the theorem only proves that for any given continuous function, there exists a particular NN with particular weight that can approximate that function to a given precision. Training is not necessarily possible, and the same NN isn't guaranteed to approximate any other function to some desired precision.
It seems pretty obvious to me that most interesting behaviors in the real world can't be modelled by a mathematical function at all (that is, for each input having a single output); if we further restrict to continuous functions, or step functions, or whatever restriction we get from our chosen activation function.
Yes, and?
> Training is not necessarily possible
That would be surprising, do you have any examples?
> and the same NN isn't guaranteed to approximate any other function to some desired precision.
Well duh. Me speaking English doesn't mean I can tell 你好[0] from 泥壕[1] when spoken.
> It seems pretty obvious to me that most interesting behaviours in the real world can't be modelled by a mathematical function at all (that is, for each input having a single output)
I think all of physics would disagree with you there, what with it being built up from functions where each input has a single output. Even Heisenberg uncertainty and quantised results from the Stern-Gerlach setup can be modelled that way in silico to high correspondence with reality, despite the result of testing the Bell inequality meaning there can't be a hidden variable.
[0] Nǐ hǎo, meaning "hello"
[1] Ní háo, which google says is "mud trench", but I wouldn't know
Neurons are not connected by a simple graph, there are plenty of neurons which affect all the neurons physically close to them. There are also many components in the body which demonstrably affect brain activity but are not neurons (hormone glands being among the most obvious).
> with weighted connections
Probably, though we don't fully understand how synapses work
> performing distributed computation (mainly hierarchical pattern matching)
This is a description of purpose, not form, so it's irrelevant.
> learning from data by gradually updating weights
We have exactly 0 idea how biological neural nets learn at the moment. What we do know for sure is that a single neuron when alone can adjust its behavior based on previous inputs, so the only thing that is really clear is that individual neurons learn as well, it's not just the synapses with their weights which modifies behavior. Even more, non-neuron cells also learn, as is obvious from the complex behaviors of many single-cell organisms, but also some non-neuron cells in multicellular organisms. So potentially, learning in a human is not completely limited to the brain's neural net, but it could include certain other parts of the body (again, glands come to mind).
> using selective attention (and/or recurrence, and/or convolutional filters).
This is completely unknown.
So no, overall, there is almost no similarity between (artificial) neural nets and brains, at least none profound enough that they wouldn't share with a GPU.
