> What it could do is experiment. It could rewire the network, add or take away nodes, change functions, etc. This could potentially be done and evaluated by it at an incredibly fast rate compared to humans.
"Potentially," but let's not get ahead of ourselves here. There are two ways a brain could be "rewired." It could first be rewired by a local process, that is to say, a group of neurons decide how they are to be connected to each other, without influence from faraway neurons except through the normal propagation of signals through the brain. That's how biological brains do it. Or it could be rewired by a global process, a coordinator that can look at the big picture and make minute changes. That would be your suggestion.
My contention is that the latter method involves a lot of hardware overhead: you basically need a global network connected to each and every neuron which can probe their state in addition to the local wiring that lets neurons communicate with their neighbours. You need space for this bus, so neurons need to be further apart than they otherwise would, which means that signals have to travel further and the brain will think slower.
Nor is "rewire the network, add or take away nodes, change functions, etc." necessarily an effective strategy. First you need to identify what to change, which is like finding a needle inside a haystack, then you need to figure out in what way to change it, which is also difficult, then you need to test whether it had the intended effect, and more importantly, whether there were harmful side-effects. Whether humans can do it or not is not relevant: What is relevant is whether this process is efficient enough to beat the baseline local learning method. It is not clear that it would be.
> Regarding the difficulties of it reading its own source code, even the hypothetical ones you cite are many orders of magnitude smaller than those faced by humans in reading their own, as long as the AI is not itself running on a biological substrate.
What makes you think these are smaller difficulties? On the contrary, if you imagine that the AI is built as a very dense, solid 3D circuit, and you need to read the value of a neuron at the center, you might have a much harder time jamming a probe in there than you would injecting one in a squishy human brain. You would need to build it in such a fashion that it can be probed easily, but that may require making the circuit twice as big and therefore slower. Furthermore, in the presence of local update rules, which is likely to be the case, your "source code" is changing all the time, even as you read it, so your self-knowledge is constantly out of date. There is a synchronization issue here.
