My take is that the neural network is a bit of a red herring -- people poked around in brains to see what was going on and noticed a network structure with many apparently simple computing nodes. So they tried making similar structures in software and quickly discovered they could do some interesting things. But it may turn out that the neural network was just nature's best implementation for "field programmable matrix manipulation". You can implement the functionality in other ways, not resembling neural networks.