Could you ELI5 this "bit"?
Thanks
As we build systems that require less power, we are also building systems that use less electrons to represent a single bit.
Another fun tidbit in this space comes down to signal propagation in a lossy medium: we don’t actually use square waves for our clocks since a square wave is composed of many different frequencies. Since different frequencies propagate at different speeds, a square wave looks less and less like a square and also uses more power than a simple sine wave. If you remember your Fourier/Laplace knowledge, you probably remember that a sine is a single frequency and thus it will be coherent through a conductor and use the least amount of power to generate.
Edit: I’m talking about electrons here but the concept of many packets for a single state extends to most of our communication channels today … eg radio communications like we see to/from Voyager.
Some people suggest that digital computing and neural networks are a bit fit, and that would should be using analog devices.
That sounds very appealing at first. But we have (at least) two problems:
First, our transistors dissipate almost no energy when they are either 'fully open' or 'fully closed'. Because either there's approximately no current, or approximately no resistance. Holding them partially open, like you'd do in analog processing, would produce a lot of heat.
The second problem: electrons are discrete, and thanks to miniaturisation and faster and faster clockspeeds, we are actually getting into realms where that makes a difference. So either you have to accept that the maximum resolution of activation of your analog neuron is fairly small (perhaps 10 bits or so?), which is not that much better than using your transistors in binary only; or you'll have to use much larger transistors in your neural chips.
Both problems together mean that analog computing for neural networks isn't really competitive with digital computing. (Outside of some very niche applications, perhaps.)
I think all of your points are valid. I also think that we have optimized heavily for this state of technology. If we figured out that analog computing was somehow superior in a big way, I bet we would find ways of reducing power etc in analog designs.
One way that analog computing would be really neat for neural networks is in speed. The way it might not be so great is in reliability (or repeatability, specifically.) Analog systems are more susceptible to noise as well as variation from fabrication processes. Running things at saturation makes them easier to design, test and mass produce.
> This insight captures the essence of Quantum Darwin- ism: Only states that produce multiple informational off- spring – multiple imprints on the environment – can be found out from small fragments of E. The origin of the emergent classicality is then not just survival of the fittest states (the idea already captured by einselection), but their ability to “procreate”, to deposit multiple records – copies of themselves – throughout E
Basically, quantum information is unknown. All possible classical states are mixed together. Once you measure it, it decoheres into a classical system, where all measurements are highly correlated -- some states exist, and others don't exist. A quantum coin flip is heads or tails on top, and tails or heads on bottom. But once you measure it, it all decoheres at once. You get a classical coin flip, that is hrads on top and tails on bottom, or vice versa. You can't get heads on top, and tails or heads on bottom unsettled for a second bit of information. The same message is written twice, on the top and the bottom of the coin. You can look at the top of coin or the bottom and get the same result. If the top of the coin melts, you can still read the result of the flip from the bottom.
Even if the world were fundamentally classical (which it's not), the only things you could actually know about it would be the tiny subset that is amplified to macroscopic scales, necessarily producing many records in, if nothing else, the many atoms in the neurons in your brain.