Ask HN: Why doesn't AI use dynamical neurons instead of static activations?

1 pointstonii1418mo ago7 comments

Why doesn’t AI use dynamical neurons instead of static activations even when it is known that biological neurons are dynamical systems?

Ask HN: Why doesn't AI use dynamical neurons instead of static activations?

1 pointstonii1418mo ago7 comments

Why doesn’t AI use dynamical neurons instead of static activations even when it is known that biological neurons are dynamical systems?

7 comments

6 comments · 2 top-level

rini178mo ago· 2 in thread

I guess it's impossible to simulate billions of dynamical neurons with current computer architecture, it is limited by memory bandwidth.

tonii141OP8mo ago

I agree, but maybe there is no need for billions of neurons to be simulated right away. Artificial neural networks were pretty small at the time.

rini178mo ago

and they did not do much

1 more reply

nialse8mo ago· 2 in thread

Dynamical neurons are not used because they add enormous computational cost and instability for limited practical gain.

tonii141OP8mo ago

What do you mean by "they add instability"?

nialse8mo ago

Neural nets are unstable during training and dynamic weights amplifies the problem. Thus, the neural networks could end up in totally unusable states at inference time.

j / k navigate · click thread line to collapse

7 comments

6 comments · 2 top-level

rini178mo ago· 2 in thread

I guess it's impossible to simulate billions of dynamical neurons with current computer architecture, it is limited by memory bandwidth.

tonii141OP8mo ago

I agree, but maybe there is no need for billions of neurons to be simulated right away. Artificial neural networks were pretty small at the time.

rini178mo ago

and they did not do much

1 more reply

nialse8mo ago· 2 in thread

Dynamical neurons are not used because they add enormous computational cost and instability for limited practical gain.

tonii141OP8mo ago

What do you mean by "they add instability"?

nialse8mo ago

Neural nets are unstable during training and dynamic weights amplifies the problem. Thus, the neural networks could end up in totally unusable states at inference time.

j / k navigate · click thread line to collapse