Neural vs Digital computing

There have been many claims that the artificial neural network emulates the processes within our human body, and that explains the vast swathes of success it enjoys. There's even hype that one day they would become more intelligent than us, an idea explored aplenty in science fiction. However, the truth is that ANNs were only inspired by the human neural circuit, and are not representative of the inner workings of our nervous system.

To elaborate on this, let’s enact a fair comparison between ANNs of digital computing, and the nervous system of our neural computing. Our first point should address the structures of the networks. While we may think deep Resnets are massive, they are tiny compared to the neural network within us. This should come as no surprise. The extent of neural connections within us is mind-boggling. For example, within our cerebral cortex alone (the convoluted outer layer that we regularly see in pictures of the brain), there are 30 billion neurons, each having 10,000 connections (or synapses). In comparion, VGG19 considered to be one of the larger networks, contains only about 138 million neurons, while complexity-wise the deepest neural networks we have been able to conjure up is a variant of Resnet with only 1000+ layers.

The next point to consider, is the layout of the network. We have observed a visual heirarchy of semantic complexity in our nervous system. For example, there are neurons in our eyes which have been verified to apply a Difference of Gaussians filter and detect edges. Going deeper in the visual cortex, we find that the orientations of these edges causes particular neurons to activate. And as we proceed even deeper, we slowly find semantic constructs such as faces or house being responsible for some neural activations. Our modern day convolutional neural networks have in fact taken inspiration directly from this and have greatly benefited.

While it is definitely tempting to think of this as a feedforward model, the reality disavows this idea. The deeper neurons send their information back to the neurons at the initial stages of the heirarchy, influencing them. There is a great degree of interconnectivity, leading to what is known as semantic expectation - “What you think you see affects what you actually see”. (Deep, huh?) I’m sure all of us have been victims of semantic expectation: say you looked at your watch with some expectation of the time, and you had to do a double take to acknowledge that it’s actually 3 hours past that. The takeaway from this is that our bio-neural circuits are highly complex and interconnected, and can carry information across multiple stages. The layout of modern day artificial networks on the other hand are woefully simple in comparison.

This leads us to the next point - the paradigm of computation. Our complex layout enables our neural network to be massively parellel in its computation. ANNs on the other hand require a mostly sequential information processing.

It’s worth adding that the updation rules also differ between the real and the artificial. In the former, we have local adaptive connectivity. Every time a neuron gets activated and sends its activation to another neuron, their connection is strengthened, and vice versa. In our artificial counterpart, the updation is global, and all parameters are simultaneously updated. This also implicates that connections are fixed and new ones cannot be formed. Local adaptive connectivity along with the highly parallel paradigm of computation enables our brain and its neural circuit to be very efficient in what it does. The artifical neural network, not that much.

So is there at least one win for our inventions? Yes! It’s the resolution time. For all of the efficiency of the bio neural circuits, the resolution time is still limited to 100 microseconds. In comparison, a typical clock speed today is 100 picoseconds, a huge difference.

All these points should convince one of the mostly loose resemblance that ANNs bear to the human brain. A more worthy endeavor in mimicking the human neural networks are Spiking Neural Networks. They enjoy a great deal of attention in neuroscience, but are yet to be found in engineering applications. This is because, while they serve as valuable computational models to mimic the processes within us, there still lies a wide gap between their theoretical power and what has been practically achieved till date. Perhaps that will change in the future, igniting the minds of many more science fiction authors!