Skip to main content

Are deep neural networks really that deep?

 

We chose to call neural networks 'deep learning'. I seem to have a minor issue with this name as I think more appropriate name (at least for now) would be 'deeper learning'. Why deeper instead of deep? Well, if you look how wide current neural networks are (how many features are coming out) and how deep they are (number of layers), it seems that for a lot of neural networks - they are way more wide than deep. For example - Resnet for CIFAR-10 has 41 layers and 512 features are coming out. It seems wider than deep, hence I believe it's more of a "deeper learning" than "deep learning".

Resnet for CIFAR-10

I tried to find today neural network, which would be deeper than wider, but I failed - but my search was not complete as I used only internet and not some university library and a lot of articles are behind the pay walls. But maybe the reason why I failed is very linear.


Disclaimer: I am no longer employee of Deep Learning Technology Center in Microsoft Research (it has been a while)