fully connected layer as 1x1 convolution

Quoting Lecun’s post:

In Convolutional Nets, there is no such thing as “fully-connected layers”. There are only convolution layers with 1x1 convolution kernels and a full connection table.

It’s a too-rarely-understood fact that ConvNets don’t need to have a fixed-size input. You can train them on inputs that happen to produce a single output vector (with no spatial extent), and then apply them to larger images. Instead of a single output vector, you then get a spatial map of output vectors. Each vector sees input windows at different locations on the input.

In that scenario, the “fully connected layers” really act as 1x1 convolutions.

References:

• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• An Analysis of the Connections Between Layers of Deep Neural Networks
• Torch nn.SpatialConvolutionMap
• 1x1 convolution - why use it?