Pooling shrinks a feature map by replacing each small window with a single summary number. It is the one layer in a ConvNet with no parameters at all — nothing about it is learned, and backpropagation has nothing to update.
Max pooling
Take the window, keep the largest value, discard the rest. With \(f = 2\), \(s = 2\) on a 4×4 input:
\[\begin{bmatrix} 1 & 3 & 2 & 1 \\ 2 & 9 & 1 & 1 \\ 1 & 3 & 2 & 3 \\ 5 & 6 & 1 & 2 \end{bmatrix} \;\longrightarrow\; \begin{bmatrix} 9 & 2 \\ 6 & 3 \end{bmatrix}\]Each 2×2 block collapses to its maximum. The output is half the height and half the width, so a quarter of the values survive.
Step through the four windows, and switch to average pooling to see the same input give a different answer:
type: pooling
mode: max
f: 2
s: 2
values: 1,3,2,1; 2,9,1,1; 1,3,2,3; 5,6,1,2
The output size rule is the same one from part 4, with the filter size now meaning the pooling window:
\[\left\lfloor \frac{n + 2p - f}{s} \right\rfloor + 1\]Padding is almost always 0 for pooling, and \(f = s = 2\) is overwhelmingly the common choice — which is why pooling breaks the odd-filter-size convention from part 3. Nothing needs a centre pixel here, and 2×2 with stride 2 tiles the input exactly, with no overlap and nothing dropped.
Average pooling
Same window, mean instead of maximum:
\[\begin{bmatrix} 1 & 3 & 2 & 1 \\ 2 & 9 & 1 & 1 \\ 1 & 3 & 2 & 3 \\ 5 & 6 & 1 & 2 \end{bmatrix} \;\longrightarrow\; \begin{bmatrix} 3.75 & 1.25 \\ 3.75 & 2 \end{bmatrix}\]Max pooling dominates inside networks, and the preference is empirical rather than principled — a direct comparison on recognition tasks found max consistently ahead of average [1]. Average pooling survives in one very important place — global average pooling, where the window is the entire feature map, turning \(n_H \times n_W \times n_C\) into \(1 \times 1 \times n_C\). That single trick is what lets an architecture drop the enormous fully connected layers discussed in part 6; it was introduced with Network in Network [2] and adopted by GoogLeNet [3] and ResNet [4].
Pooling acts per channel
This is the detail most often got wrong. Pooling does not sum over channels the way convolution does. It runs independently on each channel and the channel count is unchanged:
\[n_H \times n_W \times n_C \;\longrightarrow\; \left\lfloor \frac{n_H - f}{s} \right\rfloor + 1 \;\times\; \left\lfloor \frac{n_W - f}{s} \right\rfloor + 1 \;\times\; n_C\]Convolution mixes channels and can change their number; pooling never touches them.
What actually matters
Pooling has no parameters, but it is not free. The hyperparameters \(f\) and \(s\) are design decisions that permanently discard information — and because there is nothing to learn, the network cannot compensate for a bad choice the way it can with a badly initialised convolution. It also means pooling layers are sometimes not counted as “layers” at all, which is why depth figures for the same network differ between sources.
The invariance argument is weaker than it is usually stated. Max pooling is often justified as giving translation invariance: shift the input a pixel and the maximum in the window is often unchanged. That holds for shifts within a window and fails at window boundaries, so what you actually get is partial invariance to small shifts, not invariance. Networks are far more robust to translation because of data augmentation and parameter sharing than because of pooling.
Modern architectures increasingly drop it. If a stride-2 convolution can downsample and it has learnable weights, the argument for a fixed max is thin — the all-convolutional experiments showed no accuracy loss from replacing pooling with strided convolution [5], and ResNet uses exactly one max-pool layer, right after the stem. Pooling persists mainly at the very start (cheap early downsampling on a large feature map) and at the very end (global average pooling). The middle of a modern network usually has none.
Backpropagation through max pooling routes, it does not distribute. The gradient goes entirely to whichever input held the maximum; every other input in the window gets zero. Average pooling splits the gradient evenly instead. This is worth knowing when a network trains oddly: max pooling makes the gradient signal sparse.
Source code
Convolution_model_Step_by_Step_v1.ipynb—pool_forwardimplements both modes, and the backward pass shows the routing behaviour above explicitly.
References
- [1]D. Scherer, A. Müller, and S. Behnke, “Evaluation of pooling operations in convolutional architectures for object recognition,” in International Conference on Artificial Neural Networks (ICANN), 2010, pp. 92–101. doi: 10.1007/978-3-642-15825-4_10.
- [2]M. Lin, Q. Chen, and S. Yan, “Network in network,” in International Conference on Learning Representations (ICLR), 2014.
- [3]C. Szegedy et al., “Going deeper with convolutions,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1–9.
- [4]K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
- [5]J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” in International Conference on Learning Representations (ICLR) Workshop Track, 2015.