2.4. Neural Network Architecture

For MNIST, a 4-layer LeNet convolutional neural network (CNN) backbone is used, with two 5×5 convolution layers with 32 and 64 filters, respectively, followed by ReLU and 2×2 max pooling, and two fully connected layers with 256 and 10 neurons. For the other two datasets, a ResNet-18 [22] backbone is used, with 4 residual blocks with 64, 128, 256 and 512 filters, respectively.

For the CSNN, two architectures, illustrated in Figure 2, will be investigated. The first is a small one (called CSNN), illustrated in Figure 2a, which takes the output of the last convolutional layer of the backbone as input, normalized as described in Section 2.2 using a batch normalization layer without any learnable affine parameters. The second one is a full network (called CSNN-F), illustrated in Figure 2b, where the backbone (LeNet or ResNet) is part of the backpropagation, and a Batch Normalization layer (BN) without any learnable parameters is used between the backbone and the CSN layer.

All experiments were conducted on an MSI GS-60 Core I7 laptop with 16GB RAM and Nvidia GTX 970M GPU, running the Windows 10 operating system. The CSNN and CSNN-F networks were implemented in PyTorch 1.90.