Problem discussed: Techniques to improve neural network architecture.

Techniques: The authors propose GradNets which dynamically adapt the architecture of the model. This is done by adding identity or some other connections in parallel with the normal connection. The techniques used in the paper are shown in the table below. Capture

An example GRelu connection is shown in the figure below.Capture1

Here g is the weight which increases from 0 to 1 over the training. In the paperCapture2

where t is the epoch number and τ is a hyperparameter. The identity and ReLu in the above figure can be replaced by different  architecture components from the table. The authors experimented on CIFAR10 dataset using the above techniques and showed improvements.


-These are some simple techniques to improve the performance of the network which can be implemented easily without much increase in memory or computation.


GradNets: Dynamic Interpolation Between Neural Architectures. Diogo Almeida, Nate Sauder