Problem discussed: Techniques to improve neural network architecture.
Techniques: The authors propose GradNets which dynamically adapt the architecture of the model. This is done by adding identity or some other connections in parallel with the normal connection. The techniques used in the paper are shown in the table below.
An example GRelu connection is shown in the figure below.
Here g is the weight which increases from 0 to 1 over the training. In the paper
where t is the epoch number and τ is a hyperparameter. The identity and ReLu in the above figure can be replaced by different architecture components from the table. The authors experimented on CIFAR10 dataset using the above techniques and showed improvements.
-These are some simple techniques to improve the performance of the network which can be implemented easily without much increase in memory or computation.