How Dropout Improves Neural Network Accuracy

What is Dropout?

Dropout is a regularization technique used to reduce over-fitting in Neural Networks(NN), Deep Learning. Over-fitting occurs when a model fits very well on training data but performs poorly on new data. A complex MLP tries to make a curve that creates a sophisticated shape in order to divided different groups of data. Dropout temporarily turns off some of the model's neurons during training — which prevents the network from becoming complex and increases its generalization ability.

When to use Dropout?

Dropout is usually applied to hidden layers during training phase. The input layer usually has a low dropout rate (approximate 0.1 or 0.2) In hidden layers, it is around 0.5. Dropout is not used during testing or inference. Rather than that, use a probability of 0.5 for (this case) and multiply it with the output of each neuron, which also known as the Inverted Dropout technique.

When to use dropout:

When the model you have make is causes over-fitting.

When the training accuracy is high but the validation accuracy is low.

A deep learning neural network with many parameters.

How it works?

Set a dropout rate of 0.4 (can be change depends on architecture and dataset). so if a layer contains 100 neurons, 40 neurons will be deactivated randomly and reduce the complexity level of the neural architecture.. However, best practice is to assign different dropout rate for every layer.

self.dropout1 = nn.Dropout(p=0.2) # for layer 1
self.dropout2 = nn.Dropout(p=0.4) # for layer 2
self.dropout1 = nn.Dropout(p=0.3) # for layer 3
self.dropout2 = nn.Dropout(p=0.1) # for layer 4

Depending on the architecture of the model and the complexity of the data, you can set a different dropout rate for each layer. Dropout increases the generalization ability of the model while ensuring better performance on new data. On the other hand, using a high dropout rate can reduce the learning ability of the model. Thus, Dropout helps to improve model accuracy.