Comprehensive Guide on ReLU
Start your free 7-days trial now!
Rectified linear units, or ReLU, is an activation function that is commonly used for neural networks. The mathematical formula for ReLU is quite simple:
This formulation is actually equivalent to the following:
Graphically, ReLU would look like the following:
Implementation in Python
The implementation of ReLU is straight-forward:
import numpy as npdef relu(x):return np.maximum(0,x)
Derivative of ReLU
The derivative of ReLU is straight-forward - we just need to consider the two cases:
when $x$ is less than or equal to zero - the derivative would simply be 0 since the slope is flat
when $x$ is larger than zero - the derivative would be 1 since we just have a linear curve $y=x$.
Mathematically, this is the following: