Activation Function
An activation function is a function that is applied to the output of a neuron. They are also sometimes called transfer functions. The following is a list of a few of the popular activation functions.
Click on the plot images below to open the interactive Geogebra links:
| Name | Function | Range | Interactive Plot |
|---|---|---|---|
| Sigmoid | f(x) = 1 / (1 + e-x) | (0, 1) |  |
| Tanh (Hyperbolic Tangent) | f(x) = (ex - e-x) / (ex + e-x) | (-1, 1) |  |
| ReLU (Rectified Linear Unit) | f(x) = max(0, x) | [0, ∞) |  |
| Leaky ReLU | f(x) = x if x >= 0 else 0.01x | (-∞, ∞) |  |