FROM PROBABILITY DENSITY TO NEURAL NON-LINEARITY: THE 4RS FAMILY AS A NEW PARAMETRIC CLASS OF ACTIVATION FUNCTIONS FOR NEURAL NETWORKS

Abstract

This article presents the 4Rs-Gate, a novel parametric activation function for neural networks. While popular self-gated activation functions, such as GELU and Swish, multiply the input by the cumulative distribution function (CDF) of a fixed unimodal density, the 4Rs-Gate introduces innovation by utilizing the CDF of the 4Rs Family. This function consists of a convex mixture between a cubic B-spline and the normalized magnitude of a mother wavelet. The interpolation between the smoothness of the cubic B-spline and the spatial localization of the wavelet is controlled by a single parameter, , with values constrained between 0 and 1. In this work, we derived its analytical properties, validated the exactness of its gradients using finite difference approximations, and evaluated the model's overall performance using multilayer perceptrons (MLP) trained across two synthetic benchmark tasks with five random seeds. In numerical experiments, the 4Rs-Gate (configured with ) outperformed traditional activation functions including ReLU, GELU, Swish, and Tanh on the task featuring multi-scale local structures, while achieving competitive and equivalent performance to classical activation baselines on the task lacking such local spatial characteristics.