Last updated on May 6, 2026
I just tried some simple activation functions for UAT/Neural Network – ParaLU and QuartLU which are Parabolic Linear Unit and Quartic Linear Unit.
ParaLU and QuartLU below are all continously differentiable in all x.
1) ParaLU’s math:
if x<-scope, y=0 and y’=0;
else if x<scope, y=(x+scope)**2/(scope*4) and y’=(x+scope)/(2*scope);
else y=x and y’=1.
use scope=0.1 in test below.
2) QuartLU’s math
if x<-scope, y=0 and y’=0;
else if x<(scope/3), y=(x+scope)**4*27/(256*scope**3);
else y=x and y’=1.
use scope=0.3 in test below.
3) test
In vs code on my laptop, I programmed a neural network with adjustable parameters of layers, features and activation fuction switch.
So I just tested Sigmoid, Relu, ParaLU and QuartLU in 15 layers and 128 features through 500 loops to approximate 21 pairs of x and y.
In few tests , Sigmoid cannot work, and Relu, ParaLU and QuartLU can work similarly well. It may be like: the smaller the scope is the more ParaLU and QuartLU are similar to Relu, but the larger the scope is ParaLU and QuartLU are better at approximating smoother continuous curve than Relu at the expense of slower training speed.
4) Both Gemini and Grok searched and confirmed that there are no exactly same activation functions as ParaLU or QuartLU although there are similar ones.
Also both commented that ParaLU and QuartLU have much lower working load on hardware than Sigmoid like ReLU but at same time are continuously differentiable on all x.
Be First to Comment