I just tried some simple activation functions for UAT\/Neural Network – ParaLU and QuartLU which are Parabolic Linear Unit and Quartic Linear Unit.<\/p>\n\n\n\n
I made a 1000 features and 4 hidden layers neural network to approximate 21 pairs of x and y. In the test, ParaLU converges much slower in training than Sigmoid but similarly or sometime better than ReLU. QuartLU converges also much slower than Sigmoid but obviously better than ParaLU and Relu. Sigmoid is a really monster in training that it converges really fast.<\/p>\n\n\n\n
ParaLU and QuartLU below are all continously differentiable in all x.<\/p>\n\n\n\n
1) ParaLU’s math:<\/p>\n\n\n\n
if x<-scope, y=0 and y’=0; <\/p>\n\n\n\n
else if x<scope, y=(x+scope)**2\/(scope*4) and y’=(x+scope)\/(2*scope);<\/p>\n\n\n\n
else y=x and y’=1.<\/p>\n\n\n\n
It seems that scope=1 is good for ParaLU above.<\/p>\n\n\n\n
2) QuartLU’s math<\/p>\n\n\n\n
if x<-scope, y=0 and y’=0;<\/p>\n\n\n\n
else if x<(scope\/3), y=(x+scope)**4*27\/(256*scope**3);<\/p>\n\n\n\n
else y=x and y’=1.<\/p>\n\n\n\n
It seems that scope=1.5 is good for QuartLU above.<\/p>\n\n\n\n
Both Gemini and Grok searched and confirmed that there are no exactly same activation functions as ParaLU or QuartLU although there are similar ones. <\/p>\n\n\n\n
Also both commented that ParaLU and QuartLU have much lower working load on hardware than Sigmoid like ReLU but at same time are continuously differentiable on all x. <\/p>\n","protected":false},"excerpt":{"rendered":"
I just tried some simple activation functions for UAT\/Neural Network – ParaLU and QuartLU which are Parabolic Linear Unit and Quartic Linear Unit. I made…<\/p>\n