Just tried two new LUs – PowerLU and ExpoLU which have continuous value y but are not continuously differentiable just like ReLU.<\/p>\n\n\n\n
ExpoLU beats ReLU by huge advantages (always several magnitudes) in my few tests of 300 rounds training of a neural network of 512 features and 30 layers to approximate 21 pairs of x and y, under same setting including learning rate.<\/p>\n\n\n\n
1) PowerLU’s math:<\/p>\n\n\n\n
x<-1, y=0, y’=0<\/p>\n\n\n\n
-1<=x<1, y=(x+1)**p\/(2**p)<\/p>\n\n\n\n
1<x, y=x<\/p>\n\n\n\n
2) ExpoLU’s math<\/p>\n\n\n\n
ExpoLU has 3 parameters: a, b, p.<\/p>\n\n\n\n
x<-b, y=0<\/p>\n\n\n\n
-b<=x<b, y=(1\/a)(x+b)**p<\/p>\n\n\n\n
b<x, y=x+(1\/a)(2b)**p-b<\/p>\n\n\n\n
It’d better be a=2**n to make division as a bit shift in hardware.<\/p>\n\n\n\n
If set a=1, ExpoLU has another simplified version (first version):<\/p>\n\n\n\n
x<-b, y=0<\/p>\n\n\n\n
-b<=x<b, y=(x+b)**p<\/p>\n\n\n\n
b<x, y=x+(2b)**p-b<\/p>\n\n\n\n
3) I’m trying these Linear units for fun and also for: first to make training converge become more jumping to avoid small deep holes, second to provide some curve around x=0 for activation.<\/p>\n\n\n\n
In my few tests of a neural network of 512 features and 30 layers to approximate 21 pairs of x and y, both PowerLU and ExpoLU can work with proper settings.<\/p>\n\n\n\n
Especially in the few tests above, Relu is not a match of ExpoLU at all in same settings including learning rate, in which Relu is far way below ExpoLU’s performance.<\/p>\n\n\n\n
ExpoLU is a real monster here and it has 3 parameters (a, b, p), but I just tried several simplest combinations like a=2,b=1,p=2 and a=1,b=1,p=2, in which a=1 reach 1e-5 to 1e-9 and a=2 reach 1e-5 to 1e-7 (while Relu reach 1e-1 only once in 3 tests).<\/p>\n","protected":false},"excerpt":{"rendered":"
Just tried two new LUs – PowerLU and ExpoLU which have continuous value y but are not continuously differentiable just like ReLU. ExpoLU beats ReLU…<\/p>\n