Just thought about a simple method to do continual learning: <\/p>\n\n\n\n
a neural network after training includes FxN weights matrix in first hidden layer and NxN weights matrix for each following hidden layers;<\/p>\n\n\n\n
expand weights matrix of first layer from FxN to Fx(N+D); <\/p>\n\n\n\n
expand weights matrix of each subsequent hidden layer from NxN to (N+D)x(N+D);<\/p>\n\n\n\n
in which the weightsof the submatrices FxN and NxN of hidden layers are frozen (fixed, unchangable, untrainable) during inference or working or deployment;<\/p>\n\n\n\n
in which the added weights of added D columns and added D rows of hidden layers are dynamic (changable, trainable, adjustable) in inference or working or deployment.<\/p>\n\n\n\n
The output of the output layerof the network with expanded hidden layer matrices can be of same dimensions as the output of the network before expansion.<\/p>\n\n\n\n
This continual learning model is simple and same compatible for hardware chip deployment as present fixed models. It’s effect may be better than branch networks based on intermediate omnitoken but much simpler.<\/p>\n\n\n\n
In nature, this one is just adding additional approximators for each hidden layer and additional dimensions for each approximator of subsequent hidden layers after first hidden layer. <\/p>\n\n\n\n
PS: branch networks on intermediate omnitoken can be specialized for continual learning of individual user, and this layer expansion can be for main network learning on data of all users.<\/p>\n","protected":false},"excerpt":{"rendered":"
Just thought about a simple method to do continual learning: a neural network after training includes FxN weights matrix in first hidden layer and NxN…<\/p>\n