A fixed-dynamic networks model is based on branch networks which includes at least more than one branch networks before a main network, wherein each network here is a neural network.<\/p>\n\n\n\n
The fixed-dynamic networks model have 2 types of training, fixed trainig which is for adjusting fixed weights, and dynamic training which is for adjusting dynamic weights, as mentioned in previous posts. <\/p>\n\n\n\n
The fixed training is pre training which is the training before inference or working of the fixed-dynamic networks model. The dynamic training is along with the inference or working of the fixed-dynamic networks model.<\/p>\n\n\n\n
The fixed-dynamic networks model has 2 kinds of gradients or partial derivatatives in the backpropagation for dynamic training, in which one kind is fixed gradients which keep unchanging\/constant\/fixed during the dynamic training, and another kind is dynamic gradients which are changable\/adjustable\/dynamic during the dynamic training.<\/p>\n\n\n\n
The fixed gradients after fixed training can be stored to be used directly in dynamic training, which can save a lot of workload and time and reduce chip circuits complexity much for the fixed-dynamic networks model.<\/p>\n\n\n\n
Make sure to have a good set of fixed gradients after fixed training, not only ensure fixed gradients are not 0 but also make fixed gradients values suitable or proper to transmit backpropagation in dynamic training.<\/p>\n\n\n\n
For continual learning especially for edge\/device applications!<\/p>\n","protected":false},"excerpt":{"rendered":"
A fixed-dynamic networks model is based on branch networks which includes at least more than one branch networks before a main network, wherein each network…<\/p>\n