machine learning - Is it possible to fix the weight of a certain layer during neural network training?

As suggested by the title, is it possible to fix some part of the neural network while training?

Since Mathematica provides a way to extract part of a neural network and combine it with some layers to make a new one:

newNet = NetChain[{Take[oldNet, 3], 10, Ramp, 10}]

It would be very helpful to fix the layers taken from the old network. In this way one can reuse the neural network and investigate how transferrable are features in the neural network (c.f. https://arxiv.org/abs/1411.1792).

Answer

The LearningRateMultipliers supports using different learning rates with different layers. When setting the learning rate of a particular layer to None, the weight of that layer will be fixed during training.

You can have a look at the example in the documentation of LearningRateMultipliers.

In that example, net is constructed from part of the network trained for MNIST

net = NetChain[Join[Drop[Normal[lenet], -2],
   {LinearLayer[], SoftmaxLayer[]}],

  "Input" -> enc, "Output" -> dec]

Then at the training time, only the newly added linear layer is allowed to change. All the other layers are fixed by setting the learning rate to None.

trained = 
 NetTrain[net, trainingData, 
  MaxTrainingRounds -> Quantity[10, "Seconds"], 
  LearningRateMultipliers -> {-2 -> 1, _ -> None}, 
  ValidationSet -> Scaled[0.1]]