Transferring Knowledge from a Teacher Neural Network to a Student Neural Network

Abstract
A method for training a student neural network to adopt the behavior of a teacher neural network that is trained to perform a given processing on input images. The method includes: providing a set of training images; producing, from training image(s), one or more style-augmented versions that have the same semantic content as the original training image but differ from the original training image in their style; processing the training images and the augmented versions by the teacher neural network, and by the student neural network; evaluating, using a predetermined loss function, to which extent outputs and/or intermediate work products produced by the student neural network from each image are in agreement with the outputs and/or intermediate work products produced by the teacher neural network from the same image; and optimizing parameters that characterize the behavior of the student neural network.