Simple learning with a teacher via biased regularized least squares

In the paradigm of learning with a teacher, introduced by Vapnik, a supervised learner is trained on an augmented features space, and a student is requested to match the teacher accuracy as much as possible in a reduced feature space. In particular, in the transfer learning mode proposed by Vapnik, a method was formalized to move the knowledge from the teacher to the student. In this paper, we use biased regularized least squares as a simple yet effective method to transfer the knowledge from one learner to another, and to assess its accuracy. We achieve this by further generalizing a semi-supervised learning method, which we previously introduced. We will show that, with this approach, the teacher can be any classifier. In particular, we will employ the Relevance Vector Machine (RVM) as teacher to assess the method’s capability in transferring the knowledge in terms of classification accuracy, and in reproducing the probabilities coming from RVM. We validate the method against standard UCI datasets and systematically compare it with Vapnik’s original method in terms of accuracy and execution time. We thus demonstrate the feasibility and speed of this new approach.