The extreme variability of faces in smart environment applications, due to continuous changes in terms of pose, illumination and subject appearance (hairstyle, make-up, etc.), requires the relevant mode of variations of the subject's faces to be encoded in the templates and to be continuously updated based on new inputs. This work proposes a new video-based template updating approach suitable for home environments where the image acquisition process is totally unconstrained but a large amount of face data is available for continuous learning. A small set of labeled images is initially used to create the templates and the updating is then totally unsupervised. Although the method is here presented in conjunction with a subspace-based face recognition approach, it can be easily adapted to deal with different kinds of face representations. A thorough performance evaluation is carried out to show the efficacy and reliability of the proposed technique.