New software tracks a person’s facial expressions and maps it—in real-time—onto a digital character. This real-time and calibration-free facial performance capture framework is based on a sensor with video and depth input. In this framework, the inventors developed an adaptive PCA model using shape correctives that adjusts on-the-fly to the actor's expressions through incremental PCA-based learning. Since the fitting of the adaptive model progressively improves during the performance, they do not require an extra capture or training session to build this model. As a result, the system is highly deployable and easy to use: it can faithfully track any individual, starting from just a single face scan of the subject in a neutral pose. Like many real-time methods, they use a linear subspace to cope with incomplete input data and fast motion. To boost the training of our tracking model with reliable samples, they use a well-trained 2D facial feature tracker on the input video and an efficient mesh deformation algorithm to snap the result of the previous step to high frequency details in visible depth map regions. This shows that the combination of dense depth maps and texture features around eyes and lips is essential in capturing natural dialogues and nuanced actor-specific emotions. It also demonstrates that using an adaptive PCA model not only improves the fitting accuracy for tracking but also increases the expressiveness of the retargeted character.