Stylized Avatar Animation Based on Deep Learning

Animating 3D character rigs from human faces requires both geometry features and facial expression information. However, traditional animation approaches such as ARkit failed to connect character storytelling to the audience because the character expressions are hard to recognize. However, recent emotion-based motion capture techniques, such as ExprGen, consider using facial emotion for facial capture. But it is difficult to characterize the details of the character’s face. A network is proposed to incorporate facial expressions into animation. Firstly, an emotion recognition neural network is used to match human and character datasets. Then, an end-to-end neural network is trained to extract character facial expressions and transfer rig parameters to characters. Finally, human face geometry is utilized to refine rig parameters. Qualitative analysis of the generated character expressions, and quantitative analysis of the attractiveness and intensity of the character expression have demonstrated the accuracy and real-time of the model.

Recommended citation: Zhang, R., & Pan, Y. (2022). Stylized Avatar Animation Based on Deep Learning. Journal of Computer-Aided Design & Computer Graphics, 34(5), 675-682.
Download Paper