Real-time 3D Eyelids Tracking from Semantic Edges
Quan Wen
Feng Xu
Ming Lu
Jun-Hai Yong
Tsinghua University
Results of current real-time face tracking systems still lack of realism on local face organs, especially on eye regions. In this paper, we propose a technique that reconstructs 3D shapes and motions of eyelids in real time. Combining with the tracked face and eyeballs, our system generates full face results with more detailed eye regions in real time. To achieve this goal, we propose a generative eyelid model which decomposes the variations of eyelids into two low dimensional linear spaces which efficiently represent the shapes and motions of eyelids, respectively. Then we modify a holistically-nested DNN model to jointly perform semantic eyelid edge detection and identification on images. Next, correspondences are constructed between the vertices on the eyelid model and the 2D edges, where a polynomial curve fitting and a corresponding searching scheme is used to handle wrong and partial edge detections. Finally, the correspondences are involved in a 3D-to-2D edge fitting scheme to reconstruct the shapes and poses of eyelids. By integrating our fast fitting method into a face tracking system, the estimated eyelid results are seamlessly fused with the face and eyeball results in real time. Experiments show that our technique is applicable for different human races, eyelid shapes and motions and is robust to changes in head poses, expressions and gaze directions.
Note
@article{wen2017real,
  title={Real-time 3D Eyelids Tracking from Semantic Edges},
  author={Wen, Quan and Xu, Feng and Lu, Ming and Yong Jun-Hai},
  journal={ACM Transactions on Graphics (TOG)},
  year={2017},
  publisher={ACM}
}
Quan Wen, Feng Xu, Ming Lu and Jun-Hai Yong. 2017. "Real-time 3D Eyelids Tracking from Semantic Edges". ACM Transaction on Graphics (TOG).
Abstract
Files
[video]
[paper]
[semantic edge training set]
[eyelid model]
*
*Corresponding author (feng-xu@tsinghua.edu.cn)
Ÿ The eyelid model database and semantic edge training set are available for non-commercial research purposes only.

Ÿ You agree not to reproduce, duplicate, copy, sell, trade, resell or exploit for any commercial purposes, any portion of the models, any portion of the images and any portion of the derived data.

Ÿ You agree not to further copy, publish or distribute any portion of the eyelid model dataset and the semantic edge training set.
Citation