Title: “People Search beyond Text in Consumer Videos”

Team members

• Ryan Lei (雷禹恆, r99944007, CMLab, MiRA group): System core, Team coordinator
• Kai-Yu Tseng (曾開瑜, r99944001, CMLab, MiRA group): Clothing segmentation
• Han-Wei Liao (廖航緯, r99922059, CMLab, Graphics group): Face tracking

Introduction

This project is an extension of a previous work by Chien-Hsing Chiang (江建興) [1]. The previous work proposed a name propagation algorithm over a face similarity graph to boost people search in videos, and the results outperformed traditional text-based approach found in current online services such as Youtube.

System framework:

Name propagation based on cluster similarities:

In this project, we aim to extend from celebrity videos to consumer (family) videos, in which every individual is considered a “celebrity”. In home videos, the frame quality is much poorer, and the facial variations in poses, expressions, and lighting are even farther from ideal.

Our previous work did not make use of temporal information in video processing. To improve the performance, we plan to introduce automatic Clothing segmentation [2] and Face tracking techniques in the “face detection” step, so that after “local clustering”, we may be able to merge or split the clusters for the later steps.

Dataset

We plan to use a small number (around 15) of HD family videos collected from Youtube, then manually annotate the names at the video level for initial tagging, and at the face level for performance evaluation. We expect the videos to cover faces for a large-enough percentage of time, and several people will appear equally long in the videos, instead of being dominated by one or two.

Expected Result

The expected result is a Web-based people search system similar to the demo system of our previous work. We'll show respectively the system with 10 fully-labeled videos and the system with 5 additional unlabeled videos.

References

1. C.-H. Chiang et. al., “Boosting People Search and Disambiguation in User-Contributed Videos by Name Propagation over Face Graph,” IEEE Trans. on Circuits and Systems for Video Technology (TCSVT) (submitted)
2. Yuri Y. Boykov Marie-Pierre Jolly,"Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N-D Images" Proceedings of “Internation Conference on Computer Vision”, Vancouver, Canada, July 2001
3. Face tracking?

Any comment is appreciated! :)