Are You Watching Closely? Content-based Retrieval of Hand Gestures

Amiri Parian, Mahnaz and Rossetto, Luca and Schuldt, Heiko and Dupont, Stéphane. (2020) Are You Watching Closely? Content-based Retrieval of Hand Gestures. In: Proceedings of the 2020 International Conference on Multimedia Retrieval (ICMR 2020). New York, pp. 266-270.

Full text not available from this repository.

Official URL: https://edoc.unibas.ch/81640/

Downloads: Statistics Overview


Gestures play an important role in our daily communications. However, recognizing and retrieving gestures in-the-wild is a challenging task which is not explored thoroughly in literature. In this paper, we explore the problem of identifying and retrieving gestures in a large-scale video dataset provided by the computer vision community and based on queries recorded in-the-wild. Our proposed pipeline, I3DEF, is based on the extraction of spatio-temporal features from intermediate layers of an I3D network, a state-of-the-art network for action recognition, and the fusion of the output of feature maps from RGB and optical flow input. The obtained embeddings are used to train a triplet network to capture the similarity between gestures. We further explore the effect of a person and body part masking step for improving both retrieval performance and recognition rate. Our experiments show the ability of I3DEF to recognize and retrieve gestures which are similar to the queries independently of the depth modality. This performance holds both for queries taken from the test data, and for queries using recordings from different people performing relevant gestures in a different setting.
Faculties and Departments:05 Faculty of Science > Departement Mathematik und Informatik > Informatik > Databases and Information Systems (Schuldt)
UniBasel Contributors:Schuldt, Heiko and Parian-Scherb, Mahnaz
Item Type:Conference or Workshop Item, refereed
Conference or workshop item Subtype:Conference Paper
Publisher:Association for Computing Machinery
Note:Publication type according to Uni Basel Research Database: Conference paper
Identification Number:
Last Modified:02 Feb 2021 13:10
Deposited On:02 Feb 2021 13:10

Repository Staff Only: item control page