Multimodal Face Data Sets - A Survey of Technologies, Applications, and Contents

Introduction. Biometric verification for controlling access to digital devices has become ubiquitous for its ease-of-use but often relies on notoriously data-hungry supervised machine learning. Additionally, biometric data is privacy-sensitive, and a comprehensive review from a data set perspective is missing. This survey provides detailed insights into data acquisition and Additionally, it offers a thorough discussion of related applications and state-of-the-art performance. Methodology. We present a comprehensive and structured review of multimodal face data sets containing RGB color and other channels such as infrared or depth. This follows a trend to use such additional modalities to improve the robustness and reliability of biometric verification systems. Starting with around 1,000 papers, retrieved based on search queries and forward-/backward-referencing, we selected 200 papers in this survey following our inclusion and exclusion criteria. We discuss around 150 multimodal data sets. Principal Findings. Multimodal data opens the path to many additional applications, such as 3D face reconstruction (e.g., to create avatars for VR/AR environments), detection, registration, alignment, and recognition systems, emotion detection, anti-spoofing, etc. Our findings show that multimodal data can boost performance and robustness in many applications. However, consistent concerns include cross-domain generalization problems due to biases in ethnicity, age, and gender, and class imbalances that may lead to mispredictions or underrepresentation of minorities. We believe that the latter point deserves future research, not only to represent minorities accurately but also to tackle societal and infrastructural biases and to ensure that work based on such data sets remains fair and equitable.