Projects

This is our project page. Here, we highlight current projects, both funded and unfunded. You can check out our past projects by clicking the link below.

Face Verification with Caricatures

A good caricature looks "more like a face than the face itself" -- Brennan, 1985

Caricatures exaggerate what makes someone unique, highlighting what makes them different from the average person. Humans are able to more quickly identify caricatured faces over realistic faces. We aim to combine research in machine learning and human perception in order to fundamentally change the future of face verification to inform both automated and human-based systems.

Funding

2019-2024

National Science Foundation

#1909707, $435K

Original grant awarded 08/2019. REU Supplement awarded 10/2020.

This material is based upon work supported by the National Science Foundation under Grant No. 1909707.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Publications

Davis, Sara R., Bryson Lingenfelter, KevinMcElhinney, Shamik Sengupta, and Emily M. Hand. “CarVer: Setting the Standard for Face Verification with Caricatures.” 2023, IEEE International Joint Conference on Biometrics (IJCB). IEEE, 2023.

Thom, Nathan, Andrew DeBolt, Lyssie Brown, and Emily M. Hand. “DoppelVer: A Benchmark for Face Verification.” 2023, Springer International Symposium on Visual Computing (ISVC). Springer, 2023.

Lingenfelter, Bryson, Sara R. Davis, and Emily M. Hand. "A Quantitative Analysis of Labeling Issues in the CelebA Dataset." International Symposium on Visual Computing. Cham: Springer International Publishing, 2022.

Davis, Sara R., and Emily M. Hand. "Improving Face Recognition Using Artistic Interpretations of Prominent Features: Leveraging Caricatures in Modern Surveillance Systems." Intelligent Video Surveillance-New Perspectives. IntechOpen, 2022.

Lingenfelter, Bryson, and Emily M. Hand. "Improving evaluation of facial attribute prediction models." 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021). IEEE, 2021.

Thom, Nathan, Hung Nguyen, and Emily M. Hand. "Consensus Subspace Clustering." 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 2021.

Datasets & Code

DoppelVer

ISVC, 2023

The DoppelVer dataset is a benchmark dataset designed for evaluating and advancing face verification and recognition algorithms. This dataset contains a diverse collection of facial images. The identities depicted were selected in pairs of doppelganger identities. A doppelganger is an identity who is highly visually similar to the source identity, such that they might be mistaken for one another.

Dataset: You can request the dataset by completing and submitting the End User License Agreement.

CarVer

IJCB, 2023

The CarVer dataset is a benchmark dataset designed to set the standard for face verification using caricatures and veridical images. This dataset contains a diverse collection identities with caricature and real images for each identity.

Dataset: You can request the dataset by completing and submitting the End User License Agreement.

Face Verification with Caricatures

A good caricature looks "more like a face than the face itself" -- Brennan, 1985
Caricatures exaggerate what makes someone unique, highlighting what makes them different from the average person. Humans are able to more quickly identify caricatured faces over realistic faces. We aim to combine research in machine learning and human perception in order to fundamentally change the future of face verification to inform both automated and human-based systems.

This material is based upon work supported by the National Science Foundation under Grant No. 1909707.
Title: Towards Human-Level Face Verification Performance using Distinctive Features
PI: Dr. Emily Hand
Project Period: August 2019 - July 2022

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Gender Bias in Political Writing

In this project we seek to analyze gender bias in political news articles. It is well known that document and word embeddings exhibit severe gender biases. We aim to hone in on this line of research to identify what exactly is being saved in these embeddings that makes them biased against women? Looking at political news articles, we remove all gendered terms and build gender classifiers based on document embeddings. In the image we have four different embedding visualizations. The top row show the embeddings for Breitbart news from the uncleaned data on the left and the cleaned data on the right. The bottom row similarly shows the uncleaned and cleaned embeddings from the New York Times. The clear separation before and after cleaning demonstrates that there is a fundamental difference in the way women and men are portrayed in political writing across the political spectrum.

Automated Sample Pruning

Many datasets contain noise in the form of incorrect labels. How can we remove or relable these noisy samples without human intervention? We introduce an automated method for sample pruning in the context of attribute prediction. Facial attributes are describable features of faces, such as face shape and hair color. We build representative sets for each attribute, consisting of images that are positive instances of an attribute as well as images that are negative instances of the attribute. The image on the left shows the representative sets for the attribute "beard." We then use these representative sets to determine if a sample has a noisy label for that attribute through a verification process.

Micro-Expression Recognition

In our daily interactions with other people, we change our behavior based on the behavior of others. We take cues from their body language and their facial expression to determine if the interaction is going well or poorly. Many times these cues are very subtle and come in the form of micro-expressions. In this project, we aim to build a system capable of recognizing micro-expressions from images and video in order to improve social interactions for those with visual impairments or on the Autism spectrum.

Facial Attribute Parsing

Facial attribute recognition suffers from several challenges including attribute localization. An attribute recognition system may be given a very large image of a face and then must determine if the person in the image is wearing earrings. The earrings (or lack there of) will take up a very small portion of the image compared to other attributes like hair color. We shift the focus of attribute recognition from the problem of identification to one of segmentation. Segmenting the face into its attribute parts allows for improved localization and recognition performance. We use an automated method to provide weak segmentation labels for this task, relying on facial landmark analysis, rather than human labelers.

First Impression Recognition

First impressions consist of three social traits: trustworthiness, attractiveness and competence. We form first impressions within 100ms of meeting someone new. That's less time than it takes to blink! We are also capable of forming first impressions from just images. These first impressions are rooted in evolution, but are often times incorrect. While it has been shown that our first impressions are often wrong, they have very serious real-world consequences. They can affect your chances of getting a job, of being released from prison, and finding a partner. We aim to build a system capable of recognizing first impressions from images in order to better understand what underlying physical characteristics give rise to these judgements.

Facial Attribute Recognition

Recognizing facial attribute, or human-describable features of faces, is an extremely challenging problem with applications in face verification and human computer interaction. As a relatively young problem in the field of computer vision, there are only two large-scale datasets available for attribute recognition, collected from public figures. These datasets suffer from label imbalance, as well as issues of bias and data and label noise. We address these issues by introducing new learning mechanisms for deep learning models.