Occlusion Face Matching

Many face recognition engines can be shown to work very well on controlled, laboratory style images. Examples include passport and mugshot photos. While this is not an uncommon occurance for the enrollment gallery of images, in most scenarios, the probe image is not so well behaved. One of the most common occurances that can throw off many face reocgnition tools is the prescence of occlusions on the face. These can include things such as sunglasses, scarves, hats, masks, etc. At the CyLab Biometrics Center, we have been developing many tools to be able to handle real world images in face recognition, including tools to handle oclusions in face matching.


One of our approaches to handling occlusions in face images is to preprocess the image in a way such that the resulting image is usable by a recognition engine. By utilizing domain knowledge available and taking advantage of the large face databases available, we develop models capabale of learning how to reconstruct the occluded portions of a face given an occlusion mask.


Another approach we take is to create a classifier that is capable of recognizing the person in question even in the prescence of these occlusions.



Robust shift-invariant biometric identification from partial face images


Robust shift-invariant biometric identification from partial face images

Savvides, M., Vijaya Kumar, B. V. K., and Khosla, P.

Proc. SPIE 5404, Biometric Technology for Human Identification, 124, August, 2004


author={Savvides, M. and Vijaya Kumar, B. V. K. and Khosla, P.},
journal={Biometric Technology for Human Identification, Proc. of SPIE},
title={Robust shift-invariant biometric identification from partial face images},


In this paper, we address how these issues can be dealt efficiently with advanced correlation filter designs. We report extensive set of results on the CMU pose, illumination and expressions (PIE) dataset where training filters are designed in two experiments: (1) the training gallery has 3 images from extreme illumination (2) the training gallery has 3 images from near-frontal illumination. In the testing phase however, we test both filters with the whole illumination variations while simultaneously cropping the test images to various sizes. The results show that the advanced correlation filter designs perform very well even with partial face images of unseen illumination variations including reduced-complexity correlation filters such as the Quad-Phase Minimum Average Correlation Energy (QP-MACE) filter that requires only 2 bits/frequency storage. MACE filters are designed to give sharp peaks when correlated with training samples and generalize to other samples from the same distruibution. This is achieved by minimizing the average correlation energy of the correlation surfaces in the frequency domain. This becomes equivalent to minimizing a quadratic term as shown below.















When adding in the peak constraints, the resulting optimization is a fairly simple convex optimization.






The closed form solution becomes




which is easy to compute given that the matrices being inverted are either low in dimension or diagonal.


Even in the prescence of all the occlusions, the resulting correlation surfaces still give sharp peaks where the face is centered. These techniques allow for accurate classification and localization in the presence of these degradations.