Standoff Biometric Identification: Face Recognition

Home / Technical Inquiries / Notable

Retweet Share on Facebook Share on Linkedin reddit
Inquiry_Default_6

Posted on October 4, 2018 | Completed on September 30, 2018 | By: Thirimachos Bourlai, Ph.D., Scott E. Armistead

What are the current technologies, solutions, and limitations of standoff biometric identification (specifically in face recognition), and what U.S. government programs are active in relation to this topic?

 

The Defense Systems Information Analysis Center (DSIAC) received a technical inquiry requesting information on stand-off biometrics identification technologies (i.e., how a set of the most common biometric modalities such as face, fingerprint, iris, and gait can be used to identify individuals under different environments and conditions). The inquirer also requested information on U.S. government programs in this area and subject matter experts (SMEs) that could potentially support a bilateral workshop on challenges and solutions. DSIAC staff compiled information on biometric research, biometrics research SMEs, and current West Virginia University biometrics research programs into a report that was delivered to the inquirer.

 

1.0  Introduction

The Defense Systems Information Analysis Center (DSIAC) contacted West Virginia University (WVU) to identify subject matter experts (SMEs) who could potentially help respond to the inquirer’s request.  WVU identified Thirimachos Bourlai, Ph.D., an associate professor of the Multispectral Imagery Lab (MILab) and an SME in biometrics. Dr. Bourlai provided a brief overview of different standoff biometric technologies for potential application to military sensing, scene understanding, and human identification in challenging environments.

 

2.0  Biometrics SMEs

Dr. Bourlai identified organizations and SMEs who are leaders in biometrics research.

 

2.1  Clarkson University:  Stephanie Schuckers, Ph.D.

Dr. Schuckers is a Paynter-Krigman endowed professor in engineering science, the Director of the Biomedical Signal Analysis Lab, and the Director for the Center for Identification Technology Research (CITer), a National Science Foundation Industry/University Cooperative Research Center.  Dr. Schuckers is the advising/planning committee member for the Federal Identity Forum and Exposition and the advising board member/committee chair for the Biometrics Institute [1].

 

2.2  Michigan State University:  Anil Jain, Ph.D.

Dr. Jain’s research focus is in pattern recognition, computer vision, and biometric recognition.  He has served as a member of the National Academies of Sciences, Engineering, and Medicine Whither Biometrics and Improvised Explosive Devices panels and is a co-organizer of the Program on Forensics at the Statistical and Mathematical Sciences Institute. He was appointed as a member of the Defense Science Board, the Forensic Science Standards Board, and the American Association for the Advancement of Science Latent Fingerprint Working Group. Dr. Jain has numerous patents in fingerprint recognition and video surveillance, as well as licensed technologies related to forensics and law enforcement (tattoo-identification [ID] for matching tattoo images; AltFinger-ID for detecting whether a fingerprint image has been altered; FaceSketch-ID for matching facial sketches to mugshot images; and Face-Search for locating a person of interest in databases with hundreds of millions of faces). He has also published multiple books related to algorithms for biometrics analysis and biometrics identification, theory, and applications [2].

 

2.3  University of Texas at Dallas (UTD):  Alice O’Toole, Ph.D.

Dr. O’Toole is a professor at UTD’s School of Behavioral and Brain Sciences and holds the Aage and Margareta Moller Endowed Chair position. She focuses her research on facial recognition by humans (human perception and memory of faces) and machines and neural processing of faces and bodies.  She currently serves as an associate editor for Psychological Science and the British Journal of Psychology and has been named a fellow by the American Psychological Association. Dr. O’Toole has been the recipient of an Alexander von Humboldt Research Fellowship and has received research funding from the National Institute of Mental Health, the Defense Advanced Research Projects Agency (DARPA), the Intelligence Advanced Research Projects Agency, and the National Institute of Justice [3].

 

2.4  National Institute of Standards and Technology (NIST):  Patrick Grother, Computer Scientist

Mr. Grother is responsible for biometric algorithm evaluation and biometric performance testing standardization. He has authored nearly 100 research publications in biometrics and imaging systems used for collecting biometrics data. His research focuses in areas related to iris camera standards, performance or iris recognition algorithms, iris imaging, face recognition, tattoo recognition, automated gender classification, automated age estimation, fingerprint matching, identification from body shape, and other biometrics quality measures and error analysis.  He leads Face Recognition Vendor Tests, Face-In-Video Evaluation, and Iris Recognition Exchange projects. Mr. Grother also co-chairs NIST’s International Biometrics Performance Conference and is the editor of several International Standards Organization standards [4].

 

2.5  NIST Information Technology Laboratory:  P. Jonathon Phillips, Ph.D.

Dr. Phillips is an electronic engineer and a leading researcher in computer vision, face recognition, biometrics, and forensics, with over 100 peer-reviewed papers on face recognition, computer vision, biometrics, psychology, forensics, statistics, and neuroscience. He was also an associate editor for IEEE Transactions on Pattern Analysis and Machine Intelligence. He is an Institute of Electrical and Electronics Engineers (IEEE) Fellow and an International Association of Pattern Recognition (IAPR) Fellow. Dr. Phillips has also served as a DARPA program manager and served on the Executive Office of the President National Science and Technology Council Subcommittee on Biometrics. A hallmark of his research is collaborating with researchers from related fields; this has contributed to an improved understanding of biometric algorithm performance and the relationship between human and algorithm face recognition accuracy [5].

Examples of work at NIST on facial recognition technology can be found in the following articles:

 

2.6  ARL:  Dr. Riggan, Dr. Short, and Dr. Hu

The article “Army Develops Face Recognition Technology That Works in the Dark” provides information on Dr. Benjamin S. Riggan, Dr. Nathaniel J. Short, and Dr. Shuown “Sean” Hu’s work in the development of an artificial intelligence and machine learning technique that produces a visible face image from a thermal image of a person’s face that was captured in low-light or nighttime conditions. This development could lead to enhanced real-time biometrics and post-mission forensic analysis for covert nighttime operations. For more information, see the following link:  https://www.arl.army.mil/www/default.cfm?article=3199 [9].

 

2.7  IDEMIA National Security Solutions:  James Loudermilk, Senior Director, Innovation and Customer Solutions

Mr. Loudermilk served as the senior technologist for a major U.S. federal agency; represented the agency with the National Science Foundation Center for Identification Technology Research; and co-chaired the Interagency Biometrics and Identity Forum and its predecessor, the White House Subcommittee on Biometrics and Identity Management.  He was the chief engineer for developing a major U.S. federal agency biometric identification system [10].

IDEMIA focuses on augmented identity technologies and solutions for national security, law enforcement, and intelligence activities related to large-scale biometrics, video analytics, frictionless security solutions (contactless fingerprint access, three-dimensional [3-D], face imaging, nonintrusive iris at a distance recognition, etc.); Internet of Things (IoT)/connected objects (IoT security chain management, embedded subscriber identity module to secure IoT infrastructures, etc.); and credentialing (critical national agency support, large-scale facial databases, iris recognition software for tactical use, etc.).  Augmented identity was the primary technology provided for the Next Generation Identification program development for the Federal Bureau of Investigation (FBI) [10].

The article “Facial Recognition:  A Strategic Imperative for National Security” by B. S. Swann and J. Loudermilk provides information on face recognition. For more information, see the following link:  https://www.biometricupdate.com/201906/facial-recognition-a-strategic-imperative-for-national-security [11].

 

3.0  Biometric Recognition Systems in Military Environments

The need to quickly and accurately identify humans in challenging environments (i.e., indoors vs. outdoors, close vs. long distance, urban vs. rural settings, and day vs. nighttime) is increasing every day. There are different applications to consider, including the following:

  • Secure access by only legitimate users.
  • Mobilize platforms or other devices used in law enforcement, military, or medical-related applications.
  • Detect and identify potentially hostile persons, persons of interest, and high-value targets in the wild.

In all cases, biometrics systems are being used independently or in combination with other security measures to establish human identity. Biometrics utilize physiological (i.e., face, fingerprints, and iris) or behavioral (keystroke dynamics) traits. Different applications and security levels require the usage of a single biometric trait (unimodal) or a combination of traits (multimodal), such as face and iris, fingerprints and veins, face and voice, etc. In an access control scenario, a legitimate user is expected to enroll into the biometric system that she/he will be trying to access. However, in detecting and identifying potentially hostile individuals, the persons of interest may or may not be enrolled in the biometric system(s) at hand.

When focusing more on military-related environments and applications, a standoff biometrics capability includes the collection, storage, processing, and return of information to tactical units before they arrive on-site. Ideally, this can be achieved by using various state-of-the-art, field-tested, single- or multimodal biometric systems at a distance, including the most commonly used ones such as face-frontal, iris, ear or face-profile, and gait-based. An efficient standoff biometric capability enables tactical forces to maintain a proactive posture, maximizes the chances of capturing targeted individuals, reduces the risk to friendly forces, and supports follow-on mission objectives.

 

4.0  Work in Biometrics-Based Detection and Identification Programs

WVU (MILab) has been working with various partners and collaborators (i.e., CITeR, National Science Foundation, etc.) on the following programs related to various aspects of biometrics-based detection and recognition, including the following:

  • Long-range forensic human identification using face, iris, and soft biometrics.
  • Face recognition in the wild using cell phone devices for access control.
  • Matching passive infrared (IR) against multispectral face images using short-, mid-, and long-range imaging sensors.
  • Data collection of various biometrics modalities (face, iris, fingerprint, and gait) at controlled and challenging conditions.
  • Face identification behind tinted glass.
  • Incorporating biological models for iris authentication in mobile environments.
  • Quality of face recognition systems.
  • Document (passport and driver’s licenses) image restoration for improved live to document face matching.
  • Biometrics meets forensic: heterogeneous matching.
  • Partial face matching across the IR band.
  • Tattoo and face recognition from multispectral imagery.
  • Cross-device and cross-distance face recognition (FR) using cell phones.
  • Facial and subfacial regions in thermal image sequences.
  • Toward low-cost, deployable thermal biometrics.
  • Sample size estimation and stratification for large biometrics databases.
  • A preprocessing methodology for handling passport facial photos.
  • Ascertaining identity within human networks in night environments.
  • Face recognition and liveness detection in the middle wave infrared (MWIR) band.

 

5.0  Published Research: Biometric Recognition Systems at a Distance

This section provides a brief discussion on the different conditions used to collect biometrics [12] and descriptions of the latest publications focused on single biometric traits when collected at a distance. While different traits (e.g., face, iris, fingerprints, and gait) are discussed in this report, the focus is on face recognition.

 

5.1  Biometrics Collections Using Different Modalities

In the Handbook of Remote Biometrics for Surveillance and Security, Tistarelli et al. [13] state that when performing biometrics collection, there are three main categories, including the following:

  1. Contact-Based Biometrics
    • Contact-based biometrics require physical interaction between the user and biometrics sensor (e.g., the touch-based fingerprint sensors used in most modern cell phones).
  2. Contactless or Close-Range Biometrics Data Collection
    • Contactless data collection can be considered the category where the biometric trait is captured from about 2 cm to 1 m from the sensor. In this category, depending on the biometric trait, there is always a certain degree of cooperation between the subject and the sensor to capture biometric trait samples of sufficient quality (e.g., when extracting fingerprints [14] or the face images [15] when using the embedded camera of mobile phones).
  3. At-a-Distance (Mid- to Long-Range) Biometrics
    • When collecting biometric traits at a distance (also known as remote or standoff biometrics), the stand-off distance ranges from about 1 m to several hundred meters away, depending on the spectrum of the imaging sensor [16] and its capability [17].

Depending on the stand-off distance, the collection is focused on an individual’s face [18], gait [19], iris [20], fingerprint [21], and other attributes (e.g., tattoos [22, 23]) that may require the individual’s cooperation [13].

 

5.2  Face Recognition at a Distance

This section contains a discussion on one of the aforementioned biometric traits when collected at a distance, namely face recognition and its applications.

FR systems are gaining interest because the face has several advantages over other biometric traits (i.e., it is nonintrusive, understandable, and can be captured in a covert manner at standoff distances); however, there are various challenges when discussing the problem of face recognition at a distance. Depending on the application, the focus can be in the processing of single-spectral, multispectral, or hyperspectral face images (when captured under controlled or uncontrolled environments using a variety of imaging sensors). Such sensors range from the state-of-the-art visible (digital single lens reflex and mirrorless) and IR (near-IR [NIR], short-wave IR [SWIR], MWIR, and long-wave IR [LWIR]) imaging sensors, to using red, green, blue, and depth (RGB-D) and mobile phone image sensors. There are several challenges, including the following:

  • Document-to-live face matching
  • Faces in the wild using cell phones
  • Scalability studies (using small- to large-scale datasets)
  • Longitudinal studies (FR over time)
  • Impact of makeup, scars marks, and tattoos on FR performance
  • Long-range FR at night
  • FR behind tinted glass
  • Liveness detection
  • Spoofing

 

5.3  Additional References

The following publications and projects contain interesting information relevant to the inquiry:

  • “Collection of Multi-Spectral Biometric Data for Cross-Spectral Identification Applications” by J. Dawson et al. [12].
    • In this report, best practice collection methodologies were developed to compile large-scale datasets of both visible and SWIR face images. All aspects of data collection are provided, from Institutional Review Board preparation through data post-processing, along with instrumentation layouts for indoor and outdoor live capture setups. Details of past collections performed at WVU to compile multispectral biometric datasets (e.g., age, gender, and ethnicity of the subject populations) are included. Insight is also given on the impact of collection parameters on the general quality of images collected, as well as on how these parameters impact design decisions at the algorithmic development level.
  • “Hyperspectral Face Databases for Facial Recognition Research” by W. Cho et al. [24].
    • To overcome several major challenges specific to current FR systems, the usage of spectral imaging is important. In this work, the authors review four publicly available hyperspectral face databases (i.e., Carnegie Mellon University, Hong Kong Polytechnic University Hyperspectral Face Database, IRIS-M, and Stanford) toward providing information on the key points of each of the considered databases.
  • “MWIR-to-Visible and LWIR-to-Visible Face Recognition Using Partial Least Squares and Dictionary Learning” by S. Hu et al. [25].
    • In this work, different thermal-to-visible face recognition algorithms are discussed using a partial least squares-based approach and a dictionary learning-based FR approach. Different FR results are presented on an extensive multimodal face dataset containing facial imagery acquired under different experimental conditions. Furthermore, the authors discuss key findings and implications for MWIR-to-visible and LWIR-to-visible face recognition.
  • “Local Operators and Measures for Heterogeneous Face Recognition” by Z. Cao et al. [26].
    • This work summarizes the usage of local operators for heterogeneous FR. It analyses performance of individual operators (generalized local binary patterns, histograms of oriented gradients, etc.) and demonstrates performance of composite operators (Gabor ordinal measures, composite multilobe descriptors, etc.). The cross-matching performance of heterogeneous face images is demonstrated on two data sets composed of active IR and visible light face images; short and long standoff distances are considered.
  • “Assessment of Facial Recognition System Performance in Realistic Operating Environments” by K. R. Leonard [27].
    • This work introduces a methodology to explore the sensitivities of an FR system to blur, noise, and turbulence effects. The author uses a government-owned, open-source FR algorithm. System performance is evaluated under different optical blurs, sensor noises, and turbulence conditions. The ramifications of these results on the design of long-range FR systems are also discussed.
  • “Understanding Thermal Face Detection: Challenges and Evaluation” by J. Agrawal et al. [28].
    • In this work, the authors provide a thorough understanding of challenges in thermal face detection, as well as an experimental evaluation of traditional approaches. They discuss approaches designed specifically for aiding occluded or disguised thermal face detection. The results suggest that while thermal face detection in semicontrolled environments is relatively easy, occlusion and disguise are challenges that require further attention.
  • “Face Recognition Systems Under Spoofing Attacks” by I. Chingovska et al. [29].
    • In this work, the authors provide an overview of spoofing attacks and spoofing counter-measures for FR systems, with a focus on visual spectrum systems in two-dimensional and 3-D, as well as NIR and multispectral systems. The authors also provide a systematic overview of the existing anti-spoofing techniques, with an analysis of their advantages and limitations and prospective for future work.
  • “On the Effects of Image Alterations on Face Recognition Accuracy” by M. Ferrara et al. [30].
    • In this work, the authors discuss that when face images are captured under desirable conditions, “some intentional or unintentional face image alterations can significantly affect the recognition performance. In scenarios where the user template is created from printed photographs rather than from images acquired live during enrollment (e.g., identity documents), digital image alterations can severely affect the recognition results.”
  • “Document to Live Facial Identification” by A. D. Clark et al. [31].
    • In this work, the authors discuss the factors impacting the quality of degraded face photos from ID documents; these include hairstyle, pose, and expression variations; lamination; and security watermarks. The authors focus on investigating a set of methodological approaches to overcome most of the aforementioned limitations and achieve a high identification rate. They incorporate a combination of preprocessing and heterogeneous face-matching techniques, where comparisons are made between the original (degraded) photo, the restored photo, and the high-quality photo (mugshots). The proposed restoration approaches can be directly applied to operational scenarios that include border-crossing stations and various transit centers.
  • “Face Recognition in Challenging Environments: An Experimental and Reproducible Research Survey” by M. Günther et al. [32].
    • In this work, the authors discuss FR in mobile and other challenging environments, where both still images and video sequences are examined. They provide an experimental study of one commercial off-the-shelf and four recent open-source FR algorithms and supply an easily extensible, open-source toolbox to rerun all the experiments. The toolbox provides a detailed description on how to regenerate the results, including modelling techniques, evaluation protocols, and metrics used.
  • “Face Recognition with RGB-D Images Using Kinect” by G. Goswami et al. [33].
    • In this work, the authors discuss existing RGB-D face recognition algorithms and present a state-of-art algorithm based on extracting discriminatory features using entropy and saliency from RGB-D images. They also present an overview of available RGB-D face datasets, along with experimental results and analysis, to understand the various facets of RGB-D face recognition.
  • “Blending 2D and 3D Face Recognition” by M. Tistarelli et al. [34].
    • In this work, the authors highlight the advantages and disadvantages of 2D- and 3D-based face recognition algorithms. They also explore the advantages of blending 2D and 3D data-based techniques, also proposing a novel approach for a fast and robust matching. Several experimental results, obtained from publicly available datasets, demonstrate the effectiveness of the proposed approach.
  • “Exploiting Score Distributions for Biometric Applications” by P. Moutafis and I. A. Kakadiaris [35].
    • In this work, the authors introduce the concept of score normalization and highlight its importance in improving recognition accuracy. They also answer the questions of why score normalization effectively utilizes score distributions and why such methods have not gained popularity in the research community.
  • “Multispectral Ocular Biometrics” by S. G. Crihalmeanu and A. A. Ross [36].
    • This work discusses the use of multispectral imaging to perform bimodal ocular recognition where the eye region of the face is used for recognizing individuals. The authors demonstrate the improvement of iris recognition performance by fusing the iris and scleral patterns in nonfrontal images of the eye.
  • “Multi-Spectral Face Recognition: Identification of People in Difficult Environments” by T. Bourlai and B. Cukic [37].
    • In this paper, the authors study the problems of intraspectral and cross-spectral FR in homogeneous and heterogeneous environments. Specifically, they investigate the advantages and limitations of matching (i) SWIR face images against visible images under controlled or uncontrolled conditions, (ii) MWIR against MWIR or visible images captured under controlled conditions, and (iii) intradistance NIR against NIR images and cross-distance, cross-spectral NIR against visible images.
  • “On Designing SWIR to Visible Face Matching Algorithms” by C. Whitelam and T. Bourlai [38].
    • In this work, the authors discuss the problem of matching visible gallery face images to probe face images captured under different wavelengths of the electromagnetic spectrum. Specifically, they discuss the problem of matching visible images to images taken in the SWIR spectrum, more specifically, the 1550-nm band. There are many benefits to using the SWIR spectrum for face recognition, including covert capturing in nighttime environments, as well as imaging through certain environmental conditions such as fog and smoke. However, due to the fact that the moisture in the skin tends to absorb the 1550-nm wavelength, all subjects appear to have dark or black skin tone. Because of the stark contrast between 1550-nm and visible face images, standard face recognition protocols fail to accurately match images captured using sensors operating on different bands.
  • “Design and Evaluation of Photometric Image Quality Measures for Effective Face Recognition” by A. Abaza et al. [39].
    • In this work, the authors discuss that the performance of an automated FR system can be significantly influenced by face image quality. Designing effective image quality index is necessary in order to provide real-time feedback for reducing the number of poor quality face images acquired during enrollment and authentication, thereby improving matching performance. The authors first evaluate techniques that can measure image quality factors such as contrast, brightness, sharpness, focus, and illumination in the context of face recognition. Second, they determine whether using a combination of techniques for measuring each quality factor is more beneficial, in terms of FR performance, than using a single, independent technique. Third, they propose a new face image quality index that combines multiple quality measures and classifies a face image based on this index.
  • “Automatic Face Image Quality Prediction” by L. Best-Rowden and A. K. Jain [40].
    • In this document, the authors propose (and compare) two methods for automatic face image quality based on target face quality values from (i) human assessments of face image quality (matcher-independent), and (ii) quality values computed from similarity scores (matcher-dependent).”  This article was selected among 10 Elsevier journals as one of the Best 26 Journal Papers for 2015. It was selected to appear in “Elsevier – Virtual Special Issues Celebrating the Breadth of Biometrics Research 2015.”
  • “Face Recognition in the SWIR Band When Using Single Sensor Multi-Wavelength Imaging Systems” by N. Narang and T. Bourlai [41].
    • In this paper, the authors’ contributions include the following: “First, a SWIR database is collected when using our developed SSMW system under the following scenarios—i.e., multi-wavelength (MW) multi-pose images were captured when the camera was focused at either 1150, 1350, or 1550 nm. Second, an automated quality-based score level fusion scheme is proposed for the classification of input MW images. Third, a weighted quality-based score level fusion scheme is proposed for the automated classification of full frontal (FF) vs. non-frontal (NFF) face images. Fourth, a set of experiments is performed, indicating our proposed algorithms for the classification of (i) multi-wavelength images, and (ii) FF vs. NFF face images are beneficial when designing different steps of multi-spectral face recognition (FR) systems, including face detection, eye detection, and face recognition.”
  • “Bridging the Spectral Gap Using Image Synthesis: A Study on Matching Visible to Passive Infrared Face Images” by N. Osia and T. Bourlai [42].
    • The four primary contributions of this work include the following: “First, we assemble a database of frontal face images composed of paired VIS-MWIR and VIS-LWIR face images (using different methods for pre-processing and registration). Second, we formulate an image synthesis framework and post-synthesis restoration methodology to improve face recognition accuracy. Third, we explore cohort-specific matching (per gender) instead of blind-based matching (when all images in the gallery are matched against all in the probe set). Finally, by conducting an extensive experimental study, we establish that the proposed scheme increases system performance in terms of rank-1 identification rate. Experimental results suggest that matching visible images against images acquired with passive infrared spectrum, and vice-versa, are feasible with promising results.”
  • “On Designing Practical Long Range Near Infrared-Based Face Recognition Systems” by T. Bourlai et al. [43].
    • In this document, the authors discuss the challenges in designing a practical NIR FR system and, more specifically, study the problems of intra-spectral, cross-spectral, i.e., VIS–NIR, intra-distance, and cross-distance NIR FR, in indoors, outdoors, daytime and nighttime environments.
  • “Face Recognition Outside the Visible Spectrum” by T. Bourlai and L. Hornak [44].
    • The focus of this work is “to expose readers to a plethora of FR advancements when operating outside the visible spectrum and our perspective on where this field is going.” The discussion includes the latest studies in the ultraviolet (UV) (100–400 nm), active IR band (0.7–2.5 μm), and the passive IR band (more specifically, the mid-wave IR [3–5 μm] and long-wave IR [8–14 μm] bands). The discussion also includes research work that involves same-spectral and cross-spectral matching scenarios, where face images (probes) are first acquired under controlled or difficult conditions, including dealing with uncooperative subjects and adverse environments. The paper also suggests new avenues of inquiry that are needed in the context of current work in multispectral FR systems.
  • “On Matching Visible to Passive Infrared Face Images Using Image Synthesis and Denoising” and “Bridging the Spectral Gap using Image Synthesis: A Study on Matching Visible to Passive Infrared Face Images” by N. Osia and T. Bourlai [45, 46].
    • In this work, the authors discuss an approach that bridges the gap between the visible and IR band of the electromagnetic spectrum, namely the MWIR (3–5 μm) and LWIR (8–14 μm) bands. Specifically, “we investigate the benefits and limitations of using synthesized visible face images from thermal and vice versa in cross-spectral face recognition systems when utilizing canonical correlation analysis and manifold learning dimensionality reduction.”
  • “Learning Deep Features for Hierarchical Classification of Mobile Phone Face Datasets in Heterogeneous Environments” by N. Narang et al. [47].
    • In this work, the authors collected a unique multisensor database (using Samsung S4 Zoom, Nokia 1020, iPhone 5S, and Samsung S5 phones) containing face images indoors, outdoors, with yaw angle from -90° to +90°, and at two different distances (i.e., 1 and 10 m). The authors propose a convolutional neural network (CNN)-based, scenario-dependent, and sensor (mobile device)-adaptable hierarchical classification framework. The proposed framework is designed to automatically categorize face data captured under various challenging conditions before the FR algorithms (pre-processing, feature extraction, and matching) are used.
  • “Gender and Ethnicity Classification Using Deep Learning in Heterogeneous Face Recognition” by N. Narang and T. Bourlai. The following is an excerpt from the document’s abstract [48]:

“Although automated classification of soft biometric traits in terms of gender, ethnicity, and age is a well-studied problem with a history of more than three decades, it is still far from being considered a solved problem for the case of difficult exposure conditions, such as during nighttime, in environments with unconstrained lighting, or at large distances from the camera.”

    • In this work, the authors investigate the advantages and limitations of the automated deep learning-based classification of soft biometric traits in terms of gender and ethnicity in NIR long-range, nighttime face images. The impact of soft biometric traits in terms of gender and ethnicity is explored for the purpose of improving cross-spectral face recognition (FR) performance.
  • “Can We Match Ultraviolet Face Images Against Their Visible Counterparts?” by N. Narang and T. Bourlai [49].
    • In this work, the authors investigate the advantages and limitations of the heterogeneous problem of matching ultraviolet (from 100 nm to 400 nm in wavelength) or UV, face images against their visible (VIS) counterparts, when all face images are captured under controlled conditions.
  • “Longitudinal Study of Automatic Face Recognition” by L. Best-Rowden and A. K. Jain [50].
    • In this work, the authors investigate the permanence property of FR by addressing the following: Does face recognition’s ability of state-of-the-art systems degrade with elapsed time between enrolled and query face images? If so, what is the rate of decline with respect to the elapsed time? Longitudinal analysis shows that despite decreasing genuine scores, 99% of subjects can still be recognized at 0.01% FAR up to approximately 6 years elapsed time and age, sex, and race only marginally influence these trends.
  • “Spoofing Faces Using Makeup: An Investigative Study” by C. Chen et al. [51].
    • In this work, the authors focus on the impact of makeup on FR, as “makeup can be used to alter the facial appearance of a person.” The authors analyze the potential of using makeup for spoofing an identity, where an individual may attempt to impersonate another person’s facial appearance… Experiments suggest that automated face matchers are vulnerable to makeup-induced spoofing and that the success of spoofing is impacted by the appearance of the impersonator’s face and the target face being spoofed. Further, an identification experiment is conducted to show that the spoofed faces are successfully matched at better ranks after the application of makeup.
  • “SWIR Imaging for Facial Image Capture Through Tinted Materials” by J. Ice et al. [52].
    • In this work, the authors discuss the use of SWIR technology that introduces challenges to facial recognition when comparing cross-spectrally from a visible gallery to images captured in the SWIR. The challenges of SWIR facial recognition are compounded by the presence of tinted materials in the imaging path due to varying material types, lighting conditions, and viewing angle.
    • The authors also discuss material and optical characterization efforts undertaken to understand the effects of temperature, interior and exterior light sources, and viewing angle on the quality of facial images captured through tinted materials. Temperature vs. spectrum curves are shown for tinted architectural, automotive, and sunglass materials over the range of -10 to 55 °C.  The results of imaging under various permutations of interior and exterior lighting, along with viewing angle, are used to evaluate the efficacy of eye detection for cross-spectral facial recognition under these conditions.
  • “Overview of Polarimetric Thermal Imaging for Biometrics” by S. Hu et al. [53].
    • In this report, the authors focus on polarimetric thermal imaging, which represents an ideal modality for acquiring the naturally emitted thermal radiation from the human face, providing additional geometric and textural details not available in conventional thermal imagery. One of the main challenges lies in matching the acquired polarimetric thermal facial signature to gallery databases containing only visible facial signature, for interoperability with existing government biometric repositories.
    • This paper discusses approaches and algorithms to exploit polarization information, as represented by the Stokes vectors, through feature extraction and non-linear regression to enable polarimetric thermal-to-visible face recognition.
  • “Introduction and Fundamental Concepts” in Biometric Recognition: Challenges and Opportunities, edited by J. N. Pato and L. I. Millet [54].
    • This document provides information on biometrics challenges and opportunities.
  • “New Trends in Biometrics (March 2018) With Isabelle Moeller From the Biometrics Institute” by Gemalto [55].
    • This article discusses trends in biometrics.
  • “Securing Social Identity in Mobile Platforms” in Technologies for Social Network Analysis and Identity Management by T. Bourlai et al. [56].
  • Surveillance in Action by P. Karampelas and T. Bourlai [57]. The following are relevant chapters in this book:
    • “Video-Based Human Respiratory Wavelet Extraction and Identity Recognition” by X. Yang and T. Bourlai [58].
    • “A Study on Human Recognition Using Auricle and Side View Face Images” by S. El-Naggar et al. [59].
    • “Facial Surveillance and Recognition in the Passive Infrared Bands” by N. Osia et al. [60].
    • “Deep Feature Learning for Classification When Using Single Sensor Multi-Wavelength Based Facial Recognition Systems in SWIR Band” by N. Narang and T. Bourlai [61].
  • Face Recognition Across the Imaging Spectrum, edited by T. Bourlai [18]. The following is a relevant chapter in this book:
    • “Collection of Multi-Spectral Biometric Data for Cross-Spectral Identification Applications” by J. Dawson et al. [12].
  • “Social Signals of Deception and Dishonesty” by J. K. Burgoon et al. [62].

 

6.0  Other Resources

The following government offices, conferences, and academia may also provide information on the inquiry topic.

  • Department of Homeland Security (DHS) Biometrics [63].
    • The following is an excerpt from “Biometrics” by DHS [63]:

“Biometrics are unique physical characteristics, such as fingerprints, that can be used for automated recognition. At the Department of Homeland Security, biometrics are used to detect and prevent illegal entry into the U.S., grant and administer proper immigration benefits, vetting and credentialing, facilitating legitimate travel and trade, enforcing federal laws, and enabling verification for visa applications to the U.S.”

  • FBI Biometric Center of Excellence (BCOE) [64].
  • IEEE Biometrics Council [65].
  • IEEE Biometrics Compendium [66].
  • Biometrics Institute [67].
  • Federal Identity Forum and Expo [68].
  • Biometrics for Government and Law Enforcement conference [69].

 

References

[1] Clarkson University. “Stephanie Schuckers.” https://www.clarkson.edu/people/stephanie-schuckers, accessed September 2018.

[2] Michigan State University Department of Computer Science and Engineering. “Anil K. Jain.” https://www.cse.msu.edu/~jain/, accessed September 2018.

[3] University of Texas at Dallas School of Behavioral and Brian Sciences. “Alice J. O’Toole.” https://www.utdallas.edu/bbs/faculty/detail.php5?i=411, accessed September 2018.

[4] National Institute of Standards and Technology (NIST). “Patrick J. Grother.” https://www.nist.gov/people/patrick-j-grother, accessed September 2018.

[5] NIST. “P. Jonathon Phillips.” https://www.nist.gov/people/p-jonathon-phillips, accessed September 2018.

[6] Leonard, M. “How NIST Helps Facial Recognition Make Better Matches.” GCN, https://gcn.com/articles/2018/04/05/nist-facial-recognition.aspx, 5 April 2018.

[7] NIST. “NIST Study Shows Face Recognition Experts Perform Better With AI as Partner.” https://www.nist.gov/news-events/news/2018/05/nist-study-shows-face-recognition-experts-perform-better-ai-partner, 29 May 2018.

[8] Phillips, J. P., A. N. Yates, Y. Hu, C. A. Hahn, E. Noyes, K. Jackson, J. G. Cavazos, G. Jeckeln, R. Ranjan, S. Sankaranarayanan, J.-C. Chen, C. D. Castillo, R. Chellappa, D. White, and A. J. O’Toole. “Face Recognition Accuracy of Forensic Examiners, Superrecognizers, and Face Recognition Algorithms.” Proceedings of the National Academy of Sciences, vol. 115, no. 24, pp. 6171-6176, https://www.pnas.org/content/115/24/6171, 12 June 2018.

[9] U.S. Army Research Laboratory. “Army Develops Face Recognition Technology That Works in the Dark.” https://www.arl.army.mil/www/default.cfm?article=3199, 16 April 2018.

[10] IDEMIA. https://www.idemia-nss.com/, accessed September 2018.

[11] Swann, B. S., and J. Loudermilk. “Facial Recognition:  A Strategic Imperative for National Security.” Biometric Update, https://www.biometricupdate.com/201906/facial-recognition-a-strategic-imperative-for-national-security, 3 June 2019.

[12] Dawson, J. M., S. C. Leffel, C. Whitelam, and T. Bourlai. “Collection of Multi-Spectral Biometric Data for Cross-Spectral Identification Applications.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[13] Tistarelli, M., S. Z. Li, and R. Chellappa (editors). Handbook of Remote Biometrics for Surveillance and Security. Berlin, Germany:  Springer, https://www.springer.com/us/book/9781848823846, 2009.

[14] Derawi, M. O., B. Yang, and C. Busch. “Fingerprint Recognition with Embedded Cameras on Mobile Phones” In Security and Privacy in Mobile Information and Communication Systems, vol. 94, Berlin, Germany:  Springer, 2012.

[15] Narang, N., M. Martin, D. Metaxas, and T. Bourlai. “Learning Deep Features for Hierarchical Classification of Mobile Phone Face Datasets in Heterogeneous Environments.” 12th IEEE International Conference on Automatic Face & Gesture Recognition, June 2017.

[16] Bourlai, T., and L. A. Hornak. “Face Recognition Outside the Visible Spectrum.” Image and Vision Computing, Amsterdam, Netherlands:  Elsevier, vol. 55, pp. 14-17, https://www.sciencedirect.com/science/article/pii/S0262885616300531, November 2016.

[17] Bourlai, T., J. VonDollen, N. Mavridis, and C. Kolanko. “Evaluating the Efficiency of a Nighttime, Middle-Range Infrared Sensor for Applications in Human Detection and Recognition.” Proceedings of SPIE, vol. 8355, https://www.spiedigitallibrary.org/conference-proceedings-of-spie/8355/83551B/Evaluating-the-efficiency-of-a-night-time-middle-range-infrared/10.1117/12.917831.short?SSO=1, April 2012.

[18] Bourlai, T. (editor). Face Recognition Across the Imaging Spectrum. Berlin, Germany:  Springer, https://www.springer.com/us/book/9783319284996, 2016.

[19] Nixon, M. S., T. Tan, and R. Chellappa (editors). Human Identification Based on Gait. International Series on Biometrics, Berlin, Germany:  Springer, https://play.google.com/store/books/details?id=tD42mXCGRGcC&source=productsearch&utm_source=HA_Desktop_US&utm_medium=SEM&utm_campaign=PLA&pcampaignid=MKTAD0930BO1&gclid=Cj0KCQjwlqLdBRCKARIsAPxTGaVD8iSAJPPDsp3LM1Fs7mZIKizNG_tn1hnzHNTR1BEgT3iqPrsoq3IaAuzuEALw_wcB&gclsrc=aw.ds&dclid=CO36jYXJ1N0CFUywswodj4kKMQ, 26 May 2010.

[20] Bowyer, K. W., and M. J. Burge (editors). Handbook of Iris Recognition. London, UK:  Springer-Verlag London, https://www.springer.com/us/book/9781447167822, 2016.

[21] Maltoni, D., D. Maio, A. Jain, and S. Prabhakar (editors). Handbook of Fingerprint Recognition. London, UK:  Springer-Verlag London, https://www.springer.com/us/book/9781848822535, 2009.

[22] Martin, M., and T. Bourlai. “Enhanced Tattoo Image Quality Assessment Through Multispectral Sensing.” In IEEE Sensors Letters, vol. 1, no. 6, pp. 1-4, https://ieeexplore.ieee.org/document/8094320, December 2017.

[23] Martin, M., J. Dawson, and T. Bourlai. “Large Scale Data Collection of Tattoo-Based Biometric Data from Social-Media Websites.” IEEE European Intelligence and Security Informatics Conference (EISIC), Athens, Greece, September 2017.

[24] Cho, W., A. Koschan, and M. A. Abidi. “Hyperspectral Face Databases for Facial Recognition Research.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[25] Hu, S., N. J. Short, P. K. Gurram, K. P. Gurton, and C. Reale. “MWIR-to-Visible and LWIR-to-Visible Face Recognition Using Partial Least Squares and Dictionary Learning.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[26] Cao, Z., N. A. Schmid, and T. Bourlai. “Local Operators and Measures for Heterogeneous Face Recognition.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[27] Leonard, K. R. “Assessment of Facial Recognition System Performance in Realistic Operating Environments.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[28] Agrawal, J., A. Pant, T. I. Dhamecha, R. Singh, and M. Vatsa. “Understanding Thermal Face Detection:  Challenges and Evaluation.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[29] Chingovska, I., A. Anjos, and S. Marcel. “Face Recognition Systems Under Spoofing Attacks.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[30] Ferrara, M., A. Franco, and D. Maltoni. “On the Effects of Image Alterations on Face Recognition Accuracy.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[31] Clark, A. D., C. Whitelam, and T. Bourlai. “Document to Live Facial Identification.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[32] Günther, M., L. E. Shafey, and S. Marcel. “Face Recognition in Challenging Environments:  An Experimental and Reproducible Research Survey.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[33] Goswami, G., M. Vatsa, and R. Singh. “Face Recognition with RGB-D Images Using Kinect.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[34] Tistarelli, M., M. Cadoni, A. Lagorio, and E. Grosso. “Blending 2D and 3D Face Recognition.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[35] Moutafis, P., and I. A. Kakadiaris. “Exploiting Score Distributions for Biometric Applications.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[36] Crihalmeanu, S. G., and A. A. Ross. “Multispectral Ocular Biometrics.” In Face Recognition Across the Imaging Spectrum, Berlin, Germany:  Springer, 2016.

[37] Bourlai, T., and B. Cukic. “Multi-Spectral Face Recognition:  Identification of People in Difficult Environments.” IEEE International Conference on Intelligence and Security Informatics, (ISI), Washington, https://ieeexplore.ieee.org/document/6284307, June 2012.

[38] Whitelam, C., and T. Bourlai. “On Designing SWIR to Visible Face Matching Algorithms.” Intel Technology Journal, Edition on Biometrics and Authentication, https://multispectral-imagery-lab.sandbox.wvu.edu/files/d/7cafa40e-c9fb-4a93-ad4e-a9e7ca31de09/intel.pdf, August 2014.

[39] Abaza, A., M. A. Harrison, T. Bourlai, and A. Ross. “Design and Evaluation of Photometric Image Quality Measures for Effective Face Recognition.” Journal (IEEE) IET Biometrics, https://multispectral-imagery-lab.sandbox.wvu.edu/files/d/1c1d8c02-77f3-494e-ab8e-eadbd0a6ffe2/iet.pdf, 2014.

[40] Best-Rowden, L., and A. K. Jain. “Automatic Face Image Quality Prediction.” https://arxiv.org/pdf/1706.09887.pdf, 2017.

[41] Narang, N., and T. Bourlai. “Face Recognition in the SWIR Band When Using Single Sensor Multi-Wavelength Imaging Systems.” Image and Vision Computing Journal, vol. 33, pp. 26-43, Amsterdam, Netherlands:  Elsevier, https://multispectral-imagery-lab.sandbox.wvu.edu/files/d/7f5dd529-cb39-45c9-9ef1-f2f0449ecc2d/elsevierimavis_neeru.pdf, January 2015.

[42] Osia, N., and T. Bourlai. “Bridging the Spectral Gap Using Image Synthesis:  A Study on Matching Visible to Passive Infrared Face Images.” Machine Vision and Applications (MVA), Berlin, Germany:  Springer, https://link.springer.com/article/10.1007/s00138-017-0855-1, 2017.

[43] Bourlai, T., N. Narang, and N. Mavridis. “On Designing Practical Long Range Near Infrared-Based Face Recognition Systems.” Image and Vision Computing, Amsterdam, Netherlands:  Elsevier, https://www.sciencedirect.com/science/article/pii/S0262885616300749, 2016.

[44] Bourlai, T., and L. Hornak. “Face Recognition Outside the Visible Spectrum.” Image and Vision Computing, Amsterdam, Netherlands:  Elsevier, https://www.sciencedirect.com/science/article/pii/S0262885616300531, 2016.

[45] Osia, N., and T. Bourlai. “On Matching Visible to Passive Infrared Face Images Using Image Synthesis & Denoising.” 12th IEEE International Conference on Automatic Face & Gesture Recognition, June 2017.

[46] Osia, N., and T. Bourlai. “Bridging the Spectral Gap using Image Synthesis: A Study on Matching Visible to Passive Infrared Face Images.” MVA, Berlin, Germany:  Springer, https://par.nsf.gov/servlets/purl/10053526, 2017.

[47] Narang, N., M. Martin, D. Metaxas, and T. Bourlai. “Learning Deep Features for Hierarchical Classification of Mobile Phone Face Datasets in Heterogeneous Environments.” 12th IEEE International Conference on Automatic Face & Gesture Recognition, https://ieeexplore.ieee.org/document/7961740, June 2017.

[48] Narang, N., and T. Bourlai. “Gender and Ethnicity Classification Using Deep Learning in Heterogeneous Face Recognition.” The 9th International Association for Pattern Recognition (IAPR) International Conference on Biometrics (ICB), Halmstad, Sweden, https://ieeexplore.ieee.org/document/7550082, 13-16 June 2016.

[49] Narang, N., and T. Bourlai. “Can We Match Ultraviolet Face Images Against Their Visible Counterparts?” SPIE Proceedings, Algorithms and Technologies for Multispectral, Hyperspectral, and Ultra-Spectral Imagery XXI, Baltimore, MD, http://community.wvu.edu/~thbourlai/publications/Bourlai%20Publications/2015/SPIEUV_Neeru.pdf, April 2015.

[50] Best-Rowden, L., and A. K. Jain. “Longitudinal Study of Automatic Face Recognition.” IEEE Trans Pattern Analysis, vol. 40, no. 1, pp. 148-162, https://www.ncbi.nlm.nih.gov/pubmed/28092523, January 2018.

[51] Chen, C., A. Dantcheva, T. Swearingen, and A. Ross. “Spoofing Faces Using Makeup:  An Investigative Study.” IEEE International Conference on Identity, Security and Behavior Analysis (ISBA), New Delhi, India, https://ieeexplore.ieee.org/document/7947686, 22-24 February 2017.

[52] Ice, J., N. Narang, C. Whitelam, N. Kalka, L. Hornak, J. Dawson, and T. Bourlai. “SWIR Imaging for Facial Image Capture Through Tinted Materials.” Proceedings of SPIE, Infrared Technology and Applications XXXVIII, vol. 8353, Baltimore, MD, https://pdfs.semanticscholar.org/0a12/2076cba19d413a2370c80fcee2b0a68839b7.pdf, April 2012.

[53] Hu, S., N. Short, K. Gurton, and B. Riggan. “Overview of Polarimetric Thermal Imaging for Biometrics.” SPIE Proceedings, Polarization:  Measurement, Analysis, and Remote Sensing XIII, vol. 10655, https://www.spiedigitallibrary.org/ conference-proceedings-of-spie/10655/1065502/Overview-of-polarimetric-thermal-imaging-for-biometrics/10.1117/12.2299761.short?SSO=1, April 2018.

[54] Pato, J. N., and L. I. Millett (editors). “Introduction and Fundamental Concepts.” In Biometric Recognition:  Challenges and Opportunities, National Research Council (U.S.) Whither Biometrics Committee, Washington DC:  National Academies Press, https://www.ncbi.nlm.nih.gov/books/NBK219892/, 2010.

[55] Gemalto. “New Trends in Biometrics (March 2018) With Isabelle Moeller From the Biometrics Institute.” https://www.gemalto.com/govt/biometrics/trends-in-biometrics, accessed September 2018.

[56] Bourlai, T., P. Karampelas, and V. Patel. “Securing Social Identity in Mobile Platforms.” Technologies for Social Network Analysis and Identity Management, Berlin, Germany:  Springer, to be published.

[57] Karampelas, P., and T. Bourlai (editors). Surveillance in Action. Advanced Sciences and Technologies for Security Applications Series, Berlin, Germany:  Springer, 2018.

[58] Yang, X., and T. Bourlai. “Video-Based Human Respiratory Wavelet Extraction and Identity Recognition.” In Surveillance in Action, Berlin, Germany:  Springer, 2018.

[59] El-Naggar, S., A. Abaza, and T. Bourlai. “A Study on Human Recognition Using Auricle and Side View Face Images.” In Surveillance in Action, Berlin, Germany:  Springer, 2018.

[60] Osia, N., T. Bourlai, and L. Hornak. “Facial Surveillance and Recognition in the Passive Infrared Bands.” In Surveillance in Action, Berlin, Germany:  Springer, 2018.

[61] Narang, N., and T. Bourlai. “Deep Feature Learning for Classification When Using Single Sensor Multi-Wavelength Based Facial Recognition Systems in SWIR Band.” In Surveillance in Action, Berlin, Germany:  Springer, 2018.

[62] Burgoon, J. K., D. Metaxas, T. Bourlai, and A. Elkins. “Social Signals of Deception and Dishonesty.” Cambridge, UK:  Cambridge University Press, 2017.

[63] Department of Homeland Security. “Biometrics.” https://www.dhs.gov/biometrics, 9 May 2019.

[64] Federal Bureau of Investigation. “Biometric Center of Excellence (BCOE).” https://www.fbi.gov/services/ cjis/fingerprints-and-other-biometrics/biometric-center-of-excellence, accessed June 2019.

[65] IEEE Biometrics Council. https://ieee-biometrics.org/, accessed June 2019.

[66] IEEE. “IEEE Biometrics Compendium.” https://www.ieee.org/publications/subscriptions/ products/biometrics-compendium.html, accessed June 2019.

[67] Biometrics Institute. https://www.biometricsinstitute.org/, accessed June 2019.

[68] AFCEA. “Federal Identity Forum and Expo.” https://events.afcea.org/FedID19/Public/ enter.aspx, accessed June 2019.

[69] Institute of Defense and Government Advancement. “13th Annual Biometrics for Government & Law Enforcement.” https://www.idga.org/events-biometricsevent, accessed June 2019.

Download Full Report

Want to find out more about this topic?

Request a FREE Technical Inquiry!

New Technical Inquiry

Focus Areas