The ears have it in NIST’s facial recognition research
Nextgov/FCW - All Content ·
Trained forensic examiners can often identify someone after seeing only one or two facial features, and their preferred place to begin is surprisingly far from the eyes.
Back when I ran the Government Computer News testing lab, there was a stretch of years when biometrics was one of the hottest areas in federal technology.
My lab partner, Carlos Soto, was placed in charge of the category, in part because I thought he would do a good job with it. I helped him test many of the products, which meant we spent quite a bit of time recording one another’s voices, lifting fingerprints and gluing pieces of photographs onto our faces.
It was serious work. We were evaluating security products designed for federal agencies and trying to discover weaknesses before someone with less friendly intentions found them.
But it was also tremendously fun. Carlos and I were essentially being paid to star in our own extremely low-budget spy movie, except that the scanners, secured doors and access-control systems were real. I suspect quite a few people would have happily traded jobs with us.
We evaluated hand scanners, fingerprint readers, retina scanners, voice authentication products and facial recognition systems. In addition to measuring how well the products performed under normal conditions, we tried almost anything we could think of to fool them. The companies submitting equipment to the lab knew exactly what we were doing, and discovering a successful attack was always a thrill.
Voice recognition systems were often the easiest targets. Some required users to speak a particular passphrase, but a recording of the authorized person saying that phrase could sometimes supply both the expected voice and the correct words.
Hand scanners were much more difficult to fool. Fingerprint scanners were also challenging, although we occasionally succeeded by lifting a print left by a previous user.
Facial recognition inspired our most elaborate attacks. We would take a high-resolution photograph of an authorized user, print it in color and carefully cut it apart. We then attached individual pieces of the photograph over another person’s face and presented our creation to the scanner.
Simply cutting eyeholes in a photograph and wearing it like a mask never worked. The more successful technique required dividing the face into pieces and carefully positioning them over the corresponding features of the person attempting to gain access.
The result looked ridiculous, but some of the systems could be fooled by our improvised disguises about 10% of the time. Others rejected them consistently, and none of the techniques provided a dependable way to defeat every product.
They also shared one obvious weakness as an attack method. They worked only when nobody was watching.
A person walking toward a secure entrance with photographs of someone else’s face taped and glued to their own face would probably be stopped by any semi-alert guard, regardless of whether the scanner approved the match. The software might see a sufficiently accurate collection of facial measurements. A human would see someone who looked like he had lost a fight with a craft store.
When humans examine faces
Humans may also contribute a bit more than just common sense to facial identification. Some people develop an extraordinary ability to recognize identities from details that automated systems have traditionally overlooked.
New research from the National Institute of Standards and Technology examined how professional forensic facial specialists decide whether multiple photographs show the same person. Instead of giving participants an unobstructed face, researchers covered the probe image and allowed the specialists to reveal isolated features, such as the eyes, nose, mouth, forehead or ears.
The test included 27 facial identification specialists completing 24 trials each. In some trials, they selected which feature to uncover. In others, the researchers determined the order in which features appeared.
The specialists were remarkably efficient. In 396 of the 648 total trials, they reached a decision after seeing only one feature.
Their favorite feature was not the eyes, nose or mouth. It was the ears.
When specialists selected their own features and reached a correct decision after revealing only one, they had chosen an ear in more than 74% of those trials. Across both testing conditions, seeing an ear by itself produced a 93% probability of a correct answer.
Foreheads posted a slightly higher estimated success rate of 94%, but they were selected much less often, giving researchers far less evidence and a much wider range of statistical uncertainty. Ears combined frequent use with consistently strong performance.
The complete NIST-led study, published in PLOS One , also found that the specialists adapted when their preferred feature was unavailable or unhelpful. They did not blindly follow a single checklist. They evaluated the visible information and decided whether they needed to uncover another part of the face.
That adaptability may be as important as their preference for ears. A biometric system follows its programming. A trained human can recognize that the normally useful feature in a particular photograph is obscured, distorted or simply too similar to the same feature on another person.
The study did not directly compare human specialists with modern facial recognition software, so it does not establish that one is universally better. Machines can rapidly compare images against enormous databases, something no human could reasonably attempt. The research instead shows that trained people may approach identification differently and extract useful evidence from features that automated systems have not always emphasized.
During our lab testing, facial recognition products seldom concentrated heavily on ears. That was understandable. Hair, hats, camera angles and image cropping can conceal one or both ears, making them unreliable inputs for a system that must operate consistently.
When ears are visible, however, their complicated contours can reveal a great deal.
The Cold War’s giant ear
American intelligence analysts apparently understood the value of ears long before facial recognition became an automated technology.
In his 1999 book, Photo Fakery: The History and Techniques of Photographic Deception and Manipulation , longtime CIA imagery analyst Dino Brugioni described an effort to determine whether photographs released by China showed Mao Zedong or a substitute.
Mao had largely disappeared from public view in late 1965 amid speculation about his health. In July 1966, Chinese authorities reported that the 72-year-old leader had swum nine miles down the Yangtze River in 65 minutes. Photographs of the event and later public appearances were promoted as evidence of his continued health and vigor.
According to Brugioni’s account, analysts compared the new photographs with authenticated images of Mao. One apparent substitute had Mao’s receding hairline and distinctive mole, but the configuration of his ears was different.
“His ears gave the double away,” Brugioni wrote.
The CIA reportedly went beyond comparing ordinary photographs. Its model shop constructed a 12-inch plaster model of Mao’s ear, allowing analysts to study its contours and compare them with images of the Chinese leader.
Not every suspected appearance was conclusively identified as a double, and Brugioni’s historical account should not be interpreted as proof that every rumor about Mao’s substitutes was true. But the oversized ear demonstrates how seriously intelligence analysts treated a feature that most people barely notice.
More than half a century later, NIST researchers are beginning to quantify why those analysts may have considered ears so valuable.
Biometrics is more than a sensor
The most important lesson Carlos and I learned in the GCN lab was that biometric security depended on far more than the accuracy of a scanner.
In some cases, defeating the sensor required a recording, a lifted fingerprint or an elaborate photographic disguise. In others, an attacker could target the middleware carrying the scanner’s decision to the access-control system. If that software was not properly protected and encrypted, someone might change the acceptance tolerance, substitute an approval or simply tell the system to admit everyone.
Many products resisted our probing and performed exactly as intended. Others paired advanced biometric sensors with surprisingly vulnerable supporting software that quickly fell to our attacks.
NIST’s new research adds the human examiner to that larger security system. Machines provide speed, repeatability and the ability to compare enormous collections of images. Humans can provide context, adaptability and the common sense to notice when someone approaches a scanner with another person’s face glued over his own.
And for specially trained humans, the ears may provide something even more valuable: an identifying signature hiding in plain sight.
John Breeden II is an award-winning journalist and reviewer with over 20 years of experience covering technology. He is the CEO of the Tech Writers Bureau , a group that creates technological thought leadership content for organizations of all sizes. Twitter: @LabGuys