How Does Facial Recognition Work? What’s Wrong with FRT in NYC?

The CCTV camera system with facial recognition in New York City could be called one of the most advanced in the world. But are there any threats? How do algorithms work, and what is wrong with FRT in NYC? What is the future of biometric surveillance?

What Is Facial Recognition Technology (FRT)?  

Facial recognition is a technology that automatically recognizes (a person in a photo) or verifies (confirms a person in a photo) a person in a photo, video, or in real-time. The recognition system is based on neural networks, which can read and analyze unique facial features and then compare them to faces in the database. 

How FRT Evolved in Different Periods

• In 1964, a Texas University professor and researcher of artificial intellect in Austin, Woodrow Bledsoe, was the first to attempt “face recognition.” His team created a dataset of 800 photos of people from different angles. Then the scientists dotted faces with 46 coordinates with the help of the prototype of the modern tablet. A specially designed algorithm could rotate faces from various angles, zooming in and out. The second stage involved 22 dimensions based on Bayes’ Theorem for decision-making – to achieve the most accurate results. Consequently, Bledsoe’s system could perform the task 100 times faster than humans. 
• In 1988, Michael Kirby and Larry Sirovich from Brown University employed a new Eigenface method. It’s derived from linear algebra to analyze images representing the pixel intensity of facial features in a lower dimensional space. They used over 100 different values to mark faces. 
• In 1991, Alex Pentland and Mathew Turk from MIT improved the Eigenfaces technology using environmental factors. They managed to automate the recognition process.   
• In the late 90s, the US Department of Defense and the National Institute of Standards and Technology (NIST) launched the FERET program. Initially, it was used to find and identify criminals; later, it became available for public access. 
• From the 2010s onwards, Facebook began to use facial recognition technology to find users on public photos and offer to tag them. 
• In 2011, Panama and the US started a project known as FaceFirst. It was used to deter illicit activities at the Tocumen International Airport. The same year, the US intelligence agencies started to use facial recognition to identify corps, including Osama bin Laden. 
• Since 2014, the technology has been used in mobile phone cameras and, since 2017, in retail.   

How Does Facial Recognition Work?

The technology hinges on two neural networks: 

The first network – aligner. It takes an image from the camera, detects all faces, “cuts,” and aligns them. 

This is what it looks like:

Step 1. The program cuts faces. 

The software cuts faces (it may fail to recognize faces close to each other, turned in profile, or simply too small and blurred). 

Then it aligns them and applies dots on the eyes, nose, mouth, and other parts.

Finally, it turns the image and matches the size in a way to navigate eyes, nose, and mouth dots in specific spots.

Some algorithms use more than seven dots – to contour around the shape of the face. 

The second neural network – is recognition. It receives the aligned image from the neural network and generates a vector for a face – a string of fixed-length numbers. These vectors may differ depending on the networks. 

A huge database with faces is used to train neural networks. They give information about the person and reach the maximum outcome during training to provide the most accurate result. Trained on millions of different people, a neural network begins recognizing new faces, even those outside the database.  

How Does FRT Work in Smartphones?

The FRT is a little bit different in smartphones. The algorithm recognizes you by infrared dots and not a photo projected on your face through a camera. Furthermore, a 3D model is generated to compare to the one you entered earlier. 

Where is FRT Used?

Safety

Forensic scientists, police, and intelligence agencies use biometric identification systems (ABIS) to locate criminals, reveal crimes and prevent them.

Facial recognition cameras ensure security at mass gatherings and airports and control access in various organizations. The systems help find missing children, disoriented adults, or people being victims of trafficking.

Healthcare 

Face recognition in hospitals and nursing homes helps track whether patients are taking medication and monitor their condition through a special monitor. Neural networks can detect genetic diseases through distinct facial features – such as DiGeorge syndrome – and assess the patient’s general condition.

Retail, Catering, and Banking

Facial recognition technology helps identify the customer and prevent fraud during the purchase, analyze customers’ behavior and optimize the service to sell more.

Using online biometrics, you can open an account, get a loan and withdraw money from an ATM. For example, Chinese KFC and American Amazon Go use a Facial-Recognition Payment (FRP). In the US, the first business to benefit from the system was CaliBurger in 2018. Although slow at the moment, according to Business Wire, the US facial recognition market is projected to grow to $10.15 billion by 2025. 

In May 2022, MasterCard piloted biometric payment technology this year, like facial recognition and fingerprint scanning. And later this year, it plans to go global. 

Education

Facial recognition services help during online learning: they make sure that the student does not get distracted during the exam and does not cheat or use verbal prompts.

The Most Advanced Developments in the Industry

The FRT accuracy has improved 50 times from 2016 to 2020: the error rate was 0.8%. According to the 2019 study by the Facial Recognition Market, the FRT global market was valued at $3.2 billion. The 2024 forecast brings the figure to $7 billion, with an annual growth of 16%. 

The most advanced developments in facial recognition belong to Google, Apple, Facebook, Amazon, and Microsoft (GAFAM)

In 2014, Facebook launched a DeepFace service, which identifies whether two faces on a photo belong to the same person. The accuracy rate of the software is 97.25%. 

In 2015, Google introduced FaceNet, which achieved a record accuracy of 99.63%. The technology is used in Google Photos to sort images and tag people.

Since 2018, Amazon has been actively promoting its cloud-based facial recognition service, Rekognition, used by US law enforcement agencies. The system can recognize up to 100 people in one photo and look for them in databases of ten million faces. 

The biggest players in the market of facial recognition in NYC are Avigilon and Verkada. Avigilon Control Center (ACC) 7 employs FRT to detect people of interest based on a secure watch list. Its predecessor, ACC6, has a deep learning AI search engine for a video that uses face analytics to understand it is searching for the same person, even if they wear different clothing over time. 

What’s Wrong with Facial Recognition in New York City?

The NYPD has relied on FRT since 2011 to find images of suspects captured by cameras during the robbery, shootings, and other crimes. The data provided by Amnesty International in its study Decode Surveillance NYC project revealed that NYC has 25,000 cameras. 

However, while biometric CCTV has several benefits, many New Yorkers view the technology as biased. 

The report said NYC’s cameras showed more surveillance in boroughs with a high concentration of non-white like the Bronx, Brooklyn, and Queens. The problem was rather with how NYPD used surveillance and ClearView AI system, so some critics view it as discriminatory. In addition, there’s a report which says that FRT can be inaccurate when analyzing Black and Hispanic adults. 

A majority of Americans say the use of facial recognition software by police would be more welcoming if cops were trained in how FRT can make errors (Pew Research).

There’s also concern about privacy. In 2019, when the landlords of a New York City apartment complex in Brooklyn decided to use a facial recognition system for access control to a building’s doors, the tenants opposed the effort saying it was violating their privacy rights. Yet, a recent Pew Research stated that 50% of US citizens approve of the technology in apartments and retail stores. 

Similarly, biometric surveillance in New York state schools raised questions about its adequacy. Parents feared that information collected by the biometric software could be turned over to law enforcement or immigration authorities. Or, highly sensitive information can be abused, stolen, and hacked. In one case, the Lockport City School District’s parents sued NYSED to prohibit the tech in the school. 

While government agencies’ use of FRT has been under the public eye, experts say there’s less known about the extent of facial recognition use in private sectors. 

Until recently, datasets with faces from open sources in the United States were used to train neural networks. The public domain also contained these datasets. However, they have been removed, as it violates data protection laws. They are available with permission and for research (non-commercial) purposes only. 

How FRT Is Regulated in the USA

The US has no single law that would regulate the technology. The closest to the EU GDPR is essentially the California Consumer Privacy Act (CCPA). Some states have restrictions on the police and other agencies – for example, in Oregon, Washington, Texas, Illinois, Massachusetts, and Vermont.

IT corporations are pushing for laws to be passed at the federal level. In 2020, IBM announced that it would be winding down its facial recognition products. Following them, Amazon and Microsoft temporarily suspended cooperation with government agencies. And in July, ACLU activists filed a lawsuit against Clearview AI for violating the Illinois biometric privacy law.

What is the Future of Technology?

Experts believe facial recognition will finally become a mass technology in the next few years. It will be used in “smart” and conventional gadgets for identification and payment, access to offices and check-in at the airport, and access to online services and social media profiles. A complete switch to biometric passports is expected within the next 15-20 years. 

Although a range of concerns shadows Americans’ acceptance of FRT, the public sees facial recognition use by police as positive rather than negative for society. It would help to remember that the NYPD doesn’t use facial recognition technology to identify people on the city’s network of CCTV cameras. The footage isn’t analyzed unless it is associated with a committed crime. 

Besides, it is difficult to deny the convenience of technology for everyday life. For example, access control and intercom systems equipped with face recognition technology allow you to open the front doors of residential buildings without getting out of your  bed or while taking a bath. You see a familiar face on your smartphone, press the button, and let the person in. What’s more, you can do it thousands of miles away from home.