The role for face-recognition in customer experience

When the new iPhone X was unveiled this autumn, the phone’s capacity for face recognition instantly became its key talking point. Say goodbye to tapping in passwords—just a quick scan of your face, and the iPhone software would know who it is answering to.

But in China, this software isn’t really new to brands and marketers. “Facial recognition (FR) has been in the China market for a while,” says Carter Chow, CEO for JWT China. “Its application and usage have seen gradual pick up, but with the more recent launch of the upcoming iPhone X, the topic has resurfaced as being very hot on social media. Chinese consumers are generally very open to new tech concepts and have generally felt less resistance to sensitive topics like privacy and data security.”

In July, the Beijing government committed to making AI a “new and important” part of the economy by 2020, and China’s technology firms both old and new are rushing to apply the commercial use of FR technology, often working with the government’s image database of 700 million citizens.

In September Ant Financial, the financial services spinoff of Alibaba, rolled out a new service at KFC in Hangzhou allowing customers to pay for orders using only their faces. Although buyers had to input phone numbers as an extra layer of verification, the technology still works even if the purchaser’s phone is turned off. 

Caught in traffic: A Chinese traffic cop explains the facial recognition screen to a girl at a road crossing in Shanghai.

“We’ve had positive feedback so far from our customers, who think the idea is amazing, convenient and fun,” says Steven Li, senior vice president of Yum China. “We find our China customers are very excited by the new technology and want to try it and use it.” Li adds KFC will introduce it to more stores in November.

Although this is the first time Alibaba has rolled out the technology in commerce, it was first unveiled by Jack Ma two years ago at a technology conference in Germany, dubbing it “smile to pay.”

In the main, Chinese consumers seem to be embracing the new technology. A recent HSBC survey found nearly half of people interviewed in China believe FR to be useful, 16% higher than the global average.

Car-hailing service Didi Chuxing is using the technology to verify drivers’ identities, while ecommerce giant Suning has launched a new unmanned store that allows buyers to pay just by showing their face to the scanner after registering on Suning’s app.

It is also being experimented with fairly widely in banking. HSBC has introduced facial recognition capabilities in its mobile banking application for customers in China, meaning customers can open their HSBC mobile app, blink at their camera, and transfer up to $7,500 a day. China Construction Bank allows customers to pay with their faces at some vending machines, while China Merchants Bank does the same at about 1,000 ATMs. A mobile affiliate of Ping An Bank also uses facial recognition to authenticate a borrower or investor’s identity over the internet.

Airlines are experimenting too, and earlier this year China Southern Airlines used facial recognition in place of boarding passes for the first time.

Chinese citizens used to facing up

The technology is also proving useful to state-run organisations. In Shenzhen, Shanghai and Beijing, facial recognition is used to identify jaywalkers, with names and pictures displayed on a scene above the road, while in Beijing officials have started using the technology to catch thieves of toilet paper in public toilets.

In September, police arrested 19 people at the Qingdao beer festival who tested positive for drug use by using entrance cameras to record the faces of more than two million attendees. Using facial recognition, they identified individuals with past histories of drug abuse and tested them on the spot.

IHS Markit has estimated China has 176 million surveillance cameras in public and private hands, and it forecasts about 450 million new ones installed by 2020. The US, by comparison, only has about 50 million.

In the near future, tech firms and marketers will combine to personalise this technology to their user’s needs. Facial recognition is already being used by KFC to predict their customers’ orders, for example, depending on previous history and their demographic.

To catch a thief: These toilet paper dispensers at the Temple of Heaven in Beijing use facial recognition, aimed at ending a years-long crime spree by toilet paper thieves.

“The use of FR technology can be very wide,” says JWT’s Chow. “We are seeing more and more different applications every day. However, the true test of any innovation is if the related applications can actually bring practical benefits to the users - for example, the use of QR codes. QR codes proliferated without bringing users enough actual benefits, but this all changed when social media apps like WeChat used QR codes as an easy and quick way to add new social contacts, followed by mobile payment platforms like Alipay and WeChat Wallet.

“So, marketers and agencies need ensure that they are not using facial recognition only as a cool or trendy tool but rather, really think how the new technology can deliver meaningful benefits to their consumers.”

Aram Sinnreich, author of The Privacy Crusade, has predicted a future where advertisements will soon be able to gauge the likelihood of an individual making a purchase, and adjust their messaging in real time. For instance, a customer might see an advert for 50% off a meal deal on their phone just as they pass by a fast-food restaurant. If the phone registers disinterest when it scans their face, it could increase the discount level, to see if it persuades the customer to buy.

“If you walk into a supermarket, and examine a coffee maker but do not buy it, in a few days you could be shown an internet ad of the same item with a discount or receive a personal message on social media with relevant information about such products,” says Mikhail Ivanov, CEO of technology group Ntechlab. “Visitors’ photos may function as cookies referring to the identification and storage of user settings. In other words, loyalty cards will become obsolete. As soon as you walk into a store, the staff will already know what you bought last time, thanks to the camera’s footage and our technology.”

In 2016, Apple bought a tech startup called Emotient, which can reportedly detect up to nine different emotions by looking at a person’s face. Since every Apple device now has a user-facing camera installed in it, Sinnreich suggests that in the future marketers could in theory collaborate with Apple to record how consumers feel about their adverts, and then adjust them to capture interest in real time.

Walmart is reportedly working on a FR system to improve customer service, including cameras that detect dissatisfaction on shoppers’ faces and dispatch help.

(Story continues below)

Smile for the screen: Pay with a face scan WHO RECOGNIZES YOUR FACE? Alibaba’s “smile to pay” project is powered by Face ++ technology, developed by Beijing start-up Megvii. Founded in 2011 by three graduates of Tsinghua University, it is one of a series of Chinese start-ups that are competing globally in developing AI technology. Megvii also runs the technology behind popular apps like Chinese beauty fad Meitu. One rising star in China’s AI start-up scene is SenseTime, currently valued at $1.5bn. It uses facial ID to replace office key cards. Its software also turns ordinary traffic cameras into “smart” cameras, able to analyse the types of vehicle and human traffic flowing through them. Search giant Baidu showcased its facial-recognition technology at the company’s first AI developer conference in Beijing this August. It will use this technology at the city’s airport, initially for recognising ground crew and staff at entrance gates, but gradually expanding to verify passenger identification. Baidu has already tested this technology at tourist destination Wuzhen water town, allowing visitors to gain entry using their faces, rather than tickets.

“Retailers are starting to use facial recognition to improve the customer experience for their consumers,” adds Ivanov. “Firstly, they can monitor the facial expression of their workers and customers to ensure that the employees act according to their guidelines and provide a great experience. Secondly, they can recognise their recurring customers and provide personalised offers and discounts without even asking a card or a phone number.”

Beware of abuses

South African coffee company Douwe Egberts set up a coffee machine with FR at an airport that offered free coffee to travellers that yawned as they walked past. Yet once travellers figured out the scheme, it proved costly as word spread throughout the airport.

This shows brands and marketers need to tread carefully in this area, bearing in mind the potential for abuse by customers — or even damage to their brand.

Google, for example, has explicitly turned its back on matching faces to identities, for fear of its misuse by authoritarian regimes. Chairman Eric Schmidt has called face recognition technology ‘creepy’, and said Google will not build a database of users’ faces.

Similarly Tan Jianfeng, founder of encryption tech firm PeopleNet and president of the Shanghai Information Security Trade Association, has been outspoken against using FR technology for gimmicks.

“Once biometric libraries are breached and biometric data are stolen they can never be restored. This is the real weak spot for biological certification,” he warned.

However, the march of FR technology in China seems inevitable, with an enthusiastic government, interested consumers, and growing numbers of technology firms investing heavily. The challenge for brands and marketers will be to use it in a way that excites and interests Chinese consumers, rather than monitors them.