The Many Facets of Facial Recognition
Future Tech 2025: An interview with Brian Lovell, CTO and founder of Imagus Technolgy
In this Future Tech interview we’re speaking with Dr Brian C Lovell, CTO and founder of Imagus Technology, a private company specialising in face-in-the-crowd biometrics from CCTV, mobile and wearable devices. He is a fellow and past president of the International Association for Pattern Recognition (IAPR), and a voting member for Australia on the governing board of IAPR.
Shara Evans (SE): This is Shara Evans from Market Clarity. Today, it is my delight to be speaking with Dr Brian Lovell, who’s recognised as one of the world’s leading facial recognition experts.
Brian, thank you so much for taking the time to speak with us for our Future Tech blog.
Brian Lovell (BL): Thank you, Shara.
SE: Let’s start by talking about facial recognition systems that most people are familiar with. I believe you would call these cooperative systems. An example, in Australia, is when I’m going through border control, they try to do some smart recognition of me based on the picture of me they have in their database or on my passport.
Even then, I have found that there are a lot of failures with facial recognition when people change their hairstyle, or perhaps have glasses on, or grow facial hair, or anything that deviates from the photo. Funnily enough, every time that I try to use the system in Customs here in Australia, it never recognises my image. What kind of background can you give us on those types of systems?
BL: I call these cooperative systems because you’re being very cooperative in almost all those sessions. You get a photo taken that has to meet certain standards. You’re not allowed to smile, you have to take your glasses off. It’s very good lighting, and it’s very high-quality imagery.
Then when you go to the Australian SmartGate system, you have to basically do the same thing. The SmartGate system is all about getting a very, very good-quality image. It has three cameras to allow for people with different heights, and they choose the camera that is the right height for you. They have even lighting on the face. You have to stand still, have no expression, and take off your glasses. You’re doing everything to make it work for the system. Still there’s about a 20 percent mismatch rate.
SE: Well, I’ve certainly experienced that myself. One of the things they do in passport photos is say that, if you wear a fringe as I do, you have to push it off your face to get the picture taken. Of course, that’s not how I appear when I go through SmartGate, so it fails every single time, without a doubt.
I’m sure there are other people who experience failures all the time. How useful are systems where you have to be perfectly posed, and have your hair in the same style, or always be wearing or not wearing glasses?
BL: Those systems are very useful for their intended purpose. I suppose the agencies that use them would say their purpose is to increase security, but that’s not really true, because a lot of people don’t go through those systems. If they were for increased security, everyone would go through them.
What they’re about is passenger facilitation. When the queues are very, very long, instead of putting on more staff, people get forced into the self-service queues, and that reduces the staffing costs.
BL: No, they only check you against your own photo. They don’t check you against the wider database.
SE: Well, that’s surprising. It seems like a missed opportunity to me.
BL: It does. They’re not really face-recognition systems. They’re verification systems because, when you supply your passport, you give your purported identity. You’re saying, “This is who I am.” Then it says, “Yes, you are who you say you are,” or, “No, you’re not.” That’s the only test they do. This is why people have been able to get through border control by using other people’s passports. The face matching is done by humans, and it’s very difficult for a human to challenge someone if they’ve taken their brother’s passport; they both have a beard, look a bit similar, they’ll get through.
SE: Yes, that makes sense. There are other nefarious things that criminals can do like plastic surgery to look like someone, I suppose, as well.
BL: Yes, there’s rare cases of that. I mean, you have to be very committed to use plastic surgery for border control. There has been a case of a woman who actually had her fingerprints swapped over to get through the biometric checks. She’d been banned from Japan.
SE: Well, that’s pretty radical. Now going back to this whole area of facial recognition, one of the other techniques that could be used is real-time videos, say, from CCTV cameras or webcams. That’s an area you’ve been doing some work in. What can you tell us about that kind of facial recognition?
BL: CCTV is quite different from the border control situation. The person is usually not aware they’re being photographed, and there may not even be a person in front of the camera. You have to actually detect the faces, and they typically won’t be looking towards the camera. People will be on the phone, texting, or they’ll be talking to somebody else while you’re trying to recognise them. They’ll typically be walking past the camera.
SE: Yes. There might be other people around as well. Could you use the CCTV system for real-time policing?
BL: Absolutely, yes. Now the technologies I’m familiar with aren’t common, but they certainly perform quite well in face-in-the-crowd situations.
SE: What kind of features are used for image recognition? For instance, if you don’t have a full frontal view of a person’s face, how do you figure out that it’s actually a particular person?
BL: Well, non-cooperative systems almost always have to make do with an angled photograph. Certainly from CCTV, the cameras are usually in the ceilings, so you’re looking slightly down on each person. Usually, they don’t put the camera right in the middle of a corridor. It’s usually to the side, so you tend to get a slight off angle as well. That’s one reason why these systems tend to be much harder to build than the cooperative systems.
SE: It’s not only that, you might even have a couple of different cameras looking at someone from different angles. With the systems that you’ve developed, what sorts of things do they hone in on to be able to do facial recognition?
Facial recognition models
BL: Well, we use a statistical model of the face. We look at the low-frequency textures that make up the face. Some systems use very high-frequency features like moles on the skin and wrinkles, but that requires a very high-quality image. Whereas, in our case, with people walking past, there’s motion blur, et cetera. We can only guarantee that we’ve got low-frequency information. We concentrate on those features. These features have been considered unreliable, but we’ve been able to get very good matching just on these very low-resolution features.
SE: Can you give me an example of a low-resolution feature?
BL: The bone structure of the face is a low-level feature. A high-resolution feature would be your eyelashes or crow’s feet at the edges of the eyes.
SE: Of course those would change over time.
BL: Features like wrinkles are one of the ways that these systems can do age measurement. Typically, age measurements are always done on high-res photographs.
SE: Is that done automatically?
SE: Who would use that kind of information, or how would it be used?
BL: It’s mostly used for marketing and demographics. In a store, for example, you might like to count the number of people looking at your billboard or your display. You’d like to see how many males, how many females there are, and you’d like to know, roughly, whether they are young males or old males, et cetera. That’s an application that’s quite interesting.
SE: Security agencies wouldn’t necessarily use age recognition to find bad guys or bad gals?
BL: No, it’s more retail demographics.
SE: Okay, that makes sense. As I understand it, the software that you’ve developed at Imagus breaks down an image of a person’s face into lots of little squares to do an analysis. Is that correct?
BL: Yes. Rather than what we call holistic analysis — meaning, doing the whole face at once — we break the face into small parts and then match those parts.
SE: Is that a more reliable way of identifying a face than trying to look at the holistic image?
BL: It’s much more reliable for high pose angle photographs. If you’ve got frontal photographs, then you don’t necessarily have to do that, but, for CCTV, people are hardly ever looking straight at the camera.
SE: That would make sense because the cameras are typically mounted on ceilings and, as you said earlier, pointed at an angle looking downwards on people.
BL: It is pretty obvious why that’s the case. If you have them at eye level and they’re right in front of you, you’ll walk into them. They have to move them out of the way and keep them away from vandals.
SE: Yes, exactly. Would this type of facial recognition also be done in real time? Would it be used by security agencies to match against a forensic database?
Real time recognition
BL: Yes, that’s the idea. It’s to spot individuals. One interesting application is cross-camera recognition, where you can measure the transit time of somebody, say, through an airport, by looking at their face in one place then matching it somewhere else. That’s something that’s of great interest to large facilities like airports.
SE: In that instance, if you had someone that was loitering around an airport for a period of time, that might put up a flag, or if someone is coming back on a regular basis, might that put up a flag as well?
BL: Yes, that’s right. Typically, companies will spend a lot more money on something that’s related to their business that can be used 24 hours a day than an on the spot-the-terrorist problem, which might be used once in 10 years. You really need to design systems that work for the bad guys, but actually have some value for the 99.9 percent of good customers.
SE: In the security or law-enforcement realm, are there applications or business use cases where they’d have value other than in detecting terrorists or criminals?
BL: Most of the police just want to identify people. The problem is the people you want to identify are a very small percentage of the population. Most systems get killed because of the false-alarm rates. If you’re looking for one person in 20,000, you’ll get a lot of false matches, and then you’ll probably just find the system gets turned off.
SE: Okay. Turning to a more general application, are there any home-security applications that this technology might be useful for?
BL: Yes, very much so. If you’re thinking of, say, installing a home CCTV system, the problem is, that to determine what’s happened in your house, you have to watch a very boring video for a very long time. Most days, nothing interesting happens. An advantage of non-cooperative face recognition is that, instead of just looking at all the video, you could have the video say, “My wife’s arrived home. My daughter’s arrived home. My daughter’s boyfriend has entered the house,” by processing that video.
SE: Could you also detect when someone that’s not in a facial database of people known to the household is detected? Could you raise a flag when that event occurs?
BL: Yes. When an unknown person enters your house you could see if that unknown person is somebody you know, or say, “Who’s that person?” That’s very useful.
SE: Does your software do that now?
BL: Yes, it can do that now.
SE: Would it be sending an alert to somebody’s iPhone, or computer, or e mail, or some other mechanism?
BL: I think the way you do this is to have it web-configured. If you’re going to have something for the home market, it has to be very low cost. The sort of systems we’re considering would have small Wi-Fi cameras around the house that have motion detection so they don’t waste their battery power. When somebody goes past one it sends that video to a cloud service, which could then identify them. You’d have some quite simple infrastructure in your house and have a subscription service on a cloud.
SE: Is your technology or software powering any of these types of home-security cloud services?
BL: They’re being discussed. This could happen, but you need to get to a certain scale to roll that sort of equipment out on a cloud service.
SE: Turning back to the whole concept of facial recognition, one of the advances that I’ve seen in some of the newer chips like Intel’s RealSense is the ability to detect emotions from a facial expression. I’ve even seen prototype software from another start-up that matches a person’s expressions with what they’re saying. We’ve got so many nonverbal clues in everyday conversation that just don’t come across the same way if you see a transcription of what’s being said. Have you done much work in this area?
BL: We’ve done some work in emotion recognition. We’ve mostly concentrated on identity because it’s easier to monetise identity than emotion. It’s something we’re interested in, but it hasn’t become mainstream. For retail demographics, we’ve worked on gender, and age, and people counting because those are things that customers actually ask for.
SE: Have you done any work in the retail space with, say, being able to recognise and alert storeowners to VIP customers?
BL: Yes. That’s a space we’re very much moving into now, both in Australia and overseas.
SE: What kinds of retailers are taking that up?
BL: Large retailers and shopping malls. We’re also looking at systems for small stores, like a corner store, at different price points, and this is why you need a cloud delivery service, to make these systems affordable to mom and dad operators.
SE: How would it work? Let’s just say I’m a VIP customer of a store. I walk in. There’s a CCTV camera or a Web camera. How would the store owner or salespeople get an alert: “Hey, a VIP customer’s walked in”?
BL: They’ll typically set up a list of faces on our website that’s come off their CCTV system, and then put names against those faces, and possibly personal preferences, like whenever this guy comes in, he orders a coffee that’s made a special way, for example. It’s up to them to decide what the alerts will be. Then when that customer comes in, even if the person on the counter doesn’t know them, they’ll know they are regular customer, and what they always order.
SE: For the person behind the counter, would they be looking at a computer screen to get this alert, or would it come across on their mobile phone? How would this actually work in a store environment?
BL: It could work either way. Typically, in a store, there’ll be a desk, and there’ll be a computer screen that staff can look at. The alerts would come up on that screen. There’s no problem sending those alerts to a smartphone or any other device, such as an iPad, for example.
SE: What kind of take-up are you seeing for this kind of retail application in Australia and overseas? Are there any differences?
BL: There’s huge interest in the UK and South America. In Australia huge interest also. We’re still at the early stages, but this is progressing.
SE: You’re not able to talk about specific customers at this stage?
BL: No. We’re still doing trials in some areas, but it’s all very encouraging. It’s quite different from the security market because the price point’s much lower, but of course it’s a much larger market.
SE: Absolutely. There are only a limited number of security agencies in a given country; whereas the number of retail shops, large and small, is very, very large.
One of the other applications that you’ve developed is a payment-authentication service using facial recognition. What kind of feedback are you getting from merchants on that particular development?
BL: Everybody wants a system like that. The area’s moving extremely rapidly. There’s some big players moving in with technologies like fingerprint recognition. It’s certainly beyond our capacity to change every mobile phone in the world to have a fingerprint reader, but certainly some companies are using fingerprint readers as a biometric. I quite like the use of a face because that could work on any device, and virtually every device has a camera on it.
The other advantage of a face is you can authenticate continuously through app transactions. For example, while I’m speaking to you, a camera could be looking at me and continuously saying, “Yes, that’s Brian Lovell in front of the computer.” If somebody else decided to walk in front of it, it would say, “Brian Lovell’s no longer in front of the computer.” You can do this for a long transaction.
SE: I’m just thinking about the use of fingerprints. I’ve seen some YouTube videos that show how easy it is to fool a lot of smartphones by taking a latex print of someone’s fingerprint, and putting it on top of the phone, and unlocking their passcode, and of course then being able to make payments and do anything else that you do with a digital wallet.
BL: Yes, absolutely. That’s a problem with biometrics, fingerprint or face. You’re stuck with the same ones for the whole of your life. If somebody can make a good copy of those biometrics and present them at the appropriate time, they can get access.
SE: I understand how you can make a copy of a fingerprint. How would you make a copy of someone’s face? Would you just use a digital replay of a video or something else?
BL: Well, the obvious thing is to have a photograph of the person. That photograph could even be stored on a smartphone. You just pull up the photograph and put it in front of the camera, and some systems will accept that photograph as a good match.
SE: Will that work at, say, SmartGate?
BL: The trouble there is you’ve got a whole bunch of customs officials standing around looking at you. If you started pulling up a picture of yourself and putting that in front of SmartGate, somebody would very quickly pull you out of the line, and you’d probably get into serious trouble.
BL: It’s very different for home banking, where you’re sitting in your bedroom all by yourself, and people can try anything like that.
SE: If there were a facial-recognition payment system, wouldn’t that be subject to potentially more fraud than knowing a pin or knowing a password?
BL: Not necessarily. When you do something with a smartphone, typically that phone or device is registered with the bank. To do the attack, you have to have access to the phone or device, and you also must have the face of the person. It is not such a great risk except for high-value people, because it’s a fairly expensive attack to mount. You have to get the person’s phone and then get a face in front of it. The attacks that I’m really worried about are the attacks that can be conducted from another country by stealing the biometric from inside the phone and then replaying it at two in the morning.
SE: Right. What you’ve just been describing is two-factor authentication. You need the device, and you need to have the right biometric.
BL: And probably some secret information like a pin or an account number. You could have three factors fairly easily.
SE: Are there protections for people who are considering biometric data use in a transaction?
BL: Yes. Typically, it won’t be used by itself. I’d be worried about having the biometric as the sole means of authentication except on fairly simple transactions where there’s not much cost involved. If you were doing a transfer of $50,000 to a foreign bank, you’d need to have high levels of security. Certainly, biometrics could be part of that, but then you’d probably need other bits of information as well.
SE: Would it make sense to use two different biometrics, say, fingerprints and facial recognition? Would that be harder to fool or trick?
BL: It depends. If you’ve stolen that information through a malware attack, you’ve probably got access to all of those bits of information in any case. It won’t necessarily be harder to fool.
SE: In your view, would it actually make more sense to have two factor authentication that uses a device plus a biometric, as opposed to two biometrics?
BL: Well, biometrics can all be stolen fairly well. Secret information like passwords, they certainly can be stolen, as shown by the Edward Snowden effect, where people just gave away their passwords. As soon as you give away your password, it can be used and given to anybody. Biometrics can be stolen, and passwords can also be stolen fairly easily. Then of course the device itself, a key or a phone, can be stolen as well. If you had a combination of three or four factors, you start to make a system that’s much easier to defend.
Another point: people have to realise is there’s always the rubber-hose attack. That’s where you take a member of the family, and you start chopping their fingers off, and demand the person takes money out of their account. There’s only a certain level of security you can get before the rubber-hose attack becomes the standard method of getting access.
SE: Well, that’s a horrible thought, but I suppose someone would really have to be after whatever the person has to even consider that kind of attack.
BL: That’s typically done on high net-worth people, where they hold a member of the family to ransom. In some cases, you’re better off losing a bit of money from your account than going through the alternatives.
SE: Absolutely. I’d be giving them my bank account before giving them parts of my body, for sure, or maybe I shouldn’t say that in public. I just don’t think I’d stand up very well to that kind of interrogation.
One last thing that I’d like to talk about is the correlation that I see between facial recognition and augmented reality. In everyday life, walking down the street, there are people that you recognise, and you just can’t remember their name, or you go to a lot of conferences, like I do, or you’ve had lots and lots of clients over the years, and you pick out the face but you can’t quite correlate it with that person’s name and exactly the context where you met them. Having the combination of facial recognition and augmented reality seems, to me, a very broad-use application. I’m wondering what your views are on that.
BL: Yes. In a meeting once, somebody said that they’d give us so much money if we could solve this problem for them because they’re always meeting people and they can’t remember who they are. There is a need for that. Some people are much better at remembering people than others. About three percent of the population can’t recognise anybody, and they spend all their life pretending that they know people. They do a good job of hiding it. In fact, Oliver Sacks, the famous author who wrote the book “The Man Who Mistook His Wife for a Hat,” apparently he just couldn’t recognise anybody. It’s quite common.
That’s one side of things that’s helping people to recognise others. The problem there is that a lot of people are good at recognition, so this is another reason why we tend to concentrate on making things recognise people, like phones and cars recognising their owners.
SE: Even if you do recognise people, in a lifetime, you meet a lot of different people. Sometimes it could be years between when you first meet them and when you see them again. If you tie in augmented reality and image recognition or facial recognition, and you key it into social networks and other information that’s out there, it seems to me a quite powerful combination.
BL: Yes. That’s certainly true. The other thing that’s quite cool is using face recognition on a phone. I can export a recognition template to somebody else on the other side of the world. I can recognise you. Then when you go into Washington and get off the plane, somebody else could recognise you using that template. So you can in effect export your recognition ability.
SE: That’s interesting. You can literally meet somebody for the first time and pick out each other in a crowded place.
BL: Yes. I’ve sometimes had to do that. I’ve had to meet somebody who I’ve never met them before, in a coffee shop. So I’m in the awkward situation of looking around for somebody who’s looking around for me, and trying to figure out if that’s the person I am meant to meet.
SE: That happens to me multiple times a week. Usually, what I say to them is, “I have long red hair, and I’m wearing X colour suit,” whatever strikes my fancy at that particular time. There aren’t many women with long red hair, which makes me fairly easy to pick out.
BL: That’s right. You look for some distinguishing feature. This happened to me the other day, and I didn’t know who I was meeting exactly. I looked him up on Wikipedia and found his photo, and I then looked around and saw him, so I was actually using electronic face recognition to determine who I was meeting.
SE: Yes. LinkedIn is great for that as well.
Now one last thing that I’d like to ask you, Brian: how would you suggest that people could protect themselves against having their biometric information stolen? Is there anything that an ordinary person can do?
BL: No, not really. Your face could be anywhere. There’s a problem now with Facebook. Somebody will take a photo of you and put it up on Facebook even if you didn’t want that photo put up there. One thing we’re looking at is using face recognition to protect people’s privacy by finding photos of them that have been uploaded by other people and that they might want to have taken down.
SE: Has anybody developed that idea further because it seems like a really good use?
BL: Yes. One of my colleagues was trying to develop a start-up in that area, but I don’t know how it’s gone. There’s a lot of things you think might be worthwhile, but there’s always the problem of monetising them. There’s quite a lot of cost in the research and so on. You have to find something that has a business model.
SE: Yes. I think most people haven’t quite got the fact that their privacy may be violated because there haven’t been many very high-profile cases of identity theft, where people say, “Hang on, I want my privacy back.”
BL: There have been a number of cases of attractive women whose faces have appeared on escort sites. Their face has just been taken off their Facebook page and put on to some business site. They don’t know anything about it until something strange happens and it gets revealed that they’re being used for those purposes.
SE: It would be rather horrifying to suddenly find that all your colleagues think you’re a prostitute in your spare time.
BL: It’s happened already. Another group that’s very, very concerned is the undercover police. They have a legally protected reason to maintain another identity, but if their photos from when they went to school, or whatever, are on Facebook, and there’s some electronic face recognition systems out there that are quite powerful, they could be revealed and they could be killed.
SE: That would be a horrible outcome.
In terms of where you’re going with Imagus, what do you see as your primary markets?
BL: Well, we’re interested in airport security and other traditional markets, but we’ve really been pushing towards the consumer market. Faces are a very special biometric. People talk about fingerprints, iris recognition and so on, but a face is special because it’s what humans use. If I recognise somebody, it’s through their face, not through other features. The nice thing about face recognition is it’s very natural to be recognised by your face. Whereas fingerprints have the association with criminals and so on. If I see two fingerprints, there’s no way I can tell who I’m looking at. We don’t use fingerprints on a day-to-day basis for recognition.
SE: Very true. I don’t see anybody inspecting fingers closely. Even if you could, you’d need magnifying vision to be able to pick out one fingerprint from another.
BL: That’s right. People say face recognition is creepy, but the funny thing is that’s what we use every single day. It’s really handy not to have to identify yourself to everybody you’re speaking to and log in and give your credentials. You can just very quickly get on with business because people know who you are. They know the joke they told you at the last office party. They know any messages they had for you. As soon as they see you, they say, “There’s all these things I want to communicate with you.” Yet with computers, they’re completely dumb. You have to log in, and before you log in, you can’t do anything.
SE: Yes, and trying to remember all the passwords is just getting ridiculous. Best practise is you have a different password for every website, every application, pretty much anything you do, and they’re meant to be complicated passwords as well. How many complicated passwords can one person remember? I think there’s a finite limit, unless you store them in a secure vault somewhere on your computer, or on your phone, or somewhere else. Facial recognition or other biometrics make a lot of sense for me.
BL: Yes. There’s a HCI (Human Computer Interface) expert from University College London who did a study and said that people spend three working weeks each year authenticating themselves to computer systems or other systems. If we could bring those three weeks down to, say, one week by using biometrics appropriately, I think that’s a really good call.
SE: With the emerging Internet of Things, we’re going to have more and more devices, gadgets, wearables, you name it, that we’ll be interfacing with, and trying to make them all talk to each other without entering passwords all the time could become a bit of a logistical nightmare.
BL: That’s right. As we have to interact with more and more things, face recognition becomes more and more convenient.
SE: That’s right. We wouldn’t want somebody who just might be visiting our home to go up to our refrigerator and tell it to order a whole bunch of groceries. We’d want it to only take our commands.
BL: Yes. Those are the sorts of things that would be popular if you could get them cheap enough. Certainly, our technology can be put into very small embedded devices in a dashboard of a car or in a mobile phone, for example. That could become like a universal service.
SE: Would that be embedded in firmware, if you were going down that route?
SE: That would be an interesting application. Is that something you’re thinking of or already working on?
BL: We can do that already, but, you see, you have to get a business case and get the model right. It’s certainly possible to put it into a very small processor.
SE: Yes. That sounds like a wonderful application. Are there any other closing thoughts that you’d like to share with us in this whole world of facial recognition, or the things that you’re doing at Imagus?
BL: I think face recognition has definitely been popularised in border security. The Hollywood version of facial recognition always works. It’s wonderful stuff. Certainly, when I started researching this area, I assumed it already existed and that every major agency had some magic software that could recognise anyone anywhere. It turns out to be absolutely not the case. Cooperative recognition is pretty good, but just recognising somebody naturally in a restaurant or across a bar, for example, from a mobile phone, that system really didn’t exist. This is the space that we’re entering into now.
SE: How fabulous. Well, thank you so much for your time. Really appreciate chatting with you, Brian.
BL: Thank you, Shara.
About the author: Shara Evans is internationally acknowledged as a cutting edge technology futurist, commentator, strategy advisor, keynote speaker and thought leader, as well as the Founder and CEO of Market Clarity.