AI with a Human Face

All companies want to give their customers richer and more engaging experiences. That’s one of the most effective ways to create and sustain competitive advantage. The challenge is to offer those experiences at scale without depersonalizing or commodifying them.

Throwing people at the problem becomes prohibitively expensive very quickly. And even if a company had enough employees to offer individual service at scale, in many situations customers prefer to interact with someone of their own gender, age, or ethnic background—an impossible staffing task. Moreover, research suggests that humans do not always produce the best results for every job. For example, Deloitte UK found that human-staffed contact centers are not only more expensive to run but often deliver a less consistent customer experience than automated channels—and they sometimes generate negative customer-service experiences.

Enter the digital human. Rapid progress in computer graphics, coupled with advances in artificial intelligence (AI), is now putting humanlike faces on chatbots and other computer-based interfaces. Digital humans mimic human communication as they offer a range of services: Companies are currently using them as sales assistants, corporate trainers, and social media influencers, for example. When deployed at scale, digital humans will radically change the business landscape. They may not be as capable or versatile as human employees, but they have clear advantages when it comes to cost, customizability, and scalability. Once “hired,” they never tire, never complain, never seek a raise, and always follow company policy.

Digital humans are already making real money for their employers. Soul Machines, an autonomous animation software company, has upwards of 50 digital humans deployed in organizations around the world. According to cofounder Mark Sagar, one client in the cosmetics industry, whose digital sales assistant recommends and models products and engages with customers about how to use them, has seen sales conversion rates increase dramatically. Visitors to the client’s websites are now four and a half times more likely to complete the entire transaction and make a purchase than they were before digital sales assistants were employed.

For the past seven years we have been observing and researching the emerging field of digital humans, drawing on our decades of experience in the visual effects industry. We have worked alongside and consulted on projects with companies that create digital humans, including Pinscreen, Soul Machines, and Epic Games, witnessing firsthand the enormous growth and advances in the field. Within a decade, we believe, managers at most companies are likely to have a digital human as an assistant or an employee.

In this article we explain how different types of digital humans interact with customers and employees, discuss the situations in which using a digital human is appropriate, and present examples of digital employees working in organizations as diverse as accounting giant EY, Yahoo Japan, the Arab Banking Corporation, and the University of Southern California’s Keck School of Medicine.

What Is a Digital Human?

Meet Lil Miquela, a virtual online influencer with nearly 3 million followers on Instagram. The followers fully understand that she is not a real person—just as they know that Alexa and Siri are not “real.” What they relate to is Lil Miquela’s “authentic” and “genuine” personality, according to Isaac Bratzel, the chief design and innovation officer at Brud, the software media company that created Lil Miquela. Her personality is expressed through the products she endorses and the experiences she posts about.

Why do we ascribe a personality to what we know is an artificial construct? Because we can’t help but respond instinctively to anything that appears to be human. Research from neuroscience shows that our minds are attuned to and react emotionally to facial signals. That’s why most people prefer to communicate face-to-face rather than over the telephone. In the case of digital humans, we know that what we see on the screen is an artificial construct, but we still connect instinctively to it, and we do not have to be computer experts to interpret the facial signals and make the exchange work properly.

Digital humans are thus more likely to provide a meaningful experience than other automated channels, and customers are more likely to extend interactions with them beyond their initial search or transaction. Hao Li, cofounder and CEO of Pinscreen, a Los Angeles–based creator of digital humans, explains: “You want the customer to [explore] how things [like clothes] could be and how they might like them to look. From a user experience standpoint, you want to keep them engaged and able to explore the brand.”

Technology that was once deployed only in Hollywood blockbusters is now available to most companies.

Fashion company ZOZOtown agrees. Now part of Yahoo Japan following a $3.7 billion acquisition in 2019, ZOZOtown controls about 50% of the market for mid- to high-end fashion e-commerce in Japan. With Pinscreen’s help, ZOZOtown has deployed a group of remarkably lifelike digital humans to model fashion online and aid in customer fittings.

Another early adopter is EY, which creates digital-human doubles of its partners for use in video clips. The technology’s translation function lets users make versions of presentations in multiple languages, regardless of whether the presenter speaks them or not. Advertising and communications company WPP has used the technology to send out internal corporate videos in different languages, all made without cameras. Victor Riparbelli, cofounder and CEO of Synthesia, the company that creates digital humans for EY and WPP, says that his firm’s platform generates more than 3,000 videos for enterprise customers every day. “It is transformational to enable anyone in the company to produce uniform, on-brand video content for everything from internal training to personalized sales prospecting,” he says.

Companies can create their own simple digital humans for a range of purposes—to make video clips, for instance—by subscribing to or buying licenses from Synthesia or other platform vendors. Users choose a digital human from a gallery of options and apply a simple text script or incorporate it into automated digital channels. If a fully interactive, intelligent digital human is needed, firms will most likely need to partner with a specialist.

When Should You Deploy Digital Humans?

Digital humans are not appropriate for every application. When customers seek a quick transaction, they are likely to prefer traditional user interfaces, chatbots, and voice-only assistants such as Siri or Alexa. But digital humans can be a much better choice when it comes to communicating complex instructions or describing features of a product. This is why YouTube instruction videos—rather than pages of text—are so successful. Someone searching for clothes online might welcome seeing the outfit on someone who looks like them to get a feel for how the items go together and whether the look reflects who they are. In such cases a digital human will engage the customer more, help complete the sale, and reduce the likelihood of product returns.

These questions will help you determine if a digital human is a good fit for the job you have in mind:

1. Does the interaction involve emotional engagement?

A humanlike face will better address emotional aspects of an interaction, such as providing reassurance or empathy.

2. Are users unsure of exactly what they want from the interaction?

If customers need specific information, then normally they are keen to see the details in written form so that they can quickly digest them. But if they are unsure, scanning pages of text is painful and time-consuming, and they often prefer to be able to ask for help.

3. Is there scope to explore options and consider different approaches or outcomes?

Unlike straightforward online transactions such as buying groceries or booking a movie ticket, interactions such as shopping for clothes or working with a career coach have open-ended trajectories involving give-and-take. When speed isn’t the primary requirement, consumers often like to linger and explore.

4. Could the user benefit from a personalized explanation of a product or service?

Is there value in demonstrating some key aspect?

If the answer is yes to three or more questions, it is worth exploring a digital-human option; if they are all true, it most definitely is.

See more HBR charts in Data & Visuals

What Kind of Digital Human Is Best for You?

If the conditions are appropriate for deploying a digital human, the next step is to figure out what kind of digital human to design. First, consider the purpose of the interaction: Is the primary goal to complete a task or to engage in an experience? In many use cases, customers want to accomplish a task with measurable outcomes. Examples include booking airline tickets, filing a complaint, and retrieving order information. In others, customers want to engage with the company in some way—for example, by browsing through an online store, enjoying entertainment, or having a therapy session.

Second, consider the depth of the interaction: Is it personalized to each customer? In some cases, a person has regular exchanges with the same digital human, which “learns” and “remembers” the customer and becomes more and more personalized over time. In other cases, regardless of whether the interaction is transactional in nature or experiential, no personal relationship is developed between a specific customer and a specific digital human.

Mapping these two factors on a 2×2 matrix produces four categories of digital humans.

See more HBR charts in Data & Visuals

Virtual agent.

A virtual agent serves multiple users and does not develop a personal relationship with them. The agent’s role is to complete specific, onetime tasks. For companies already using chatbots, virtual customer-service agents may be the logical next step. They have all the advantages of chatbots and amplify them through their realistic humanlike appearance. They can respond in any language and can tailor their appearance to the background or ethnicity of each customer.

Companies are also using digital instructors to engage employees in various types of training. Synthesia provides organizations with platforms for generating videos or professional presentations using noninteractive digital humans, obviating the need for actors, film crews, or expensive equipment. At U.S. international airports, for example, you may encounter a digital human giving instructions on how to clear security. Such videos can be developed directly from written text with text-to-speech tools. Victor Riparbelli sees real demand here: “If you are a warehouse worker and you get the choice between reading a five-page PDF manual or watching a two-minute video, it’s a no-brainer. They don’t care about how it was created.” Not only are the videos less expensive and less time-consuming to create, he says, employees prefer the experience and remember more of the content.

A more sophisticated form of virtual agent is the digital patient, which the University of Southern California is researching for use in training the next generation of doctors and mental health professionals. Digital humans can simulate patients experiencing specific symptoms with a high degree of fidelity and realism. For example, they are capable of mimicking flushing, breathing responses, facial responses, slurred speech, and symptoms of PTSD or brain injury. Health care organizations could hire actors instead, but that approach cannot easily be scaled up and quality is inconsistent. Digital patients also provide reliable measurements for training outcomes, such as which symptoms are recognized or missed.

Virtual assistant.

This type of digital human supports the user in completing specific tasks, and over time they develop a personal relationship. There are many types of virtual assistants: personal shoppers, home organizers, and physical therapists, to name a few.

The digital humans shown here, created by Epic Games, Pinscreen, Soul Machines, and Synthesia, have been implemented at companies in industries ranging from banking to cosmetics.Photos courtesy of Soul Machines, Pinscreen, Epic Games, and Synthesia

Digital Domain, best known for creating digital characters for the Disney Marvel films, is developing digital assistants for Zoom, the video communications company. Its digital human Zoey, for example, attends Zoom meetings and monitors the conversation. Zoey can be cued to join the conversation by the phrase “Hey, Zoey” and deactivated by “Thank you, Zoey.”

While activated, the assistant can answer questions and arrange schedules. Because Zoey has an active monitoring memory, she can associate comments and personal profiles with specific meeting attendees. Zoey can translate conversations into text documents and produce meeting summaries. She also analyzes the meeting content with natural language processing, especially sentiment analysis, and responds with appropriate facial expressions and micromovements, such as nods and eye glances, demonstrating her attentiveness and engagement.

Another example comes from a study of military veterans, which found that many people prefer to provide personal or sensitive information to a digital assistant. In the study, veterans wanted their doctors to know about problems they were experiencing but were reluctant to speak openly about their issues. They found that describing their symptoms to a digital-human assistant felt less intimidating and allowed them to communicate the information in a way that was more comfortable for them.

Digital humans are also often preferred in other contexts, such as education, human resource management, and dispute resolution. That’s partly because they do not get angry, impatient, or distracted and can moderate their tone and speed of speech to match users’ needs.

Of course, virtual assistants cannot replace humans for complex, nuanced interactions, such as explaining a serious illness or detecting mental health issues, because they lack any true understanding of what is being communicated. The digital human may question a person on a topic and react to a range of responses, but it never “thinks” in any real sense of the word.

Virtual influencer.

Virtual influencers supply their human followers with experiences, but they are not personalized. Everyone sees the same content on social media, for example. Although virtual influencers are carefully designed to appeal to specific user segments, any relationship a person might feel with them stems from that person’s projection and not from any individual customization.

Virtual influencers operate in much the same way that human ones do. They share images of their experiences and post virtual photos of themselves having a great time somewhere wonderful in order to market a company’s products. They have two key advantages over their human counterparts: They are much cheaper and require far less management. The people who follow them have no problem with the fact that the influencers aren’t real because it is the experiences that they care about.

Virtual influencers have been especially successful in the fashion industry: Lil Miquela’s carefully curated online presence has helped Brud achieve a valuation of more than $125 million prior to being acquired by Dapper Labs. ZOZOtown also deploys a range of virtual influencers to help market the company’s products.

In the Middle East, Soul Machines has created a digital celebrity in Fatema, the public face of Bahrain-based Arab Banking Corporation. She was one of the first virtual, AI-enabled customer agents to humanize financial information. She talks to customers and responds, providing a humanlike presence. As a virtual influencer, Fatema is involved in Bank ABC’s social media efforts and has a presence on Instagram aimed at helping customers feel more connected to the bank and learn about new offerings.

Virtual companion.

A virtual companion develops a deeply personal relationship with the user. Rather than focusing on completing tasks, the primary goal is simply to be with the user. Like a digital assistant, a virtual companion does not get frustrated or bored and has no competing demands on its time.

A promising application is in eldercare. Virtual companions enable older people to stay in their homes longer, which is known to be better for their physical and mental health. They are also much cheaper than assisted living or nursing homes. Digital humans not only provide companionship to stave off isolation, they also remind people to take their medication, facilitate communications, and alert medical professionals in case of emergency.

Similar opportunities exist in education. Children are more engaged when they watch other children. Thus a child-aged digital instructor could, at times, be a more effective tutor than a human adult teacher. The digital instructor could even be slightly older than the actual student—perhaps six months older—and shown to have mastered the subject, demonstrating that it is possible and serving as inspiration.

Designing Digital Humans

In creating a digital human, firms must address two questions: Does it look right? and Can it communicate appropriately with users in the given context?

Appearance and sound.

Appearance includes human features (such as nose shape and eye color), demographic characteristics (such as gender, age, and ethnicity), and taste (such as tattoos, makeup, accessories, and clothes). How the digital human sounds depends on its accent or dialect, vocabulary, tone of voice, and other factors. A great deal of design time is spent creating a “personality” for the digital human through appearance and sound.

The personality should fit the context. A user may feel more comfortable accepting medical advice from an older and wiser-looking digital human wearing a lab coat. However, that digital human would be ill-suited for a sales assistant role in the fashion industry. A smiling, perky digital human in customer service might annoy complainers. Conversely, a serious, mature digital human would hardly be a convincing virtual influencer for a hip lifestyle brand.

Next In

The New Human-Machine Relationship

A Smarter Strategy for Using Robots

[

Automation should focus more on flexibility than on productivity.

](/2023/03/a-smarter-strategy-for-using-robots)

Brand image must be kept top of mind in designing digital humans, which serve as company ambassadors. Appearance and personality should reflect core company values and reinforce the brand image; mismatches may confuse customers or even damage the brand.

Digital humans must have a minimum level of human realism in the way they look and sound, otherwise they will be off-putting. But this is far less of a problem than it used to be. Rapid advancement in graphics technology and AI have dramatically improved standards. Tech that was once deployed only in Hollywood blockbusters is now available to most companies. Recent studies show that today’s realistic digital humans are considered by customers to be more trustworthy and are more likely to instill a sense of affinity or trust than other visual forms such as visual chatbots and animated characters. While people don’t require visual perfection, they want their virtual counterparts to be expressive and appealing, because what they’re seeking is an authentic interaction—which brings us to the second design challenge.

Communication.

Soul Machines’ Mark Sagar argues that for face-to-face interaction, the challenge is enabling digital humans to process multiple signals of intent and information. He has spent years working on how digital humans respond to and give verbal and nonverbal signals. “You’ve got to combine everything,” he told us. “Speech-related gesturing, iconic gesturing, semantic gesturing, and all kinds of body language.” Designers can’t rely on scripted or branched prerecorded dialogues, because a conversation can head in any direction at any time. “Every time you add new dimensions that adds to the number of conversational combinations the digital human has to address.”

Soul Machines uses advanced AI to circumvent this problem. The digital humans work from text and sentiment analysis and from camera input containing the human counterpart’s emotional feedback, such as body language and facial expressions. Sagar is the first to admit that digital humans can never know a user’s emotional state of mind; however, the more accurately they are able to analyze users, reflect their concerns, and apply machine-learning programs, the more meaningful they become to the customer.

Understanding nuances in human conversation can be challenging. Researchers at Digital Domain have made great advances in natural language understanding with their digital humans, but some things such as irony are still very hard to navigate. The problem is not just understanding and emotionally interpreting a person’s comments; the AI engine also has to take into account previous interactions and the broader context of the situation. As impressive as the advances of the past few years have been in AI, companies must have a realistic understanding of the uses and limits of digital humans.

. . .

Digital humans are disrupting how firms engage with customers, suppliers, employees, and external stakeholders by offering personalized attention at scale. They are also being applied to internal corporate processes by transforming video production, training programs, and administrative support. And an emerging set of providers are creating, training, and supporting the deployment of many new types of digital humans. Firms that embrace this new technology will lower costs, increase revenues, and gain a sustainable first-mover advantage that slower adopters may find hard to overcome as customers become attached to their digital counterparts.