Virtual humans and decision-making: A conversation with Stacy Marsella

TC: Oh that’s so interesting. What’s one application of that sort of social training?

SM: For instance medical schools are interested in using virtual humans to train medical students in bedside manner. We also had a project a couple of years ago where we were looking at virtual humans which would interview people to triage whether or not they might be suffering from post‑traumatic stress. If the interaction with the virtual human could detect some indication a patient might be suffering from post‑traumatic stress, then the patient would be sent to a medical professional that would work with them.

TC: Very cool. What are some of the other projects are you working on?

SM: More recently I’ve been looking at decision‑making behavior of people under stressful conditions. We have a project with engineers in the College of Engineering where we’re looking at what factors lead people to decide whether or not to evacuate in the face of an approaching natural disaster like a hurricane. We collect survey data from people right before hurricanes like Hurricane Irma and Hurricane Harvey, and try to model what factors seem to be influencing them. Is it their friends’ decision, is it the information they’re getting over media? What’s actually influencing the decision of whether or not to evacuate? And once you have some sense of what’s influencing people to evacuate or not evacuate perhaps you can craft better messages people do what authorities think is the most reasonable things to do in the face of a particular disaster.

Another area we’re interested in is people’s decision-making in supply chains. A supply chain is the sequence of steps a product goes through as it produced, proceeds to distributors and eventually ends up with a customer. Again with people in Engineering, we’ve been looking at pharmaceuticals, and the persistent shortages in pharmaceutical drugs. For instance there’s a shortage right now in saline solution, and there have been persistent shortages in saline solution over the years. We’re interested in looking at the supply chain and modelling the decision makers and seeing what’s causing these shortages. Now, in some cases it’s obvious – for instance Hurricane Maria heavily damaged Puerto Rico, and Puerto Rico is a source of saline solution. So, the current shortage is pretty obviously a result of a natural disaster. But there are times where it’s not clear what the cause is. The role I have in that project is really looking at the whole question of what drives the decision-making. How are people making decisions? How are those decisions actually influencing shortages?

TC: So what advice would you give children as they try to find a problem in their community that they can solve using technology? You’ve talked about a very wide-ranging set of problems that you are working on.

SM: I think the first thing I would do is try to figure out what the problems are and what is causing them. And how are people helping to solve those problems, or making the problems worse in some way?  And then figuring out, is there some information that could help make the response to this problem more effective?

So, it would depend on the community and what the problems were, but understanding, for example, something like how the community responds to a natural disaster, would be really very interesting. And so they would do something similar to what we did, which is survey people. And we collect data on what’s driving their decision‑making and then we put that into various kinds of statistical or machine‑learning models to actually figure out the underlying decision process.  Can we model what is driving the decision‑making? Can we make predictions about what people’s decision‑making will be? How many people, for instance, would evacuate in the face of a hurricane? How might that decision be influenced?

TC: So, in your opinion what are some characteristics of a good problem?

SM: Well, first of all, you have to be able to identify that there actually is a problem, and that it’s impacting people. But I think the other thing that’s really important is understanding the possibility of getting access to some data that’s going to help understand the problem and lead to address it.

So, for instance, you mentioned students interviewing people. Well, is there a way, for instance, for the students to use social media or to use things like Amazon Turk to gather lots of information to understand and address the problem? And so that’s the key – can you get data to actually study the problem and potentially find out what its causes are and potentially how it can be solved?

TC: Have you used Amazon Turk?

SM: Yeah. We’ve used Amazon Turk in the hurricane data actually. We used Twitter data, but we also use Amazon Turk so for instance right after a hurricane passes through we will run surveys in Florida or Texas, wherever the hurricane hit, over Amazon Turk. And I think that’s a really interesting way for anyone, including students, to get data really quickly, and then analyze it.

TC: Wow. That’s very interesting to know. Okay. So, I’m curious, what do you find difficult in your research, and how do you overcome that difficulty?

SM: The most difficult thing is always getting the data. And the other thing is – well it depends on the particular thing I’m doing. So, if I’m building models of virtual humans, the things that are difficult are getting the virtual human to understand what people are saying, getting the virtual human to behave, and getting gestures to look natural. For something like these large surveys, it really is a question of getting access to people’s internal state, essentially. Why someone chose to evacuate or not isn’t something that they are necessarily going to answer accurately, so you always have to worry about that. Because what you’re trying to do is figure out something about their internal mental behavior, and that internal behavior you don’t have direct access to.

TC: That’s very interesting. So, were you able to do this in a way where you could figure out, okay, this is what somebody said before and this is what they actually did?

SM: We haven’t done that. But other people have actually surveyed them moment‑by‑moment and seeing how their decision-making evolved.

TC: And what was the accuracy of how humans predict what they’re going to do?

SM: Well, it’s never going to be extremely accurate, but when we build models of these things we can get up to the low 80 percent in terms of accuracy. What you’re trying to do is get enough accuracy so that the policy-making is improved. And often it’s not about prediction. You’re using the prediction as a measure of how good the overall approach is, but you’re really not trying to predict how people are going to behave. You’re trying to figure out what factors influence their decision-making, because in the end what you’re trying to do from a policy perspective is figure out how to create a context in which they do what’s best for them.

TC: I see, I see. I have one more question: what field of AI or aspect excites you the most?

SM: Hmm. Well, what excites me the most is probably rather unique. There’s two types of AI researchers historically:  those who don’t really care about how people do things, and those of us who care about how people do things. And so, what excites me the most is really figuring out if I can use these computational techniques to model human behavior. That’s what excites me the most.

For instance, I build models of human emotion – what are the factors that lead to emotion, how does that actually influence decision-making, how does that influence expressive behavior?  Those kinds of things, that’s the area that interests me the most: Using computational models to essentially develop better understandings of how humans think and act.

And that’s an offshoot of AI. It’s not the main thread. The main thread of AI that most AI researchers pursue, is this notion of how can I do something that you or I might characterize as intelligent behavior?  What is the best method for doing that? And they don’t necessarily care for it to be aligned at all with how humans do it. So how humans learn is not typically the way AI systems learn. Humans can learn incredible things from very limited data. AI systems do not. They use lots of data.

TC: It’s like a mirror to yourself.

SM: Yeah, it’s a mirror. I’m very interested in using computational methods as essentially a way to understand what’s going on inside people’s heads.

TC: Very cool. That is very unique.  Thank you so much, Stacy. This was a very, very interesting conversation.