An Interview with Yoshua Bengio: thoughts on neural networks, deep learning and being a dreamer

Yoshua Bengio is best known for his groundbreaking work in artificial intelligence, particularly his work on artificial neural networks and deep learning. He is a Professor at the University of Montreal, scientific director of Mila (the Montreal Institute for Learning Algorithms), and co-founder of Element AI, a company that delivers software products for practical business applications of AI. He recently won the 2019 Turing Award  alongside collaborators Yann LeCun and Geoffrey Hinton.

Tara Chklovski: This is a real honor to sit down with the godfather of neural networks! Thank you so much for doing this interview. To start right off, how would you explain unsupervised learning in simple terms?

Yoshua Bengio: Most easily by placing it in opposition to supervised learning. Let’s say a student is doing homework and she’s trying to produce answers on her own, and then her teacher gives her the correct answers so that she can compare the answers she gave to the correct answers and try to figure out what she did wrong. That’s supervised learning.

Unsupervised learning is, for instance, children playing during recess, when they are just observing, interacting with the world, and trying to make sense of it. Nobody is telling them, “this is what you should be doing” Or, “this is the explanation of what you’re seeing” or “this is the cause of that.” They are just observing and they might do experiments by themselves. Playing is an experiment.

With supervised learning a teacher uses the concepts they know to guide the learner in a much stronger way. You basically go through all the little steps, teaching the high-level concepts that help the learner figure out what is going on. Whereas in unsupervised learning the learner figures things out by observation and exploration and the learner is the one making links between the things they observe so they can understand what is happening.

Tara Chklovski: Is unsupervised learning a more complex problem to work on?

Yoshua Bengio: Yes, and it’s harder to evaluate. If you go back to my example of homework, if you have the right answers, you can just score the work the student submitted as correct or incorrect or partially correct, but for unsupervised learning, the child is playing outside, and how do you know that it’s good? So usually we  measure the value of unsupervised learning by also doing some supervised learning and seeing whether the learner performs better on the supervised part as a result.

For instance, in unsupervised learning you may start by training your learning model with ‘unlabeled images’ – where you don’t indicate the right answers, or what to do with them. Then, you give it a supervised task, where you ask it to identify an image and determine if it’s a cat, or a dog, or a chair. If it gets better at it because it was exposed to all of that unlabeled data before, then you could say that the unsupervised exercise was valuable.

Tara Chklovski: What are some practical applications you hope will emerge from your work?

Yoshua Bengio: I have lots of hopes! I’d like to see more applications of machine learning and deep learning that impact many people positively.  The humanitarian problems [we try to solve] should be coming from the people on the ground. We have tools that could be useful for solving big problems, but only by talking to people who are actually addressing social problems do we amplify the potential and impact of technology.

Recently there’s been huge progress in computers’ abilities to understand images. We can do things that were impossible just a few years ago. I’m not saying it always works, but there’s been incredible progress.

But there’s no magic in all this – you need data. It’s going to be difficult to solve problems, especially when there’s no existing data or  path to collect that data. There’s also the additional obstacle of access to these tools. We’re building bigger neural nets, and they require specialized, expensive hardware and computers. So if people want to use current machine learning technology, they need computers that are getting more and more expensive. We want to create an international organization, AI Commons, to help connect problem owners, problem solvers, and funders, to apply AI for the benefit of all, and especially of those who would need it the most.

Tara Chklovski: Big problems like the ones you have tackled and are working on can take many years; you hit many dead ends or obstacles like the ones you just mentioned. How do you motivate yourself to keep going?

Yoshua Bengio: I’m a dreamer. How do you motivate yourself? I don’t know – it just happens. I believe we can build a much better world, that the possibilities are huge, and that we underestimate them. I believe there are a few simple principles that explain our intelligence, and if we make progress in understanding them, it could have a hugely beneficial impact. Or, it could be badly used, so it’s important people understand that.

Tara Chklovski: How do you think people can go about finding those basic principles?

Yoshua Bengio: Well, that’s what we do here. And that’s what researchers do. That’s why we need to invest in research, that’s why we need to invest in university education. I think it is particularly important in developing countries that they invest in graduate education.

Tara Chklovski: I think that to become an explorer, you do need some basic confidence.

Yoshua Bengio: Yes, absolutely, part of the answer to your bigger question is self-confidence. If I was a person with very little self-confidence, I would not be in this career. There are good and bad aspects of that, but the good part is that it keeps you going. The bad part is that many people who could contribute intellectually are not doing it because of a lack of self-confidence. Competition is playing too much of a role in academia and research.

Tara Chklovski: And how do you build that self-confidence?

Yoshua Bengio: Parents. I have parents who believe in me. I mean, very simple, basic things of loving and trusting and being an example of emotional maturity. Especially when the children are very young, of course.

Tara Chklovski: I think it gets hard when you don’t have an ecosystem of role models and technical support that can help you to not only build self-confidence, but to also persevere through obstacles.

Yoshua Bengio: Okay, here’s another thing that’s more subtle than the usual story about being loved by your parents. There were times in my life, in my young life, and in my young adult life, when I experienced the pleasure of understanding things – the ‘eureka’ moment. If you start experiencing that enough, it becomes like an addiction, but it’s a good one. You get pleasure out of understanding things, you get pleasure out of suddenly realizing that you can have ideas, that your brain is powerful, that you can solve problems.

Tara Chklovski: How do you convince people to believe in you or in an idea?

Yoshua Bengio: It took a long time to convince the AI research community. Being patient and persistent and not distracted by the trend of the day is important.

Again, it’s about self-confidence, and self-confidence is not just about “I’m good and you should listen to me.” It’s about listening to your inner voice and trusting in it despite what others say. I’ve seen students who were really smart but didn’t trust their impulses, their intuition, their inner voice, and those students have difficulty succeeding in a research career. If you don’t allow yourself to say things that could be wrong and make mistakes, you’re not going to be an entrepreneur, you’re not going to be a researcher– you’re going to try to find a place in society where everything is safe, and easy. And that’s a loss for everyone.

Tara Chklovski: Sometimes it takes experience to be able to understand whether you have the right tools, or right framing for a question.

Yoshua Bengio: What you learn in university or elsewhere is how to be both open to your own intuitions and willing to take risks, and be able to listen to reason, to evidence, to others’ arguments.

A good scientist needs to be able to see the problems and obstacles, but if you only focus on the problems and you don’t see the potential for something new and important, then it doesn’t work. You need a good balance, and you need rigor.  You have to look for something coherent and consistent, and things need to be thought out rationally, but the first thing that comes is intuition. It’s something that comes from inside, it’s in your experience, your brain is bringing up this new idea, and then, you can apply reason and improve it or discard it. But, if you don’t let that idea out in the first place, then it’s not going to work.

Tara Chklovski: You’re not inspired either.

Yoshua Bengio: Yes, there is a strong emotional aspect. I have an emotional attachment to my ideas. And that’s the thing that keeps me going. And sometimes, it’s like a love story. Sometimes it works, sometimes it doesn’t work, and you have to be ready to go through the disappointments and exhilarations. And the time isn’t wasted if it doesn’t work – because you learned something.

Tara Chklovski: Last question: What advice would you give children and parents if they are trying to figure out what problems they can solve? What is the first step in trying to find a problem?

Yoshua Bengio: Write down a number of problems that you have where technology might help. Because if it’s something that comes to your mind, it’s probably something you care about, and it’s going to be much easier and motivating to work on. And then, I would take that list and talk to people who understand the relevant technology, or experts of whatever solution you are considering, and see if there’s potential match between the problem and the possible solutions. That’s what I would do.

Tara Chklovski: That’s a good start. Thank you so much for your time!