The Power of Inner Monologue: How Language Can Revolutionize AI Learning

Introduction

Artificial intelligence (AI) has made significant advancements and can now perform complex tasks that were once thought to be exclusive to humans. However, AI still has limitations, and researchers are constantly exploring new ways to improve its performance. A recent study has shown that tying language to actions can enhance AI's ability to learn complex tasks, potentially enabling it to learn from instructional videos and gain more human-like cognitive abilities.

The Experiment

Shengran Hu and Jeff Clune, computer scientists at the University of British Columbia, designed an AI agent that could carry out missions in a virtual 2D world. The agent had two components, each containing a neural network. The first component used visual information, the mission, and the agent's previous thoughts to create a new thought, such as "open blue door to explore" or "go to purple box." The second component combined the thought with the mission and observations to choose actions.

The AI had to be trained, and to do that, the researchers relied on a large data set of missions completed by a bot designed specifically to solve such problems and also to generate text that described step by step what it was doing. For comparison, they also trained an agent using an existing technique called "behavioral cloning." It learned to predict actions based on the mission and observations, without the benefit of articulated thoughts.

The trained agents were then assigned new missions in new mazes. On the most complex missions, the agent trained to imitate both actions and thoughts succeeded about 80% of the time, whereas the one trained to imitate only actions succeeded only about 30% of the time. Hu explains that language helps one learn concepts at different levels of abstraction and then combine them in new ways. The advanced agent could even rethink plans after encountering unexpected obstacles, which Hu thought was "cool."

The Benefits of Thought Cloning

Beyond improving performance, an AI trained using what Hu and Clune call "thought cloning" offers users something rare in the world of neural networks: a chance to see what the agent is thinking. This should help debug systems and also benefit safety, the researchers say: If an AI is planning something dangerous, a human operator can tell and can intervene. With existing, mute systems, Hu says, "when you see your agent rushing to a bank, you don't know whether it will try to rob the bank or just try to get some cash for you."

In tests of such "precrime intervention," the researchers showed they could stop an agent before it performed a prohibited action such as touching a red item. They just added a rule triggered by its thoughts, without having to retrain the model. Clune was surprised by how well the approach worked. "You probably pick up knives all the time without having the words 'I'm going to pick up the knife' show up in your head," he said. But the agent consistently premeditated, enabling safeguards to kick in.

The Future of Thought Cloning

The researchers trained their systems from scratch. Hu says in the future they might try to add an inner monologue component to pretrained models such as OpenAI's GPT-4 Vision, which already contains elements of general knowledge and reasoning. Ultimately, they hope their thought-cloning agents will learn useful skills from the mass of information in sources like YouTube videos, where a narrator describes each step. "Every single video of somebody saying, 'I will now show you how to make a croissant,' or cook saag paneer, or replace the carburetor on this old Chevy, or fix a flat tire, or edit a photo in Photoshop, or book a flight on Expedia, or build a house in Minecraft"—that would all be fodder for learning, Clune says.

Conclusion

Tying language to actions can enhance AI's ability to learn complex tasks, potentially enabling it to learn from instructional videos and gain more human-like cognitive abilities. This innovative approach could revolutionize the way AI learns and interacts with the world, making it more adaptable and efficient. As AI continues to advance, incorporating human-like thought processes and language abilities could be the key to unlocking its full potential.

https://www.shengranhu.com/ThoughtCloning/