Hey everyone!
I’m currently finishing the next release of Update All, but since I haven’t posted for a while I decided to take some time to share something that’s really blown my mind these past few weeks. It’s a bit of a special tale, about something I learned thanks to the master’s degree I’m currently studying in AI (I’m almost done!), and it’s pretty much off-topic although Atari 2600 emulation plays a big role in it. I hope you like it, though!
AI Agents hype
Since this tale is gonna be all about agents, let’s start first with a small introduction on what agents are. If you know what they are, feel free to skip this section and move to the next one named “The state of the art”.
AI Agents are becoming kind of a popular buzzword nowadays as it is being sold as a productivity tool to automate some of your workflows without having to code. In simple terms, an agent is just a smart bot. It receives stimuli from an environment and takes actions in it to try to achieve goals. And it’s “smarter” than your typical Bomberman CPU because it has some neural networks in its decision routines that have been trained with machine learning techniques instead of being just a set of “if-this-do-that” rules like the “dumb” bots do. LLMs like ChatGPT 5 or AI Art Generators are not considered agents, as they just perform one action when given one prompt.
Agents constantly "move" in their environment. In futuristic movies, an agentic AI is what’s in the brain of robots like Terminator, and it also shows up in virtual worlds like the Agent Smith in The Matrix. And current productivity tools with agentic capabilities can autonomously plan and execute multi-step tasks across digital environments—such as browsers, file systems, and apps—adapting their actions in real time based on changes in the environment.
An agent observes an environment and acts within it until it reaches a goal.
In my studies, I’ve been playing with implementing agents that play Atari 2600 games until they outperform seasoned players so, I’ve come to understand the subject a bit more than your average AI bro.
The state of the Art
Now that we have a clear picture on what an AI agent is, I’ll talk about an area of research that blew my mind.
I assume you all know that AI models (neural networks) have to be trained on tons of data before they are useful, the same happens with agents. In fact, they use AI models under the hood. But when training agents, you face a big additional problem, you also need to run an environment, as they learn from the data that's observed from the environment. And running such an environment is expensive in terms of resources and time, especially if the environment is the real world (in this case, instead of running the environment, you are sampling it with cameras and sensors).
For learning to really happen, you need to run the environment an obscene number of times. Because of this, researchers opted for training agents on “cheap” virtual environments that consume few resources and that they can fast-forward. This is why a lot of researchers in this area are using Atari 2600 emulators to do the training, such as Stella, which is a cycle-accurate core that runs on CPUs and can run up to 30000 frames per second, or CuLE, which is not that accurate but runs on the GPU and can go up to 180000 FPS. Despite this, your environment can still be the bottleneck of your training, especially if you are designing your agent to interact with something more complex than an Atari 2600 game.
The area of research I’m talking about here proposes a very crazy but smart solution for the environment bottleneck problem. The solution is conceptually stupid simple: remove the environment (which is the expensive part) during the training. Which doesn’t make any sense, right? An agent by definition does need an environment to be trained. So what is the trick? The trick is dreaming. We can make the agent dream its own fake environment so that it doesn’t need any direct observation from a real one to learn.
The dreams happen in its own neural networks. They generate new data, in a process similar to what we see in generative AI tools, like those that are able to generate videos, images, music, and text. These dreams instead generate new observations of a non existing environment and feed it to the agent itself. Sounds kind of nuts, but it could well be something that we humans also do during our dreams. In fact, a number of top neuroscience dream-theories are the main source of inspiration for the researchers to try this approach. And surprisingly, it works for our agents!
Snapshot of the dream of a proof-of-concept agent, as shown in the later referenced article “World Models”.
What are Dreamer Agents
To understand how “dreaming” really works in these agents, we need to check the architecture in terms of neural network models. I’ll do my best to explain it with simple terms, but first we need to talk about how basic agents work.
A very simple non-dreaming agent may consist of two models, one for observing the environment, and another for planning the action to be taken by the agent:
The observer model is usually pretty simple, because it’s often just image processing — for example, the agent gets a screenshot from the Atari 2600 emulator and processes it. This observer part digests the current game frame and produces an encoded input for the planner model, which we’ll refer to as “processed observation”. It’s basically acting like “the eyes” of the agent.
The planner model is a deep neural network that outputs an action — for example, pressing the left direction button in the Atari 2600 controller. It determines how the agent should move given a processed observation of the environment. In other words, it’s “the brain” of the agent.
Having first a model observing the environment is an important part of the agent because getting this processed observation greatly simplifies the data coming from the environment. The observer model just outputs the important parts of the observation to the next model, so that the planner can be more focused and therefore much easier to train. Making training easier means consuming less resources: less data, less time. This is essential to make these agents feasible, as it can turn millions of years of hypothetical required training time into just hours in many cases.
The architecture of the discussed simple agent that consists of two models.
For the dreamer agent, the architecture is pretty similar, but instead it consists of three models. We keep the observer and planner models, but we add “the predictor”. The role of the predictor model is to generate a prediction on how the next processed observation would look given the current processed observation and an action to be performed by the agent. In formula form:
Predictor model:
Current Processed Observation + Action → Next Processed Observation Guess
Notice that it consumes processed observations, and an action, and outputs processed observations... but the output is just a guess! It isn't guaranteed to be right — it’s just the model's best prediction, based on what it has learned so far. Sometimes it's close to reality, sometimes it's off.
This new predictor model fits into the agent architecture to produce an additional input to the planner: A speculation on the future of the environment! That way the planner model also has some “intuition” about how the environment would look like given the action it’s considering. This intuition empowers the planner to make better actions, and therefore become more skilled, which makes the agent more successful at their tasks.
The new dreamer agent architecture, with the predictor model addition.
The way the predictor model improves the agent is easy to understand with an example. Imagine that the agent is playing Space Invaders, and the predictor model shows that in the next frame it would get hit by a shot if the considered action is taken. Let’s say that action was “moving left”. Once the planner gets this input, it can decide to move right instead, making the agent able to survive.
One of the advantages of this architecture is that you can train each model separately. The observer model, and the predictor model need the environment to be trained, but far less than the planner. So an efficient training strategy would be to train both models in isolation, using random actions to modify the environment, and once they are kind of competent, start training the planner by connecting it to both trained models and to the environment. It will learn quicker that way. As you can see, this architecture saves a lot of training time and environment iterations as it is, but the point of this research is that we can do much better. In fact, we can dream!
We can take even more advantage of the fact that we can train the observer and predictor models in isolation. With a well trained predictor model, we can temporarily modify the architecture of the agent during the training of the planner to simplify the training further. We do so by skipping the running of the environment and the observer model altogether.
The dreamer agent “dreams” to train the planning model.
The trick is that, if we connect the predictor model to itself, we are guessing the next processed observation over a current processed observation, even when the current processed observation is also a guess. And we can keep doing that as much as we want. With this setup, we just need to kickstart the predictor model with a single real observation, and then unplug the observer model and the real environment, like the figure shows.
This loop produces a hallucinated environment. A dream for the planner to be used during its training, instead of running the real environment. This makes training super efficient, as only the predictor model and the planner model have to run on the GPU. Thanks to that, we are sparing all the resources required to run the observer model and the environment during the training of the planner. And that way we can put all these extra resources into training the planner, which is the most expensive part to train, and the most valuable too — remember, it is the brain of the agent after all!
Once the planner model is trained, we can deploy the full architecture for the agent (with the three models plugged in) to operate on the real environment with great results.
Learning a World Model
You may wonder how a hallucinated environment may substitute for the real environment. We surely all know how AI hallucination gives bad results in ChatGPT and other chat bots. But the truth is, what the predictor model really learns in this case is a “world model” of the environment, so it can be quite accurate.
In the case of an Atari 2600 game, the predictor model will somehow learn game engine physics, gameplay mechanics, the game levels and frame rendering. And in the case of a model trained by sampling the real world, it’ll learn how some physics work in our real world. So more than a hallucinated environment, this agent is imagining the environment during the dreaming phase. Perhaps like what we do in our dreams too!
NOTE: In the literature, the name for the predictor model is actually the “ world model”, so we can use both terms interchangeably. From now on I’ll start to use the term world model instead to reflect the added semantics.
Maybe this world model is also similar to how our intuition works. We can catch a fast moving ball instinctively and very competently and we do it without having to calculate the parabolic trajectory with equations of motion and so on. We just guess that the ball will be somewhere in the immediate future, and move the hand in advance to catch it without doing any maths. Maybe this is also thanks to our own inner world model, feeding good guesses to the more sophisticated areas of our brain that quickly decide our movements to catch that ball.
A World Model, from Scott McCloud’s Understanding Comics.
This is just the beginning
Dreamer agents are right now at the forefront of AI research. It could well be one of the key pillars in bringing truly intelligent robots to reality.
If you liked this article and would like to expand more, I recommend you to check out this previously mentioned “World Models” article which has interactive parts that show what dreams look like. There, they show that they built a proof-of-concept dreamer agent with a very simple planner that outperformed other agents at the time (year 2018), and inspired this area of research.
And if you’d like to check the state of the art in this subject, see the Dreamer V3 (by Google DeepMind) and MuDreamer (by the University of Würzburg) agents. Dreamer V3 is a refinement of the Dreamer architecture, while MuDreamer integrates ideas from MuZero (itself a successor to AlphaZero, the agent famous for mastering the game of Go) to produce a highly adaptive agent.
As of today, the best benchmark results come from the EfficientZero family, which also builds on the MuZero paradigm. EfficientZero does not perform explicit “dreaming,” but future agents may combine both approaches to push efficiency even further.
Closing up
That’s all! I hope you liked this introduction on such a cool concept, even though it’s not related to my work on MiSTer. Note that I introduced a couple of small inaccuracies to be able to explain hard concepts in simpler ways, but the gist of the subject is kept intact.
I loved sharing this, but I don’t plan to make more posts on AI. In any case, if you liked this one, please let me know in the comments and I will consider making more in the future.
See you very soon!