Training an AI Agent for 76 Hours Reveals Surprising Reinforcement Learning Lessons
A developer trained a from-scratch deep Q-network on a side-scrolling runner game for 76 hours, running 226 million steps to measure learning, failures, and network plasticity.
A developer recently undertook a unique reinforcement learning experiment, building a Deep Q-Network (DQN) completely from scratch on a laptop CPU using NumPy. Without relying on frameworks like PyTorch or Gym, the system trained for 76 hours across 226 million steps in a side-scrolling runner game featuring low and tall blocks, gaps, and sliding obstacles. Over the course of the experiment, the agent learned to navigate increasingly difficult tiers, eventually outperforming a hand-written rule-based player at higher levels.
Running without a GPU meant every gradient was visible, operating at a median speed of 870 to 1,180 environment steps per second. The project aimed to explore continuous learning from failure, utilizing specialized mechanisms such as a death bank that restarted episodes from moments prior to a crash, and prioritized experience replay. However, 15 ablation runs revealed counterintuitive results regarding which features actually helped performance.
Fixing Fundamental Bottlenecks Early
Before analyzing complex training dynamics, three major design flaws had to be resolved. First, the agent initially struggled with gaps because its observation rays did not pinpoint gap edges. Expanding the terrain scan to sample every half unit for 12 units ahead solved this issue entirely, proving that an agent's sensory ceiling limits its potential regardless of learning algorithms.
Second, exploration hindered performance evaluations. With a 3% random action rate, the agent failed frequently during training episodes, making the curriculum promote levels based on flawed data. Implementing greedy evaluation exams resolved this discrepancy. Finally, policy churn caused the neural network's greedy actions to shift while updating during an episode. Using Polyak averaging to act with an exponential moving average of the weights stabilized performance significantly.
Ablations and Catastrophic Forgetting
When testing the value of each component through ablation runs, the results challenged common assumptions. Clever mechanisms like the death bank and prioritized experience replay actually degraded overall held-out success compared to a plain configuration. Although local metrics showed the death bank successfully resolving thousands of failures, the narrow focus cost the agent globally.
Conversely, replaying old tiers proved crucial. Without old-tier replay, the agent suffered from catastrophic forgetting, rapidly clearing high tiers while nearly completely forgetting how to play the simpler, early tiers. Developers working on advanced applications can easily experiment with top AI models cheaply through one API at https://apixoai.online.
What it means for developers
The experiment offers practical insights for machine learning engineers dealing with continual learning and network plasticity. Over 226 million steps, the network exhibited a well-documented loss of plasticity, with 21% of hidden units dying and feature rank dropping significantly. Implementing continual backprop—which periodically replaces the least useful hidden units with fresh ones—successfully maintained network health metrics.
However, the connection between a healthy network and better gameplay proved nuanced. While continual backprop prevented performance dips on hard levels in some seeds, a degraded network could still perform reasonably well. Ultimately, the research highlights that local metrics can look exceptional while failing to improve global outcomes, emphasizing the necessity of rigorous ablation testing and evaluating multiple seeds before drawing firm conclusions.
Source: I Trained an AI Agent to Play a Game for 76 Hours (And It Learned From Its Mistakes) — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
