Training an RL agent to play Atari Pong using Deep Q-Networks (DQN), plus Double DQN, and optional difficulty scaling (difficulty 2 and 3). Built on Gymnasium + ALE and a classic CNN Atari-style architecture.

• Project layout
• Install
• Environment
• Methods
• Experiments and results
  • Baseline DQN experiments (1–5)
  • Extra credit: Double DQN
  • Extra credit: Difficulty 2 and 3
• Run training
• Evaluate a saved model
• Hyperparameter sweeps
• License

pong-dqn/
├─ README.md
├─ LICENSE
├─ requirements.txt
├─ .gitignore
│
├─ docs/
│  ├─ report.md
│  ├─ figures/
│  └─ videos/
│
├─ src/
│  ├─ env.py
│  ├─ preprocess.py
│  ├─ replay_buffer.py
│  ├─ models.py
│  ├─ agent.py
│  ├─ train.py
│  ├─ evaluate.py
│  ├─ config.py
│  └─ utils.py
│
├─ scripts/
│  ├─ train_dqn.py
│  ├─ train_double_dqn.py
│  ├─ train_difficulty.py
│  ├─ eval.py
│  └─ sweep.py
│
├─ experiments/
│  ├─ README.md
│  └─ .gitkeep
│
├─ models/
│  ├─ README.md
│  └─ .gitkeep
│
└─ tests/
   ├─ test_preprocess.py
   ├─ test_replay_buffer.py
   └─ test_model_shapes.py

python -m venv .venv
source .venv/bin/activate

pip install -r requirements.txt

pytest -q

• Gymnasium ALE environment: ALE/Pong-v5
• Observations: raw RGB frames (210×160×3)
• Actions: discrete actions (usually 6 for Pong)
• Preprocessing:
  • grayscale → 84×84
  • normalize to [0, 1]
  • stack 4 frames → (4, 84, 84) for temporal context

We approximate Q(s, a) with a convolutional neural network:

• Conv(32, 8×8, stride 4) + ReLU
• Conv(64, 4×4, stride 2) + ReLU
• Conv(64, 3×3, stride 1) + ReLU
• FC(256) + ReLU
• Output layer: Q-values for each action

Key training components:

• Experience replay
• Target network (periodic hard updates)
• Epsilon-greedy exploration with exponential decay
• Huber loss (SmoothL1Loss) for stability
• Gradient accumulation (default: 4 steps) to emulate larger batches more stably

Same network, but target computation uses:

• action selection from the policy network
• action evaluation from the target network to reduce overestimation bias.

The progression focused on:

• Increasing gamma from 0.95 → 0.99 (more long-term value in Pong)
• Slower and broader exploration: epsilon from 1.0 → 0.1, decayed over 200k–300k steps
• Lower learning rate: 5e-4 → 1e-4 for stability
• Larger FC layer: 64/128 → 256 hidden units
• Larger replay: 10k → 50k transitions for diversity

Plot:

Plot:

Plot:

Plot:

Plot:

GitHub renders videos inconsistently; link directly:

• docs/videos/video.mp4

Plot:

• docs/videos/video_double_dqn_episode_1271.mp4

Plot:

Hyperparameters

Video demo

• docs/videos/diff2_best_video_episode_785.mp4

Download Difficulty 2 Video

Plot:

Hyperparameters

Video demo

• docs/videos/diff3_best_video_episode_755.mp4

Download Difficulty 3 Video

Run from repo root.

python scripts/train_dqn.py --exp-dir experiments/dqn_run --episodes 10000 --device cuda

Optional: record videos (every N episodes):

python scripts/train_dqn.py --exp-dir experiments/dqn_run --episodes 10000 --record-video --record-every 100

python scripts/train_double_dqn.py --exp-dir experiments/ddqn_run --episodes 10000 --device cuda

python scripts/train_difficulty.py --exp-dir experiments/diff2_run --difficulty 2 --algo dqn --episodes 10000
python scripts/train_difficulty.py --exp-dir experiments/diff3_run --difficulty 3 --algo dqn --episodes 10000

python scripts/eval.py --checkpoint experiments/dqn_run/best.pth --episodes 20

Record evaluation video:

python scripts/eval.py --checkpoint experiments/dqn_run/best.pth --episodes 5 --record-video

A small (illustrative) sweep is included:

python scripts/sweep.py --base-exp-dir experiments/sweep --episodes 2000 --device cuda

This will generate experiment folders like:

• experiments/sweep/dqn_g0.99_lr0.0001_decay300000/
• experiments/sweep/double_dqn_g0.99_lr0.0001_decay200000/ …and so on.

MIT (See LICENSE).