Deep Q-learning visualizer
Why a target network stabilizes DQN
A 20-step illustrative MountainCar comparison. Both sides start identically at step 0. The left side uses a slowly updated target network; the right side bootstraps from the same online network it is changing.
Soft target
Single network
Q-learning equations — variables only
γ = 0.99 · α = 0.25 · τ = 0.01
Bellman target and TD error
ybefore = r + γ(1 − d) maxa′ Qboot,old(s′,a′)
δ = ybefore − Qonline,old(s,a)
δ = ybefore − Qonline,old(s,a)
Online value update
Qonline,new(s,a) ≈ Qonline,old(s,a) + αδ
Target movement
Qtarget,new ≈ (1 − τ)Qtarget,old + τQonline,new
Δy = yafter − ybefore
Δy = yafter − ybefore
True initialization
Initialization · Step 0 of 19
Two-network DQN
Online prediction + soft target bootstrap
Substituted calculationscontrolled target
Single-network DQN
Online prediction + online bootstrap
Substituted calculationscoupled target