Clark Whitehead

Some of the most powerful ideas don’t work at toy scale.

← All research

Continual learning research · proposed test · November 2026

Does an AI that learns beat one that takes notes?

The gated test only asks whether a lesson reached the weights. This one asks the harder question: is learning in the weights any better than a notebook the AI rereads?

Proposed test · November 2026

Does an AI that learns beat one that takes notes?

The main benchmark · play the levels on the homepage · no AI run on it yet · the same Skirmish levels as the side test with context wiped

Two copies of the same AI play the same levels.

One writes everything down and rereads it before every move.

The other changes its own weights after every level.

What you’re looking at: the same levels, two ways of remembering them
Two players on the same levels. The notebook player’s notebook grows every level and is reread before every move. The learning player’s weights change after every level and stay the same size.Level 1Level 2Level 3Level 4Level 5 …Notebook playerwrites it all down,rereads it every movegrows;slower to useLearning playerchanges its own weightsafter every levelsame size;dots shift

The lessons are chosen so a notebook struggles.

A pattern spread across hundreds of tries has no single line to write down.

A timing skill can’t be learned by reading.

The two kinds of lesson in the test
Left: a pattern spread across hundreds of tries, where no single try tells you the rule. Right: a timing skill, where the right moment to act is a feel, not a fact.A pattern across hundreds of triesact nowA timing skill, like aim

Then we keep adding levels until one player falls behind.

The result we’re looking for
Win rate against number of levels. If the bet is right, the learning player keeps rising while the notebook player flattens and falls as the notebook outgrows what it can use. If notes are enough, the two lines stay together.more levels played →levels wonlearning player keeps risingnotebook player falls behind: the bet is rightdashed: notes are enough, the lines stay together

If the notebook player falls behind, learning beats notes, and that is the case for continual learning.

If the two lines stay together, notes are enough, and the bet is wrong.

Why this is the main test, not the context-wiped one

The gated level test wipes the player’s context between levels, so it can only ask whether a lesson reached the weights. It cannot say whether weights are better than a notebook, because the notebook player never gets to play. This test lets both play, with memory on, and makes the lessons the kind that notes carry badly.

How it is scored

Levels won, as the number of levels grows, for each player. Alongside it, the cost of a move: the notebook player rereads a notebook that grows every level, the learning player pays the same every time. A notebook player that keeps up but takes ten times longer per move has not kept up.

What would prove the bet wrong

The two lines staying together out to a thousand levels, at a cost per move that stays reasonable. Then today’s approach, a frozen model with memory and tools, is enough for long projects, and the forgetting problem matters less than this site claims.

More research