Letting the AI decide how it learns

A motivating analogy

Perhaps learning is the process of building and revising a map of memory: organizing experience so it can guide future decisions. This is an intuition to investigate, not a claim that learning and memory are identical. For this research, distinguish stored experiences and lessons, retrieval that makes them available now, and learning that changes how the agent interprets information or acts. A useful change could live in an external memory, a learned retrieval strategy, or model weights.

Let the agent make learning decisions

A proposed learning controller could choose among collecting new experience, retrieving a saved lesson, replaying selected examples, or updating the model. For updates, it could choose a bounded learning rate and number of training steps. Record which examples it selected and the resulting changes so “learning to learn” means a measurable mechanism.

Saving “do this next time” could begin as an explicit memory that the agent can retrieve and revise. A later experiment could test whether replaying the relevant experience turns that lesson into behavior that persists without the reminder. Retrieval, saved intentions and weight updates should be measured separately; remembering a rule does not guarantee applying it well.

First comparison

Start with fixed model code and a small sequence of tasks. Compare uniform replay and a tuned fixed learning-rate schedule with learned data selection alone, learned learning-rate selection alone, and both together. Include a retrieval-only condition to test when access to a saved lesson is enough. Match total interaction and compute budgets, including the cost of choosing what to learn.

Use development tasks to train and select the learning controller, then freeze its selection before evaluating on untouched task sequences and seeds. Measure new-task learning speed, retention of earlier skills, ability to keep learning, and cost. Evaluate with memory available and with retrieval disabled to distinguish dependence on reminders from changes retained in the model. This would test whether choosing how to learn helps beyond a good fixed schedule and replay policy.

What the value function scores

Treat attention targets as a set of candidates: regions of the observation, stored memories and lessons, recent events, or sub-goals. The attention value function maps the current state and a candidate target to an estimate of how much attending to it would improve later return, net of the cost of attending. The agent then allocates a bounded attention budget to the highest-value candidates and processes only those at full resolution.

This differs from attention weights inside a transformer, which are computed in a single forward pass with no explicit cost and no credit assignment to the choice itself. Here the choice of where to look is a decision whose consequences are measured over time, so the agent can learn that something boring now matters later, or that a salient distraction is not worth the compute.

How it could be trained

One route is reinforcement learning over an augmented action space: at each step the agent chooses both what to attend to and what to do, and the attention choice is credited with the return that follows. A second route is a counterfactual signal: compare the agent's prediction or value estimate with and without a candidate attended, and train the attention value to predict that gain. A third route reuses the planner: in MuZero-style search, attention targets that change the chosen action or sharpen the value estimate earn higher value.

The same function could rank memories for retrieval, which connects it to learning what and how to learn: deciding what deserves attention now and deciding what deserves to be learned later may be the same estimate applied at two time scales.

First comparison

Start in a gridworld or the survival testbed with partial observability, a few distractor signals, and a small set of saved lessons. Compare dense processing of the full observation, fixed heuristic attention (nearest hazard, nearest resource), uniformly random attention under the same budget, and the learned attention value function. Match total compute per step, including the cost of scoring candidates.

Measure return, survival time, and how often the agent attends to a target that later turns out to matter. Add a condition where the cost of attention is raised to see whether the learned function becomes more selective. Evaluate on unseen seeds and on a world where the distractors change, to test whether the agent learned a general sense of relevance or memorized which signals mattered in training.

A shared model with a game-specific interface

Each game supplies an adapter that maps observations into a common input format and maps policy outputs to valid game actions. It also supplies action availability, reward and episode boundaries. Include task identity or rules so the same input or action index is not silently assigned conflicting meanings. Keep the shared model architecture and weights across game switches; reset episode memory and environment state at those boundaries.

For the first experiment, use a declared collection of discrete games with a shared action vocabulary and a mask for legal actions. Define the vocabulary before training so switching games does not replace the policy head. An action index must have an explicit meaning; merely resizing or relabeling an output layer does not transfer a skill. Observation adapters must likewise preserve meaning, not just tensor shape.

More varied games may require learned observation encoders or action heads, especially when switching between discrete and continuous control. In that extension, distinguish retained shared weights from newly initialized interface parameters. A policy that scores descriptions of available actions is another possible design, requiring training to interpret those descriptions. Neither approach guarantees immediate competence in an unfamiliar game.

Put the switch inside the training loop

At a declared episode boundary, save the current training state, select the next game from a fixed schedule, load its adapter and continue optimization without reinitializing shared weights. Preserve compatible optimizer state; tag replay by game and interface version so old transitions are decoded correctly. Archive adapters and checkpoints with the environment versions. Environment changes here mean switching the game being played, not learning to generate or redesign games.

Measure transfer and forgetting separately

Evaluate each new game before training to measure immediate transfer, then track learning speed against a fresh agent with a matched budget. Revisit earlier games using the latest weights with learning disabled during evaluation. Compare sequential training with and without replay, repeat across seeds and task orders, and report each game’s scores separately when reward scales differ. Reserve fresh final evaluation seeds from all training and selection.

Connect this to code editing

The cross-game loop supplies ongoing experience to the same learner. The separate code-editing loop below could propose model changes at explicit checkpoints. First test game switching with the model code fixed; then compare adding learned code edits under the same total budget. This separates gains from carried-over weights, replay and code changes.

Give the agent an explicit editing action

At training boundaries, an editing controller reads the code version, checkpoint summary, learning curves, past edit outcomes and remaining budget. It chooses whether to edit, which allowed module to target and what change to request. An external code editor receives the relevant source and instruction, then produces a patch in a versioned candidate copy. A separate runner checks the patch, trains the candidate and measures the result.

Start with a small editable policy or memory module with a fixed input/output interface. Preserve compatible weights; initialize changed parameters by a declared rule and reset incompatible optimizer state. Larger architecture changes and edits to the learning algorithm can follow after this limited setting works. Lana’s current numeric policy does not generate code. The editing controller and external code editor would be additional components of the overall agent.

Use an external editor without needing its weights

The editor can be a coding agent, a hosted model accessed through an API, or a local model. It needs a common contract: receive source, instructions and selected development history; return a candidate patch, no change or a failure. The training runner owns evaluation and promotion. The editor’s weights can remain fixed; no gradient needs to pass through it.

Train the editing controller from downstream outcomes

The controller’s action is an editing instruction or a decision to skip. The external editor’s patch, bounded candidate training and evaluation form the resulting transition. Store instructions, patches, successful and unsuccessful outcomes, invalid attempts and costs. Use the measured reward to update the controller’s action probabilities, rather than the external editor’s token probabilities. Start with a small set of edit objectives and one candidate per outer step so credit can be assigned to the instruction that produced it.

A proposed reward is the candidate’s evaluation return minus a matched unchanged parent’s return, minus penalties for added compute and lost performance on earlier tasks. Fix those coefficients and any rejection thresholds before the experiment. Both branches receive equal training budgets and paired evaluation seeds; the parent keeps learning too. Invalid patches and timeouts receive a predefined failure reward and consume budget.

Start with fixed editing instructions, a frozen external editor and selection by measured results. Supplying previous outcomes gives the editor useful context, but does not update its weights. The next comparison adds a controller trained from those rewards while holding the editor fixed. Calling that outer loop RL requires actual controller updates; accepting patches or retaining history alone is not an RL update. Ordinary model-weight learning remains in the inner loop.

Keep the measurement independent

The editable surface excludes the task reward, evaluator, test data and execution limits. Keep the editor’s access separate from candidate execution. Run candidate training and evaluation without credentials or network access, with fixed time and memory limits. Promote a candidate only at a checkpoint boundary after it satisfies the predeclared improvement and retention criteria; otherwise restore the parent. Archive source hashes, checkpoints, migrations, rewards, failures and total compute so each decision can be reproduced.

First experiment: does learning to edit help?

Use a small sequence of tasks and compare ordinary RL with no edits, random valid edits, a fixed instruction policy using an external editor with selection, and a learned instruction policy using the same editor. Match total compute, including code generation, failed trials and evaluation, and repeat across seeds. Measure new-task performance, retention, subsequent learning speed, invalid-edit rate and cost. This tests whether learning to edit contributes beyond extra search.

Use development evaluations for edit rewards and promotion. Keep a separate final task/seed suite inaccessible to the editor, freeze the selected system before testing it, and never feed those final results back into the loop. Repeated development evaluation can still overfit. Improving one task or one patch does not establish cumulative improvement or continual learning.

Related work and the open question

The Darwin Gödel Machine studies code modification with empirical evaluation and an evolving archive of agents. Hyperagents extends editable programs to include the mechanism that proposes improvements. These motivate the direction; they do not validate this proposed RL-trained editor for my model.

The research question is whether an agent can learn edits that improve future learning while preserving earlier capabilities, after accounting for the cost of finding those edits. The first experiment holds the external editor, controller-training algorithm and evaluator fixed; only the controller’s learned policy and the candidate model change.