A continual-learning agent.

My research hypothesis is that continual learning could be a path toward superintelligence: systems that keep acquiring skills, testing ideas, and building on experience. Learning from an evolving stream of interaction could let capabilities compound beyond what a fixed training dataset provides. This is a research direction, not a demonstrated result. See Silver and Sutton’s Welcome to the Era of Experience, a position paper on learning from experience.

One potential bottleneck lies in the train-then-deploy pattern seen in systems from OpenAI and Anthropic. When deployed model weights remain fixed, context, tools, and external memory can still enable useful adaptation, but new experience does not automatically update those weights. A key question is when retrieved experience is enough, and when durable changes to learned capabilities are more effective. Durable system-level learning can also live in memory and tools; weight updates are not the only route. Anthropic’s Effective context engineering for AI agents describes approaches to managing context over long-running tasks.

My concern is a possible ceiling on efficient, cumulative adaptation: repeatedly retrieving experience or running separate training cycles may add cost and delay. Continual learning could reduce that bottleneck by integrating new experience while retaining earlier skills. This is a hypothesis to test, including against strong memory and replay baselines. Forgetting, unreliable feedback, and safe adaptation remain hard problems. For the retention challenge, see Kirkpatrick and colleagues’ Overcoming catastrophic forgetting in neural networks.

There is no demonstrated intelligence ceiling for either company, and their approaches keep evolving. OpenAI has reported gains from reinforcement learning and inference-time compute in Learning to reason with LLMs; Anthropic has described memory and tools for long-running agents. These may complement continual learning. Continual learning alone is not known to be sufficient for superintelligence.

The prototype is implemented; the proposed continual-learning evaluation remains uncompleted. The immediate goal is to test learning and retention honestly before making broader claims. Review the proposed evaluation and research progress.

Changing worlds, interacting agents

The proposal is to create simulations where agents must survive as resources, hazards, and other agents change. Competition, cooperation, resource management, and adaptation could all matter. Worlds with millions of interacting agents are a long-term ambition, not an implemented capability. Start with small, controlled populations and establish what can be learned and measured before increasing scale.

Why synthetic survival environments?

Finite operating power, irreversible lifetime wear, integrity damage, unstable zones and system faults provide artificial pressures to investigate. Agents could face resource allocation, coordination and changing conditions without copying biological needs. These proposed systems are not all implemented, and optimizing operational survival does not by itself establish transferable intelligence. Useful controlled tests matter more than visual realism.

Learn across worlds without losing earlier skills

Train across a sequence of environments with changing conditions, then revisit earlier worlds to test whether prior capabilities remain. Compare continual adaptation with sequential training and replay under matched resource budgets. A longer survival time in the latest world would not, by itself, show that the agent retained what it learned before.

Proposed tests

Keep survival as the central objective; compare survival-only rewards with survival plus auxiliary learning signals. Measure retention of earlier abilities separately from learning new tasks and continued learnability. Evaluate frozen checkpoints against archived opponents so changing competitors are not mistaken for forgetting; compare conditions with memory available and reset. These are proposed tests, not a finalized protocol. Retaining past skills and preserving the ability to learn new ones are separate questions; see Loss of plasticity in deep continual learning.

Test the reward, not just the agent

Survival may encourage useful behavior, but an agent might also learn to hide, avoid exploration, exploit a simulator, or act aggressively when the reward favors it. Compare reward designs and inspect behavior alongside survival, adaptation, retention, and resource costs. Survival alone is not evidence of general intelligence, safe behavior, or a path guaranteed to reach superintelligence.

Prior work motivates both the opportunity and the caution: multiagent autocurricula demonstrate evolving strategies in a particular simulated setting, while Neural MMO 2.0 describes limits of earlier survival objectives and introduces a broader task system. Neither establishes survival as the best reward or validates this project’s hypothesis.

The survival objective is confined to simulated agents inside research environments; it is not a real-world self-preservation objective.

The arena results above measure winning in one task; retention across tasks remains untested. A lightweight survival testbed and a local 3D base world are available; broader survival systems and large-scale populations remain future work. Reward design and evaluation still need controlled tests.

The small gridworld diagnostic is a first check of learning and forgetting, not the final research scope.

A motivating analogy

Perhaps learning is the process of building and revising a map of memory: organizing experience so it can guide future decisions. This is an intuition to investigate, not a claim that learning and memory are identical. For this research, distinguish stored experiences and lessons, retrieval that makes them available now, and learning that changes how the agent interprets information or acts. A useful change could live in an external memory, a learned retrieval strategy, or model weights.

Let the agent make learning decisions

A proposed learning controller could choose among collecting new experience, retrieving a saved lesson, replaying selected examples, or updating the model. For updates, it could choose a bounded learning rate and number of training steps. Record which examples it selected and the resulting changes so “learning to learn” means a measurable mechanism.

Saving “do this next time” could begin as an explicit memory that the agent can retrieve and revise. A later experiment could test whether replaying the relevant experience turns that lesson into behavior that persists without the reminder. Retrieval, saved intentions and weight updates should be measured separately; remembering a rule does not guarantee applying it well.

First comparison

Start with fixed model code and a small sequence of tasks. Compare uniform replay and a tuned fixed learning-rate schedule with learned data selection alone, learned learning-rate selection alone, and both together. Include a retrieval-only condition to test when access to a saved lesson is enough. Match total interaction and compute budgets, including the cost of choosing what to learn.

Use development tasks to train and select the learning controller, then freeze its selection before evaluating on untouched task sequences and seeds. Measure new-task learning speed, retention of earlier skills, ability to keep learning, and cost. Evaluate with memory available and with retrieval disabled to distinguish dependence on reminders from changes retained in the model. This would test whether choosing how to learn helps beyond a good fixed schedule and replay policy.

What the value function scores

Treat attention targets as a set of candidates: regions of the observation, stored memories and lessons, recent events, or sub-goals. The attention value function maps the current state and a candidate target to an estimate of how much attending to it would improve later return, net of the cost of attending. The agent then allocates a bounded attention budget to the highest-value candidates and processes only those at full resolution.

This differs from attention weights inside a transformer, which are computed in a single forward pass with no explicit cost and no credit assignment to the choice itself. Here the choice of where to look is a decision whose consequences are measured over time, so the agent can learn that something boring now matters later, or that a salient distraction is not worth the compute.

How it could be trained

One route is reinforcement learning over an augmented action space: at each step the agent chooses both what to attend to and what to do, and the attention choice is credited with the return that follows. A second route is a counterfactual signal: compare the agent's prediction or value estimate with and without a candidate attended, and train the attention value to predict that gain. A third route reuses the planner: in MuZero-style search, attention targets that change the chosen action or sharpen the value estimate earn higher value.

The same function could rank memories for retrieval, which connects it to learning what and how to learn: deciding what deserves attention now and deciding what deserves to be learned later may be the same estimate applied at two time scales.

First comparison

Start in a gridworld or the survival testbed with partial observability, a few distractor signals, and a small set of saved lessons. Compare dense processing of the full observation, fixed heuristic attention (nearest hazard, nearest resource), uniformly random attention under the same budget, and the learned attention value function. Match total compute per step, including the cost of scoring candidates.

Measure return, survival time, and how often the agent attends to a target that later turns out to matter. Add a condition where the cost of attention is raised to see whether the learned function becomes more selective. Evaluate on unseen seeds and on a world where the distractors change, to test whether the agent learned a general sense of relevance or memorized which signals mattered in training.

A shared model with a game-specific interface

Each game supplies an adapter that maps observations into a common input format and maps policy outputs to valid game actions. It also supplies action availability, reward and episode boundaries. Include task identity or rules so the same input or action index is not silently assigned conflicting meanings. Keep the shared model architecture and weights across game switches; reset episode memory and environment state at those boundaries.

For the first experiment, use a declared collection of discrete games with a shared action vocabulary and a mask for legal actions. Define the vocabulary before training so switching games does not replace the policy head. An action index must have an explicit meaning; merely resizing or relabeling an output layer does not transfer a skill. Observation adapters must likewise preserve meaning, not just tensor shape.

More varied games may require learned observation encoders or action heads, especially when switching between discrete and continuous control. In that extension, distinguish retained shared weights from newly initialized interface parameters. A policy that scores descriptions of available actions is another possible design, requiring training to interpret those descriptions. Neither approach guarantees immediate competence in an unfamiliar game.

Put the switch inside the training loop

At a declared episode boundary, save the current training state, select the next game from a fixed schedule, load its adapter and continue optimization without reinitializing shared weights. Preserve compatible optimizer state; tag replay by game and interface version so old transitions are decoded correctly. Archive adapters and checkpoints with the environment versions. Environment changes here mean switching the game being played, not learning to generate or redesign games.

Measure transfer and forgetting separately

Evaluate each new game before training to measure immediate transfer, then track learning speed against a fresh agent with a matched budget. Revisit earlier games using the latest weights with learning disabled during evaluation. Compare sequential training with and without replay, repeat across seeds and task orders, and report each game’s scores separately when reward scales differ. Reserve fresh final evaluation seeds from all training and selection.

Connect this to code editing

The cross-game loop supplies ongoing experience to the same learner. The separate code-editing loop below could propose model changes at explicit checkpoints. First test game switching with the model code fixed; then compare adding learned code edits under the same total budget. This separates gains from carried-over weights, replay and code changes.

Give the agent an explicit editing action

At training boundaries, an editing controller reads the code version, checkpoint summary, learning curves, past edit outcomes and remaining budget. It chooses whether to edit, which allowed module to target and what change to request. An external code editor receives the relevant source and instruction, then produces a patch in a versioned candidate copy. A separate runner checks the patch, trains the candidate and measures the result.

Start with a small editable policy or memory module with a fixed input/output interface. Preserve compatible weights; initialize changed parameters by a declared rule and reset incompatible optimizer state. Larger architecture changes and edits to the learning algorithm can follow after this limited setting works. Lana’s current numeric policy does not generate code. The editing controller and external code editor would be additional components of the overall agent.

Use an external editor without needing its weights

The editor can be a coding agent, a hosted model accessed through an API, or a local model. It needs a common contract: receive source, instructions and selected development history; return a candidate patch, no change or a failure. The training runner owns evaluation and promotion. The editor’s weights can remain fixed; no gradient needs to pass through it.

Train the editing controller from downstream outcomes

The controller’s action is an editing instruction or a decision to skip. The external editor’s patch, bounded candidate training and evaluation form the resulting transition. Store instructions, patches, successful and unsuccessful outcomes, invalid attempts and costs. Use the measured reward to update the controller’s action probabilities, rather than the external editor’s token probabilities. Start with a small set of edit objectives and one candidate per outer step so credit can be assigned to the instruction that produced it.

A proposed reward is the candidate’s evaluation return minus a matched unchanged parent’s return, minus penalties for added compute and lost performance on earlier tasks. Fix those coefficients and any rejection thresholds before the experiment. Both branches receive equal training budgets and paired evaluation seeds; the parent keeps learning too. Invalid patches and timeouts receive a predefined failure reward and consume budget.

Start with fixed editing instructions, a frozen external editor and selection by measured results. Supplying previous outcomes gives the editor useful context, but does not update its weights. The next comparison adds a controller trained from those rewards while holding the editor fixed. Calling that outer loop RL requires actual controller updates; accepting patches or retaining history alone is not an RL update. Ordinary model-weight learning remains in the inner loop.

Keep the measurement independent

The editable surface excludes the task reward, evaluator, test data and execution limits. Keep the editor’s access separate from candidate execution. Run candidate training and evaluation without credentials or network access, with fixed time and memory limits. Promote a candidate only at a checkpoint boundary after it satisfies the predeclared improvement and retention criteria; otherwise restore the parent. Archive source hashes, checkpoints, migrations, rewards, failures and total compute so each decision can be reproduced.

First experiment: does learning to edit help?

Use a small sequence of tasks and compare ordinary RL with no edits, random valid edits, a fixed instruction policy using an external editor with selection, and a learned instruction policy using the same editor. Match total compute, including code generation, failed trials and evaluation, and repeat across seeds. Measure new-task performance, retention, subsequent learning speed, invalid-edit rate and cost. This tests whether learning to edit contributes beyond extra search.

Use development evaluations for edit rewards and promotion. Keep a separate final task/seed suite inaccessible to the editor, freeze the selected system before testing it, and never feed those final results back into the loop. Repeated development evaluation can still overfit. Improving one task or one patch does not establish cumulative improvement or continual learning.

Related work and the open question

The Darwin Gödel Machine studies code modification with empirical evaluation and an evolving archive of agents. Hyperagents extends editable programs to include the mechanism that proposes improvements. These motivate the direction; they do not validate this proposed RL-trained editor for my model.

The research question is whether an agent can learn edits that improve future learning while preserving earlier capabilities, after accounting for the cost of finding those edits. The first experiment holds the external editor, controller-training algorithm and evaluator fixed; only the controller’s learned policy and the candidate model change.

My starting hypothesis is that the brain’s much larger number of connections might be part of the explanation. A commonly cited rough estimate is around 100 trillion synapses in the human brain. Comparing that with a hypothetical trillion-parameter AI model suggests a striking difference in scale—but AI has no single parameter count, and a biological synapse is not equivalent to one artificial parameter.

Connection count alone does not establish why one system learns from fewer examples. How connections are organized and updated may matter as much as their number. Could greater capacity improve sample efficiency under otherwise matched conditions? That is a testable question, not a conclusion from the counts.

Scale reference: the National Institute on Aging’s Unraveling the Mystery gives the approximate 100-trillion-synapse figure. Treat it as an estimate, not a precise measurement or a direct AI comparison.

A person approaching a new task brings years of sensory experience, learned concepts, and biological structure shaped by evolution. A fair comparison should distinguish learning from scratch from adapting with prior knowledge, including an AI model’s pretraining.

Human-level one-shot learning has been demonstrated in a computational model for handwritten characters using structured prior knowledge. That narrow result suggests that what a learner brings to a task matters; it does not settle the broader human–AI experience gap. See Lake, Salakhutdinov, and Tenenbaum’s Human-level concept learning through probabilistic program induction.

What should count as experience: examples, interactions, sensory information, or time? How much of rapid learning is reusing earlier knowledge, memory, or internal rehearsal?

This leads to another question: do humans learn through something like backpropagation, or through a different mechanism? Backpropagation computes how each parameter contributes to an error, allowing a neural network to adjust its weights. The biological counterpart is the credit-assignment problem: how do local synaptic changes improve the behavior of the whole system?

Whether brains implement backpropagation, approximate parts of it, or use other mechanisms remains an open research question. Biological feedback and local learning rules could support backpropagation-like updates, but a plausible mechanism is not proof that the brain uses it. See Lillicrap and colleagues’ Backpropagation and the brain.

The experience gap alone cannot tell us which learning rule the brain uses. I want to understand which combinations of network structure, prior knowledge, memory, and learning rules could make artificial agents learn more efficiently.