Skip to content

Checkpointing

Checkpointing lets you save and resume evolution runs without losing progress, and seed one experiment with the results of another. A long evolution run can take hours; checkpoints mean a crash or a configuration change does not cost you everything. Cross-phase seeding lets you chain experiments - evolve basic capabilities first, then build on them with different fitness pressure.

checkpoint {
every: 100 generations // save interval
keep: 3 // retained checkpoints
}
.quale FieldConfig FieldDefaultDescription
everyCheckpointInterval100Generations between periodic checkpoint saves
keepCheckpointKeep3Maximum number of checkpoint files to retain (older ones are deleted)

Each checkpoint (.quale-ckpt) contains:

  • Full population (all genomes)
  • Current generation number
  • Best genome observed so far
  • Generation history (fitness stats)
  • Innovation tracker state (next node ID, next innovation number)
  • Sensor and actuator name lists (for pathway analysis in quale inspect)

Checkpoints are saved to checkpoints/<run_name>/ relative to the current working directory. The run name is derived from the evolve block name, lowercased.

Note: Checkpoint pruning sorts filenames lexicographically (e.g., checkpoint_gen1000.quale-ckpt sorts before checkpoint_gen200.quale-ckpt). This can cause unexpected behavior when generation numbers differ in digit count, potentially pruning a more recent checkpoint instead of the oldest.

Note: There is an off-by-one in checkpoint generation numbering. The save trigger checks stats.Generation (set before the generation counter increments), but the filename uses ee.Generation (already incremented). So a checkpoint triggered at generation 100 is saved as checkpoint_gen101.quale-ckpt.


Use quale evolve --resume <path> to continue from a checkpoint. The checkpoint restores:

  • Population
  • Generation counter
  • Best genome
  • History
  • Innovation tracker state

The evolution configuration (mutation rates, speciation settings, etc.) comes from the .quale file, not the checkpoint - so you can adjust parameters when resuming.


evolve Phase2 {
seed_from: Phase1
// or: seed_from: "checkpoints/phase1_gen474.quale-ckpt"
// ... all other fields ...
}

seed_from initializes the population from a prior experiment:

  • When referencing an evolve block name (seed_from: Phase1), the runtime looks for the most recent checkpoint produced by that block
  • When referencing a file path (seed_from: "checkpoints/...") , the runtime loads that specific checkpoint
  • The generation counter and history are reset to zero
  • No configuration is inherited - the evolve block is fully self-contained
  • If the source population is smaller than the target population, additional genomes are created by mutating copies of the seed genomes

This enables multi-phase experiments where Phase 1 evolves basic capabilities and Phase 2 builds on them with different fitness pressure or world configuration.