Score by model and condition
Effort and result
Scored runs only. Medians unless noted. "Beats best published" compares each run with the best published method run on that task (CAST, SLAT, SPACEL, PASTE, …), where one exists.
Task × model
Exploration space vs final approach
For each keyword: the share of a model's scored runs that tried it at any point (light bar) and the share whose submitted code uses it (dark bar). The last column is the change in mean within-task percentile when the final code uses the keyword, computed inside each model and then averaged, so it is not just a model effect. These are associations, not causal effects.
apply_patch events carry no file contents, so for Codex models (gpt-5.5, gpt-6-astra) the code signal only sees shell commands, inline code and the final snapshot. It undercounts what they tried. The narration option partly fills that gap.What the submission is
Final code that calls a published package is labelled by that package. Other code is labelled by the most flexible transform found in it.
Approach and score
Mean of the selected metric by final approach, with its n. Click a row to list those runs.
Packages tried, in order, then submitted
Published packages in the order the run first ran them (via code), then what the submission is. The 20 most common sequences.
What the agent is doing, through the run
Each tool call gets one phase. Progress is measured in tool calls: each run is sampled at 20 equally spaced points, so every run has equal weight. Hover a panel to read the shares.
Wall-clock time per phase
Mean minutes per run. Each tool call is charged the time until the next event. Codex logs no timestamps, so GPT models are absent. Background jobs make "wait" large: the agent waits on a script it launched earlier.
Working habits
Scored runs. "Baseline first" means a submission was written before any alignment method was tried. "Evaluated before final write" means an evaluate step came before the last submission write.
How the phase mix relates to the score
Runs split into thirds by within-task percentile, separately for each model. Bars show the mean share of tool calls in each phase.
Alignments
Every panel is drawn from the coordinates the run or method submitted, exactly as scored: the input, the true positions where they are known, each published package at its best configuration, the best naive fit, and the one analysis-set run of each model under each prompt. For a pair, the reference section is grey and the moving section is coloured by the region or cell-type labels the agents never saw. Click a panel to enlarge it or to overlay a second one; drag a 3D panel to rotate the stacks.
Published methods on the benchmark
Every published alignment package we reran, scored by the same hidden scorer as the agents. Each package is shown at its best configuration on each task (click a cell in the last table to see all of them). Published means the 13 packages run on the same inputs as the agents: the naive geometric fits (identity, centroid, PCA, ICP), the synthesized programs and the SPACEL variants that read the true labels are not counted, and the first two can be added as extra rows. The within-task percentile ranks an entry against the published packages on that task (100 = better than every package); "best published" is the best package on that task. The family filter at the top applies.
Package × task family
Mean within-task percentile among the published packages, by task family. Hover for the number of tasks.
Task × package
Publication figures
Nature double-column figures (183 mm, Arial) made from the same tags. The PDF, SVG and TIFF versions, the source data for every panel and the LaTeX write-up sit next to these images in explorer/figures/.