Conceptual method animation · 12 seconds
Poster PNG ↗ Video MP4 ↗

ChronoReplay

One Answer Interface Is Not Enough
for Streaming Spatial Intelligence

Paper Code Data guide Cite this work

Yihao Hu1,*Yitong Lin2,*Pan Wang3,*Zhihao Wen4,†Hang Wang1Yiwen Zhao5Deyuan Liu1Yutong Li6Mingda WAN1Yuchen Mao7Tao Lin1,†

1Westlake University2ZJU3USTC4SMU5FDU6ECNU7SJTU

* Equal contribution † Corresponding authors

+13.41 pts

OST-Bench over the official base

3 benchmarks

Improved under matched observation access

39,561

Sequences in the causal replay pool

Explore the paper case

A history worth
replaying.

Explore two questions grounded in one observation history. Step through the original paper frames and see how the temporal and reference-frame answers use different evidence.

Read the full paper ↗
A history worth replaying5 observation rounds
Observation frame from the paper's qualitative example.Round 2Shared observation history
Question from the paperWhen were both the bathtub and folded towel on the observer’s left?
Round 2Temporal answer · asked in Round 4

Explore an illustrated case from the paper. Answers are the reported outputs. Read the case ↗

01 / The idea

Different answers need
different spatial distinctions.

Knowing which side an object lies on does not necessarily preserve how far away it is, or when it first appeared.

Training on one answer interface can concentrate improvements on that interface while transfer to others remains weak. We call this observed specialization interface lock-in. ChronoReplay combines heterogeneous supervision in chronological episodes, keeps source-native targets, and balances their contribution to learning.

A single Qwen3-VL-8B checkpoint improves across OVO-S-Bench, OST-Bench, and UCS-Bench under the same observation access. Adaptive frame selection provides a separate inference-time gain with the trained model held fixed.

Read the full abstract
Training a spatial VLM to answer which side an object lies on does not require it to preserve the metric displacement that determines that answer, while supervision for a distance estimate may leave categorical relations largely unconstrained. When post-training emphasizes one readout, improvements can therefore concentrate on that interface while transfer to others remains weak or even degrades; we call this observed specialization interface lock-in. ChronoReplay addresses it by replaying heterogeneous spatial supervision as chronologically ordered episodes, retaining source-native targets, and balancing supervision across families. Under the same observation access, a single ChronoReplay checkpoint improves OVO-S-Bench, OST-Bench, and UCS-Bench together, including a 13.41-point gain on OST-Bench, and achieves the highest three-benchmark mean among the supervision settings compared here. Same-data controls further separate the ingredients of this transfer: answer serialization redistributes gains across OST and UCS, family-balanced learning improves over ordinary joint SFT and natural turn weighting, and chronological replay exceeds a shifted-order control by about 2.41 OST points across two paired seeds. Adaptive frame selection then changes evidence access while keeping the trained model fixed, providing an additional inference-time gain at lower visual cost. Streaming spatial performance depends on what distinctions supervision preserves, how heterogeneous histories are organized during learning, and what causal evidence inference makes available at test time.

Code: github.com/yihaohu0118/ChronoReplay
Project Homepage: yihaohu0118.github.io/ChronoReplay/

ChronoReplay brings categorical, metric, temporal, and reference-frame supervision together, addressing the uneven transfer of specialized training.
From specialized readouts to complementary spatial abilities. Figure 1 from the paper · Click to enlarge
02 / The method

Preserve the target.
Respect its history.

Two stages turn complementary spatial supervision into one autoregressive training objective.

Stage I constructs chronological trajectory episodes and single-turn preservation examples. Stage II combines normalized family risks in one family-balanced learning objective.
Causal replay construction and family-balanced learning. Observations beyond the query's causal frontier remain out of scope.
I

Causal replay construction

Retain categorical options, metric values, and temporal targets in their source-native formats. Replay trajectory turns in chronological order, with each target tied to its observation boundary.

Native targetsChronological episodes
II

Family-balanced learning

Normalize the preservation and trajectory losses, then balance their contribution to shared parameters. The language model and visual merger are trained; the visual encoder stays frozen.

Preservation + trajectoryOne shared model
03 / Evidence

One checkpoint.
Transfer across interfaces.

Matched comparisons isolate the effects of training. Adaptive observation access is measured separately.

Same model size. Same observation access.

Official Qwen3-VL-8B versus ChronoReplay. OVO-S uses Uniform-128; OST and UCS follow their fixed causal evaluation protocols. Higher is better.

Matched causal-access evaluation, scores in percent.
ModelOVO-S ↑OST ↑UCS ↑
Official Qwen3-VL-8B48.9850.5245.33
ChronoReplay53.8963.9352.77

Author-reported results from the paper; this release does not bundle saved predictions. See protocols and full tables ↗

+2.41 OST points

Chronological order matters.

Chronological replay exceeds shifted-order replay across two paired seeds (64.22 versus 61.81 mean OST score), with the same turn content.

37.11% fewer frames

Evidence access adds another gain.

Adaptive access uses 80.50 frames on average versus Uniform-128 and improves OVO-S from 53.89 to 57.98 for the same checkpoint.

04 / A closer look

One origin.
Two native readouts.

Spatial answers depend on the history that defines them. This OST-Bench episode pairs metric and temporal questions with the same causal observation sequence.

Qualitative example reproduced from the paper. Full examples and limitations are included in the manuscript.

Explore the paper
Qualitative OST-Bench episode from the paper showing metric and temporal readouts grounded in a shared observation history.
Paper example · Click to inspect at full resolution
05 / Build on the work

Start with the code.

Implementation, configurations, tests, and a frozen training-selection manifest are available in the repository.

01

Installation ↗

Environment setup and model requirements.

 02

Data preparation ↗

Native records, selection manifests, and source access.

 03

Training ↗

The replay recipe and controlled comparisons.

 04

Evaluation ↗

Inference and scoring for all three benchmarks.

A small example, without a model download.

The CPU smoke test generates synthetic images and native records, checks media decoding, and builds a weighted Parquet pool.

Fine-tuned weights, benchmark media, and saved predictions are not included. See the data guide for available manifests and source-specific requirements.

Terminal 
git clone https://github.com/yihaohu0118/ChronoReplay.git
cd ChronoReplay
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python -m scripts.smoke_test --output outputs/cpu-demo

Linux / macOS · For GPU setup, follow the installation guide.

06 / Citation

Cite ChronoReplay.

If you use the implementation or build on this work, please cite our paper.

BibTeX 
@misc{hu2026chronoreplay,
  title={ChronoReplay: One Answer Interface Is Not Enough
         for Streaming Spatial Intelligence},
  author={Yihao Hu and Yitong Lin and Pan Wang and
          Zhihao Wen and Hang Wang and Yiwen Zhao and
          Deyuan Liu and Yutong Li and Mingda WAN and
          Yuchen Mao and Tao Lin},
  year={2026},
  url={https://yihaohu0118.github.io/ChronoReplay/}
}