Visible-State Supervision
Train on source-grounded probes of event identity, order, accumulation, relation change, and answer sufficiency. One fresh epoch updates the language model and multimodal projector while the vision tower stays frozen.
ViSS
Supervising the Visible State:
Separating Representation from View Selection
in Streaming Video
86.33%
StreamingBench · ViSS + TVS
69.77%
OVO-Bench · ViSS + TVS
7,677
Training examples · One epoch
Explore the paper case
Same query.
Different evidence.
Follow the paper’s frozen-checkpoint example through recent, event and history views. The original frames show how evidence access changes the proposed answer.
Read the full paper ↗Same model. A different view.Query @ 145 s
145.00 s4 recent framesQuestion from the paperWhy is Mr. Bean under the bed now?
D · Investigate something strangeRecent proposal · margin 0.834
Replay the paper’s frozen-checkpoint trace. Illustrative playback, not live inference. Read the case ↗
01 / The idea
An action label does not
tell you how many times.
The same action, seen once or three times, can receive the same label. A model can learn that label without learning to count.
We call this state under-specification: histories that differ in order, accumulation, or relations share an answer, leaving those distinctions unconstrained. Visible-State Supervision (ViSS) asks for them directly, including whether the visible evidence is sufficient to answer at all.
Test-Time Causal View Search (TVS) addresses a separate limitation. With the trained model frozen, it acquires complementary views of the observed video prefix when recent frames are insufficient.
Read the full abstract
An action seen once and the same action seen three times receive the same action-recognition label, so that supervision does not require a model to count. Answers shared across histories that differ in event order, accumulation, or relations leave those distinctions unconstrained; we call this state under-specification. Visible-State Supervision (ViSS) addresses it by training on source-grounded probes of event identity, order, accumulation, relation change, and answer sufficiency; under the same recent-frame interface it improves OVO-Bench over a strict-matched generic temporal supervision baseline while remaining comparable on StreamingBench, with the largest gain in hallucination detection. Test-Time Causal View Search (TVS) then adapts evidence access while keeping the learned model fixed, expanding from recent frames to sparse event and compact history views on demand. The two interventions separate representation from view selection: supervision particularly benefits hallucination detection and causal reasoning, access particularly benefits event recall and forward reasoning. Together they reach the best reported accuracy among the published systems compared here on StreamingBench and OVO-Bench at a fraction of the processed frames of exhaustive selection, and strict replay shows that most of the remaining gap to a three-view oracle is a correct answer that was later replaced. Streaming accuracy depends on what supervision asks the model to keep, what evidence inference exposes, and whether the model keeps a correct answer once it has one.
Code: github.com/yihaohu0118/ViSS
Project Homepage: yihaohu0118.github.io/ViSS/
02 / The method
Learn the visible state.
Search the causal views.
Two controlled interventions separate representation learning from evidence access.
Test-Time Causal View Search
Start with four recent frames. Acquire a sparse event view when the recent confidence margin is low; consult compact history when the proposals disagree. Separate acquisition and commitment thresholds are fixed on a source-disjoint holdout.
03 / Evidence
Two levers.
Different gains.
Full evaluation splits: 2,495 StreamingBench queries and 3,035 OVO-Bench queries. All controlled comparisons share one backbone and official scoring.
Separate what the model learns from what it sees.
One Qwen3-VL-8B backbone. First fix recent-4 access; then freeze the ViSS checkpoint and enable TVS. Higher is better.
| Model / view policy | StreamingBench ↑ | OVO-Bench ↑ |
|---|---|---|
| Qwen3-VL-8B · recent-4 | 80.52 | 58.97 |
| ViSS · recent-4 | 84.77 | 65.21 |
| ViSS + TVS | 86.33 | 69.77 |
Author-reported results from the manuscript. The source release includes scoring utilities, not saved predictions. See full protocols and tables ↗
+1.72 OVO points
Beyond generic temporal supervision.
The mean ViSS gain over strict-matched generic supervision across three paired seeds is 1.72 OVO points (95% paired-bootstrap CI: 0.80–2.60). StreamingBench remains comparable.
17.08 frames per query
Adaptive access on StreamingBench.
TVS reaches 86.33% with 17.08 processed frames on average, compared with 86.45% and 29.97 frames for a single full-history call. TVS averages 1.60 model calls.
The remaining oracle gap is conditional on correct proposals being available. Strict replay finds that replacing an already correct answer dominates these selection errors; the calibrated commitment margin does not show a consistent aggregate benefit.
04 / A closer look
One query
for each lever.
Changing the checkpoint and changing the view can repair different failures. These qualitative examples come directly from the paper.
05 / Get started
From visible-state probes
to causal-view evaluation.
The source release covers data construction, training, inference, policy replay, and scoring.
01
Installation ↗
CPU checks and GPU runtime setup.
02Data preparation ↗
Source-grounded records and media bindings.
03Training ↗
One-epoch ViSS training recipe.
04Evaluation ↗
View proposals, TVS, and official scoring.
Clone and run the source checks.
The tests use synthetic inputs and do not download model weights or benchmark media.
Trained ViSS weights, training records, benchmark media, and calibration artifacts are obtained or prepared separately. See the repository for the release contents.
Quick start
git clone https://github.com/yihaohu0118/ViSS.git
cd ViSS
python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[test]"
python -m pytest -q testsPython 3.10+ · Bash shown; use the platform-specific activation command on Windows.
06 / Citation
Build on this work.
If you use ViSS or TVS in your research, please cite the accompanying manuscript.
Download BibTeX ↗BibTeX
@misc{hu2026viss,
title={Supervising the Visible State: Separating Representation from View Selection in Streaming Video},
author={Yihao Hu and Pan Wang and Yitong Lin and Zhihao Wen and Deyuan Liu and Hang Wang and Mingda Wan and Yiwen Zhao and Yutong Li and Yuchen Mao and Tao Lin},
year={2026},
note={Manuscript},
url={https://yihaohu0118.github.io/ViSS/}
}