Ground truth, the VT-WM baseline and the Cosmos3 model side by side on the ten short-horizon
windows published in the vtwm-cosmos inference study.
What is being compared
Episode motherboard_0911_episode_000, windows at t_start
255, 337, 453, 790, 1107, 1235, 1918, 2341, 2940, 3058. Each window is 16 frames at 15 fps: 8 recorded history,
8 predicted. Only the 8 predicted frames are shown.
These are the exact windows the inference study used, and they land on our window grid without
resampling: that episode is stored with stride 1 and window 16, so their t_start is our
window index. Every clip below was checked against the dataset's own t_start field
rather than assuming the mapping.
Our decoder emits 13 frames for a 4-latent window (1 + 4 + 4 + 4; Wan2.2 is 4×
temporal) while theirs rebuilds all 16 with pad_latent.PadLatents. The two framings
disagree about the history, but the predicted halves are the same 8 raw frames — ours is
decode[5:13] (latents 2-3 = raw 8..15), theirs is frames[8:16]. Showing only
the predicted half removes that ambiguity rather than papering over it.
Side by side
rows: env camera · wrist L · wrist R · tactile L · tactile R | columns: ground truth · VT-WM · Cosmos3 at 35 steps | 8 predicted frames
Grey border = ground truth. Red border = predicted.
The one number that is actually comparable
F0 latent MSE — squared error on the first predicted latent block, in normalized latent
space, averaged over the same ten windows. This is the metric their ablations.json
reports, computed here with the same definition. No decoder is involved, so unlike PSNR or
LPIPS it is not contaminated by the fact that the two models decode differently.
| stream | VT-WM
no step axis | Cosmos3
1 step | Cosmos3
2 steps | Cosmos3
4 steps | Cosmos3
10 steps | VT-WM ahead from |
|---|
| env camera | 0.0537 | 0.0278 | 0.0289 | 0.0361 | 0.0474 | never |
| wrist left | 0.1035 | 0.0717 | 0.0772 | 0.0952 | 0.1195 | 10 steps |
| wrist right | 0.2154 | 0.1554 | 0.1738 | 0.2277 | 0.2764 | 4 steps |
| tactile left | 0.0038 | 0.0046 | 0.0041 | 0.0044 | 0.0056 | 1 step |
| tactile right | 0.0115 | 0.0134 | 0.0133 | 0.0166 | 0.0223 | 1 step |
Green marks the step counts at which VT-WM is already ahead on that stream.
35 steps is what the Cosmos3 side runs, and its F0 latent MSE at that setting was never
published. The released ablations.json solver block stops at 10 steps, so the
table above ends there. The trend across the published points is monotone
(video 0.0278 → 0.0289 → 0.0361 → 0.0474 from 1 to 10), so 35 steps would sit higher still —
but extrapolating a number into a comparison table would be inventing it, so the column is
absent rather than estimated. The sharpness table below does cover 35 steps, because
that was measured here from the published preview grids rather than taken from their file.
There is no single honest step count, so the curve is the result. VT-WM is an
autoregressive predictor (predictor: ar, one forward per latent frame): it has no
denoising-step axis and is a horizontal line here. Cosmos3 is a curve. Picking one column is a
choice, and the choice changes who wins — at 1–2 steps Cosmos3 leads on all three
camera streams, by 4 steps VT-WM is ahead on wrist_right, and by 10 steps it leads
on both wrists and both tactile streams.
Do not read "more steps makes Cosmos3 worse" literally. This is squared error against the
one recorded future, and few-step samples sit near the conditional mean, which is exactly what
squared error rewards. As steps rise, Cosmos3's samples get sharper and more diverse and the
metric punishes it — for a capability VT-WM does not have. VT-WM is trained with an L1
next-latent objective, so it emits the conditional median, the MSE-favourable kind of output, at
every step count.
If one column has to be the headline, it is 1 step. That is the only setting which is both
compute-matched (one forward per block on each side) and mode-matched (both outputs
near the conditional mean, so the error measures the same thing on both sides). It is also the
column most favourable to Cosmos3. Quoting the 10-step column instead would let VT-WM "win" four
of five streams by scoring a sampler with a metric that penalises sampling. For perceptual
realism or multi-window coherence Cosmos3 needs ≥ 4 steps — but there the metric has to
change too, not stay MSE.
Sharpness — the axis MSE cannot see
Gradient energy of the prediction divided by gradient energy of that model's own
ground truth, over the same ten windows. 1.000 = as sharp as real footage. Below 1 is
blur, above 1 is high-frequency detail the recording does not contain.
| model | env camera | wrist left | wrist right | tactile left | tactile right |
|---|
| VT-WM (no step axis) | 0.937 | 0.969 | 0.862 | 0.832 | 0.828 |
| Cosmos3, 35 steps | 1.115 | 1.128 | 1.113 | 1.518 | 1.582 |
| Cosmos3, 10 steps | 1.114 | 1.123 | 1.108 | 1.502 | 1.566 |
| Cosmos3, 4 steps | 1.089 | 1.108 | 1.076 | 1.461 | 1.512 |
| Cosmos3, 2 steps | 1.090 | 1.101 | 1.036 | 1.467 | 1.523 |
| Cosmos3, 1 steps | 1.070 | 1.070 | 0.996 | 1.418 | 1.469 |
The two models fail in opposite directions, and neither is "sharper = better". VT-WM sits
at 0.83–0.97: it is under-sharp, which is exactly what an L1 next-latent objective
produces — the conditional median of the futures, i.e. their average, i.e. blur. Cosmos3 sits
at 1.00–1.58 and climbs with steps: it is over-sharp, synthesising high-frequency
content that is not in the recording. On the tactile streams its high-frequency share reaches
2.4× the ground truth's. Blur discards information; hallucination invents it. The target is
1.0, so the error is |ratio − 1|, and by that measure neither model is close on tactile.
Why the ratio, not the raw number. The Cosmos3 clips were decoded by their Wan2.2 VAE plus
the tactile LoRA and ours by the stock diffusers decoder, so absolute sharpness is not comparable.
Dividing by a ground truth that went through the same decoder cancels that to first order. The
residual was measured rather than assumed: the same eight ground-truth frames through the two
decoders differ by 1.09–1.26× in gradient energy (ours is the higher one). That
bounds the systematic error at roughly a quarter, which is far smaller than the VT-WM 0.83 vs
Cosmos3 1.42 gap on tactile, so the direction of the result survives it.
Decoder calibration
The Cosmos3 column was decoded by their wan22_vae.Wan2_2_VAE plus a tactile LoRA and
an anchor-restoration step. Ground truth and VT-WM were decoded by the stock diffusers
AutoencoderKLWan; the LoRA and the wan22_vae package live on their cluster and
are not available here. So part of any visible difference between column 2 and column 3 is the
decoder, not the model.
Rather than asking you to take that on faith, here is the same ground truth through both
decoders. Whatever difference you see here is decoder; subtract it before reading the comparison above.
the SAME ground-truth frames through the two decoders
Expect the gap to be largest on the tactile rows. Their LoRA was fine-tuned
on tactile frames specifically, which is exactly where a stock decoder has least to work with.
Measured, not asserted — and it is large enough to rule a metric out. LPIPS between the
same ground-truth latents decoded the two ways is 0.194 / 0.188 / 0.206 / 0.257 / 0.234
(env camera, wrist L, wrist R, tactile L, tactile R). On four of the five streams that floor is
bigger than either model's distance to its own ground truth, so a cross-model LPIPS or PSNR
number here measures the decoders, not the models. That is why the table above uses F0 latent MSE,
which never leaves latent space, and why the sharpness section uses a ratio to each model's
own ground truth, where the offset cancels to first order (residual measured at 1.09–1.26×).
The comparable perceptual numbers need the latent handover:
scripts/mot_predict_windows.py exports predicted latents in the format
score_external_predictions.py reads, so that scorer decodes both sides with one decoder.
It is also the only route that can settle the tactile comparison, since tactile latent MSE is
saturated (≥ 98 % of variance explained for both models) and only a perceptual metric separates
them there.
What this comparison cannot tell you
This is a training episode. motherboard_0911_episode_000 is in the train split of the
shared split file — for both models. The inference study used it legitimately, because denoising-step
and history-source ablations do not need held-out data. But a model-vs-model comparison on it measures
how well each fits data it has seen, not how well it predicts. Read the clips as qualitative behaviour,
not as a generalization result.
Why not run both on held-out episodes? We can run VT-WM anywhere. Running Cosmos3 needs their
cosmos-framework and their VAE package, neither of which is on this cluster, so the only
Cosmos3 outputs available are the ones they published — and those are all on this one episode.
The held-out comparison that is possible is the latent handover: predicted latents for every
val window of all three tasks are exported for
vtm_cosmos/scripts/score_external_predictions.py, which decodes both sides with the same
decoder. That is the route to numbers that settle anything.