# 768p → 2× VR Shootout — evaluation

Site: https://sr-2x-shootout.pages.dev
Machine: `viggle_new_5090` (8 × RTX 5090 32G)
Evaluation date: **2026-08-21**

Two independent evaluations. They answer different questions and they do not agree —
that is the point, not a bug.

| | MiniMax-20 | ×2 GT Benchmark |
|---|---|---|
| sources | 20 real MiniMax-H3 production outputs, 768p, 5 aspect ratios | UDM10 (10 clips) + YouHQ40 (40 clips) |
| ground truth | none | yes |
| question | how good does it look, and how much did it change the source | how accurately does it reconstruct the truth |
| metrics | MUSIQ, CLIP-IQA, MANIQA, NIQE, DOVER, Downsample LPIPS | PSNR, SSIM, LPIPS, DISTS + the no-reference set |

---

## Methods under test (6)

| key | what it is |
|---|---|
| `src_nearest_2x` | reference floor — input upscaled ×2, nearest neighbour, no GPU |
| `src_bicubic_2x` | reference floor — input upscaled ×2, bicubic, no GPU |
| `swiftvr_official` | SwiftVR 5B generative DiT, shipped `restore_video` 4-thread pipeline, clip_len 24 with OOM fallback, no torch.compile |
| `swiftvr_optimised` | SwiftVR, single-threaded driver, clip_len 4, torch.compile |
| `cdavsr_official` | CDA-VSR 3.11M discriminative compressed-domain VSR, fp32, official eval loop (one fix: hardcoded `t = 100` → `t = len(frames)`) |
| `cdavsr_optimised` | CDA-VSR, bf16 autocast + channels_last + torch.compile, torchvision `deform_conv2d` in place of mmcv (bit-identical, max\|diff\| = 0), prefetch thread, ×4→×2 downscale on GPU |

CDA-VSR's released head is ×4 only, so it serves ×2 as `native ×4 → area downscale`.
That extra compute is counted in its runtime.

---

## Metric definitions

Arrows: ↑ higher is better, ↓ lower is better.

**No-reference (no GT needed)** — all from PyIQA 0.1.16, default checkpoints:

| metric | dir | what it measures |
|---|---|---|
| MUSIQ | ↑ | multi-scale transformer IQA, KonIQ-10k weights |
| CLIP-IQA | ↑ | CLIP-space "good photo" vs "bad photo" affinity |
| MANIQA | ↑ | ViT-based no-reference IQA, KonIQ-10k weights |
| NIQE | ↓ | classical natural-scene-statistics distance |
| DOVER overall / technical / aesthetic | ↑ | official `VQAssessment/DOVER` + `DOVER.pth` v0.1.0. Video model, run with the repo's own `spatial_temporal_view_decomposition` and `val-l1080p` sample_types — **not** decomposed into single frames. `overall` is the repo's own score-level fusion squashed to [0,1]; technical/aesthetic are the raw head outputs. |

**Full-reference (GT benchmark only)**:

| metric | dir | notes |
|---|---|---|
| PSNR-Y / SSIM-Y | ↑ | Y channel, the VSR-literature convention |
| PSNR-RGB / SSIM-RGB | ↑ | kept alongside so the two conventions can never be confused |
| LPIPS | ↓ | AlexNet variant |
| DISTS | ↓ | |

**Source consistency (MiniMax-20 only)**:

| metric | dir | definition |
|---|---|---|
| Downsample LPIPS | ↓ | `2× output → F.interpolate(mode="area") back to source resolution → LPIPS(AlexNet) vs the MiniMax source`. **Not a quality score.** Low = faithful to the source, high = the restoration rewrote the content. It cannot tell you whether the rewrite looks good. |

---

## Frame sampling

* **MiniMax-20, image IQA**: 24 uniformly sampled frames per video. Every method uses the
  **identical frame indices**, frozen into `manifest.json` by the integrity pass before any
  metric runs. 20 × 24 × 6 = 2880 evaluations per metric.
* **MiniMax-20, Downsample LPIPS**: **no sampling** — all 3724 common frames. The two videos are
  decoded in lockstep and a short read is a hard error, so frame *i* of the output is always
  compared against frame *i* of the source.
* **GT benchmark**: **no sampling** — every frame of every clip (~32 per clip).
* **DOVER**: its own official temporal sampling protocol, never split into frames.
* Per-clip length is `n_common = min(frame count)` over the source and all methods, because
  SwiftVR rounds its processing length down to `4k+1`.

---

## ×2 degradation protocol (GT benchmark)

Our production task is `768p → 2×`, not the ×4 the papers report, so the datasets' own LQ is
**not** used (UDM10 ships a ×4 LQ at 318×180). The ×2 input is built from GT:

```
GT png ──(centre-crop to a multiple of 32)──▶ aligned GT       ← the scoring reference, lossless png
   │
   └──(bicubic 0.5×)──▶ LQ ──(libx264 crf 23, GOP 25, bframes 0, refs 1, weightp 0)──▶ ONE shared LQ.mp4
                                                                                          │
             ┌────────────────────────┬──────────────────────────┬──────────────────────────┘
             ▼                        ▼                          ▼
       nearest/bicubic ×2       SwiftVR ×2         CDA-VSR native ×4 → area → ×2
             └────────────── all outputs lossless png at GT resolution ──────────────┘
```

* **All six methods read the same LQ file** (md5 per clip in `bench_manifest_*.json`). The LQ has
  to be a compressed video because CDA-VSR is a compressed-domain model and reads motion vectors
  out of the bitstream; its `deLR/`, `mvs/`, `residual_y/` are all decoded from that one file.
  This also fixes an asymmetry in the MiniMax run, where SwiftVR read the pristine source while
  CDA-VSR read a re-encode.
* **Scoring is PNG on both sides.** Quantitative metrics never pass through a lossy container.
  Exception: DOVER needs a container, so all six methods are encoded identically with
  `libx264 -crf 0 -pix_fmt yuv420p` before it runs.
* **GT losslessness is verified, not assumed**: GT png → lossless video → decode → per-pixel
  compare, `max|diff|` must be 0 or the clip fails.
* **GT is centre-cropped to a multiple of 32** (YouHQ40 1080→1056, UDM10 720→704 / 1272→1248)
  because of the SwiftVR crop bug below. Native size and crop offset per clip are in the manifest.

### Datasets

| dataset | source | clips | frames | GT after alignment |
|---|---|---|---|---|
| UDM10 | SparkVSR (ECCV 2026) repackaging of the PFNL test set, Google Drive `1AmGVSCwMm_OFPd3DKgNyTwj0GG2H-tG4` | 10 | 32 | 1248×704 (native 1272×720) |
| YouHQ40 | Upscale-A-Video (CVPR 2024) `YouHQ40-Test`, Google Drive `1rkeBQJMqnRTRDtyLyse4k6Vg2TilvTKC` | 40 | 33 (one clip 31) | 1920×1056 / 1056×1056 (native 1080) |

The PFNL original is only on TeraBox/Baidu Pan, neither of which is scriptable; the SparkVSR
repackaging was used instead. SPMCS was not run — see below.

---

## Two upstream findings

1. **SwiftVR silently drops rows when the 2× target height is not a multiple of 32.** It pads up
   to a multiple of 32 before the network and unconditionally crops the padding off afterwards,
   but the padding never appears in the decoder output — so a 1080p target comes out **1072**
   rows tall. Both the shipped `restore_video` and the single-threaded driver do it. The MiniMax
   set never triggered it (all its output dimensions are multiples of 32); YouHQ40's 1080 hits it
   immediately. Production impact: any output height that is not a multiple of 32, 1080p first.
2. **SwiftVR drops up to 3 trailing frames**, rounding its processing length to `4k+1`. All 10
   UDM10 clips (32 → 29 frames) and the one 31-frame YouHQ40 clip are affected.

Both are recorded per clip in `bench_issues.csv`; scoring uses the common prefix and never pads.

---

## Statistics

All six methods see the same sources, so comparisons are **paired**: the per-video difference is
reported, not the gap between two independent means (which between-clip variance can swamp).
Each pair gets a mean and median per-video delta, a win/tie/loss count in the metric's own
direction, and a 95% percentile bootstrap CI (2000 resamples, seed 20260821 — reproducible).
"Significant" on the site means the CI does not cross zero.

**No composite score.** MUSIQ, CLIP-IQA, DOVER, LPIPS and PSNR measure different things and are
deliberately not normalised into one number. The site marks a winner per metric and shows an
average-rank "consensus", explicitly labelled
`Heuristic ranking — not a calibrated quality score`.

**SPMCS was not run.** The plan was to add a third dataset only if the first two failed to
separate the models. They separate them decisively (>8 dB PSNR-Y spread, and every
model-vs-model pair's 95% CI clear of zero), so a third dataset would not change the conclusion.

---

## Files

Under `sr_eval/evaluation/` (also published under `/evaluation/` on the site):

| file | contents |
|---|---|
| `results_raw.csv` | MiniMax-20, one row per (source_video, method, metric) |
| `summary.csv` / `summary.json` | MiniMax-20 per-method mean/median/std/rank, paired deltas, consensus |
| `manifest.json` | MiniMax-20 per-clip sizes, frame counts, the frozen 24 IQA frame indices |
| `issues.csv` | MiniMax-20 integrity: 0 fatal, 28 warn (SwiftVR tail frames), 20 note (delivery bitrate spread) |
| `bench_{udm10,youhq40}_results_raw.csv` | benchmark, one row per (clip, method, metric) |
| `bench_{udm10,youhq40}_summary.{csv,json}` | benchmark per-method stats, paired deltas, runtime, peak VRAM |
| `bench_manifest_{udm10,youhq40}.json` | per-clip GT/LQ sizes, native size, crop offset, LQ md5, ffmpeg command |
| `bench_issues.csv` | benchmark integrity: 0 fatal, 22 warn (SwiftVR frame counts), 50 note (32-alignment crops) |

Per-frame scores are kept on the machine in `~/sr_eval/raw/*.jsonl` — every record carries the
full per-frame array, not just the mean.

## Exact commands

```bash
# environments
~/sriqa_venv/bin/python          # pyiqa 0.1.16, torch 2.10.0+cu128  (venv over conda env `cdavsr`)
~/dover_venv/bin/python          # official DOVER, torch 2.10.0+cu130 (venv over conda env `swiftvr`)
export HF_HOME=/home/kevin/hf_cache TORCH_HOME=/home/kevin/torch_home

# --- MiniMax-20 ---
python make_baselines.py                                   # nearest/bicubic x2 mp4, libx264 crf 17
python integrity.py                                        # -> manifest.json + issues.csv (gate)
bash launch_phase1.sh                                      # 8 GPUs: musiq x2, clipiqa, dslpips x3, maniqa, niqe
CUDA_VISIBLE_DEVICES=$g python run_dover.py --shard $g --nshard 6
python aggregate.py                                        # -> results_raw.csv, summary.*, site_eval.json

# --- x2 GT benchmark (per dataset) ---
python bench_prep.py <gt_root> <benchroot> --jobs 12 --align 32
bash launch_bench.sh <benchroot>                           # 4 methods x 2 shards = 8 GPUs, + CPU baselines
CUDA_VISIBLE_DEVICES=$g python bench_metrics.py <benchroot> \
  --metrics psnr,psnr_y,ssim,ssim_y,lpips,dists,musiq,clipiqa,maniqa,niqe --shard $g --nshard 8
python bench_dover_videos.py <benchroot> --jobs 20
CUDA_VISIBLE_DEVICES=$g python run_dover.py --dataset <ds> --videos <benchroot>/dover_videos.txt --shard $g --nshard 4
python bench_integrity.py <benchroot> ...                  # -> bench_issues.csv
python bench_aggregate.py <benchroot>                      # -> bench_<ds>_summary.*, site_bench_<ds>.json
```

Every job is resume-safe: a finished (clip, method, metric) is never recomputed, and results are
appended and fsynced per record, so a killed job loses at most the record in flight.