Studio: HunyuanVideo 1.5 family and video quality gate#6877
Studio: HunyuanVideo 1.5 family and video quality gate#6877danielhanchen wants to merge 36 commits into
Conversation
HunyuanVideo-1.5 (the 8.3B DiT with Qwen2.5-VL + ByT5 text encoders) loads through the hunyuanvideo-community Diffusers repacks; tencent's own repo is the original non-diffusers layout and cannot load as a pipeline, so only the community 480p/720p t2v repos are trusted. Two pipeline quirks, both verified against pipeline_hunyuan_video1_5.py in diffusers 0.39, shape the wiring: - __call__ takes no guidance kwarg: CFG lives on the pipeline's guider component (ClassifierFreeGuidance, shipped at scale 6.0). The family gains guidance_via_guider and generate() writes the requested scale onto pipe.guider instead of passing cfg_kwarg, which the pipeline would reject. - __call__ has no callback_on_step_end: progress and cancellation fall back to a scheduler.step wrapper (one call per denoise step), installed for the duration of the call and always restored. Cancellation unwinds the loop by raising through the wrapper and surfaces the same cancelled sentinel the callback path uses. The VAE compresses 16x spatial / 4x temporal, so sizes snap to /16 and frame counts to 4k+1. The transformer declares _repeated_blocks and CacheMixin, so the regional compile profile and the step cache both apply unchanged. scripts/video_quality.py is the video accuracy gate, the analogue of scripts/diffusion_quality.py with the same pure-numpy PSNR/SSIM math so image and video budgets compare: fixed prompt/seed/shape, one short clip per candidate against a reference clip, per-frame SSIM/PSNR over sampled frames, a temporal-consistency deviation (motion-energy series error, catching flicker SSIM alone misses), black-frame/NaN collapse checks, an audio RMS silence trip-wire for LTX-2, and wall time + peak VRAM per candidate. Verdicts map the standing budget: ssim >= 0.75 passes, >= 0.50 warns, anything lower or any collapse fails. --selftest runs the metric path on synthetic clips with no GPU or model.
The A14B gate run crashed mid-denoise: accelerate offload hooks move modules with Module.to(), and torchao quantized tensors reject that (aten._has_compatible_shallow_copy_type is unimplemented, raised from compute_should_use_set_data). The 114 GB dual DiT plans model offload by default, so int8 there was a guaranteed crash. The loader now skips quant whenever the plan resolves to any offload policy, logs why, and surfaces the reason in the resolved record; a user who wants both can pin a resident memory mode. A dense DiT under offload beats a crashed one.
for more information, see https://pre-commit.ci
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for the HunyuanVideo-1.5 model family across the frontend and backend, introducing a custom scheduler-step progress wrapper for pipelines lacking native step callbacks, and skipping transformer quantization under offload to prevent crashes. It also introduces a video quality-vs-cost evaluation harness (video_quality.py). The review feedback identifies two robustness issues in the new quality harness: a potential IndexError when video generation returns zero frames, and a loophole where NaN audio RMS values bypass the silent collapse check.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| ref_frames: Any, | ||
| cand_frames: Any, |
There was a problem hiding this comment.
If either the reference or candidate video generation fails or returns an empty clip, n will be 0. In this case, idx will contain 0, leading to an IndexError when attempting to access ref_frames[0] or cand_frames[0]. Adding a defensive check for n == 0 prevents the quality gate script from crashing and allows it to fail gracefully.
n = min(len(ref_frames), len(cand_frames))
if n == 0:
return {
"frames_compared": 0,
"psnr_mean": 0.0,
"ssim_mean": 0.0,
"temporal_deviation": math.inf,
"min_luma": 0.0,
"has_nan": True,
}
idx = sorted({int(round(i * (n - 1) / max(1, sample_count - 1))) for i in range(sample_count)})| ) | ||
| return {"ref_rms": ref_rms, "cand_rms": cand_rms, "silent_collapse": silent_collapse} |
There was a problem hiding this comment.
If the candidate audio generation collapses to NaN values, cand_rms will be nan. Since comparisons with nan (like cand_rms < 1e-4) always evaluate to False, the silent_collapse check will not be triggered, and the NaN collapse in audio will go undetected. Adding an explicit check for NaN in the candidate's audio RMS ensures that such failures are correctly caught and marked as FAIL.
| ) | |
| return {"ref_rms": ref_rms, "cand_rms": cand_rms, "silent_collapse": silent_collapse} | |
| cand_rms = audio.get("cand_rms") | |
| audio_nan = cand_rms is not None and math.isnan(cand_rms) | |
| if metrics["has_nan"] or audio_nan or metrics["min_luma"] < 0.02 or audio.get("silent_collapse"): | |
| return "FAIL" |
There was a problem hiding this comment.
Both fixed: empty decodes now return a zero-frame FAIL record, and a NaN candidate RMS counts as a silent collapse inside audio_metrics so the verdict logic stays unchanged.
# Conflicts: # studio/backend/tests/test_video_families.py
# Conflicts: # studio/backend/core/inference/video.py
|
@codex review |
An empty or corrupt clip decode crashed clip_metrics on frame indexing; it now returns a zero-frame FAIL record. A NaN candidate audio RMS compared False against every threshold and slipped past the silence trip-wire; NaN now counts as a collapse.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6d9c8cce63
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| except _VideoGenerationCancelled: | ||
| raise RuntimeError(VIDEO_CANCELLED_MSG) from None |
There was a problem hiding this comment.
When HunyuanVideo-1.5 is loaded with model/group/sequential CPU offload and a user cancels, this exception path exits the pipeline before its normal maybe_free_model_hooks() cleanup runs. The callback-based path lets the pipeline return normally before raising the cancelled sentinel, but the scheduler-wrapper path skips that cleanup, so cancelled HV15 generations can leave modules/hooks in the onloaded state and retain VRAM for the next request. Please run the pipeline cleanup before mapping _VideoGenerationCancelled to VIDEO_CANCELLED_MSG.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Real: the wrapper path skipped the pipeline end-of-call cleanup. generate() now calls maybe_free_model_hooks before raising the cancelled sentinel, and the cancel test asserts it ran.
| # original non-diffusers layout: config.json, no model_index.json, so it | ||
| # cannot load through HunyuanVideo15Pipeline at all). | ||
| "hunyuanvideo-community/hunyuanvideo-1.5-diffusers-480p_t2v", | ||
| "hunyuanvideo-community/hunyuanvideo-1.5-diffusers-720p_t2v", |
There was a problem hiding this comment.
When someone loads hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v through the API, this allowlist lets the load pass but the only Hunyuan family entry still hard-codes 480p presets; generate() then fills omitted width/height from fam.resolution_presets[0], so the 720p checkpoint silently defaults to 832x480 and the loader also plans runtime memory for that smaller shape. Please add a 720p-specific family/default selection or avoid trusting this repo until the defaults match it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Added a dedicated 720p family entry (same guider config, 6.0 on both repos; verified) whose repo-id alias outranks the generic token, so the 720p repack now defaults to 1280x720-class presets.
# Conflicts: # studio/backend/tests/test_video_backend.py
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
…ks on cancel The trusted-repo allowlist admitted the 720p t2v repack while the only Hunyuan family entry carried 480p presets, so a 720p load silently defaulted to 832x480. It now resolves a dedicated entry whose repo-id alias outranks the generic token (same guider config, verified: both repos ship guidance 6.0). A scheduler-wrapped cancel unwinds pipe call by exception and skips the pipeline's end-of-call maybe_free_model_hooks, leaving onloaded offload modules on the GPU until the next request; generate() now frees them before surfacing the cancelled sentinel.
# Conflicts: # studio/backend/tests/test_video_backend.py
|
@codex review |
for more information, see https://pre-commit.ci
|
@codex review |
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Stacked on #6876. Last of the video stack: the HunyuanVideo 1.5 family plus the accuracy gate that backs the quant and speed defaults.
What this adds
HunyuanVideo 1.5 (
hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v)guidance_via_guiderfamily flag), and progress plus cancel run through a scheduler.step wrapper that counts steps and always restores the original method.Quant guard under offload
Video quality gate (
scripts/video_quality.py)Gate results (B200, 832x480x33 at 20 steps unless noted)
Interpretation for the LTX rows: at few-step distilled schedules quant noise shifts the trajectory, so the clips compose differently rather than degrade. Q2 and Q4 score the same SSIM and side-by-side frames are at visual parity, so the WARN is composition divergence, not quality loss. int8 on B200 is a memory unlock, not a speed win: bf16 eager stays the default on large GPUs.
Verification