Studio: video inference backend with LTX-2 and LTX-2.3 support#6874
Studio: video inference backend with LTX-2 and LTX-2.3 support#6874danielhanchen wants to merge 38 commits into
Conversation
Text-to-video lands as a SIBLING of the image diffusion backend, not a mode of
it: video pipelines take frame/fps arguments, return frame stacks plus, for
LTX-2, synchronized audio, and persist MP4s -- none of the image module's
img2img/inpaint/ControlNet/LoRA surface applies. The image backend's hardware
and optimisation layers are imported unchanged (device/dtype resolution, memory
planning + offload tiers, attention backends, speed profiles, FBCache), and the
load-token/cancel-event concurrency skeleton is copied verbatim so lifecycle
behaviour cannot diverge.
core/inference/video_families.py: a pure VideoFamily registry (no torch) with
the ltx-2 entry -- LTX2Pipeline + LTX2VideoTransformer3DModel, base
Lightricks/LTX-2, unsloth/LTX-2.3-GGUF as the curated GGUF source, audio on,
frame lattice k*8+1, /32 resolutions with a vertical preset, and measured bf16
component sizes (the Gemma3-27B text encoder outweighs the 19B DiT itself).
MoE fields (transformer_2, guidance_scale_2) are declared now so the Wan2.2
A14B family lands later without churning the schema.
core/inference/video.py: VideoBackend with async begin_load + cache-scan
download progress, GGUF / single-file / full-pipeline loads (the GGUF DiT
assembles onto the base repo exactly like the image path), generation with
frame/size snapping BEFORE latents allocate, per-step progress + ETA and
cooperative cancel via the standard diffusers callback, and MP4 (H.264) export
through diffusers' PyAV encoder with the audio track muxed when the family
produces one. VAE tiling is always on: decoding a 100+ frame clip is the
memory peak, and the frames-aware estimate_video_runtime_mib (new, in
diffusion_memory) feeds the planner where the pixel-area image estimate would
badly undershoot. Loads are gated to unsloth/*, the official Lightricks base
repos, or local paths; PyAV availability is checked at load time so a missing
encoder cannot fail a clip after a multi-minute denoise.
core/inference/video_gallery.py: {id}.mp4 + {id}.json recipe sidecar pairs
under studio_root()/videos (an MP4 has no PNG text chunk to embed the recipe
in), with the image gallery's id/containment guards, newest-first listing that
skips orphans, delete/clear.
gpu_arbiter gains the VIDEO owner: ownership is exclusive, so the existing
evict-the-current-owner already generalises to chat/image/video all evicting
each other. The av (PyAV) dependency joins requirements/studio.txt.
Tests: video family detection/snapping/defaults, backend lifecycle on a faked
torch/diffusers runtime (GGUF assembly, shape snapping, distilled defaults,
cancel/progress, sentinel), gallery roundtrip/containment/orphans. 52 new
tests green plus the arbiter suite.
…dels routes/video.py mirrors the /images/* routes one-for-one: validate-before-evict load ordering (a bad pick must not evict a working chat model and then 400), the training-active interlock, the device-gated GPU arbiter handoff with the new VIDEO owner, the exact-match sentinel mapping (VIDEO_NOT_LOADED_MSG / VIDEO_CANCELLED_MSG to 409, ValueError/FileNotFoundError to 400 with native paths redacted, everything else a sanitized 500), and the gallery CRUD shape with fetch-one-extra has_more paging. Generate persists the encoded MP4 plus its full recipe through video_gallery.save and returns the gallery record; the file endpoint serves video/mp4 with an immutable Cache-Control, 404 on any id that fails the containment check. models/inference.py gains the video request/response set (VideoLoadRequest, VideoGenerateRequest/Response, GalleryVideo, gallery list, both progress shapes, VideoGenerationDefaults nested in VideoStatusResponse), reusing DiffusionResolvedControl for the resolved-provenance badges. The router is registered in main.py after the images router under the same /api/inference prefix and auth dependency. Tests: 20 route tests on a stubbed backend + real tmp gallery (load happy path and arbiter acquisition, 400/409 mappings, generate persistence round trip, mp4 file serving + 404, delete/clear, unload releases ownership); the diffusion route suite stays green after the shared models edit.
diffusers 0.39 ships every LTX-2.3 model class but its single-file loader maps all LTX-2 checkpoints to the 2.0 config, so 2.3 checkpoints (9-row modulation tables, gated attention, per-modality connectors) fail a shape check at load. The community transformer-only GGUFs also lack the text projections, VAEs, and vocoder that 2.3 moved out of the transformer. New core/inference/video_ltx2.py detects a 2.3 checkpoint from its header (6 vs 9 modulation rows, no weight data read) and assembles the full pipeline: the DiT through from_single_file with the 2.3 config overrides and the prompt_adaln key renames the stock converter lacks, the 8-layer per-modality connectors from the same checkpoint plus the text projection companion file, and the 2.3 video VAE, audio VAE, and BWE vocoder from the companion files in unsloth/LTX-2.3-GGUF. Configs and rename tables mirror diffusers' own scripts/convert_ltx2_to_diffusers.py, which the library loader has not absorbed yet. Assembled through the constructor because the base repo pins LTX2Vocoder while 2.3 needs LTX2VocoderWithBWE and the from_pretrained type gate rejects the substitution. Verified on a B200: distilled-1.1 Q4_K_M GGUF loads in 37s, generates a 49-frame 768x512 clip with synchronized audio in 18s (8 steps), frames on-prompt and non-black, container decodes fully. Meta-tensor validation confirms exact key and shape match for all five converted components. Unit tests cover 2.3 detection (gguf + safetensors headers), combined checkpoint partitioning, and companion-set choice.
The Lightricks/LTX-2.3-fp8 checkpoints store float8 weights with per-tensor weight_scale and input_scale companions (verified from the file headers: 1496 F8_E4M3 tensors, 2924 scale tensors). A plain dtype cast would silently corrupt every quantized layer, so the 2.3 assembly now detects the companions and raises with a pointer to the GGUF quants, which offer comparable fidelity through the supported path. Dequantizing the scaled fp8 layout is a possible follow-up.
for more information, see https://pre-commit.ci
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a local text-to-video inference backend using diffusers, including model loading, memory planning, video generation, and a disk-backed gallery for persistence. Key feedback includes using FileResponse instead of Response to serve video files efficiently with range request support, avoiding a redundant resolution of checkpoint_path in the pipeline loading phase, and optimizing gallery paging with high offsets to reduce synchronous disk I/O.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| headers = {"Cache-Control": "private, max-age=31536000, immutable"}, | ||
| ) | ||
|
|
||
|
|
||
| @router.delete("/video/gallery/{video_id}") | ||
| async def delete_gallery_video(video_id: str, current_subject: str = Depends(get_current_subject)): | ||
| from core.inference import video_gallery |
There was a problem hiding this comment.
Using Response(content=data) to serve video files requires reading the entire file into memory synchronously, which can lead to high memory usage and potential Out-Of-Memory (OOM) crashes under concurrent load. Additionally, it does not support HTTP range requests, which prevents browsers from seeking or scrubbing through the video.
Using FileResponse avoids loading the file into memory and automatically supports streaming and range requests.
| headers = {"Cache-Control": "private, max-age=31536000, immutable"}, | |
| ) | |
| @router.delete("/video/gallery/{video_id}") | |
| async def delete_gallery_video(video_id: str, current_subject: str = Depends(get_current_subject)): | |
| from core.inference import video_gallery | |
| from fastapi.responses import FileResponse | |
| # Use FileResponse to support streaming, range requests, and avoid loading the entire file into memory. | |
| return FileResponse( | |
| path, | |
| media_type = "video/mp4", | |
| headers = {"Cache-Control": "private, max-age=31536000, immutable"}, | |
| ) |
| "config": base, | ||
| "subfolder": "transformer", |
There was a problem hiding this comment.
The checkpoint path is already resolved and stored in checkpoint_path during the memory planning phase (line 444) under the same kind != "pipeline" condition. Resolving it again here is redundant and can be avoided by reusing the existing checkpoint_path variable.
| "config": base, | |
| "subfolder": "transformer", | |
| transformer_cls = getattr(diffusers, fam.transformer_class) |
References
- To improve efficiency, avoid redundant data iterations. Combine checks and transformations into a single loop and return computed values for callers to reuse.
| want = None if limit is None else offset + limit | ||
| records = [] | ||
| for path in paths: | ||
| meta = _read_meta(_sidecar_path(path.stem)) | ||
| if meta is None: # no readable sidecar (orphan mp4) — skip | ||
| continue | ||
| records.append(_record(path.stem, meta)) | ||
| if want is not None and len(records) >= want: | ||
| break | ||
| return records[offset:] if limit is None else records[offset : offset + limit] |
There was a problem hiding this comment.
Paging with high offsets can cause a massive amount of synchronous disk I/O because it reads and parses the JSON sidecar for every single video up to the offset. We can optimize this by only checking file existence with is_file() for skipped records. For records within the requested window, we should call _read_meta directly and use its result to check for existence, avoiding a redundant is_file() check beforehand to prevent redundant file traversals.
| want = None if limit is None else offset + limit | |
| records = [] | |
| for path in paths: | |
| meta = _read_meta(_sidecar_path(path.stem)) | |
| if meta is None: # no readable sidecar (orphan mp4) — skip | |
| continue | |
| records.append(_record(path.stem, meta)) | |
| if want is not None and len(records) >= want: | |
| break | |
| return records[offset:] if limit is None else records[offset : offset + limit] | |
| records = [] | |
| valid_count = 0 | |
| for path in paths: | |
| sidecar = _sidecar_path(path.stem) | |
| if valid_count < offset: | |
| if sidecar.is_file(): | |
| valid_count += 1 | |
| else: | |
| meta = _read_meta(sidecar) | |
| if meta is not None: | |
| records.append(_record(path.stem, meta)) | |
| valid_count += 1 | |
| if limit is not None and len(records) >= limit: | |
| break | |
| return records |
References
- To improve efficiency, avoid redundant file traversals. If a function calculating a property (e.g., size) returns a zero/null value when no files are found, use its result to check for existence instead of calling a separate existence-checking function.
There was a problem hiding this comment.
Galleries here are small (page size 50) and the sidecar read IS the validity check; counting skipped records by is_file alone would make pagination disagree with the read path on a corrupt sidecar. Keeping the simple loop.
The 1.22.0 release adds _sandbox.py (the client for HF Jobs sandboxes, including the bootstrap that downloads HF's own sandbox server binary) and the scanner flags it plus three long-standing while True retry and pagination loops as CRITICAL. Reviewed all four against the upstream repo and the 1.22.0 wheel: legitimate library code. Entries generated with scan_packages.py --write-baseline and verified to suppress with exit 0.
The video backend never cleared FBCache residuals between clips, so with the step cache engaged a second generation at a different resolution would hit stale state from the first. Mirrors the image backend fix from #6872: call the transformer level _reset_stateful_cache (reset_stateful_hooks only exists on the HookRegistry in diffusers 0.39), covering transformer_2 for the Wan dual expert, only when a cache is engaged.
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3c91c60687
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if components is not None | ||
| else None | ||
| ) | ||
| model_dense_mib = transformer_mib |
There was a problem hiding this comment.
When loading GGUF or single-file video checkpoints, model_dense_mib is set to only the transformer size while the ~50+ GiB text encoder/VAE companions are passed separately. plan_diffusion_memory() uses model_dense_mib for the resident required check, so auto/fast modes can choose OFFLOAD_NONE on GPUs that fit the quantized transformer but not the LTX-2 companions, then OOM while from_pretrained() loads the base components. The resident estimate needs to include companion_mib as well, while still passing it separately for the group-offload floor.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catches in this batch. The resident estimate, the generation exit barrier, the cache before compile ordering, the GGUF dequant uninstall, the base_repo gate, the local path preflight, the placement recheck, the engaged only status flags, and the FileResponse streaming are all in.
| if self._active_generate_cancel is not None: | ||
| self._active_generate_cancel.set() | ||
| self._teardown_state() |
There was a problem hiding this comment.
If /video/load is called while generate() is still denoising, this only signals the cancel event and immediately tears down _state before allocating the replacement pipeline. The running generation holds its own reference until the next callback/return, so the old video model can remain in VRAM concurrently with the new load and cause warm model swaps to OOM. This should mirror the image backend by taking _generate_lock after signaling cancel so the previous denoise exits before freeing/reallocating the pipeline.
Useful? React with 👍 / 👎.
| fam = detect_video_family(kwargs["repo_id"], kwargs.get("family_override")) | ||
| kind = resolve_video_model_kind(kwargs.get("gguf_filename"), kwargs.get("model_kind")) | ||
| base = ( | ||
| kwargs["repo_id"] | ||
| if kind == "pipeline" | ||
| else resolve_video_base_repo(fam, kwargs.get("base_repo")) |
There was a problem hiding this comment.
For a local directory or generic repo whose family is only present in gguf_filename (for example /models plus ltx-2.3-...gguf), validate_load_request() accepts the request by checking the combined path, but the background loader recomputes fam from repo_id alone. That leaves fam as None and resolve_video_base_repo() fails after the route has already handed off the GPU, so valid local checkpoint picks never load unless the directory name also contains ltx.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
The loader thread calls the same validate_load_request with gguf_filename, which does the combined repo plus filename detection, so a local dir plus an ltx named GGUF resolves the family in both places.
| if kind in ("gguf", "single_file") and not gguf_filename: | ||
| raise ValueError("A gguf/single_file load needs the checkpoint filename.") |
There was a problem hiding this comment.
This preflight only checks that a single-file request has some filename, so a path-shaped local model like /tmp/ltx-2-missing or a missing child under an existing directory passes validation when the family is inferred or overridden. The route then calls acquire_for(VIDEO) before the background _resolve_checkpoint_path() fails, evicting a working chat/image model for a request that could have been rejected synchronously; mirror the image backend's local path and child-file checks here.
Useful? React with 👍 / 👎.
| speed_optims = apply_speed_optims( | ||
| pipe, | ||
| target, | ||
| is_gguf = kind == "gguf", | ||
| family = fam, | ||
| speed_mode = effective_speed, | ||
| offload_active = plan.offload_policy != "none", | ||
| ) |
There was a problem hiding this comment.
When a user requests transformer_cache=fbcache together with a compiled tier such as speed_mode=max (or a dense non-GGUF load where default compiles repeated blocks), this runs apply_speed_optims() before the cache is installed and never passes cache_active=True. That lets the speed layer compile with fullgraph=True; the later FBCache hook adds a graph break, so the first generation can fail instead of running cached. Resolve/apply the cache before speed, as the image backend does, or pass a cache-active flag based on the request.
Useful? React with 👍 / 👎.
| with self._lock: | ||
| state, self._state = self._state, None | ||
| if state is not None: | ||
| restore_backend_flags(state.backend_flags) |
There was a problem hiding this comment.
GGUF video loads default to the default speed profile, so apply_speed_optims() can install the process-wide compiled GGUF dequantizer. This teardown only restores torch backend flags; after unloading or replacing a video model, later GGUF image/video loads that requested speed_mode=off can still inherit the compiled dequant patch, breaking the bit-identical off path. Call the same diffusion_gguf_compile.uninstall_all() cleanup used by the image backend.
Useful? React with 👍 / 👎.
| model_kind = model_kind, | ||
| ) | ||
| kind = resolve_video_model_kind(gguf_filename, model_kind) | ||
| base = repo_id if kind == "pipeline" else resolve_video_base_repo(fam, base_repo) |
There was a problem hiding this comment.
For GGUF/single-file video requests, an arbitrary base_repo is accepted here and later used to load the pipeline companions, bypassing the non-GGUF trusted-repo gate that only checked repo_id. In a shared Studio, a user can pick a safe GGUF plus an untrusted remote base_repo and make the backend deserialize that repo's components after the GPU handoff. Apply the same official/unsloth/local allowlist to base_repo or ignore arbitrary remote bases.
Useful? React with 👍 / 👎.
| vae_tiling = vae_tiling, | ||
| memory_mode = plan.requested_mode, | ||
| speed_mode = effective_speed, | ||
| speed_optims = tuple(speed_optims or ()), |
There was a problem hiding this comment.
apply_speed_optims() returns a dict of optimization flags, but tuple(speed_optims or ()) stores every key, including entries whose value is False. As a result /video/status can claim channels_last, compiled, tf32, etc. are active even when speed_mode=off or an optimization was unsupported, which misleads the UI/status provenance. Filter the dict to keys with truthy values before committing state.
Useful? React with 👍 / 👎.
| speed_mode = effective_speed, | ||
| offload_active = plan.offload_policy != "none", | ||
| ) | ||
| offload_policy, vae_tiling = apply_memory_plan(pipe, plan, device = device, logger = logger) |
There was a problem hiding this comment.
If chat or image inference evicts VIDEO while a new video load is in this finalization section, unload() bumps _load_token while _state is still None and returns to the new owner, but the stale load worker does not check the token again until after apply_memory_plan() has moved/offloaded the pipeline. That lets a cancelled video load allocate VRAM after the arbiter has handed the GPU to another backend, causing the same cross-backend OOM this arbiter is meant to prevent. Recheck cancellation immediately before placement or serialize this placement phase with unload.
Useful? React with 👍 / 👎.
| path = await asyncio.to_thread(video_gallery.video_path, video_id) | ||
| if path is None: | ||
| raise HTTPException(status_code = 404, detail = "Video not found.") | ||
| data = await asyncio.to_thread(path.read_bytes) |
There was a problem hiding this comment.
Fetching a gallery clip currently reads the entire MP4 into a Python bytes object before building the response. At the allowed video sizes/frame counts, a generated clip can be hundreds of MB, and concurrent downloads will duplicate that memory in the backend; returning a FileResponse/streaming response preserves the same auth check and cache headers without buffering the whole file.
Useful? React with 👍 / 👎.
Review round on the video backend: - the resident memory check now budgets transformer plus companions like the image backend, instead of letting auto pick a resident placement that OOMs while the LTX text encoder and VAEs load - a new load waits for the signalled in flight generation to exit before tearing down the old pipeline, so two models never share VRAM during a swap - the load worker rechecks its token right before placement, narrowing the window where a cancelled load could put weights on a GPU the arbiter already handed to another backend - the step cache installs before the speed profile and compile now keys fullgraph off an active cache, matching the image order; compiling fullgraph first crashed the first cached generation - teardown uninstalls the process wide compiled GGUF dequantizer so a later speed off load gets the bit identical path - explicit base_repo goes through the same trust gate as non GGUF repo ids, and local checkpoint paths are verified during validation, before the route evicts a resident model - status reports only the speed optimisations that actually engaged - the gallery file route streams via FileResponse with range support instead of buffering whole clips - the build step reuses the checkpoint path resolved during planning
for more information, see https://pre-commit.ci
A bare from_pretrained snapshot of Lightricks/LTX-2 pulls the whole 314 GB repo: 170 GB of packaged root checkpoints and a second 50 GB text-encoder shard set, when the pipeline reads about 93 GB. Build the needed file list once (shared with the progress estimate so the two cannot disagree), download it per file with cancellation, and hand from_pretrained the local snapshot dir. Clamp the progress counter to the estimate so stale cache blobs can no longer report over 100 percent.
for more information, see https://pre-commit.ci
A 2.3 checkpoint (GGUF or single file) carries the DiT and, with its extras files, the connectors, both VAEs and the vocoder; only the 2.0 base repo's scheduler, text encoder and tokenizer are read. Detect 2.3 from the checkpoint header after the pull, re-estimate, and scope the base pre-download accordingly (about 6 GB less per fresh install).
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
… for LTX-2.3 A warm-cache predownload sweep never consults the cancel event (each cached file returns instantly), so an unload during it was ignored until a cold file hit the network. The 2.3 detection also only probed bare-file local repos; resolve directory repos through the same child resolver the loader uses so their base pull is scoped too.
for more information, see https://pre-commit.ci
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Stacked on #6873. First of four video PRs: backend engine, routes, and the LTX-2 family.
What this adds
Video backend (
core/inference/video.py)gpu_arbitergains a VIDEO owner and now evicts all non-owner backends on acquire, so chat, image, and video loads always displace each other cleanly./api/inference/video/*route surface.avdependency for mp4 export with synchronized audio.LTX-2 family, including LTX 2.3 single files and GGUF
video_ltx2.pyfixes this end to end: detects 2.3 by the transformer scale_shift_table row count, applies the 2.3 config overrides, renames checkpoint keys to the diffusers layout, pulls the companion text projection, video VAE, audio VAE, and vocoder weights the transformer-only GGUFs do not carry, and assembles the pipeline from components. Works for safetensors and GGUF (unsloth/LTX-2.3-GGUF).Lightricks/LTX-2.3-fp8) are refused with a clear error pointing at the GGUF path, instead of producing garbage.Verification
test_video_families.py,test_video_backend.py,test_video_routes.py,test_video_gallery.py, arbiter three-way eviction tests. CI-sim green.