Add LTX-2.X IC LoRA and HDR Pipelines by dg845 · Pull Request #13572 · huggingface/diffusers

dg845 · 2026-04-28T05:18:33Z

What does this PR do?

This PR adds two new LTX-2.X pipelines: LTX2InContextPipeline, which supports in-context (IC) conditioning (used for example by some IC LoRAs) and LTX2HDRPipeline, which supports the newly released high dynamic range (HDR) pipeline (and HDR IC-LoRA) introduced in the LTX LumiVid paper.

This PR also updates LTX2ConditionPipeline to follow the LTX-2 repo's current image conditioning strategy, which overwrites the noisy latents only for first-frame (I2V) conditions and treats non-first-frame as keyframe conditions which are appended to the noisy latents. (Previously, the pipeline overwrote the noisy latents at all latent indices.)

Here is an example of using LTX2InContextPipeline with the Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-In IC LoRA, which biases the generated video toward panning in on the subject:

LTX-2.3 IC LoRA Example Script

import torch

from diffusers import LTX2InContextPipeline
from diffusers.pipelines.ltx2.export_utils import encode_video
from diffusers.pipelines.ltx2.utils import DEFAULT_NEGATIVE_PROMPT


pipe = LTX2InContextPipeline.from_pretrained("dg845/LTX-2.3-Diffusers", torch_dtype=torch.bfloat16)
pipe.enable_model_cpu_offload(device="cuda:0")
pipe.load_lora_weights(
    "Lightricks/LTX-2-19b-LoRA-Camera-Control-Dolly-In",
    adapter_name="ic_lora",
    weight_name="ltx-2-19b-lora-camera-control-dolly-in.safetensors",
)
pipe.set_adapters("ic_lora", 1.0)

# If the IC LoRA uses reference conditions, you can specify them as follows:
# reference_video = load_video("reference.mp4")
# ref_cond = LTX2ReferenceCondition(frames=reference_video, strength=1.0)

prompt = "A flowing river in a forest"
frame_rate = 24.0
video, audio = pipe(
    prompt=prompt,
    negative_prompt=DEFAULT_NEGATIVE_PROMPT,
    # reference_conditions=[ref_cond],
    width=768,
    height=512,
    num_frames=121,
    frame_rate=frame_rate,
    num_inference_steps=30,
    guidance_scale=3.0,
    output_type="np",
    return_dict=False,
)

encode_video(
    video[0],
    fps=frame_rate,
    audio=audio[0].float().cpu(),
    audio_sample_rate=pipe.vocoder.config.output_sampling_rate,
    output_path="ltx2_ic_lora_output.mp4",
)

ltx2_ic_lora_output.mp4

And here is an example of using LTX2HDRPipeline with the Lightricks/LTX-2.3-22b-IC-LoRA-HDR HDR IC-LoRA, using the video above as the reference:

LTX-2.3 HDR IC LoRA Example Script

import torch
from safetensors import safe_open

from diffusers import LTX2HDRPipeline
from diffusers.pipelines.ltx2.export_utils import encode_hdr_tensor_to_mp4
from diffusers.pipelines.ltx2.pipeline_ltx2_hdr_lora import LTX2HDRReferenceCondition
from diffusers.pipelines.ltx2.utils import DISTILLED_SIGMA_VALUES
from diffusers.utils import load_video


pipe = LTX2HDRPipeline.from_pretrained("dg845/LTX-2.3-Distilled-Diffusers", torch_dtype=torch.bfloat16)
pipe.enable_model_cpu_offload(device="cuda")
pipe.load_lora_weights(
    "Lightricks/LTX-2.3-22b-IC-LoRA-HDR",
    adapter_name="hdr_lora",
    weight_name="ltx-2.3-22b-ic-lora-hdr-0.9.safetensors",
)
pipe.set_adapters("hdr_lora", 1.0)

reference_video = load_video("ltx2_ic_lora_output.mp4")
ref_cond = LTX2HDRReferenceCondition(frames=reference_video, strength=1.0)

# Load pre-computed HDR LoRA connector embeddings.
with safe_open(
    "/path/to/ltx-2.3-22b-ic-lora-hdr-scene-emb.safetensors", framework="pt", device="cuda"
) as f:
    connector_video_embeds = f.get_tensor("video_context")
    connector_audio_embeds = f.get_tensor("audio_context")

# `hdr_video` is a linear HDR tensor of shape (batch, frames, H, W, C).
hdr_video = pipe(
    reference_conditions=[ref_cond],
    connector_video_embeds=connector_video_embeds,
    connector_audio_embeds=connector_audio_embeds,
    width=768,
    height=512,
    num_frames=121,
    frame_rate=24.0,
    num_inference_steps=8,
    sigmas=DISTILLED_SIGMA_VALUES,
    guidance_scale=1.0,
    output_type="pt",
    return_dict=False,
)[0]

# Convert the HDR video to a SDR sRGB-tonemapped `.mp4` video.
# You can also save the output to EXR using `save_hdr_video_frames_as_exr`.
# A custom tone-mapper can be specified via the `tone_mapping_fn` argument.
encode_hdr_tensor_to_mp4(
    hdr_video[0],
    output_mp4="ltx2_hdr_lora_output.mp4",
    frame_rate=24.0,
)

ltx2_hdr_lora_output.mp4

This uses the default tone-mapper that simply clips HDR values to [0, 1], which is also used by the original LTX-2 code.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@yiyixuxu
@sayakpaul
@linoytsaban

…nector embeddings

… suggested params for the LTX-2.3 model

…eline to LTX2HDRPipeline

…preserved positional arg ordering

HuggingFaceDocBuilderDev · 2026-04-28T05:27:21Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

yiyixuxu · 2026-05-06T04:07:01Z

+
+# Adapted from ltx_pipelines.utils.media_io.save_exr_tensor
+# https://github.com/Lightricks/LTX-2/blob/41d924371612b692c0fd1e4d9d94c3dfb3c02cb3/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io.py#L609
+def save_exr_tensor(


I think with this PR, encode_hdr_tensor_to_mp4 is enough, no?
can we remove all the other functions here?

yiyixuxu · 2026-05-06T04:18:49Z

        num_frames: int = 121,
        frame_rate: float = 24.0,
-        num_inference_steps: int = 40,
+        num_inference_steps: int = 30,


did they change the default since the release? 🤯

I ended up changing the LTX2ConditionPipeline.__call__ defaults to the LTX-2.3 defaults (since the implementation now follows the newest behavior rather than the original LTX-2.0 behavior); num_inference_steps=30 is the default for LTX-2.3, while num_inference_steps=40 is the default for LTX-2.0.

Let's treat the update on image conditioning strategy as a bug fix and not change anything else

Changed the params back to the previous default values.

… (LTX-2.0) defaults

yiyixuxu

left a bit more feedbacks,
can we add tests too?

yiyixuxu · 2026-05-11T22:37:47Z

    container.close()
+
+
+def simple_tone_map(x: np.ndarray) -> np.ndarray:


let's just move this line of code inside encode_hdr_tensor_to_mp4

yiyixuxu · 2026-05-11T22:37:57Z

+
+# Adapted from ltx_pipelines.utils.medio_io._linear_to_srgb
+# https://github.com/Lightricks/LTX-2/blob/41d924371612b692c0fd1e4d9d94c3dfb3c02cb3/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io.py#L644
+def linear_to_srgb(x: np.ndarray) -> np.ndarray:


same here, let's move this small function

yiyixuxu · 2026-05-11T22:46:54Z

        decode_timestep: float | list[float] = 0.0,
        decode_noise_scale: float | list[float] | None = None,
-        use_cross_timestep: bool = False,
+        use_cross_timestep: bool = True,


why do we change this?

yiyixuxu · 2026-05-11T22:57:40Z

+                prompt_attention_mask = torch.cat([negative_prompt_attention_mask, prompt_attention_mask], dim=0)
+
+            tokenizer_padding_side = "left"
+            if getattr(self, "tokenizer", None) is not None:


can we move this to __init__?

yiyixuxu · 2026-05-11T23:01:29Z

+        )
+        audio_num_frames = round(duration_s * audio_latents_per_second)
+
+        num_mel_bins = self.audio_vae.config.mel_bins if getattr(self, "audio_vae", None) is not None else 64


can you move this to init

yiyixuxu · 2026-05-11T23:01:36Z

+        num_mel_bins = self.audio_vae.config.mel_bins if getattr(self, "audio_vae", None) is not None else 64
+        latent_mel_bins = num_mel_bins // self.audio_vae_mel_compression_ratio
+        num_channels_latents_audio = (
+            self.audio_vae.config.latent_channels if getattr(self, "audio_vae", None) is not None else 8


same here too

yiyixuxu · 2026-05-11T23:02:39Z

+            self.scheduler.config.get("base_shift", 0.95),
+            self.scheduler.config.get("max_shift", 2.05),
+        )
+        audio_scheduler = copy.deepcopy(self.scheduler)


maybe adding another scheduler component? audio_sehcduler?

yiyixuxu · 2026-05-11T23:03:23Z

+                    xm.mark_step()
+
+        # Silence unused-variable lints for `audio_latent_model_input` / `latent_mel_bins`.
+        del audio_latent_model_input, latent_mel_bins


Suggested change

del audio_latent_model_input, latent_mel_bins

yiyixuxu · 2026-05-11T23:04:18Z

+            video = video.permute(0, 2, 3, 4, 1).contiguous()
+            if output_type == "np":
+                video = video.cpu().numpy()
+            elif output_type != "pt":


move this inot check_inputs?

yiyixuxu · 2026-05-11T23:06:48Z

+            # VAE decode returns a video tensor in the VAE's native range ([-1, 1]).
+            decoded = self.vae.decode(latents, timestep, return_dict=False)[0]
+            # HDR postprocess: LogC3 decompress → linear HDR [0, ∞). Always float32 for HDR fidelity.
+            video = self.hdr_video_processor.postprocess_hdr_video(decoded)


we already have a custom image processor, should we just make it take output_type and return expected output?

…mp4 for HDR export

…_video

…e ruff complains

yiyixuxu

thanks!

dg845 added 14 commits April 23, 2026 11:51

LTX-2.X IC LoRA + HDR IC LoRA pipeline implementation draft from Claude

1b0a6d4

Refactor HDR export to accept custom tone-mapping functions

f18211d

Apply parity fixes + refactor + allow HDR LoRA pipeline to accept con…

f5f9656

…nector embeddings

Change LTX2ConditionPipeline default __call__ parameters to match the…

29be81d

… suggested params for the LTX-2.3 model

Improve IC LoRA example and fix some LTX2ICLoraPipeline bugs

5800b0c

Improve HDR IC LoRA example

a3a784d

Clean up the code a bit

4938f67

make style and make quality

e5414c8

Rename LTX2ICLoraPipeline to LTX2InContextPipeline and LTX2HDRLoraPip…

b1e8907

…eline to LTX2HDRPipeline

Improve LTX2InContextPipeline and LTX2HDRPipeline docstrings

0be15f3

Add export function to directly convert HDR tensors to .mp4 files

cd25a51

Clean up the code/docstrings some more

dac2e20

Move new video_self_attention_mask LTX-2.X transformer arg to end to …

7fee0ca

…preserved positional arg ordering

make fix-copies

9460fbc

github-actions Bot added models utils pipelines size/L PR with diff > 200 LOC labels Apr 28, 2026

dg845 requested review from sayakpaul and yiyixuxu April 28, 2026 05:20

Merge branch 'main' into ltx2-hdr-ic-lora-pipeline

d135b24

github-actions Bot added size/L PR with diff > 200 LOC and removed size/L PR with diff > 200 LOC labels Apr 28, 2026

Merge branch 'main' into ltx2-hdr-ic-lora-pipeline

46e5cff

github-actions Bot added size/L PR with diff > 200 LOC and removed size/L PR with diff > 200 LOC labels May 1, 2026

yiyixuxu reviewed May 6, 2026

View reviewed changes

Merge branch 'main' into ltx2-hdr-ic-lora-pipeline

006af7d

github-actions Bot added size/L PR with diff > 200 LOC and removed size/L PR with diff > 200 LOC labels May 6, 2026

Simplify HDR export functions to only export to mp4

094d3d3

github-actions Bot added size/L PR with diff > 200 LOC and removed size/L PR with diff > 200 LOC labels May 6, 2026

dg845 added 2 commits May 9, 2026 04:14

Merge branch 'main' into ltx2-hdr-ic-lora-pipeline

323729b

Revert LTX2ConditionPipeline default __call__ values back to previous…

946f149

… (LTX-2.0) defaults

yiyixuxu reviewed May 11, 2026

View reviewed changes

dg845 and others added 9 commits May 12, 2026 02:13

Inline simple_tone_map and linear_to_srgb into encoder_hdr_tensor_to_…

90ac1dd

…mp4 for HDR export

Apply suggestions from review

eecf6dc

Refactor HDR video processor to handle output_type in postprocess_hdr…

6cf5467

…_video

Rewrite simple_tone_map as function rather than lambda since otherwis…

d65c0e8

…e ruff complains

make style and make quality

283b9df

Merge branch 'main' into ltx2-hdr-ic-lora-pipeline

982a36f

Refactor HDR pipeline to use audio_scheduler as an optional component

c87eab9

Add initial LTX2ConditionPipeline tests

ec42d04

make style and make quality

a7c24fa

github-actions Bot added the tests label May 12, 2026

dg845 and others added 8 commits May 12, 2026 07:30

Add initial LTX2InContextPipeline tests

7411471

Add initial LTX2HDRPipeline tests

2280786

Merge branch 'main' into ltx2-hdr-ic-lora-pipeline

3bcd684

Try to fix failing tests for new LTX-2.X pipelines

bb01569

Merge branch 'main' into ltx2-hdr-ic-lora-pipeline

ba9c9b0

Propagate suggestions from HDR pipeline to IC and condition pipelines

7083eaf

Merge branch 'main' into ltx2-hdr-ic-lora-pipeline

47d3031

Merge branch 'main' into ltx2-hdr-ic-lora-pipeline

1dbd344

yiyixuxu approved these changes May 14, 2026

View reviewed changes

Merge branch 'main' into ltx2-hdr-ic-lora-pipeline

b96eba4

dg845 merged commit 037efda into main May 15, 2026
13 of 16 checks passed

dg845 deleted the ltx2-hdr-ic-lora-pipeline branch May 15, 2026 00:27

		container.close()


		def simple_tone_map(x: np.ndarray) -> np.ndarray:

Uh oh!

Conversation

dg845 commented Apr 28, 2026

What does this PR do?

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Apr 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants