Add Anima modular pipeline

[rmatif]

What does this PR do?

Adds modular-only support for Anima, a text-to-image model built on top of the Cosmos Predict2 DiT architecture.

This PR adds:

• AnimaModularPipeline and AnimaAutoBlocks
• AnimaTextConditioner
• checkpoint conversion for the original Anima weights
• LoRA loading support for the Cosmos transformer and Anima text conditioner
• docs and fast modular pipeline tests

Converted weights:
https://huggingface.co/mrfatso/anima-preview3-diffusers

Fixes #13067

cc @tdrussell

Testing

uv run pytest tests/modular_pipelines/anima/test_modular_pipeline_anima.py -q
uv run ruff check src/diffusers/modular_pipelines/anima src/diffusers/models/condition_embedders/condition_embedder_anima.py scripts/convert_anima_to_diffusers.py tests/modular_pipelines/anima/test_modular_pipeline_anima.py
uv run python utils/check_dummies.py
uv run python utils/modular_auto_docstring.py src/diffusers/modular_pipelines/anima

Tested the converted checkpoint locally with txt2img generation and LoRA loading

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? [Feature] Add support for Anima #13067
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

@yiyixuxu @asomoza

[tdrussell]

I'm the creator of Anima. I've run this and confirmed it looks good in terms of correct model implementation and producing images that match the quality of other implementations. I'll let the Diffusers team comment on any Diffusers-specific questions or concerns.

[asomoza]

@rmatif thanks a lot for this, can you please post an example code and a couple of generations with this?

[rmatif]

can you please post an example code and a couple of generations with this?

Sure, here is a small txt2img example using the converted checkpoint:

import torch
from diffusers import AnimaPipeline

pipe = AnimaPipeline.from_pretrained(
    "mrfatso/anima-preview3-diffusers",
    torch_dtype=torch.bfloat16,
)
pipe.to("cuda")

prompt = (
    "masterpiece, best quality, very aesthetic, absurdres, 1girl, solo, silver hair, blue eyes, "
    "long hair, school uniform, sailor collar, cherry blossoms, petals, spring, soft lighting, "
    "looking at viewer, upper body, detailed background"
)
negative_prompt = (
    "worst quality, low quality, score_1, score_2, score_3, blurry, jpeg artifacts, "
    "sepia, signature, artist name"
)

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=1024,
    height=1024,
    num_inference_steps=25,
    guidance_scale=4.0,
    generator=torch.Generator(device="cuda").manual_seed(12341),
).images[0]

image.save("anima.png")

[sayakpaul]

@claude can you do a review?

[github-actions]

Claude finished @sayakpaul's task in 4m 49s —— View job

---

Review of PR #13732: Add Anima pipeline

• Read repository guidelines (AGENTS.md, models.md, pipelines.md, modular.md)
• Review modeling_anima.py (AnimaTextConditioner)
• Review pipeline_anima.py (AnimaPipeline)
• Review __init__.py changes
• Review LoRA integration (lora_pipeline.py, lora_conversion_utils.py)
• Review tests
• Review docs
• Trace call path & check for dead code
• Post final review

---

Overall this is a clean PR that reuses existing infra well (Cosmos transformer, QwenImage VAE, FlowMatchEuler scheduler). The code is readable and the LoRA conversion logic is well-structured. A few issues to fix before merge:

---

Blocking issues

1. _toctree.yml — broken YAML structure (docs won't build)

docs/source/en/_toctree.yml lines 502-506 use tab characters for indentation (the rest of the file uses spaces), and the Anima + AnimateDiff entries are structurally nested under the "Audio" section title (line 501). Anima is an image pipeline and should be a sibling of AuraFlow, Bria, etc., not a child of Audio.

The insertion also appears to have displaced the AnimateDiff entry from its original position.

Fix: insert the Anima entry at the same indentation level as the other image pipelines (e.g. between the Audio section and AuraFlow), using spaces, not tabs. AnimateDiff should remain in its original position.

2. Missing dummy objects

AnimaPipeline and AnimaTextConditioner need entries in src/diffusers/utils/dummy_torch_and_transformers_objects.py. Without them, from diffusers import AnimaPipeline will fail with a confusing ImportError when torch or transformers isn't installed, instead of the friendly "requires torch and transformers" message. Every other pipeline has these — see AnimateDiffPipeline at line 845 for the pattern.

Fix this →

3. AnimaTextConditioner import ordering in src/diffusers/__init__.py

In the _import_structure dict (line 505-506), AnimaPipeline and AnimaTextConditioner are adjacent. But in the TYPE_CHECKING block, AnimaTextConditioner is at line 1314 (after all the AnimateDiff entries), separated from AnimaPipeline at line 1306. Should be right after AnimaPipeline to keep alphabetical consistency with the lazy import structure. Run make style to fix.

---

Non-blocking suggestions

4. F.scaled_dot_product_attention in AnimaTextConditionerAttention

modeling_anima.py:114 calls F.scaled_dot_product_attention directly. The repo convention for ModelMixin subclasses is to use dispatch_attention_fn with the AttentionModuleMixin / processor pattern (see .ai/models.md). AnimaTextConditioner inherits ModelMixin + PeftAdapterMixin and acts like a model. Using F.scaled_dot_product_attention directly means it won't benefit from attention backend dispatch (flash, sage, etc.).

This isn't blocking since the text conditioner is small relative to the transformer and runs once per generation (not in the denoising loop), but it diverges from repo convention. Consider adopting the dispatch_attention_fn pattern in a follow-up.

5. Unused target_token_weights parameter

AnimaTextConditioner.forward() declares target_token_weights (line 251) and uses it in the body (lines 287-288), but the pipeline never passes this argument. From the pipeline's call path, it's dead code. Per AGENTS.md: "do not carry unused method parameters 'for API consistency'". If this is intended for LoRA or external callers, document it; otherwise, remove it.

6. text_embed_dim in the test conditioner doesn't match target_dim

In test_anima.py:77, CosmosTransformer3DModel is created with text_embed_dim=16. The AnimaTextConditioner at line 97 has target_dim=16, which means prompt embeddings are 16-dim. However, the conditioner's out_proj maps model_dim → target_dim (16→16), and the output is what gets passed to the transformer as encoder_hidden_states. The pipeline feeds the output of text_conditioner (dim=target_dim) into the transformer which expects text_embed_dim. These match in the test (both 16), but the naming convention is worth documenting to avoid confusion — target_dim in the conditioner must equal text_embed_dim in the transformer config.

7. Consider # Copied from annotation for retrieve_timesteps

pipeline_anima.py:58 has # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.retrieve_timesteps — good. Just making sure this passes make fix-copies.

8. Denoising loop differs significantly from Cosmos2

The Anima denoising loop (lines 498-544) doesn't use c_in/c_skip/c_out scaling, unlike the Cosmos2 pipeline which computes current_t = current_sigma / (current_sigma + 1) and applies velocity scaling. Since the model creator confirmed parity, this is presumably intentional (Anima was fine-tuned with a different training formulation). Just noting this for maintainer awareness — a brief comment in the pipeline noting the difference from standard Cosmos2 denoising would help future readers.

---

Suggestions / additional info: Dead code trace

I traced the call path from AnimaPipeline.__call__ through the CosmosTransformer3DModel.forward():

Parameters passed by the pipeline:

• hidden_states ✅
• timestep ✅
• encoder_hidden_states ✅
• padding_mask ✅
• return_dict=False ✅

Parameters NOT passed (using defaults):

• block_controlnet_hidden_states=None — controlnet residuals, not applicable for base txt2img
• attention_mask=None — text attention mask is not passed through. The text conditioner already zero-masks padded positions via target_attention_mask, so the output embeddings have zeros for padding, but the transformer's cross-attention still attends to those zero-padded positions. This works because the embeddings are zeroed out, not because attention is masked. Fine in practice.
• fps=None — video-only parameter, correct for image generation
• condition_mask=None — used for video conditioning, correct for image generation

Under default configs (concat_padding_mask=True, extra_pos_embed_type=None in the test config):

• self.learnable_pos_embed is None when extra_pos_embed_type=None → extra_pos_emb path is dead in this config
• use_crossattn_projection=False (default) → crossattn_proj is never instantiated or called
• img_context_dim_in=None (default) → img_context_proj never instantiated

These are all expected to be dead under the Anima config — they're Cosmos-specific features that Anima doesn't use. No issues here.

---

[yiyixuxu]

@rmatif

thanks for the PR! super excited to have Anima in diffusers

on high-level I have two feedbacks/questions:

(1) Can we explore having an AnimaTransformer3DModel instead of a standalone AnimaTextConditioner? Is there any reason you went with this design? I think it'd be more consistent with our conventions to have a dedicated model that (1) has shared layers copied from Cosmos with # Copied from statements and (2) includes this additional conditioning layer as a submodule. Additionally, it'd be closer to the original checkpoint format, which will make future support easier.

(2) Can we support this pipeline from Modular Diffusers directly? Given the very active community ecosystem and the continuous training/release style, Modular is a better fit — see the docs here: https://huggingface.co/docs/diffusers/main/en/modular_diffusers/overview. since you've already implemented in the standard pipeline, it would take a little refactor - happy to provide more info if interseted, we have pretty good docs for AI agents on this and I can point you to reference PRs as well

[tdrussell]

Regarding (1), what if we just subclassed the Cosmos DiT, like ComfyUI does?

The main reason to try to avoid duplicating code is that Anima's DiT architecture is identical to the Cosmos-Predict2 DiT. The only change is the LLM Adapter module (called AnimaTextConditioner in this PR). In ComfyUI the adapter lives as a submodule of the DiT for convenience, but it's not called in the forward() method since it only needs to run once for the entire diffusion process. So regardless of the structure, the pipeline code is going to be calling the adapter "manually" only once.

[yiyixuxu]

@tdrussell

what if we just subclassed the Cosmos DiT?

This isn't something we do in diffusers — all our models are self-contained and inherit from ModelMixin directly. We try to keep the code structure flat and easy to read

In ComfyUI the adapter lives as a submodule of the DiT for convenience, but it's not called in the forward() method since it only needs to run once for the entire diffusion process.

ohh, we usually include text condition layers in forward as well for simplicity — the performance tradeoff is typically non-significant. But if that's not the case for Anima, keeping it as a separate component like this PR would makes sense

[rmatif]

@yiyixuxu My preference is also to keep AnimaTextConditioner as a separate component in this PR

The main reason is that Anima’s DiT is not a new architecture. The denoiser weights and forward path are the Cosmos Predict2 DiT, the Anima-specific part is the LLM adapter that turns Qwen3 hidden states + T5 token ids into the encoder_hidden_states consumed by Cosmos

Since subclassing CosmosTransformer3DModel is not a Diffusers pattern, an AnimaTransformer3DModel would probably mean copying the Cosmos transformer into a new self-contained class just to add one adapter submodule. That feels worse to me for many reasons

The checkpoint conversion does split net.llm_adapter.* into text_conditioner/, but it is still strict and direct, there is no architectural remapping beyond separating the adapter from the unchanged Cosmos DiT

If the preference is still to make an AnimaTransformer3DModel, I can do that, but I think it would mostly be a wrapper/copy around Cosmos rather than a meaningfully different model class

And agree that Modular diffusers is a good fit for Anima, Would it be okay to handle Modular support in a follow-up PR?

[tdrussell]

@yiyixuxu

ohh, we usually include text condition layers in forward as well for simplicity — the performance tradeoff is typically non-significant. But if that's not the case for Anima

It's not the case for Anima. The LLM Adapter is 6 transformer layers with both self- and cross-attention, which is heavier than what is typical in most models (often just a single MLP projection layer). Anima basically has a mini text encoder that is converting from Qwen3 embedding space to T5XXL embedding space for input to the model. It's been a while since I ran the numbers, and I didn't write it down, but I recall the LLM Adapter as being ~10% of the full forward pass. IMO this is enough to warrant being called just once for the entire diffusion loop (and is what ComfyUI does as well).

[yiyixuxu]

sounds good to keep AnimaTextConditione then!

Can we support Anima only through Modular Diffusers, rather than maintaining both? We've been supporting new pipelines through both, but now that Modular is officially released we're looking to shift new pipelines to modular-only. Especially since we expect Anima to be be a very actively developed model, both from the author and the community, the maintenance cost from the standard pipeline could be quite high for us.

[rmatif]

Can we support Anima only through Modular Diffusers, rather than maintaining both?

Fair enough, I moved everything into Modular. Looking forward to your review

Here’s the updated example:

import torch
from diffusers import AnimaAutoBlocks
from diffusers.guiders import ClassifierFreeGuidance

pipe = AnimaAutoBlocks().init_pipeline("mrfatso/anima-preview3-diffusers")
pipe.load_components(torch_dtype=torch.bfloat16)
pipe.update_components(guider=ClassifierFreeGuidance(guidance_scale=4.0))
pipe.to("cuda")

prompt = (
    "masterpiece, best quality, very aesthetic, absurdres, 1girl, solo, silver hair, blue eyes, "
    "long hair, school uniform, sailor collar, cherry blossoms, petals, spring, soft lighting, "
    "looking at viewer, upper body, detailed background"
)
negative_prompt = (
    "worst quality, low quality, score_1, score_2, score_3, blurry, jpeg artifacts, "
    "sepia, signature, artist name"
)

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=1024,
    height=1024,
    num_inference_steps=25,
    generator=torch.Generator(device="cuda").manual_seed(12341),
).images[0]

image.save("anima.png")

[yiyixuxu]

i left one comment,
overall looks good to me, thanks for working on this

[ghunkins]

Thank you for the great work @rmatif!

The new model id of circlestone-labs/Anima-Base-v1.0-Diffusers results in errors due to missing the organization in the pretrained_model_name_or_path part of the modular config.

e.g. https://huggingface.co/circlestone-labs/Anima-Base-v1.0-Diffusers/blob/main/modular_model_index.json#L9

Trace:

Traceback (most recent call last):
    File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 761, in hf_raise_for_status
      response.raise_for_status()
    File "/usr/local/lib/python3.10/site-packages/httpx/_models.py", line 829, in raise_for_status
      raise HTTPStatusError(message, request=request, response=self)
  httpx.HTTPStatusError: Client error '401 Unauthorized' for url
  'https://huggingface.co/Anima-Base-v1.0-Diffusers/resolve/main/text_conditioner/config.json'
      raise HTTPStatusError(message, request=request, response=self)
  httpx.HTTPStatusError: Client error '401 Unauthorized' for url
  'https://huggingface.co/Anima-Base-v1.0-Diffusers/resolve/main/scheduler/scheduler_config.json'
  For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401
  The above exception was the direct cause of the following exception:
  Traceback (most recent call last):
    File "/usr/local/lib/python3.10/site-packages/diffusers/configuration_utils.py", line 424, in load_config
      config_file = hf_hub_download(
    File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 88, in _inner_fn
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1010, in hf_hub_download
      return _hf_hub_download_to_cache_dir(
    File "/usr/local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1161, in _hf_hub_download_to_cache_dir
      _raise_on_head_call_error(head_call_error, force_download, local_files_only)
    File "/usr/local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1795, in _raise_on_head_call_error
      raise head_call_error
    File "/usr/local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1682, in _get_metadata_or_catch_error
      metadata = get_hf_file_metadata(
    File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 88, in _inner_fn
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1604, in get_hf_file_metadata
      response = _httpx_follow_relative_redirects_with_backoff(
    File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 692, in
  _httpx_follow_relative_redirects_with_backoff
      hf_raise_for_status(response)
    File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 835, in hf_raise_for_status
      raise _format(RepositoryNotFoundError, message, response, repo_type=repo_type, repo_id=repo_id) from e
  huggingface_hub.errors.RepositoryNotFoundError: 401 Client Error. (Request ID:
  Root=1-6a1096b9-14602f58443e114a5cb34725;991c7c5a-d821-4d20-8a0d-2c6e3a742e7c)
  Repository Not Found for url: https://huggingface.co/Anima-Base-v1.0-Diffusers/resolve/main/scheduler/scheduler_config.json.
  Please make sure you specified the correct `repo_id` and `repo_type`.
  If you are trying to access a private or gated repo, make sure you are authenticated and your token has the required permissions.
  For more details, see https://huggingface.co/docs/huggingface_hub/authentication
  Invalid username or password.
  tests/test_anima.py::test_anima_basic FAILED

You can override this now by explicitly passing:

pipe.load_components(
    torch_dtype=torch.bfloat16,
    pretrained_model_name_or_path=ANIMA_V1_HF_PATH,
)

@tdrussell can you please update the model config to include the org id so that loading can work out of the box?

Thanks all

[ghunkins]

PR on V1 model is here: https://huggingface.co/circlestone-labs/Anima-Base-v1.0-Diffusers/discussions/1

@tdrussell 🙏

[ghunkins]

PR on V1 model is now merged, thank you @tdrussell.

Is there anything we can do to get this over the finish line? Can't wait to see this on main. Thank you team!

[ghunkins]

More expansive LoRA conversion script to support kohya, etc: https://gist.github.com/ghunkins/05839b2e6b0111be3a98ba2af13e3dda

[yiyixuxu]

hi @rmatif
sorry but I think we are missing a docstring https://github.com/huggingface/diffusers/actions/runs/26606725730/job/78425463328?pr=13732#step:5:25
(hopefully last thing to fix)

[rmatif]

hi @rmatif sorry but I think we are missing a docstring https://github.com/huggingface/diffusers/actions/runs/26606725730/job/78425463328?pr=13732#step:5:25 (hopefully last thing to fix)

Thanks, fixed!
Please let me know if everything is looking good now

[yiyixuxu]

thanks a lot @rmatif @tdrussell @ghunkins