Studio: keep training from failing when a namespace-package shadows unsloth#6269
Conversation
…of unsloth A directory named unsloth without an __init__.py on sys.path (a stray checkout, a partial clone, or a polluted PYTHONPATH) makes the path finder return a namespace package, so the worker's 'from unsloth import FastLanguageModel' fails with 'cannot import name ... (unknown location)'. A normal site-packages install always wins this race, so only source/editable installs are exposed. Before the import, drop the offending sys.path entries, bind the real packages, then restore sys.path so other modules on those entries keep importing. It is a no-op when unsloth already resolves to a real package.
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 82bd9f5adc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| sys.path[:] = saved | ||
|
|
||
|
|
||
| _ensure_real_packages("unsloth_zoo", "unsloth") |
There was a problem hiding this comment.
When the namespace shadow is for unsloth_zoo (for example an editable/source install plus a stray unsloth_zoo directory), this call imports unsloth_zoo directly inside _ensure_real_packages before unsloth.__init__ reaches _gpu_init. _gpu_init deliberately runs fixes before its first import unsloth_zoo (see unsloth/_gpu_init.py lines 69-79 and 103-139), including ROCm bitsandbytes and cache setup, so the recovery path can import zoo with those guards skipped and fail on the same ROCm/Windows cases those fixes handle. Let the guard unshadow paths and import unsloth, allowing unsloth to import zoo after its setup.
Useful? React with 👍 / 👎.
_ensure_real_packages imported the requested names in argument order, so a unsloth_zoo namespace shadow made it import unsloth_zoo directly before unsloth. That skips unsloth.__init__ -> _gpu_init, which runs its ROCm and Windows bitsandbytes fixes before its own import unsloth_zoo, so the recovery path could import zoo with those guards skipped and fail on the ROCm/Windows cases those fixes handle. Import parent-first via reversed(names) so unsloth is imported first and pulls in the real unsloth_zoo after _gpu_init has run; the later cached import is a no-op. This also covers the case where only unsloth_zoo is shadowed: the bad sys.path entry is still dropped and unsloth owns the zoo import. Detection, sys.path pruning, shadow-cache clearing, and restoration are unchanged. Add tests/test_namespace_shadow_guard_pr6269.py: CPU-only subprocess scenarios (only zoo shadowed, both shadowed, only unsloth shadowed, healthy no-op, real package absent, multiple shadow entries) that assert unsloth imports before unsloth_zoo and that sys.path is restored.
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e5c3e5cc8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if not bad: | ||
| return | ||
| saved = list(sys.path) | ||
| sys.path[:] = [e for e in sys.path if e not in bad] |
There was a problem hiding this comment.
When a single sys.path entry contains a valid source checkout for unsloth and a namespace-only unsloth_zoo directory (for example running Studio from an Unsloth checkout while unsloth_zoo is supplied by an editable/meta-path install), this removes that whole entry to hide the zoo namespace. The later importlib.import_module("unsloth") then runs while the only path to the real unsloth package is gone, so the guard turns that source/editable setup into ModuleNotFoundError instead of recovering from the zoo shadow. Consider only hiding the shadowed package path or deferring the unsloth import until its source path is available.
Useful? React with 👍 / 👎.
…w guard Move importlib.invalidate_caches() inside the try/finally so a failure there still restores sys.path, and tighten the import-order comment. Add a test that forces invalidate_caches to raise and asserts sys.path is restored.
… workers A directory named `unsloth` (or `unsloth_zoo`) without an __init__.py on PYTHONPATH/sys.path, a stray source checkout or a polluted PYTHONPATH, makes `import unsloth` resolve to an empty namespace package, so a worker's `from unsloth import FastLanguageModel` dies with a cryptic "cannot import name ... (unknown location)". The LLM training path already recovered from this via `_ensure_real_packages` in trainer.py (PR unslothai#6269), but the inference, export, and embedding-training subprocesses imported Unsloth directly with no guard. Extract that helper into a shared, dependency-free core/import_guards.py and call it before the Unsloth import in every subprocess: it drops the offending sys.path entries, imports the real packages (unsloth before unsloth_zoo so the pre-zoo GPU fixes run), then restores sys.path. trainer.py now imports the shared helper instead of its local copy. Covers both unsloth and unsloth_zoo and both namespace origin forms (None and "namespace"). The existing PR unslothai#6269 test now exercises the shared helper.
…ages (#6532) * Studio: self-heal unsloth namespace-package shadows in all subprocess workers A directory named `unsloth` (or `unsloth_zoo`) without an __init__.py on PYTHONPATH/sys.path, a stray source checkout or a polluted PYTHONPATH, makes `import unsloth` resolve to an empty namespace package, so a worker's `from unsloth import FastLanguageModel` dies with a cryptic "cannot import name ... (unknown location)". The LLM training path already recovered from this via `_ensure_real_packages` in trainer.py (PR #6269), but the inference, export, and embedding-training subprocesses imported Unsloth directly with no guard. Extract that helper into a shared, dependency-free core/import_guards.py and call it before the Unsloth import in every subprocess: it drops the offending sys.path entries, imports the real packages (unsloth before unsloth_zoo so the pre-zoo GPU fixes run), then restores sys.path. trainer.py now imports the shared helper instead of its local copy. Covers both unsloth and unsloth_zoo and both namespace origin forms (None and "namespace"). The existing PR #6269 test now exercises the shared helper. * Studio: distinguish a failed model load from no model in the attach gates A failed load never sets the checkpoint, so the image and audio attach gates fell through to "Load a model before adding images/audio", which reads as if the user simply forgot to pick a model rather than that the load errored. Add a dedicated lastModelLoadError to the chat runtime store, set only when an actual load attempt fails (not on refresh, list, status, or unload errors, which keep using modelsError) and cleared when the next load starts. The image gate (all three call sites) and the audio gate now use it to report a failed load and point at the server logs, while still blocking in exactly the same cases. * Tighten namespace-shadow guard and load-error comments
1 participant
Problem
Starting a training run can fail before it begins with:
The traceback ends at the worker's
from unsloth import FastLanguageModelincore/training/trainer.py.Root cause
(unknown location)meansimport unslothresolved to a namespace package: a directory namedunsloththat has no top-level__init__.py(a stray checkout, a partial clone, or aPYTHONPATHentry that happens to contain one). The path finder returns that directory as a namespace package, which of course has noFastLanguageModel.A normal
pip install unsloth(a regular package in site-packages) always wins this race: the path finder keeps scanning and returns the real package even when a namespace directory appears earlier on the path. The failure only happens whenunslothis reachable solely through a meta path finder that runs after the standard path finder, i.e. a source / editable (PEP 660) install, combined with a strayunslothdirectory onsys.path. The training worker is spawned with the parent'ssys.pathandPYTHONPATH, so it inherits any such directory.Fix
Before the worker imports unsloth, run a small guard that:
unsloth/unsloth_zoocurrently resolve to a namespace shadow,sys.pathentries that hold a shadow directory (one without__init__.py),sys.pathso any other modules living on those entries keep importing.It is a no-op when unsloth already resolves to a real package, so normal installs are unaffected.
Testing
unslothdirectory onPYTHONPATH; confirmedfrom unsloth import FastLanguageModelfails before the guard and succeeds after it, resolving to the real package, withsys.pathrestored.sys.pathrestoration on both success and failure, and a clearModuleNotFoundErrorwhen no real package exists at all.py_compileon the changed file.