Package scanners: close fail-open gaps in the sdist fallback and hidden-payload paths (#6359)

* Package scanners: close fail-open gaps in the sdist fallback and hidden-payload paths

Follow-up hardening on the now-blocking scanners so the enforcing gate cannot
report clean while a malicious artifact goes unscanned.

scan_packages.py
- Hidden payload: also flag a network call AND an os/subprocess exec that live
  only in a blanked docstring/string of an exec/eval file (the fetch-then-run
  shape of an exec(__doc__) dropper). Either alone in real code was already
  covered; hidden together they are the payload.
- Pinned releases fail closed: _release_files no longer falls back to the latest
  artifact when a pinned version is missing or empty, so a yanked/bad pin is an
  error instead of a different file being scanned in its place.
- requires_dist is read from the pinned release's metadata, not the project-level
  (latest) document, so a sdist-only pin follows its own dependency tree.
- Environment markers are evaluated (PEP 508) instead of dropping any marker that
  merely contains the word extra, so default-true markers like extra != 'dev' are
  kept; conservative fallback keeps a dep on any uncertainty.
- Transitive recovery is a depth-bounded worklist: a wheel dependency whose own
  child is sdist-only is fetched (--no-deps) and scanned, then its children are
  recovered in turn, rather than being silently skipped.

scan_npm_packages.py
- Baseline keys use the package-relative path instead of the basename, so the
  same basename in a different directory is not over-suppressed.

Tests cover each case; full scripts pass AST and ruff checks.

* Address review: tighten marker scope, decoy-proof the dropper check, fail closed on missing pin metadata

- Markers: keep any dep whose marker can hold on another install target
  (sys_platform == 'win32', python_version == '3.13'); only drop a marker that
  depends solely on extra and is false with no extra. A scanner runs on one
  target but must cover code installed on others. Pure-extra markers are
  evaluated against default_environment() with extra unset.
- Hidden dropper: the network+exec docstring check now inspects the removed
  (blanked) span directly, so a benign visible network or subprocess call cannot
  mask a payload that still lives in a docstring. Carrier checks stay
  blanked-only (an in-code carrier is already caught by the normal check), so
  corpus findings are unchanged.
- requires_dist: a pinned version whose own metadata cannot be fetched recovers
  nothing rather than substituting the latest release's dependency tree.
- Transitive recovery: the last-ditch direct-sdist branch also chases the
  recovered package's declared deps, matching the other branches.
- npm baseline: schema bumped to v2 (package-relative keys); a pre-v2 baseline
  with entries is ignored (fail closed) instead of mis-applying basename keys.

Tests cover each case; scripts pass AST, ruff, and the import-hoist verifier.

* Scanner: exclude comments from hidden-payload check, flag missing pin metadata as incomplete

Hidden network+exec detection now inspects only docstring/string spans (what exec(__doc__)/exec(<str>) can actually run), so a real exec() beside comments that mention a network and a subprocess call no longer false-positives. Missing pinned-release metadata in transitive recovery records a download_error so the --with-deps path fails closed instead of treating it as no dependencies. Adds regression tests for both.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: danielhanchen

scripts/scan_npm_packages.py

@@ -1445,14 +1445,20 @@ def scan_one(pkg: PackageEntry, workspace: Path) -> tuple[list[Finding], str | N
 # ─────────────────────────────────────────────────────────────────────
 # Baseline allowlist: triaged known-good HIGH/CRITICAL findings so the gate
 # can enforce without red-failing on rare legitimate-library behavior.
-# Matched on ``(normalized package, basename(filename), pattern)`` -- not
+# Matched on ``(normalized package, package-relative path, pattern)`` -- not
 # evidence text -- so a version bump does not reopen a finding, but a *new*
 # kind of finding in a listed file is a different pattern and still fails.
 # Mirrors scan_packages.py. Regenerate with ``--write-baseline``.
 # ─────────────────────────────────────────────────────────────────────
 
 _DEFAULT_BASELINE_PATH = str(Path(__file__).resolve().parent / "scan_npm_packages_baseline.json")
 
+# Bumped when the entry-key semantics change. v2 keys on the package-relative
+# path; v1 stored only a basename, so a v1 entry could suppress a same-named file
+# in a different directory. A pre-v2 baseline with entries is ignored (fail
+# closed) rather than mis-applied.
+_BASELINE_SCHEMA_VERSION = 2
+
 
 def _norm_pkg_name(display: str) -> str:
     """``@scope/pkg@1.2.3`` / ``pkg@1.2.3`` -> name without the version.
@@ -1468,9 +1474,21 @@ def _norm_pkg_name(display: str) -> str:
     return s.lower()
 
 
+_NPM_TARBALL_ROOT = "package/"
+
+
+def _relpath_in_package(filename: str) -> str:
+    """Path within the published package, stable across version bumps. npm
+    tarballs root every file at ``package/``; strip it so the key is the real
+    source path (``dist/index.js``) and a new file with the same basename in a
+    different directory is not silently suppressed."""
+    f = (filename or "").replace("\\", "/")
+    return f[len(_NPM_TARBALL_ROOT) :] if f.startswith(_NPM_TARBALL_ROOT) else f
+
+
 def _finding_key(f: Finding) -> tuple[str, str, str]:
-    """Stable allowlist key: normalized package, file basename, pattern."""
-    return (_norm_pkg_name(f.package), os.path.basename(f.filename), f.pattern)
+    """Stable allowlist key: normalized package, package-relative path, pattern."""
+    return (_norm_pkg_name(f.package), _relpath_in_package(f.filename), f.pattern)
 
 
 def _load_baseline(path: str) -> set[tuple[str, str, str]]:
@@ -1483,10 +1501,18 @@ def _load_baseline(path: str) -> set[tuple[str, str, str]]:
     except (OSError, json.JSONDecodeError) as exc:
         print(f"  [WARN] could not read baseline {path}: {exc}", file = sys.stderr)
         return set()
+    entries = data.get("entries", [])
+    if entries and data.get("version") != _BASELINE_SCHEMA_VERSION:
+        print(
+            f"  [WARN] baseline schema v{data.get('version')} predates package-relative "
+            f"keys; ignoring {len(entries)} entr(y/ies). Regenerate with --write-baseline.",
+            file = sys.stderr,
+        )
+        return set()
     keys: set[tuple[str, str, str]] = set()
-    for e in data.get("entries", []):
+    for e in entries:
         try:
-            keys.add((_norm_pkg_name(e["package"]), os.path.basename(e["file"]), e["pattern"]))
+            keys.add((_norm_pkg_name(e["package"]), _relpath_in_package(e["file"]), e["pattern"]))
         except (KeyError, TypeError):
             continue
     return keys
@@ -1506,7 +1532,7 @@ def _write_baseline(path: str, findings: list[Finding], threshold_rank: int) ->
         entries.append(
             {
                 "package": _norm_pkg_name(f.package),
-                "file": os.path.basename(f.filename),
+                "file": _relpath_in_package(f.filename),
                 "pattern": f.pattern,
                 "severity": f.severity,
                 "evidence": (f.evidence or f.detail)[:240],
@@ -1516,10 +1542,10 @@ def _write_baseline(path: str, findings: list[Finding], threshold_rank: int) ->
         "_comment": (
             "scan_npm_packages.py allowlist. Each entry is a HIGH/CRITICAL "
             "finding manually judged benign. Matched on (package, "
-            "basename(file), pattern); evidence/severity are for review only. "
-            "Regenerate with --write-baseline AFTER reviewing every line."
+            "package-relative path, pattern); evidence/severity are for review "
+            "only. Regenerate with --write-baseline AFTER reviewing every line."
         ),
-        "version": 1,
+        "version": _BASELINE_SCHEMA_VERSION,
         "entries": entries,
     }
     with open(path, "w", encoding = "utf-8") as fh:

scripts/scan_npm_packages_baseline.json

@@ -1,5 +1,5 @@
 {
-  "_comment": "scan_npm_packages.py allowlist. Each entry is a HIGH/CRITICAL finding manually judged benign. Matched on (package, basename(file), pattern); evidence/severity are for review only. Regenerate with --write-baseline AFTER reviewing every line. EMPTY by design: a full scan of studio/frontend/package-lock.json (915 packages) produced 0 findings, so nothing needs suppressing and the CI gate can run enforcing (SCAN_ENFORCE=1) as-is. If a future dependency adds a reviewed-benign HIGH/CRITICAL, add it here rather than weakening a pattern.",
-  "version": 1,
+  "_comment": "scan_npm_packages.py allowlist. Each entry is a HIGH/CRITICAL finding manually judged benign. Matched on (package, package-relative path, pattern); evidence/severity are for review only. Regenerate with --write-baseline AFTER reviewing every line. EMPTY by design: a full scan of studio/frontend/package-lock.json (915 packages) produced 0 findings, so nothing needs suppressing and the CI gate can run enforcing (SCAN_ENFORCE=1) as-is. If a future dependency adds a reviewed-benign HIGH/CRITICAL, add it here rather than weakening a pattern.",
+  "version": 2,
   "entries": []
 }