feat(skill): Sigma.js viz fallback + batched community-label reconciliation

[docwilde]

Summary

Adds a Sigma.js + graphology (WebGL) fallback for graph.html once a view exceeds ~300 nodes, plus a batched-subagent flow for community labeling at scale, both validated against a real production run (apa, a 20,097-node/1083-community monorepo) — then substantially hardened across six follow-up review rounds.

Sigma.js viz (graph_sigma.html):

• Layout precomputed offline in Python (no client-side physics), using linlog=True + a node_size repulsion halo + a high scaling_ratio, tuned across three passes against real production output rather than a numeric proxy alone.
• Icon per community's dominant content kind (@sigma/node-image's NodePictogramProgram), color per dominant module, node labels on their own background box (no sigma built-in for this; used the documented defaultDrawNodeLabel override point), edges colored + labeled by their dominant relation bucket.
• Left-side filter panel (entity kind + relation type, grouped so non-code relations like cites/rationale_for are represented too), a clickable module legend, and draggable nodes for manual layout tidying.
• Click panel lists every one of a community's source files (filterable, scrollable, uncapped), and clicking a file opens a movable dialog with an embedded content preview (bounded to the first 8 files per community, since full content for every file would bloat the self-contained HTML far more than paths alone do), with a file:// link when opened on the machine that generated the graph.
• Module clustering: a separate layout-only edge weight biases same-module edges before running forceAtlas2, so modules cluster spatially instead of scattering by topology alone. A translucent convex-hull region is drawn behind each module's cluster (pure-Python hull, no scipy dependency), redrawn in sync with the graph so it tracks pan/zoom without visibly desyncing from the nodes. Edges below a weight percentile are hidden by default, with a "show all edges" opt-in checkbox, so a well-separated layout also reads as structured rather than just less bunched.
• The browser window can be resized after the page has already loaded — including immediately after load, or rapidly (a real drag-resize) — and the graph reliably re-fits to the new viewport with no dead space and no stale camera framing.
• The header shows the repo name, scan-root path, and git branch a given graph_sigma.html was generated from, so one open copy can be told apart from another (e.g. comparing branches, or a stale copy left over from a previous run).

Community labeling (Step 5): a batched-subagent flow for 100+ communities (sample top-degree nodes, dispatch one subagent per ~15-20-community batch in parallel) replacing one-at-a-time labeling and a naive path-prefix heuristic, plus a required cross-batch label-reconciliation pass — batches only see their own slice, so duplicate labels across siblings are the common case at scale (31% of communities collided on the same production run), not an edge case.

Why: Sigma.js instead of vis-network past ~300 nodes

vis-network runs a live, single-threaded JS forceAtlas2 physics simulation on load; past a few hundred nodes this is genuinely slow/laggy regardless of hardware. The fix is removing client-side physics entirely: precompute the layout once, offline, and render only via WebGL. Ships as a separate output file, not a replacement — vis-network is fine below the threshold and has features (a community filter dropdown, confidence-styled edges) this lighter template doesn't replicate.

Review history

1. Initial feature, landed via two parallel review agents (code review + sigma.js API fact-check) plus execution of the Python precompute step against real graph.json samples.
2. A black-screen regression, found only by actually opening a generated file in a real browser: autoRescale: false (added for a sizing fix) silently broke camera framing, and a separate esm.sh peer-dependency resolution issue broke the icon/utils imports entirely before any script code ran. An independent second-opinion review caught a third latent bug (icon SVG sizing) before it shipped.
3. A usability pass after actually using the rendered graph surfaced: layout still too bunched (re-tuned a second time), a near-invisible label background color, file previews needing a script-tag auto-escape (a previewed HTML file's own <script> tag collided with the page's), and a floating info panel getting clipped independently of the main sidebar.
4. A structure pass: a well-separated layout with every edge at full opacity still read as a hairball, so this added module-clustering (layout-only edge weight boost), translucent module hulls, and default edge decluttering by weight. Also surfaced that nothing had ever called the filter/highlight/decluttering reducer function at startup — harmless before since its default state was a no-op, not harmless once edge decluttering became a real default-hiding behavior. The sidebar panel also had a viewport-anchoring bug (position: absolute drifting out of sync with the visible viewport on any page scroll) fixed to position: fixed with height instead of max-height.
5. A resize regression, fixed twice before it stuck: growing the browser window after load left the graph pinned at its original size with dead space around it, since sigma's own resize listener never re-fits the camera. The first fix attempt (fitViewportToNodes called synchronously in the resize handler) was worse than the bug — it read node data mid-flight through sigma's own pending re-normalization and rendered a blank canvas. The second attempt deferred that call by one requestAnimationFrame, which fixed the common case but still assumed frame ordering rather than verifying it — it raced and failed on an immediate post-load resize or a rapid multi-step resize.
6. The actual fix: tie the re-fit (and the hull redraw, which had the same class of bug — briefly showing the new camera state a frame before the WebGL canvases caught up) to sigma's own afterRender event instead of a guessed frame count, since that event only fires once a render pass has genuinely finished. This round also re-tuned the module-clustering boost after a report that the two largest modules were still visibly clumping into their own sub-hairball — a too-high boost was found to actively compound on modules with more same-module edges rather than plateauing harmlessly — and added the repo/branch header line.

Scope note

Only the shared core.md fragment (covering all 14 split-platform skills, rendered via tools/skillgen) was updated. The aider and devin monolith skill bodies maintain their own independent Step 5/6 sections and were intentionally left untouched.

Test plan

• uv run python -m tools.skillgen (+ --check / --audit-coverage / --schema-singleton / --monolith-roundtrip / --always-on-roundtrip) — all pass
• uv run python -m pytest -q — full suite green (2962 passed, 3 skipped, with [all] optional extras installed)
• Step 1's Python precompute block executed against real graph.json samples, including forced edge cases (empty-communities fallback, edges-vs-links key, synthetic hyperedges)
• The full generated graph_sigma.html opened in a real browser (Chrome via Playwright, served over local HTTP) against a synthetic multi-type example and the real 20,097-node/1083-community production graph — zero console errors, correct rendering, icons/colors/labels/filter panel/legend/click-highlight/dragging/file dialog/module hulls/edge decluttering/repo-branch header all confirmed working
• Resize robustness verified against the real production graph: load-then-resize-immediately in both directions (900x700↔1800x1100), and a simulated rapid multi-step drag-resize — all settle with the graph filling the viewport, hulls staying aligned with nodes throughout, no dead space, no blank canvas
• Layout re-tune verified both numerically (near-center node fraction 0.28 → 0.12 on the real graph) and visually, per the lesson from round 3 that the numeric proxy alone previously under-predicted "still too bunched"

[docwilde]

Pushed a follow-up commit addressing review feedback:

• Fixed the node-sizing bug: autoRescale: false is required alongside itemSizesReference: "positions", per sigma's own fit-sizes-to-positions example — otherwise the one-time auto-fit repositions the graph without symmetrically rescaling sizes. Node size is now also calibrated against the layout's own median nearest-neighbor distance instead of a fixed constant, so communities stay legible instead of overlapping into a blob as node count grows.
• Added a left-side filter panel for entity kind (code/document/paper/image/rationale/concept, shown as icons via @sigma/node-image's NodePictogramProgram) and relation type, grouped into calls/structure/imports/references/documentation-and-concepts/groups so non-code relations (cites, rationale_for, semantically_similar_to, etc.) are represented, not just calls/imports.
• Added module/top-level-directory color-coding with a clickable legend that doubles as an isolate-by-module toggle — a lightweight grouping tool without introducing a new library.
• Fixed several bugs an independent code review caught: a crash when no community meets MIN_COMMUNITY_SIZE, a missing links/edges key compatibility guard (present elsewhere in the codebase but missing here), hyperedges never being processed (so the "Groups" filter bucket was permanently empty), an unescaped innerHTML injection of LLM-generated community labels and directory names, a case-sensitive </script> safety check, search results ignoring active filters, and a camera-pan target using raw graph coordinates instead of sigma's own display-space coordinates (confirmed against sigma's demo SearchField.tsx).

Also merged in the latest v8 to pick up the 0.9.7 release.

[docwilde]

Critical fix, found by actually opening the generated HTML in a real browser — the previous fix reported here made the graph render as a solid black screen with nothing visible. Root causes:

1. autoRescale: false (added for the sizing fix) also disables sigma's automatic camera-fit-to-graph on load. Fixed with fitViewportToNodes from @sigma/utils, called on init and in the reset handler (replacing camera.animatedReset(), which had the identical bug).
2. More fundamentally: the @sigma/node-image/@sigma/utils ESM imports were throwing at module-load time — esm.sh resolved their sigma peer dependency to a broken literal path instead of the pinned version, so the import never completed and the entire script died before anything else ran. Fixed by pinning explicitly via ?deps=sigma@3.0.3 on both imports.
3. An independent second-opinion review (a separate agent instance, told to be skeptical of every camera/coordinate claim) caught a third bug before shipping: icon SVGs need explicit width/height for @sigma/node-image's raster path to size them correctly.
4. maxCameraRatio raised 3 → 10 for headroom on narrow/short viewports, per the same review.

Verified end-to-end in Chrome (via Playwright) against both a small synthetic multi-type example and the real 20,097-node/1083-community production graph (apa) that motivated this feature — zero console errors, correct rendering, icons/module-colors/filter panel/legend all working.

[docwilde]

Second follow-up round, driven by direct feedback after actually using the rendered graph:

• Layout re-tuned again: the first spread fix (gravity=0.5, scaling_ratio=20) measured better on a numeric proxy but was still reported as too bunched to read labels without manually dragging nodes apart. gravity=0.15, scaling_ratio=80 actually fixes it — verified by reading real labels at default zoom on the 342-community production graph, not by trusting the proxy metric a second time.
• Node labels get a background box — sigma's default label renderer is bare fillText with no background option; defaultDrawNodeLabel is the documented override point for a custom renderer. First attempt used a near-black fill indistinguishable from the canvas's own background color, fixed to match the UI panel's tone.
• Edges colored + labeled by their dominant relation bucket, draggable nodes (sigma's downNode/moveBody/upNode/upStage pattern, with the setCustomBBox freeze required to stop one dragged node from panning every other one), an uncapped filterable/scrollable file list (no more "+N more"), and clicking a file now opens a movable dialog with an embedded content preview (bounded to the first 8 files per community — full content for every file would bloat the self-contained HTML far more than paths alone do).
• Embedding real file content surfaced a new instance of the script-tag-collision risk: a previewed HTML/JS file routinely contains a literal </script (confirmed on the real corpus — a Vite index.html's <script type="module"> tag). This used to be a check-after-the-fact step; Step 3 now escapes it unconditionally before embedding.
• The entity-properties panel, previously a separate floating box that could get clipped by the viewport independently of the main panel, is now nested inside the main sidebar — same width, same scroll.

A parallel Sonnet review pass over the accumulated diff found no functional bugs, just two cosmetic nitpicks (an unused data attribute, a stray character in placeholder text), both fixed. Verified end-to-end in a real browser (Chrome via Playwright): the label-background fix, merged panel layout, scrollable/filterable file list, and draggable preview dialog were all confirmed on a synthetic example (switched to exact-coordinate clicks via getNodeDisplayData after eyeballed pixel coordinates kept missing targets); the layout re-tune and script-tag auto-escape were confirmed on the real 20,097-node/1083-community production graph.

[docwilde]

Third follow-up round: "more structure" + sidebar fixes.

• Module clustering: same-module edges get a boosted layout-only weight (a separate layout_weight attribute, not the real weight that drives rendered thickness) before forceAtlas2 runs, so modules cluster spatially instead of scattering by topology alone. Verified via the ratio of within-module to between-module node distance on the real production graph: 0.96 (no boost) → 0.48 at the chosen factor, with a higher factor not improving further.
• Module hulls: a soft translucent region drawn behind each module's cluster, computed with a small pure-Python convex-hull implementation (Andrew's monotone chain — not worth a scipy dependency for one shape per module) and rendered on a 2D canvas layer behind sigma's own WebGL canvas, redrawn on every afterRender event via renderer.graphToViewport so it tracks pan/zoom.
• Edge decluttering: edges below a weight percentile are hidden by default (a "show all edges" checkbox opts back in; anything touching the highlighted node always shows regardless of weight). This surfaced a real gap: nothing ever called applyReducers() at startup, so the filter/highlight reducers had been silently inert until the first user interaction — harmless before (a no-op identity function by default), not harmless once hiding-by-weight became a real default-view behavior.
• Sidebar bug: position: absolute let the panel drift out of sync with the actual viewport whenever the page gained any scroll, which read as "the panel is truncated on my laptop" on a short screen — fixed to position: fixed, plus a visible custom scrollbar (macOS hides scrollbars until actively scrolling, making an already-working scroll region look dead) and height instead of max-height so the sidebar always spans the full viewport.

Verified end-to-end on the real 20,097-node/1083-community production graph: 12 module hulls render correctly and track pan/zoom, the show-all-edges toggle visibly changes edge density in both directions, module-isolate correctly gates its hull via the same activeModules check nodes/edges already use, and the sidebar renders as a full-height, properly-scrolling panel at both a short laptop viewport (1440×760) and a large one (2560×1440).