Skip to content

fix(core): skip oversized ripgrep match lines instead of aborting grep#35699

Open
C0d3N1nja97342 wants to merge 1 commit into
anomalyco:devfrom
C0d3N1nja97342:fix/grep-skip-oversized-lines-dev
Open

fix(core): skip oversized ripgrep match lines instead of aborting grep#35699
C0d3N1nja97342 wants to merge 1 commit into
anomalyco:devfrom
C0d3N1nja97342:fix/grep-skip-oversized-lines-dev

Conversation

@C0d3N1nja97342

Copy link
Copy Markdown

Issue for this PR

Closes #35523

Type of change

  • Bug fix

What does this PR do?

grep aborts the entire search with Ripgrep JSON record exceeded 65536 bytes whenever a single match sits on a line longer than ~64 KiB (minified bundles, base64, large data rows) — every other match in the same call is thrown away.

The guard is opencode's own, in packages/core/src/ripgrep.ts (the parse callback inside run, grep branch):

Buffer.byteLength(line, "utf8") > MAX_RECORD_BYTES  // 64 * 1024
  ? Effect.fail(failure(`Ripgrep JSON record exceeded ${MAX_RECORD_BYTES} bytes`))
  : JSON.parse(line)

ripgrep itself exits 0 and emits the record fine — the failure is purely opencode rejecting its own output. I confirmed with real rg 15.1.0 that --max-columns does not help here: in --json mode it leaves lines.text at the full length, and --max-columns-long drops the match entirely. So the fix belongs in the parse guard, not the rg args.

Change: when a record exceeds MAX_RECORD_BYTES, return undefined (skipped by the existing Stream.filter just downstream) instead of failing. The remaining matches are still returned. Truly invalid JSON at normal length still fails with the same error as before.

Why this is the right trade: the display path already truncates lines.text to 2000 chars (in Match.make further down in grep), so the full content of an oversized line was never surfaced to the model/user anyway. Skipping one unusable line is strictly better than discarding the whole result set.

(Note: a previous version of this PR targeted v2; reopened against dev, which is where 1.17.x ships and where GitHub links the issue correctly.)

How did you verify your code works?

Ran in an oven/bun:1.3.14 container with git installed (the existing "keeps ignored files out of catch-all find results" test shells out to git init), against the dev branch tip:

cd packages/core
bun test test/ripgrep.test.ts
bun run typecheck
  • Added a regression test skips oversized match lines instead of failing the whole grep: writes big.txt (a 70000-byte line containing the pattern) and small.txt (a normal line), greps the pattern, asserts small.txt is returned, big.txt is skipped, and the call does not throw.
  • Confirmed the regression test fails with the original Effect.fail and passes with the fix.
  • All 3 tests in ripgrep.test.ts pass.
  • bun run typecheck (tsgo --noEmit) in packages/core is clean.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

A match whose JSON record exceeds the 64 KiB parse guard (a line longer
than ~65536 bytes — minified bundles, base64, data rows) used to fail the
entire grep with "Ripgrep JSON record exceeded N bytes", discarding every
other match. rg itself exits 0 and emits the record fine; the guard was
opencode's own.

Skip the oversized record (return undefined, filtered out of the stream)
instead of failing, so the remaining matches are still returned. The
display layer already truncates lines.text to 2000 chars, so the full
oversized content was never surfaced anyway. Truly invalid JSON still
fails.

Closes anomalyco#35523
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Grep tool fails with "Ripgrep JSON record exceeded 65536 bytes" on files with long matching lines

1 participant