Skip to content

perf(weekly-report): Batch key errors Snuba query#117329

Merged
amy-chen23 merged 9 commits into
masterfrom
amyc/refactor-weekly-report-snuba
Jun 12, 2026
Merged

perf(weekly-report): Batch key errors Snuba query#117329
amy-chen23 merged 9 commits into
masterfrom
amyc/refactor-weekly-report-snuba

Conversation

@amy-chen23

Copy link
Copy Markdown
Contributor

Resolves ID-1618

Summary

  • Replace N per-project Snuba JOIN queries for key errors with a single batched org-wide query, reducing Snuba round-trips from N to ceil(N/100)
  • Gate behind organizations:weekly-report-batched-key-errors feature flag with fallback to existing per-project path
  • Chunk project IDs into batches of 100 to prevent query timeouts for large orgs

Context

The key_errors portion of weekly report generation is the slowest step. For each project in an org, it fires a separate events ⨝ group_attributes JOIN query to Snuba. For an org with 100 projects, that's 100 sequential round-trips.

This PR introduces a batched alternative: one query per 100 projects using project_id IN [...] and GROUP BY (project_id, group_id), then extracts top 3 per project in Python. For a 100-project org, this is 1 query instead of 100.

Test plan

  • All 35 existing tests pass with flag off (old path)
  • 2 new tests exercise the batched path (test_org_key_errors_batched, test_message_builder_filter_resolved_batched)
  • ruff, mypy clean

@linear-code

linear-code Bot commented Jun 10, 2026

Copy link
Copy Markdown

ID-1618

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Jun 10, 2026
@amy-chen23

Copy link
Copy Markdown
Contributor Author
  • Chunk project IDs into batches of 100 to prevent query timeouts for large orgs

Arbitrarily picked 100 projects --> can increase/decrease if necessary

Comment thread src/sentry/tasks/summaries/utils.py Outdated
@amy-chen23 amy-chen23 force-pushed the amyc/refactor-weekly-report-snuba branch from 843a1e3 to 534083c Compare June 11, 2026 16:49
FEEDBACKS_LABEL_QUERY = "feedbacks.label_query"
EU_DATA_EXPORT = "sentry.internal.eu-compliance-data-export"
REPORTS_KEY_ERRORS = "reports.key_errors"
REPORTS_KEY_ERRORS_BATCHED = "reports.key_errors.batched"

This comment was marked as resolved.

@amy-chen23 amy-chen23 marked this pull request as ready for review June 11, 2026 18:57
@amy-chen23 amy-chen23 requested review from a team as code owners June 11, 2026 18:57
@amy-chen23 amy-chen23 requested a review from shashjar June 11, 2026 18:58
Comment thread src/sentry/tasks/summaries/utils.py Outdated
Comment thread src/sentry/tasks/summaries/utils.py

@shashjar shashjar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

query impl itself looks good to me, we can simplify by removing the EAP query path

Comment thread src/sentry/tasks/summaries/utils.py Outdated
snuba_results = _org_key_errors_snuba(ctx=ctx, project_ids=project_ids, referrer=referrer)

callsite = "tasks.summaries.project_key_errors"
if EAPOccurrencesComparator.should_check_experiment(callsite):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we implement this as just a Snuba query to start? we can add the parallel EAP query later on

query_result = project_key_transactions_last_week(
ctx, project, key_transactions_this_week
)
# Join this week with last week

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like a useful comment to leave

Comment thread src/sentry/tasks/summaries/utils.py Outdated

def _org_key_errors_snuba(
ctx: OrganizationReportContext,
project_ids: list[int],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: for typing, we typically type parameters as Sequence, etc. to be more general, and return the more specific list type

_KEY_ERRORS_CHUNK_SIZE = 100


def _org_key_errors_snuba(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i notice that we have eap versions of the 'project key' variant of this function, do we need that here too?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for now, we won't add the EAP versions. we'll add the parallel eap version in another PR (if needed)

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4f0be31. Configure here.

Comment thread src/sentry/tasks/summaries/utils.py Outdated
@amy-chen23 amy-chen23 merged commit 1669e4d into master Jun 12, 2026
86 checks passed
@amy-chen23 amy-chen23 deleted the amyc/refactor-weekly-report-snuba branch June 12, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants