perf(weekly-report): Batch key errors Snuba query#117329
Conversation
Arbitrarily picked 100 projects --> can increase/decrease if necessary |
843a1e3 to
534083c
Compare
| FEEDBACKS_LABEL_QUERY = "feedbacks.label_query" | ||
| EU_DATA_EXPORT = "sentry.internal.eu-compliance-data-export" | ||
| REPORTS_KEY_ERRORS = "reports.key_errors" | ||
| REPORTS_KEY_ERRORS_BATCHED = "reports.key_errors.batched" |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
shashjar
left a comment
There was a problem hiding this comment.
query impl itself looks good to me, we can simplify by removing the EAP query path
| snuba_results = _org_key_errors_snuba(ctx=ctx, project_ids=project_ids, referrer=referrer) | ||
|
|
||
| callsite = "tasks.summaries.project_key_errors" | ||
| if EAPOccurrencesComparator.should_check_experiment(callsite): |
There was a problem hiding this comment.
can we implement this as just a Snuba query to start? we can add the parallel EAP query later on
| query_result = project_key_transactions_last_week( | ||
| ctx, project, key_transactions_this_week | ||
| ) | ||
| # Join this week with last week |
There was a problem hiding this comment.
seems like a useful comment to leave
|
|
||
| def _org_key_errors_snuba( | ||
| ctx: OrganizationReportContext, | ||
| project_ids: list[int], |
There was a problem hiding this comment.
nit: for typing, we typically type parameters as Sequence, etc. to be more general, and return the more specific list type
| _KEY_ERRORS_CHUNK_SIZE = 100 | ||
|
|
||
|
|
||
| def _org_key_errors_snuba( |
There was a problem hiding this comment.
i notice that we have eap versions of the 'project key' variant of this function, do we need that here too?
There was a problem hiding this comment.
for now, we won't add the EAP versions. we'll add the parallel eap version in another PR (if needed)
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4f0be31. Configure here.

Resolves ID-1618
Summary
organizations:weekly-report-batched-key-errorsfeature flag with fallback to existing per-project pathContext
The
key_errorsportion of weekly report generation is the slowest step. For each project in an org, it fires a separateevents ⨝ group_attributesJOIN query to Snuba. For an org with 100 projects, that's 100 sequential round-trips.This PR introduces a batched alternative: one query per 100 projects using
project_id IN [...]andGROUP BY (project_id, group_id), then extracts top 3 per project in Python. For a 100-project org, this is 1 query instead of 100.Test plan
test_org_key_errors_batched,test_message_builder_filter_resolved_batched)