feat(clock-pulse): rollout TaskProducer as checkin_producer#117107
Merged
Conversation
github-actions
Bot
added
the
Scope: Backend
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9786a7997f1415c0fc2e749df98b79a7c435d7ca. Configure here.
| if in_random_rollout("tasks.producer.clock-pulse.rollout"): | ||
| _checkin_task_producer.produce(dest, payload) | ||
| else: | ||
| _checkin_producer.produce(dest, payload) |
Contributor
There was a problem hiding this comment.
Rollout evaluated per partition
Medium Severity
in_random_rollout runs inside the partition loop, so one clock_pulse invocation can send some pulses via _checkin_task_producer and others via _checkin_producer. Task completion then tracks only the TaskProducer futures, while other pulses may still be in flight on the singleton path.
Reviewed by Cursor Bugbot for commit 9786a7997f1415c0fc2e749df98b79a7c435d7ca. Configure here.
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
bmckerry
force-pushed
the
ben/clock-pulse-taskproducer
branch
from
9786a79 to
8b5b5d4
Compare
Member
Author
|
Converting to draft, needs a taskbroker version bump to work properly |
untitaker
reviewed
Jun 9, 2026
| _occurrence_task_producer = get_task_producer( | ||
| producer_name="sentry.issues.tasks.producer", | ||
| producer_factory=partial(_get_occurrence_producer, name="sentry.issues.tasks.producer"), | ||
| metrics_backend=SentryMetricsBackend(), |
Member
There was a problem hiding this comment.
was removing this intentional?
Member
Author
There was a problem hiding this comment.
yes, get_task_producer() now handles instantiating the metrics backend: https://github.com/getsentry/sentry/pull/117107/changes#diff-c6948a9b5ad1d35d6c2187062eb4063c51988ac1159f17f1c16ea420ee5773beR19
untitaker
approved these changes
Jun 9, 2026
wedamija
approved these changes
Jun 9, 2026
bmckerry
added a commit
that referenced
this pull request
Jun 10, 2026
…er (#117360) Followup to #117107 This PR updates the snapshot producer used by several tasks to use the new `TaskProducer`. The reason for this change is that SingletonProducer doesn't guarantee at-least-once delivery when used inside a task. TaskProducer keeps track of any producer futures generated as a part of task execution, and only marks a task as successful if all producer futures succeed. To ensure a safe rollout, workers will instantiate an instance of the old and new producer, and cut over to the new producer based on a rollout option. This means that task futures won't be properly tracked while the rollout option is <1.0, but since our base state is to just ignore producer futures that's acceptable.
3 tasks
bmckerry
added a commit
that referenced
this pull request
Jun 16, 2026
`KafkaPublisher` was a deprecated Kafka producer API which was only still used in replay deletion tasks and replay linking in postprocess. This PR moves all instances where `KafkaPublisher` was used over to the existing `sentry.replays.lib.kafka.publish_replay_event` function, which is backed by `SingletonProducer`. It also fixes corresponding tests, and completely deletes `KafkaPublisher` from the codebase. This is a step towards migrating replay-related tasks to use taskbroker-client's `TaskProducer` abstraction (similar to what I've done in #117081, #117107, #117360). I'll open up another PR stacked on top of this one that adds the rollout code for `TaskProducer`.
sehr-m
pushed a commit
that referenced
this pull request
Jun 23, 2026
`KafkaPublisher` was a deprecated Kafka producer API which was only still used in replay deletion tasks and replay linking in postprocess. This PR moves all instances where `KafkaPublisher` was used over to the existing `sentry.replays.lib.kafka.publish_replay_event` function, which is backed by `SingletonProducer`. It also fixes corresponding tests, and completely deletes `KafkaPublisher` from the codebase. This is a step towards migrating replay-related tasks to use taskbroker-client's `TaskProducer` abstraction (similar to what I've done in #117081, #117107, #117360). I'll open up another PR stacked on top of this one that adds the rollout code for `TaskProducer`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Followup to #116788
This PR updates the checkin producer used by the
clock_pulsetask to use the newTaskProducer.The reason for this change is that SingletonProducer doesn't guarantee at-least-once delivery when used inside a task. TaskProducer keeps track of any producer futures generated as a part of task execution, and only marks a task as successful if all producer futures succeed.
To ensure a safe rollout, workers will instantiate an instance of the old and new producer, and cut over to the new producer based on a rollout option. This means that task futures won't be properly tracked while the rollout option is <1.0, but since our base state is to just ignore producer futures that's acceptable.
This PR also adds a
get_task_producer()helper function that fills in theTaskProducermetrics backend automatically.