Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(job-runner): track job status #6332

Merged
merged 17 commits into from
Oct 1, 2024
Merged

feat(job-runner): track job status #6332

merged 17 commits into from
Oct 1, 2024

Conversation

xurui-c
Copy link
Member

@xurui-c xurui-c commented Sep 20, 2024

Track job status to prevent double executions

Making a few changes here:

  1. Make JobSpec the main way to call run_job and run_job the main interface for executing manual jobs
  2. Introduce a new Redis cluster to store job status
  3. for run_job, acquire a lock via a redis key or abort the attempted execution
  4. for run_job, also abort execution if the job_id contains a non-null/"not started" status
  5. new command to fetch job status given a job ID

JIRA: https://getsentry.atlassian.net/browse/SNS-2869

@xurui-c xurui-c changed the title c feat(job-runner): Implement a status subcommand for snuba jobs Sep 20, 2024
@xurui-c xurui-c changed the title feat(job-runner): Implement a status subcommand for snuba jobs feat(job-runner): track job status Sep 20, 2024
tests/jobs/test_job.py Outdated Show resolved Hide resolved
Copy link

codecov bot commented Sep 20, 2024

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
580 1 579 1
View the top 1 failed tests by shortest run time
tests.cli.test_jobs test_cmd_line_valid
Stack Traces | 0s run time
No failure message available

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

@onewland onewland marked this pull request as ready for review September 27, 2024 21:08
@onewland onewland requested a review from a team as a code owner September 27, 2024 21:08
snuba/manual_jobs/runner.py Outdated Show resolved Hide resolved
snuba/manual_jobs/runner.py Outdated Show resolved Hide resolved
snuba/manual_jobs/runner.py Outdated Show resolved Hide resolved
snuba/manual_jobs/runner.py Show resolved Hide resolved
snuba/manual_jobs/runner.py Outdated Show resolved Hide resolved
snuba/manual_jobs/runner.py Show resolved Hide resolved
@onewland onewland merged commit a94f00f into master Oct 1, 2024
30 checks passed
@onewland onewland deleted the rachel/status branch October 1, 2024 17:53
@getsentry-bot
Copy link
Contributor

PR reverted: bb814d3

getsentry-bot added a commit that referenced this pull request Oct 1, 2024
onewland added a commit that referenced this pull request Oct 2, 2024
onewland added a commit that referenced this pull request Oct 2, 2024
This is #6332 unchanged, but now Redis clusters have been defined in all
SaaS environments (getsentry/ops#12293 deployed
everywhere)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants