Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow GoodJob::Job#retry_job's transaction to be joinable (fix CI for current rails-head) #1496

Merged
merged 2 commits into from
Oct 7, 2024

Conversation

Earlopain
Copy link
Collaborator

It is now >= 3.2 only

It is now >= 3.2 only
@Earlopain
Copy link
Collaborator Author

Earlopain commented Oct 2, 2024

Locks like the failures are legit, I can reliably reproduce with rails-head locally.

Failures:

  1) GoodJob::Job#retry_job when job is retried updates the original job
     Failure/Error: expect(POSTGRES_NOTICES).to be_empty
       expected `["Connection returned to pool with 1 advisory locks", "Connection returned to pool with 1 advisory locks"].empty?` to be truthy, got false
     # ./spec/support/postgres_notices.rb:34:in `block (2 levels) in <main>'
     # ./spec/support/logger.rb:7:in `block (2 levels) in <main>'
     # ./spec/support/database_cleaner.rb:17:in `block (2 levels) in <main>'

  2) GoodJob::Job#retry_job when job is retried when run inline executes the job
     Failure/Error: expect(POSTGRES_NOTICES).to be_empty
       expected `["Connection returned to pool with 1 advisory locks", "Connection returned to pool with 1 advisory locks"].empty?` to be truthy, got false
     # ./spec/support/postgres_notices.rb:34:in `block (2 levels) in <main>'
     # ./spec/support/logger.rb:7:in `block (2 levels) in <main>'
     # ./spec/support/database_cleaner.rb:17:in `block (2 levels) in <main>'

  3) GoodJob::Job#retry_job when job arguments cannot be deserialized ignores the error and discards the job
     Failure/Error: expect(POSTGRES_NOTICES).to be_empty
       expected `["Connection returned to pool with 1 advisory locks", "Connection returned to pool with 1 advisory locks"].empty?` to be truthy, got false
     # ./spec/support/postgres_notices.rb:34:in `block (2 levels) in <main>'
     # ./spec/support/logger.rb:7:in `block (2 levels) in <main>'
     # ./spec/support/database_cleaner.rb:17:in `block (2 levels) in <main>'

Finished in 2 minutes 15.5 seconds (files took 1.81 seconds to load)
567 examples, 3 failures

Failed examples:

rspec ./spec/app/models/good_job/job_spec.rb:135 # GoodJob::Job#retry_job when job is retried updates the original job
rspec ./spec/app/models/good_job/job_spec.rb:159 # GoodJob::Job#retry_job when job is retried when run inline executes the job
rspec ./spec/app/models/good_job/job_spec.rb:210 # GoodJob::Job#retry_job when job arguments cannot be deserialized ignores the error and discards the job

Will spend some time bisecting later

@Earlopain
Copy link
Collaborator Author

Caused by rails/rails#53118. There's rails/rails#53147 (which probably is something different, since the PR that caused this was a proposed fix for that kind of problem). Will try to reduce into something reportable

@bensheldon
Copy link
Owner

bensheldon commented Oct 2, 2024

@Earlopain thank you so much for looking into this!

It is interesting that it looks like it's just GoodJob::Job#retry_job that has problems. I could imagine that GoodJob itself is doing something unexpected (or wrong!) because GoodJob jumps through a lot of hoops in #retry_job to simulate an Active Job retry.

From reading the Rails change, it looks like it heavily enforces "if there is not an open transaction, the current database connection is checked in" when a Rails executor completes. This should not a problem because nested Rails Executors should be noops, and it's only the outermost executor that should run the callbacks (I'm 90% sure of this; if I am wrong that's a problem!) and by the outermost executor GoodJob should have released all it's advisory locks.

@Earlopain
Copy link
Collaborator Author

I believe your assumption is correct. Here are my findings:

Rails own executor_around_test_case config is documented as using wrap but uses perform instead. I think that's just a docs error though, the caveat from rails/rails#43550 is still there. I'll just open a PR for that to fix.

Does is seem reasonable for you to switch to wrap instead of perform? Seems a bit weird to me and I don't understand it enough anyways. Maybe its just an order-of-execution thing since the rails PR did a new hook. But even if, I wouldn't know how to fix. It would be nice if you could take this over when you got some time.

@bensheldon
Copy link
Owner

bensheldon commented Oct 2, 2024

hmmmmm.

By the time any test gets to good_job/spec/support/rails_executor.rb, there should not be any Advisory Locks so I believe that executor#perform is the correct one to use here. If there are still advisory locks remaining, that seems like GoodJob (or the test) has a problem.

It does seem interesting that it is just those tests, which makes me suspect that maybe this is GoodJob's problem.

I would be curious about the callers stacktrace when the connection is checked back in:

warning = "Connection returned to pool with #{count} advisory locks"

...cause I am unsure whether the Notice is being triggered at the very end of the test run (when I do expect all advisory locks to be released), or somewhere in the middle of the test run (when there would be open advisory locks and lots of Rails Executor boundaries being being cross that I believe should be noops).

Also, reading the the test file, this is sorta curious. Now why did I do that?!:

around do |example|
Rails.application.executor.wrap { example.run }
end

@Earlopain
Copy link
Collaborator Author

Here are caller locations:

1

["/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:428:in `instance_exec'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:428:in `block in make_lambda'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:178:in `block in call'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:667:in `block (2 levels) in default_terminator'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:666:in `catch'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:666:in `block in default_terminator'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:179:in `call'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `block in invoke_before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `each'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `invoke_before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:108:in `run_callbacks'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:912:in `_run_checkin_callbacks'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:579:in `block (2 levels) in checkin'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/3.3.0/monitor.rb:201:in `synchronize'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/3.3.0/monitor.rb:201:in `mon_synchronize'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:576:in `block in checkin'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:575:in `checkin'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:392:in `release_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:197:in `block in complete'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_handler.rb:107:in `block (2 levels) in each_connection_pool'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/pool_manager.rb:31:in `each_value'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/pool_manager.rb:31:in `block in each_pool_config'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/pool_manager.rb:30:in `each_value'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/pool_manager.rb:30:in `each_pool_config'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_handler.rb:106:in `block in each_connection_pool'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/map.rb:265:in `block in each_value'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/collection/map/non_concurrent_map_backend.rb:101:in `block in each_pair'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/collection/map/non_concurrent_map_backend.rb:100:in `each_pair'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/collection/map/non_concurrent_map_backend.rb:100:in `each_pair'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/map.rb:276:in `each_pair'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/concurrent-ruby-1.3.4/lib/concurrent-ruby/concurrent/map.rb:265:in `each_value'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_handler.rb:105:in `each_connection_pool'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:193:in `complete'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/execution_wrapper.rb:36:in `before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:384:in `block in make_lambda'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:178:in `block in call'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:667:in `block (2 levels) in default_terminator'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:666:in `catch'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:666:in `block in default_terminator'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:179:in `call'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `block in invoke_before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `each'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `invoke_before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:108:in `run_callbacks'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/execution_wrapper.rb:142:in `complete'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/execution_wrapper.rb:136:in `complete!'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/execution_wrapper.rb:96:in `wrap'",
 "/home/earlopain/Documents/good_job/lib/good_job/adapter.rb:140:in `enqueue_at'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/enqueuing.rb:131:in `raw_enqueue'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/enqueue_after_transaction_commit.rb:24:in `raw_enqueue'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/enqueuing.rb:118:in `block in enqueue'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:120:in `block in run_callbacks'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/instrumentation.rb:40:in `block in instrument'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/notifications.rb:210:in `block in instrument'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/notifications/instrumenter.rb:58:in `instrument'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/notifications.rb:210:in `instrument'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/instrumentation.rb:39:in `instrument'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/railties/job_runtime.rb:18:in `instrument'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/instrumentation.rb:21:in `block (2 levels) in <module:Instrumentation>'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:129:in `instance_exec'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:129:in `block in run_callbacks'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/tagged_logging.rb:143:in `block in tagged'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/tagged_logging.rb:38:in `tagged'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/tagged_logging.rb:143:in `tagged'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/broadcast_logger.rb:241:in `method_missing'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/logging.rb:39:in `tag_logger'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/logging.rb:28:in `block (2 levels) in <module:Logging>'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:129:in `instance_exec'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:129:in `block in run_callbacks'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:140:in `run_callbacks'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/enqueuing.rb:117:in `enqueue'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/exceptions.rb:153:in `block in retry_job'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/instrumentation.rb:40:in `block in instrument'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/notifications.rb:210:in `block in instrument'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/notifications/instrumenter.rb:58:in `instrument'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/notifications.rb:210:in `instrument'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/instrumentation.rb:39:in `instrument'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/railties/job_runtime.rb:18:in `instrument'",
 "/home/earlopain/Documents/rails/activejob/lib/active_job/exceptions.rb:152:in `retry_job'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:483:in `block (3 levels) in retry_job'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/transaction.rb:626:in `block in within_new_transaction'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/transaction.rb:623:in `within_new_transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:367:in `within_new_transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:359:in `transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/transactions.rb:234:in `block in transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:414:in `with_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_handling.rb:310:in `with_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/transactions.rb:233:in `transaction'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:482:in `block (2 levels) in retry_job'",
 "/home/earlopain/Documents/good_job/lib/good_job/current_thread.rb:113:in `within'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:478:in `block in retry_job'",
 "/home/earlopain/Documents/good_job/app/models/concerns/good_job/advisory_lockable.rb:368:in `with_advisory_lock'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:457:in `retry_job'",
 "/home/earlopain/Documents/good_job/spec/app/models/good_job/job_spec.rb:138:in `block (5 levels) in <top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/matchers/built_in/change.rb:395:in `perform_change'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/matchers/built_in/change.rb:223:in `perform_change'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/matchers/built_in/change.rb:193:in `matches?'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/handler.rb:51:in `block in handle_matcher'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/handler.rb:27:in `with_matcher'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/handler.rb:48:in `handle_matcher'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/expectation_target.rb:65:in `to'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/expectation_target.rb:139:in `to'",
 "/home/earlopain/Documents/good_job/spec/app/models/good_job/job_spec.rb:136:in `block (4 levels) in <top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:263:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:263:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:511:in `block in with_around_and_singleton_context_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:468:in `block in with_around_example_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:486:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:626:in `block in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-rails-6.1.3/lib/rspec/rails/adapters.rb:75:in `block (2 levels) in <module:MinitestLifecycleAdapter>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/reset_rails_queue_adapter.rb:29:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/reset_good_job.rb:29:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/rails_executor.rb:19:in `block (3 levels) in <main>'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/execution_wrapper.rb:104:in `perform'",
 "/home/earlopain/Documents/good_job/spec/support/rails_executor.rb:17:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/postgres_notices.rb:35:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/logger.rb:7:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/database_cleaner.rb:17:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:629:in `run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:486:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:468:in `with_around_example_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:511:in `with_around_and_singleton_context_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:259:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:646:in `block in run_examples'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:642:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:642:in `run_examples'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:607:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `block (3 levels) in run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `block (2 levels) in run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/configuration.rb:2091:in `with_suite_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:116:in `block in run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/reporter.rb:74:in `report'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:115:in `run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:89:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:71:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:45:in `invoke'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/exe/rspec:4:in `<top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/rspec:25:in `load'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/rspec:25:in `<top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:58:in `load'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:58:in `kernel_load'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:23:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli.rb:455:in `exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli.rb:35:in `dispatch'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli.rb:29:in `start'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/exe/bundle:28:in `block in <top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/exe/bundle:20:in `<top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/bundle:25:in `load'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/bundle:25:in `<main>'"]

2

["/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:428:in `instance_exec'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:428:in `block in make_lambda'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:178:in `block in call'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:667:in `block (2 levels) in default_terminator'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:666:in `catch'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:666:in `block in default_terminator'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:179:in `call'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `block in invoke_before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `each'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:558:in `invoke_before'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:108:in `run_callbacks'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/callbacks.rb:912:in `_run_checkin_callbacks'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:579:in `block (2 levels) in checkin'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/3.3.0/monitor.rb:201:in `synchronize'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/3.3.0/monitor.rb:201:in `mon_synchronize'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:576:in `block in checkin'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:575:in `checkin'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:392:in `release_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:423:in `with_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_handling.rb:310:in `with_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/transactions.rb:410:in `with_transaction_returning_status'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/transactions.rb:366:in `save!'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/suppressor.rb:56:in `save!'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:485:in `block (3 levels) in retry_job'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/transaction.rb:626:in `block in within_new_transaction'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/transaction.rb:623:in `within_new_transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:367:in `within_new_transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:359:in `transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/transactions.rb:234:in `block in transaction'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:414:in `with_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/connection_handling.rb:310:in `with_connection'",
 "/home/earlopain/Documents/rails/activerecord/lib/active_record/transactions.rb:233:in `transaction'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:482:in `block (2 levels) in retry_job'",
 "/home/earlopain/Documents/good_job/lib/good_job/current_thread.rb:113:in `within'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:478:in `block in retry_job'",
 "/home/earlopain/Documents/good_job/app/models/concerns/good_job/advisory_lockable.rb:368:in `with_advisory_lock'",
 "/home/earlopain/Documents/good_job/app/models/good_job/job.rb:457:in `retry_job'",
 "/home/earlopain/Documents/good_job/spec/app/models/good_job/job_spec.rb:138:in `block (5 levels) in <top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/matchers/built_in/change.rb:395:in `perform_change'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/matchers/built_in/change.rb:223:in `perform_change'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/matchers/built_in/change.rb:193:in `matches?'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/handler.rb:51:in `block in handle_matcher'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/handler.rb:27:in `with_matcher'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/handler.rb:48:in `handle_matcher'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/expectation_target.rb:65:in `to'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-expectations-3.13.1/lib/rspec/expectations/expectation_target.rb:139:in `to'",
 "/home/earlopain/Documents/good_job/spec/app/models/good_job/job_spec.rb:136:in `block (4 levels) in <top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:263:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:263:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:511:in `block in with_around_and_singleton_context_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:468:in `block in with_around_example_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:486:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:626:in `block in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-rails-6.1.3/lib/rspec/rails/adapters.rb:75:in `block (2 levels) in <module:MinitestLifecycleAdapter>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/reset_rails_queue_adapter.rb:29:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/reset_good_job.rb:29:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/rails_executor.rb:19:in `block (3 levels) in <main>'",
 "/home/earlopain/Documents/rails/activesupport/lib/active_support/execution_wrapper.rb:104:in `perform'",
 "/home/earlopain/Documents/good_job/spec/support/rails_executor.rb:17:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/postgres_notices.rb:35:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/logger.rb:7:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/Documents/good_job/spec/support/database_cleaner.rb:17:in `block (2 levels) in <main>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:457:in `instance_exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:390:in `execute_with'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:628:in `block (2 levels) in run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:352:in `call'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:629:in `run_around_example_hooks_for'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/hooks.rb:486:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:468:in `with_around_example_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:511:in `with_around_and_singleton_context_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example.rb:259:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:646:in `block in run_examples'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:642:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:642:in `run_examples'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:607:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `block in run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/example_group.rb:608:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `block (3 levels) in run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `map'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:121:in `block (2 levels) in run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/configuration.rb:2091:in `with_suite_hooks'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:116:in `block in run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/reporter.rb:74:in `report'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:115:in `run_specs'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:89:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:71:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/lib/rspec/core/runner.rb:45:in `invoke'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rspec-core-3.13.0/exe/rspec:4:in `<top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/rspec:25:in `load'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/rspec:25:in `<top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:58:in `load'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:58:in `kernel_load'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli/exec.rb:23:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli.rb:455:in `exec'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli.rb:35:in `dispatch'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/cli.rb:29:in `start'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/exe/bundle:28:in `block in <top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'",
 "/home/earlopain/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/bundler-2.5.11/exe/bundle:20:in `<top (required)>'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/bundle:25:in `load'",
 "/home/earlopain/.rbenv/versions/3.3.5/bin/bundle:25:in `<main>'"]

Previously there where 9 checkins, now there are 11, which checks out since there are 2 warnings. I briefly looked at these already but couldn't make sense of them. It is too deep into internals for me.

self.class.transaction(joinable: false, requires_new: true) do

This uses joinable: false which seems relevant. If you add some pp debugging around it:

"BEFORE TRANSACTION"
"INSIDE TRANSACTION BEFORE AJ.retry_job"
Connection returned to pool with 1 advisory locks
"INSIDE TRANSACTION AFTER AJ.retry_job"
Connection returned to pool with 1 advisory locks
"AFTER TRANSCATION"

@bensheldon
Copy link
Owner

This uses joinable: false which seems relevant.

I can admit I don't fully understand nested transactions in Rails; I probably copied that from writing applications.

Short term: I wonder if just removing the joinable/requires_new would fix this (I see many other instances in GoodJob where I didn't use those options

Long term: I noticed this is a place where the connection-based advisory lock should be replaced with a transaction-based one.

@bensheldon
Copy link
Owner

I can remove that right now and see what happens 🤞🏻

@bensheldon bensheldon changed the title Fix CI for current rails-head Allow GoodJob::Job#retry_job's transaction to be joinable (fix CI for current rails-head) Oct 7, 2024
@bensheldon bensheldon changed the title Allow GoodJob::Job#retry_job's transaction to be joinable (fix CI for current rails-head) Allow GoodJob::Job#retry_job's transaction to be joinable (fix CI for current rails-head) Oct 7, 2024
@bensheldon bensheldon added the bug Something isn't working label Oct 7, 2024
@bensheldon bensheldon merged commit b714be6 into bensheldon:main Oct 7, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Development

Successfully merging this pull request may close these issues.

2 participants