If a job fails because it "has been attempted too many times", the original error never gets reported

[tkoop]

Horizon Version

5.29.0

Laravel Version

11.10.0

PHP Version

8.2

Redis Driver

PhpRedis

Redis Version

7.2.4

Database Driver & Version

No response

Description

In our logs, we are noticing that a job is failing. It says this job "has been attempted too many times". But the stack trace in the log only shows the stack trace for /vendor/laravel/framework/src/Illuminate/Queue/MaxAttemptsExceededException.php:24, not the original error. I want to know the real reason the job is failing, and the stack trace and exception of the original error.

Steps To Reproduce

This is our horizon.php config file.

<?php

use Illuminate\Support\Str;

return [
	/*
    |--------------------------------------------------------------------------
    | Horizon Domain
    |--------------------------------------------------------------------------
    |
    | This is the subdomain where Horizon will be accessible from. If this
    | setting is null, Horizon will reside under the same domain as the
    | application. Otherwise, this value will serve as the subdomain.
    |
    */

	"domain" => env("HORIZON_DOMAIN"),

	/*
    |--------------------------------------------------------------------------
    | Horizon Path
    |--------------------------------------------------------------------------
    |
    | This is the URI path where Horizon will be accessible from. Feel free
    | to change this path to anything you like. Note that the URI will not
    | affect the paths of its internal API that aren't exposed to users.
    |
    */

	"path" => env("HORIZON_PATH", "horizon"),

	/*
    |--------------------------------------------------------------------------
    | Horizon Redis Connection
    |--------------------------------------------------------------------------
    |
    | This is the name of the Redis connection where Horizon will store the
    | meta information required for it to function. It includes the list
    | of supervisors, failed jobs, job metrics, and other information.
    |
    */

	"use" => "default",

	/*
    |--------------------------------------------------------------------------
    | Horizon Redis Prefix
    |--------------------------------------------------------------------------
    |
    | This prefix will be used when storing all Horizon data in Redis. You
    | may modify the prefix when you are running multiple installations
    | of Horizon on the same server so that they don't have problems.
    |
    */

	"prefix" => env("HORIZON_PREFIX", Str::slug(env("APP_NAME", "laravel"), "_") . "_horizon:"),

	/*
    |--------------------------------------------------------------------------
    | Horizon Route Middleware
    |--------------------------------------------------------------------------
    |
    | These middleware will get attached onto each Horizon route, giving you
    | the chance to add your own middleware to this list or change any of
    | the existing middleware. Or, you can simply stick with this list.
    |
    */

	"middleware" => ["web"],

	/*
    |--------------------------------------------------------------------------
    | Queue Wait Time Thresholds
    |--------------------------------------------------------------------------
    |
    | This option allows you to configure when the LongWaitDetected event
    | will be fired. Every connection / queue combination may have its
    | own, unique threshold (in seconds) before this event is fired.
    |
    */

	"waits" => [
		"redis:default" => 60,
	],

	/*
    |--------------------------------------------------------------------------
    | Job Trimming Times
    |--------------------------------------------------------------------------
    |
    | Here you can configure for how long (in minutes) you desire Horizon to
    | persist the recent and failed jobs. Typically, recent jobs are kept
    | for one hour while all failed jobs are stored for an entire week.
    |
    */

	"trim" => [
		"recent" => 60,
		"pending" => 60,
		"completed" => 60,
		"recent_failed" => 10080,
		"failed" => 10080,
		"monitored" => 10080,
	],

	/*
    |--------------------------------------------------------------------------
    | Silenced Jobs
    |--------------------------------------------------------------------------
    |
    | Silencing a job will instruct Horizon to not place the job in the list
    | of completed jobs within the Horizon dashboard. This setting may be
    | used to fully remove any noisy jobs from the completed jobs list.
    |
    */

	"silenced" => [
		// App\Jobs\ExampleJob::class,
	],

	/*
    |--------------------------------------------------------------------------
    | Metrics
    |--------------------------------------------------------------------------
    |
    | Here you can configure how many snapshots should be kept to display in
    | the metrics graph. This will get used in combination with Horizon's
    | `horizon:snapshot` schedule to define how long to retain metrics.
    |
    */

	"metrics" => [
		"trim_snapshots" => [
			"job" => 24,
			"queue" => 24,
		],
	],

	/*
    |--------------------------------------------------------------------------
    | Fast Termination
    |--------------------------------------------------------------------------
    |
    | When this option is enabled, Horizon's "terminate" command will not
    | wait on all of the workers to terminate unless the --wait option
    | is provided. Fast termination can shorten deployment delay by
    | allowing a new instance of Horizon to start while the last
    | instance will continue to terminate each of its workers.
    |
    */

	"fast_termination" => false,

	/*
    |--------------------------------------------------------------------------
    | Memory Limit (MB)
    |--------------------------------------------------------------------------
    |
    | This value describes the maximum amount of memory the Horizon master
    | supervisor may consume before it is terminated and restarted. For
    | configuring these limits on your workers, see the next section.
    |
    */

	"memory_limit" => 64,

	/*
    |--------------------------------------------------------------------------
    | Queue Worker Configuration
    |--------------------------------------------------------------------------
    |
    | Here you may define the queue worker settings used by your application
    | in all environments. These supervisors and settings handle all your
    | queued jobs and will be provisioned by Horizon during deployment.
    |
    */

	"defaults" => [],

	"environments" => [
		// in dev, just have one queue worker for everything
		"local" => [
			"supervisor-default" => [
				"connection" => "redis",
				"queue" => [
					"default",
					"notifications",
					"notifications-priority",
					"notifications-bulk",
					"background-processes",
				],
				"balance" => false,
				"minProcesses" => 1,
				"maxProcesses" => 1,
				"timeout" => config("app.browsershot_timeout"),
				"memory" => 128,
				"tries" => 1,
			],
		],

		"*" => [
			// supervisor that manages the default queue and other medium priority tasks
			"supervisor-default" => [
				"connection" => "redis",
				"queue" => ["default"],
				"balance" => false,
				"minProcesses" => 1,
				"maxProcesses" => 1,
				"tries" => 3,
			],

			// supervisor that manages queues that are high priority
			"supervisor-notifications" => [
				"connection" => "redis",
				"queue" => ["notifications", "notifications-priority", "notifications-bulk"],
				"balance" => "simple",
				"minProcesses" => 1,
				"maxProcesses" => 3,
				"tries" => 3,
			],

			// supervisor that manages processes which can use a lot of resources
			"supervisor-background" => [
				"connection" => "redis",
				"queue" => ["background-processes"],
				"balance" => false,
				"minProcesses" => 1,
				"maxProcesses" => 1,
				"timeout" => config("app.browsershot_timeout"),
				"memory" => 128,
				"tries" => 3,
			],
		],
	],
];

[github-actions]

Thank you for reporting this issue!

As Laravel is an open source project, we rely on the community to help us diagnose and fix issues as it is not possible to research and fix every issue reported to us via GitHub.

If possible, please make a pull request fixing the issue you have described, along with corresponding tests. All pull requests are promptly reviewed by the Laravel team.

Thank you!

[crynobone]

Feel free to send a PR if you want this feature.

[javis]

Having the same issue since upgrading to Laravel 11. I suspect Horizon is having a bug at killing gracefully the process that has to be terminated

[marensas999]

It's a bug not a feature.

[benjibee]

It's a bug not a feature.

I'd like to second this, it's quite a big issue. The jobs I've had fail where the only visible exception is that the job was tried to many times are painful to debug and track down. The reason I'd say this is a bug and not a feature is that reporting that the job failed because it was tried too many times is simply not reporting the reason the job failed. The exception is visible and right there, but not reported.

[crynobone]

Feel free to submit a PR to fix the issue. Bug or feature this is an open source project where the community can contribute.

[benjibee]

Feel free to submit a PR to fix the issue. Bug or feature this is an open source project where the community can contribute.

I looked into it before posting my comment. It's far outside of my abilities at the moment. Too many areas I've wholly unfamiliar with. If it's not a priority for the team then maybe I'll get there at some point and then I'd be happy to contribute.