Speedtest batch failed for unknown reasons

[AlliedMineral]

Describe the bug
Speedtests began failing for me after midnight local time on Jan 6. All schedulded and manually run speedtests show in the results chart as "Status: Checking" and do not complete.

In the container logs I can see the error message below that the speedtest batch is failing for unknown reasons. I get this regardless of the selection for the destination test server.

Environment (please complete the following information):

• Unraid Docker
• Version 1.0.3

Logs
[2025-01-07 17:42:56] production.ERROR: Speedtest batch "9de9ba3f-43e0-4e86-9f4e-c953582dcc98" failed for an unknown reason.
[2025-01-07 17:42:56] production.ERROR: App\Jobs\Ookla\RunSpeedtestJob has been attempted too many times. {"exception":"[object] (Illuminate\Queue\MaxAttemptsExceededException(code: 0): App\Jobs\Ookla\RunSpeedtestJob has been attempted too many times. at /app/www/vendor/laravel/framework/src/Illuminate/Queue/MaxAttemptsExceededException.php:24)
[stacktrace]
#0 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(794): Illuminate\Queue\MaxAttemptsExceededException::forJob()
#1 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(528): Illuminate\Queue\Worker->maxAttemptsExceededException()
#2 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(431): Illuminate\Queue\Worker->markJobAsFailedIfAlreadyExceedsMaxAttempts()
#3 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(392): Illuminate\Queue\Worker->process()
#4 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(178): Illuminate\Queue\Worker->runJob()
#5 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(149): Illuminate\Queue\Worker->daemon()
#6 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(132): Illuminate\Queue\Console\WorkCommand->runWorker()
#7 /app/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\Queue\Console\WorkCommand->handle()
#8 /app/www/vendor/laravel/framework/src/Illuminate/Container/Util.php(43): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#9 /app/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(95): Illuminate\Container\Util::unwrapIfClosure()
#10 /app/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod()
#11 /app/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(694): Illuminate\Container\BoundMethod::call()
#12 /app/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(213): Illuminate\Container\Container->call()
#13 /app/www/vendor/symfony/console/Command/Command.php(279): Illuminate\Console\Command->execute()
#14 /app/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(182): Symfony\Component\Console\Command\Command->run()
#15 /app/www/vendor/symfony/console/Application.php(1094): Illuminate\Console\Command->run()
#16 /app/www/vendor/symfony/console/Application.php(342): Symfony\Component\Console\Application->doRunCommand()
#17 /app/www/vendor/symfony/console/Application.php(193): Symfony\Component\Console\Application->doRun()
#18 /app/www/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(198): Symfony\Component\Console\Application->run()
#19 /app/www/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(1205): Illuminate\Foundation\Console\Kernel->handle()
#20 /app/www/artisan(13): Illuminate\Foundation\Application->handleCommand()
#21 {main}
"}

[AlliedMineral]

I reverted to 1.0.2 and I am able to run Speedtests again

[alexjustesen]

@AlliedMineral what frequency are you running tests?

[lmcastaneda]

I reverted to 1.0.2 and I am able to run Speedtests again

Had the same issue, reverted back to 1.0.2 and can report the speed tests are running again.

@alexjustesen If it helps, I have the tests running every hour.

[AlliedMineral]

@AlliedMineral what frequency are you running tests?

I had set them to run hourly, (0 * * * *) but beginning with the scheduled test on 1/6/2024 at 01:00 all of the scheduled tests failed. The test at 00:00 was successful. Manually queued tests that I attempted to run this morning after discovering this issue all failed regardless of the selected server for the test.

[albeedelacruz]

ohh same issue. Running tests hourly/ Reverted to 1.02--> issue gone.

[MindScorpio]

Same here as @albeedelacruz

[svenvg93]

I have not been able to replicate this problem. I'm running tests every hour myself as well with 1.0.3 without any problems.
Can you share your docker compose, it might help us replicate what happens 😄 As well, are you running any kind of pi-hole or similar on your network?

[albeedelacruz]

Not sure how to replicate this too but I have 3 identical containers in one pi running speedtests for each of my ISP every hour (not simultaneously) and only 1 of 3 had this problem. Reverting to 1.02 for that one got it back up and running again. I left the other 2 as is and they are still running fine to this day.On Jan 11, 2025, at 7:52 PM, Sven van Ginkel ***@***.***> wrote: I have not been able to replicate this problem. I'm running tests every hour myself as well with 1.0.3 without any problems. Can you share your docker compose, it might help us replicate what happens 😄 As well, are you running any kind of pi-hole or similar on your network? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[AlliedMineral]

I have not been able to replicate this problem. I'm running tests every hour myself as well with 1.0.3 without any problems. Can you share your docker compose, it might help us replicate what happens 😄 As well, are you running any kind of pi-hole or similar on your network?

I don't exactly have a docker compose to share because I'm running this in Unraid. (Or if there is somewhere I don't know how to get it).

I am running a pihole w/cloudflared and I block/redirect all port 53 traffic at the firewall.

[APetri36]

Getting this same error, but it only appears to be on manually run tests for me. I have scheduled tests running on the same test server ID that run fine (every 8 hours). Haven't reverted to 1.02 yet, but can if need be. No pihole on my network. Compose file in case it helps.

[eightalex]

I have the same problem, revert to 1.0.2 fixed the issue

[albeedelacruz]

Just to note: After switching to 1.0.2 and resolving the issue, I switched back to the latest and it seems to work fine now too. Will observe for the next 24 hours.

[svenvg93]

Just to note: After switching to 1.0.2 and resolving the issue, I switched back to the latest and it seems to work fine now too. Will observe for the next 24 hours.

Intresting and more confusing at the same time 😅 .

@albeedelacruz @eightalex @APetri36 @MindScorpio @lmcastaneda And everybody else that I forget, can you try with the 1.1.0 release from yesterday. Nothing specific has been fixed in terms of this problems. But as switching back and forth fixed it for @albeedelacruz It might work again when upgrading to the latest version.

[albeedelacruz]

Yes, post 24 hours and tests are still running fine.

[alexjustesen]

Yes, post 24 hours and tests are still running fine.

For those that are having this issue did you have a failed or incomplete speedtest in the last 24-48h?

[AlliedMineral]

Updated to 1.1.0 and the initial manually triggered speed test ran successfully. I will continue to monitor.

Pulling image: lscr.io/linuxserver/speedtest-tracker:1.1.0
IMAGE ID [1.1.0]: Pulling from linuxserver/speedtest-tracker.
IMAGE ID [5789772697fb]: Already exists.
IMAGE ID [076a58089b36]: Already exists.
IMAGE ID [ba44afedd04c]: Already exists.
IMAGE ID [1eb0e96d6e78]: Already exists.
IMAGE ID [497bd28cf8ee]: Already exists.
IMAGE ID [e2cfb95fd4d9]: Already exists.
IMAGE ID [94d3442d6383]: Already exists.
IMAGE ID [2023842854b7]: Already exists.
IMAGE ID [762808a55b90]: Already exists.
IMAGE ID [ee3e86816113]: Pulling fs layer. Downloading 100% of 29 MB. Verifying Checksum. Download complete. Extracting. Pull complete.
IMAGE ID [6ba1dbcbd945]: Pulling fs layer. Downloading 100% of 2 KB. Verifying Checksum. Download complete. Extracting. Pull complete.
Status: Downloaded newer image for lscr.io/linuxserver/speedtest-tracker:1.1.0

TOTAL DATA PULLED: 29 MB

Stopping container: speedtest-tracker
Successfully stopped container 'speedtest-tracker'

Removing container: speedtest-tracker
Successfully removed container 'speedtest-tracker'

Command execution
docker run
-d
--name='speedtest-tracker'
--net='bridge'
--pids-limit 2048
-e TZ="America/Denver"
-e HOST_OS="Unraid"
-e HOST_HOSTNAME="Tower"
-e HOST_CONTAINERNAME="speedtest-tracker"
-e 'DB_CONNECTION'='mysql'
-e 'APP_KEY'='xxx'
-e 'DB_HOST'='192.168.0.222'
-e 'DB_PORT'='3306'
-e 'DB_DATABASE'='speedtest_tracker'
-e 'DB_USERNAME'='xxx'
-e 'DB_PASSWORD'='xxx'
-e 'CONTENT_WIDTH'='full'
-e 'DISPLAY_TIMEZONE'='America/Denver'
-e 'SPEEDTEST_SCHEDULE'='8 * * * *'
-e 'SPEEDTEST_SERVERS'='8862'
-e 'PUID'='99'
-e 'PGID'='100'
-e 'UMASK'='022'
-l net.unraid.docker.managed=dockerman
-l net.unraid.docker.webui='http://[IP]:[PORT:80]'
-l net.unraid.docker.icon='https://raw.githubusercontent.com/linuxserver/docker-templates/master/linuxserver.io/img/speedtest-tracker-logo.png'
-p '9442:80/tcp'
-v '/mnt/user/appdata/speedtest-tracker-1':'/config':'rw' 'lscr.io/linuxserver/speedtest-tracker:1.1.0'
780ea1ac27a7ffb12199c046b5c7a1da92588aa4f0318f2a863ba59b464e4122

The command finished successfully!

[lmcastaneda]

Just to note: After switching to 1.0.2 and resolving the issue, I switched back to the latest and it seems to work fine now too. Will observe for the next 24 hours.

Intresting and more confusing at the same time 😅 .

@albeedelacruz @eightalex @APetri36 @MindScorpio @lmcastaneda And everybody else that I forget, can you try with the 1.1.0 release from yesterday. Nothing specific has been fixed in terms of this problems. But as switching back and forth fixed it for @albeedelacruz It might work again when upgrading to the latest version.

Just switched my container to 1.1.0. It did work for me.

[MoShiKB]

1.1.0 didn't work for me, but reverting to 1.0.3 did.
also it was pretty weird, cause the first ~25 tests failed, but then it started to work. (on 1.0.3)
(I'm running one every 15 minutes)

that's the errors I received:

[2025-01-17 01:45:02] production.ERROR: Speedtest batch "9dfc8182-a466-4005-88e0-7c9517002726" failed for an unknown reason. [2025-01-17 01:45:02] production.ERROR: App\Jobs\Ookla\RunSpeedtestJob has been attempted too many times. {"exception":"[object] (Illuminate\\Queue\\MaxAttemptsExceededException(code: 0): App\\Jobs\\Ookla\\RunSpeedtestJob has been attempted too many times. at /app/www/vendor/laravel/framework/src/Illuminate/Queue/MaxAttemptsExceededException.php:24) [stacktrace] #0 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(794): Illuminate\\Queue\\MaxAttemptsExceededException::forJob() #1 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(528): Illuminate\\Queue\\Worker->maxAttemptsExceededException() #2 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(431): Illuminate\\Queue\\Worker->markJobAsFailedIfAlreadyExceedsMaxAttempts() #3 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(392): Illuminate\\Queue\\Worker->process() #4 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(178): Illuminate\\Queue\\Worker->runJob() #5 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(149): Illuminate\\Queue\\Worker->daemon() #6 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(132): Illuminate\\Queue\\Console\\WorkCommand->runWorker() #7 /app/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\\Queue\\Console\\WorkCommand->handle() #8 /app/www/vendor/laravel/framework/src/Illuminate/Container/Util.php(43): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}() #9 /app/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(95): Illuminate\\Container\\Util::unwrapIfClosure() #10 /app/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod() #11 /app/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(694): Illuminate\\Container\\BoundMethod::call() #12 /app/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(213): Illuminate\\Container\\Container->call() #13 /app/www/vendor/symfony/console/Command/Command.php(279): Illuminate\\Console\\Command->execute() #14 /app/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(182): Symfony\\Component\\Console\\Command\\Command->run() #15 /app/www/vendor/symfony/console/Application.php(1094): Illuminate\\Console\\Command->run() #16 /app/www/vendor/symfony/console/Application.php(342): Symfony\\Component\\Console\\Application->doRunCommand() #17 /app/www/vendor/symfony/console/Application.php(193): Symfony\\Component\\Console\\Application->doRun() #18 /app/www/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(198): Symfony\\Component\\Console\\Application->run() #19 /app/www/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(1205): Illuminate\\Foundation\\Console\\Kernel->handle() #20 /app/www/artisan(13): Illuminate\\Foundation\\Application->handleCommand() #21 {main} "}

[svenvg93]

1.1.0 didn't work for me, but reverting to 1.0.3 did. also it was pretty weird, cause the first ~25 tests failed, but then it started to work. (on 1.0.3) (I'm running one every 15 minutes)

Did it stop working after updating to 1,1,0? What happens if you now switch back to 1.1.0.

[MoShiKB]

1.1.0 didn't work for me, but reverting to 1.0.3 did. also it was pretty weird, cause the first ~25 tests failed, but then it started to work. (on 1.0.3) (I'm running one every 15 minutes)

Did it stop working after updating to 1,1,0? What happens if you now switch back to 1.1.0.

It stopped working after the update to 1.1.0, sounds to me like the first speedtest got stuck and the rest didn't run as planned, restarting the container didn't help also not deleting the old jobs.
now its working. dunno what changed.

[itsmemac]

After much trying all the above given solutions, narrowed it down to these scenarios where the test fails (atleast for me)

• There is a server on list (.env) but the server is offline atm (doesn't go for the next available server from list)
• If there is no preferred servers defined in the .env (doesn't automatically select nearest server)

Preferred server is not marked as Required in docs so it shouldn't cause this issue (in earlier version, you didn't have to set preferred server in .env)
Using latest version ATM and having working speedtest server ID in config fixed the issue
Manual test always runs successfully in either cases

[alexjustesen]

• There is a server on list (.env) but the server is offline atm (doesn't go for the next available server from list)

This would make sense on why the tests are failing, what was the error message in the logs from the CLI?

[itsmemac]

• There is a server on list (.env) but the server is offline atm (doesn't go for the next available server from list)

This would make sense on why the tests are failing, what was the error message in the logs from the CLI?

no error messages in logs. Completely normal

[rpschube]

I am experiencing the same thing, all of my tests are suddenly stuck in a "Checking" status. Batches have been failing for days with this same error

[2025-01-23 19:34:01] production.ERROR: App\Jobs\Ookla\RunSpeedtestJob has been attempted too many times. {"exception":"[object] (Illuminate\\Queue\\MaxAttemptsExceededException(code: 0): App\\Jobs\\Ookla\\RunSpeedtestJob has been attempted too many times. at /app/www/vendor/laravel/framework/src/Illuminate/Queue/MaxAttemptsExceededException.php:24) [stacktrace] #0 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(794): Illuminate\\Queue\\MaxAttemptsExceededException::forJob() #1 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(528): Illuminate\\Queue\\Worker->maxAttemptsExceededException() #2 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(431): Illuminate\\Queue\\Worker->markJobAsFailedIfAlreadyExceedsMaxAttempts() #3 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(392): Illuminate\\Queue\\Worker->process() #4 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(178): Illuminate\\Queue\\Worker->runJob() #5 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(149): Illuminate\\Queue\\Worker->daemon() #6 /app/www/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(132): Illuminate\\Queue\\Console\\WorkCommand->runWorker() #7 /app/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\\Queue\\Console\\WorkCommand->handle() #8 /app/www/vendor/laravel/framework/src/Illuminate/Container/Util.php(43): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}() #9 /app/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(95): Illuminate\\Container\\Util::unwrapIfClosure() #10 /app/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod() #11 /app/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(694): Illuminate\\Container\\BoundMethod::call() #12 /app/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(213): Illuminate\\Container\\Container->call() #13 /app/www/vendor/symfony/console/Command/Command.php(279): Illuminate\\Console\\Command->execute() #14 /app/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(182): Symfony\\Component\\Console\\Command\\Command->run() #15 /app/www/vendor/symfony/console/Application.php(1094): Illuminate\\Console\\Command->run() #16 /app/www/vendor/symfony/console/Application.php(342): Symfony\\Component\\Console\\Application->doRunCommand() #17 /app/www/vendor/symfony/console/Application.php(193): Symfony\\Component\\Console\\Application->doRun() #18 /app/www/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(198): Symfony\\Component\\Console\\Application->run() #19 /app/www/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(1208): Illuminate\\Foundation\\Console\\Kernel->handle() #20 /app/www/artisan(13): Illuminate\\Foundation\\Application->handleCommand() #21 {main} "}

I have tried all of the "workarounds" mentioned in this thread, have not been able to get a single successful test since this started unfortunately.