READONLY You can't write against a read only replica.

[Genaker]

[2020-04-01 01:25:38] main.CRITICAL: READONLY You can't write against a read only replica. {"report_id":"91c1378c517f87a1ffc8455f1ababbe16d709494e48087c35b46fb92b0f70f51","exception":"[object] (CredisException(code: 0): READONLY You can't write against a read only replica. at /var/www/html/magento/vendor/colinmollenhour/credis/Client.php:1157, RedisException(code: 0): READONLY You can't write against a read only replica. at /var/www/html/magento/vendor/colinmollenhour/credis/Client.php:1119)"} []
[2020-04-01 01:25:38] main.CRITICAL: READONLY You can't write against a read only replica. {"report_id":"91c1378c517f87a1ffc8455f1ababbe16d709494e48087c35b46fb92b0f70f51","exception":"[object] (CredisException(code: 0): READONLY You can't write against a read only replica. at /var/www/html/magento/vendor/colinmollenhour/credis/Client.php:1157, RedisException(code: 0): READONLY You can't write against a read only replica. at /var/www/html/magento/vendor/colinmollenhour/credis/Client.php:1119)"} []

Magento configuration

backend_options' => [
'server' => 'redis.0wfhud.ng.0001.use2.cache.amazonaws.com',
'database' => '0',
'port' => '6379',
'password' => '',
'persistent' => 'redis',
'compress_data' => '9',
'connect_retries' => 5,
'compression_lib' => 'gzip',
'load_from_slave' => 'tcp://redis-ro.0wfhud.ng.0001.use2.cache.amazonaws.com:6379',
'retry_reads_on_master' => '1',
]

[colinmollenhour]

Thanks for the report, can you provide a fuller stack trace?

[Genaker]

{"0":"READONLY You can't write against a read only replica.","1":"#1 Cm\RedisSession\Handler->read() called at [vendor/magento/framework/Session/SaveHandler/Redis.php:97]\n#2 Magento\Framework\Session\SaveHandler\Redis->read() called at [vendor/magento/framework/Session/SaveHandler.php:138]\n#3 Magento\Framework\Session\SaveHandler->callSafely() called at [vendor/magento/framework/Session/SaveHandler.php:83]\n#4 Magento\Framework\Session\SaveHandler->read()\n#5 session_start() called at [vendor/magento/framework/Session/SessionManager.php:206]\n#6 Magento\Framework\Session\SessionManager->start() called at [vendor/magento/framework/Interception/Interceptor.php:58]\n#7 Magento\Framework\Session\Generic\Interceptor->___callParent() called at [vendor/magento/framework/Interception/Interceptor.php:138]\n#8 Magento\Framework\Session\Generic\Interceptor->Magento\Framework\Interception\{closure}() called at [vendor/magento/framework/Interception/Interceptor.php:153]\n#9 Magento\Framework\Session\Generic\Interceptor->___callPlugins() called at [generated/code/Magento/Framework/Session/Generic/Interceptor.php:26]\n#10 Magento\Framework\Session\Generic\Interceptor->start() called at [vendor/magento/framework/Session/SessionManager.php:140]\n#11 Magento\Framework\Session\SessionManager->__construct() called at [generated/code/Magento/Framework/Session/Generic/Interceptor.php:14]\n#12 Magento\Framework\Session\Generic\Interceptor->__construct() called at [vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php:121]\n#13 Magento\Framework\ObjectManager\Factory\AbstractFactory->createObject() called at [vendor/magento/framework/ObjectManager/Factory/Compiled.php:108]\n#14 Magento\Framework\ObjectManager\Factory\Compiled->create() called at [vendor/magento/framework/ObjectManager/Factory/Compiled.php:150]\n#15 Magento\Framework\ObjectManager\Factory\Compiled->get() called at [vendor/magento/framework/ObjectManager/Factory/Compiled.php:79]\n#16 Magento\Framework\ObjectManager\Factory\Compiled->create() called at [vendor/magento/framework/ObjectManager/Factory/Compiled.php:150]\n#17 Magento\Framework\ObjectManager\Factory\Compiled->get() called at [vendor/magento/framework/ObjectManager/Factory/Compiled.php:79]\n#18 Magento\Framework\ObjectManager\Factory\Compiled->create() called at [vendor/magento/framework/ObjectManager/Factory/Compiled.php:150]\n#19 Magento\Framework\ObjectManager\Factory\Compiled->get() called at [vendor/magento/framework/ObjectManager/Factory/Compiled.php:79]\n#20 Magento\Framework\ObjectManager\Factory\Compiled->create() called at [vendor/magento/framework/ObjectManager/Factory/Compiled.php:150]\n#21 Magento\Framework\ObjectManager\Factory\Compiled->get() called at [vendor/magento/framework/ObjectManager/Factory/Compiled.php:125]\n#22 Magento\Framework\ObjectManager\Factory\Compiled->parseArray() called at [vendor/magento/framework/ObjectManager/Factory/Compiled.php:86]\n#23 Magento\Framework\ObjectManager\Factory\Compiled->create() called at [vendor/magento/framework/ObjectManager/ObjectManager.php:70]\n#24 Magento\Framework\ObjectManager\ObjectManager->get() called at [vendor/magento/framework/App/FrontController.php:71]\n#25 Magento\Framework\App\FrontController->__construct() called at [generated/code/Magento/Framework/App/FrontController/Interceptor.php:14]\n#26 Magento\Framework\App\FrontController\Interceptor->__construct() called at [vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php:121]\n#27 Magento\Framework\ObjectManager\Factory\AbstractFactory->createObject() called at [vendor/magento/framework/ObjectManager/Factory/Compiled.php:108]\n#28 Magento\Framework\ObjectManager\Factory\Compiled->create() called at [vendor/magento/framework/ObjectManager/ObjectManager.php:70]\n#29 Magento\Framework\ObjectManager\ObjectManager->get() called at [vendor/magento/framework/App/Http.php:115]\n#30 Magento\Framework\App\Http->launch() called at [vendor/magento/framework/App/Bootstrap.php:261]\n#31 Magento\Framework\App\Bootstrap->run() called at [pub/index.php:40]\n","url":"/test-saas-product.html?_tideways%5Bmethod%5D=&_tideways%5Btime%5D=1585705200&_tideways%5Buser%5D=19900&_tideways%5Bhash%5D=d070227bd9cdeebe24698917bce7b91b7405cbd0c13f4656247a64c6d41e3ccb&_tideways_ref=d8b81646-2923-4235-ac80-ea659fb68d7a","script_name":"/index.php","report_id":"91c1378c517f87a1ffc8455f1ababbe16d709494e48087c35b46fb92b0f70f51"}

What's interesting sessions don't use slaves. Sessions writes to the different Redis

'session' => [
'save' => 'redis',
'redis' => [
'host' => '152.14.20.80',
'port' => '6379',
'password' => '',
'timeout' => '2.5',
'persistent_identifier' => 'redis1',
'database' => '2',
'compression_threshold' => '2048',
'compression_library' => 'gzip',
'log_level' => '1',
'max_concurrency' => '6',
'break_after_frontend' => '5',
'break_after_adminhtml' => '30',
'first_lifetime' => '600',
'bot_first_lifetime' => '60',
'bot_lifetime' => '7200',
'disable_locking' => '1',
'min_lifetime' => '60',
'max_lifetime' => '2592000',
'sentinel_master' => '',
'sentinel_servers' => '',
'sentinel_connect_retries' => '5',
'sentinel_verify_master' => '0'
]
],

It is realy rarely issue but I'm having it each day several times.

As I can see from the repost it tries to write the session to the read-only slave DB it uses cache Redis somehow

[Genaker]

It is my full config:

'session' => [
'save' => 'redis',
'redis' => [
'host' => '152.14.20.80',
'port' => '6379',
'password' => '',
'timeout' => '2.5',
'persistent_identifier' => 'redis1',
'database' => '2',
'compression_threshold' => '2048',
'compression_library' => 'gzip',
'log_level' => '1',
'max_concurrency' => '6',
'break_after_frontend' => '5',
'break_after_adminhtml' => '30',
'first_lifetime' => '600',
'bot_first_lifetime' => '60',
'bot_lifetime' => '7200',
'disable_locking' => '1',
'min_lifetime' => '60',
'max_lifetime' => '2592000',
'sentinel_master' => '',
'sentinel_servers' => '',
'sentinel_connect_retries' => '5',
'sentinel_verify_master' => '0'
]
],
'cache' => [
'frontend' => [
'default' => [
'id_prefix' => 'def_',
'backend' => 'Cm_Cache_Backend_Redis',
'backend_options' => [
'server' => 'redis.0wfhud.ng.0001.use2.cache.amazonaws.com',
'database' => '0',
'port' => '6379',
'password' => '',
'persistent' => 'redis',
'compress_data' => '9',
'connect_retries' => 5,
'compression_lib' => 'gzip',
'load_from_slave' => 'tcp://redis-ro.0wfhud.ng.0001.use2.cache.amazonaws.com:6379',
'retry_reads_on_master' => '1',
]
],
'page_cache' => [
'id_prefix' => 'def_',
'backend' => 'Cm_Cache_Backend_Redis',
'backend_options' => [
'server' => '152.14.20.80',
'database' => '1',
'port' => '6379',
'password' => '',
'persistent' => 'redis2',
'compress_data' => '0',
'compression_lib' => ''
]
]
]
],

[kanevbg]

Experiencing the same issue running on Redis sentinel of 3 nodes.

In my observations error happens every day after full cache flush when a lot of data is present (trough Magento's PHP API, cant say if flushed from Redis directly). Types of errors observed:

• SELECT failed while reconnecting
• read error on connection to 127.0.0.1:6378
• READONLY You can't write against a read only replica.

Errors are not the same each day, however they are the same after flush, eg. there is only one kind of errors after the given flush for less than a minute, than starts functioning normally.

[colinmollenhour]

Are you both using the Magento patch or Magento LTS which has a fix for the config cache stampeding issue? Without this patch flushing the cache on a site with moderate traffic can easily cause the cache backend to get overwhelmed. Also I recommend using a cache refresh rather than flush so that the config cache gets atomically updated rather than clobbered and rebuilt by the next request hitting a cold cache.

[kanevbg]

My project has Mage_Core_Model_Config class taken from the openmage lts (from master or last release) project, so yes?

This morning all of the three errors had happened after the cache flush (all of the three mentioned types). I have cache flush configured at 5:05 am trough crontab and shell script. From 5:05:07 to 5:05:33 Redis related errors are logged in Magento reports. I will implement cache clearance trough mass refresh to see if this makes difference, however if this do it still seems something is not in place.

EDIT: By the way these two has significant differences, one of which is the useCache flag. Which version is correct:

• https://github.com/OpenMage/magento-lts/blob/4c72dcd46a20ec1032b638c79280b453fc84e357/app/code/core/Mage/Core/Model/Config.php
• https://gist.github.com/colinmollenhour/7a91c4a92ccfd2adaeb6
  Maybe the gist is outdated?

[colinmollenhour]

I didn't do a full diff, can you point out the difference more precisely?

Here is the atomic ache update I was referring to: https://gist.github.com/colinmollenhour/2715268
I think n98-magerun also has this functionality built in.

[kanevbg]

See the diff here:
Report.zip

[colinmollenhour]

Ahh, thanks. Inspecting the code I think that shouldn't be a material difference because the value is declared as false by default and if it is set to true that code in loadModulesCache isn't evaluated anyway.

[Genaker]

I have found the issue

If Redis can't remove keys according to the policy, or if the policy is

set to 'noeviction', Redis will start to reply with errors to commands

that would use more memory, like SET, LPUSH, and so on, and will continue

to reply to read-only commands like GET.

it happens when Redis doesn't have enough memory however looks like Read-only error

Fatal error: Uncaught RedisException: READONLY You can't write against a read only replica. in /var/www/html/magento/vendor/colinmollenhour/credis/Client.php:1119 Stack trace: #0 /var/www/html/magento/vendor/colinmollenhour/credis/Client.php(1119): Redis->exec() #1 /var/www/html/magento/vendor/colinmollenhour/cache-backend-redis/Cm/Cache/Backend/Redis.php(629): Credis_Client->__call('exec', Array) #2 /var/www/html/magento/vendor/magento/zendframework1/library/Zend/Cache/Core.php(515): Cm_Cache_Backend_Redis->save('a:1532:{s:11:"p...', 'def_GLOBAL__DIC...', Array, NULL) #3 /var/www/html/magento/vendor/magento/framework/Cache/Core.php(74): Zend_Cache_Core->save('a:1532:{s:11:"p...', 'def_GLOBAL__DIC...', Array, NULL, 8) #4 /var/www/html/magento/vendor/magento/framework/Cache/Frontend/Adapter/Zend.php(63): Magento\Framework\Cache\Core->save('a:1532:{s:11:"p...', 'GLOBAL::DICONFI...', Array, NULL) #5 /var/www/html/magento/vendor/magento/framework/Cache/Frontend/Decorator/Bare.php(75): Magento\Framework\Cache\Frontend\Adapter\ in /var/www/html/magento/vendor/colinmollenhour/credis/Client.php on line 1157