New Relic Agent - Breaks OpCache

[sebastianthulin]

Description

I’ve been dealing with a persistent and puzzling issue related to OPCache on a production server running PHP 8.3.10 (though it has also occurred on PHP 8.2.x). After investigating and discussing the issue with the PHP team, I’ve concluded that the culprit is the New Relic extension (Issue report on PHP-SRC: php/php-src#15539).

The issue

About once a day, we encounter file inclusion errors where filenames and function names appear to be "shifted" by one letter, leading to file not found or function does not exist errors. Examples include:

• /includes/config.php being misread as /includes/comfig.php [The "n" in "config" has been shifted to letter "m"]
• wp-includes/class-wp-textdomain-registry.php being misread as wp-includes/class-wp-textdomain-registry.pho [The "p" in "php" has been shifted to letter "o"]
• wp_get_current_user being misread as wp_get_current_urer() [The "s" in "user" has been shifted to letter "r"].

Pattern:

We have identified a very clear pattern here. It is on any file include or function call, and are always one single letter that has been shifted to the left in the alphabet if an "b" in a string is replaced, it will be the letter "a" that replaces the correct char.

These issues seem to be tied to periods of high traffic when many uncached requests hit the server simultaneously (normally cached in pagecache). The problem resolves temporarily when PHP processes are restarted, but it eventually recurs. In som cases the issue may occur in 10 minutes after a reset, and in some cases up to 24 hours.

We have not concluded if this also affects variable names as this won't neccesarily result in a fatal error (in most cases we would se a warning about undefined var).

Steps to Reproduce

Install and enable NewRelic Extension in PHP 8.2 or 8.3 (8.3.10). The error may show after some time (up to 24 hours).

Expected Behavior

The expected behaviour is that opcache is not messed up.

Your Environment

PHP Version: 8.3.10 (Issue also present in 8.2.x, a PHP update was made in order to mitigate the error)
OPCache Version: 8.3.10
Web Server: LiteSpeed (with lsphp)
Caching: OPCache enabled
Traffic: High-traffic site
Memory: 120GB RAM (utilize max 15GB of these)
Disk: 1.5 TB (utilize 49GB)
Hosting: Openstack Compute
OS: Ubuntu 22.04.3 LTS (Jammy Jellyfish)
Number of vhosts: 2 (two nearly identical codebases are deployed in /var/www/prod/ and /var/www/stage/)
Deployed solution: WordPress (highly adapted to usecase)

New Relic deamon

dpkg -l | grep newrelic
ii newrelic-daemon 11.0.0.13 amd64 The New Relic proxy daemon
ii newrelic-infra 1.56.1 amd64 New Relic Infrastructure provides flexible, dynamic server monitoring. With real-time data collection and a UI that scales from a handful of hosts to thousands, Infrastructure is designed for modern Operations teams with fast-changing systems.
ii newrelic-php5 11.0.0.13 amd64 The New Relic agent for PHP
ii newrelic-php5-common 11.0.0.13 all Scripts and samples for the New Relic PHP agent

[workato-integration]

https://new-relic.atlassian.net/browse/NR-306842

[newrelic-php-agent-bot]

Hi @sebastianthulin, this issue has been identified as requiring additional information to debug.
Due to the potentially sensitive nature of the information we may have to collect,
such as PHPInfo and Debug Logs, please follow these steps
to create a support case, where a member of our New Relic Support Team will work
with you to gather the necessary information securely and help debug your issue.

If your subscription level does not include technical support, the
other option available to customers is to use New Relic's community support channel: Explorers Hub. From there, a member of our New Relic Support Team will work
with you to gather the necessary information securely and help debug your issue.
For all available options on how to get support, please see these resources.