From 17936c3229457781d9c53ce9c6de3222269eeb19 Mon Sep 17 00:00:00 2001 From: Zach Neumann Date: Wed, 31 Jul 2024 12:10:53 -0600 Subject: [PATCH 1/7] feat(php): add seg fault troubleshooting page --- .../troubleshooting/segmentation-faults.mdx | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx diff --git a/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx new file mode 100644 index 00000000000..b9b134f7ede --- /dev/null +++ b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx @@ -0,0 +1,113 @@ +--- +title: Segmentation Faults +type: troubleshooting +tags: + - Agents + - PHP agent + - Troubleshooting +metaDescription: Learn what to do if your site seg faults with the New Relic PHP Agent +redirects: + - /docs/agents/php-agent/troubleshooting/segmentation-faults +freshnessValidatedDate: never +--- + +## Problem + +Your site segmentation faults when the New Relic PHP Agent is enabled on PHP 8.0+. + +## Description + +PHP’s Observer API currently has some bugs that can lead to segmentation faults. The New Relic PHP Agent utilizes OAPI for instrumenting your application on PHPs 8.0+ since agent version 10.18. These seg faults only happen when the agent is enabled. This is because PHP’s OAPI codepaths are only executed if there is an observer hooked in, and in all likelihood, the New Relic PHP Agent is the only thing in your environment hooking into OAPI. + +### Trace + +In these situations, the stack trace seems to be relatively consistent. What we see is, as always, the NR fatal signal handler at the top. Then, we see that the signal handler was called during ```zend_observer_fcall_end_all```. This happens when PHP accesses invalid memory when trying to call all registered observers. Any time we see ```zend_observer_*``` in the stack trace without any NR code before the signal handler should raise some eyebrows on whether PHP is the issue + +``` +0 nr_php_backtrace_get_call_site () at php_stack.c:220 + +1 nr_php_frame_info () at php_stack.c:267 + +2 nr_php_backtrace_fd () at php_stack.c:462 + +3 0x00007fa6d6df026c in nr_php_fatal_signal_handler () at php_minit.c:740 + +<signal handler called> + +5 0x00007fa6db184c63 in zend_observer_fcall_end_all () from libphp8.2.so + +6 0x00007fa6db081abd in php_request_shutdown () from libphp8.2.so +``` + +## zend_test Observer + +Luckily, we have an easy way to test if the issue is PHP and or the agent. PHP has a built-in extension called zend_test which has a dummy observer. Enabling this (while disabling the agent), allows the OAPI codepaths to be executed by something other than our agent. Here are the steps to test this: + +- fully disable the agent by setting newrelic.enabled=0. Test here to ensure that it is indeed disabled. +- enable the PHP extension zend_test. This can be done by modifying your Dockerfile as seen below. +- set the ini settings ```zend_test.observer.enabled=1``` and ```zend_test.observer.observe_all=1```. This should enable PHP's observer API +- see if the issue can be reproduced under this circumstance. + +Here is an example for Debian using a PHP docker image: + +``` +FROM php:8.0 + +RUN apt update && apt install -y libxml2-dev && \ + EXTRA_CFLAGS="$(xml2-config --cflags) -I/usr/src/php" docker-php-ext-install zend_test + +RUN echo "zend_test.observer.enabled=1" >> $(php-config --ini-dir)/docker-php-ext-zend_test.ini +RUN echo "zend_test.observer.observe_all=1" >> $(php-config --ini-dir)/docker-php-ext-zend_test.ini +``` + + and an example using an Alpine PHP docker image: + +``` +FROM php:8.0-alpine + +RUN apk add --no-cache libxml2-dev && \ + EXTRA_CFLAGS="$(xml2-config --cflags) -I/usr/src/php" docker-php-ext-install zend_test + +RUN echo "zend_test.observer.enabled=1" >> $(php-config --ini-dir)/docker-php-ext-zend_test.ini +RUN echo "zend_test.observer.observe_all=1" >> $(php-config --ini-dir)/docker-php-ext-zend_test.ini +``` + +Here is an example from phpinfo() output if the extension is installed and enabled as instructed above: + +``` +zend_test + +zend_test extension => enabled + +Directive => Local Value => Master Value +zend_test.limit_copy_file_range => -1 => -1 +zend_test.not_empty_str_test => val => val +zend_test.observe_opline_in_zendmm => Off => Off +zend_test.observer.enabled => On => On +zend_test.observer.execute_internal => Off => Off +zend_test.observer.fiber_destroy => Off => Off +zend_test.observer.fiber_init => Off => Off +zend_test.observer.fiber_switch => Off => Off +zend_test.observer.observe_all => On => On +zend_test.observer.observe_declaring => Off => Off +zend_test.observer.observe_function_names => no value => no value +zend_test.observer.observe_functions => Off => Off +zend_test.observer.observe_includes => Off => Off +zend_test.observer.show_init_backtrace => Off => Off +zend_test.observer.show_opcode => Off => Off +zend_test.observer.show_opcode_in_user_handler => no value => no value +zend_test.observer.show_output => On => On +zend_test.observer.show_return_type => Off => Off +zend_test.observer.show_return_value => Off => Off +zend_test.print_stderr_mshutdown => Off => Off +zend_test.quantity_value => 0 => 0 +zend_test.register_passes => Off => Off +zend_test.replace_zend_execute_ex => Off => Off +zend_test.str_test => no value => no value +``` + +## Known reproduction and tracking the fix + +As explored in [this issue](https://github.com/newrelic/newrelic-php-agent/issues/865), the Attribute syntax causes seg faults on PHP < 8.2 + +Here is [an issue](https://github.com/php/php-src/issues/13817) on the PHP source tracking a fix for PHP 8.3. From e550bb6d3eda2eb4134711424a7ab6e5f74cfce2 Mon Sep 17 00:00:00 2001 From: ZNeumann Date: Wed, 31 Jul 2024 12:22:07 -0600 Subject: [PATCH 2/7] Update src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx --- .../agents/php-agent/troubleshooting/segmentation-faults.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx index b9b134f7ede..66219b950bf 100644 --- a/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx +++ b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx @@ -13,7 +13,7 @@ freshnessValidatedDate: never ## Problem -Your site segmentation faults when the New Relic PHP Agent is enabled on PHP 8.0+. +Your site segmentation faults when the New Relic PHP Agent (version >= 10.18.0.8) is enabled on PHP 8.0+. ## Description From 2f3a1822dc50cad2156737ee352b365d1468fe9d Mon Sep 17 00:00:00 2001 From: Zach Neumann Date: Mon, 19 Aug 2024 08:52:50 -0600 Subject: [PATCH 3/7] fixup --- .../php-agent/troubleshooting/segmentation-faults.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx index 66219b950bf..7fb0bade621 100644 --- a/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx +++ b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx @@ -13,7 +13,7 @@ freshnessValidatedDate: never ## Problem -Your site segmentation faults when the New Relic PHP Agent (version >= 10.18.0.8) is enabled on PHP 8.0+. +Your site generates a segmentation faults when the New Relic PHP Agent (version >= 10.18.0.8) is enabled on PHP 8.0+. ## Description @@ -21,7 +21,7 @@ PHP’s Observer API currently has some bugs that can lead to segmentation fault ### Trace -In these situations, the stack trace seems to be relatively consistent. What we see is, as always, the NR fatal signal handler at the top. Then, we see that the signal handler was called during ```zend_observer_fcall_end_all```. This happens when PHP accesses invalid memory when trying to call all registered observers. Any time we see ```zend_observer_*``` in the stack trace without any NR code before the signal handler should raise some eyebrows on whether PHP is the issue +In these situations, the stack trace seems to be relatively consistent. What we see is, as always, the NR fatal signal handler at the top. Everything in the stack above ```nr_php_fatal_signal_handler``` will always be present for a segmentation fault when the PHP Agent is installed, as we catch the fault for logging purposes. Then, we see that the signal handler was called during ```zend_observer_fcall_end_all```. This happens when PHP accesses invalid memory when trying to call all registered observers. Any time we see ```zend_observer_*``` in the stack trace without any NR code before the signal handler should raise some eyebrows on whether PHP is the issue ``` 0 nr_php_backtrace_get_call_site () at php_stack.c:220 @@ -41,7 +41,7 @@ In these situations, the stack trace seems to be relatively consistent. What we ## zend_test Observer -Luckily, we have an easy way to test if the issue is PHP and or the agent. PHP has a built-in extension called zend_test which has a dummy observer. Enabling this (while disabling the agent), allows the OAPI codepaths to be executed by something other than our agent. Here are the steps to test this: +Luckily, we have an easy way to test if the issue is PHP and or the agent. PHP has a built-in extension called zend_test which has a test observer. Enabling this (while disabling the agent), allows the OAPI codepaths to be executed by something other than our agent. Here are the steps to test this: - fully disable the agent by setting newrelic.enabled=0. Test here to ensure that it is indeed disabled. - enable the PHP extension zend_test. This can be done by modifying your Dockerfile as seen below. From e559fff21c0a3c9f7bc22425bf40eb1a1daa99ea Mon Sep 17 00:00:00 2001 From: Zach Neumann Date: Tue, 27 Aug 2024 12:42:03 -0600 Subject: [PATCH 4/7] reword --- .../agents/php-agent/troubleshooting/segmentation-faults.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx index 7fb0bade621..cf1f1990540 100644 --- a/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx +++ b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx @@ -108,6 +108,6 @@ zend_test.str_test => no value => no value ## Known reproduction and tracking the fix -As explored in [this issue](https://github.com/newrelic/newrelic-php-agent/issues/865), the Attribute syntax causes seg faults on PHP < 8.2 +Here is a link to issues on php-src that are related to OAPI and segmentation faults: https://github.com/php/php-src/issues?q=is%3Aissue+segfault+observer -Here is [an issue](https://github.com/php/php-src/issues/13817) on the PHP source tracking a fix for PHP 8.3. +We recommend updating to the most recent PHP version to receive the relevant fixes as they are released. From c022eb18b23ac3bb172d5aa4dc5449bef4f7b027 Mon Sep 17 00:00:00 2001 From: Rob Siebens Date: Tue, 27 Aug 2024 11:54:12 -0700 Subject: [PATCH 5/7] fix(PHP agent): Add navigation and make minor formatting changes. --- .../troubleshooting/segmentation-faults.mdx | 38 +++++++++---------- src/nav/apm.yml | 2 + 2 files changed, 21 insertions(+), 19 deletions(-) diff --git a/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx index 7fb0bade621..65eb872a308 100644 --- a/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx +++ b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx @@ -1,27 +1,27 @@ --- -title: Segmentation Faults +title: Segmentation faults type: troubleshooting tags: - Agents - PHP agent - Troubleshooting -metaDescription: Learn what to do if your site seg faults with the New Relic PHP Agent +metaDescription: Learn what to do if your site seg faults with the New Relic PHP agent redirects: - /docs/agents/php-agent/troubleshooting/segmentation-faults -freshnessValidatedDate: never +freshnessValidatedDate: 2023-08-27 --- -## Problem +## Problem [#problem] -Your site generates a segmentation faults when the New Relic PHP Agent (version >= 10.18.0.8) is enabled on PHP 8.0+. +Your site generates a segmentation faults when the New Relic PHP agent (versions 10.18.0.8 or higher) is enabled on PHP 8.0+. -## Description +### Description [#description] -PHP’s Observer API currently has some bugs that can lead to segmentation faults. The New Relic PHP Agent utilizes OAPI for instrumenting your application on PHPs 8.0+ since agent version 10.18. These seg faults only happen when the agent is enabled. This is because PHP’s OAPI codepaths are only executed if there is an observer hooked in, and in all likelihood, the New Relic PHP Agent is the only thing in your environment hooking into OAPI. +PHP’s Observer API currently has some bugs that can lead to segmentation faults. The New Relic PHP agent utilizes OAPI for instrumenting your application on PHPs 8.0+ since agent version 10.18. These seg faults only happen when the agent is enabled. This is because PHP’s OAPI codepaths are only executed if there is an observer hooked in, and in all likelihood, the New Relic PHP agent is the only thing in your environment hooking into OAPI. -### Trace +### Trace [#trace] -In these situations, the stack trace seems to be relatively consistent. What we see is, as always, the NR fatal signal handler at the top. Everything in the stack above ```nr_php_fatal_signal_handler``` will always be present for a segmentation fault when the PHP Agent is installed, as we catch the fault for logging purposes. Then, we see that the signal handler was called during ```zend_observer_fcall_end_all```. This happens when PHP accesses invalid memory when trying to call all registered observers. Any time we see ```zend_observer_*``` in the stack trace without any NR code before the signal handler should raise some eyebrows on whether PHP is the issue +In these situations, the stack trace seems to be relatively consistent. What we see is, as always, the `nr` fatal signal handler at the top. Everything in the stack above `nr_php_fatal_signal_handler` will always be present for a segmentation fault when the PHP agent is installed, as we catch the fault for logging purposes. Then, we see that the signal handler was called during `zend_observer_fcall_end_all`. This happens when PHP accesses invalid memory when trying to call all registered observers. Any time we see ```zend_observer_*``` in the stack trace without any NR code before the signal handler should raise some eyebrows on whether PHP is the issue ``` 0 nr_php_backtrace_get_call_site () at php_stack.c:220 @@ -39,14 +39,14 @@ In these situations, the stack trace seems to be relatively consistent. What we 6 0x00007fa6db081abd in php_request_shutdown () from libphp8.2.so ``` -## zend_test Observer +## `zend_test` observer [#zend-test-observer] -Luckily, we have an easy way to test if the issue is PHP and or the agent. PHP has a built-in extension called zend_test which has a test observer. Enabling this (while disabling the agent), allows the OAPI codepaths to be executed by something other than our agent. Here are the steps to test this: +You can test if the issue is PHP and or the agent. PHP has a built-in extension called `zend_test` which has a test observer. Enabling this (while disabling the agent), allows the OAPI codepaths to be executed by something other than our agent. Here are the steps to test this: -- fully disable the agent by setting newrelic.enabled=0. Test here to ensure that it is indeed disabled. -- enable the PHP extension zend_test. This can be done by modifying your Dockerfile as seen below. -- set the ini settings ```zend_test.observer.enabled=1``` and ```zend_test.observer.observe_all=1```. This should enable PHP's observer API -- see if the issue can be reproduced under this circumstance. +* Fully disable the agent by setting `newrelic.enabled=0`. Test here to ensure that it is indeed disabled. +* Enable the PHP extension `zend_test`. This can be done by modifying your Dockerfile as shown below. +* Set the `ini` settings `zend_test.observer.enabled=1` and `zend_test.observer.observe_all=1`. This should enable PHP's observer API. +* See if the issue can be reproduced under this circumstance. Here is an example for Debian using a PHP docker image: @@ -60,7 +60,7 @@ RUN echo "zend_test.observer.enabled=1" >> $(php-config --ini-dir)/docker-php-ex RUN echo "zend_test.observer.observe_all=1" >> $(php-config --ini-dir)/docker-php-ext-zend_test.ini ``` - and an example using an Alpine PHP docker image: +Here's an example using an Alpine PHP docker image: ``` FROM php:8.0-alpine @@ -72,7 +72,7 @@ RUN echo "zend_test.observer.enabled=1" >> $(php-config --ini-dir)/docker-php-ex RUN echo "zend_test.observer.observe_all=1" >> $(php-config --ini-dir)/docker-php-ext-zend_test.ini ``` -Here is an example from phpinfo() output if the extension is installed and enabled as instructed above: +Here is an example from `phpinfo()` output if the extension is installed and enabled as instructed above: ``` zend_test @@ -106,8 +106,8 @@ zend_test.replace_zend_execute_ex => Off => Off zend_test.str_test => no value => no value ``` -## Known reproduction and tracking the fix +## Known reproduction and tracking the fix [#known-issues] -As explored in [this issue](https://github.com/newrelic/newrelic-php-agent/issues/865), the Attribute syntax causes seg faults on PHP < 8.2 +As explored in [this issue](https://github.com/newrelic/newrelic-php-agent/issues/865), the attribute syntax causes seg faults on PHP version 8.1 or lower. Here is [an issue](https://github.com/php/php-src/issues/13817) on the PHP source tracking a fix for PHP 8.3. diff --git a/src/nav/apm.yml b/src/nav/apm.yml index 5a185fb6e45..4c9ec65dc7b 100644 --- a/src/nav/apm.yml +++ b/src/nav/apm.yml @@ -771,6 +771,8 @@ pages: path: /docs/apm/agents/php-agent/troubleshooting/performance-issues-long-running-task - title: Symfony 4.4 overhead with opcache.preload path: /docs/apm/agents/php-agent/troubleshooting/symfony44-performance + - title: Segmentation faults + path: /docs/apm/agents/php-agent/troubleshooting/segmentation-faults - title: Python monitoring path: /docs/apm/agents/python-agent pages: From f005bf91274aae7cca7115ae17f55e5035731acd Mon Sep 17 00:00:00 2001 From: Rob Siebens Date: Tue, 27 Aug 2024 12:37:49 -0700 Subject: [PATCH 6/7] fix(PHP agent): Fix link and make some minor edits --- .../php-agent/troubleshooting/segmentation-faults.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx index a98981310df..04369621f2a 100644 --- a/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx +++ b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx @@ -21,7 +21,7 @@ PHP’s Observer API currently has some bugs that can lead to segmentation fault ### Trace [#trace] -In these situations, the stack trace seems to be relatively consistent. What we see is, as always, the `nr` fatal signal handler at the top. Everything in the stack above `nr_php_fatal_signal_handler` will always be present for a segmentation fault when the PHP agent is installed, as we catch the fault for logging purposes. Then, we see that the signal handler was called during `zend_observer_fcall_end_all`. This happens when PHP accesses invalid memory when trying to call all registered observers. Any time we see ```zend_observer_*``` in the stack trace without any NR code before the signal handler should raise some eyebrows on whether PHP is the issue +In these situations, the stack trace seems to be relatively consistent. What we see is, as always, the `nr` fatal signal handler at the top. Everything in the stack above `nr_php_fatal_signal_handler` will always be present for a segmentation fault when the PHP agent is installed, as we catch the fault for logging purposes. Then, we see that the signal handler was called during `zend_observer_fcall_end_all`. This happens when PHP accesses invalid memory when trying to call all registered observers. Any time we see `zend_observer_*` in the stack trace without any New Relic code before the signal handler should raise some questions about whether PHP is the issue. ``` 0 nr_php_backtrace_get_call_site () at php_stack.c:220 @@ -41,7 +41,7 @@ In these situations, the stack trace seems to be relatively consistent. What we ## `zend_test` observer [#zend-test-observer] -You can test if the issue is PHP and or the agent. PHP has a built-in extension called `zend_test` which has a test observer. Enabling this (while disabling the agent), allows the OAPI codepaths to be executed by something other than our agent. Here are the steps to test this: +You can test if the issue is PHP or the agent. PHP has a built-in extension called `zend_test` which has a test observer. Enabling this (while disabling the agent), allows the OAPI codepaths to be executed by something other than our agent. Here are the steps to test this: * Fully disable the agent by setting `newrelic.enabled=0`. Test here to ensure that it is indeed disabled. * Enable the PHP extension `zend_test`. This can be done by modifying your Dockerfile as shown below. @@ -108,6 +108,6 @@ zend_test.str_test => no value => no value ## Known reproduction and tracking the fix [#known-issues] -Here is a link to issues on php-src that are related to OAPI and segmentation faults: https://github.com/php/php-src/issues?q=is%3Aissue+segfault+observer +Check out this [php-src page](https://github.com/php/php-src/issues?q=is%3Aissue+segfault+observer) that shows issues related to OAPI and segmentation faults. We recommend updating to the most recent PHP version to receive the relevant fixes as they are released. From a474e15fc1c3b1a87f0e9700183bd4745cc38ff6 Mon Sep 17 00:00:00 2001 From: Rob Siebens Date: Tue, 27 Aug 2024 13:09:38 -0700 Subject: [PATCH 7/7] fix(PHP agent): Fix freshness date year --- .../agents/php-agent/troubleshooting/segmentation-faults.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx index 04369621f2a..30fcfd6b348 100644 --- a/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx +++ b/src/content/docs/apm/agents/php-agent/troubleshooting/segmentation-faults.mdx @@ -8,7 +8,7 @@ tags: metaDescription: Learn what to do if your site seg faults with the New Relic PHP agent redirects: - /docs/agents/php-agent/troubleshooting/segmentation-faults -freshnessValidatedDate: 2023-08-27 +freshnessValidatedDate: 2024-08-27 --- ## Problem [#problem]