[BUG] Failed to attach uprobe at client side

[nalreddy]

Describe the bug
Trying to attach uprobe to application functions. Issue occurred when client trying to instrument application failed to inject or attach probe to certain functions.

Same bpf code able to attach uprobes to appilcation with kernel uprobes.
After adding some debugs ,looks like we are hitting following issue GUM_INSTRUMENTATION_ERROR_WRONG_SIGNATURE.

To Reproduce
Attaching sample programs user space bpf and kernel bpf code.

Application : https://github.com/ls1mardyn/ls1-mardyn

you may need to increase file descriptor limit.(system limits ulimit -n  ). below app with "*" , comes around 3500 functions (return probes 2X that atleast)
bpftime code fix  DEFAULT_MAX_FD = 4024 * 6

How to build :
make;g++ -g bpftracer.cpp -o bpftracer -lbpf -lelf
Make sure bpftime bundled libbpf and system installed libbpf is same (libbpf.so.1.3.0)

How to run :
./bpftracer "absulute path to executable" "function pattern or name"

tracer_code.zip
functionpattern "*" traces all functions in executable ".text" section.

Server :
LD_PRELOAD=build/runtime/syscall-server/libbpftime-syscall-server.so /home/user/bpftracer

Client :
application https://github.com/ls1mardyn/ls1-mardyn ( build clang one )
cd ls1-MarDyn/build/src
LD_PRELOAD=/home/user/bpftime/build/runtime/agent/libbpftime-agent.so ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 50

[nalreddy]

adding sample code for reproducing issue
tracer_code.zip

[Officeyutong]

I'm sorry that i'm a bit confusing. Could please make the following things clearer?

• Which executable are you attaching to? (i.e the one injected by libbpftime-agent.so) How to get this executable?
• What's the behavior of the failure and success?

[nalreddy]

Which executable : https://github.com/ls1mardyn/ls1-mardyn
Build steps mentioned in Readme
cd ls1-mardyn
mkdir build
cd build
CC=clang CXX=clang++ ccmake ..
make -j $(nproc)

How to run : standalone application

    export OMP_NUM_THREADS=4
    cd ls1-mardy/build/src
    ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 10

=====================================================
Success :
With mardyn application and above kernel bpf program mentioned. we are able to attach uprobes.
Please refer above comments for how get bpf program , how to build, and run.

Failure : Tried same app with bpftime.

Server :
LD_PRELOAD=build/runtime/syscall-server/libbpftime-syscall-server.so /home/user/bpftracer

cd ls1-MarDyn/build/src
LD_PRELOAD=/home/user/bpftime/build/runtime/agent/libbpftime-agent.so ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 50

Error : Failed to attach uprobe to some functions ;
error : Failed to execute frida gum_interceptor_attach for function 55b2caf0a3a0

case GUM_INSTRUMENTATION_ERROR_WRONG_SIGNATURE:
result = GUM_ATTACH_WRONG_SIGNATURE;

Please do let me know if you are anymore queries.

[nalreddy]

@Officeyutong please let me know if you need more clarity . I am kind off stuck here , we had to delay integration of bpftime into our project.

bpftime working fine with other application but failing for this application. https://github.com/ls1mardyn/ls1-mardyn

[Officeyutong]

@Officeyutong please let me know if you need more clarity . I am kind off stuck here , we had to delay integration of bpftime into our project.

bpftime working fine with other application but failing for this application. https://github.com/ls1mardyn/ls1-mardyn

Sorry I'm too busy these days, I'll investigate into this issue and give you a reply in two days

[Officeyutong]

Which executable : https://github.com/ls1mardyn/ls1-mardyn Build steps mentioned in Readme cd ls1-mardyn mkdir build cd build CC=clang CXX=clang++ ccmake .. make -j $(nproc)

How to run : standalone application

    export OMP_NUM_THREADS=4
    cd ls1-mardy/build/src
    ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 10

===================================================== Success : With mardyn application and above kernel bpf program mentioned. we are able to attach uprobes. Please refer above comments for how get bpf program , how to build, and run.

Failure : Tried same app with bpftime.

Server : LD_PRELOAD=build/runtime/syscall-server/libbpftime-syscall-server.so /home/user/bpftracer

cd ls1-MarDyn/build/src LD_PRELOAD=/home/user/bpftime/build/runtime/agent/libbpftime-agent.so ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 50

Error : Failed to attach uprobe to some functions ; error : Failed to execute frida gum_interceptor_attach for function 55b2caf0a3a0

case GUM_INSTRUMENTATION_ERROR_WRONG_SIGNATURE: result = GUM_ATTACH_WRONG_SIGNATURE;

Please do let me know if you are anymore queries.

Could you please send me a precompiled (statically linked is better) bpftracer for x86-64? It's difficult to get your source compiled

[nalreddy]

Attaching bpftracer built with following tools
ldd (Ubuntu GLIBC 2.37-0ubuntu2.2) 2.37
gcc version 9.5.0 (Ubuntu 9.5.0-3ubuntu1)
Ubuntu clang version 15.0.7

Need root access and check ulimit limits for that user , may need to increase base on number of uprobes.
by default attaching to all functions in symtab

For attaching normal kernel uprobes
sudo ./bpftracer "path to binary"

bpftracer.zip

[Officeyutong]

Unable to repdocude except limitation of DEFAULT_MAX_FD

• For limitation of DEFAULT_MAX_FD, you can change

  bpftime/runtime/src/handler/handler_manager.hpp

  Line 45 in 87a8901

  const size_t DEFAULT_MAX_FD = 1024 * 6;

  to a larger value and recompile to temporarily resolve it. In a future version, we will support specifying this value at runtime

I didn't encounter any issues related with attach, I have done the following test:

• In terminal 1, run bpftime load ./bpftracer /root/bpftime-312/ls1-mardyn/build/src/MarDyn. bpftracer is the binary you have uploaded
• In terminal 2, run bpftime start ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 50
  After a few seconds, uprobe enter and uprobe return are kept being printed to console infinitely

bpftime is built with the newest commit.

So now I think there might be some issues on the binary you produced. Can you send me a copy of MarDyn , libbpftime-agent.so, libbpftime-syscall-server.so you compiled? (Debug profile is preferred)

[nalreddy]

I could attach uprobes for default binary and working fine. I will check if there are any issues with generated MarDyn binary.
Thanks for looking into it and quick response.

sorry for deviating from main question :

Does bpftime support attaching multiple pids (of running processes)

like example
sudo bpftime attach 101771,101772,...

https://github.com/eunomia-bpf/bpftime (Quick start section)
You can also dynamically attach the eBPF program with a running process:
Does with work with multiple pids, how do we provide

[nalreddy]

Hi @Officeyutong . I managed to create small reproducer for the issue.

Issue happening with AMD compiler(AOCC) you can download from here: https://www.amd.com/en/developer/aocc.html
Frida seems to inject probes with wrong function address. AOCC compiler is llvm based.

Same application compiled with GCC and Clang working fine with bpftime.

Reproduced with :
Compile below application or any small program with function with AOCC comipler (llvm based compiler).
Try to attach function.

Bpf client side error :

2024-08-27 05:00:43][error][1401954] Failed to execute frida gum_interceptor_attach for function 555da3ee37b0
[2024-08-27 05:00:43][error][1401954] Unable to instantiate handlers: Failed to attach uprobe/uretprobe

Bpf Server side : Success

2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0
[2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0

Process maps :

Maps of app
555da3ee3000-555da3ee4000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits (555da3ee37b0 address in bpftime falls in non-executable region ?? )
555da3ee4000-555da3ee5000 r-xp 00000000 08:04 28574075 /home/user/data/expr/probe_limits ( function address supposed to fall in this region ?)
555da3ee5000-555da3ee6000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits
555da3ee6000-555da3ee7000 rw-p 00000000 08:04 28574075 /home/user/data/expr/probe_limits
555da3ee7000-555da3ee8000 r--p 00000000 00:00 0
555da3ee8000-555da3eef000 rwxp 00000000 00:00 0
555da5bb3000-555da5bf5000 rw-p 00000000 00:00 0

Failed function address 555da3ee37b0, falling outside execution mapped region of process.

Server side attached using proper file offset
objdump -D -F -C binary

00000000000017b0 <op_0> (File Offset: 0x7b0): we are u
17b0: c3 ret
17b1: 66 66 66 66 66 66 2e data16 data16 data16 data16 data16 cs nopw 0x0(%rax,%rax,1)
17b8: 0f 1f 84 00 00 00 00
17bf: 00

Application

#include <stdio.h>
#include <unistd.h>

void attribute ((noinline)) op_0()
{
for (int x = 0; x < 10; x++) { int sum = x+ 100; }
return;
}

int main()t
{
printf("test limits pid : %d\n",getpid());
getchar();
return 0;
}

[Officeyutong]

I could attach uprobes for default binary and working fine. I will check if there are any issues with generated MarDyn binary. Thanks for looking into it and quick response.

sorry for deviating from main question :

Does bpftime support attaching multiple pids (of running processes)

like example sudo bpftime attach 101771,101772,...

https://github.com/eunomia-bpf/bpftime (Quick start section) You can also dynamically attach the eBPF program with a running process: Does with work with multiple pids, how do we provide

Just attach them seperately, call bpftime attach 101771 and bpftime attach 101772 and so on. Currently bpftime attach can only attach one pid at a time

[Officeyutong]

Hi @Officeyutong . I managed to create small reproducer for the issue.

Issue happening with AMD compiler(AOCC) you can download from here: https://www.amd.com/en/developer/aocc.html Frida seems to inject probes with wrong function address. AOCC compiler is llvm based.

Same application compiled with GCC and Clang working fine with bpftime.

Reproduced with : Compile below application or any small program with function with AOCC comipler (llvm based compiler). Try to attach function.

Bpf client side error :

2024-08-27 05:00:43][error][1401954] Failed to execute frida gum_interceptor_attach for function 555da3ee37b0 [2024-08-27 05:00:43][error][1401954] Unable to instantiate handlers: Failed to attach uprobe/uretprobe

Bpf Server side : Success

2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0 [2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0

Process maps :

Maps of app 555da3ee3000-555da3ee4000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits (555da3ee37b0 address in bpftime falls in non-executable region ?? ) 555da3ee4000-555da3ee5000 r-xp 00000000 08:04 28574075 /home/user/data/expr/probe_limits ( function address supposed to fall in this region ?) 555da3ee5000-555da3ee6000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee6000-555da3ee7000 rw-p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee7000-555da3ee8000 r--p 00000000 00:00 0 555da3ee8000-555da3eef000 rwxp 00000000 00:00 0 555da5bb3000-555da5bf5000 rw-p 00000000 00:00 0

Failed function address 555da3ee37b0, falling outside execution mapped region of process.

Server side attached using proper file offset objdump -D -F -C binary

00000000000017b0 <op_0> (File Offset: 0x7b0): we are u 17b0: c3 ret 17b1: 66 66 66 66 66 66 2e data16 data16 data16 data16 data16 cs nopw 0x0(%rax,%rax,1) 17b8: 0f 1f 84 00 00 00 00 17bf: 00

Application

#include <stdio.h> #include <unistd.h>

void attribute ((noinline)) op_0() { for (int x = 0; x < 10; x++) { int sum = x+ 100; } return; }

int main()t { printf("test limits pid : %d\n",getpid()); getchar(); return 0; }

Will AOCC work on an Intel CPU? I don't have any AMD cpu machine now

[nalreddy]

AOCC should work with intel CPU, its same x86 . Except for some special optimization flags.

above application compiled with aocc/clang application.c -o application

Make sure its aocc/clange not system default clang . Please do "which clang" and Give absolute path of AOCC compiler.

[nalreddy]

Hi @Officeyutong . I managed to create small reproducer for the issue.
Issue happening with AMD compiler(AOCC) you can download from here: https://www.amd.com/en/developer/aocc.html Frida seems to inject probes with wrong function address. AOCC compiler is llvm based.
Same application compiled with GCC and Clang working fine with bpftime.
Reproduced with : Compile below application or any small program with function with AOCC comipler (llvm based compiler). Try to attach function.
Bpf client side error :
2024-08-27 05:00:43][error][1401954] Failed to execute frida gum_interceptor_attach for function 555da3ee37b0 [2024-08-27 05:00:43][error][1401954] Unable to instantiate handlers: Failed to attach uprobe/uretprobe
Bpf Server side : Success
2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0 [2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0
Process maps :
Maps of app 555da3ee3000-555da3ee4000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits (555da3ee37b0 address in bpftime falls in non-executable region ?? ) 555da3ee4000-555da3ee5000 r-xp 00000000 08:04 28574075 /home/user/data/expr/probe_limits ( function address supposed to fall in this region ?) 555da3ee5000-555da3ee6000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee6000-555da3ee7000 rw-p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee7000-555da3ee8000 r--p 00000000 00:00 0 555da3ee8000-555da3eef000 rwxp 00000000 00:00 0 555da5bb3000-555da5bf5000 rw-p 00000000 00:00 0
Failed function address 555da3ee37b0, falling outside execution mapped region of process.
Server side attached using proper file offset objdump -D -F -C binary
00000000000017b0 <op_0> (File Offset: 0x7b0): we are u 17b0: c3 ret 17b1: 66 66 66 66 66 66 2e data16 data16 data16 data16 data16 cs nopw 0x0(%rax,%rax,1) 17b8: 0f 1f 84 00 00 00 00 17bf: 00
Application
#include <stdio.h> #include <unistd.h>
void attribute ((noinline)) op_0() { for (int x = 0; x < 10; x++) { int sum = x+ 100; } return; }
int main()t { printf("test limits pid : %d\n",getpid()); getchar(); return 0; }

Will AOCC work on an Intel CPU? I don't have any AMD cpu machine now

"yes, AOCC should work with intel CPU, its same x86 . Except for some special optimization flags.

above application compiled with aocc/clang application.c -o application

Make sure its aocc/clange not system default clang . Please do "which clang" and Give absolute path of AOCC compiler."

[nalreddy]

@Officeyutong
Our current product ,bpf programs run without sudo ( we set bpf_cap to executable before running the bpf program).

How do we achieve same with bpftime .

Are there any places we have to run bpftime with sudo (must ?? )
from : https://eunomia.dev/bpftime/documents/usage/

Modes of running bpftime

1. ./example/malloc/victim & echo $! # The pid is 101771

attach to it:

$ sudo bpftime attach 101771 # You may need to run make install in root
Inject: "/root/.bpftime/libbpftime-agent.so"
Successfully injected. ID: 1

2. bpftime load ./example/malloc/malloc
   bpftime start ./example/malloc/victim

Run with LD_PRELOAD directly

Why do we need to use sudo in 1st method , is it necessary and
also System call tracing examples are run with Sudo , is it must.

Could you please let me know which cases we need to use sudo and which use case not.

We would like to avoid use of Sudo along with bpf programs. (run bpf program as non root).

we would like to use attach method without sudo , is it possible.