Generating mappings for transient java processes / agentlib mode

[craiig]

Hi all,
I'm trying to profile systems that consist of lots of java processes being started and stopped over time. Does anyone know of a way to ensure that I can get a symbol mapping of each java process that gets run on my system? Just running once won't capture these transient processes, so I'd ideally like something that automatically makes a symbol dump, maybe on process termination? Any thoughts would be appreciate.

Not sure if this is the right place to ask, feel free to point me somewhere else!
Thanks a lot for this tool.

[jrudolph]

The first question would be whether those JVM processes are running long enough to have sufficient code compiled at all. Otherwise, you'd just get a lot of opaque Interpreter frames.

The actual functionality would probably need the perf-map-agent library to be loaded at the start of each of the Java processes to capture all JIT compilation from the start of the program till the end. The problem then is that it is likely that the JVM discards some compiled methods and reuses the same memory for new methods. That means that over the run of a Java process, a memory address may map to different symbols changing over time. So, you will need a dynamic symbol mapping mechanism that associates timestamps from perf traces with timestamps from the JVM compilation events.

To get there we'd need:

• to resurrect the persistent mode for the agent
• to output load/unload events with a timestamp that can be correlated with timestamps collected with perf
• a custom symbol mapping step that doesn't rely on perf-tools and perf-.map files but directly maps (timestamp, address) tuples to the correct symbol at that time

All of these would be interesting to implement but not a short-term goal for now.

[craiig]

Thanks for the response! I'm motivated to get something like this working, since full system stack trace profilers that can examine transient JITs seem rare (or nonexistent?), but I'd like something like this for my research.

What do you think it would take to resurrect the persistent mode for the agent? I'm new to the JVMTI, but it looks like the agent would just need to implement Agent_OnLoad to register the same callbacks at startup, and just not close the the perf-map file, right? In addition to the right JAVA_TOOL_OPTIONS to get the agent loaded.

The load/unload events you're referring to are CompiledMethod(Load / Unload)?

[jrudolph]

Relevant new perf features:

https://lkml.org/lkml/2015/3/30/749
https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/jitdump-specification.txt

[libbkmz]

This is old thread, but to get mappings after java process termination you can use -XX:+DumpPerfMapAtExit flag for that

[jrudolph]

Cool, thanks for sharing!