Fine grained CPU profiling

[sandreim]

There have been efforts to solve this earlier like paritytech/polkadot#4871 but the results we got were not providing enough insights due to the low sampling rate/storage - https://pyroscope.io/docs/storage-design/ . We should continue the effort to implement something that works better for our usecase. We need as fine grained as possible CPU profiling (100us) with visualisation tooling to increase accuracy and decrease the scope of debugging when dealing with node performance issues or optimization work.

The solution should also consider this must also work easily with Zombienet to test performance regression in the CI pipeline.

[ordian]

Just to clarity, the profiler frequency used in pyroscope is configurable. The problem is that it accumulates the profiling info into segments of 10s, which is hardcoded in their server in many places. Cf grafana/pyroscope#901.
Thus the critical section we want to profile gets lost in the noise of other tasks such as networking.

Also worth mentioning that we don't need to run it on every node. One validator and one collator per parachain would be fine.

We could try another tool in the same category of continuous profilers or try to fork their server.

[alindima]

After my latest experiences re-enabling pyroscope and pprof-rs, I can confirm that we need a different solution because:

• pprof-rs has a high overhead, since it's doing profiling being based on signal handlers (and pyroscope unregisters/registers the signal handlers every 10 seconds).
• libunwind mechanism is buggy and causes SIGABORTs: SIGABORT when profiling with pyroscope-rs tikv/pprof-rs#219
• pyroscope-rs repo does not seem under a lot of active maintenance. I also discovered a bug there: fix spurious exit when epoll_wait is interrupted by a signal grafana/pyroscope-rs#125
• even with frame pointer unwinding and the above fix, after a couple of hours of running the validator with pyroscope enabled, most of the nodes tasks are killed, including the pyroscope agent (I assume there's a memory leak that triggers an OOM somewhere), which leaves the node in an inconsistent but still running state.

Ideally, we'd want to use perf on linux to profile, since it's very mature and flexible. However, I don't know how feasible it is considering that I couldn't find a continuous profiler with flamegraph visualisations that uses perf as a data source.

And there is also the storage problem. Profiling spits out a ton of data. If the frequency is too small, the data is not representative. If the frequency is too high, it can overwhelm the server and become unmanageable.

[petethepig]

@alindima FWIW it's possible to send perf or eBPF profling data to pyroscope in collapsed (or folded) format, we have documentation on how to upload that data to Pyroscope using an HTTP API here.

[alexggh]

Superseeded by using subsystem benchmarks with pyroscope.