1502377 - Excessive overhead from the Gecko Profiler on Linux

Status: NEW

(Core :: Gecko Profiler, enhancement, P3)

Description

[Randell Jesup [:jesup] (needinfo me)]

Attachment: webtooling performance with profiler, various settings, on dual-XEON

When profiling the Web Tooling benchmark, I noticed (again) that the Gecko Profiler can have a problematically large impact on performance.

To summarize, benchmark scores (basically all JS/GC/etc code) drop 20% with default options, 40% if I also capture "JS" threads (JS Helper, JS Watchdog).  If we try to capture all threads, there's a much bigger hit on performance.

This is very problematical, since large effects on performance from the profiler itself can warp the captured profiles in unexpected ways.  If everything was equally slower, this may not matter, but it's not - network data will appear to be 'faster'; non-sampled threads will run faster than sampled ones (and since we suspend-and-sample one thread at a time, during that sampling time other threads effectively get "free" time to do stuff).  Ordering of things can change. Etc, etc 

Any profiler that disturbs the thing it's measuring to this extent makes it hard to consider the captured profiles reliable.


We need to reduce the overhead for the profiler, and also consider if there are ways to account for the impacts, or compensate for them - perhaps by pausing all threads while sampling, for example, and sampling multiple threads in parallel.

There also seem to be issues around how the buffer is handled; with buffer expansions eating significant time during capture.  This might also account for some of the sampling gaps we see.

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

Perf profile of part of a Web Tooling run with the profiler on (1ms, with JS threads).  Note that the Sampling thread is using a lot of time - and that almost 1/3 of the CPU used by the thread was in DoSharedSample's memset's and memmoves for (apparently, hidden due to inlining) to expand and copy the profiler buffer.

We should allocate the profiler buffer at the configured size when the profiler is turned on.

http://bit.ly/2ORFgNk