1331578 - stylo: Stylo is slower than Gecko on etc/ci/performance TP5 tests

Status: RESOLVED WORKSFORME

(Core :: CSS Parsing and Computation, defect, P1)

Description

[Shing Lyu [:shinglyu]]

I ran the performance test in Servo's etc/ci/performance on TP5 test suite. Here are the result for comparing Stylo (release build) with Gecko 52

=================
 595.0  774.0 +179.0  +30.08%   http://localhost:8000/page_load_test/tp5n/globo.com/www.globo.com/index.html.
 360.0  792.0 +432.0 +120.00%   http://localhost:8000/page_load_test/tp5n/56.com/www.56.com/index.html.
 287.0  524.0 +237.0  +82.58%   http://localhost:8000/page_load_test/tp5n/rakuten.co.jp/www.rakuten.co.jp/index.html.
 135.0  228.0  +93.0  +68.89%   http://localhost:8000/page_load_test/tp5n/chemistry.about.com/chemistry.about.com/index.html.
 293.0  397.0 +104.0  +35.49%   http://localhost:8000/page_load_test/tp5n/huffingtonpost.com/www.huffingtonpost.com/index.html.
 203.0  275.0  +72.0  +35.47%   http://localhost:8000/page_load_test/tp5n/imgur.com/imgur.com/gallery/index.html.
 135.0  212.0  +77.0  +57.04%   http://localhost:8000/page_load_test/tp5n/ezinearticles.com/ezinearticles.com/index.html@Migraine-Ocular---The-Eye-Migraines&id=4684133.html.
 259.0  661.0 +402.0 +155.21%   http://localhost:8000/page_load_test/tp5n/myspace.com/www.myspace.com/albumart.html.
 160.0  310.0 +150.0  +93.75%   http://localhost:8000/page_load_test/tp5n/mail.ru/mail.ru/index.html.
 230.0  345.0 +115.0  +50.00%   http://localhost:8000/page_load_test/tp5n/tudou.com/www.tudou.com/index.html.
  91.0  198.0 +107.0 +117.58%   http://localhost:8000/page_load_test/tp5n/goo.ne.jp/goo.ne.jp/index.html.
 198.0  353.0 +155.0  +78.28%   http://localhost:8000/page_load_test/tp5n/bbc.co.uk/www.bbc.co.uk/news/index.html.
 381.0  564.0 +183.0  +48.03%   http://localhost:8000/page_load_test/tp5n/youku.com/www.youku.com/index.html.
 104.0  150.0  +46.0  +44.23%   http://localhost:8000/page_load_test/tp5n/w3.org/www.w3.org/standards/webdesign/htmlcss.html.
 136.0  218.0  +82.0  +60.29%   http://localhost:8000/page_load_test/tp5n/beatonna.livejournal.com/beatonna.livejournal.com/index.html.
 170.0  362.0 +192.0 +112.94%   http://localhost:8000/page_load_test/tp5n/dailymotion.com/www.dailymotion.com/us.html.
 190.0  307.0 +117.0  +61.58%   http://localhost:8000/page_load_test/tp5n/cgi.ebay.com/cgi.ebay.com/ALL-NEW-KINDLE-3-eBOOK-WIRELESS-READING-DEVICE-W-WIFI-/130496077314@pt=LH_DefaultDomain_0&hash=item1e622c1e02.html.
 172.0  281.0 +109.0  +63.37%   http://localhost:8000/page_load_test/tp5n/reddit.com/www.reddit.com/index.html.
 231.0  362.0 +131.0  +56.71%   http://localhost:8000/page_load_test/tp5n/stackoverflow.com/stackoverflow.com/questions/184618/what-is-the-best-comment-in-source-code-you-have-ever-encountered.html.
 455.0  599.0 +144.0  +31.65%   http://localhost:8000/page_load_test/tp5n/dailymail.co.uk/www.dailymail.co.uk/ushome/index.html.
 249.0  373.0 +124.0  +49.80%   http://localhost:8000/page_load_test/tp5n/yelp.com/www.yelp.com/biz/alexanders-steakhouse-cupertino.html.
 143.0  280.0 +137.0  +95.80%   http://localhost:8000/page_load_test/tp5n/youtube.com/www.youtube.com/music.html.
 407.0  647.0 +240.0  +58.97%   http://localhost:8000/page_load_test/tp5n/amazon.com/www.amazon.com/Kindle-Wireless-Reader-Wifi-Graphite/dp/B002Y27P3M/507846.html.
  56.0   75.0  +19.0  +33.93%   http://localhost:8000/page_load_test/tp5n/google.com/www.google.com/search@q=mozilla.html.
 214.0  444.0 +230.0 +107.48%   http://localhost:8000/page_load_test/tp5n/spiegel.de/www.spiegel.de/index.html.
 173.0  283.0 +110.0  +63.58%   http://localhost:8000/page_load_test/tp5n/page.renren.com/page.renren.com/index.html.
======================

The columns means (from left to right)
* Gecko page load time
* Stylo page load time
* (Stylo - Gecko) page load time difference
* page load time difference in percentage
* Test case

All the page load time for Stylo is slower then Gecko, ranging from 30% to 240% slower.

You can find instructions on how to run this yourself here: https://github.com/servo/servo/tree/master/etc/ci/performance

Comment 1

[Shing Lyu [:shinglyu]]

cc Chris

Comment 2

[Emilio Cobos Álvarez [:emilio]]

Yeah, this wouldn't surprise me a lot right now. We'd be faster in a full restyle I think, but I suspect these results may mean one of two things:

 * That our incremental restyle isn't quite there yet (and that's known).
 * That our CSS parser is slower (see https://github.com/servo/servo/issues/13778) for an example.

CCing Bobby and Simon for that.

Shing, could you run an instrumented build against these test cases to see how many elements are we restyling? That'd mean adding a counter in recalc_style_at in Servo, and in ElementRestyler::Restyle in Gecko, and check the difference.

It'd be nice to know if the page load time drop from stylo comes from CSS parsing, or restyling (or both).

Comment 3

[Chris Peterson [:cpeterson]]

Shing, great work getting Talos running with Stylo!

Blocking the stylo-nightly milestone bug 1330412 because we don't want to enable Stylo in Nightly until it is faster than Gecko.

Comment 4

[Shing Lyu [:shinglyu]]

Emilio: Is the instrumented build code available? Or are you suggesting me writing one myself?

Chris: It's not Talos yet, I'm using Servo's performance test, but running the same test case as Talos.

Comment 5

[Bobby Holley (:bholley)]

(In reply to Shing Lyu [:shinglyu] from comment #4)
> Emilio: Is the instrumented build code available? Or are you suggesting me
> writing one myself?

I think he may have been suggesting the latter, but it might be a bit involved. I filed bug 1331856 to do this.

Comment 6

[Shing Lyu [:shinglyu]]

Some new results, to summarize: 

1. Stylo (parallel, 6 threads) is slightly slow than Gecko in general, but we do have some big wins like amazon.com & hoffintonpost.com. On the other hand, some sites are extremely slow Stylo like myspace and bild.de

2. Parallel stylo is not faster than sequential stylo. (see below.) The difference is around -5% to 5%, centered around 0%. This might indicate a big risk in our claim that parallelism is useful for the style system. 
 

Gecko vs Stylo (parallel)

test                     gecko     stylo	difference
----------------------------------------------------------------
tudou.com                315.36    402.52	-27.64%
youtube.com              267.44    308.92	-15.51%
sohu.com                 462.68    506.72	-9.52%
store.apple.com          354.28    509.2	-43.73%
people.com.cn            380.96    410.08	-7.64%
rakuten.co.jp            440.92    539.76	-22.42%
myspace.com              447.4     700.92	-56.67%
stackoverflow.com        403.92    475.68	-17.77%
ifeng.com                494.32    486.68	1.55%
cgi.ebay.com             301.12    329.44	-9.40%
ezinearticles.com        209.64    214.6	-2.37%
thepiratebay.org         298.8     323.32	-8.21%
media.photobucket.com    221.64    245.76	-10.88%
digg.com                 367.64    421.28	-14.59%
huffingtonpost.com       586.4     505.36	13.82%
noimpactman.typepad.com  328.8     319.2	2.92%
163.com                  609.32    673.36	-10.51%
dailymotion.com          304.84    385.84	-26.57%
chinaz.com               278.16    305.16	-9.71%
w3.org                   179.6     192.52	-7.19%
chemistry.about.com      183.52    205.64	-12.05%
mashable.com             575.52    649.84	-12.91%
aljazeera.net            462.64    465.44	-0.61%
youku.com                475.4     506.36	-6.51%
reuters.com              444.76    477.36	-7.33%
reddit.com               286.68    311.88	-8.79%
dailymail.co.uk          579.56    592.48	-2.23%
uol.com.br               455.08    486.32	-6.86%
naver.com                414.52    462.16	-11.49%
page.renren.com          251.64    267.76	-6.41%
xinhuanet.com            826.96    966.04	-16.82%
cnn.com                  432.28    495.96	-14.73%
amazon.com               813.76    652.44	19.82%
slideshare.net           321.88    373.72	-16.11%
56.com                   601.2     680.52	-13.19%
imgur.com                254.4     268.12	-5.39%
wsj.com                  496.04    601.92	-21.35%
bbc.co.uk                312.96    290.28	7.25%
guardian.co.uk           365.96    399.12	-9.06%
beatonna.livejournal.com 158.24    163.52	-3.34%
globo.com                492.56    678.64	-37.78%
google.com               129.92    136.0	-4.68%
xunlei.com               773.4     860.76	-11.30%
mail.ru                  349.24    422.04	-20.85%
homeway.com.cn           436.16    517.52	-18.65%
bild.de                  1042.68   1470.52	-41.03%
spiegel.de               397.92    431.92	-8.54%
imdb.com                 413.68    439.2	-6.17%
goo.ne.jp                156.88    178.56	-13.82%
indiatimes.com           433.2     466.36	-7.65%
yelp.com                 501.36    522.6	-4.24%


Sequential Stylo vs Parallel Stylo (6 threads)

test                     sequential parallel	difference
----------------------------------------------------------------------
tudou.com                424.6     402.52	5.20%
youtube.com              320.28    308.92	3.55%
sohu.com                 514.12    506.72	1.44%
store.apple.com          510.28    509.2	0.21%
people.com.cn            408.52    410.08	-0.38%
rakuten.co.jp            511.32    539.76	-5.56%
myspace.com              673.28    700.92	-4.11%
stackoverflow.com        500.6     475.68	4.98%
ifeng.com                496.2     486.68	1.92%
cgi.ebay.com             318.36    329.44	-3.48%
ezinearticles.com        204.44    214.6	-4.97%
thepiratebay.org         339.24    323.32	4.69%
media.photobucket.com    226.2     245.76	-8.65%
digg.com                 391.04    421.28	-7.73%
huffingtonpost.com       531.44    505.36	4.91%
noimpactman.typepad.com  306.84    319.2	-4.03%
163.com                  636.52    673.36	-5.79%
dailymotion.com          378.4     385.84	-1.97%
chinaz.com               293.24    305.16	-4.06%
w3.org                   175.96    192.52	-9.41%
chemistry.about.com      195.32    205.64	-5.28%
mashable.com             634.48    649.84	-2.42%
aljazeera.net            452.84    465.44	-2.78%
youku.com                537.72    506.36	5.83%
reuters.com              461.76    477.36	-3.38%
reddit.com               312.36    311.88	0.15%
dailymail.co.uk          639.04    592.48	7.29%
uol.com.br               474.48    486.32	-2.50%
naver.com                439.12    462.16	-5.25%
page.renren.com          261.28    267.76	-2.48%
xinhuanet.com            940.16    966.04	-2.75%
cnn.com                  483.56    495.96	-2.56%
amazon.com               677.68    652.44	3.72%
slideshare.net           356.6     373.72	-4.80%
56.com                   665.84    680.52	-2.20%
imgur.com                251.76    268.12	-6.50%
wsj.com                  609.56    601.92	1.25%
bbc.co.uk                278.64    290.28	-4.18%
guardian.co.uk           398.4     399.12	-0.18%
beatonna.livejournal.com 161.84    163.52	-1.04%
globo.com                701.16    678.64	3.21%
google.com               130.08    136.0	-4.55%
xunlei.com               803.32    860.76	-7.15%
mail.ru                  412.08    422.04	-2.42%
homeway.com.cn           509.76    517.52	-1.52%
bild.de                  1613.04   1470.52	8.84%
spiegel.de               436.52    431.92	1.05%
imdb.com                 416.88    439.2	-5.35%
goo.ne.jp                167.12    178.56	-6.85%
indiatimes.com           443.96    466.36	-5.05%
yelp.com                 533.56    522.6	2.05%

Comment 7

[Shing Lyu [:shinglyu]]

Attachment: Histogram of performance improvements

The blue one is Stylo vs Gecko comparison. The orange is parallel stylo vs sequential stylo.

Comment 8

[Bobby Holley (:bholley)]

Looked at this a bit last week, there's a bunch of dumb stuff we're doing. We haven't spent much time optimizing the actual styling yet, which I plan to do over the next couple of weeks. We should watch what that does to the numbers here, since I expect them to improve considerably.

Comment 9

[Bobby Holley (:bholley)]

The graphs are pretty much all at parity at this point, mostly because it turns out that the 2011-era sites don't have that much styling in them. We'll keep an eye on these numbers, but this bug as-filed is fixed.