Lies, big fat lies and benchmarks

April 7, 2010 at 11:14 pm 5 comments

There is the old saying, ‘there are lies, big fat lies and there are benchmarks’, but after my previous post about the jsanim, it was time to try to measure how good or bad the javascript based animations are when comparing it against Plasma::Animations (written in C++).

== what to measure? ==

The first task was to have a meaningful number and frames per second (FPS) sounds like a good idea. The only issue is that QGraphicsView will only update parts in the scene that need redrawing (besides it seems that there is not an easy way to access the draw events without subclassing). My solution was to set a counter for the Animation::updateCurrentTime() method that is called while the animation is running, given that the documentation says that “… neither the interval between calls nor the number of calls to this function are defined; though, it will normally be 60 updates per second.”. Maybe I should call it Function calls Per Second (FCPS). 🙂

Another important factor that can impact hugely is this kind of benchmark is both the hardware as also the graphics system used by Qt (and finally if the window manager has composite effect turned on). My test computer is a macbook (version 4.1), Intel Core 2 Duo (T8300@2.4Ghz), video Intel Mobile GM965/GL960), Qt 4.7 from master repository and I run the tests with composite disabled and got this results.

As the test application, I used the code available in playground/plasma/libs/jsanim, where a set of QGraphicsWidgets (each one drawing a pixmap) will run an animation group where 2 animations (zoom and fade) run in parallel followed next by a rotation.

== equal or not? ==

Next task is to extract any meaning from the results and I remembered what an old statistician professor that I had used to joke “Torture the data enough and it will confess”. I used to be good on this subject (back in the day I wrote some Bootstrap simulations and model fitting for biological data as also geostatistics  studies using R), but it is being 5 years since I last have done any statistical analysis, I was worried about making some mistakes while using more advanced methods.

So I went with the lazier approach and decided to use a boxplot, following bellow:

The js based animations seems to have a somewhat bigger variance (see second boxplot) but overall you can’t really tell if there is a difference between them (just in case, the p-values for t-student means tests was for each animation respectively: 0.54, 0.43, 0.86).

So… what about mobile? I did the same tests in a Nokia N900 (using the opengl and raster engines) and got this data, which yielded the following boxplot:

this time without the js variance and again without any sensible difference between js and C++ animations (p-values: 0.42, 0.16, 1).

== wrapping up ==

From the data (and after asking 4 different people to inspect visually how the animations were running in js X C++ apps), I can say that following the path of using javascript to code the logic of animations in Qt seems to be a valid approach with no sensible difference in performance. Kudos to QScriptEngine (and Qt development frameworks) for making it perform really fast.

There are indeed some issues in this study, following:

  • CPU and memory usage: I measure it informally using ‘top’ and they seems to be on the same level, but a more formal benchmarking ideally should measure this variables;
  • samples: I was a bit in a hurry and run each animation only 3 times (ideally a bigger sample like 20 should be used).

Entry filed under: KDE, mobile, Qt.

jsanim: current status 6 months of plasma in 10 minutes

5 Comments Add your own

  • 1. simon  |  April 8, 2010 at 8:19 am

    how about putting labels to the graphs that indicate wether big is good or bad?

  • 2. maninalift  |  April 8, 2010 at 8:20 am

    (1) Surely all these results tell you is that the tests did not stretch the machine enough to slow the frame rate – the frame rate was fixed by the screen refresh rate.

    (2) If I understand correctly, each box on your box-plot represents only three animation runs. In this case I don’t think you can make anything of the difference in variance.

  • 3. savago  |  April 8, 2010 at 2:40 pm

    1) Actually it shows the result of 6 executions using 2 different rendering engines (3 samples each). I guess it stretched the machine enough, as you can see by looking at the data, the 3rd animation FPS drops to 37-40 when using xrender in the macbook and 8-11 in the N900 (look at lower bar in the third boxplot).

    If I run each animation solo, it will have a 60-65 FPS.

    2) Nopes, is actually total 6 samples (3 samples * 2 rendering systems).

    Best regards


  • 4. The User  |  April 9, 2010 at 11:09 pm

    I still hate JS-Syntax… I hate all this focussing on JS.

    • 5. The Developer  |  April 14, 2010 at 2:07 pm

      You’re just a troll, dude. Doesn’t matter if it’s JS or pure C++, if it’s more convenient for you, then use it, otherwise DON’T!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Trackback this post  |  Subscribe to the comments via RSS Feed


April 2010
« Mar   May »

Most Recent Posts

%d bloggers like this: