[LLVMdev] Using the test suite to benchmark patches
Tanya M. Lattner
tonic at nondot.org
Wed May 21 13:16:30 CDT 2008
> just a quick email. I've been working on a patch to simplifycfg last week and
> want to test its performance. I've ran the test-suite succesfully, both with
> the patched and unpatched versions. However, I could find no easy way to
> compare both results. I see that the web pages of the nightly tester provide
> nice results (changes compared to the day before, together with percentages
> and colors etc). Something like that should be supported for two local test
> runs as well, but I couldn't find how.
Currently, the nightly tester scripts only compare current day to previous
and do not have the ability to compare between two test runs that are
more than 1 day/run apart. There is a GSOC student who will be working to
improve this and add this feature.
However, the nightly tester scripts are PHP based and require the results
to be in a database. You could just send them to the LLVM server to do
your comparisions if you wanted. Another option is to do a local set up,
but this is more work and really only necessary if you have results that
you don't want out in the public.
> I did a bit of hacking on the HTMLColDiff.pl script that I found lying around,
> the (rough) patch is attached. Is this script a quick hackup that got
> forgotten, or is it still used by people for a purpose that I don't see right
I do not know if people are using this frequently. If it doesn't work, I
am sure the answer is no. :) It was probably used before the nightly
tester was around.
> Any thoughts or suggestions on how to do this testing in a structured manner?
> In particular, it would be useful to have some means of running a test a few
> times and taking mean values for comparison or something...
I would look at the TEST.*.Makefile and TEST.*.report for a way to do your
multiple run testing. You still have the problem with comparing 2
different sets of results though.. and that would require a new script.
Of course, someone else may have a better suggestion.
More information about the LLVMdev