[LLVMdev] QMTest vs. Dejagnu
vadve at cs.uiuc.edu
Sun Nov 28 12:09:38 CST 2004
dejagnu sounds like the better choice to me too. I recommend keeping
qmtest for this release and then removing it for 1.5.
On Nov 27, 2004, at 3:24 PM, Tanya Lattner wrote:
> I've finished adding the -rundejagnu option to the nightly tester
> which was the last step to fully support Dejagnu. I think now is the
> appropriate time to discuss keeping QMTest or switching to Dejagnu. A
> of work went into using QMTest, so I think we should make this decision
> carefully and before the 1.4 release.
> Here are the pros and cons in my eyes, please feel free to add your own
> (Note: RUN lines are basically sequences of commands that are put into
> bash script and executed. Here is an example: RUN: llvm-as < %s |
> | llvm-as)
> Pros of Dejagnu:
> 1) Adding a test is very quick and easy. In most cases it requires
> just adding a RUN line to the file. No gui needed.
> 2) XFAILing a test is also quick and easy. You just add a XFAIL
> line to a single test with a * for all platforms, or specify part
> of a target triplet to match. No gui needed.
> 3) Because you add the RUN and XFAIL lines to each individual
> test, it's all located in one file. This makes it very easy
> understand how the test is run and easy to change.
> 4) You can run just a single sub directory of tests by specifying
> that directory as the TESTSUITE. (ex: make check-dejagnu
> 5) All intermediate output is put into an Output directory and
> easily cleaned up w/ proper Makefile support.
> 6) It is faster then QMTest (54% faster on the sparc, 20% faster
> on ppc)
> 7) The output logs are easy to parse.
> 8) You can easily search for XFAIL tests.
> Cons of Dejagnu:
> 1) There must be a dg.exp file in each subdirectory to run tests
> in that directory. (easily copied though)
> Pros of QMTest:
> 1) Can add names to subdirectories to create testsuites (kinda
> like a shortcut).
> 2) Has a gui (some prefer this).
> Cons of QMTest:
> 1) You have to use the gui to add directories.
> 2) You have to use the gui to XFAIL a test.
> 3) It uses something called expectation files that you must load
> to view which tests XFAIL. There is no way (that I have found) to
> get a complete list of XFAILs..
> 4) It is also hard to XFAIL across platforms, because it requires
> hacking an expectation file for each
> target, which must be done with the gui.
> 5) Intermediate output placement can not be controlled.
> 6) The output logs are not as clean.
> 7) Right now we are dependent on a specific version of QMTest.
> 8) It has several bugs that can cause it to forkbomb when tools
> are not built correctly.
> I propose that we switch over to using dejagnu by default, renaming
> check-dejagnu to check, and deprecate QMTest. We can either remove
> for this release or keep it until 1.5.
> I'd appreciate your opinion or any feedback you may have.
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu
More information about the LLVMdev