diff options
Diffstat (limited to 'test/README')
| -rw-r--r-- | test/README | 319 |
1 files changed, 319 insertions, 0 deletions
diff --git a/test/README b/test/README new file mode 100644 index 000000000..f1433944b --- /dev/null +++ b/test/README @@ -0,0 +1,319 @@ +Regression test suite for cairo. + +How to use cairo's test suite +============================= +Using this test should be as simple as running: + + make test + +assuming that the cairo distribution in the directory above has been +configured and built. The test suite here goes through some effort to +run against the locally compiled library rather than any installed +version, but those efforts may fall short depending on the level of your +libtool madness. + +The results of the test suite run are summarized in an index.html +file, which, when viewed in a web browser makes it quite easy to +visually see any failed renderings alongside the corresponding +reference image, (and a diff image as well). + +The test suite needs to be run before any code is committed and before +any release. See below for hints and rules governing the use of the suite. + +The test suite is built as a single binary, which allows you to choose +individual or categories of tests to run. For example, if you want to +run all text related tests you can use: + ./cairo-test-suite text +Or if you want to check the current status of known failures: + ./cairo-test-suite XFAIL +Or to run a subset of tests, use the -k option to run only the tests +that include the given keyword: + ./cairo-test-suite -k downscale +The binary also permits controlling which backend is used via the +CAIRO_TEST_TARGET environment variable, so for instance: + CAIRO_TEST_TARGET=gl ./cairo-test-suite -k blur +This binary should be backwards-compatible with all library versions, +allowing you to compare current versus past behaviour for any test. + +Tailoring tests running +----------------------- +There are some mechanisms to limit the tests run during "make test". +These come very handy when doing development, but should not be used +to circumvent the "pass" requirements listed below. + +make's TARGETS environment variable can be used to limit the backends when +running the tests. It should contain a (space-, comma-separated) list of +backends. CAIRO_TESTS environment variable, which is a comma-, space-seperated +lists, can be used to limit the tests run. +For example: + + CAIRO_TESTS="zero-alpha" make test TARGETS=image,ps + +make's FORMAT variable can also be used to limit the content formats when +running the tests. It should contain a (space-, comma-separated) list of +content formats to test. +For example: + + CAIRO_TESTS="zero-alpha" make test TARGETS=image,ps FORMAT="rgb,rgba" + +Another very handy mechanism when trying to fix bugs is: + + make retest + +This will re-run the test suite, but only on tests that failed on the +last run. So this is a much faster way of checking if changes actually +fix bugs rather than running the entire test suite again. + +The test suite first compares the output from the current run against the +previous in order to skip more expensive image comparisons . If you think +this is interfering with the results, you can clear the cached results using: + + make clean-caches + +Running tests under modified environments or tools +------------------------------------------------- +To run tests under a tool like gdb, one can use the run target and +the TOOL variable. For example: + + CAIRO_TESTS=user-font make run TOOL=gdb TARGETS=pdf + +If you want to run under valgrind, there is a specific target for that +that also sets a bunch of useful valgrind options. Try: + + CAIRO_TESTS=user-font make check-valgrind + +You can run tests under a modified environment you can use the ENV +make variable. However, that environment will also affect the libtool +wrapper of the tests. To only affect the actual test binaries, pass +such environment as TOOL: + + CAIRO_TESTS=user-font make run TOOL="LD_PRELOAD=/path/to/something.so" + +Getting the elusive zero failures +--------------------------------- +It's generally been very difficult to achieve a test run with zero +failures. The difficulties stem from the various versions of the many +libraries that the test suite depends on, (it depends on a lot more +than cairo itself), as well as fonts and other system-specific +settings. If your system differs significantly from the system on +which the reference images were generated, then you will likely see +the test suite reporting "failures", (even if cairo is working just +fine). + +We are constantly working to reduce the number of variables that need +to be tweaked to get a clean run, (for example, by bundling fonts with +the test suite itself), and also working to more carefully document +the software configuration used to generate the reference images. + +Here are some of the relevant details: + + * Your system must have a copy of the DejaVu font, the sha1sum of + the version used are listed in [...]. These are + "DejaVu Sans" (DejaVuSans.ttf) [1cd336329f45f241002ded61893d91e3acd04436]; + "DejaVu Sans Mono" (DejaVuSansMono.ttf) [0458c0f0fb57f3eb8ced62f26fe7c5ed4e6a9a68]; + "DejaVu Serif" (DejaVuSerif.ttf) [93502d0d0445d1fe1c9f51e51b3e0169266346ce]; + [the DejaVu fonts can be installed from the ttf-dejavu 2.33-2 Debian package] + and also + "Nimbus Sans L" (n019003l.pfb) + [which can be found in the gsfonts Debian package]. + + * Currently, you must be using a build of cairo using freetype + (cairo-ft) as the default font backend. Otherwise all tests + involving text are likely to fail. + + * To test the pdf backend, you will want the very latest version of + poppler as made available via git: + + git clone git://anongit.freedesktop.org/git/poppler/poppler + + As of this writing, no released version of poppler contains all + the fixes you will need to avoid false negatives from the test + suite. + + * To test the ps backend, you will need ghostscript version 9.04. + + * Testing the xlib backend is problematic since many X server + drivers have bugs that are exercised by the test suite. (Or, if + not actual bugs, differ slightly in their output in such a way + that the test suite will report errors.) This can be quite handy + if you want to debug an X server driver, but since most people + don't want to do that, another option is to run against a headless + X server that uses only software for all rendering. One such X + server is Xvfb which can be started like this: + + Xvfb -screen 0 1680x1024x24 -ac -nolisten tcp :2 + + after which the test suite can be run against it like so: + + DISPLAY=:2 make test + + We have been using Xvfb for testing cairo releases and ensuring + that all tests behave as expected with this X server. + +What if I can't make my system match? +------------------------------------- +For one reason or another, you may be unable to get a clean run of the +test suite even if cairo is working properly, (for example, you might +be on a system without freetype). In this case, it's still useful to +be able to determine if code changes you make to cairo result in any +regressions to the test suite. But it's hard to notice regressions if +there are many failures both before and after your changes. + +For this scenario, you can capture the output of a run of the test +suite before your changes, and then use the CAIRO_REF_DIR environment +variable to use that output as the reference images for a run after +your changes. The process looks like this: + + # Before code change there may be failures we don't care about + make test + + # Let's save those output images + mkdir /some/directory/ + cp -r test/output /some/directory/ + + # hack, hack, hack + + # Now to see if nothing changed: + CAIRO_REF_DIR=/some/directory/ make test + +Best practices for cairo developers +=================================== +If we all follow the guidelines below, then both the test suite and +cairo itself will stay much healthier, and we'll all have a lot more +fun hacking on cairo. + +Before committing +----------------- +All tests should return a result of PASS or XFAIL. The XFAIL results +indicate known bugs. The final message should be one of the following: + + All XX tests behaved as expected (YY expected failures) + All XX tests passed + +If any tests have a status of FAIL, then the new code has caused a +regression error which should be fixed before the code is committed. + +When a new bug is found +----------------------- +A new test case should be added by imitating the style of an existing +test. This means adding the following files: + + new-bug.c + reference/new-bug.ref.png + reference/new-bug.xfail.png + +Where new-bug.c is a minimal program to demonstrate the bug, following +the style of existing tests. The new-bug.ref.png image should contain +the desired result of new-bug.c if the bug were fixed while +new-bug.xfail.png contains the current results of the test. + +Makefile.sources should be edited by adding new-bug.c to test_sources. +And last but not least, don't forget to "git add" the new files. + +When a new feature is added +--------------------------- +It's important for the regression suite to keep pace with development +of the library. So a new test should be added for each new feature. +The work involved is similar the work described above for new bugs. +The only distinction is that the test is expected to pass so it +should not need a new-bug.xfail.png file. + +While working on a test +----------------------- +Before a bugfix or feature is ready, it may be useful to compare +output from different builds. For convenience, you can set +CAIRO_REF_DIR to point at a previous test directory, relative +to the current test directory, and any previous output will be +used by preference as reference images. + +When a bug is fixed +------------------- +The fix should be verified by running the test suite which should +result in an "unexpected pass" for the test of interest. Rejoice as +appropriate, then remove the relevant xfail.png file from git. + +Before releasing +---------------- +All tests should return a result of PASS for all supported (those enabled by +default) backends, meaning all known bugs are fixed, resulting in the happy +message: + + All XX tests passed + +Some notes on limitations in poppler +==================================== +One of the difficulties of our current test infrastructure is that we +rely on external tools to convert cairo's vector output (PDF, +PostScript, and SVG), into an image that can be used for the image +comparison. This means that any bugs in that conversion tool will +result in false negatives in the test suite. + +We've identified several such bugs in the poppler library which is +used to convert PDF to an image. This is particularly discouraging +because 1) poppler is free software that will be used by *many* cairo +users, and 2) poppler calls into cairo for its rendering so it should +be able to do a 100% faithful conversion. + +So we have an interest in ensuring that these poppler bugs get fixed +sooner rather than later. As such, we're trying to be good citizens by +reporting all such poppler bugs that we identify to the poppler +bugzilla. Here's a tracking bug explaining the situation: + + Poppler does not yet handle everything in the cairo test suite + https://bugs.freedesktop.org/show_bug.cgi?id=12143 + +Here's the rule: If a cairo-pdf test reports a failure, but viewing +the resulting PDF file with acroread suggests that the PDF itself is +correct, then there's likely a bug in poppler. In this case, we can +simply report the poppler bug, (making it block 12143 above), post the +PDF result from the test suite, and list the bug in this file. Once +we've done this, we can capture poppler's buggy output as a +pdf-specific reference image (as reference/*.xfail.png) so that the +test suite will regard the test as passing, (and we'll ensure there +is no regression). + +Once the poppler bug gets fixed, the test suite will start reporting a +false negative again, and this will be easy to fix by simply removing +the pdf-specific reference image. + +Here are the reported poppler bugs and the tests they affect: + +Poppler doesn't correctly handle gradients with transparency +https://bugs.freedesktop.org/show_bug.cgi?id=12144 +-------------------------------------------------- +fill-alpha-pattern +gradient-alpha +gradient-constant-alpha +linear-gradient +linear-gradient-reflect +radial-gradient +trap-clip + +Poppler should paint images with CAIRO_EXTEND_PAD +https://bugs.freedesktop.org/show_bug.cgi?id=14578 +-------------------------------------------------- +paint-source-alpha +paint-with-alpha +rotate-image-surface-paint +scale-source-surface-paint + +Incorrect clipping of group object (regression?) +https://bugs.freedesktop.org/show_bug.cgi?id=14580 +-------------------------------------------------- +push-group + +spurious horizontal stripes in color gradients +https://bugs.freedesktop.org/show_bug.cgi?id=10942 +-------------------------------------------------- +smask +smask-fill +smask-image-mask +smask-mask +smask-paint +smask-stroke +smask-text + +Ghostscript does not correctly render small miters +http://bugs.ghostscript.com/show_bug.cgi?id=690098 +-------------------------------------------------- +miter-precision |