summaryrefslogtreecommitdiff
path: root/perf/README
diff options
context:
space:
mode:
Diffstat (limited to 'perf/README')
-rw-r--r--perf/README239
1 files changed, 239 insertions, 0 deletions
diff --git a/perf/README b/perf/README
new file mode 100644
index 000000000..9e402098a
--- /dev/null
+++ b/perf/README
@@ -0,0 +1,239 @@
+This is cairo's micro-benchmark performance test suite.
+
+One of the simplest ways to run this performance suite is:
+
+ make perf
+
+which will give a report of the speed of each individual test. See
+more details on other options for running the suite below.
+
+A macro test suite (with full traces and more intensive benchmarks) is
+also available; for this, see http://cgit.freedesktop.org/cairo-traces.
+The macro-benchmarks are better measures of actual real-world
+performance, and should be preferred over the micro-benchmarks (and over
+make perf) for identifying performance regressions or improvements. If
+you copy or symlink this repository at cairo/perf/cairo-traces, then
+make perf will run those tests as well.
+
+Running the micro-benchmarks
+----------------------------
+The micro-benchmark performance suite is composed of a series of
+hand-written, short, synthetic tests that measure the speed of doing a
+simple operation such as painting a surface or showing glyphs. These aim
+to give very good feedback on whether a performance related patch is
+successful without causing any performance degradations elsewhere.
+
+The micro-benchmarks are compiled into a single executable called
+cairo-perf-micro, which is what "make perf" executes. Some
+examples of running it:
+
+ # Report on all tests with default number of iterations:
+ ./cairo-perf-micro
+
+ # Report on 100 iterations of all gradient tests:
+ ./cairo-perf-micro -i 100 gradient
+
+ # Generate raw results for 10 iterations into cairo.perf
+ ./cairo-perf-micro -r -i 10 > cairo.perf
+ # Append 10 more iterations of the paint test
+ ./cairo-perf-micro -r -i 10 paint >> cairo.perf
+
+Raw results aren't useful for reading directly, but are quite useful
+when using cairo-perf-diff to compare separate runs (see more
+below). The advantage of using the raw mode is that test runs can be
+generated incrementally and appended to existing reports.
+
+Generating comparisons of separate runs
+---------------------------------------
+It's often useful to generate a chart showing the comparison of two
+separate runs of the cairo performance suite, (for example, after
+applying a patch intended to improve cairo's performance). The
+cairo-perf-diff script can be used to compare two report files
+generated by cairo-perf.
+
+Again, by way of example:
+
+ # Show performance changes from cairo-orig.perf to cairo-patched.perf
+ ./cairo-perf-diff cairo-orig.perf cairo-patched.perf
+
+This will work whether the data files were generate in raw mode (with
+cairo-perf -r) or cooked, (cairo-perf without -r).
+
+Finally, in its most powerful mode, cairo-perf-diff accepts two git
+revisions and will do all the work of checking each revision out,
+building it, running cairo-perf for each revision, and finally
+generating the report. Obviously, this mode only works if you are
+using cairo within a git repository, (and not from a tar file). Using
+this mode is as simple as passing the git revisions to be compared to
+cairo-perf-diff:
+
+ # Compare cairo 1.2.6 to cairo 1.4.0
+ ./cairo-perf-diff 1.2.6 1.4.0
+
+ # Measure the impact of the latest commit
+ ./cairo-perf-diff HEAD~1 HEAD
+
+As a convenience, this common desire to measure a single commit is
+supported by passing a single revision to cairo-perf-diff, in which
+case it will compare it to the immediately preceding commit. So for
+example:
+
+ # Measure the impact of the latest commit
+ ./cairo-perf-diff HEAD
+
+ # Measure the impact of an arbitrary commit by SHA-1
+ ./cairo-perf-diff aa883123d2af90
+
+Also, when passing git revisions to cairo-perf-diff like this, it will
+automatically cache results and re-use them rather than re-running
+cairo-perf over and over on the same versions. This means that if you
+ask for a report that you've generated in the past, cairo-perf-diff
+should return it immediately.
+
+Now, sometimes it is desirable to generate more iterations rather than
+re-using cached results. In this case, the -f flag can be used to
+force cairo-perf-diff to generate additional results in addition to
+what has been cached:
+
+ # Measure the impact of latest commit (force more measurement)
+ ./cairo-perf-diff -f
+
+And finally, the -f mode is most useful in conjunction with the --
+option to cairo-perf-diff which allows you to pass options to the
+underlying cairo-perf runs. This allows you to restrict the additional
+test runs to a limited subset of the tests.
+
+For example, a frequently used trick is to first generate a chart with
+a very small number of iterations for all tests:
+
+ ./cairo-perf-diff HEAD
+
+Then, if any of the results look suspicious, (say there's a slowdown
+reported in the text tests, but you think the text test shouldn't be
+affected), then you can force more iterations to be tested for only
+those tests:
+
+ ./cairo-perf-diff -f HEAD -- text
+
+Generating comparisons of different backends
+--------------------------------------------
+An alternate question that is often asked is, "how does the speed of one
+backend compare to another?". cairo-perf-compare-backends can read files
+generated by cairo-perf and produces a comparison of the backends for every
+test.
+
+Again, by way of example:
+
+ # Show relative performance of the backends
+ ./cairo-perf-compare-backends cairo.perf
+
+This will work whether the data files were generate in raw mode (with
+cairo-perf -r) or cooked, (cairo-perf without -r).
+
+
+Creating a new performance test
+-------------------------------
+This is where we could use everybody's help. If you have encountered a
+sequence of cairo operations that are slower than you would like, then
+please provide a performance test. Writing a test is very simple, it
+requires you to write only a small C file with a couple of functions,
+one of which exercises the cairo calls of interest.
+
+Here is the basic structure of a performance test file:
+
+ /* Copyright © 2006 Kind Cairo User
+ *
+ * ... Licensing information here ...
+ * Please copy the MIT blurb as in other tests
+ */
+
+ #include "cairo-perf.h"
+
+ static cairo_time_t
+ do_my_new_test (cairo_t *cr, int width, int height)
+ {
+ cairo_perf_timer_start ();
+
+ /* Make the cairo calls to be measured */
+
+ cairo_perf_timer_stop ();
+
+ return cairo_perf_timer_elapsed ();
+ }
+
+ void
+ my_new_test (cairo_perf_t *perf, cairo_t *cr, int width, int height)
+ {
+ /* First do any setup for which the execution time should not
+ * be measured. For example, this might include loading
+ * images from disk, creating patterns, etc. */
+
+ /* Then launch the actual performance testing. */
+ cairo_perf_run (perf, "my_new_test", do_my_new_test);
+
+ /* Finally, perform any cleanup from the setup above. */
+ }
+
+That's really all there is to writing a new test. The first function
+above is the one that does the real work and returns a timing
+number. The second function is the one that will be called by the
+performance test rig (see below for how to accomplish that), and
+allows for multiple performance cases to be written in one file,
+(simply call cairo_perf_run once for each case, passing the
+appropriate callback function to each).
+
+We go through this dance of indirectly calling your own function
+through cairo_perf_run so that cairo_perf_run can call your function
+many times and measure statistical properties over the many runs.
+
+Finally, to fully integrate your new test case you just need to add
+your new test to three different lists. (TODO: We should set this up
+better so that the lists are maintained automatically---computed from
+the list of files in cairo/perf, for example). Here's what needs to be
+added:
+
+ 1. Makefile.am: Add the new file name to the cairo_perf_SOURCES list
+
+ 2. cairo-perf.h: Add a new CAIRO_PERF_DECL line with the name of your
+ function, (my_new_test in the example above)
+
+ 3. cairo-perf-micro.c: Add a new row to the list at the end of the
+ file. A typical entry would look like:
+
+ { my_new_test, 16, 64 }
+
+ The last two numbers are a minimum and a maximum image size at
+ which your test should be exercised. If these values are the same,
+ then only that size will be used. If they are different, then
+ intermediate sizes will be used by doubling. So in the example
+ above, three tests would be performed at sizes of 16x16, 32x32 and
+ 64x64.
+
+
+How to run cairo-perf-diff on WINDOWS
+-------------------------------------
+This section explains the specifics of running cairo-perf-diff under
+win32 plateforms. It assumes that you have installed a UNIX-like shell
+environment such as MSYS (distributed as part of MinGW).
+
+ 1. From your Mingw32 window, be sure to have all of your MSVC environ-
+ ment variables set up for proper compilation using 'make'
+
+ 2. Add the %GitBaseDir%/Git/bin path to your environment, replacing the
+ %GitBaseDir% by whatever directory your Git version is installed to.
+
+ 3. Comment out the "UNSET CDPATH" line in the git-sh-setup script
+ (located inside the ...Git/bin directory) by putting a "#" at the
+ beginning of the line.
+
+you should be ready to go !
+
+From your mingw32 window, go to your cairo/perf directory and run the
+cairo-perf-diff script with the right arguments.
+
+Thanks for your contributions and have fun with cairo!
+
+TODO
+----
+Add a control language for crafting and running small sets of micro
+benchmarks.