[matplotlib-devel] Automated test harness discussion...

Continued from: requesting permission to remove traits and configobj...

Gael,
There might be ways to handle these problems.  A lot of depends on what we're trying to test.  I agree that if we take the example scripts, run them, and save the plots, we'll never get an automated test harness to figure things out because of machine differences.

However, if we set the goal of the testing to be that we make sure that MPL runs, that it accepts the correct options, and produces a plot, then we can be more creative with the testing harness.  We can do significant amounts of testing with a fixed back end (Agg probably) that generates an image for comparison.  We've tried a number of things in our own testing work which could help.  The first step is to identify why plots on different machines are different:  numeric differences in input data, agg output, font differences, colors, etc, etc.

Some ideas that might (or might not) help:

- Use wide lines that are grey in color for everything.  The plot looks crazy but then if you get one pixel shifts, it isn't a case of the pixels going "white, black, white" on one machine and "white, white, black" on another - you end up with most of the line overlapping which makes image comparisons easier.

- Never generate the input data on the machine you're on.  For example, never do this:
   t = arange(0.0,3.01,0.01)
   s = sin(2*pi*t)
Because you can get differences between machines.  A better way is to run this on the machine that will generate the "correct" image and then save the numbers using pickle or by embedding them in the script.

- Embed a font with the tests to eliminate font server differences (no experience with this so I'm not sure how hard this would be).  We could even create a dummy font that just has black squares for each character - it still tests that everything is drawn in the correct place and runs properly and eliminates subtle character differences.

- Create a testing backend that records the drawing commands as a set of meta-data like (draw red line from point 1 to point 2).  The test case then checks that the proper commands were issued by the test script.  This eliminates drawing completely.  A nice  comparison suite would allow loose comparisons like "make sure a vertical line was drawn from (10,20) to (30,40) with a pixel slop of 2 pixels.

- Smarter image comparison algorithms.  We currently use something that processes the image with PIL and looks at an averaged pixel difference (it's not perfect by any means).  I'll try to talk to some of the people here who work in image processing to see if there are any fuzzy image comparison algorithms they can recommend.

Ted

> -----Original Message-----
> From: Gael Varoquaux [mailto:gae...@no...]
> Sent: Thursday, December 11, 2008 10:06 PM
> To: Andrew Straw
> Cc: Drain, Theodore R; mat...@li...
> Subject: Re: [matplotlib-devel] requesting permission to remove traits
> and configobj
>
> On Thu, Dec 11, 2008 at 01:37:01PM -0800, Andrew Straw wrote:
> > Prabhu Ramachandran has also done similar things for mayavi2 using
> VTK's
> > image comparison (see compare_image_with_saved_image in
> >
> https://svn.enthought.com/enthought/browser/Mayavi/trunk/integrationtes
> ts/mayavi/common.py
> > ).
>
> Yeah, and it always fails due to the hardware rendering being slightly
> different on different computers :). I think we are giving up on this
> approach.
>
> Gaël
>
> No virus found in this incoming message.
> Checked by AVG - http://www.avg.com
> Version: 8.0.176 / Virus Database: 270.9.17/1844 - Release Date:
> 12/11/2008 8:58 PM