Metadata-driven Testing

Why you ever meta data you didn’t like:
Testing twisty little passages all alike.
Steven Lembark
Workhorse Computing
lemark@wrkhors.com

What is testing?
Smoke, white- & black-box, integration, regression...
Set up controlled environment.
See if software fails.
Lather, rinse repeat...
Controls include data, environment, handlers.

The usual way: Template tests
Write a test.
Make it work.
Copy it.
Edit it.
Copy it.
Edit it.
Copy...

“Red flags”
Cut + paste
Works until you find a bug...
Need a change...
Update 45 files.

Common result: A mess
Single directory.
Semi-random basenames.
Difficult to run in order.

First piece of metadata
Filesystem is powerful medicine.
Cures all sorts of problems.

Basenames
prove uses lexical order.
100-perlcritic.t
101-perltidy.t
102-pod.t
103-podt.t
10-init.t
200-player-TestRoo.t
201-TestRoo-Action.t
201-TestRoo-Actor.t
201-TestRoo-Adventure.t
201-TestRoo-Base.t
201-TestRoo-Item.t
201-TestRoo-Location.t
201-TestRoo-Player.t
202-TestClassMoose.t
20-player.t
300-black-hole.t
60-command.t

Basenames
Might not be what you
want.
100-perlcritic.t
101-perltidy.t
102-pod.t
103-podt.t
10-init.t
201-TestRoo-Actor.t
201-TestRoo-Base.t
201-TestRoo-Item.t
20-player.t
300-black-hole.t
60-command.t

Basenames
Might not be what you
want.
Use consistent names.
010-init.t
020-player.t
060-command.t
100-perlcritic.t
101-perltidy.t
102-pod.t
103-podt.t
201-TestRoo-Actor.t
201-TestRoo-Base.t
201-TestRoo-Item.t
300-black-hole.t

Directories
Need every test every time?
010-init.t
020-player.t
060-command.t
100-perlcritic.t
101-perltidy.t
102-pod.t
103-podt.t
201-TestRoo-Actor.t
201-TestRoo-Base.t
201-TestRoo-Item.t
300-black-hole.t

Directories
What about:
Quick tests for check-in.
Complete for release.
010-init.t
020-player.t
060-command.t
100-perlcritic.t
101-perltidy.t
102-pod.t
103-podt.t
201-TestRoo-Actor.t
201-TestRoo-Base.t
201-TestRoo-Item.t
300-black-hole.t

Directories
What about:
prove && commit;
prove -r && release;
010-init.t
020-player.t
060-command.t
100-perlcritic.t
101-perltidy.t
102-pod.t
103-podt.t
200-Modules
300-Missions

Directories
What about:
prove && commit;
prove -r && release;
01-init.t
02-player.t
06-command.t
10-perlcritic.t
11-perltidy.t
12-pod.t
13-podt.t
20-Modules
30-Missions

Directories
Baseline & config checks.
prove t/0*t;
01-init.t
02-player.t
06-command.t
10-perlcritic.t
11-perltidy.t
12-pod.t
13-podt.t
20-Modules
30-Missions

Directories
Simple integration tests:
prove -v t/1*t;
01-init.t
02-player.t
06-command.t
10-perlcritic.t
11-perltidy.t
12-pod.t
13-podt.t
20-Modules
30-Missions

Tests don’t need to be stupid.
Adding a little logic avoids copying.
Lazy: Write once, recycle in place.

Choose your mission
Defines the map, monsters, goal.
The mission files start with:
my $pkg = 'Adventure';
my $path = 't/etc/EmptyMap00.yaml';

Choose your mission
One set of tests starts with:
Same tests for any config file.
my $path = 't/etc/EmptyMap00.yaml';

Choose your mission
Don’t copy the file:
my $base = basename $0, ‘.t’;
my $path = “t/etc/$base.yaml”;

Choose your mission
Don’t copy the file:
Symlink them all to “./bin/01-mission_t”.
my $path = “t/etc/$base.yaml”;

Choose your mission
cd $(dirname $0)/../20-Missions;
for i in ../etc/*-mission.yaml;
do
base=$(basename $i .yaml);
ln -fs ../bin/01-mission_t ./01-$base.t;
done
One “test” for many configs:

Generic tests: well-formed Perl
“use_ok” is unfairly maligned.
It does something quite useful.
Better off if all modules pass it.

“use_ok” is unfairly maligned.
It does something quite useful.
Better off if all modules pass it.
And have a working package name.
With a version.

use Test::More;
use_ok ‘Frobnicate’;
can_ok Frobnicate => ‘VERSION’;
ok Frobnicate->VERSION,
“Frobnicate has a version”;
done_testing;
__END__

use Test::More;
my $package= ‘Frobnicate’;
use_ok $package;
can_ok $package => ‘VERSION’;
ok $frobnicate->VERSION,
“$package has a version”;
done_testing;

my $package= ‘Frobnicate’;

my @partz = split /W+/, $base;
my $package
= join ‘::’, @partz[ 1 .. $#partz ];

ln -fs ../bin/01-generic_t
./01-Acme-Eyedrops.t;

Test them all
cd $(dirname $0)/..;
rm -f 01*.t;
find .. -name $glob |
perl -n
-E 'state $path = ( glob "./bin/01-*_t" )[0];'
-E 'chomp;'
-E 'my @a = split m{[/]}, substr $_, 3;'
-E 'my $b = join "-", @a;'
-E 'symlink $path => "01-$b.t" or warn' ;

Testing groups of files
Runs same test on all modules:
prove 10-*.t ;
Test class & children:
prove *-Parent-*.t;
find t -name $glob | xargs prove;

Combine with commits
Git tags mark prove success.
Combine prove with git tags to track progress:
prove && git tag …
Merge tags into main branch for Q/A.
Use “prove -r” in master for Q/A pass.

Generic tests: well formed configs.
Ever fat-finger some JSON?
Leave out an XML tag?
Mis-quote an .ini?

Generic tests: config files.
Ever fat-finger some JSON?
Leave out an XML tag?
Mis-quote an .ini?
Then waste debugging code to find it?

Generic tests: config files
To the rescue: Config::Any.
If its read-able, we can check it.
At least for readabiliy...

Generic tests: config files
Same basic trick: symlink a reader.
my $base = basename $0;
my @partz= split /W+/, $base;
my $test = join ‘~’ => ‘01’, @partz;
symlink ‘../bin/00-config_t’ =>
$test . ‘.t’;

#!/bin/bash
cd $(dirname $0)/..; # run from ./t/bin
rm -f 0*.t; # remove generic tests
i='-1';
for glob in '*.yaml' '*.pm' # test config & modules
do
export j="0$((++i))";
echo "Pass: $j";
ls ./bin/$j-*_t;
find .. -name $glob |
perl -n
-E 'state $path = ( glob "./bin/$ENV{j}-*_t" )[0];'
-E 'chomp;'
-E 'my @a = split m{[/]}, substr $_, 3;'
-E 'my $b = join "-", @a;'
-E 'symlink $path => "$ENV{j}-$b.t" or warn' ;
done
exit 0;

Re-duce, Re-use, Re-cycle
Tests only depend on “.pm”.
Re-use on multiple directories.
Across projects.

Metadata: ./t is for testing
Ever test a production database?
Destructively?
Ouch...
Metadata: test configs are in ./t/etc.

# tests find ./t/lib/Foo/Config.pm
# ./bin files find ./lib/Foo/Config.pm
use FindBin::libs;
use Foo::Config;

# tests prefer ./t/etc
use FindBin::libs qw( base=etc export );
my $base = ‘Database.config.yaml’;
my $found
= first {-e “$_/$base” } @etc
or die “Oops... no database config.n”;
my $path = “$found/$base”;

Looking inside yourself
Or, at least, inside of Perl.
Overloads.
Closures.
Non-OO Polymorphism.

Dispatch via scalar.
Say you want to test Madness for a Method.
$madness->method( … );

Dispatch via scalar.
Say you want to test Madness for a Method.
$madness->$method( … ); # Perl5
$method can be text or a subref...

Aside: Dispatch in Perl6
my $code = sub { … };
$madness.$code;
$madness.$code( @argz );
my $name = ‘subname’;
$madness.”$name”();
$madness.”$name”( @argz );

Testing many methods
Iterate an object over many methods:
$object->$_( @argz )
for @methodz;

Example: Object::Exercise
my @plan =
(
[
[ method => ( ‘a’, ‘b’ ) ],
[ ‘Expected Return Value’ ],
[ ‘Your message here’ ],
],

use Object::Exercise;
my @plan = ( [ … ], [ … ], … );
$madness
->new( … )
->$exercise( @plan );

use Object::Exercise;
my @plan = generate_plan( … );
$madness
->new( … )
->$exercise( @plan );

my @plan = generate_plan( … );
Generate plan can read a YAML file...
Query a database...
Read a symlink...

Multiple tests in YAML
---
-
- - method
- a
- b
- - Expected Return Value
- - Your Message Here
-
- - frobnicate
- ...

Testing system failures
Check for file-read failures.
Bad way: Hack the filesystem.
Beter way: Hack Perl CORE.

All politics is local
So are values.
“local” provides scoped values.
local $ = “n”;
local *STDOUT = $fh;

Perl Testers Notebook
Great book, even with the fake coffee stains.
One nice technique described in detail:
Hacking CORE.
Say you want to open to fail.

Hack open???
sub
{
# see Perl Testing Notebook
local *CORE::open
= sub { die “No such file.n” };
$madness->$method( @_ );
}

Fail on one specific path
sub fail_on_open
{
my $pkg = shift;
my $path = shift;
my $open = $pkg->can( ‘open_config’ );
sub
{
$_[1] eq $path
and die “Failed open: ‘$path’n”;
goto &$open;
}
}

Wrap a method to fail on a specific file
my @plan
= map
{
my $sub = fail_on_open $pkg, $_;
[
[ $sub, “$_” ],undef,[ “Fail: $_” ]
]
}
glob “/etc/frobnicate/*.config.*”;

Testing runt reads
.ini or data files may not have bookends.
Lacking a closing marker, can you detect runts?
Bad test: Write hacked files to temp dir’s.
Better test: Hack your reader.

Dispatch a partial read
my $path = shift;
sub
{
$_[1] eq $path or goto &$wrapped;
my $data = &$wrapped;
substr $data, 0, rand length $data
}

my $method = ‘do_something’;
my @plan
= map
{
my $path = $_;
my $runt = gen_runt_read $path;
my $ref = qualify_to_ref read_cfg => $pkg;
my $sub
= sub
{
local *{ $ref } = $runt;
$object->$method( $path )
};
[
[ $sub, $path ], undef, [ “Failed read: ‘$path’” ]
]
}
glob $glob;
$object->$exercise( @plan ); # verify failing on each conf

Checking mods
Overload stat to return zero size, hacked mods.
Force fail on -s, -r checks for data files.
Return non-existant or zero UID, GID.

Mapped tests
Good: One test file, one test result.
Bad: Have to test them all each time.
Alternatives:
Symlink individual tests.
Break glob-lists into smaller pieces.

Aside: “Testable” code.
Monolithic code is harder to test.
Testing a find-and-check-and-validate-and-open-
and-read-and-close-and-evaluate-and-install-values-
and-use-values-and-return is hard.
Faking an open is relatively easy.
So is faking a read.
Un-testable code is less maintainable.

Adventure: Your mission...
Given a data file, play the game.
Twisty little passages, nasty trolls, you name it.

Sanity checking size
Generic wrapper: call a method, check size of
object.
Same basic wrapper:
Store size.
Call something.
Re-examine size.

---
name: Empty Map 00
namespace : EmptyMap00
locations:
blackhole:
name: BlackHole
description : You are standing in a small Black Hole.
exits :
Out : blackhole
items: {}
player:
location : blackhole
items : {}

Entering a black hole
use FindBin::libs qw(base=etc export scalar);
my $madness = ‘Adventure’;
use_ok $madness;
$madness->init(“$etc/blackhole.yaml);
my $player = $madness->player;
is_ok ‘blackhole’, $player->location;
$player->location_object
->use_exit('blackhole');
is_ok ‘blackhole’, $player->location;

Look for memory leaks
ok $player->move( blackhole ), “Can move out”
for 1 .. 1_000_000;

Look for memory leaks
ok $player->move( out ), “Can move out”
for 1 .. 1_000_000;
Downside: a million OK’s.

Avoid a million OK’s.
my $expect = ‘out’;
my $found= ‘’;
my $i;
for $i ( 1 .. 1_000_000 )
{
$player->move( $expect );
$found = $player->location;
$expect eq $found or last;
}
is $found, $expect, “’$found’-‘$expect’ at $i”;

How big are you?
Memory footprint:
sum_size
{
sum map { size $_ } @_
}

Yes, guys, size() does matter
my $loc = $player->location_object;
my $prior = 1.1*sum_size $madness, $player;
$loc->use_exit('blackhole')
for( 1 .. 1_000_000 );
my $after = sum_size $madness, $player;
ok $after < $prior, “$after < $prior”;

Generic Tests
Init the game with a mission file.
Check the initial location.
Write a black-hole file with 1 .. N stages.
Check that the size doesn’t grow.

my $limit = ( split / W /x, $base )[ -1 ];
my $path= make_daisy_chain_map $limit;
Adventure->init( $path );
my $player = Adventure->player;
my $loc = $player->location_object;
my $prior = 1.1 * size $player;
for my $i ( 1 .. $limit )
{
my $next = ‘room_’ . $i;
$loc->use_exit( $next );
$next eq $loc->location or last;
my $after = size $player;
$prior > $after or last;
}

Playing with the web
Ever have test a web app?
But didn’t have a “back end”?
You are doing it wrong!

Selenium Sandwich
Sandwich the browser between tasty layers of Perl:
use Plack;
use Selenium;

Metadata-driven selenium testing
Four structs define an iteration:
[
[ DOM contents ],
[ Expected values ],
[ return struct ],
[ result DOM ]
]

Failing successfully
Return the HTTP failure code:
[
[ DOM contents ],
[ Expected values ],
[ 404, “Oops...”, ],
[ result DOM ]
]

Summary
Avoid “red flags”.
Use data to drive tests.
Metadata to generate the data.
Abstract your tests.

Bedside reading
Object::Exercise
Selenium Sandwich
Getting Testy with Perl
https://www.slideshare.net/lembark

Bedside reading
Adventure:
git@github.com:rizen/Adventure.git

Metadata-driven Testing

Recommended

More Related Content

What's hot (20)

Similar to Metadata-driven Testing (20)

More from Workhorse Computing (18)

Recently uploaded (20)

Metadata-driven Testing