Testing real world applications

Real world?

Why test?

• Business Value

How to test?

• Manual - exploratory
• Automated - regression

Manual Tests (exploratory tests)

• New feature is working
• Bug was eliminated
• General view of the system

Automated Tests (regression tests)

• Avoid regression
• Better Software Design (TDD)
• Your Sanity

Two cases of Automated tests

• TDD
• Real world

Real World

• Application mostly works
• Huge system
• Interdependent code
• Long functions
• It is hard to test parts of the system

Testing modes

• Unit testing
• Integration testing
• Acceptance testing (BDD ?)

Testing Environment

• Git
• Virtualization (Docker? VirtualBox?)
• One-click and fast setup

Setup - Fixture

• Web server
• Databases
• Windows machines
• Other devices
• External services

Fake the world

• Simulator
• Mocking
• Monkey patching = runtime changing of code.

Test Double

• Martin Fowler: Mocks aren't Stubs
• Gerard Meszaros: XUnit Patterns

Test Doubles

• Dummy
• Fake
• Stubs
• Spies
• Mocks

Mocking what?

• User created functions (or classes)
• 3rd party functions (or classes)
• Functions that are part of the language
• System calls (STDIO, files, time, etc.)
• Whole fiilesystem
• Database
• Network access
• External device/service

Perl

Fake (temporary) filesytem

Use temporary directories and files - locate pathes that are hard-coded in the code and replace them with configuration options from a configuration file or environment variables. File::Temp

use File::Temp qw(tempdir);
my $dir = tempdir( CLEANUP => 1 );

Fake library

Have a private implementation of the used library and make sure this one gets loaded instead of the real class. (e.g. by tweaking @INC and loading the module early.

package WWW::Mechanize;
use strict;
use warnings;

our $VERSION = 'fake';

sub new {
    return bless {}, shift;
}

sub get {
    ...
}

1;

Use the real library

use strict;
use warnings;
use Test::More;

plan tests => 1;

use WWW::Mechanize;
diag $WWW::Mechanize::VERSION;
like $WWW::Mechanize::VERSION, qr/^\d+\.\d+$/;

Use the fake library

use strict;
use warnings;
use Test::More;

plan tests => 1;

use Cwd qw(abs_path);
use File::Basename qw(dirname);
use lib dirname(abs_path($0)) . '/lib';

use WWW::Mechanize;
diag $WWW::Mechanize::VERSION;
is $WWW::Mechanize::VERSION, 'fake';

Mocking IO - module

package MyModule1;
use strict;
use warnings;

sub game {
    print("Enter your name: ");
    my $name = <STDIN>;
    chomp $name;
    print("Hello '$name'\n");

    return;
}

42;

Mocking IO - test

use strict;
use warnings;
use Test::More;

use MyModule1;
plan tests => 1;

{
    my $stdout;
    open my $out_fh, '>', \$stdout or die "Cannot open STDOUT to write to string: $!";
    my $stdin = "qwert\n";
    open my $in_fh, '<', \$stdin or die "Cannot open STDIN to read from string: $!";
    local *STDIN = $in_fh;
    local *STDOUT = $out_fh;

    MyModule1::game();

    is($stdout, "Enter your name: Hello 'qwert'\n");
}

Mocking IO

package MyModule2;
use strict;
use warnings;

sub game {
    print("Enter your name: ");
    my $name = <STDIN>;
    chomp $name;
    print("Hello '$name'\n");
    print("Guess a number: ");
    my $number = <STDIN>;
    chomp $number;
    print("Your number '$number' is good\n");
    
    return;
}

42;

Mocking IO - test

use strict;
use warnings;
use Test::More;

use MyModule2;
plan tests => 1;

{
    my $stdout;
    open my $out_fh, '>', \$stdout or die "Cannot open STDOUT to write to string: $!";
    my $stdin = join('',
        "qwert\n",
        "42\n",
    );
    open my $in_fh, '<', \$stdin or die "Cannot open STDIN to read from string: $!";
    local *STDIN = $in_fh;
    local *STDOUT = $out_fh;
    
    MyModule2::game();
    my @expected_stdout = (
        "Enter your name: ",
        "Hello 'qwert'\n",
        "Guess a number: ",
        "Your number '42' is good\n",
    );
    is($stdout, join('',@expected_stdout));
}

Mocking IO - script

use strict;
use warnings;

print("Enter your name: ");
my $name = <STDIN>;
chomp $name;
print("Hello '$name'\n");

Mocking IO - test script

use strict;
use warnings;
use Test::More;


plan tests => 1;

{
    my $stdout;
    open my $out_fh, '>', \$stdout or die "Cannot open STDOUT to write to string: $!";
    my $stdin = "abc def\n";
    open my $in_fh, '<', \$stdin or die "Cannot open STDIN to read from string: $!";
    local *STDIN = $in_fh;
    local *STDOUT = $out_fh;

    do "game.pl";

    is($stdout, "Enter your name: Hello 'abc def'\n");
}

Mocking function of web access

• Test::Mock::Simple
• [Mocking function to fake environment](https://perlmaven.com/mocking-function-to-fake-environme.t" %}

package MyWebAPI;
use strict;
use warnings;

use LWP::Simple qw(get);

my $URL = 'http://www.dailymail.co.uk/';

sub new {
    return bless {}, shift;
}

sub count_strings {
    my ($self, @strings) = @_;

    my $content = get $URL;

    my %data;
    foreach my $str (@strings) {
        $data{$str} = () = $content =~ /$str/ig;
    }
    return \%data;
}

1;

Test live web server

use strict;
use warnings;

use FindBin qw($Bin);
use lib $Bin;

use Test::More;
plan tests => 1;

use MyWebAPI;

my $w = MyWebAPI->new;

is_deeply $w->count_strings('Beyonce', 'Miley Cyrus'), 
    {
        'Beyonce'     => 26,
        'Miley Cyrus' => 3,
    };

1..1
not ok 1
#   Failed test at webapi.t line 14.
#     Structures begin differing at:
#          $got->{Miley Cyrus} = '4'
#     $expected->{Miley Cyrus} = '3'
# Looks like you failed 1 test of 1.

Mocking the get method

use strict;
use warnings;

use FindBin qw($Bin);
use lib $Bin;

use Test::More;
plan tests => 1;

use MyWebAPI;

use Test::Mock::Simple;
my $mock = Test::Mock::Simple->new(module => 'MyWebAPI');
$mock->add(get => sub {
    return 'Beyonce Beyonce Miley Cyrus';
});

my $w = MyWebAPI->new;

is_deeply $w->count_strings('Beyonce', 'Miley Cyrus'), 
    {
        'Beyonce'     => 2,
        'Miley Cyrus' => 1,
    };

1..1
ok 1

More test cases

use strict;
use warnings;

use FindBin qw($Bin);
use lib $Bin;

use Test::More;
plan tests => 3;

use MyWebAPI;

use Test::Mock::Simple;
my $mock = Test::Mock::Simple->new(module => 'MyWebAPI');
$mock->add(get => sub {
    return 'Beyonce Beyonce Miley Cyrus';
});

my $w = MyWebAPI->new;

is_deeply $w->count_strings('Beyonce', 'Miley Cyrus'), 
    {
        'Beyonce'     => 2,
        'Miley Cyrus' => 1,
    };

$mock->add(get => sub {
    return 'Beyonce';
});
is_deeply $w->count_strings('Beyonce', 'Miley Cyrus'), 
    {
        Beyonce => 1,
        'Miley Cyrus' => 0,
    };

$mock->add(get => sub {
    return '';
});
is_deeply $w->count_strings('Beyonce', 'Miley Cyrus'), 
    {
        Beyonce => 0,
        'Miley Cyrus' => 0,
    };

Test exception

$mock->add(get => sub {
    die 'Something went wrong'; 
});
is_deeply $w->count_strings('Beyonce', 'Miley Cyrus'), 
    {
        Beyonce => 0,
        'Miley Cyrus' => 0,
    };

1..2
ok 1
Something went wrong at exception.t line 25.
# Looks like you planned 2 tests but ran 1.
# Looks like your test exited with 255 just after 1.

Linewrap bug

$mock->add(get => sub {
    return 'Beyonce Miley Cyrus Miley
Cyrus';
});
is_deeply $w->count_strings('Beyonce', 'Miley Cyrus'), 
    {
        Beyonce => 1,
        'Miley Cyrus' => 2,
    };

1..4
ok 1
ok 2
ok 3
not ok 4
#   Failed test at linewrap.t line 46.
#     Structures begin differing at:
#          $got->{Miley Cyrus} = '1'
#     $expected->{Miley Cyrus} = '2'
# Looks like you failed 1 test of 4.

Mocking time: the session module

• Test::MockTime
• Testing sessions by mocking time

package MySession;
use strict;
use warnings;

my %SESSION;
my $TIMEOUT = 60;

sub new {
    return bless {}, shift;
}

sub login {
    my ($self, $username, $pw) = @_;
    # ...
    $SESSION{$username} = time;
    return;
}

sub logged_in {
    my ($self, $username) = @_;
    if ($SESSION{$username} and time - $SESSION{$username} < $TIMEOUT) {
        return 1;
    }
    return 0;
}

1;

Test session timeout

Takes 61 seconds to run...

use strict;
use warnings;

use FindBin qw($Bin);
use lib $Bin;

use Test::More;
plan tests => 3;

use MySession;

my $s = MySession->new;
$s->login('foo', 'secret');
ok $s->logged_in('foo'),  'foo logged in';
ok !$s->logged_in('bar'), 'bar not logged in';
sleep 61;
ok !$s->logged_in('foo'),  'foo not logged in - timeout';

Test session timeout faking the time

use strict;
use warnings;

use FindBin qw($Bin);
use lib $Bin;

use Test::MockTime qw(set_relative_time);
use Test::More;
plan tests => 3;

use MySession;

my $s = MySession->new;
$s->login('foo', 'secret');
ok $s->logged_in('foo'),  'foo logged in';
ok !$s->logged_in('bar'), 'bar not logged in';
set_relative_time(61);
ok !$s->logged_in('foo'),  'foo not logged in - timeout';

Testing a database driven application

• Have a set of test data (small set of data)
• Have a copy of the production data
• Use an in-memory database for testing to make it much faster than a disk-based one.
• Mock the database access
• DBD::Mock
• DBI::Mock

Build Schema by code

Have scripts that can create the database schema and load the initial data and scripts for migration among schema versions.

Manual Schema change

Have DBAs (or developers) change the schema manually on the development and later on the production server. As a minimum have some code that will regularly dump the schema and add it to some version control system.

Mocking in Java

• EasyMock
• JMock
• Mockito
• JMockit

How to move forward?

• Hard to justify as it usually does not find (new) bugs.
• Write tests for new features
• Write tests for bugs reported

How to find bugs?

• Use a linter.
• Turn on stricter compilation (and runtime) warnings.

How to find bugs in Perl?

use strict;
use warnings;

Run Perl::Critic on the code.

Testing culture

• Automated tests are like insurance.
• Get management support
• Build a culture