Kaputt – version 1.1Copyright © 2008-2011 Xavier Clerc – kaputt@x9c.fr |
Kaputt is a unit testing tool for the Objective Caml language1. Its name stems from the following acronym: Kaputt is A Popperian Unit Testing Tool. The adjective popperian is derived from the name of Karl Popper, a famous philosopher of science who is known for forging the concept of falsifiability. The tribute to Popper is due to the fact that Kaputt, like most test-based methodologies, will never tell you that your function is correct; it can only point out errors.
Kaputt features two main kinds of tests:
When writing assertion-based tests, the developer explicitly encodes input values and checks that output values satisfy given assertions. When writing specification-based tests, the developer encodes the specification of the tested function and then requests the library to either generate random values, or enumerate values to be tested against the specification.
Kaputt also provides shell-based tests that barely execute commands such as grep, diff, etc. They can be regarded as a special kind of assertion-based tests, and can be useful to run the whole application and compare its output to reference runs whose output has been stored into files.
Kaputt, in its 1.1 version, is designed to work with version 3.12.0 of Objective Caml. Kaputt is released under the GPL version 3. This licensing scheme should not cause any problem, as test versions of applications are intended to be used during developement but should not be released publicly. Bugs should be reported at http://bugs.x9c.fr.
Bolt can be built from sources using make (in its GNU Make 3.81 flavor), and Objective Caml version 3.12.0. No other dependency is needed. Following the classical Unix convention, the build and installation process consists in these three steps:
During the first step, one can specify elements if they are not correctly inferred by the ./configure script; the following switches are available:
During the third and last step, according to local settings, it may be necessary to acquire privileged accesses, running for example sudo make install.
The Java5 version will be built only if the ocamljava6 compiler is present and located by the makefile.
To use Kaputt, it is sufficient to compile and link with the library. This is usually done by adding of the following to the compiler invocation:
Since version 1.0, to access bigarray- and num-specific elements, it is necessary to link with respectively kaputtBigarray.cm[oxj] and kaputtNums.cm[oxj].
Typically, the developer wants to compile the code for tests only for internal (test) versions, and not for public (release) versions. Hence the need to be able to build two versions. The IFDEF directive of camlp4 can be used to fulfill this need. Code sample 1 shows a trivial program that is designed to be compiled either to debug or to release mode.
let () =
IFDEF DEBUG THEN
print_endline "debug mode on"
ELSE
print_endline "debug mode off"
ENDIF
To compile the debug version, one of the following commands (according to the compiler used) should be issued:
At the opposite, to compile the release version, one of following commands should be executed:
This means that the developer can choose the version to compile by only specifying a different preprocessor (precisely by enabling/disabling a preprocessor argument) to be used by the invoked OCaml compiler.
Code sample 2 shows how to use Kaputt from a toplevel session. First, the Kaputt directory is added to the search path. Then, the library is loaded and the module containing shorthand definitions is opened. Finally, the check method is used in order to check that the successor of an odd integer is even.
Objective Caml version 3.11.1
# #directory "+kaputt";;
# #load "kaputt.cma";;
# open Kaputt.Abbreviations;;
# check Gen.int succ [Spec.is_odd_int ==> Spec.is_even_int];;
Test 'untitled no 1' ... 100/100 cases passed
- : unit = ()
#
When writing assertion-based tests, one is mainly interested in the Assertion and Test modules. The Assertion module provides various functions performing tests over values. Then, the Test module allows to run the tests and get some report about their outcome. An assertion-based test built by the Test.make_assert_test function is made of four elements:
The idea of the set up and tear down functions is that they bracket the execution of the test function. If there is no data to pass to the test function (i.e. its signature is unit -> unit), the obvious choices for set up and tear down are respectively Test.return () and ignore; another possibility is to use the make_simple_test function. Code sample 3 shows a short program declaring and running two tests, the first one uses no data while the second one does. The second test also exhibits the fact that the title is optional.
open Kaputt.Abbreviations
let t1 =
Test.make_simple_test
~title:"first test"
(fun () -> Assert.equal_int 3 (f 2))
let t2 =
Test.make_assert_test
(fun () -> open_in "data")
(fun ch -> Assert.equal_string "waited1" (f1 ch); ch)
close_in_noerr
let () = Test.run_tests [t1; t2]
When writing specification-based tests, one is mainly interested in the Generator, Specification, and Test modules. The Generator module defines the concept of generator that is a function randomly producing values of a given type, and provides implementations for basic types and combinators. The Specification module defines the concept of specification that is predicates over values and their images through the tested function, as well as predicates over basic types and combinators. A specification-based test built by Test.make_random_test is made of nine elements (the six first ones being optional):
The generator, of type ’a Generator.t, is used to randomly produce test cases. Tests cases are produced until the requested number has be reached. One should notice that a test case is counted if and only if the generated value satisfies one of the preconditions of the specification.
The classifier is used to characterize the generated test cases to give the developer an overview of the coverage of the test (in the sense that the classifier gives hints about the portions of code actually executed). For complete coverage information, one is advised to use the Bisect tool8 by the same author.
The specification is a list of ⟨precondition, postcondition⟩ couples. This list should be regarded as a case-based definition. When checking if the function matches its specification, Kaputt will determine the first precondition from the list that holds, and ensure that the corresponding postcondition holds: if not, a counterexample has been found.
Assuming that the tested function has a signature of ’a -> ’b, a precondition has type ’a predicate (that is ’a -> bool) and a postcondition has type (’a * ’b) predicate (that is (’a * ’b) -> bool). The preconditions are evaluated over the generated values, while the postconditions are evaluated over ⟨generated values, image by tested function⟩ couples.
An easy way to build ⟨precondition, postcondition⟩ couples is to use the => infix operator. Additionally, the ==> infix operator can be used when the postcondition is interested only in the image through the function (ignoring the generated value), thus enabling lighter notation.
Code sample 4 shows how to build a test for function f whose domain is the string type. The classifier stores generated values into two categories, according to the length of the string. The pre_i functions are of type string -> bool, while the post_i functions are of type (string * t) -> bool where t is the codomain (also sometimes refered to as the “range”) of the tested function f.
open Kaputt.Abbreviations
let t =
Test.make_random_test
~title:"random test"
~nb_runs:128
~classifier:(fun s -> if (String.length s) < 4 then "short" else "long")
(Gen.string (Gen.make_int 0 16) Gen.char)
f
[ pre_1 => post_1 ;
...
pre_n => post_n ]
let () = Test.run_test t
The previous sections have exposed how to run tests using the Test.run_tests function. When only passed a list of tests, the outcome of these tests is written to the standard output in a (hopefully) user-friendly text setting. It is however possible to change both the destination and the layout by supplying an optional output parameter of type Test.output_mode, that is a sum type with the following constructors:
The passed channel is closed if it is neither stdout, nor stderr.
Test 'succ test' ... 100/100 cases passed
Test 'untitled no 1' ... 10/10 cases passed
Test 'sum of odds' ... 200/200 cases passed
Test 'strings' ... 0/2 case passed
counterexamples: "eYbHu", "UEggsF"
categories:
short -> 1 occurrence
long -> 1 occurrence
Test 'lists' ... 0/2 case passed
counterexamples: [ 6; 5; 1; 3; 6; ], [ 3; 2; 6; ]
<kaputt-report>
<random-test name="succ test" valid="100" total="100" uncaught="0">
</random-test>
<random-test name="untitled no 1" valid="10" total="10" uncaught="0">
</random-test>
<random-test name="sum of odds" valid="200" total="200" uncaught="0">
</random-test>
<random-test name="strings" valid="0" total="2" uncaught="0">
<counterexamples>
<counterexample value=""OAsdUXKf""/>
<counterexample value=""dhVMK""/>
</counterexamples>
<categories>
<category name="long" total="1"/>
<category name="short" total="1"/>
</categories>
</random-test>
<random-test name="lists" valid="0" total="2" uncaught="0">
<counterexamples>
<counterexample value="[ 5; 1; 6; ]"/>
<counterexample value="[ 6; 3; ]"/>
</counterexamples>
</random-test>
</kaputt-report>
random-test (stats)|succ test|100|100|0 random-test (stats)|untitled no 1|10|10|0 random-test (stats)|sum of odds|200|200|0 random-test (stats)|strings|0|2|0 random-test (counterexamples)|strings|"SHwJpJ"|"tbMlVNwqh" random-test (stats)|lists|0|2|0 random-test (counterexamples)|lists|[ 3; 6; 6; ]|[ 3; 4; 5; ]
<!ELEMENT kaputt-report (passed-test|failed-test|uncaught-exception|random-test|enum-test|shell-test)*> <!ELEMENT passed-test EMPTY> <!ATTLIST passed-test name CDATA #REQUIRED> <!ELEMENT failed-test EMPTY> <!ATTLIST failed-test name CDATA #REQUIRED> <!ATTLIST failed-test expected CDATA> <!ATTLIST failed-test not-expected CDATA> <!ATTLIST failed-test actual CDATA #REQUIRED> <!ATTLIST failed-test message CDATA> <!ELEMENT uncaught-exception EMPTY> <!ATTLIST uncaught-exception name CDATA #REQUIRED> <!ATTLIST uncaught-exception exception CDATA #REQUIRED> <!ELEMENT random-test (counterexamples?,categories?)> <!ATTLIST random-test name CDATA #REQUIRED> <!ATTLIST random-test valid CDATA #REQUIRED> <!ATTLIST random-test total CDATA #REQUIRED> <!ATTLIST random-test uncaught CDATA #REQUIRED> <!ELEMENT enum-test (counterexamples?)> <!ATTLIST enum-test name CDATA #REQUIRED> <!ATTLIST enum-test valid CDATA #REQUIRED> <!ATTLIST enum-test total CDATA #REQUIRED> <!ATTLIST enum-test uncaught CDATA #REQUIRED> <!ELEMENT counterexamples (counterexample*)> <!ELEMENT counterexample EMPTY> <!ATTLIST counterexample value CDATA #REQUIRED> <!ELEMENT categories (category*)> <!ELEMENT category EMPTY> <!ATTLIST category name CDATA #REQUIRED> <!ATTLIST category total CDATA #REQUIRED> <!ELEMENT shell-test EMPTY> <!ATTLIST shell-test name CDATA #REQUIRED> <!ATTLIST shell-test exit-code CDATA #REQUIRED>
This document was translated from LATEX by HEVEA.