Doctests are generated by recursing the source files and looking for
Example: Sphinx directives in the comments.
The corresponding Sphinx block, as defined by its indentation, represents a single doctest.
A doctest is composed of a series of statements.
Each statement is composed of a command and its expected output. The first line of the statement has the following structure:
>>> command-text [...] [% [comment-text] [doctest: flags]]
The first three characters,
>>> define the start of a statement.
The command text
command-text is MATLAB/Octave code.
The command text is optionally terminated by
..., in which case the command text of the next line will be concatenated.
An optional comment, which will be stripped from the constructed doctest. However, only on the first line of a statement,
the comment may contain a
doctest: substring, and the part of the comment following that substring will be interpreted as statement flags.
If a statement line representing a comment has its command part ending with
..., the next line is of the form:
`` command-text […] [% [comment-text]]``
and such lines are concatenated to the first as long as they end with
.... Flags are only parsed on the first line of a statement.
A statement is followed by the expected output, which can be empty. The expected test output ends
when the next line is no longer a comment line,
when another comment line starts with
when two blank comment lines are provided.
command-text can have different forms, and the expected output is interpreted accordingly.
In expected test output, blank lines are ignored. Errors are not captured, and will crash the test handler.
Test files are written in the MoXUnit format, using functions and subfunctions as the rest of the RepLAB test suite; helpers functions are used to verify the test output.
Our inspiration for the framework comes from http://doc.sagemath.org/html/en/developer/coding_basics.html#writing-testable-examples.
We distinguish the following three kinds of statements.
A silent statement finishes with a semicolumn. It can be of the form
a = statement; or
[a, b] = statement;.
In that case, the text written to the standard output/error streams is captured and compared to the data provided by the doctest.
A REPL statement is a non-assigning statement which does not finish with a semicolumn: it is of the form
statement, and is expected to returns a single output. If the command does not return any output (as would an
assert do, for example), it should be followed by a semicolon in the form of a silent statement instead.
There are two forms of expected output for REPL statements. The long form is used when the command displays text on the standard output:
>>> verbose_f() Log: computing the f value ans = 1
ans = line separates what the code displays on the standard output (before the
ans =) line, and what the returned value is (here,
>>> silent_f() 1
We diverge from the Matlab output conventions in one case. When we display the value of a variable, without assignment, we do not repeat the variable name. The following would be a valid Matlab session:
>> x = 2; >> x x = 2
whereas the corresponding doctest would be:
>>> x = 2; >>> x 2
A command of an assignment statement is of one of the forms:
a = statement, in which case the value of the variable
a is captured and compared to the doctest data;
[a, b] = statement, in which case several variables are captured and compared to the doctest data.
These statements do not finish with a semicolumn. The expected output contains first the text written by the command to the standard output, if any, before the value of each of the variables, where the value of each variable is preceded by a
varname = line.
>>> a = verbose_f() Log: computing the f value a = 1 >>> a = 2 a = 2 >>> [a, b] = deal(1, 2) a = 1 b = 2
>>> a = silent_f() 1
For now, we support the following types in the expected output.
Strings are char row vectors. In the expected output, they can be described in two ways:
They are written between single quotes as in
'between quotes', and quotes in the string are doubled (as in the standard Matlab string syntax:
'String with a ''double'' quote'). In that case, the string must not contain newline characters.
They are written without a beginning single quote, and they can be multiline.
double expected values are parsed using eval and compared for equality (the NaN case is not supported yet)
replab.cyclotomic expected scalars are written as expressed by the
vpi expected values are written on one line as a base 10 string
logical expected values are evaluated by
eval and converted to the logical type