Last updated at 2016-11-30
- This is not an official Google product.
- This is tested only on Ubuntu though it would be running on any platforms which installed python 3 and depending python packages.
Sprockets is a framework for conformance testing based on state transitions.
To run conformance tests, Sprockets users should provide:
- a test manifest specifying the STL files, roles, and tests to run
- STL files describing a system defined with a set of protocols (or specifications)
- External Python libraries implementing external events and message encoders
State transitions are specified with STL (State Transition Language). See STL.md for details of STL.
test_driver.py is the main program made of Python to run conformance tests specified by users with a test manifest, STL files, and python libraries.
To run test_driver.py, extra python packages below are necessary:
- google.protobuf
- ply
- networkx packages
- websockets
If they are not installed already, follow commands below (tested only on Ubuntu):
$ pip install protobuf
$ pip install ply
$ pip install networkx
$ apt-get install python3-dev
$ apt-get install libgraphviz-dev
$ pip install pygraphviz \
--install-option="--include-path=/usr/include/graphviz" \
--install-option="--library-path=/usr/lib/graphviz/"
$ pip install websockets
$ python test_driver.py [options] <test manifest>
$ python test_driver.py -h
usage: test_driver.py [-h] [-a MANIFEST_ARGS] [-d] manifest
positional arguments:
manifest The manifest (*.test) file to run.
optional arguments:
-h, --help show this help message and exit
-a MANIFEST_ARGS, --manifest-args MANIFEST_ARGS
A list of <key>=<value> pairs. Each instance of
'$<key>' in the manifest file is replaced by <value>
verbatim. In particular, if you want to pass a string,
it must be explicitly quoted, e.g.: 'ip="0.0.0.0"'
-d, --debug Increase logging verbosity to debug level.
The test manifest describes the tests to be run. The test manifest file typically has a .test extension. The manifest file is formatted as a python dictionary with three keys: ‘stl_files’, ‘roles’, and ‘test’.
example.test:
{
'stl_files': [
'example_base.stl',
'example.stl',
],
'roles': [ # Role information
{ 'role': 'example::rReceiver',
'ipAddress': $ip, # Must pass --manifest-args 'ip="0.0.0.0"'
'transportId': 'receiver-0',
},
],
'constants': {
'example::kHelloWorld': $hello_world, # Must pass --manifest-args \
# 'hello_world="Hello, world!"'
},
'test': ['example::rReceiver'],
}
$ python test_driver.py example/example.test \
--manifest-args 'ip="0.0.0.0"' 'hello_world="Hello, world!"'
The ‘stl_files’ field must be a list of strings naming the STL files that include the transitions to be tested. The filenames here should be relative to the manifest .test file. In the example above, example.test and example_base.stl are in the same directory.
The ‘roles’ field must be list of dicts. Each dict element in the list describes one role. These are the roles described in the stl_files listed in the ‘stl_files’ field. In the above example, example_base.stl contains the following:
example.stl:
module example;
...
role rReceiver {
string ipAddress;
string transportId;
}
...
This gets transformed into
{
'role': 'example::rReceiver',
'ipAddress': $ip, # Must pass --manifest-args 'ip="0.0.0.0"'
'transportId': 'receiver-0',
}
The ‘role’ key maps to the ‘module:rRoleName’, which in this case is ‘example::rReceiver’.
The remaining key: value pairs map to the fields of the role as described in the STL file.
The ‘constants’ field must be a dictionary. Each entry in the dictionary defines a declared constant described in the stl_files listed in the ‘stl_files’ field. In the above example, example_base.stl contains the following:
example_base.stl:
module example;
...
const string kHelloWorld;
...
This value gets defined in the manifest here:
'constants': {
'example::kHelloWorld': $hello_world, # Must pass --manifest-args \
# 'hello_world="Hello, world!"'
}
The key maps to the ‘::’, which in this case is ‘example::kHelloWorld’.
The ‘test’ field must be a list of strings naming the roles to be tested. Each entry in the test list must correspond to a ‘role’ value in the ‘roles’ list.
Values can be substituted from the command line by passing a space separated list of = pairs to the --manifest-args option of test_driver.py. The manifest .test file will then replace any occurence of $ with . $ can be located anywhere in the manifest file (e.g. in the list of stl_files, as a key or value in any of the manifest dictionaries).
parameter_example.test:
{
'stl_files': ['$STL_FILE'],
'roles': [
{ 'role': '$TEST_ROLE',
$FIELD: $VALUE,
'my_number': $my_number,
'my_boolean': $my_boolean,
'str_no_quotes': '$str_no_quotes',
'str_with_quotes': $str_with_quotes,
'str_with_spaces': $str_with_spaces,
},
],
'test': ['$TEST_ROLE'],
}
$ python test_driver.py parameter_example.test --manifest-args \
STL_FILE=example.stl \
TEST_ROLE=example::rReceiver \
FIELD=\'new_field\' \
VALUE=\'new_value\' \
my_number=3 \
my_boolean=True \
'str_no_quotes=Hello, World' \
str_with_quotes=\'bar\' \
'str_with_spaces=\'foo bar baz\''
This is exactly equivalent to running with the following manifest:
parameter_example_with_substitues.test:
{
'stl_files': ['example.stl'],
'roles': [
{ 'role': 'example::rReceiver',
'new_field': 'new_value',
'my_number': 3,
'my_boolean': True,
'str_no_quotes': 'Hello, World',
'str_with_quotes': 'bar',
'str_with_spaces': 'foo bar baz',
},
],
'test': ['example::rReceiver'],
}