-
Notifications
You must be signed in to change notification settings - Fork 42
Sources
Sources are types of tests that OK can recognize. Each source describes how to load, interact with, and dump tests from a given file.
Staff members can specify which sources they want to use for a particular assignment in the configuration file's tests
field:
'tests': {
'file pattern': 'source',
...
}
The value of tests
is a mapping of file patterns to sources:
- file pattern: a UNIX-file pattern. Patterns may include wildcards
*
,?
and[]
-style ranges. - source: All files in the current directory that are matched by the file pattern will be loaded as the specified source.
Here's an example. Suppose we want to load all Python files in the tests/
directory as OK tests. We would specify this behavior with the following snippet:
'tests': {
'tests/*.py': 'ok_test'
}
The following is a list of currently supported sources:
Sources can be found in the directory client/sources/
:
client/
sources/
common/
doctest/
ok_test/
Each type of source is its own module (in the schematic above, doctest
and ok_test
are packages). Whether the source module is a standalone Python file (for lightweight sources) or a package (for more complex sources) is up to the developer. The only requirements are the following:
-
The source is importable in Python code. This places restrictions on possible names for sources (e.g. no hyphens)
-
The source module must have a function called
load
that returns a dictionary mapping test names toTest
objects (or subclasses ofTest
).load
has the following function signature:def load(file, parameter, cmd_args): """Loads the specified file as a particular source type. PARAMETERS: file -- str; a filepath to a test. This file is not guaranteed to exist. parameter -- str; parameters that may be necessary for loading this source. The source can ignore this parameter if it is not needed. cmd_args -- Namespace; command-line arguments as returned by argparse.ArgumentParser.parse_args(). RAISES: client.exceptions.LoadingException; in cases where the test cannot be loaded. RETURNS: dict; a mapping of test names to client.sources.common.models.Test objects, or subclasses of Test.
The names in the name-to-Test
mapping are only used for students to specify certain tests. Each source can choose what to name its tests.
The abstract class client.sources.common.models.Test
is a serializable object that defines an interface for Test objects.
- Fields:
name = core.String()
points = core.Float()
partner = core.Int(optional=True)
- Methods:
-
run()
: Defines how to run tests -
score()
: Defines how to score tests -
unlock(interact)
: Defines how to unlock tests. Theinteract
parameter is a function used for processing student input -
lock(hash_fn)
: Defines how to lock tests. Thehash_fn
parameter is the hash function used for generating the locked version of text -
dump
: Defines how to serialize the test.Test
's are expected to retain information about where tests should be dumped -- this information is provided by theload
function described above.
-