dovetail Package¶

`dovetail` Package¶

Dovetail API implementation.

Class-relationship diagram¶

          Module: loader.py                            Module: model.py

                        build file                                 +--------------+
+-------------+       +-------------+        +-------------+       | Dependencies |
|             |------>|             |<------*|             |       +--------------+
| PackageNode |       | BuildModule |        |    Task     |
|             |*-+    |             |*-+     |             |       +--------------+
+-------------+  |    +-------------+  |     +-------------+       | TaskWrapper  |
             \___/                 \___/            ^              +--------------+
            Package                 Loaded          |
           Structure                                |           Objects in model.py are related
                                                    |           but look each other up dynamically
                                                    *
                                             +--------------+
                                             |              |
           +-------------+ <<creates>>       |  Execution   |
           |             |~~~~~~~~~~~~~~~~~~>|              |*-+
           |             |                   +--------------+  |
           |  Processor  |                          ^      \___/
           |             |                          |
           |             | <<creates>>       +--------------+
           |             |~~~~~~~~~~~~~~~~~~>|              |
           +-------------+                   |    Result    |
                                             |              |
                                             +--------------+

                          Module: engine.py

Links:

PackageNode: Model of package/directory structure of the build files

BuildModule: Models build files

Task: Model of a Task

Dependencies: Stores dependencies between Tasks

TaskWrapper: Directives (such as @do_if or @adjust_env)

Processor: The class that runs the build

Execution: The running and completed state of the build

Result: Container for a build’s result

Flow of events¶

The major flow of events when running within the same Python VM:

.   main.py        .   loader.py   .   engine.py                               .
.                  .               .                                           .
. +-------------+  .               .                                           .
. |    main()   |  .               .                                           .
. +-------------+  .               .                                           .
.        |         .               .                                           .
.        v         .               .                                           .
. +-------------+  .   +--------+ This instantiates the                        .
. |   inner()   |----->| load() | PackageNode, BuildModule                     .
. +-------------+  .   +--------+ and Task objects                             .
.        |         .               .                                           .
.        v         .               .                                           .
. +-------------+  .               .                                           .
. | execute_in_ |  .               .  +---------+    +----------------------+  .
. |    vm()     |-------------------->| build() |--->| Processor._execute() |  .
. +-------------+  .               .  +---------+    +----------------------+  .
.                  .               .                                           .

Links:

main(): Entry point from the command line

load(): Loads build files creating:

PackageNode: Models the packages

BuildModule: Models build files

Task: Task instances (and other associated structures)

build(): Runs the build

Note

execute_in_vm() calls stamp_environment() to set a number of environment variables before the call to build()

Processor._execute implementation¶

The Processor ‘runs’ a build using the following algorithm to execute a single Task, recording activity in Execution instances on a stack. The Execution states are defined in the Enumeration dovetail.engine.STATE:

Method Processor._execute(self, task):

Begin: Push a new Execution frame onto the stack

Check for errors:

2.1. Check if we have already run this task. If update the frame to STATE.REPEATED

and go to #10

2.2. Check if there is a circular invocation which has lead to

an infinite recursion. If so, fail the build and go to #10

CWD: Change working directory unless specifically disabled with @cwd(None)

Directives: Execute the TaskWrapper.before() method of all directives

Dependencies: If Execution.state is not STATE.SKIPPED or STATE.FAILED:

5.1. Loop over Task.dependencies()

5.1.1. Recursive call to Processor._execute() to execute the Task.

If the execution was not successful (Execution.is_ok() is False), fail this Task with STATE.ABORT and go to #8

CWD: Check the working directory to ensure that anything the dependents did is reverted here before the main body of the Task runs

Body: Execute the Task body by execute the function returned by Task.get_function()

Directives: Execute the TaskWrapper.after() method of all directives in reverse order

If nothing went wrong, mark this Execution with STATE.SUCCEEDED

CWD: Reset the working directory

End: Pop the Execution stack frame

This method is heavily error-trapped and guarded - in the event that any failures occur in the directives (in the TaskWrapper.before() and TaskWrapper.after() methods) or the task dependencies or the task’s function, the method will catch the error and fail gracefully and safely.

Note

Exceptions raised during this process can trigger two different outcomes:

If the exception occurs executing a Task function, then the Task is failed (STATE.FAIL), and the build is failed

Otherwise, the exception must have come from the Dovetail framework, so the exception is propagated up the the highest level as a system failure.

`config` Module¶

Handles the command-line parameter configuration and the configuration files

dovetail.config.BOOLEAN = ['true', 't', 'on', 'yes', 'y', '1', 'ok', 'false', 'f', 'off', 'no', 'n', '0']¶: A conjunction of TRUE and FALSE

dovetail.config.FALSE = ['false', 'f', 'off', 'no', 'n', '0']¶: Values that are considered ‘True’ in the configuration file

class dovetail.config.Option(name, validator=None, mapper=None)¶

Bases: object

A class which encapsulates an option in the Dovetail section of the configuration file, and the code to validate the entry and map it to the internal data model.

Parameters:

name (string) – Name of an Option group
validator (function(string)) – Defaults None. A function that validates a value in the config file If None, no validation is performed
mapper (function(string)) – Defafult None. A function that converts a value found in a config file to the appropriate internal type and value. If None, the value is isomorphic - it is not mapped.

static lookup(name)¶: Returns the Option with the given name.

map(value)¶

Converts the value into the appropriate representation for the option using the Option ‘mapper’ function.

Parameters:	value (string) – A value to look up
Returns:	The mapped value
Raises :	`ValueError` if value does not validate

dovetail.config.TRUE = ['true', 't', 'on', 'yes', 'y', '1', 'ok']¶: Values that are considered ‘True’ in the configuration file

dovetail.config.apply_config_to_args(parser, args)¶

Takes a config file (in parser) and applies the Dovetail section to the argparser (or dictionary) in args.

Parameters:	parser (ConfigParser.ConfigParser) – A ConfigParser with loaded file args (argparser.ArgumentParser) – The ArgumentParser constructed from the command line
Raises :	`dovetail.util.exception.InvalidEnvironment`

dovetail.config.apply_config_to_environ(parser)¶

Reads the Environment section of a ConfigParser, writing all entries to os.environ.

Parameters:	parser (class:ConfigParser.ConfigParser) – The ConfigParser

For each item in [Environment], an environment variable will be set in os.environ with the item’s value.

dovetail.config.coerce_to_boolean(boolean_as_string)¶

Returns the string as a boolean.

Parameters:	boolean_as_string (string) – A string, eg from the config file
Returns:	The boolean representation of boolean_as_string
Return type:	boolean

coerce_to_boolean() is not case sensitive. It looks up appropriate values for True from TRUE.

It is assumed that boolean_as_string is valid (i.e. has already been passed to is_a_boolean() without ValueError

dovetail.config.create_parser()¶

Creates the command line parser.

Returns:	An ArgumentParser instance
Return type:	`argparse.ArgumentParser`

dovetail.config.find_best_config()¶

Looks at the various ini file locations and probes them; if it finds a config file, it will use load_config_file() to load the file, and return the then return the ConfigParser.SafeConfigParser for it.

Returns:	A ConfigParser loaded from the best configuration file found or None
Return type:	`ConfigParser.SafeConfigParser`

See load_config() for more details

dovetail.config.is_a_boolean(boolean_as_string)¶

Returns without error if the string can be mapped to a boolean.

Parameters:	boolean_as_string (string) – The string from a config file
Returns:	The value as a boolean
Return type:	boolean
Raises :	`ValueError` if boolean_as_string is not in `BOOLEAN`

dovetail.config.load_config(command_line_args)¶

Loads the configuration for Dovetail from configuration files and the command-line arguments; command line overrides configuration files.

Parameters:	command_line_args (list of string) – The command line arguments
Returns:	An ArgumentParser with modifications from the config file
Return type:	`argparse.ArgumentParser`

The configuration file is found with find_best_config(). It is in two sections:

Section Dovetail:
- Items are defined by the Option instances defined in this file.
- apply_config_to_args() applies the config file to the arguments
Section Environment:
- For each item in this section, an environment variable will be set in os.environ with the item’s value
- apply_config_to_environ() reads and applies the values

dovetail.config.load_config_file(config_file)¶

Loads the config file named in config_file into a SafeConfigParser, validating the file argument

Parameters:	config_file (string) – A path to a configuration file to be read
Returns:	A configuration parser with the file loaded in it
Return type:	`ConfigParser.SafeConfigParser`

dovetail.config.map_to_log_level(level)¶

Returns the correct enumeration value for the Log Level.

Parameters:	level (string) – A Log `dovetail.util.logging.LEVEL` value
Returns:	The log level enumeration value
Return type:	int
Raises :	`ValueError` if level is not a valid log level

dovetail.config.print_help(out_file=None)¶

Utility function to print the help of the Dovetail argparser.

Parameters:	out_file (File Object) – Default None (std_out). An optional file stream to write the help to

dovetail.config.print_usage(out_file=None)¶

Utility function to print the usage instructions of the Dovetail argparser.

Parameters:	out_file (File Object) – Default None (std_out). An optional file stream to write the help to

dovetail.config.split_on_spaces(words)¶

Returns a list of space-separated items in the string; if the string is empty, an empty list is returned

Parameters:	words (string) – A space-separated series of items
Returns:	Splits the string on spaces, returning a list of items. If words if None or empty, the empty list is returned
Return type:	list

`constants` Module¶

Version and product information constants

dovetail.constants.DESCRIPTION = 'Dovetail: A light-weight, multi-platform build tool with Continuous Integration servers like Jenkins in mind'¶: Description of Dovetail for setup.py

dovetail.constants.DEVELOPMENT_STATUS = 'Development Status :: 4 - Beta'¶: Development status for setup.py

dovetail.constants.VERSION = '1.0beta3'¶: The version of Dovetail, for setup.py

dovetail.constants.read(filename)¶: Returns the contants of the file as a string

`engine` Module¶

The processing engine and state objects.

class dovetail.engine.Execution(task)¶

Bases: object

A stack frame recording the internal and external aspects of processing a Task and its dependents.

Parameters:	task (`Task`) – The task whose execution is recorded

The Execution stack is returned to by build() function wrapped in a Result.

Attributes:

Attribute	Type	Description
task	`Task`	A reference to the Task that ran.
state	Enum from `STATE`	Record of the state of the processing. See state table see `STATE`
start, finish	`datetime.datetime`	Start and end time of processing. If the processing has not finished, finish is None
parent	`Execution`	The `Execution` instance that directly or indirectly invoked this task
dependents	list of `Execution`	List of `Execution` instances of direct or indirect task invocations
skip_reasons	list of string	Accumulated messages from @skip_if, @do_if or any other cause of this task being skipped
result	any	The return value of the Task (if successful)
exception	`Exception`	If the task failed, the Exception that was thrown from the Task
exc_type, exc_value, exc_traceback		Details from the exception retrieved by `sys.exc_info()`.

abort()¶: Set the state to indicate that a dependency of the Task failed; this Task will be aborted

add_dependent(execution)¶

Adds another Execution instance as a child of this.

Parameters:	execution (`Execution`) – A child task of this

call_ended()¶: Set the state of the class:Execution to indicate that a Task has called build() and that task has completed

call_started()¶

Set the state of the Execution to indicate that a Task has called build() to run another Task.

This state continues only until the called Task returns

count()¶

Returns the number of Execution instances in the tree.

If this has no children, it will return 1

depth()¶

Returns the depth of this call in the call stack.

Returns:	Depth of this `Execution` in the call stack
Return type:	int

duration()¶

Returns how long the task took, including dependencies.

Returns:	Duration of execution
Return type:	float

duration_dependents()¶

Returns the duration that the dependencies of this Execution took to execute

Returns:	Duration of dependency execution
Return type:	float

duration_internal()¶

Returns the time inherent in processing the Task, i.e. less its dependencies

Returns:	Duration of execution of the `Task`
Return type:	float

executions_by_duration()¶: Returns a list of all executions from this point down the graph ordered by their elapsed internal duration (eg exclude) with the longest elapsed executions first

fail(exception)¶

Set the state to indicate the Task failed with an exception.

Parameters:	exception (`Exception`) – The exception thrown by the task

find_failed()¶

Searches the Execution graph for the Execution object which failed the build, either returning the Execution object or None

Returns:	The `Exception` which maps to the `Task` which failed the build. Returns None if no exception
Return type:	`Execution` or None

flatten()¶: Returns a list containing this and all dependent Executions

format_exception()¶

Returns a formatted exception message, exactly as Python does, for the exc_type, exc_value and exc_traceback attributes.

Returns:	Formatted exception
Return type:	string

get_messages()¶

Returns a list of Message objects that were logged.

Returns:	Messages recorded during the execution of the task
Return type:	list of `Message`

has_stack_trace()¶: Returns True if this node failed and captured a stack trace

imprint_stack_trace()¶: Runs sys.exc_info() to stamp this object with the stack trace of the current exception

internal()¶: Set the state of the Execution to indicate processing of the internal task itself (dependencies complete)

is_complete()¶

Is the Task complete?

Returns:	Returns True if the Task has completed execution, successfully or not
Type :	boolean

is_ok()¶

Returns True if the Task has completed execution without error.

Returns:	Returns True if `Execution.state` is SKIPPED, SUCCEEDED or REPEATED
Type :	boolean

log(message)¶

Log a message to be recorded on the Execution object.

Parameters:	message (string) – A message

This may later be retrieved programmatically or reported

received_stderr()¶

Returns true if the Execution received any error messages from stderr.

Returns:	Did this task generate output on stderr?
Return type:	boolean

repeated()¶: Set the state to indicate that this Task has already run

retrieve(key)¶

Retrieves a value from the Execution frame store

Works in conjunction with store().

set_result(result)¶: Sets the result of the Task.

Note

This has to be set before the Task is said to be complete because the TaskWrapper.after() directives may fail the overall result

skip(reason)¶

Set the state to indicate the Task was skipped.

Parameters:	reason (string) – The reason for skipping the `Task`

store(key, value)¶

Stores the key-value pair in a dictionary in the Execution frame for later retrieval allowing multiple participants to share data.

store() works in conjunction with retrieve() to create a:

Shared whiteboard - use simple keys such as strings, or

Private rendezvous for mementos - use a private key such as a function reference

succeed()¶: Set the state to indicate the execution was successful, capturing the result

system_exception(exception)¶

Update the state to record that a system exception has occurred, and to fail the ongoing build.

Parameters:	exception (`Exception`) – A system exception thrown by Dovetail, a directive or predicate

dovetail.engine.PROCESSOR = <dovetail.engine.Processor object at 0x10c5ee490>¶: Reference to the Processor singleton

class dovetail.engine.Processor¶

Bases: object

The processor that runs tasks and their decorators and maintains a call stack of Execution instances.

To run (or build) a Task, use the build() function.

class dovetail.engine.Result(execution)¶

Bases: object

The results of executing a build.

Result is constructed by Processor after the build has completed, successfully or otherwise.

Parameters:	execution (`Execution`) – The top-level stack frame

Attributes:

Attribute	Type	Description
task	`Task`	A reference to the Task that ran. If the build was launched with multiple tasks this will be the `NullTask`
success	boolean	A boolean indicating if this run was successful or not
result	any	The return value of the Task (if successful)
execution	`Execution`	A reference to the Execution instance containing a complete history of the run
exception	`Exception`	If the run failed, the Exception that was thrown from the failing Task
exc_type, exc_value, exc_traceback		Details from the exception retrieved by `sys.exc_info()`.

BANNER = "\n===============================================================================\n{0}\n\n Task '{1}' {2}\n\n Environment:\n Hostname: {3}\n Username: {4}\n Python: {5}\n Working dir: {6}\n Build file: {7}\n Tasks: {8}\n Elapsed: {9:.3f}s\n\n===============================================================================\n{10}\n"¶

banner()¶: Returns a report banner including most information required to identify this build

format_exception()¶

Returns a formatted exception message, exactly as Python does, for the exc_type, exc_value and exc_traceback attributes.

Convenience method - delegates to the topmost execution object.

If no exception this method returns the empty string

report_slowest()¶: Report the slowest tasks in the build (comprising >80% of build time, and being faster than 1ms)

report_tree()¶: Produce a report of the structure of the Tasks and their results

top_level_tasks()¶

Returns a list of the the top-level Tasks that were executed.

If the top-level task is the logical grouping task, this will return the logical grouping’s dependents

dovetail.engine.STATE¶

An enumeration capturing the state of execution.

Execution is modelled as a state machine with the following states:

STATE.STARTED: The Execution instance has been freshly created

STATE.DEPENDENTS: The Processor is executing the Task dependencies from Dependencies

STATE.RUNNING: The Processor is executing the main body of the Task

STATE.CALLING: A Task is programmatically calling another Task

STATE.SKIPPED: Directives caused the Task to be skipped (not executed) but treated as successful

STATE.SUCCEEDED: The Task completed normally

STATE.FAILED: The Task itself failed (raising an exception)

STATE.ABORTED: A dependent Task of this Task failed. The Processor makes no attempt to run the main body of the Task

STATE.REPEATED: A Task had already been executed (perhaps as a dependency or directly invoked); it is not executed again

And models state transitions as:

     .---------------------------------------------------.
    /  .-----------------------------------> SKIPPED >--.|
   /  /  .----------------------.                       ||
  /  /  /                       |   .------> SUCCEEDED  ||
 /  /  /                        v  /                    ||
STARTED ---> DEPENDENTS ---> RUNNING >-----> FAILED <---+'
   |             |            |   ^            ^
   v             v            v   |            |
REPEATED      ABORTED        CALLING >---------'

Note

If a directive fails in TaskWrapper.before() or TaskWrapper.after() the state will transition directly to FAILED and the system will stop
If @fail_if_skipped is used, then SKIPPED will transition to FAILED
The DEPENDENTS state will be skipped if there are no dependents
If a dependent Task fails, the parent is then ABORTED
RUNNING transits to CALLING if the Task calls build(). Normally this returns to RUNNING, but if the called Task fails, the calling task transitions to FAILED

alias of Enum

dovetail.engine.build(task, handle_exceptions=False)¶

Execute one task, optionally handling exceptions.

Parameters:	task (function or string) – The task to build. Either be the name of the Task, eg see Task names), or the function reference itself. handle_exceptions (boolean) – Default False. Configures how exceptions should be treated. See below
Returns:	The `Execution` call stack wrapped by a `Result`
Return type:	`Result`
Raises :	If handle_exceptions is False, `build()` will raise the Exception raised by the `Task` that failed

handle_exception behaviour:

False (default): If a Task throws an exception, or fails, the exception is propagated to the caller.
True: The any exception thrown is handled by the function and the exception is wrapped in the Result object. The result of the call can be checked with:
if result.success is False:
    # do error handling
    print result.exception

Warning

This function is for use within a task. Use run() to start a build as discussed in Running Dovetail

`loader` Module¶

A PEP 302 Python import hook for loading build scripts with classes that model the loaded packages and modules.

This behaves differently from the built-in loader by loading Python files from subdirectories without the presence of the __init__.py.

The loader is created on a base directory and searches by converting them into a directory under the base. For example:

>>> # assuming a file "build.py" in /path/to/build/root
>>> load("/path/to/build/root/build.py")
Building in /path/to/build/root
>>> # assuming build.py in directory ./path/to/directory under /path/to/build/root
>>> load("path/to/directory/build.py")
>>> # assuming @task declared on a function test() in build.py above
>>> build("path.to.directory.build:test")

class dovetail.loader.BuildLoader(base_directory)¶

Bases: object

A PEP 302 Python loader to import build scripts and their dependencies.

class DirectoryLoader(loader, location)¶

Bases: object

A class implementing a PEP 302 load_module import hook for a directory.

Directories loaded into Dovetail with this loader contain ONLY references to the packages loaded from their directory. Otherwise they are empty and do not have the __init__.py semantics

load_module(fullname)¶: PEP 302 load_module implementation

class BuildLoader.FileLoader(loader, file_name)¶

Bases: object

A class implementing a PEP 302 load_module import hook for a file.

Files can be loaded from any subdirectory of base_directory. Functions and classes can be referenced like:

>>> import path.to.file
>>> path.to.file.function()

Or:

>>> from path.to.file import function
>>> function()

load_module(fullname)¶: PEP 302 load_module implementation

BuildLoader.find_module(fullname, locations=None)¶: A PEP 302 import hook for BuildLoader

static BuildLoader.loaded()¶: Returns true if a BuildLoader import hook as been installed

static BuildLoader.register(base_directory)¶

Creates and registers a BuildLoader instance with the Python loader.

Parameters:	base_directory (string) – The base directory for the build. Files outside this directory will not load
Returns:	The build loaded singleton
Return type:	`BuildLoader`

Warning

This must be called precisely once

class dovetail.loader.BuildModule(fullname, file_name, module=None)¶

Bases: object

An object that represents a single build file loaded in Dovetail.

All build files must be loaded by Dovetail using load(). Dovetail will not run Task in files loaded by, for example, the import statement. Indeed, if the Task functions are called directly, Dovetail will decline to run them.

BuildModule works with PackageNode. The latter represent each loaded package. Leaves on the package tree may be modules. These modules are modelled by a BuildModule.

Parameters:	fullname (string) – The unqualified name of the module containing the build file (the name of the file, less the .py extension) file_name (string) – The path to the file being loaded

Attribute	Type	Description
name	string	The unqualified name of the module containing the build file (the name of the file, less the .py extension)
file_name	string	The file name of the loaded build file
module	`module`	Reference to the Python module containing the build. Note that `module.__file__` is equal to `file_name`
parent	`BuildModule`	The BuildModule that loaded (directly or indirectly) this BuildModule
tasks	dict of {string: `Task`}	A dict of task-name -> `Task` instance. The key is the short name for the `Task`
loaded	list of `BuildModule`	A list of BuildModule instances which this build file directly (or indirectly) loaded. The order of the list is the order the dependencies where loaded.

base_directory = '/Users/aalcock/Development/dovetail/env/bin'¶

Type :	string

The base directory for the build

static current()¶

Get the build module that’s currently running.

Returns:	the highest `BuildModule` found in the stack.
Return type:	`BuildModule`

static find(module)¶

Finds the BuildModule for the Python module argument

Parameters:	module (Python `module` instance or the name of the module) – The module to search for
Returns:	The `BuildModule` that wraps the `module`
Return type:	`BuildModule`
Raises :	`NoSuchModule` if the module was not loaded by Dovetail

find_task(name)¶

Searches for task name in this BuildModule and it’s children.

Parameters:	name (string) – The name of a `Task`. Accepts both non-qualified and fully-qualified `Task` names
Returns:	The task
Return type:	`Task`
Raises :	`NoSuchTask`

get_child_tasks_with_name(name)¶

Interrogates children directly loaded by this module and returns a list of Tasks with the name given in the argument.

This method allows a Task, say ‘clean’, in a parent build file to automatically call ‘clean’ in each of its descendants.

Parameters:	name (string) – The name of a `Task`. The name must be the function name, not the fully qualified name.
Returns:	All the tasks with that name under this `BuildModule`
Return type:	list of `Task` (may be an empty list)

get_relative_file_name()¶: Returns the file name, relative to the build root, of this module

static register_task(task)¶

Register a task with the BuildModule database.

Parameters:	task (`Task`) – The task that has just been loaded
Raises :	`NoSuchModule` if the task was declared in a file that was not loaded via `load()`

static report_load_graph()¶: Print a tree of all build files (modules) loaded by calls to load()

root = <dovetail.loader.BuildModule object at 0x10cac1fd0>¶

Type :	`BuildModule`

A reference to the build’s root build file

class dovetail.loader.PackageNode(full_name)¶

A data structure for describing package structure and which packages are associated with BuildModule instances.

PackageNode is a model of the tree structure of the build files loaded by Dovetail and their relative directory locations. It closely resembles the package structure in Python and uses the same dotted notation as Python.

Parameters:	full_name (string) – The full name of the package being loaded

Attribute	Type	Description
name	string	The non-qualified name of this file
full_name	string	The full Python-style package name for this package, eg for top/subdir/build.py, full name is top.subdir.build
parent	`PackageNode`	The parent of this instance
children	set of `PackageNode`	The child packages of this package
build_module	`BuildModule`	For a `PackageNode` representing a file (rather than directory) this points to the corresponding `BuildModule`, otherwise None

add(node)¶

Adds a PackageNode as a child of this one.

Parameters:	node (`PackageNode`) – The `PackageNode` to be added to `children`

child(name)¶

Find a child of the given name.

Parameters:	name (string) – The name to search for in `children`
Returns:	The child `PackageNode` if present, else None
Return type:	`PackageNode`

delete()¶: Recursively removes this node and children from the graph

static find(module)¶

Finds a PackageNode instance for the given package.

Parameters:	module (string or `BuildModule`) – Name of the package to find
Returns:	The associated `PackageNode`
Return type:	`PackageNode`
Raises :	`KeyError` if there is no such `PackageNode`

The PackageNode graph does not include packages that are loaded but are independent of the build files loaded by load() (and have corresponding BuildModule instances)

static get_root()¶: Returns the root PackageNode of the package graph

static invalidate()¶

Invalidates the package graph forcing re-discovery next time get_root is called.

This is called whenever the PEP 302 hook ‘BuildLoader’ detects that import has been called

is_leaf()¶: Returns True if the node is a leaf (eg no PackageNodes have been loaded by this node

static report_modules()¶: Prints a graph of the PackageNode instances to the console

dovetail.loader.is_loaded()¶: Returns whether Dovetail has loaded itself and is ready to load directives

dovetail.loader.load(file_path)¶

Load a build file which may be specified as a relative or absolute path (relative paths are relative to the file of the calling script).

Load will only load files in or under the build root (which is set as the directory containing the first build file loaded by Dovetail).

Paths may be specified with either forward or backward slash.

>>> # Load another.py in the same directory as the calling script
>>> load("another.py")

>>> # Load another.py in directory 'dir' under the directory of the calling
>>> # script:
>>> load("dir/another.py")
>>> # or
>>> load("dir\another.py")

>>> # Load a specific file:
>>> load("/path/to/build/root/build.py")

The very first call to load() should either supply an absolute path which will set the build’s base directory. Failing this, the current working directory is used to establish the build root.

dovetail.loader.load_from_import()¶

`main` Module¶

Command-line entry-point.

dovetail.main.execute_in_vm(module, tasks, reports)¶

Execute the build in the current Python VM.

Parameters:	module (sys.module) – A reference to the module of the build root tasks (list of string) – One or more tasks to run. Can be non-qualified task names if they are present in build_file, or fully-qualified task names for those in other files reports (list of string) – The reports to run after the build completes
Returns:	0 if successful, 1 if the build failed
Return type:	int
Raises :	`Terminate` if Dovetail itself crashes. Build steps will not raise this exception.

Note

execute_in_vm() calls stamp_environment() to set a number of environment variables. These can be used by the build script to better understand the environment

dovetail.main.main(*kw)¶

Dovetail main entry point.

Exceptions thrown due to errors in command line arguments or system errors are caught and an appropriate message and/or banner is displayed to the user rather than cryptic stack traces, and Dovetail exits the Python interpreter.

dovetail.main.print_reports(result, reports)¶: Print all the reports specified in the second argument to the console on the Results passed as the first argument.

dovetail.main.report_tasks(module)¶: Report the tasks in the specified module in a simple one-level tree

dovetail.main.run(tasks)¶

Used to run tasks from in a build script when the script has been loaded directly by Python.

When a script is run in the following way:

$ python build.py task1 task2

It can invoke Dovetail functionality using this fragment:

from dovetail import *
import sys

...

if __name__ == "__main__":
    run(sys.argv[1:])

Parameters:	tasks (list of string) – One or more tasks to run. Can be non-qualified task names if they are present in build_file, or fully-qualified task names for those in other files
Returns:	0 if successful, 1 if the build failed
Return type:	int
Raises :	`Terminate` if Dovetail itself crashes. Build steps will not raise this exception.

Note

If you want to simply execute a task from within another task, use the: build() method.

Warning

Read the warnings in Running Dovetail

dovetail.main.task_help(build_file, tasks)¶: Validate arguments and attempt to print help on the tasks

dovetail.main.validate_tasks(task_list, module=None)¶

Processes the list of Task targets into a list of normalized task names.

The method also validates the list of tasks in the module name provided.

`model` Module¶

The object model for Dovetail.

class dovetail.model.Dependencies¶

Bases: object

A simple dictionary-based store for recording dependencies of one Task on another.

This is an internal object used by the depends() directive and the Task class.

This class does not handle automatic dependencies (see Automatic dependencies).

static add(task_or_func, *dependencies)¶

Adds a list of dependencies to a Task.

Parameters:	task_or_func (`Task` or string) – A task defined either by name or by its function dependencies (list of function or string) – The dependant tasks defined by name or function

static get(task)¶

Retrieves a task’s dependencies.

Parameters:	task (`Task`) – The task for which dependencies are sought
Returns:	list of dependencies for a Task, or the empty list if none.
Return type:	list of function or string

class dovetail.model.LoadedTask(func)¶

Bases: dovetail.model.Task

A minor specialization of Task that registers with BuildModule so that its dependencies can be resolved.

dependencies()¶

Gets this task’s dependencies.

Returns:	The `Task` instances on which this instance depends. If no dependencies, it will be the empty list
Return type:	list of `Task`

Note

This method returns the tasks of the same name in child build modules.

dovetail.model.NULL_TASK = NullTask singleton¶: A reference to the NullTask singleton

class dovetail.model.NullTask¶

Bases: dovetail.model.Task

A logical root of the Task dependency hierarchy when the user specifies multiple Tasks from the command line. This allows the Task graph to be logically single-rooted.

This object is stored in dovetail.model.NULL_TASK

dependencies()¶

Returns a list of Task instances on which this Task depends.

This includes tasks of the same name in child modules

set_logical_module(module)¶

Called to anchor the Null Task as logically associated with the build file specified on the command line.

This is needed so that children of the NullTask have a build root.

class dovetail.model.Task(func)¶

Bases: object

An object representing a Task in a build system.

Tasks are functions, taking no arguments, identified with the ‘@task’ directive. A task’s short name is the name of the function. A hello-world for Dovetail is:

from dovetail import task

@task
def hello_world():
    pass

Parameters:	func (function()) – A reference to the task’s function

Tasks have the following members:

Attribute	Type	Description
module	module	The module in which the Task function is declared
func_name	string	The name of the function (f.__name__)
name	string	The fully qualified name of the Task
source	string	The absolute path of the file in which the Task is declared
dir	string	The directory in which source is situated

At instantiation time, the task instance captures the name and module of the task’s function - it does not capture a reference to the function. This approach means the function can be decorated or otherwise modified after the @task decorator is processed and the Task instance will ‘see’ the additional decoration.

At the time of resolution in get_function(), the module’s namespace is interrogated for the function name.

A task (in the build sense) is modelled by a combination of classes:

Class	Description
`Task`	The class models the core aspects of a task and provides a in-memory datastore for all tasks.
`TaskWrapper`	A algorithmic class that captures the wrapping behaviour of directives such as @Env and @do_if
`Dependencies`	A data store capturing all the dependencies of every task loaded. Dependencies are not resolved until runtime to allow forward references in build files. The class provides an API to look up any task’s dependencies.
`TaskWorkingDirectories`	A data store capturing the path specified in the @cwd directive. This is needed because the Task has not been created when the @cwd directive is processed.

Implementation note: There is a special Null Task always present that is used to to logically root a graph of Tasks. This can be retrieved from dovetail.model.NULL_TASK.

static all_tasks()¶: Returns a set of all Task instances known to Dovetail.

static as_task_name(func)¶

A utility to create a Task name for a function.

Parameters:	func (function()) – The function
Returns:	Task name. The name is typically of the form “module:function”, but module name is dropped for the `__main__` module
Return type:	string

build_dep_tree()¶

Builds out a tree of Task dependencies for reporting.

Returns:	A directed graph of `Task` dependencies
Return type:	`Node`

Node Class
Attribute	Type	Description
task	function() or string	The `Task`
children	list of function() or string	Dependencies of the `task`

dependencies()¶

Returns a list of Task instances on which this Task depends.

Returned dependencies include both the explicit dependencies created by @depends and stored in Dependencies, and also automatic dependencies between tasks of the same name in different files (see Automatic dependencies).

This is an abstract method implemented in:

LoadedTask.dependencies()

NullTask.dependencies()

static find(name_or_function, module=None)¶

Return the Task instance for the task of this name or function.

The Task can be looked up by passing a function reference - this is the easiest approach. You can also look up by Task name, which is relative to the package you are in. Tasks in the same file have the same name as the function. Otherwise Tasks declared at or below this use a relative package naming scheme, eg “package:function_name” - see Task names

The module argument can be specified to determine the starting point of the relative name search. If None, the call stack is examined to start the search from the file highest on the stack loaded by dovetail.load().

Parameters:	name_or_function (function() or string) – The name of the `Task` or the `Task` name module (module) – Default None. Optional `module` to guide the search for the task.
Returns:	The requested `Task`
Return type:	`Task`
Raises :	`NoSuchTask`

Note

If using the module argument, only use the short name for the task
If the task is specified by function, the module argument is ignored

get_doc()¶

Gets the Task instance’s documentation string.

Returns:	The reformatted docstring from the Task’s implementation
Return type:	string

get_function()¶

Returns the function implementing this task.

This is looked up by name within the module namespace at the point of request.

get_working_directory()¶

Returns the directory in which the Task wants to be run.

Returns:	The `Task`‘s working directory
Return type:	string

This defaults to the Task.dir, but can be set directly. The working directory has several levels of default, in this order:

Value set using TaskWorkingDirectories or the cwd() directive

Task.dir

If TaskWorkingDirectories or cwd() explicitly sets the working directory to None (eg @cwd(None)), then the the Dovetail engine switches of working directory functionality for this task, and the task will execute in the working directory of the calling task.

get_wrappers()¶

Returns the TaskWrapper instances associated with this Task.

Returns:	The directives
Return type:	list of `TaskWrapper`

report()¶: Prints a report of this Task and all its children, recursively; each Task’s docstring is printed

report_line(full_name=True)¶: Writes one line in the Task report report - the Task name and docstring

class dovetail.model.TaskWorkingDirectories¶

Bases: object

A simple dictionary for storing @cwd directive working directory values and their retrieval.

This class is necessary as the Task has not been fully created when the @cwd decorator is executed.

See Task.get_working_directory() for more details

static get(task)¶: Retrieves the working directory set by @cwd for a Task; if not present, it throws KeyError

static set(task_or_func, directory)¶

Sets a working directory value for a specific Task.

Working directories may be declared by Task instance or by their function

class dovetail.model.TaskWrapper(name, func, before, after)¶

Bases: object

TaskWrappers are abstractions of directives that have been applied to a Task, eg @adjust_env().

A few directives are implemented directly in the Dovetail Processor and are not present as TaskWrapper instances:

@task

@depends

@cwd

Directive implementation contains a pre-Task function or a post-Task function or both. The pre- and post-Task functions may interrogate the environment, modify it, perform any calculation or modify the state of execution (eg by calling Execution.skip() or Execution.fail())

Parameters:

name (string) – Task name
func (function()) – The Function that underlies a Task. Mandatory
before (function(Execution)) – Default None. Function called before the Task and Dependencies Return result ignored.
after (function(Execution, result)) – Default None. Function called after the Task and Dependencies. Second argument, result, will be the return value from the Task or None. The return result ignored by Dovetail.

WRAPPERS = {}¶

after(execution, result)¶: Executes the after() method of the wrapper (if any)

before(execution)¶: Executes the before() method of the wrapper (if any)

static decorate(name, func, before=None, after=None)¶

Allows a zero-argument directive to register its pre- and post- functions.

Parameters:	func (function()) – The function (`Task`) being decorated by the directive before (function(`Execution`)) – Default None. Function called before the Task and Dependencies Return result ignored. after (function(`Execution`, result)) – Default None. Function called after the Task and Dependencies. Second argument, result, will be the return value from the Task or None. The return result ignored by Dovetail.
Returns:	func
Return type:	function()

Note

This method has the important side-effect that it creates and registers a TaskWrapper that will be used by the Dovetail engine

static decorator_maker(name, before=None, after=None)¶

Allows a directive with arguments to register its pre- and post- functions.

This method returns a decorator which uses by TaskWrapper.decorate().

Parameters:	name (string) – Name of the `Task` to wrap before (function(`Execution`)) – Default None. Function called before the Task and Dependencies Return result ignored. after (function(`Execution`, result)) – Default None. Function called after the Task and Dependencies. Second argument, result, will be the return value from the Task or None. The return result ignored by Dovetail.
Returns:	A decorator of the `TaskWrapper`
Return type:	function(function)

static get(task)¶

Gets a list of TaskWrappers for this Task.

Parameters:	task (`Task`) – A Task
Returns:	The TaskWrappers for task
Return type:	list of `Task`

dovetail.model.null()¶: The function used to create the NullTask held in dovetail.model.NULL_TASK

`test_config` Module¶

py.test tests for config.py

class dovetail.test_config.TestConfig¶

Bases: object

test_apply_args()¶

test_apply_environment()¶

test_build_file()¶

test_clear()¶

test_illegal_file()¶

test_illegal_option()¶

test_load()¶

test_load_missing()¶

test_load_negative()¶

test_load_not_config()¶

test_logger()¶

test_nested()¶

test_reports()¶

test_task()¶

test_virtualenv()¶

dovetail.test_config.setup_module()¶

dovetail.test_config.teardown_module()¶

dovetail.test_config.write_file(file_name, content)¶

`virtual` Module¶

Interface to virtualenv.

The main purpose is to support running the build in:

An existing virtual environment

A new virtual environment

A freshly recreated virtual environment

dovetail.virtual.BOOTSTRAP = '\n# Check to see if Dovetail is running\nimport sys\nimport os\ntry:\n import dovetail\n print "Dovetail is already installed"\nexcept ImportError:\n print "Dovetail is not installed in virtual environment. Modifying system path"\n path = sys.argv[1]\n sys.path.insert(0, path)\n try:\n import dovetail\n print "Dovetail loaded from", path\n except ImportError as exception:\n print "Path", path, "did not work - got exception:", exception\n print "Failing build"\n exit(254)\n\nvirtualenv = os.path.dirname(os.path.dirname(sys.executable))\ndovetail.util.utilities.set_virtual_environment(virtualenv)\ndovetail.main.main(*sys.argv[2:])\n'¶: A script used to bootstrap Dovetail in a virtualenv sub-process

dovetail.virtual.build_command_line(python, bootstrap, arguments)¶

Produce a command line for calling a virtual environment.

Parameters:	python (string) – Path to the Python executable in a virtual environment bootstrap (string) – Path to the bootstrap.py file which bootstraps the build in the virtual environment arguments (list of string) – Additional command line arguments
Returns:	The command line to bootstrap the build in a virtual environment
Return type:	string

This is basically the same as the command line which called this, but we remove the virtual environment arguments

dovetail.virtual.find_self_on_sys_path()¶

Returns the sys.path entry that loaded this module (dovetail)

Returns:	Path to `dovetail`
Return type:	string

dovetail.virtual.get_virtualenv_command(directory, clear=False)¶

Returns the command to create a virtualenv environment.

Parameters:	directory (string) – A path to the virtual environment to be tested clear (boolean) – Erase any pre-existing environment?
Returns:	A command suitable for executing with `subprocess.call()`
Return type:	list of string

dovetail.virtual.install_virtualenv()¶

Attempts to install and validate virtualenv returning the path to the virtualenv executable.

If the installation fails, the function raises InvalidEnvironment exception.

Returns:	The full path to the newly-created virtualenv executable
Return type:	string
Raises :	`InvalidEnvironment`

dovetail.virtual.prepare_virtual(proposed, clear)¶

Prepare the proposed virtual environment, returning the path to the Python executable to use.

This function will call virtualenv to create the environment if either the environment is not present, or the –clear option was specified.

Parameters:	proposed (string) – A path to the virtual environment to be created or used clear (boolean) – Erase any pre-existing environment?
Returns:	The full path to the virtualenv executable under proposed
Return type:	string
Raises :	`InvalidEnvironment`

dovetail.virtual.probe_existing(directory, python_exe)¶

Validates the virtual environment to ensure it can be used, raising an exception if it is not.

Parameters:	directory (string) – A path to the virtual environment to be tested python_exe (string) – A path to the Python executable in the environment
Raises :	`InvalidEnvironment`

dovetail.virtual.probe_new(directory, python_exe)¶

Validates whether we can create a virtual environment - can we write to the specified directory?

Parameters:	directory (string) – A path to the virtual environment to be tested python_exe (string) – A path to the Python executable in the environment
Raises :	`InvalidEnvironment`

dovetail.virtual.run_virtual(virtualenv, clear, kw)¶

Runs the test in a virtualenv environment, creating and clearing it if necessary.

Parameters:	virtualenv (string) – A path to a existing or to-be-built virtual environment clear (boolean) – Erase any pre-existing environment? kw (list of string) – Command line arguments
Returns:	The return of the `subprocess.call()` process
Return type:	int

The arguments from the original invocation should be passed untouched and will (after manipulation) be passed to the virtual environment

dovetail.virtual.write_bootstrap(directory)¶

Creates a bootstrap Python file in a directory, returning the path to the bootstrap file.

Parameters:	directory (string) – Path to directory in which to write the bootstrap
Returns:	Path to the bootstrap.py file
Return type:	string

dovetail Package¶