directives Package

directives Package

This package contains the Dovetail’s core ‘language’ and markup for build scripts.

Directives

Fundamental directives for declaring Tasks, dependencies and conditions to execute a Task:

Name Parameters Description
task() none Declares a function to be a build dovetail.model.Task. Once declared, the Task can be referenced either by passing a reference to the function itself, or by it’s name.
depends() 1 or more Task names or function references Declares that a Task is dependent on another Task.
do_if() 1 or more predicates Executes a Task if predicate returns True, otherwise Task is skipped.
skip_if() 1 or more predicates Skips a Tasks if predicate returns True, otherwise Task is executed

This collection of directives allow the build engineer to fail a build when specific conditions occur:

Name Parameters Description
fail_if() 1 or more predicates

Fails the task if the predicate returns True. @fail_if is evaluated after the Task has otherwise completed. This is often combined with the following predicates:

fail_if_skipped() none Fails the task if the task was skipped, eg by @skip_if or @do_if. This can be useful if there are several @skip_if and/or @do_if predicates and you need to ensure that at least one allowed the task to execute
check_result() none Checks the return value of the task for Unix process semantics, failing the task if the result is non-null and non-zero

Directives that modify the execution environment of the Task:

Name Parameters Description
adjust_env() Environment definition arguments Modifies the os.environ for the duration of the Task and its dependents
cwd() directory Changes the current working directory to the nominated location before the task runs. This is reverted after the task is complete.
mkdirs() 1 or more directories Ensure that the directories exist, making them if not
requires() 1 or more package requirements Ensure that the Python packages are present, installing if necessary.

Predicates

Predicates are logical statements used to determine when a Task executes, is skipped or should be failed. Predicates are used by:

  • @do_if
  • @skip_if
  • @fail_if

A Predicate is a Callable which will be evaluated during the build and returns a boolean.

Predicates that test the os.environ:

Name Parameters Description
Env Environment definition arguments Test whether environment variables are set, or have a specific value
EnvRE 1 or more NAME=”regex” Test whether environment variables are match the supplied regular expressions

Predicates that test the file system:

Name Parameters Description
Access path, mode Test is a path has certain mode flags. Wrapper to os.access(). Note: On Windows, only os.R_OK is respected
IsDir path Tests if the path is a directory.
IsFile path Tests if the path is a normal file
IsLink path Tests if the path is a link
IsMount path Tests if the path is a mount-point (disk).
LargerThan path, size Tests if the file pointed to by path is larger than size bytes
Newer source iterator, target iterator Tests if any file in the source iterator is newer than any files in the target iterator. Useful if you want to skip a Task if nothing has changed. Often used in conjunction with Glob or Formic

Predicates for testing whether a package or executable is present:

Name Parameters Description
Installed requirement specifications Tests if the easy_install-like requirements are present.
Which executable_name Tests if the named executable is on the path.

Predicates to test a Task’s result:

Name Parameters Description
engine.StdErr None True if the current Task sent any output to sys.stderr
ResultEquals value True if the return value from the Task function equals the supplied argument
ResultNone None True if the return value from the Task function equals is None
ResultZero None True if the return value from the Task function is 0

Predicates to test the processing state of the Dovetail build engine:

Name Parameters Description
DependentSucceeded None True if the current Task has at least one dependent Task that succeeded. False if there are no subtasks
StateAborted None True if the current Task has been aborted (dependent failed)
StateFailed None True if the current Task has failed
StateRepeated None True if the current Task has already been executed successfully
StateSkipped None True if the current Task was skipped
StateStarted None True if the current Task has started but not yet finished or failed.
StateSucceeded None True if the current Task has succeeded

Logic

A number of Task decorators take Predicates as arguments; if the decorator takes multiple predicates they will be ‘anded’ together - True only if all predicates are True.

Predicates can be combined into arbitrary logical statements using logical operator predicates:

Operator Predicate Description
OR Any True iff any predicate is True
AND All True iff all predicates are True
NOT Not The negation of the single predicate argument

(iff stands for ‘if and only if’)

By combining these operators, you can make constructions like:

Not(Any(p1, p2, All(p3, p4)))

Environment definition arguments

Environment definitions are used by:

  • The Env predicate, and
  • The adjust_env() directive

Several methods use the *args and **kwargs argument magic to tersely specify:

  • Tests to environment, such as whether a specific environment variable is set, or has a specific value
  • Modifications to the environment, so environment variables can be set, modified or unset.

The *args section represents set or unset variables (either tests that a variable is set, or unsets a variable):

Env('BASH')               # True if environment variable $BASH is set
Env('BASH', 'HOSTNAME')   # True if $BASH and $HOSTNAME are both set
@adjust_env('A')          # Unsets $A
@adjust_env('A', 'B')     # Unsets $A and $B

The **kwargs represents the value of a specific environment variable:

Env(DOVETAIL_NODE='pluto.example.com')   # Run only on the pluto build machine
@adjust_env(A="new")                     # Sets $A=new

*args and **kwargs can be combined:

Env('BASH', DOVETAIL_NODE='pluto.example.com')

And finally, a **kwargs argument with a value of None is exactly the same as a *args entry. The following have identical behavior:

Env('BASH')
Env(BASH=None)

Modification dictionary

Modification dictionaries are used by:

This is a dictionary that is used to record or make changes to another dictionary, the target. It may:

  • Create new entries
  • Overwrite an existing entry
  • Delete an entry (a key)

The entries in the modification dictionary have the following semantics

  • Has value: the modification will either set or overwrite the entry with the same key in the target dictionary
  • No value: the modification will delete any entry with the same key in the target dictionary

Keys in the target not matched by an entry in the modification dictionary will not be modified.

core Module

The core directives.

class dovetail.directives.core.All(*predicates)

Bases: object

A predicate that returns True if all of its arguments are True.

Any instances are callable, optionally with an dovetail.engine.Execution. This predicate has the same semantics as Pythons all() function.

Instantiate with a comma separated list of predicates, or use Python’s *kw argument modifier:

a = All(p1, p2, p3)

Or:

l = [ p1, p2, p3 ]
a = All(*l)

Any’s value is evaluated on demand, not when it is instantiated:

print a()
predicates = None
Type :iterable
class dovetail.directives.core.Any(*predicates)

Bases: object

A predicate that evaluates to True if any of its arguments is True.

Any instances are callable, optionally with an dovetail.engine.Execution. This predicate has the same semantics as Pythons any() function.

Instantiate with a comma separated list of predicates, or use Python’s *kw argument modifier:

a = Any(p1, p2, p3)

Or:

l = [ p1, p2, p3 ]
a = Any(*l)

Any’s value is evaluated on demand, not when it is instantiated:

print a()
predicates = None
Type :iterable
class dovetail.directives.core.Not(predicate)

Bases: object

A negation predicate which wraps another predicate.

Not instances are callable, optionally with an dovetail.engine.Execution. Their return value is the boolean negation of the predicate which they wrap.

Not’s value is evaluated on demand, not when it is instantiated:

n = Not(Env("BASH"))
print p()    # Evaluation occurs here
predicate = None
Type :function or callable returning boolean
dovetail.directives.core.call_predicate(predicate, execution)

Calls a predicate, resolving whether the predicate uses the optional Execution argument, and returns the result of the call.

This method caches whether the predicate uses the optional argument in the PREDICATE_ARG_CACHE.

Parameters:
  • predicate (function, callable) – A function or callable which, when called, returns True or False
  • execution (Execution) – An Execution object containing current state
Returns:

The current value of the predicate

Return type:

boolean

dovetail.directives.core.check_result(f)

Checks the return of the decorated function as if it were a shell script - anything other than None or 0 is considered an error and an exception will be thrown.

This is similar to the longer:

@fail_if(Not(ResultZero()))

but issues a more specific message and exception

dovetail.directives.core.coalesce(predicate, predicates)

Internal function to produce a single list from a mandatory predicate and an optional list of further predicates.

Parameters:
  • predicate (function, callable) – A predicate
  • predicates (duck-typed iterable of Predicate) – A list or iterable of further Predicates, or None
Returns:

A list consisting of all Predicates

Rtype list:
dovetail.directives.core.depends(*tasks)

This is a declares that a task is dependent on another task completing before it can run.

Parameters:tasks (string, function) – A comma separated list of task names or functions
dovetail.directives.core.do_if(predicate, *predicates)

Executes this Task if all predicates are True.

Parameters:
  • predicate (function or callable returning boolean) – A predicate
  • predicates (function or callable returning boolean) – Optional; additional predicates
dovetail.directives.core.fail_if(predicate, *predicates, **kwargs)

Fails the Task if all the predicates are True with optional failure message.

The message may be assigned using **kwargs: message=’my message’ If given, this message will be the message of the FailIf exception raised if the predicates return True. If not given, a default message will be constructed.

fail_if runs after the main body of the Task has executed.

Parameters:
  • predicate (function or callable returning boolean) – A predicate
  • predicates (function or callable returning boolean) – Optional; additional predicates
  • kwargs – kwargs. See above
dovetail.directives.core.fail_if_skipped(f)

If the wrapped Task was skipped by any directive, the Task will be failed.

This is useful to force an error if the environment is, for example, not correct.

This is similar to the longer:

@fail_if(StateSkipped())

but issues a more specific message and exception

dovetail.directives.core.skip_if(predicate, *predicates, **kwargs)

Skips this Task if all predicates are True; and optional message can be specified to make the logic clearer.

The optional message is specified using **kwargs: message=’my message’. When given, this message will be logged to the Execution object (and the logging mechanism). This can be used to make complex logic more understandable in the output.

Parameters:
  • predicate (function or callable returning boolean) – A predicate
  • predicates (function or callable returning boolean) – Optional; additional predicates
  • kwargs – kwargs. See above
dovetail.directives.core.task(f)

The task directive declares that a function is a Dovetail Task and causes an Instance of LoadedTask (subclass of Task) to be instantiated and registered in the Dovetail model.

engine Module

Predicates about the processing of the engine.

class dovetail.directives.engine.DependentSucceeded

Bases: object

A predicate that returns True if the current task has at least one dependent Task that executed successfully.

This is useful to detect a Task whose dependents all skipped

class dovetail.directives.engine.ResultEquals(result)

Bases: object

A predicate that returns True if a task returns a specific value.

Parameters:result (any) – The value to test for
Returns:True if the result equals result
Return type:boolean
class dovetail.directives.engine.ResultNone

Bases: object

A predicate that returns True if a task returns None

class dovetail.directives.engine.ResultZero

Bases: object

A predicate that returns True if a task returns 0 - i.e. what Unix considers a successful execution

class dovetail.directives.engine.StateAborted

Bases: object

A predicate that returns True if the state of the current Task is engine.STATE.FAILED

class dovetail.directives.engine.StateFailed

Bases: object

A predicate that returns True if the state of the current Task is engine.STATE.FAILED

class dovetail.directives.engine.StateRepeated

Bases: object

A predicate that returns True if the state of the current Task is engine.STATE.FAILED

class dovetail.directives.engine.StateSkipped

Bases: object

A predicate that returns True if the state of the current Task is engine.STATE.SKIPPED

class dovetail.directives.engine.StateStarted

Bases: object

A predicate that returns True if the state of the current Task is engine.STATE.STARTED

class dovetail.directives.engine.StateSucceeded

Bases: object

A predicate that returns True if the state of the current Task is engine.STATE.SUCCEEDED

class dovetail.directives.engine.StdErr

Bases: object

A predicate that returns True if a task has produced output to sys.stderr

environment Module

Environment-based directives.

class dovetail.directives.environment.Env(*if_set, **if_equals)

Bases: object

A predicate that returns True if the environment meets all the specified criteria.

Parameters:
  • if_set – List of environment variables to test to see if they exist
  • if_equals – List of environment variable equality tests
Return type:

boolean

The if_set and if_equals parameters are more fully described at Modification dictionary.

class dovetail.directives.environment.EnvRE(**tests)

Bases: object

A Predicate for testing environment variables against regular expressions.

Parameters:tests (string) – List of NAME=RE arguments where NAME is the environment variable to be checked, and RE is the regular expression
Return type:boolean

The Python re.search() method is used, rather than re.match(), so the whole value of the environment variable is tested, rather than just the anchoring the search to the beginning of the value.

Examples:

>>> predicate = EnvRE(DOVETAIL_NODE="mercury")

A predicate that matches hostnames of mercury.example.com, mercuryrising.example.org, www.mercury.ru, etc

>>> predicate = EnvRE(DOVETAIL_NODE="mercury.*")

Same as previous

>>> predicate = EnvRE(DOVETAIL_NODE="^mercury\.")

A predicate that matches the host “mercury” in any domain.

>>> predicate = EnvRE(DOVETAIL_NODE="\.example\.com$")

A predicate than matches hosts in any subdomain of example.com

>>> predicate = EnvRE(A="^start", B="end$")

A predicate that returns True if the value of A starts with ‘start’ and the value of B ends with ‘end’

dovetail.directives.environment.adjust_env(*remove, **merge)

Temporarily sets environment variables for the duration of the decorated function.

Parameters:
  • remove (string) – Environment variables to unset
  • merge (string) – List of environment variables to set, the argument name being mapped to the environment variable name

The parameters are more fully described in Environment definition arguments.

For example:

@adjust_env("REMOVE", NEW='value')

will unset $REMOVE and set $NEW to ‘value’.

All environment variables are reset to their original values at the end of the task even if the calling function modified their value. Other modifications to the environment (for example new variables) are preserved.

dovetail.directives.environment.apply_environment(environment)

Internal method that applies a dictionary of values to the os.environ while recording the original values.

Parameters:environment (dict) – A Modification dictionary that will be applied to os.environ.
Returns:an object which, when passed to revert_environment(), will reset the environment variables modified by this function
dovetail.directives.environment.pp_adjust_env(environment)

Pretty-print a Modification dictionary.

Parameters:environment (dict) – The modification dictionary to print
dovetail.directives.environment.revert_environment(memento)

Resets the environment variables modified by apply_environment().

Parameters:memento – The return value from apply_environment()

Warning

Other environment variables modified after the call to apply_environment() and revert_environment() will be left unchanged.

Warning

If several ‘nested’ snapshots are takes, they must be reverted in the inverse order, otherwise the environment may not be restored as expected

dovetail.directives.environment.unset(key)

Unset an environment variable using del environ[key], ignoring any KeyError

files Module

Predicates and directives for the file system.

class dovetail.directives.files.Access(path, mode)

Bases: object

A predicate version of os.access().

Parameters:
  • path (string) – The path to test
  • mode (int) – Same semantics and values as os.access():
Returns:

Same as: os.access(path, mode)

Return type:

boolean

os provides some useful constants for mode:

  • os.R_OK
  • os.W_OK
  • os.X_OK
pp_mode()

Pretty-prints the instance

class dovetail.directives.files.Glob(glob)

Bases: object

A Container Object wrapper for glob.iglob.

Parameters:glob (string) – A fnmatch glob, eg “*.py”
Returns:An iterator over the files discovered by the glob
Return type:iterator

Usage:

>>> glob = Glob("*.txt")
>>> for file in glob:
...     print file
>>> for file in glob:
...     print file

Whenever the __iter__() method is called, the object returns a new iterator created by a call to glob.iglob(). This means that, unlike glob.iglob(), Glob can be iterated over any number of times.

class dovetail.directives.files.IsDir(path)

Bases: object

A predicate version of os.path.isdir().

Parameters:path (string) – The path to test
Returns:True if the path is a directory
Return type:boolean
class dovetail.directives.files.IsFile(path)

Bases: object

A predicate version of os.path.isfile().

Parameters:path (string) – The path to test
Returns:True if the path is a file
Return type:boolean

Bases: object

A predicate version of os.path.islink().

Parameters:path (string) – The path to test
Returns:True if the path is a link
Return type:boolean
class dovetail.directives.files.IsMount(path)

Bases: object

A predicate version of os.path.ismount().

Parameters:path (string) – The path to test
Returns:True if the path is a mount point
Return type:boolean
class dovetail.directives.files.LargerThan(path, size=1)

Bases: object

A predicate that returns True if the file is larger than the specified size.

Parameters:
  • path (string) – A path to a file
  • size (int) – Default 1; the size to test for, in bytes
Returns:

True if the file is larger than the specified size in bytes

Return type:

boolean

class dovetail.directives.files.Newer(source_iterator, target_iterator)

Bases: object

A Predicate for testing if some files are newer than another.

Example: This can be used to compile some sources only if they have changed since the last compile:

@do_if(Newer("src/*.py", "build/*.*"))
Parameters:
  • source_iterator (container) – A set of source files
  • target_iterator (container) – A set of target files
Returns:

True if the newest file in source_iterator is younger than the oldest file in target_iterator

Return type:

boolean

Newer computes the result using this pseudo-logic:

  1. Find the newest file in source_iter
    • Call file SOURCE
  2. Find the oldest file in target_iter
    • Call file TARGET
  3. Test age of SOURCE against age of TARGET
    • If SOURCE is newer (younger) than TARGET, return True
    • Else return False

Note

If the newest source_iter has exactly the same timestamp as the oldest file in the target_iter, Newer will return False.

Note

The iterator is likely to be called several times in a single build which means an iterator is insufficient - it must be a Container Object - an object with the __iter__() method: http://docs.python.org/library/stdtypes.html#container.__iter__:

  1. The arguments should evaluate only when called, not created, otherwise Newer will test only those files present when the build file was loaded (and files created or modified in the build may be missed)
  2. If the argument is an iterator/generator instance, then the second call to Newer will return the iterator after it has performed an iteration, so the iterator will yield no more files.

Dovetail provides a simple wrapper of glob.iglob() in the form of Glob:

source = Glob(os.path.join("src", "mypackage", "*.py"))
target = Glob(os.path.join("build", "doc", "mypackage*.rst"))

@task
@Newer(source, target)
def generate_api_doc():
    return call("sphinx-apidoc -f -o src/mypackage build/doc".split(' '))

In this example, Newer runs if any top-level .py file in src/mypackage is newer than the corresponding mypackage*.rst file in build/doc.

Newer works well with Formic (http://www.aviser.asia/formic), which allows an easy construction of rich (Apache Ant) globs. In the example below:

  • Source file are all Python files in all subdirectories of src/mypackage, excluding tests,
  • Target files are the corresponding mypackage*.rst files in build/doc
import formic
source = formic.FileSet(directory="src/mypackage",
                        include="*.py",
                        exclude=["**/*test*/**", "test_*"]
                        )
target = formic.FileSet(directory="build/doc",
                        include="mypackage*.rst")

@task
@Newer(source, target)
def generate_api_doc():
    return call("sphinx-apidoc -f -o src/mypackage build/doc".split(' '))
static best(file_iterator, newer=True)

Returns the timestamp of the newest (or oldest) file in the file iterator argument.

Parameters:
  • file_iterator (iterator) – An iterator whose items are file paths
  • newer (boolean) – Defaults True. Determines if the selector is for the newest (newer=True) or oldest (newer=False) file
Returns:

The timestamp of the newest (or oldest) file in file_iterator

Return type:

the os.path.getmtime() value of the file

class dovetail.directives.files.Which(executable_name)

Bases: object

A predicate for testing whether an executable is present or on the path.

Parameters:executable_name (string) – The file (base) name to look for
Return type:boolean

Uses a Python implementation of the Unix which command: dovetail.util.which.which()

dovetail.directives.files.cwd(directory)

The wrapped Task will be run in the specified working directory, and the current working directory restored on completion.

Parameters:directory (string) – The directory to run the Task in; if None, Dovetail will not set _any_ directory for this Task, but leave the Task in the most recently set directory.

Implementation note: cwd() uses the TaskWorkingDirectories.set() method to store the preferred directory.

dovetail.directives.files.mkdirs(*required_dirs)

Ensures that the give path(s) are present before the decorated function is called, creating if necessary.

Parameters:required_dirs (string) – A number of paths. For each path that does not exist, os.makedirs() is called to create it

packages Module

Directives for package/egg requirements and resolution

class dovetail.directives.packages.Installed(requirement, *requirements)

Bases: object

A predicate that returns True if all requirements are met in the Python environment.

Parameters:
  • requirement (string) – A requirement specifications as per easy_install, eg: “pylint” and “coverage>3”
  • requirements (string) – Additional requirements (optional)
Returns:

True if all specified requirements are satisfied

Return type:

boolean

dovetail.directives.packages.install(requirements)

Uses setuptools.commands.easy_install to install a series of package requirements and adjust the system path so they are immediately available.

Parameters:requirements (Either a string or a list of string) – Requirement specifications as per easy_install, eg: “pylint” and “coverage>3”
Raises :dovetail.util.MissingRequirement if easy_install cannot locate or install the requirement

Note

If you need to set specific easy_install behaviour, such as loading from a local host, then modify the easy_install configuration as described in:

dovetail.directives.packages.not_present(requirements, stop_on_error=True)

Checks to see which requirements are currently satisfied, returning a list of those requirements not satisfied.

Parameters:
  • requirements (list of string) – Requirements in the form “pylint” or “coverage>3”
  • stop_on_error (boolean) – Default True; if True, if pkg_resources.VersionConflict is raised, this is propagated to the caller. Otherwise the exception is handled.
Returns:

A list of unsatisfied requirements - may be empty

Return type:

list of string

Raises :

pkg_resources.VersionConflict if a requirement is in conflict with the environment

dovetail.directives.packages.pp_requirements(requirements)

Pretty-print a list of requirements.

Parameters:requirements (list of string) – Requirements in the form “pylint” or “coverage>3”
Return type:string
dovetail.directives.packages.requires(*requirements)

Ensures all package requirements are installed before executing Task.

Parameters:requirements (string) – One or more requirement specifications as per easy_install, eg: “pylint” and “coverage>3”
Raises :dovetail.util.exception.MissingRequirement if easy_install cannot locate or install the requirement
Raises :pkg_resources.VersionConflict if a requirement is in conflict with the current environment.=

Note

If you need to set specific easy_install behaviour, such as loading from a local host, then modify the easy_install configuration as described in:

test_environment Module

py.test test script for environment.py

class dovetail.directives.test_environment.TestEnvironment

Bases: object

testApplyAddition()
testApplyModification()
testApplyNothing()
testRemoveVariables()
testRevertNothing()
test_env_predicate()
test_env_re_predicate()
test_pp_adjust()
test_pp_test()
dovetail.directives.test_environment.filter(d)

Filters a dictionary to include only those keys we are testing for

dovetail.directives.test_environment.setup_module()

test_files Module

py.test test script for predicates in core.py, environment.py and files.py.

NOTE: Windows file access permissions is, ahem, somewhat different from Posix. As a result, we test ONLY THE READ BIT.

class dovetail.directives.test_files.TestFiles

Bases: object

test_access_base()
test_access_exec_only()
test_access_print()
test_access_private_only()
test_access_read_only()
test_access_write_only()
test_dir_file()
test_is_mount()
test_larger()
test_newer()
test_sanity()
test_which_predicate()
dovetail.directives.test_files.set(base, file_name, add=0, remove=0)
dovetail.directives.test_files.setup_module(module)
dovetail.directives.test_files.teardown_module(module)

test_packages Module

py.test test script for packages.py

class dovetail.directives.test_packages.TestModules

Bases: object

test_cannot_install()
test_install()
test_mixed()
test_multiple_not_present()
test_not_present()
test_predicate()
test_present()
test_version_conflict()
dovetail.directives.test_packages.setup_module(module)

test_predicates Module

py.test test script for predicates in core.py, environment.py and files.py

class dovetail.directives.test_predicates.AlwaysFalse

Bases: object

class dovetail.directives.test_predicates.AlwaysTrue

Bases: object

class dovetail.directives.test_predicates.TestLogic

Bases: object

test_all()
test_any()
test_basics()

Make sure primitives work

test_not()
test_predicate_function()
dovetail.directives.test_predicates.always_false()
dovetail.directives.test_predicates.always_true()