This package contains the Dovetail’s core ‘language’ and markup for build scripts.
Fundamental directives for declaring Tasks, dependencies and conditions to execute a Task:
Name | Parameters | Description |
---|---|---|
task() | none | Declares a function to be a build dovetail.model.Task. Once declared, the Task can be referenced either by passing a reference to the function itself, or by it’s name. |
depends() | 1 or more Task names or function references | Declares that a Task is dependent on another Task. |
do_if() | 1 or more predicates | Executes a Task if predicate returns True, otherwise Task is skipped. |
skip_if() | 1 or more predicates | Skips a Tasks if predicate returns True, otherwise Task is executed |
This collection of directives allow the build engineer to fail a build when specific conditions occur:
Name | Parameters | Description |
---|---|---|
fail_if() | 1 or more predicates | Fails the task if the predicate returns True. @fail_if is evaluated after the Task has otherwise completed. This is often combined with the following predicates: |
fail_if_skipped() | none | Fails the task if the task was skipped, eg by @skip_if or @do_if. This can be useful if there are several @skip_if and/or @do_if predicates and you need to ensure that at least one allowed the task to execute |
check_result() | none | Checks the return value of the task for Unix process semantics, failing the task if the result is non-null and non-zero |
Directives that modify the execution environment of the Task:
Name | Parameters | Description |
---|---|---|
adjust_env() | Environment definition arguments | Modifies the os.environ for the duration of the Task and its dependents |
cwd() | directory | Changes the current working directory to the nominated location before the task runs. This is reverted after the task is complete. |
mkdirs() | 1 or more directories | Ensure that the directories exist, making them if not |
requires() | 1 or more package requirements | Ensure that the Python packages are present, installing if necessary. |
Predicates are logical statements used to determine when a Task executes, is skipped or should be failed. Predicates are used by:
- @do_if
- @skip_if
- @fail_if
A Predicate is a Callable which will be evaluated during the build and returns a boolean.
Predicates that test the os.environ:
Name | Parameters | Description |
---|---|---|
Env | Environment definition arguments | Test whether environment variables are set, or have a specific value |
EnvRE | 1 or more NAME=”regex” | Test whether environment variables are match the supplied regular expressions |
Predicates that test the file system:
Name | Parameters | Description |
---|---|---|
Access | path, mode | Test is a path has certain mode flags. Wrapper to os.access(). Note: On Windows, only os.R_OK is respected |
IsDir | path | Tests if the path is a directory. |
IsFile | path | Tests if the path is a normal file |
IsLink | path | Tests if the path is a link |
IsMount | path | Tests if the path is a mount-point (disk). |
LargerThan | path, size | Tests if the file pointed to by path is larger than size bytes |
Newer | source iterator, target iterator | Tests if any file in the source iterator is newer than any files in the target iterator. Useful if you want to skip a Task if nothing has changed. Often used in conjunction with Glob or Formic |
Predicates for testing whether a package or executable is present:
Name | Parameters | Description |
---|---|---|
Installed | requirement specifications | Tests if the easy_install-like requirements are present. |
Which | executable_name | Tests if the named executable is on the path. |
Predicates to test a Task’s result:
Name | Parameters | Description |
---|---|---|
engine.StdErr | None | True if the current Task sent any output to sys.stderr |
ResultEquals | value | True if the return value from the Task function equals the supplied argument |
ResultNone | None | True if the return value from the Task function equals is None |
ResultZero | None | True if the return value from the Task function is 0 |
Predicates to test the processing state of the Dovetail build engine:
Name | Parameters | Description |
---|---|---|
DependentSucceeded | None | True if the current Task has at least one dependent Task that succeeded. False if there are no subtasks |
StateAborted | None | True if the current Task has been aborted (dependent failed) |
StateFailed | None | True if the current Task has failed |
StateRepeated | None | True if the current Task has already been executed successfully |
StateSkipped | None | True if the current Task was skipped |
StateStarted | None | True if the current Task has started but not yet finished or failed. |
StateSucceeded | None | True if the current Task has succeeded |
A number of Task decorators take Predicates as arguments; if the decorator takes multiple predicates they will be ‘anded’ together - True only if all predicates are True.
Predicates can be combined into arbitrary logical statements using logical operator predicates:
Operator | Predicate | Description |
---|---|---|
OR | Any | True iff any predicate is True |
AND | All | True iff all predicates are True |
NOT | Not | The negation of the single predicate argument |
(iff stands for ‘if and only if’)
By combining these operators, you can make constructions like:
Not(Any(p1, p2, All(p3, p4)))
Environment definitions are used by:
- The Env predicate, and
- The adjust_env() directive
Several methods use the *args and **kwargs argument magic to tersely specify:
- Tests to environment, such as whether a specific environment variable is set, or has a specific value
- Modifications to the environment, so environment variables can be set, modified or unset.
The *args section represents set or unset variables (either tests that a variable is set, or unsets a variable):
Env('BASH') # True if environment variable $BASH is set
Env('BASH', 'HOSTNAME') # True if $BASH and $HOSTNAME are both set
@adjust_env('A') # Unsets $A
@adjust_env('A', 'B') # Unsets $A and $B
The **kwargs represents the value of a specific environment variable:
Env(DOVETAIL_NODE='pluto.example.com') # Run only on the pluto build machine
@adjust_env(A="new") # Sets $A=new
*args and **kwargs can be combined:
Env('BASH', DOVETAIL_NODE='pluto.example.com')
And finally, a **kwargs argument with a value of None is exactly the same as a *args entry. The following have identical behavior:
Env('BASH')
Env(BASH=None)
Modification dictionaries are used by:
This is a dictionary that is used to record or make changes to another dictionary, the target. It may:
- Create new entries
- Overwrite an existing entry
- Delete an entry (a key)
The entries in the modification dictionary have the following semantics
- Has value: the modification will either set or overwrite the entry with the same key in the target dictionary
- No value: the modification will delete any entry with the same key in the target dictionary
Keys in the target not matched by an entry in the modification dictionary will not be modified.
The core directives.
Bases: object
A predicate that returns True if all of its arguments are True.
Any instances are callable, optionally with an dovetail.engine.Execution. This predicate has the same semantics as Pythons all() function.
Instantiate with a comma separated list of predicates, or use Python’s *kw argument modifier:
a = All(p1, p2, p3)
Or:
l = [ p1, p2, p3 ]
a = All(*l)
Any’s value is evaluated on demand, not when it is instantiated:
print a()
Type : | iterable |
---|
Bases: object
A predicate that evaluates to True if any of its arguments is True.
Any instances are callable, optionally with an dovetail.engine.Execution. This predicate has the same semantics as Pythons any() function.
Instantiate with a comma separated list of predicates, or use Python’s *kw argument modifier:
a = Any(p1, p2, p3)
Or:
l = [ p1, p2, p3 ]
a = Any(*l)
Any’s value is evaluated on demand, not when it is instantiated:
print a()
Type : | iterable |
---|
Bases: object
A negation predicate which wraps another predicate.
Not instances are callable, optionally with an dovetail.engine.Execution. Their return value is the boolean negation of the predicate which they wrap.
Not’s value is evaluated on demand, not when it is instantiated:
n = Not(Env("BASH"))
print p() # Evaluation occurs here
Type : | function or callable returning boolean |
---|
Calls a predicate, resolving whether the predicate uses the optional Execution argument, and returns the result of the call.
This method caches whether the predicate uses the optional argument in the PREDICATE_ARG_CACHE.
Parameters: |
|
---|---|
Returns: | The current value of the predicate |
Return type: | boolean |
Checks the return of the decorated function as if it were a shell script - anything other than None or 0 is considered an error and an exception will be thrown.
This is similar to the longer:
@fail_if(Not(ResultZero()))
but issues a more specific message and exception
Internal function to produce a single list from a mandatory predicate and an optional list of further predicates.
Parameters: |
|
---|---|
Returns: | A list consisting of all Predicates |
Rtype list: |
This is a declares that a task is dependent on another task completing before it can run.
Parameters: | tasks (string, function) – A comma separated list of task names or functions |
---|
Executes this Task if all predicates are True.
Parameters: |
|
---|
Fails the Task if all the predicates are True with optional failure message.
The message may be assigned using **kwargs: message=’my message’ If given, this message will be the message of the FailIf exception raised if the predicates return True. If not given, a default message will be constructed.
fail_if runs after the main body of the Task has executed.
Parameters: |
|
---|
If the wrapped Task was skipped by any directive, the Task will be failed.
This is useful to force an error if the environment is, for example, not correct.
This is similar to the longer:
@fail_if(StateSkipped())
but issues a more specific message and exception
Skips this Task if all predicates are True; and optional message can be specified to make the logic clearer.
The optional message is specified using **kwargs: message=’my message’. When given, this message will be logged to the Execution object (and the logging mechanism). This can be used to make complex logic more understandable in the output.
Parameters: |
|
---|
The task directive declares that a function is a Dovetail Task and causes an Instance of LoadedTask (subclass of Task) to be instantiated and registered in the Dovetail model.
Predicates about the processing of the engine.
Bases: object
A predicate that returns True if the current task has at least one dependent Task that executed successfully.
This is useful to detect a Task whose dependents all skipped
Bases: object
A predicate that returns True if a task returns a specific value.
Parameters: | result (any) – The value to test for |
---|---|
Returns: | True if the result equals result |
Return type: | boolean |
Bases: object
A predicate that returns True if a task returns None
Bases: object
A predicate that returns True if a task returns 0 - i.e. what Unix considers a successful execution
Bases: object
A predicate that returns True if the state of the current Task is engine.STATE.FAILED
Bases: object
A predicate that returns True if the state of the current Task is engine.STATE.FAILED
Bases: object
A predicate that returns True if the state of the current Task is engine.STATE.FAILED
Bases: object
A predicate that returns True if the state of the current Task is engine.STATE.SKIPPED
Bases: object
A predicate that returns True if the state of the current Task is engine.STATE.STARTED
Bases: object
A predicate that returns True if the state of the current Task is engine.STATE.SUCCEEDED
Bases: object
A predicate that returns True if a task has produced output to sys.stderr
Environment-based directives.
Bases: object
A predicate that returns True if the environment meets all the specified criteria.
Parameters: |
|
---|---|
Return type: | boolean |
The if_set and if_equals parameters are more fully described at Modification dictionary.
Bases: object
A Predicate for testing environment variables against regular expressions.
Parameters: | tests (string) – List of NAME=RE arguments where NAME is the environment variable to be checked, and RE is the regular expression |
---|---|
Return type: | boolean |
The Python re.search() method is used, rather than re.match(), so the whole value of the environment variable is tested, rather than just the anchoring the search to the beginning of the value.
Examples:
>>> predicate = EnvRE(DOVETAIL_NODE="mercury")
A predicate that matches hostnames of mercury.example.com, mercuryrising.example.org, www.mercury.ru, etc
>>> predicate = EnvRE(DOVETAIL_NODE="mercury.*")
Same as previous
>>> predicate = EnvRE(DOVETAIL_NODE="^mercury\.")
A predicate that matches the host “mercury” in any domain.
>>> predicate = EnvRE(DOVETAIL_NODE="\.example\.com$")
A predicate than matches hosts in any subdomain of example.com
>>> predicate = EnvRE(A="^start", B="end$")
A predicate that returns True if the value of A starts with ‘start’ and the value of B ends with ‘end’
Temporarily sets environment variables for the duration of the decorated function.
Parameters: |
|
---|
The parameters are more fully described in Environment definition arguments.
For example:
@adjust_env("REMOVE", NEW='value')
will unset $REMOVE and set $NEW to ‘value’.
All environment variables are reset to their original values at the end of the task even if the calling function modified their value. Other modifications to the environment (for example new variables) are preserved.
Internal method that applies a dictionary of values to the os.environ while recording the original values.
Parameters: | environment (dict) – A Modification dictionary that will be applied to os.environ. |
---|---|
Returns: | an object which, when passed to revert_environment(), will reset the environment variables modified by this function |
Pretty-print a Modification dictionary.
Parameters: | environment (dict) – The modification dictionary to print |
---|
Resets the environment variables modified by apply_environment().
Parameters: | memento – The return value from apply_environment() |
---|
Warning
Other environment variables modified after the call to apply_environment() and revert_environment() will be left unchanged.
Warning
If several ‘nested’ snapshots are takes, they must be reverted in the inverse order, otherwise the environment may not be restored as expected
Unset an environment variable using del environ[key], ignoring any KeyError
Predicates and directives for the file system.
Bases: object
A predicate version of os.access().
Parameters: |
|
---|---|
Returns: | Same as: os.access(path, mode) |
Return type: | boolean |
os provides some useful constants for mode:
- os.R_OK
- os.W_OK
- os.X_OK
Pretty-prints the instance
Bases: object
A Container Object wrapper for glob.iglob.
Parameters: | glob (string) – A fnmatch glob, eg “*.py” |
---|---|
Returns: | An iterator over the files discovered by the glob |
Return type: | iterator |
Usage:
>>> glob = Glob("*.txt")
>>> for file in glob:
... print file
>>> for file in glob:
... print file
Whenever the __iter__() method is called, the object returns a new iterator created by a call to glob.iglob(). This means that, unlike glob.iglob(), Glob can be iterated over any number of times.
Bases: object
A predicate version of os.path.isdir().
Parameters: | path (string) – The path to test |
---|---|
Returns: | True if the path is a directory |
Return type: | boolean |
Bases: object
A predicate version of os.path.isfile().
Parameters: | path (string) – The path to test |
---|---|
Returns: | True if the path is a file |
Return type: | boolean |
Bases: object
A predicate version of os.path.islink().
Parameters: | path (string) – The path to test |
---|---|
Returns: | True if the path is a link |
Return type: | boolean |
Bases: object
A predicate version of os.path.ismount().
Parameters: | path (string) – The path to test |
---|---|
Returns: | True if the path is a mount point |
Return type: | boolean |
Bases: object
A predicate that returns True if the file is larger than the specified size.
Parameters: |
|
---|---|
Returns: | True if the file is larger than the specified size in bytes |
Return type: | boolean |
Bases: object
A Predicate for testing if some files are newer than another.
Example: This can be used to compile some sources only if they have changed since the last compile:
@do_if(Newer("src/*.py", "build/*.*"))
Parameters: |
|
---|---|
Returns: | True if the newest file in source_iterator is younger than the oldest file in target_iterator |
Return type: | boolean |
Newer computes the result using this pseudo-logic:
- Find the newest file in source_iter
- Call file SOURCE
- Find the oldest file in target_iter
- Call file TARGET
- Test age of SOURCE against age of TARGET
- If SOURCE is newer (younger) than TARGET, return True
- Else return False
Note
If the newest source_iter has exactly the same timestamp as the oldest file in the target_iter, Newer will return False.
Note
The iterator is likely to be called several times in a single build which means an iterator is insufficient - it must be a Container Object - an object with the __iter__() method: http://docs.python.org/library/stdtypes.html#container.__iter__:
- The arguments should evaluate only when called, not created, otherwise Newer will test only those files present when the build file was loaded (and files created or modified in the build may be missed)
- If the argument is an iterator/generator instance, then the second call to Newer will return the iterator after it has performed an iteration, so the iterator will yield no more files.
Dovetail provides a simple wrapper of glob.iglob() in the form of Glob:
source = Glob(os.path.join("src", "mypackage", "*.py"))
target = Glob(os.path.join("build", "doc", "mypackage*.rst"))
@task
@Newer(source, target)
def generate_api_doc():
return call("sphinx-apidoc -f -o src/mypackage build/doc".split(' '))
In this example, Newer runs if any top-level .py file in src/mypackage is newer than the corresponding mypackage*.rst file in build/doc.
Newer works well with Formic (http://www.aviser.asia/formic), which allows an easy construction of rich (Apache Ant) globs. In the example below:
- Source file are all Python files in all subdirectories of src/mypackage, excluding tests,
- Target files are the corresponding mypackage*.rst files in build/doc
import formic
source = formic.FileSet(directory="src/mypackage",
include="*.py",
exclude=["**/*test*/**", "test_*"]
)
target = formic.FileSet(directory="build/doc",
include="mypackage*.rst")
@task
@Newer(source, target)
def generate_api_doc():
return call("sphinx-apidoc -f -o src/mypackage build/doc".split(' '))
Returns the timestamp of the newest (or oldest) file in the file iterator argument.
Parameters: |
|
---|---|
Returns: | The timestamp of the newest (or oldest) file in file_iterator |
Return type: | the os.path.getmtime() value of the file |
Bases: object
A predicate for testing whether an executable is present or on the path.
Parameters: | executable_name (string) – The file (base) name to look for |
---|---|
Return type: | boolean |
Uses a Python implementation of the Unix which command: dovetail.util.which.which()
The wrapped Task will be run in the specified working directory, and the current working directory restored on completion.
Parameters: | directory (string) – The directory to run the Task in; if None, Dovetail will not set _any_ directory for this Task, but leave the Task in the most recently set directory. |
---|
Implementation note: cwd() uses the TaskWorkingDirectories.set() method to store the preferred directory.
Ensures that the give path(s) are present before the decorated function is called, creating if necessary.
Parameters: | required_dirs (string) – A number of paths. For each path that does not exist, os.makedirs() is called to create it |
---|
Directives for package/egg requirements and resolution
Bases: object
A predicate that returns True if all requirements are met in the Python environment.
Parameters: |
|
---|---|
Returns: | True if all specified requirements are satisfied |
Return type: | boolean |
Uses setuptools.commands.easy_install to install a series of package requirements and adjust the system path so they are immediately available.
Parameters: | requirements (Either a string or a list of string) – Requirement specifications as per easy_install, eg: “pylint” and “coverage>3” |
---|---|
Raises : | dovetail.util.MissingRequirement if easy_install cannot locate or install the requirement |
Note
If you need to set specific easy_install behaviour, such as loading from a local host, then modify the easy_install configuration as described in:
Checks to see which requirements are currently satisfied, returning a list of those requirements not satisfied.
Parameters: |
|
---|---|
Returns: | A list of unsatisfied requirements - may be empty |
Return type: | list of string |
Raises : | pkg_resources.VersionConflict if a requirement is in conflict with the environment |
Pretty-print a list of requirements.
Parameters: | requirements (list of string) – Requirements in the form “pylint” or “coverage>3” |
---|---|
Return type: | string |
Ensures all package requirements are installed before executing Task.
Parameters: | requirements (string) – One or more requirement specifications as per easy_install, eg: “pylint” and “coverage>3” |
---|---|
Raises : | dovetail.util.exception.MissingRequirement if easy_install cannot locate or install the requirement |
Raises : | pkg_resources.VersionConflict if a requirement is in conflict with the current environment.= |
Note
If you need to set specific easy_install behaviour, such as loading from a local host, then modify the easy_install configuration as described in:
py.test test script for environment.py
Bases: object
Filters a dictionary to include only those keys we are testing for
py.test test script for predicates in core.py, environment.py and files.py.
NOTE: Windows file access permissions is, ahem, somewhat different from Posix. As a result, we test ONLY THE READ BIT.
Bases: object
py.test test script for packages.py
Bases: object
py.test test script for predicates in core.py, environment.py and files.py
Bases: object
Bases: object
Bases: object
Make sure primitives work