Dovetail script tutorial

Preparation

Before starting this tutorial, please ensure that:

  • You have a working Python 2.7 installation
  • You have installed setuptools
  • You have installed Dovetail

Consider installing virtualenv. It helps you keep your Python system clean when trying out new packages.

For brevity in the examples below there are no import statements. Standard packages, such as shutil or subprocess, are always fully qualified so it is easy for you to interpret the import. All other functions and classes are present in the dovetail package and can be imported:

from dovetail import *

With correct name qualifications, Dovetail will work with any standard import approach.

Declaring tasks

It is traditional to introduce a language or technology with the simplest working program - one that prints “Hello world”. Here it is for Dovetail.

Create a temporary working directory somewhere, say ~/dovetail_tmp. In it, create a file build.py:

from dovetail import task

@task
def hello_world():
    print "hello world!"

This example declares a single task - hello_world.

The task may be executed from the command line:

$ cd dovetail_tmp
$ dovetail hello_world
STARTING Task build:hello_world
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
hello world!
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
SUCCEEDED: Task build:hello_world

===============================================================================
2012-07-20 14:16:19

  Task 'build:hello_world' SUCCESS, returning None

  Environment:
    Hostname:    macbook.local
    Username:    builder
    Python:      /usr/bin/python
    Working dir: /Users/builder/dovetail_tmp
    Build file:  build.py
    Tasks:       1
    Elapsed:     0.000s

===============================================================================

Dovetail produces a detailed log of what it did, but you can see the “hello world!” between the row of “v“‘s and “^“‘s.

A task:

  • Is a Python function with the @task decorator. The function has no arguments,
  • May return any value it wants (which will be captured by Dovetail and printed)
  • May reference and use any libraries you want
  • Can raise an exception to tell Dovetail it has failed

Note

By default, the dovetail command will load the script called build.py. If you want a script with a different name, run it using the -f option:

$ dovetail -f my_build.py hello_world

You can find out more about the dovetail command in the Running Dovetail.

Warning

Dovetail will not load build files from a package directory - a directory with the __init__.py file.

This is because Dovetail does not want to get itself muddled up with the Python packages it is building.

Task dependencies

Tasks may have dependencies declared on other tasks. Dovetail guarantees that a task’s dependents will all have executed before the task’s function is called.:

from dovetail import task, depends

@task
def bar():
    print "bar() runs before foo()"

@task
@depends(bar)
def foo():
    print "foo runs after bar"

Tasks can depend on several other tasks, and dependencies can be ‘nested’:

from dovetail import task, depends

@task
def frist():
    print "frist"

@task
@depends(frist)
def second():
    print "second"

@task
def third():
    print "third"

@task
@depends(second, third)
def foo():
    print "foo"

In this example, the tasks run in this order:

  1. first
  2. second
  3. third
  4. foo

In the examples above, the dependencies were declared by directly referencing the task’s function. You can also reference task’s by name (which can be necessary if you have complex code dependencies):

from dovetail import task, depends

@task
@depends("bar")
def foo():
    print "foo"

@task
def bar():
    print "Bar by name"

Within the same build file, the name of a task is simply the name of the function. Things become a bit more complex if you have tasks in several files. This is discussed in Loading build files.

Running in a virtual environment

Builds often need to install specific and precise versions of packages against which to compile and link. They may also be sensitive to the presence of specific packages. Also, the builder might not have write permissions to install into the system package or even user library. For all these reasons and more, it is good practice to build in a ‘clean room’ using virtualenv.

Dovetail makes it easy to run your build in a virtualenv, even if a the actual environment doesn’t yet exist. From the command line, use the -e option to tell Dovetail to run the build in the specified virtualenv:

$ dovetail -e ~/my_private_env foo

Dovetail test that ~/my_private_env is a valid virtualenv and use it if it is. If there is no virtualenv there, Dovetail will create it. Finally, the entire build will be run in that environment.

Sometimes you absolutely need a clean environment (production builds for example). Use the --clear option in conjunction with the -e to guarantee a clean environment:

$ dovetail -e ~/my_private_env --clear foo

Declarative approach

Introduction

Dovetail provides a library of classes and functions to allow a build script to externalize a number of typical areas of complexity. This makes a typical build script shorter, more understandable and much easier to maintain.

For example, many builds use environment variables to interact with other scripts or build programs. Dovetail makes this easy:

@task
@adjust_env('HIDE', A="10", B="20")
def foo():
    ...

The @adjust_env() decorator does three things:

  • Unsets the environment variable $HIDE
  • Sets $A to 10
  • Sets $B to 20

These changes are present only within the foo() task body. Dovetail:

  • Sets the environment variable before the task starts, then
  • Runs all of the task’s dependencies, then
  • Runs the body of the task, then
  • Resets the environment variables to their original values.

Dovetail also allows the build script to declare when to skip (or execute) a task, for example:

@task
@skip_if(Env('SKIP_OPTIONAL'))
def optional():
    ...

The @skip_if decorator is executed at the point the task is run. If the environment variable $SKIP_OPTIONAL is set (has any value), then optional() and any dependent tasks are not executed and the build continues.

Note

This works because @skip_if accepts a predicate - a function or callable which, when executed, returns a boolean. Dovetail executes the predicate only at the point where it decides whether to run the task (not at the point the predicate is loaded from file)

In the example above, the Env predicate is constructed, returning a Python callable object. When Env.__call__() is called, it tests whether the key os.environ['SKIP_OPTIONAL'] exists.

Declarative execution

Dovetail has two declarative ways of controlling whether a task should execute:

  • @do_if
  • @skip_if

We have seen @skip_if; @do_if is the inverse. A decorated task will be executed only if the predicate returns True.

For example, to execute the task only when an environment variable is set:

@task
@do_if(Env('DO_OPTIONAL'))
def optional():
    ...

Some more predicates

Dovetail provides a number of predicates. Some of the important ones are:

  • IsFile: Tests whether a path exists and is a file
  • IsDir: Tests whether a path exists and is a directory
  • Installed: Tests whether one or more packages are installed.
  • Which: Tests whether an executable is on the system path

In the example below, the task runs and deletes a directory - but only if it is present:

BUILD_DIR = os.path.join("..", "build")

@task
@do_if(IsDir(BUILD_DIR))
def clean():
    shutil.rmtree(BUILD_DIR)

Logic

So far we have seen some simple predicates. Dovetail has predicate logic operators to allow the simple predicates to be combined in complex expressions. For example, use All to test that a number of conditions are all true:

# BUILD_DIR is a global reference to the 'build' directory
BUILD_DIR = os.path.join("..", "build")

@task
@do_if(All(IsDir(BUILD_DIR), Env("DO_CLEAN")))
def clean():
    shutil.rmtree(BUILD_DIR)

Here the clean task runs only if the build directory exists and the $DO_CLEAN is set.

Dovetail supports the following logical operators:

  • Not: Inverts a predicate’s value
  • All: True only if all the predicate arguments are true (semantics are the same as Python’s all() function).
  • Any: True if any of the predicate arguments are true (semantics are the same as Python’s any() function).

A more complex example:

# BUILD_DIR is a global reference to the 'build' directory
BUILD_DIR = os.path.join("..", "build")

@task
@do_if(All(IsDir(BUILD_DIR), Not(Env("SKIP_CLEAN"))))
def clean():
    shutil.rmtree(BUILD_DIR)

Environment variables

Dovetail sets a number of environment variables so that it is easy to tailor your build scripts to different build environments. In conjunction with standard OS environment variables, it is is easy to tailor build scripts to:

  • Different operating systems, which have different file system layouts and different tools and utilities
  • Different machines in a cluster, with different capabilities
  • Different Python versions
  • Different users

Some particularly useful environment variables are:

Environment Variable Name How obtained (eg Python API) Eg Win Eg Mac
About the OS
DOVETAIL_OS_NAME os.name nt posix
DOVETAIL_SYSTEM platform.system() Windows Darwin
DOVETAIL_RELEASE platform.release() XP 11.3.0
Information about Python itself
DOVETAIL_PYTHON_IMPLEMENTATION platform.python_implementation() CPython, IronPython, PyPy, Jython
DOVETAIL_PYTHON_MAJOR_VERSION platform.python_version_tuple() 2 2
DOVETAIL_PYTHON_MINOR_VERSION platform.python_version_tuple() 2.7 2.7
DOVETAIL_PYTHON_VERSION platform.python_version_tuple() 2.7.1 2.7.1
About the environment
DOVETAIL_NODE platform.node() Machine’s hostname
DOVETAIL_USER getpass.getuser() Username of build user
DOVETAIL_VIRTUALENV_SLAVE Set if running in a virtualenv slave Path to virtualenv

(The full list is defined in stamp_environment()).

You can set environment variables in several different ways:

  • For the whole build: Before the script is called, eg export MY_VAR=10 from the Bash prompt
  • For a task and dependencies: Use the @adjust_env declaration
  • Programmatically: Directly modify os.environ

You can use environment variables to control the flow of the build using the Env predicate:

  1. Test if is set or not set, eg Not(Env('FOR'))
  2. Test if equals a specific value, eg Env(BAR="10")
  3. Test if matches an RE, eg EnvRE(FOO='fo+bar')

The EnvRE regular expression matcher is works as shown below:

@task
@do_if(EnvRE(DOVETAIL_NODE="\\.example\\.com$"))
def example-dot-com():
    ...

This task runs only if the network name of the machine ends in .example.com. More details can be found at EnvRE.

Interacting with the file system

By default, all tasks will run in the same directory as the build file in which they are defined - even if they are called by a task which changed the working directory. If this were not so, the sub-processes run by the task may produce output in unpredictable locations.

To run a task in a specific directory, for example in a build or working directory, use the @cwd directive:

@task
@cwd("/tmp")
def in_temporary_dir():
    ...

Just like @adjust_env, the scope of @cwd is the task itself and its dependencies. The original working directory will be restored after the task completes.

Using @cwd is simpler than changing directory in the code, and helps other engineers using the build script see what is going on.

Many build scripts need a set of directories created for both temporary and final artifacts. Dovetail makes this explicit with the @mkdirs directive:

BUILD_ROOT = os.path.join("..", "build")
DOC_ROOT = os.path.join(BUILD_ROOT, "doc")
EGG_DIR = os.path.join(BUILD_ROOT, "egg")

@task
@mkdirs(DOC_ROOT, EGG_DIR)
def prepare():
    ...

@mkdirs ensures that the document root (../build/doc) and the location the Egg will be written to (../build/egg) are created before the prepare task starts. Dovetail ensures all intermediate directories are created.

Dovetail also has a number of predicates for examining the file system. We have met a few like IsFile. A complete list is:

  • IsFile
  • IsDir
  • IsLink
  • IsMount
  • Which: Tests if an executable is on the system path
  • Access: Tests the mode bits of a file
  • LargerThan: Tests whether a file is larger than a specified size
  • Newer: Tests whether the one or more files are newer than another file or files.

Interacting with installed packages

Warning

It is generally considered bad practice to develop directly on the Python installation - virtualenv explains why.

We strongly recommend using virtualenv to create repeatable build using isolated and clean environments. See Running in a virtual environment.

Dovetail uses setuptools to manage package requirements for the build.

A single build script typically contains a collection of quite different tasks - from building a software package, running code analysis and quality tools through to generating documentation. There are typically significant differences in the packages needed for any two tasks. For example:

  • Team members do not need or want a specific packages installed on their machines, for example, developers might not need Sphinx while documenters don’t need Pylint.
  • License restrictions mean that the package cannot be deployed to all machines
  • Building on different architectures or operating systems require packages unique to that architecture or OS, for example py2exe for Windows and py2app for Mac.
  • Different versions of Python need different versions of specific libraries.

Wouldn’t it be nice if the build script loaded and installed packages only as needed? Dovetail supports this with the following features:

  • install() function: Uses setuptools easy_install to install one or more packages and modifies the package path so the package is immediately available. Can be used either as the script loads, or within the body of a task
  • @requires directive: Works as install(), but is called only when a task is invoked. In other words, if a build is run which does not execute the task, no packages are installed
  • Installed predicate: Returns True if the package or packages are all installed in the current Python VM.

For example, a documentation build task might start:

@task
@requires("sphinx")
def doc():
    ...

And the predicate can be used to test if a site package has been installed:

@task
@do_if(Installed("custom_package"))
def custom():
    ...

Note

The order of directives can be important. In Python, the decorators of a function are predictably invoked from outside to inside. This means that a @skip_if or @do_if directive can guard a @requires if it comes before:

@task
@skip_if(Env('SKIP_DOC'))
@requires('sphinx')
def doc():
    ...

If $SKIP_DOC is set, then the @requires will not be executed, and sphinx will not be installed.

Note

If you need to set specific easy_install behaviour, such as loading from a local host, then create the easy_install configuration files.

Deliberately failing the build

So far we have discussed how to set up a build script and execute dependencies, and how to optionally execute or skip tasks. What if the build reaches a state where it should fail?

Especially with a long-running build, it is much better to follow the Rule of Repair and fail fast and fail noisily.

The first approach is for the task to simply raise an exception (or allow an exception to propagate out of the function). Dovetail will catch the exception and fail the build.

Dovetail also supports a more explicit approach using the @fail directive. If the @fail predicate is True after the task completes, then the task will be failed. This is very useful if you need to check if an external program produced output, for example:

@task
@fail_if(Not(IsFile("/path/to/tool/output")))
def run_tool():
    ...

dovetail.directives.engine.StdErr is a predicate that detects output to standard error for that specific task (which is normally, but not always, a good indicator that there was an issue):

@task
@fail_if(StdErr())
def run_tool():
    ...

It is also very common to check the return value of a process. You can do this explicitly with @check_result. This works according to Unix semantics, so if the return value is not 0, the task is failed:

@task
@check_result
def run_tool():
    return subprocess.call(["my_tool"])

Of course, these can be combined:

@task
@fail_if(Not(IsFile("/path/to/tool/output")))
@fail_if(StdErr())
@check_result
def run_tool():
    return subprocess.call(["my_tool"])

Complete list of directives and predicates

For a complete list of predicates and directives, go to the directives Package

Write your own Predicates

If the predicates that come packaged with Dovetail are insufficient, for example maybe you need to call a network service, you can readily extend Dovetail and create your own.

A predicate is a callable - this means a function or an object instance that has a __call__. The callable will be invoked only when Dovetail is ready to run your task. The callable should:

  • Have no arguments
  • Return a boolean
  • Not raise an exception (exceptions from predicates fail the build with a system error)

Which approach should you use? A function or an object with a __call()__?

  • No arguments or configuration: If the test has no arguments or configuration, for example something that queries a hard-coded path or network address, use a function
  • Configurable predicates: If the predicate has arguments, perhaps a path to interrogate, a host name to contact, etc, then use a object.

Here is an example of a simple predicate that tests whether the host is a specific server:

class IsHost(object):
    def __init__(self, hostname):
        self.hostname = hostname

    def __call__(self):
        return os.environ['DOVETAIL_NODE'] == self.hostname

    def __str__(self):
        if self():
            return "DOVETAIL_NODE == '{0}'".format(self.hostname)
        else:
            return "DOVETAIL_NODE == '{0}' (actually is '{1}')".format(self.hostname, os.environ['DOVETAIL_NODE'])


@task
@do_if(IsHost('macbook.example.com'))
def test_is_hostname():
    pass

The salient points are:

  1. The constructor takes the configuration arguments and stores them in the instances members. The configuration arguments are given when the predicate is instantiated in the @do_if, @skip_if or @fail_if directive: IsHost('macbook.example.com')
  2. The __call__() method takes only the self argument. It compares the environment with the values stored in its members
  3. A __str__() method is helpful - Dovetail will use this to log the decisions it made, and a simple explanation will make the flow of the build much more understandable.

Loading build files

In many cases it is best that a single project has several build files, for example when:

  • The project consists of a master-build of several full-fledged modules, and developers may be working on just one module at any one time
  • Complex builds where it is desirable to separate concerns into separate files, for example separating the Python build and packaging from the documentation and web site publishing.

To support this, Dovetail has the following features:

  • load(): Loads another build file on the file system
  • Cross-file Task names
  • Automatic dependencies: When a build file is loaded, tasks with the same name are set as dependants in the loading file.

In the following sections, let us assume a simple project layout. The project consists of two separate packages, module1 and app:

+-buildroot/
  +-build.py        <- Master build script
  +-library/        <- A directory containing a single Python package used as a library
  | +-build.py      <- Library build script
  | +-setup.py      <- Standard distutils setup.py
  | +-src/
  |   +- ...
  +-app/            <- Another directory containing a Python application and documentation source
  | +-build.py      <- Application build script
  | +-setup.py      <- Standard distutils setup.py
  | +-doc/          <- Documentation
  | | +-build.py    <- Documentation build script
  | | +- ...
  | +-src/
  |   +- ...

The load() function

The buildroot/build.py script will call load() for both the library’s and the application’s build scripts:

import dovetail
dovetail.load("library/build.py")
dovetail.load("app/build.py")

And the app/build.py would load the documentation build script (relative to its own directory:

import dovetail
dovetail.load("doc/build.py")

Paths can be given with either a forward or backward slash - Dovetail will convert to the slashes appropriate for the platform.

As discussed earlier, Dovetail will refuse to load a build file from Python package directories (those with an __init__.py). If the build script were in the package directory, you would end up packaging the script in your distributable, and then require the runtime to have a dependency on Dovetail.

Warning

Because build scripts cannot be placed in Python package directories, they cannot be loaded by Pythons import statement. If a script is placed in a package and is loaded with an import, Dovetail will not load its tasks and will not allow its tasks to be referenced.

Task names

A script can reference tasks in other files using the ‘fully-qualified’ approach. Suppose the master build script needed to reference the generate_api task in the app/doc/build.py file - it would do so like:

@task
@depends("app.doc.build:generate_api")
def document():
    ...

While the same reference from the app‘s build file would be:

@task
@depends("doc.build:generate_api")
def document():
    ...

Note how the task name is relative to the current build file. You can reference tasks that are:

  • Self: In the same build file
  • Siblings: In other build files in the same directory (as the current build file)
  • Descendents: In build files in sub directories (of the current build file).

You cannot reference tasks in either parent directories or ‘cousin’ files.

Automatic dependencies

In projects that have multiple build files, most build file will contain the same ‘master’ tasks - a typical set might be:

  • clean: Removes all temporary and working directories and files
  • prepare: Set up necessary working directories. Run code generation utilities (if any).
  • test: Run unit tests, optionally gathering metrics
  • dist: Package the Python project (eg: python setup.py dist)
  • doc: Run the documentation utility to produce a documentation bundle
  • publish: Upload artifacts to a central repository, perhaps even the PyPI/Cheeseshop.

(There are many other possibilities, including tasks to deploy to a test server, perform integration tests, tag and release and so on).

Dovetail automatically ‘links’ tasks of the same name in descendant build files in the order the build files were loaded. For example:

Build files:                    +------------+
                                | ./library/ |
                                |   build.py |
                              2 +------------+
    +------------+          ,~~>| clean      |
    | ./build.py |         /    | prepare    |
  1 +------------+        /     | test       |
~~~>| clean      |~~~~~~~'      | dist       |
    | prepare    |   \          |            |
    | test       |    \         +------------+
    | dist       |     \
    | doc        |      \       +------------+              +------------+
    +------------+       \      | ./app/     |              | ./app/doc/ |
                          \     |   build.py |              |   build.py |
                           \  3 +------------+            4 +------------+
                            `~~>| clean      |~~~~~~~~~~~~~>| clean      |
                                | prepare    |              | prepare    |
                                | test       |              |            |
                                | dist       |              |            |
                                | doc        |              | doc        |
                                +------------+              +------------+

This this example, running dovetail clean at the top level automatically invokes:

  1. build:clean
  2. library.build:clean
  3. app.build:clean
  4. app.doc.build:clean

In a similar fashion, running doc in either ./build.py or ./app/build.py ultimately causes the doc in ./app/doc/build.py to run.

Automatic dependencies work even if a tasks are ‘missing’. In this example, doc might be declared only in two files:

Build files:                    +------------+
                                | ./library/ |
                                |   build.py |
                                +------------+
    +------------+              | ...        |
    | ./build.py |              +------------+
    +------------+
    | ...        |~~~.          +------------+              +------------+
    | doc        |    \         | ./app/     |              | ./app/doc/
    +------------+     \        |   build.py |              |   build.py |
                        \       +------------+              +------------+
                         \      | ...        |              | ...        |
                          `~~~~>|            |~~~~~~~~~~~~~>| doc        |
                                +------------+              +------------+

Assuming that build files are largely similar, automatic dependencies makes it much easier to manage a large project distributed over many build scripts.

Note

A master build script often contains tasks without a body to act as placeholders for automatic dependencies. For example:

@task
def doc():
    pass

Reports

If the build does not go as planned, you might need to start debugging it. Dovetail has a number of reports that are useful in this instance:

dovetail -r slow <task>

After a run, show a report showing the slowest tasks. Relatively fast tasks are not listed.

dovetail -r tasks <task>

Runs the build and produces a hierarchical graph showing the tasks that ran and the reasons why tasks either were skipped or failed.

dovetail -r modules <task>

Runs the build, and then generates a report showing the package structure and modules available as loaded by the build.

dovetail -r depend <task>

Runs the build and then generates a report showing the dependency graph (more specifically, shows the hierarchical graph of calls to load()).

Configuration files

Sometimes you may find your command line becoming too long to be easily manageable. Instead of retyping the command line options every time, you can enter these values in a configuration file.

Dovetail example

Create a file names .dovetail.ini in the same directory as your build file:

[Dovetail]
task: clean dist
virtualenv: /path/to/virtual/environment
loglevel: ERROR
reports: slow

In this example, if you enter run dovetail with no arguments, the configuration file will make the effective command:

$ dovetail -e /path/to/virtual/environment -q -report slow clean dist

If you specify values on the command line, they will always override the corresponding values in the configuration file. For example, consider the command dovetail doc. The command-line specified the tasks to build, so the configuration file’s clean dist will be overridden, and the effective command is:

$ dovetail -e /path/to/virtual/environment -q -report slow doc

Note

If your configuration file sets log output nesting on, then you can override the option using the -nn option:

$ dovetail -nn ...

-nn is short for No Nesting.

Environment variable example

You can also use the configuration file to set environment variables for the duration of the build - these values will be visible to all tasks. To achieve this, use the Environment section:

[Environment]
MYVAR: VALUE
BUILD_FLAG: special-options

This file will set two environment variables:

$MYVAR=VALUE
$BUILD_FLAG=special-options

Note

This is especially useful for automated builds - use this to keep your precise environment and options under configuration management in your VCS.

Configuration file format

The configuration file should conform to ConfigParser norms, eg the file is sections with names in square brackets. Items are expressed as “name: value”.

The formal definition of the configuration file is:

  • Section Dovetail:
    • build_file: The name of the build file to use (rather than the default build.py)
    • task: A space-separated list of Task names
    • virtualenv: Path to the virtual environment
    • clear: Boolean flag; if True any virtual environment will be cleared before the build
    • nested: Boolean flag: Produce nested log files?
    • loglevel: The level to log at (see Logger and dovetail.util.logging.LEVEL)
    • reports: A list of reports to run (see Running Dovetail)
  • Section Environment:
    • For each item in this section, an environment variable will be set in os.environ with the item’s value

For a boolean flag, you can use the following values (case is ignored):

True False
true false
t f
on off
yes no
y n
1 0
ok  

Configuration file location

The ideal situation for an automated build is to be able to run the build script with as few arguments as possible. This makes it easier to maintain, and makes it trivial for everyone else to generate a complete build.

The best way of achieving this is to configure the arguments in the configuration file as described above - with the bonus that now all arguments and environment variables are under configuration management and version control. However:

  • What if you need to configure different environment variables on different build machines? (for example to point to a different location of an important resource)
  • How can developers and testers run partial builds quickly and conveniently?

Dovetail provides a solution to these issues by parameterization of the configuration file name - the parameters are the user id of the process and the network name of the computer.

Dovetail searches the following locations, taking the first configuration file that matches:

  1. In the current working directory
    1. .dovetail.<user id>.ini (<user id> is substituted by getpass.getuser())
    2. .dovetail.<hostname>.ini (<hostname> is substituted by platform.node())
    3. .dovetail.ini
  2. In the user’s home directory (from os.path.expanduser("~"))
    1. .dovetail.ini

For example, if you want your own personal configuration file, and your user id is ‘builder’, create a file .dovetail.builder.ini. Its values will override all the values in .dovetail.ini, even if it’s completely empty.

Note

Remember that entries on the command line always override that entry in the configuration file.

How to automate your build

Use a VCS...

... and preferably a DVCS like Git or Mercurial.

All your project’s files should be checked into it, including your build scripts and configuration files. This should go without saying...

Use virtualenv

If you are, then excellent. If you aren’t, please do, you will thank yourself later.

Set up the build script

Create a Dovetail build script in the root of your code. Create tasks for all your major build goals. Some might be:

  • clean: Remove all non-vcs files from the build
  • dist, sdist, dist_egg: Tasks to build your Python project’s distributables.
  • test: Run your unit, system and integration tests, preferably with code Coverage
  • metrics: Run Pylint and other code quality tools
  • doc: Run documentation tasks

Check this script into your VCS.

Set up configuration files

You may want to create a configuration file as well. This configuration file might be host specific or build-user specific so it doesn’t clash with the developer environments. This can be used to:

  • Specify a virtual environment to run the whole build in (this means that Dovetail won’t try to install packages into the system library)
  • Run specific reports after the build (like slow)
  • Set environment variables which can influence your build.
  • Specify the tasks to run, eg clean, dist_egg, metrics and doc

With this configuration file done, a default build is as simple as:

$ cd /path/to/build/root
$ dovetail

You have a decision: Do you want developers to check in their own configuration files? One the plus side, it is easier for them to manage their environment or move between machines and versions. On the negative side, you may have a large number of files checked in which are not specifically product related. Changes to these files may also trigger meaningless builds.

If you created a configuration file, check it into your VCS.

Set up the automated build machine and slaves

The installation of the build automation server and its slaves are outside the scope of this tutorial - follow the official instructions for your tool.

After the build server (and slaves) are running, you need to ensure that each server has both virtualenv and Dovetail installed. You probably want them installed in Python’s system library:

$ sudo easy_install virtualenv dovetail

This is the only requirement Dovetail puts on the build infrastructure. Dovetail does all the rest in the virtual environment.

Warning

If you call Dovetail with arguments that need virtualenv, Dovetail will attempt to download and install virtualenv using EasyInstall. This will work in a number of environments (Windows, or with Python installed by the build user), but may not be what you want.

Note

Dovetail works equally well from a virtual environment, but you will need to activate this in your build plan (see below) making this slightly more complicated. Dovetail running in a virtual environment will happily run its build in another virtual environment.

Set up the build

Each build automation server is different, but share common steps. To set up a build (often called a build plan) you need to configure:

  • How the tool checks the project out of the VCS
  • How the build is triggered (eg manual, periodic, polling the VCS)
  • The build script: dovetail (note - you can give arguments)
  • Post-build actions (eg triggering other builds, uploading the artifacts, who to email on success/failure, etc)

You may want to create a separate build plan for metrics or documentation, since these do not strictly need to be complete for the binary to be useful to developers. This build plan might run only nightly where the main build runs after each checkin.

You want to keep the build plan configuration to be as simple as possible because this aspect is likely not checked into your VCS. In the event of catastrophic damage to the build server (well, it is running builds...) the simpler the build configuration the easier it is to restore.

Next steps

Where do you go from here? Here are a few suggestions:

  1. Give Dovetail a try and use it to script your builds. It should fit on top of your existing tools (if you have any) and co-ordinate them
  2. Look at the complete library of predicates and directives: directives Package
  3. Look at the examples in the source distribution (under ‘test’). The examples cover most aspects of build script use (and abuse)
  4. If you need more detail on the specific algorithms and implementation, take a look at the API and model documentation: dovetail Package
  5. If you need help or assistance, either email the maintainer or raise an issue on BitBucket
  6. Join the Dovetail developer community - we’d be delighted to have you! Email the maintainer or visit on Facebook