Web development projects often rely on third-party libraries that don’t really belong in version control alongside the application source code. For example, a PHP web application might depend on Zend Framework, Doctrine 2, and jQuery. To support a fully automated build process, I want to be able to retrieve and install the right versions of these dependencies into my application code base automatically.
There are several package management tools out there that support this kind of thing, but most seem overly complex or overly restrictive. Enter Apache Ivy, a simple and flexible dependency manager that integrates with the Apache Ant build tool. Unlike the other dependency managers I found, Ivy makes it easy to run your own repository and choose your own directory layouts and file formats.
The one drawback is that Ivy is designed to work with individual Java JAR files: archives that are typically extracted in memory by Java at runtime. I want to work with dependencies that are directories of library code, already extracted from their archives. This article shows how to set up Ivy and Ant to work together to manage such directory dependencies.
Managing directory dependencies
To manage dependency directories, in addition to the original archive files they were extracted from, Ivy needs some help from the Apache Ant build tool. Dependency management is done by invoking Ant build targets, which run Ivy tasks as needed.
Ivy takes care of resolving dependencies, figuring out what versions of what modules are needed, and downloading dependency archive files from the repository. But Ivy doesn’t know anything about the directories into which the archive files are extracted.
Ant is used to extract the archive files after Ivy retrieves them, and to manage the extracted directories.
Setting up the Ivy repository
Ivy allows downloading dependencies from a variety of different artifact repositories. To keep things as simple as possible, my Ivy repository is just a plain-old web server; an Apache virtual host which serves up a directory in which the dependency artifacts are stored.
My dependency “artifacts” are simply tar/gzip archives stored in subdirectories with a specific naming convention (in this case, [organization]/[module]/[module]-[version].tgz), along with corresponding metadata stored in Ivy XML files.
While the repository format is very simple, you should still use Ivy to publish new artifacts to the repository instead of manually copying them in yourself. Ivy automatically generates the ivy.xml file descriptor, along with MD5 and SHA1 signatures which are used to verify the downloads.
Creating an Apache vhost for the Ivy repository is pretty straightforward.
Add a DNS record
In this example, I use ivy-repo.example.com.
Create the repository directory
This is the directory in which all dependencies will be stored, and which will be the document root for the ivy-repo web server. I use SSH to publish new artifacts to the repository, so I make sure that developer accounts have write access to this directory and everything created within it.
sudo mkdir /srv/ivyrepo sudo chgrp developers /srv/ivyrepo sudo chmod g+ws /srv/ivyrepo
Set up the Apache virtual host
There are any number of ways to do this. As one example:
Add an Apache virtual host config in /etc/apache2/sites-available/ivy-repo:
<VirtualHost *:80> ServerName ivy-repo.example.com DocumentRoot /srv/ivyrepo </VirtualHost> <Directory /srv/ivyrepo> Options +Indexes AllowOverride All Order allow,deny Allow from all </Directory>
Enable the ivy-repo virtual host:
sudo a2ensite ivy-repo
It’s also possible to secure the repo with SSL and HTTP basic authentication, if you choose. See this Ivy mailing list post by Mike Shea for details.
Installing Ivy in the development environment
The development environment needs Java, Ivy, Ant, Ant-Contrib, and a Java SSH library called Jsch.
Packages are available for most Linux distributions. You can also go to the Ivy download page.
On Debian/Ubuntu, do this:
sudo apt-get install ant ant-contrib ivy libjsch-java
I also needed to symlink the jsch library into Ant’s classpath.
sudo ln -s /usr/share/java/jsch.jar /usr/share/ant/lib/jsch.jar
Configuring Ant and Ivy to work together
Add Ant targets in build.xml
Add the following to your Ant build file. This adds support for dependency management targets like get-deps, clean-deps, and publish-deps.
Add Ant properties in build.properties
The build file above relies on the following properties. Define them in your properties file (typically build.properties) and edit as appropriate.
build.dir=${basedir}/build
build.scripts.dir=${basedir}/scripts/build
build.config.dir=${build.scripts.dir}/config
antcontrib.dir=/usr/share/java
# Ivy dependency management settings
ivy.lib.dir=/usr/share/java
deparchive.dir=${build.dir}/deps
ivy.dep.file=${build.config.dir}/ivy.xml
ivy.settings.file=${build.config.dir}/ivysettings.xml
repo.dir=/srv/ivyrepo
repo.server=ivy-repo.example.com
repo.url=http://${repo.server}
repo.ssh.port=22
Add Ivy repository settings in ivysettings.xml
Create an ivysettings.xml file which defines how to interact with the repository. Store it in the directory defined in the properties file.
// ]]>
This tells Ivy to access the repository via the web when downloading dependencies. If the web URL is not accessible, it will try to access via SSH instead.
The trigger at the end tells Ivy to call the “extract-dep” Ant target after it retrieves a dependency from the repository. The extract-dep target extracts the archive file to the appropriate directory, as specified in the build properties (described below).
Publishing dependencies to the repository
When publishing a dependency to the repository, Ant packages the directory into a tar/gzip archive, and then Ivy copies it into the repository over SSH.
To add a new dependency, start by manually adding it to your development environment. For example, to add Zend Framework, download it from the Zend site and extract it into your library directory. (Make sure to tell your version control system to ignore the dependency directory, ex. via .gitignore or “svn propset svn:ignore”.) Then publish it like this:
- Add an ivy.xml file to the directory that tells Ivy its name, version, and organization.
- Add an Ant build property to build.properties that tells Ant about the dependency directory.
- Run ant publish-deps to tell Ant to iterate over each dependency directory, package it up if necessary, and tell Ivy to publish it to the repository if it has changed.
The following sections explain how to do this in more detail.
Describing the module in the dependency’s ivy.xml file
Each dependency’s directory needs to contain an ivy.xml file, describing the “module” that Ivy will create. In most cases this file can be very simple, specifying only the name, version, and organization. More complex specifications are also possible–see the Ivy file documentation for details.
Here’s an example of an ivy.xml file for a Zend Framework module that I created.When publishing the dependency to the Ivy repository, both Ant and Ivy look in this file to determine its name and version.
Defining the directory dependency in build.properties
To let Ant know about the directory dependency, add a property to the build.properties file. The property should have the format dep.NAME.dir=/path/to/dependency/dir, where NAME is the “module” value you defined in its ivy.xml file above. For example:
# Define the directories into which each dependency archive should be extracted.
# Note that these dependencies must be defined in ivy.xml in order to be retrieved at all.
#
# Format: dep.[name].dir
# (where [name] is the dependency's "module" or "name" attribute used in ivy.xml)
#
# WARNING: Make sure to specify directories that will contain nothing but the dependencies themselves.
# *** THESE DIRECTORIES WILL BE DELETED *** when running the clean-deps target!
#
dep.jQuery.dir=${basedir}/public/js/jquery
dep.ZendFramework.dir=${basedir}/library/Zend
dep.ZendX.dir=${basedir}/library/ZendX
dep.Doctrine.dir=${basedir}/library/Doctrine
Running the publish-deps target
Use the following Ant target to package up dependency directories into archive files and publish them as modules in the Ivy repository.
ant publish-deps
This command tells Ant to iterate over each dependency directory configured in build.properties, and do the following for each:
- Determine the current revision defined in the dependency’s ivy.xml.
- Check if there is already an archive file in ${deparchive.dir}/{module}-{version}.tgz.
- If not, create the archive by tarring up the dependency directory.
- Check if a module with the same version already exists in the repository.
- If not, publish the dependency archive to the Ivy repository over SSH. It will prompt for a username and password to log in the repo server.
Getting dependencies into your project
Specifying dependencies in a project ivy.xml file
The overall web application project has an ivy.xml file too. This is the same type of Ivy file used to describe each dependency module. But in this case, we are less interested in publishing the web application as a module, and more interested in defining the application’s dependencies. Store this in the location specified in the ${ivy.dep.file} property in build.properties, above.
Each dependency “name” listed here should have a matching ${dep.NAME.dir} property in build.properties, if it is to be extracted to a directory.
The ivy.xml file in each dependency’s directory could also contain a list of dependencies like this, if necessary. In that case, Ivy would resolve those dependencies and retrieve them as well.
Similarly, the web application itself could be packaged up and published as a module to the Ivy repository, just like the dependencies are.
Running the get-deps target
Use Ant to download all dependencies from the Ivy repository and extract them to the appropriate directories.
ant get-deps
This causes Ivy to do the following:
- Look up the list of application dependencies in ${build.scripts.dir}/ivy.xml
- Download the archive files for those dependencies from the repository to ${deparchive.dir}, if necessary.
- Tell Ant to extract the dependency archives to the appropriate directories, as defined in build.properties.
Remember that Ivy only knows about the archive files, not the directories they are extracted into. This has the following consequences when running ant get-deps:
- If you delete the archive file from ${deparchive.dir} but don’t delete the extracted directory, Ivy will download the archive file again and Ant will extract it again.
- If you delete the extracted directory but don’t delete the archive file, Ivy will think the dependency is up to date, and Ant won’t know that it needs to be extracted again. You’ll need to manually delete the archive file (or run ant clean-deps) in order for it to be retrieved and extracted again.
Running the clean-deps target
The following Ant command will delete all dependency directories and all downloaded dependency archives.
ant clean-deps
This command does the following:
- Delete the ${deparchive.dir} directory, which stores the dependency archive files.
- Delete each dependency directory defined in build.properties.
Why this is awesome
Now, to get a complete working copy of all source code and dependencies, a developer or continuous integration system just has to check out the source code from version control and run ant get-deps. Throw in some Ant targets to automatically update the database (ex. with DbMaintain or Liquibase) and run automated tests (ex. with PHPUnit, etc.), and you’ve got a pretty awesome automated build system.