Jenkins Choreography

Last modified by Harpreet Singh on 2012/02/11 02:38

As you progressively put more automation on Jenkins/Nectar, it often becomes necessary to choreograph executions of multiple jobs in a complex way.

For example, you might execute a setup job (that deploys a webapp to a QA environment), then run a quick smoke test, then fan out and execute the server test and several load-generating client tests simultaneously. Once all that is over, you might execute the final cleanup job. You further want this whole thing to be packaged as a Jenkins job, to ensure that this sequence executes as an unit.

While there are a number of plugins (such as join and parameterized trriger) that can be used to achieve something similar. In this post, I'll discuss the approach to do this via Jenkins CLI.

Jenkins CLI

The key building block of this technique is the "build" command in the Jenkins CLI. 

ScreenShot2012-02-10at6.31.16PM.png

As you can see, this command can invoke arbitrary jobs, with arbitrary parameters (if your build is parameterized), and also wait for the completion.

With this, the sample workflow discussed above can be achieved by a single shell script build step that looks like the following (assuming that the name of the jobs are "setup", "smoke-test", etc:

ScreenShot2012-02-10at6.31.30PM.png

More Sophisticated Workflow

In the above example, I've used the background process execution mechanism in Unix shells to achieve parallelism. While that is enough for simple workflow, more complex workflow would benefit from Unix tools like xargs and GNU parallel.

It is also possible (and in fact desirable) to use your favorite scripting language, such as Perl, Ruby, or Groovy, to drive Jenkins CLI. For example, the following Groovy code uses GPars to executes a large number of child projects concurrently, but only up to 4 at a time to be considerate for other users on the system:

ScreenShot2012-02-10at6.31.39PM.png

You can very easily extend this to read the list of projects from another file, do a try/finally block to ensure that the tear down job runs all the time, and so on.

Tags:
Created by Harpreet Singh on 2012/02/11 02:36

Resources