I’ve had some problems regarding the triggering of our nightly builds and has not been able to solve this with the standard triggers that comes with ccnet (using v1.4.4). So finally I took some time (night time, kids are sleeping and there is peace and quiet in the house :) and wrote a trigger that fixed the issue at hand. I will come to the trigger in just a moment but I’ll take some time to explain our ccnet setup.
Interval builds
All our projects are built using CruiseControl.Net and for each project we setup two builds. One interval build that triggers when the source has been changed. This build uses the intervalTrigger that checks the source repository at regular intervals. The interval build compiles the project and performs unit testing and provides feedback to the programmer. This is quite standard setup I guess so there’s no rocket science here…
Nightly builds
The other build is a nightly build that (in addition to the tasks for the interval build) performs analysis, generates documentation and installation packages ready to deploy to a target system. It also creates deliverable folder in a specific ”drop zone” for the project. This folder is named using the label of the nightly build.
Labels
All projects uses the project name and build no as label (like MyProject – 234). We reuse the build number of the interval build for the nightly build so if the last interval build was ”MyProject – 234”, the next time the nightly build is triggered the label for the nightly build is ”MyProject-Nightly – 234”. This creates a side effect that if the nightly build is triggered twice and the interval build hasn’t been triggered, the label is the same. If you remember the drop zone folder it uses the label of the nightly build as a name so if it is triggered twice using the same label, it tries to overwrite the drop zone folder.
This is a big no-no and to solve this I added a nant task that fires in the beginning of the nightly build that fails if the drop zone folder exists.
Triggering the nightly build
So to avoid failing the nightly build, it should only be triggered if an interval build has successfully been completed and the nightly build for that interval build hasn’t been performed.
Normally nightly triggers are defined to run at a specific time every night using a ScheduleTrigger that fires at that time if any changes to the source has been made during the day. In combination with a projectTrigger that only fires if the interval build has been successful we can accomplish the trigger to fire only if the interval is successful.
Problem with projectTrigger
There is a limitation to the projectTrigger that affects our setup severely. It doesn’t remember the project status after the ccnet server has restarted. This causes the nightly projects to be rebuilt after a server restart. Since we have lots of projects on the server, we need to restart it atleast once a week and this causes the nightly builds to fail.
New trigger?
To solve this we need another way to trigger the build. The first solution that comes to mind is if there was a trigger that could check if a specific folder (the drop zone folder in this case) was missing, we could replace the projectTrigger with a ”Missing folder trigger”.
Perhaps could look like this.
<missingFolderTrigger path="Z:\PublishedBuilds\MyProject-Nightly\1.0.0.234"/>
Problem here is that we don’t know the last folder name. The ”1.0.0.234”comes from the file version for our compiled assemblies.
The first part of the version number can be any numbers and the 234 refers to the build number of the project. This changes for each build so we need a way to look this up.
Final design
I speak highly of the KISS principle, but I’m also great at creating generic things and sometimes this ”generic gene” gets some overhand (and sometimes also get way out of hand, but I think I found a good level this time :)
So the final design was a trigger syntax like below:
<fileExistsTrigger
triggerOnMissing="true"
seconds="30"
path="Z:\PublishedBuilds\MyProject-Nightly\"
match="\d+\.\d+\.\d+\.[Regex.Match([Projects(MyProject).LastSuccessfulBuildLabel],\d+$)]"/>
- This trigger is now a more ”generic” one allowing us to trigger on both files and directories.
- The ”triggerOnMissing” allows us to trigger on both exists and not exists.
- It’s based on an interval trigger and the ”seconds” parameter allows us to set on how often a file/directory should be checked.
- The ”path” allows us to set the path that should be checked. If we only need to check a file/directory that we know the name of, this is enough. End with a \ to check a directory.
The match attribute
The ”match” allows us to perform a regular expression search that searches for a match for a directory/file in the supplied path.
- The \d+ will match one or more digits
- The \. will match a dot.
- Repeating this match three times \d+\.\d+\.\d+\. will match the ”1.0.0.” part of the file (or any other valid version major.minor.build. combo)
Now the fun part begins :). I added some syntax to perform string operations and replacements. Any valid command is surrounded by [] and are replaced before the file/directory regex matching begins.
In this example we have two commands, one property and one method call. Properties are evaluated first so
[Projects(MyProject).LastSuccessfulBuildLabel] allows us to instruct the trigger to lookup the project named ”MyProject” and return the LastSuccessfulBuildLabel. In this case this would be ”MyProject – 234”.
The method call would look like this after the property replacement.
[Regex.Match(MyProject - 234,\d+$)]. This will call the Regex.Match method and return the matched pattern. In this case \d+$ will match the last digits in the ”MyProject – 234” and would return ”234”.
So the final match attribute will be \d+\.\d+\.\d+\.234 and would in this case find a folder named 1.0.0.234. If the folder is missing, the build will be triggered, but if it is there, no trigger is fired. If the interval build fires and is successful, the LastSuccessfulBuildLabel increases and is 235. When the nightly build is triggered again, the folder is missing and the trigger will fire.
Writing the trigger
The first part to think about when writing extensions/plugins to ccnet is that the assembly must be named ccnet.*.plugins.dll otherwise ccnet wont find it. Secondly, the trigger class needs to be decorated with the ReflectionTypeAttribute to tell the class what the xml element name for the class looks like (like [ReflectorType("fileExistsTrigger")]). Every field/property that should be read into the trigger should also be decorated with the ReflectorPropertyAttribute (like [ReflectorProperty("path", Required = true)]). I mainly looked at the ProjectTrigger that comes with ccnet and added stuff along the way.
Beware that the code don't fix all special cases, like if your project name contains , ) or other characters that messes up the expressions, but this is what I needed (remember TDD).
You can use the code below as it is or modify it to your needs. If you want the whole solution (including unit tests), just drop me a mail.
Good luck :)
using System;
using System.Collections.Generic;
using System.Linq;
using ThoughtWorks.CruiseControl.Remote;
using ThoughtWorks.CruiseControl.Core.Triggers;
using ThoughtWorks.CruiseControl.Core.Util;
using Exortech.NetReflector;
using System.Text.RegularExpressions;
using System.IO;
namespace Meridium.CruiseControl.Net.Triggers {
[ReflectorType("fileExistsTrigger")]
public class FileExistsTrigger : IntervalTrigger {
/// <summary>
/// If the trigger should be active if the file is missing, default is false
/// </summary>
[ReflectorProperty("triggerOnMissing", Required = false)]
public bool TriggerOnMissing;
/// <summary>
/// The url to the ccnet server, defaults to a local ccnet installation tcp://localhost:21234/CruiseManager.rem
/// </summary>
[ReflectorProperty("serverUri", Required = false)]
public string ServerUri;
/// <summary>
/// The file/directory path to see if it exists or not
/// </summary>
[ReflectorProperty("path", Required = true)]
public string Path;
/// <summary>
/// Optional parameter to use if a regular expression should be used to match a file or directory in the path. Default is an empty string.
/// </summary>
[ReflectorProperty("match", Required = false)]
public string Match;
private readonly ICruiseManagerFactory _managerFactory;
public IDictionary<string, ProjectStatus> ProjectStatus {
get {
lock (ProjectStatusLock) {
//if cache has timed out, clear cache.
if (_projectStatus != null && DateTime.Now > _cacheValidUntil) {
_projectStatus = null;
Log.Debug("Cache was deleted, was valid until " + _cacheValidUntil.ToString("yyyy-MM-dd HH:mm:ss,fff"));
}
if (_projectStatus == null) {
Log.Debug("Updating ProjectStatus cache from server: " + ServerUri);
_projectStatus = new Dictionary<string, ProjectStatus>();
foreach (ProjectStatus status in _managerFactory.GetCruiseManager(ServerUri).GetProjectStatus()) {
_projectStatus.Add(status.Name, status);
}
_cacheValidUntil = DateTime.Now + CacheTime;
Log.Debug("Cache valid until " + _cacheValidUntil.ToString("yyyy-MM-dd HH:mm:ss,fff"));
}
return _projectStatus;
}
}
}
private static DateTime _cacheValidUntil = DateTime.MinValue;
private static readonly TimeSpan CacheTime = new TimeSpan(0, 10, 0);
private static Dictionary<string, ProjectStatus> _projectStatus;
private static readonly object ProjectStatusLock = new object();
public FileExistsTrigger()
: this(new DateTimeProvider(), new RemoteCruiseManagerFactory()) {
}
public FileExistsTrigger(DateTimeProvider dtp, ICruiseManagerFactory managerFactory)
: base(dtp) {
ServerUri = "tcp://localhost:21234/CruiseManager.rem";
_managerFactory = managerFactory;
}
public override IntegrationRequest Fire() {
//only check on intervals
if (base.Fire() != null) {
try {
Log.Debug(string.Format("More than {0} seconds since last integration, checking url.", IntervalSeconds));
if (FileExists() != TriggerOnMissing) {
Log.Debug("Trigger matched, fire IntegrationRequest");
return new IntegrationRequest(BuildCondition, Name);
}
} catch (Exception ex) {
Log.Error(ex);
} finally {
IncrementNextBuildTime();
}
}
return null;
}
private string HandleProjectPropertyMatches(Match m) {
string projectName = m.Groups["projectName"].Value;
string property = m.Groups["property"].Value;
var ps = GetCurrentProjectStatus(projectName);
switch (property.ToLower()) {
case "lastsuccessfulbuildlabel":
return ps.LastSuccessfulBuildLabel;
case "name":
return ps.Name;
case "buildstatus":
return ps.BuildStatus.ToString();
default:
throw new NotImplementedException(string.Format("Support for property {0} is not implemented yet!", property));
}
}
private static string HandleRegexMatchMatches(Match m) {
string input = m.Groups["input"].Value;
string pattern = m.Groups["pattern"].Value;
Match match = Regex.Match(input, pattern);
return match.Success ? match.Value : string.Empty;
}
private bool FileExists() {
string fp = TranslateValue(Path);
string match = TranslateValue(Match);
if (!string.IsNullOrEmpty(match)) {
if (!fp.EndsWith(@"\"))//"
fp += @"\";//"
var dir = new DirectoryInfo(fp);
if (!dir.Exists) {
Log.Debug(string.Format("Matching path {0} failed. Directory does not exist.", fp));
return false;
}
foreach (var fs in dir.GetFileSystemInfos().Where(fs => Regex.IsMatch(fs.Name, match))) {
Log.Debug(string.Format("Match successful with fileSystemInfo {0}", fs.FullName));
return true;
}
Log.Debug(string.Format("No match for {0}", match));
return false;
}
bool isDirectory = fp.EndsWith(@"\");//"
bool exists = isDirectory ? Directory.Exists(fp) : File.Exists(fp);
Log.Debug(string.Format("Checking if {0} {1} exists: {2}", (isDirectory ? "directory" : "file"), fp, exists));
return exists;
}
/// <summary>
/// Translates the supplied value and expands all methods and variables
/// </summary>
/// <param name="val">The value to translate</param>
/// <returns>The translated value</returns>
private string TranslateValue(string val) {
//handle replacements...
//like [Projects(MyProject).LastSuccessfulBuildLabel]
val = Regex.Replace(val, @"\[Projects\((?<projectName>[^\)]+)\)\.(?<property>[^\]]+)\]", HandleProjectPropertyMatches, RegexOptions.IgnoreCase);
//handle Regexp
//like [Regex.Match(string,pattern)]
val = Regex.Replace(val, @"\[Regex\.Match\((?<input>[^,]+),(?<pattern>[^\)]+)\)\]", HandleRegexMatchMatches, RegexOptions.IgnoreCase);
return val;
}
private ProjectStatus GetCurrentProjectStatus(string project) {
if (!ProjectStatus.ContainsKey(project)) {
throw new NoSuchProjectException(project);
}
return ProjectStatus[project];
}
}
}
