2010-06-09

TF53010: Errors in the metadata manager when rebuilding OLAP cube

I was adding new fields to our TFS workitems the other day and wanted them to be part of the olap cube. But when I added them (marked them with the reportable attribute) and processed the cube, i couldn't see them in the cube.
I went reading the Event log on the Data tier of the TFS and found log entries like below


TF53010: The following error has occurred in a Team Foundation component or extension:
Date (UTC): 2010-06-09 09:31:26
Machine: MyTfsServer
Application Domain: /LM/W3SVC/830720315/Root/Warehouse-3-129205184075330719
Assembly: Microsoft.TeamFoundation.Warehouse, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727
Process Details:
Process Name: w3wp
Process Id: 9316
Thread Id: 8484
Account name: DOMAIN\admin

Detailed Message: The pending configuration changes were not successfully added to
the cube because of the following error: Microsoft.AnalysisServices.OperationException:
XML parsing failed at line 1, column 0: A document must contain exactly one root element.
.
Errors in the metadata manager. An error occurred when instantiating a metadata
object from the file, '\\?\C:\Program Files\Microsoft SQL Server\MSSQL.3\OLAP\Data
\TfsWarehouse.0.db\Team System.63837.cub.xml'.

at Microsoft.AnalysisServices.AnalysisServicesClient.SendExecuteAndReadResponse(ImpactDetailCollection impacts, Boolean expectEmptyResults, Boolean throwIfError)
at Microsoft.AnalysisServices.AnalysisServicesClient.Alter(IMajorObject obj, ObjectExpansion expansion, ImpactDetailCollection impact, Boolean allowCreate)
at Microsoft.AnalysisServices.Server.Update(IMajorObject obj, UpdateOptions options, UpdateMode mode, XmlaWarningCollection warnings, ImpactDetailCollection impactResult)
at Microsoft.AnalysisServices.Server.SendUpdate(IMajorObject obj, UpdateOptions options, UpdateMode mode, XmlaWarningCollection warnings, ImpactDetailCollection impactResult)
at Microsoft.AnalysisServices.MajorObject.Update(UpdateOptions options, UpdateMode mode, XmlaWarningCollection warnings)
at Microsoft.AnalysisServices.MajorObject.Update(UpdateOptions options)
at Microsoft.TeamFoundation.Warehouse.OlapCreator.CreateOlap(WarehouseConfig whConf, String accessUser, String[] dataReaderAccounts, Boolean dropDB, Boolean processCube)
at Microsoft.TeamFoundation.Warehouse.AdapterScheduler.EnsureCubeIsUpToDate()

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


I searched the path in the error message above (on the database server, little bit confusing since the path seems to be a local one, but since the error originates from the database server it makes sense).

I the folder i found several files named Team System.xxxx.cub.xml (where xxxx is digits) and some where 0 bytes long and some contained error xml (seemed to be cut in the middle of writing).
I removed these files and processed the cube again. (I used the Warehousecontroller webservice on the tfs server, https://mytfsserver:8143/Warehouse/v1.0/WarehouseController.asmx?op=Run)

This time the processing went through without any problems and the fields apperared in my cube.

I wonder how these error xml files where created, perhaps if the cube was instructed to reprocess while the iis was restarted or for some other reason.
I'll better check the folder once and a while....

2010-05-21

Writing a custom trigger for CruiseControl.Net

I’ve had some problems regarding the triggering of our nightly builds and has not been able to solve this with the standard triggers that comes with ccnet (using v1.4.4). So finally I took some time (night time, kids are sleeping and there is peace and quiet in the house :) and wrote a trigger that fixed the issue at hand. I will come to the trigger in just a moment but I’ll take some time to explain our ccnet setup.

Interval builds

All our projects are built using CruiseControl.Net and for each project we setup two builds. One interval build that triggers when the source has been changed. This build uses the intervalTrigger that checks the source repository at regular intervals. The interval build compiles the project and performs unit testing and provides feedback to the programmer. This is quite standard setup I guess so there’s no rocket science here…

Nightly builds

The other build is a nightly build that (in addition to the tasks for the interval build) performs analysis, generates documentation and installation packages ready to deploy to a target system. It also creates deliverable folder in a specific ”drop zone” for the project. This folder is named using the label of the nightly build.

Labels

All projects uses the project name and build no as label (like MyProject – 234). We reuse the build number of the interval build for the nightly build so if the last interval build was ”MyProject – 234”, the next time the nightly build is triggered the label for the nightly build is ”MyProject-Nightly – 234”. This creates a side effect that if the nightly build is triggered twice and the interval build hasn’t been triggered, the label is the same. If you remember the drop zone folder it uses the label of the nightly build as a name so if it is triggered twice using the same label, it tries to overwrite the drop zone folder.

This is a big no-no and to solve this I added a nant task that fires in the beginning of the nightly build that fails if the drop zone folder exists.

Triggering the nightly build

So to avoid failing the nightly build, it should only be triggered if an interval build has successfully been completed and the nightly build for that interval build hasn’t been performed.

Normally nightly triggers are defined to run at a specific time every night using a ScheduleTrigger that fires at that time if any changes to the source has been made during the day. In combination with a projectTrigger that only fires if the interval build has been successful we can accomplish the trigger to fire only if the interval is successful.

Problem with projectTrigger

There is a limitation to the projectTrigger that affects our setup severely. It doesn’t remember the project status after the ccnet server has restarted. This causes the nightly projects to be rebuilt after a server restart. Since we have lots of projects on the server, we need to restart it atleast once a week and this causes the nightly builds to fail.

New trigger?

To solve this we need another way to trigger the build. The first solution that comes to mind is if there was a trigger that could check if a specific folder (the drop zone folder in this case) was missing, we could replace the projectTrigger with a ”Missing folder trigger”.

Perhaps could look like this.

<missingFolderTrigger path="Z:\PublishedBuilds\MyProject-Nightly\1.0.0.234"/>
Problem here is that we don’t know the last folder name. The ”1.0.0.234”comes from the file version for our compiled assemblies.
The first part of the version number can be any numbers and the 234 refers to the build number of the project. This changes for each build so we need a way to look this up.

Final design

I speak highly of the KISS principle, but I’m also great at creating generic things and sometimes this ”generic gene” gets some overhand (and sometimes also get way out of hand, but I think I found a good level this time :)
So the final design was a trigger syntax like below:
<fileExistsTrigger
triggerOnMissing="true"
seconds="30"
path="Z:\PublishedBuilds\MyProject-Nightly\"
match="\d+\.\d+\.\d+\.[Regex.Match([Projects(MyProject).LastSuccessfulBuildLabel],\d+$)]"/>
  • This trigger is now a more ”generic” one allowing us to trigger on both files and directories.
  • The ”triggerOnMissing” allows us to trigger on both exists and not exists.
  • It’s based on an interval trigger and the ”seconds” parameter allows us to set on how often a file/directory should be checked.
  • The ”path” allows us to set the path that should be checked. If we only need to check a file/directory that we know the name of, this is enough. End with a \ to check a directory.

The match attribute

The ”match” allows us to perform a regular expression search that searches for a match for a directory/file in the supplied path.
  • The \d+ will match one or more digits
  • The \. will match a dot.
  • Repeating this match three times \d+\.\d+\.\d+\. will match the ”1.0.0.” part of the file (or any other valid version major.minor.build. combo)
Now the fun part begins :). I added some syntax to perform string operations and replacements. Any valid command is surrounded by [] and are replaced before the file/directory regex matching begins.
In this example we have two commands, one property and one method call. Properties are evaluated first so [Projects(MyProject).LastSuccessfulBuildLabel] allows us to instruct the trigger to lookup the project named ”MyProject” and return the LastSuccessfulBuildLabel. In this case this would be ”MyProject – 234”.
The method call would look like this after the property replacement. [Regex.Match(MyProject - 234,\d+$)]. This will call the Regex.Match method and return the matched pattern. In this case \d+$ will match the last digits in the ”MyProject – 234” and would return ”234”.
So the final match attribute will be \d+\.\d+\.\d+\.234 and would in this case find a folder named 1.0.0.234. If the folder is missing, the build will be triggered, but if it is there, no trigger is fired. If the interval build fires and is successful, the LastSuccessfulBuildLabel increases and is 235. When the nightly build is triggered again, the folder is missing and the trigger will fire.

Writing the trigger

The first part to think about when writing extensions/plugins to ccnet is that the assembly must be named ccnet.*.plugins.dll otherwise ccnet wont find it. Secondly, the trigger class needs to be decorated with the ReflectionTypeAttribute to tell the class what the xml element name for the class looks like (like [ReflectorType("fileExistsTrigger")]). Every field/property that should be read into the trigger should also be decorated with the ReflectorPropertyAttribute (like [ReflectorProperty("path", Required = true)]). I mainly looked at the ProjectTrigger that comes with ccnet and added stuff along the way.
Beware that the code don't fix all special cases, like if your project name contains , ) or other characters that messes up the expressions, but this is what I needed (remember TDD).
You can use the code below as it is or modify it to your needs. If you want the whole solution (including unit tests), just drop me a mail.
Good luck :)
using System;
using System.Collections.Generic;
using System.Linq;
using ThoughtWorks.CruiseControl.Remote;
using ThoughtWorks.CruiseControl.Core.Triggers;
using ThoughtWorks.CruiseControl.Core.Util;
using Exortech.NetReflector;
using System.Text.RegularExpressions;
using System.IO;

namespace Meridium.CruiseControl.Net.Triggers {
[ReflectorType("fileExistsTrigger")]
public class FileExistsTrigger : IntervalTrigger {
/// <summary>
/// If the trigger should be active if the file is missing, default is false
/// </summary>
[ReflectorProperty("triggerOnMissing", Required = false)]
public bool TriggerOnMissing;
/// <summary>
/// The url to the ccnet server, defaults to a local ccnet installation tcp://localhost:21234/CruiseManager.rem
/// </summary>
[ReflectorProperty("serverUri", Required = false)]
public string ServerUri;
/// <summary>
/// The file/directory path to see if it exists or not
/// </summary>
[ReflectorProperty("path", Required = true)]
public string Path;
/// <summary>
/// Optional parameter to use if a regular expression should be used to match a file or directory in the path. Default is an empty string.
/// </summary>
[ReflectorProperty("match", Required = false)]
public string Match;

private readonly ICruiseManagerFactory _managerFactory;

public IDictionary<string, ProjectStatus> ProjectStatus {
get {
lock (ProjectStatusLock) {
//if cache has timed out, clear cache.
if (_projectStatus != null && DateTime.Now > _cacheValidUntil) {
_projectStatus = null;
Log.Debug("Cache was deleted, was valid until " + _cacheValidUntil.ToString("yyyy-MM-dd HH:mm:ss,fff"));
}
if (_projectStatus == null) {
Log.Debug("Updating ProjectStatus cache from server: " + ServerUri);
_projectStatus = new Dictionary<string, ProjectStatus>();
foreach (ProjectStatus status in _managerFactory.GetCruiseManager(ServerUri).GetProjectStatus()) {
_projectStatus.Add(status.Name, status);
}
_cacheValidUntil = DateTime.Now + CacheTime;
Log.Debug("Cache valid until " + _cacheValidUntil.ToString("yyyy-MM-dd HH:mm:ss,fff"));
}
return _projectStatus;
}
}
}
private static DateTime _cacheValidUntil = DateTime.MinValue;
private static readonly TimeSpan CacheTime = new TimeSpan(0, 10, 0);
private static Dictionary<string, ProjectStatus> _projectStatus;
private static readonly object ProjectStatusLock = new object();

public FileExistsTrigger()
: this(new DateTimeProvider(), new RemoteCruiseManagerFactory()) {
}

public FileExistsTrigger(DateTimeProvider dtp, ICruiseManagerFactory managerFactory)
: base(dtp) {
ServerUri = "tcp://localhost:21234/CruiseManager.rem";
_managerFactory = managerFactory;
}
public override IntegrationRequest Fire() {
//only check on intervals
if (base.Fire() != null) {
try {
Log.Debug(string.Format("More than {0} seconds since last integration, checking url.", IntervalSeconds));
if (FileExists() != TriggerOnMissing) {
Log.Debug("Trigger matched, fire IntegrationRequest");
return new IntegrationRequest(BuildCondition, Name);
}
} catch (Exception ex) {
Log.Error(ex);
} finally {
IncrementNextBuildTime();
}
}
return null;
}
private string HandleProjectPropertyMatches(Match m) {
string projectName = m.Groups["projectName"].Value;
string property = m.Groups["property"].Value;
var ps = GetCurrentProjectStatus(projectName);
switch (property.ToLower()) {
case "lastsuccessfulbuildlabel":
return ps.LastSuccessfulBuildLabel;
case "name":
return ps.Name;
case "buildstatus":
return ps.BuildStatus.ToString();
default:
throw new NotImplementedException(string.Format("Support for property {0} is not implemented yet!", property));
}
}
private static string HandleRegexMatchMatches(Match m) {
string input = m.Groups["input"].Value;
string pattern = m.Groups["pattern"].Value;
Match match = Regex.Match(input, pattern);
return match.Success ? match.Value : string.Empty;
}
private bool FileExists() {
string fp = TranslateValue(Path);
string match = TranslateValue(Match);
if (!string.IsNullOrEmpty(match)) {
if (!fp.EndsWith(@"\"))//"
fp += @"\";//"
var dir = new DirectoryInfo(fp);
if (!dir.Exists) {
Log.Debug(string.Format("Matching path {0} failed. Directory does not exist.", fp));
return false;
}
foreach (var fs in dir.GetFileSystemInfos().Where(fs => Regex.IsMatch(fs.Name, match))) {
Log.Debug(string.Format("Match successful with fileSystemInfo {0}", fs.FullName));
return true;
}
Log.Debug(string.Format("No match for {0}", match));
return false;

}
bool isDirectory = fp.EndsWith(@"\");//"
bool exists = isDirectory ? Directory.Exists(fp) : File.Exists(fp);
Log.Debug(string.Format("Checking if {0} {1} exists: {2}", (isDirectory ? "directory" : "file"), fp, exists));
return exists;
}
/// <summary>
/// Translates the supplied value and expands all methods and variables
/// </summary>
/// <param name="val">The value to translate</param>
/// <returns>The translated value</returns>
private string TranslateValue(string val) {
//handle replacements...
//like [Projects(MyProject).LastSuccessfulBuildLabel]
val = Regex.Replace(val, @"\[Projects\((?<projectName>[^\)]+)\)\.(?<property>[^\]]+)\]", HandleProjectPropertyMatches, RegexOptions.IgnoreCase);
//handle Regexp
//like [Regex.Match(string,pattern)]
val = Regex.Replace(val, @"\[Regex\.Match\((?<input>[^,]+),(?<pattern>[^\)]+)\)\]", HandleRegexMatchMatches, RegexOptions.IgnoreCase);

return val;
}
private ProjectStatus GetCurrentProjectStatus(string project) {
if (!ProjectStatus.ContainsKey(project)) {
throw new NoSuchProjectException(project);
}
return ProjectStatus[project];
}
}
}

2010-04-13

Adding buttons to the TinyMce editor in EPiServer 6 programmatically

While developing ImageVault we needed to add buttons to insert images in the TinyMce editor that ships with EPiServer CMS 6. The buttons are decorated with a TinyMCEPluginButtonAttribute and they can be added manually by entering admin mode and modifying the settings for the XHTML property (see Oskars post "Add functionality to the rich text editor in EPiServer cms 6").
To do this by code this was rather simple since EPiServer has a great Api (even that it could contain more examples)...

The settings are stored in the EPiServer database and is accessed using the PropertySettingsRepository class.
To get the current settings for the TinyMce editor, just create a repository and use the GetDefault method


PropertySettingsRepository p = new PropertySettingsRepository();
PropertySettingsWrapper defaultSettings = p.GetDefault(typeof(TinyMCESettings));

The PropertySettingsWrapper contains metadata for the settings, like DisplayName, Id etc.
It also contains the specific setting for the TinyMce editor in the property PropertySettings.

To create a new settings for the TinyMce editor with some buttons added we can use the code below.

PropertySettingsRepository p = new PropertySettingsRepository();
PropertySettingsWrapper defaultSettings = p.GetDefault(typeof(TinyMCESettings));

//create a copy of the default settings
PropertySettingsWrapper copy = defaultSettings.Copy();
copy.DisplayName+="(My copy)";
TinyMCESettings settings = copy.PropertySettings as TinyMCESettings;

//create a new toolbar row for my buttons
ToolbarRow row = new ToolbarRow();
settings.Toolbars.Add(row);

//the buttons are added using the name defined in the
//ButtonName property of the TinyMCEPluginButtonAttribute.
row.Buttons.Add("mybutton1");
row.Buttons.Add("separator");
row.Buttons.Add("mybutton2");

//Save the copy as a global setting
p.SaveGlobal(copy);

//Set it as default
p.SetDefault(copy.Id);

Quite easy. The separator button is included in EPiServer 6 the others need to be added using classes decorated with the TinyMCEPluginButtonAttribute.

Note!
To actually use this code in EPiServer you need to insert it somewhere :) The easiest way is to create a InitializationModule class (implement IInitializaionModule and mark it with the InitializationModuleAttribute). Then in the Initialize method, add your code to the InitializationEngine.InitComplete event.

public void Initialize(InitializationEngine context) {
context.InitComplete+=(sender,args)=> MyInstallButtonsMethod();
}

More about EPiServer Initialization on episerver world.