Concurrent Unit Testing with xUnit – The answers come in dreams
Well, not precisely in dreams, but in blog posts. No sooner had I written (http://www.damonpayne.com/2008/04/17/ConcurrentUnitTesting.aspx) about how the over-design of NUnit was going to make it hard for me to implement concurrent unit testing than I see Scott Hanselman feature xUnit on his Daily Source Code (http://www.hanselman.com/blog/TheWeeklySourceCode24ExtensibilityEditionPlugInsProvidersAttributesAddInsAndModulesInNET.aspx ). The words that caught my eye: the source is extremely tidy. Scott probably meant that the organization of the solution was tidy but I grabbed it from CodePlex and started investigating. The design is tidy too, could this be a better platform on which to complete my research?
I am generally liking xUnit.net so far, and I strongly expect I’ll be ditching NUnit in favor of this across the board assuming the integration with TestDriven and NCover work as I’d expect. It’s nice to just say “using xUnit” instead of “using NUnit.Framework”, and I like that I don’t have to place a [TextFixture] attribute on the class. But, these are small concerns saving a few keystrokes. What we’re really concerned about is the original goal I wrote about:
“On a sizable project, with a meaningful suite of unit tests, a developer practicing proper due diligence during the development lifecycle will spend a tremendous amount of time running Unit Tests. This is an unfortunate disincentive for the developer to run said tests.”
In general, the “Command” strucuture of a unit test foreshadows parallel-ability Properly designed unit tests should be easy to run in parallel: a unit test should Stand Alone, meaning each test case does not depend on state set up elsewhere. xUnit does two more things that help us out here. The first is by removing the notions of “TestFixtureSetup/Teardown” they’ve made it much harder to shoot yourself in the foot at the Class level by relying on state, though for my example this is merely food for future thought as we’ll see later. The second is that it would appear they use a Randomizer to make sure the [Fact] methods in a Class do not run in any dependable order.
I set up a suite of 17 unit tests, implemented in 7 classes. The tests do incredibly useful things like divide int.MaxValue by things and SpinWait(). Using xUnit is simple:
using XUnit;
namespace DamonPayne.xUnit.Tests
{
public class FooTester : TestBase
{
[Fact]
public void Fact1()
{
Console.WriteLine("FooTester::Fact1");
System.Threading.Thread.SpinWait(int.MaxValue / 2);
Assert.False(false);
}
The last thing to do before jumping into code is to establish a baseline. My 17 tests take 51.78 seconds to run in the xUnit GUI in all their spin-waiting glory.
Payneallel Revisited
While doing the research for this article, I found a few minor issues with my Payneallel.ForEach code I used with the Tree Concurrency articles (http://www.damonpayne.com/2008/04/03/ManagingConcurrencyWithTrees0.aspx ). The first issue dealt with the code I used to wait for all concurrent iterations to be done before returning to the calling thread. If the number of tasks was less than the number of processors on the machine, the “never touched” worker threads would never Finish(). The second dealt with an interesting thread-timing issue related to when a Worker declared itself “Busy”. Concurrency is fun! At any rate, here is the revised Payneallel code I used within xUnit:
using System;
using System.Collections.Generic;
using System.Threading;
namespace XUnit.Sdk
{
/// <summary>
/// Contains static methods and internal helper classes for executing concurrent operations
/// </summary>
public static class Payneallel
{
/// <summary>
/// Concurrently perform the body action on each item in source
/// </summary>
/// <typeparam name="TSource"></typeparam>
/// <param name="source"></param>
/// <param name="body"></param>
public static void ForEach<TSource>(IEnumerable<TSource> source, Action<TSource> body)
{
ForEach<TSource>(source, body, true);
}
/// <summary>
///
/// </summary>
/// <typeparam name="TSource"></typeparam>
/// <param name="source"></param>
/// <param name="body"></param>
/// <param name="waitAll"></param>
public static void ForEach<TSource>(IEnumerable<TSource> source, Action<TSource> body, bool waitAll)
{
WorkerPool<TSource> pool = new WorkerPool<TSource>();
foreach (TSource src in source)
{
Worker<TSource> worker = pool.GetWorker();
//Console.WriteLine("Using worker " + worker.Name);
worker.Arg = src;
worker.Work = body;
worker.Go();
}
if (waitAll)
{
pool.WaitAll();
}
}
}
/// <summary>
///
/// </summary>
/// <typeparam name="T"></typeparam>
class WorkerPool<T>
{
public WorkerPool()
{
_workers = new List<Worker<T>>(Environment.ProcessorCount);
for (int i = 0; i < Environment.ProcessorCount; ++i)
{
Worker<T> worker = new Worker<T>("Payneallel " + i);
_workers.Add(worker);
worker.Done = new Action<T>(WorkerDone);
worker.Go();
}
_workerDoneEvent = new ManualResetEvent(false);
}
private ManualResetEvent _workerDoneEvent;
private static List<Worker<T>> _workers;
private object _syncRoot = new object();
/// <summary>
///
/// </summary>
public void WorkerDone<T>(T arg)
{
lock (_syncRoot)
{
_workerDoneEvent.Set();
}
}
public Worker<T> GetWorker()
{
Worker<T> worker = GetFreeWorker();
while (null == worker)
{
_workerDoneEvent.WaitOne(5, true);
worker = GetFreeWorker();
}
_workerDoneEvent.Reset();
return worker;
}
private Worker<T> GetFreeWorker()
{
foreach (Worker<T> w in _workers)
{
if (!w.Busy)
{
//Console.WriteLine("returning worker from pool");
return w;
}
}
return null;
}
public void WaitAll()
{
while (true)
{
foreach (Worker<T> w in _workers)
{
string name = w.Name;
if (w.Busy)//Don't block if the thread is not working
{
w.Finish();
}
else
{
w.Active = false;
}
}
return;
}
}
}
/// <summary>
///
/// </summary>
/// <typeparam name="T"></typeparam>
class Worker<T>
{
/// <summary>
/// Set up initial state
/// </summary>
/// <param name="name"></param>
public Worker(string name)
{
Name = name;
Work = null;
_active = true;
Busy = false;
_syncRoot = new object();
_acceptingWorkEvent = new ManualResetEvent(true);
_gotWorkEvent = new ManualResetEvent(false);
ThreadStart ts = new ThreadStart(DoWork);
Thread t = new Thread(ts);
t.Name = Name;
t.Start();
}
/// <summary>
/// set _active to false so the thread will exit
/// </summary>
~Worker()
{
_active = false;
}
private ManualResetEvent _acceptingWorkEvent;
private ManualResetEvent _gotWorkEvent;
/// <summary>
/// The name of the worker, used to Name the thread for easier debugging
/// </summary>
public string Name { get; set; }
/// <summary>
/// The worker is currently doing something
/// </summary>
public bool Busy { get; set; }
private bool _active;
private object _syncRoot;
public bool Active
{
get
{
return _active;
}
set
{
_active = value;
}
}
/// <summary>
///
/// </summary>
public Action<T> Work { get; set; }
/// <summary>
/// The Argument to pass to Work
/// </summary>
public T Arg
{
get
{
return _arg;
}
set
{
Busy = true;
_arg = value;
}
}
private T _arg;
/// <summary>
/// Call back to the Pool to signify we're done
/// </summary>
public Action<T> Done { get; set; }
/// <summary>
/// Block until done
/// </summary>
public void Finish()
{
_acceptingWorkEvent.WaitOne();
_active = false;
}
/// <summary>
/// Reset the wait flag so the thread running DoWork will start up again
/// </summary>
public void Go()
{
_gotWorkEvent.Set();
}
public void DoWork()
{
while (_active)
{
_gotWorkEvent.WaitOne();//Wait for someone to call Go on us
if (null != Work && null != Arg) //make sure they set the callback action and the argument to the action
{
lock (_syncRoot)
{
Busy = true;
_acceptingWorkEvent.Reset();//Not accepting work now..
}
Work(Arg);//Do the action
Work = null;
Arg = default(T);
lock (_syncRoot)
{
Busy = false;
_acceptingWorkEvent.Set();//We are accepting work
_gotWorkEvent.Reset();//Don't have work
Done(Arg);//Call back to the pool in case someone is waiting on a free worker
}
}
}
}
}
}
Now, let’s take a brief look at how xUnit works, starting with the xUnit GUI and working down.
Click on the sequence diagram to see it full size. I have paraphrased a bit, and the authors of xUnit might recognize some methods they don’t recall writing. There are some interesting things here, starting with ExecutorWrapper.CreateObject:
object CreateObject(string typeName, params object[] args)
{
try
{
return appDomain.CreateInstanceAndUnwrap(xUnitAssemblyName.FullName, typeName, false, 0, null, args, null, null, null);
}
catch (TargetInvocationException ex)
{
RethrowWithNoStackTraceLoss(ex.InnerException);
return null;
}
}
So we’re going to be executing the tests inside a new AppDomain. This was confusing at first. The code for RunAssembly.DoSequential should make it fairly easy to understand the rest of the execution pipeline:
protected void DoSequential(Executor executor, object _handler)
{
ICallbackEventHandler handler = _handler as ICallbackEventHandler;
AssemblyResult results = new AssemblyResult(new Uri(executor.assembly.CodeBase).LocalPath);
Type[] exportedTypes = executor.assembly.GetExportedTypes();
int count = exportedTypes.Length;
foreach (Type type in exportedTypes)
{
ITestClassCommand testClassCommand = TestClassCommandFactory.Make(type);
if (testClassCommand != null)
{
ClassResult classResult = TestClassCommandRunner.Execute(testClassCommand,
null,
result => OnTestResult(result, handler));
results.Add(classResult);
}
}
OnTestResult(results, handler);
}
}
For each class with Tests inside the assembly, pass the Type off to TestClassCommandRunner and Execute on down the chain. When the results are available, we call OnTestResult, which is the callback we passed all the way from the XUnit GUI. This allows the UI to update itself as the tests are run.
Now we have an understanding of what xUnit is doing. In the next article, we’ll look at using multiple threads to reduce that test execution time.