Building Your Own Unit Testing Framework in C# - Part 1

2/5/2011

About two weeks ago I became bored and when I am bored, I write code. In this case I was curious how difficult it would be to write my own unit testing framework. I mean I use xUnit.net for most of my day to day testing, but I never gave much thought into what goes into creating the framework itself. It turns out that it's fairly simple. Truth be told there are only a couple of parts that the framework needs:

A test finder/loader/runner
A set of assertions
A set of exceptions
Something to hold the results of a test

For something basic, that's it. And to be honest, none of those take much time to write. The approach that I'm taking is very similar to xUnit.net, so if you're familiar with it, this will look very similar. Anyway, let's start with the bit of code that finds our tests and loads them:

   1: /*

   2: Copyright (c) 2011 <a href="http://www.gutgames.com">James Craig</a>

3:

   4: Permission is hereby granted, free of charge, to any person obtaining a copy

   5: of this software and associated documentation files (the "Software"), to deal

   6: in the Software without restriction, including without limitation the rights

   7: to use, copy, modify, merge, publish, distribute, sublicense, and/or sell

   8: copies of the Software, and to permit persons to whom the Software is

   9: furnished to do so, subject to the following conditions:

10:

  11: The above copyright notice and this permission notice shall be included in

  12: all copies or substantial portions of the Software.

13:

  14: THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR

  15: IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,

  16: FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE

  17: AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER

  18: LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,

  19: OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN

  20: THE SOFTWARE.*/

21:

  22: #region Usings

  23: using System;

  24: using System.Collections.Generic;

  25: using System.Linq;

  26: using System.Text;

  27: using Utilities.DataTypes.Patterns.BaseClasses;

  28: using System.Reflection;

  29: using MoonUnit.Attributes;

  30: using MoonUnit.Exceptions;

  31: using MoonUnit.InternalClasses;

  32: using Utilities.Profiler;

  33: #endregion

34:

  35: namespace MoonUnit

  36: {

  37:     /// <summary>

  38:     /// Manager class for unit testing framework

  39:     /// </summary>

  40:     public class Manager : Singleton<Manager>

  41:     {

  42:         #region Constructor

43:

  44:         /// <summary>

  45:         /// Constructor

  46:         /// </summary>

  47:         protected Manager()

  48:             : base()

  49:         {

  50:         }

51:

  52:         #endregion

53:

  54:         #region Functions

55:

  56:         /// <summary>

  57:         /// Tests an assembly

  58:         /// </summary>

  59:         /// <param name="AssemblyToTest">Assembly to test</param>

  60:         /// <returns>The XML string of the results</returns>

  61:         public string Test(Assembly AssemblyToTest)

  62:         {

  63:             Type[] Types = AssemblyToTest.GetTypes();

  64:             return Test(Types);

  65:         }

66:

  67:         /// <summary>

  68:         /// Test a list of types

  69:         /// </summary>

  70:         /// <param name="Types">Types to test</param>

  71:         /// <returns>The XML string of the results</returns>

  72:         public string Test(Type[] Types)

  73:         {

  74:             Output TempOutput = new Output();

  75:             foreach (Type Type in Types)

  76:                 TestClass(Type, TempOutput);

  77:             return TempOutput.ToString();

  78:         }

79:

  80:         /// <summary>

  81:         /// Tests a type

  82:         /// </summary>

  83:         /// <param name="Type">Type to test</param>

  84:         /// <returns>The XML string of the results</returns>

  85:         public string Test(Type Type)

  86:         {

  87:             Output TempOutput = new Output();

  88:             TestClass(Type, TempOutput);

  89:             return TempOutput.ToString();

  90:         }

91:

  92:         private static void TestClass(Type Type, Output TempOutput)

  93:         {

  94:             MethodInfo[] Methods = Type.GetMethods();

  95:             foreach (MethodInfo Method in Methods)

  96:             {

  97:                 object[] Attributes = Method.GetCustomAttributes(false);

  98:                 foreach (Attribute TempAttribute in Attributes)

  99:                 {

 100:                     if (TempAttribute is TestAttribute)

 101:                     {

 102:                         TestAttribute Attribute = (TestAttribute)TempAttribute;

 103:                         if (!Attribute.Skip)

 104:                         {

 105:                             StopWatch Watch = new StopWatch();

 106:                             object TestClass = Type.Assembly.CreateInstance(Type.FullName);

 107:                             try

 108:                             {

 109:                                 Watch.Start();

 110:                                 Method.Invoke(TestClass, Type.EmptyTypes);

 111:                                 Watch.Stop();

 112:                                 if (Attribute.TimeOut > 0 && Watch.ElapsedTime > Attribute.TimeOut)

 113:                                     throw new TimeOut("Method took longer than expected");

 114:                                 TempOutput.MethodCalled(Method);

 115:                             }

 116:                             catch (Exception e)

 117:                             {

 118:                                 if(e.InnerException!=null)

 119:                                     TempOutput.MethodCalled(Method, e.InnerException);

 120:                                 else

 121:                                     TempOutput.MethodCalled(Method, e);

 122:                             }

 123:                         }

 124:                         else

 125:                         {

 126:                             TempOutput.MethodCalled(Method, new Skipped(Attribute.ReasonForSkipping));

 127:                         }

 128:                     }

 129:                 }

 130:             }

 131:         }

 132:

 133:         #endregion

 134:     }

 135: }

I'm using code from my utility library and will skip over those parts but you should be able to figure out what they're doing. Anyway this class only has a couple of functions of note. The first couple are simply Test. These functions are called to actually run tests (either sending in a Type, array of Types, or an assembly). These then call TestClass which goes through and sees if any methods on the class are marked as a test. We do this by using the following attribute:

   1: #region Usings

   2: using System;

   3: using System.Collections.Generic;

   4: using System.Linq;

   5: using System.Text;

   6: #endregion

7:

   8: namespace MoonUnit.Attributes

   9: {

  10:     /// <summary>

  11:     /// Attribute used to denote a test function

  12:     /// </summary>

  13:     [AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]

  14:     public class TestAttribute : Attribute

  15:     {

  16:         #region Constructor

17:

  18:         /// <summary>

  19:         /// Constructor

  20:         /// </summary>

  21:         public TestAttribute(long TimeOut=0)

  22:             : base()

  23:         {

  24:             Skip = false;

  25:             this.TimeOut = TimeOut;

  26:         }

27:

  28:         /// <summary>

  29:         /// Constructor

  30:         /// </summary>

  31:         /// <param name="ReasonForSkipping">Reason for skipping this test</param>

  32:         public TestAttribute(string ReasonForSkipping,long TimeOut=0)

  33:             : base()

  34:         {

  35:             this.Skip = true;

  36:             this.ReasonForSkipping = ReasonForSkipping;

  37:             this.TimeOut = TimeOut;

  38:         }

39:

  40:         #endregion

41:

  42:         #region Properties

43:

  44:         /// <summary>

  45:         /// Determines if the test should be skipped

  46:         /// </summary>

  47:         public bool Skip { get; private set; }

48:

  49:         /// <summary>

  50:         /// The reason for skipping this test

  51:         /// </summary>

  52:         public string ReasonForSkipping { get; private set; }

53:

  54:         /// <summary>

  55:         /// If it takes longer than this, consider it a timeout/failed test

  56:         /// </summary>

  57:         public long TimeOut { get; private set; }

58:

  59:         #endregion

  60:     }

  61: }

With this attribute, we can mark a method to be run by the system. For instance:

   1: class Example

   2: {

   3:     public void DoesNotRun(){}

   4:     [Test]

   5:     public void Runs(){}

   6: }

In this case Runs would be called by the system and DoesNotRun would not be. You can also tell the system to skip a test by giving an explanation. You can also give it an amount of time for it to wait. If it takes longer than a specified time, the test is considered a failure. The last bit of code that is of interest is when an error is detected (and thus the individual test is considered a failure), we need some way to record this. We do this with the aid of Output:

   1: #region Usings

   2: using System;

   3: using System.Collections.Generic;

   4: using System.Linq;

   5: using System.Text;

   6: using MoonUnit.Exceptions;

   7: using System.Reflection;

   8: #endregion

9:

  10: namespace MoonUnit.InternalClasses

  11: {

  12:     /// <summary>

  13:     /// Output class

  14:     /// </summary>

  15:     internal class Output

  16:     {

  17:         #region Constructor

18:

  19:         public Output()

  20:         {

  21:             MethodsCalled = new List<TestMethod>();

  22:         }

23:

  24:         #endregion

25:

  26:         #region Functions

27:

  28:         public void MethodCalled(MethodInfo MethodInformation, Exception Exception=null)

  29:         {

  30:             MethodsCalled.Add(new TestMethod(MethodInformation, Exception));

  31:         }

32:

  33:         public override string ToString()

  34:         {

  35:             string AssemblyLocation = Assembly.GetAssembly(this.GetType()).Location;

  36:             AssemblyName Name = AssemblyName.GetAssemblyName(AssemblyLocation);

  37:             StringBuilder Builder = new StringBuilder();

  38:             Builder.Append("<?xml version=\"1.0\" encoding=\"UTF-8\" ?>");

  39:             Builder.Append(string.Format("<MoonUnit><Header><FileLocation>{0}</FileLocation><Version>{1}</Version></Header><Tests>", AssemblyLocation, Name.Version));

  40:             foreach (TestMethod Method in MethodsCalled)

  41:             {

  42:                 Builder.Append(Method.ToString());

  43:             }

  44:             Builder.Append("</Tests><Footer></Footer></MoonUnit>");

  45:             return Builder.ToString();

  46:         }

47:

  48:         #endregion

49:

  50:         #region Properties

51:

  52:         public virtual List<TestMethod> MethodsCalled { get; private set; }

53:

  54:         #endregion

  55:     }

  56: }

This class simply gets called when a test method is called and the result, it then stores the results in a list and can later export the results into an XML string. The individual test records are stored in the following class:

   1: #region Usings

   2: using System;

   3: using System.Collections.Generic;

   4: using System.Linq;

   5: using System.Text;

   6: using MoonUnit.Exceptions;

   7: using System.Reflection;

   8: #endregion

9:

  10: namespace MoonUnit.InternalClasses

  11: {

  12:     /// <summary>

  13:     /// Holds information about test methods

  14:     /// </summary>

  15:     internal class TestMethod

  16:     {

  17:         #region Constructor

18:

19:

  20:         public TestMethod(MethodInfo MethodInformation,Exception Exception)

  21:         {

  22:             this.MethodInformation = MethodInformation;

  23:             this.Exception = Exception;

  24:         }

25:

  26:         #endregion

27:

  28:         #region Functions

29:

  30:         public override string ToString()

  31:         {

  32:             StringBuilder Builder = new StringBuilder();

  33:             Builder.Append(string.Format("<Test><Class name=\"{0}\" /><Method name=\"{1}\" />",

  34:                                             MethodInformation.DeclaringType.Name,

  35:                                             MethodInformation.Name));

  36:             if (Exception == null)

  37:                 Builder.Append("<Passed />");

  38:             else

  39:                 Builder.Append(string.Format("<Failed>{0}</Failed>", Exception.ToString()));

  40:             Builder.Append("</Test>");

  41:             return Builder.ToString();

  42:         }

43:

  44:         #endregion

45:

  46:         #region Properties

47:

  48:         public virtual MethodInfo MethodInformation { get; private set; }

49:

  50:         public virtual Exception Exception { get; set; }

51:

  52:         #endregion

  53:     }

  54: }

This class simply stores the method information and the result for an individual test and can export that data it to a string later. That's it. With these four classes, we have our set of classes for finding tests to run, running them, and storing the results. That just leaves us with assertions and exceptions. Assertions are simply predicates that should be true at a specific position in our code. Which really just means they're a set of functions that determine if something is truly in the state that we're expecting. In some systems like nUnit, there's like 60 of these functions (most are overloaded versions of themselves). In xUnit, there's much fewer (mostly because they use generics, etc. to achieve the same things). In our case we're going to use the xUnit style (in fact many of them are the same name and look about the same, because... Well... There's not much to them really). Anyway, let's take a look at a couple:

   1: #region Usings

   2: using System;

   3: using System.Collections.Generic;

   4: using System.Linq;

   5: using System.Text;

   6: using MoonUnit.Exceptions;

   7: using System.Collections;

   8: using System.Text.RegularExpressions;

   9: #endregion

10:

  11: namespace MoonUnit

  12: {

  13:     /// <summary>

  14:     /// Used to make assertions

  15:     /// </summary>

  16:     public static class Assert

  17:     {

  18:         #region Functions

19:

  20:         #region DoesNotThrow

21:

  22:         /// <summary>

  23:         /// Used when a specific type of exception is not expected

  24:         /// </summary>

  25:         /// <typeparam name="T">Type of the exception</typeparam>

  26:         /// <param name="Delegate">Delegate called to test</param>

  27:         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>

  28:         public static void DoesNotThrow<T>(VoidDelegate Delegate, string UserFailedMessage = "Assert.DoesNotThrow<T>() Failed")

  29:         {

  30:             try

  31:             {

  32:                 Delegate();

  33:             }

  34:             catch (Exception e)

  35:             {

  36:                 if (e is T)

  37:                     throw new DoesNotThrow(typeof(T), e, UserFailedMessage);

  38:             }

  39:         }

40:

  41:         /// <summary>

  42:         /// Used when a specific type of exception is not expected and a return value is expected when it does not occur.

  43:         /// </summary>

  44:         /// <typeparam name="T">Exception type</typeparam>

  45:         /// <typeparam name="R">Return type</typeparam>

  46:         /// <param name="Delegate">Delegate that returns a value</param>

  47:         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>

  48:         /// <returns>Returns the value returned by the delegate or thw appropriate exception</returns>

  49:         public static R DoesNotThrow<T, R>(ReturnObjectDelegate<R> Delegate, string UserFailedMessage = "Assert.DoesNotThrow<T,R>() Failed")

  50:         {

  51:             try

  52:             {

  53:                 return Delegate();

  54:             }

  55:             catch (Exception e)

  56:             {

  57:                 if (e is T)

  58:                     throw new DoesNotThrow(typeof(T), e, UserFailedMessage);

  59:             }

  60:             return default(R);

  61:         }

62:

  63:         #endregion

64:

  65:         #region Empty

66:

  67:         /// <summary>

  68:         /// Determines if an IEnumerable is empty or not

  69:         /// </summary>

  70:         /// <param name="Collection">Collection to check</param>

  71:         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>

  72:         public static void Empty(IEnumerable Collection, string UserFailedMessage = "Assert.Empty() Failed")

  73:         {

  74:             if (Collection == null)

  75:                 throw new ArgumentNullException("Collection");

  76:             foreach (object Object in Collection)

  77:                 throw new NotEmpty(Collection,UserFailedMessage);

  78:         }

79:

  80:         #endregion

81:

  82:         #region Equal

83:

  84:         /// <summary>

  85:         /// Determines if two objects are equal

  86:         /// </summary>

  87:         /// <typeparam name="T">Object type</typeparam>

  88:         /// <param name="Expected">Expected result</param>

  89:         /// <param name="Result">Actual result</param>

  90:         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>

  91:         public static void Equal<T>(T Expected, T Result,string UserFailedMessage="Assert.Equal() Failed")

  92:         {

  93:             Equal(Expected, Result, new InternalClasses.EqualityComparer<T>(), UserFailedMessage);

  94:         }

95:

  96:         /// <summary>

  97:         /// Determines if two objects are equal

  98:         /// </summary>

  99:         /// <typeparam name="T">Object type</typeparam>

 100:         /// <param name="Expected">Expected result</param>

 101:         /// <param name="Result">Actual result</param>

 102:         /// <param name="Comparer">Comparer used to compare the objects</param>

 103:         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>

 104:         public static void Equal<T>(T Expected, T Result, IEqualityComparer<T> Comparer, string UserFailedMessage = "Assert.Equal() Failed")

 105:         {

 106:             if (!Comparer.Equals(Expected, Result))

 107:                 throw new NotEqual(Expected, Result, UserFailedMessage);

 108:         }

 109:

 110:         #endregion

 111:

 112:         #region False

 113:

 114:         /// <summary>

 115:         /// Determins if something is false

 116:         /// </summary>

 117:         /// <param name="Value">Value</param>

 118:         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>

 119:         public static void False(bool Value, string UserFailedMessage = "Assert.False() Failed")

 120:         {

 121:             if (Value)

 122:                 throw new NotFalse(UserFailedMessage);

 123:         }

 124:

 125:         #endregion

 126:

 127:         #endregion

 128:     }

 129: }

The code above only contains four of the assertions that I've added to the system. Anyway, the basic Assert class is static with a number of static functions. In this case we're looking at DoesNotThrow, Empty, Equal, and False. DoesNotThrow takes in a delegate, which it calls, and sees if it throws a specific error, the Empty function just checks if an IEnumerable is empty (not null, but empty), the equal function checks if two items are equal using an IEqualityComparer, and False just checks if a boolean value is false. Every single assertion is like this, very simple, maybe 5 lines of code, and that's it. The Equals function (and others like it) are a bit more complex as you need to write an IEqualityComparer but I've written one that the system will use by default:

   1: #region Usings

   2: using System;

   3: using System.Collections.Generic;

   4: using System.Linq;

   5: using System.Text;

   6: using MoonUnit.Exceptions;

   7: using System.Reflection;

   8: using System.Collections;

   9: #endregion

10:

  11: namespace MoonUnit.InternalClasses

  12: {

  13:     /// <summary>

  14:     /// Internal equality comparer

  15:     /// </summary>

  16:     /// <typeparam name="T">Data type</typeparam>

  17:     internal class EqualityComparer<T>:IEqualityComparer<T>

  18:     {

  19:         #region Functions

20:

  21:         public bool Equals(T x, T y)

  22:         {

  23:             if (!typeof(T).IsValueType

  24:                 || (typeof(T).IsGenericType

  25:                 && typeof(T).GetGenericTypeDefinition().IsAssignableFrom(typeof(Nullable<>))))

  26:             {

  27:                 if (Object.Equals(x, default(T)))

  28:                     return Object.Equals(y, default(T));

  29:                 if (Object.Equals(y, default(T)))

  30:                     return false;

  31:             }

  32:             if (x.GetType() != y.GetType())

  33:                 return false;

  34:             if (x is IEnumerable && y is IEnumerable)

  35:             {

  36:                 EqualityComparer<object> Comparer = new EqualityComparer<object>();

  37:                 IEnumerator XEnumerator = ((IEnumerable)x).GetEnumerator();

  38:                 IEnumerator YEnumerator = ((IEnumerable)y).GetEnumerator();

  39:                 while (true)

  40:                 {

  41:                     bool XFinished = !XEnumerator.MoveNext();

  42:                     bool YFinished = !YEnumerator.MoveNext();

  43:                     if (XFinished || YFinished)

  44:                         return XFinished & YFinished;

  45:                     if (!Comparer.Equals(XEnumerator.Current, YEnumerator.Current))

  46:                         return false;

  47:                 }

  48:             }

  49:             if (x is IEquatable<T>)

  50:                 return ((IEquatable<T>)x).Equals(y);

  51:             if (x is IComparable<T>)

  52:                 return ((IComparable<T>)x).CompareTo(y) == 0;

  53:             if (x is IComparable)

  54:                 return ((IComparable)x).CompareTo(y) == 0;

  55:             return x.Equals(y);

  56:         }

57:

  58:         public int GetHashCode(T obj)

  59:         {

  60:             throw new NotImplementedException();

  61:         }

62:

  63:         #endregion

  64:     }

  65: }

Oh and if you're at all experienced with xUnit's code base, it's pretty much the same as the one they use. Slightly different but there's only so many ways to write these things. Anyway, if you look at the code, if the assertion fails you will notice that it throws an exception. These exceptions use the following base class:

   1: #region Usings

   2: using System;

   3: using System.Collections.Generic;

   4: using System.Linq;

   5: using System.Text;

   6: #endregion

7:

   8: namespace MoonUnit.BaseClasses

   9: {

  10:     /// <summary>

  11:     /// Base exception class

  12:     /// </summary>

  13:     public class BaseException:Exception

  14:     {

  15:         #region Constructor

16:

  17:         /// <summary>

  18:         /// Constructor

  19:         /// </summary>

  20:         /// <param name="ExceptionText">Exception Text</param>

  21:         /// <param name="Expected">Expected value</param>

  22:         /// <param name="Result">Actual result</param>

  23:         public BaseException(object Expected, object Result,string ExceptionText)

  24:             : base(ExceptionText)

  25:         {

  26:             this.Expected = Expected;

  27:             this.Result = Result;

  28:         }

29:

  30:         #endregion

31:

  32:         #region Properties

33:

  34:         /// <summary>

  35:         /// Expected result

  36:         /// </summary>

  37:         public virtual object Expected { get; private set; }

38:

  39:         /// <summary>

  40:         /// Actual result

  41:         /// </summary>

  42:         public virtual object Result { get; private set; }

43:

  44:         #endregion

45:

  46:         #region Functions

47:

  48:         public override string ToString()

  49:         {

  50:             StringBuilder Builder = new StringBuilder();

  51:             Builder.Append(string.Format("<Expected>{0}</Expected><Result>{1}</Result><ErrorText>{2}</ErrorText><Trace>{3}</Trace><ErrorType>{4}</ErrorType>",

  52:                                         Expected,

  53:                                         Result,

  54:                                         this.Message,

  55:                                         this.StackTrace,

  56:                                         this.GetType().Name));

  57:             return Builder.ToString();

  58:         }

59:

  60:         #endregion

  61:     }

  62: }

The exception base class just takes in the expected value and the result and spits out the result to text when ToString is called. Fairly simple. An an example of an exception class would look like this:

   1: #region Usings

   2: using System;

   3: using System.Collections.Generic;

   4: using System.Linq;

   5: using System.Text;

   6: using MoonUnit.BaseClasses;

   7: #endregion

8:

   9: namespace MoonUnit.Exceptions

  10: {

  11:     /// <summary>

  12:     /// Exception thrown if two items are equal

  13:     /// </summary>

  14:     public class Equal : BaseException

  15:     {

  16:         #region Constructor

17:

  18:         /// <summary>

  19:         /// Constructor

  20:         /// </summary>

  21:         /// <param name="ExceptionText">Exception Text</param>

  22:         /// <param name="Expected">Expected value</param>

  23:         /// <param name="Result">Actual result</param>

  24:         public Equal(object Expected, object Result,string ExceptionText)

  25:             : base(Expected, Result, ExceptionText)

  26:         {

  27:         }

28:

  29:         #endregion

  30:     }

  31: }

The vast majority of the exception classes just look like that (note that the one above is actually thrown from the NotEqual function that I didn't show you and not the Equal function). So create about 20 assertions, have 20 exceptions that match up, and we're done. With this we can create a class library, have the system load it, look through for our tests and give us results in an XML doc. The only thing left is to write a front end of some description, but I'll leave that for part 2. Oh and if you've paid attention to the namespaces, yes, I have indeed named this thing MoonUnit, and yes I will be releasing this on Codeplex. Anyway, take a look, leave feedback, and happy coding.

Comments

Name *

Email (used to grab Gravatar image)

Website

Comment *

Other Posts in Libraries And Releases

Building Your Own Unit Testing Framework in C# - Part 1

Comments

Recent Posts

Recent Comments