Building Your Own Unit Testing Framework in C# - Part 1

2/5/2011

About two weeks ago I became bored and when I am bored, I write code. In this case I was curious how difficult it would be to write my own unit testing framework. I mean I use xUnit.net for most of my day to day testing, but I never gave much thought into what goes into creating the framework itself. It turns out that it's fairly simple. Truth be told there are only a couple of parts that the framework needs:

  • A test finder/loader/runner
  • A set of assertions
  • A set of exceptions
  • Something to hold the results of a test

For something basic, that's it. And to be honest, none of those take much time to write. The approach that I'm taking is very similar to xUnit.net, so if you're familiar with it, this will look very similar. Anyway, let's start with the bit of code that finds our tests and loads them:

 

   1: /*
   2: Copyright (c) 2011 <a href="http://www.gutgames.com">James Craig</a>
   3: 
   4: Permission is hereby granted, free of charge, to any person obtaining a copy
   5: of this software and associated documentation files (the "Software"), to deal
   6: in the Software without restriction, including without limitation the rights
   7: to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
   8: copies of the Software, and to permit persons to whom the Software is
   9: furnished to do so, subject to the following conditions:
  10: 
  11: The above copyright notice and this permission notice shall be included in
  12: all copies or substantial portions of the Software.
  13: 
  14: THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
  15: IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
  16: FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
  17: AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
  18: LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
  19: OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
  20: THE SOFTWARE.*/
  21:  
  22: #region Usings
  23: using System;
  24: using System.Collections.Generic;
  25: using System.Linq;
  26: using System.Text;
  27: using Utilities.DataTypes.Patterns.BaseClasses;
  28: using System.Reflection;
  29: using MoonUnit.Attributes;
  30: using MoonUnit.Exceptions;
  31: using MoonUnit.InternalClasses;
  32: using Utilities.Profiler;
  33: #endregion
  34:  
  35: namespace MoonUnit
  36: {
  37:     /// <summary>
  38:     /// Manager class for unit testing framework
  39:     /// </summary>
  40:     public class Manager : Singleton<Manager>
  41:     {
  42:         #region Constructor
  43:  
  44:         /// <summary>
  45:         /// Constructor
  46:         /// </summary>
  47:         protected Manager()
  48:             : base()
  49:         {
  50:         }
  51:  
  52:         #endregion
  53:  
  54:         #region Functions
  55:  
  56:         /// <summary>
  57:         /// Tests an assembly
  58:         /// </summary>
  59:         /// <param name="AssemblyToTest">Assembly to test</param>
  60:         /// <returns>The XML string of the results</returns>
  61:         public string Test(Assembly AssemblyToTest)
  62:         {
  63:             Type[] Types = AssemblyToTest.GetTypes();
  64:             return Test(Types);
  65:         }
  66:  
  67:         /// <summary>
  68:         /// Test a list of types
  69:         /// </summary>
  70:         /// <param name="Types">Types to test</param>
  71:         /// <returns>The XML string of the results</returns>
  72:         public string Test(Type[] Types)
  73:         {
  74:             Output TempOutput = new Output();
  75:             foreach (Type Type in Types)
  76:                 TestClass(Type, TempOutput);
  77:             return TempOutput.ToString();
  78:         }
  79:  
  80:         /// <summary>
  81:         /// Tests a type
  82:         /// </summary>
  83:         /// <param name="Type">Type to test</param>
  84:         /// <returns>The XML string of the results</returns>
  85:         public string Test(Type Type)
  86:         {
  87:             Output TempOutput = new Output();
  88:             TestClass(Type, TempOutput);
  89:             return TempOutput.ToString();
  90:         }
  91:  
  92:         private static void TestClass(Type Type, Output TempOutput)
  93:         {
  94:             MethodInfo[] Methods = Type.GetMethods();
  95:             foreach (MethodInfo Method in Methods)
  96:             {
  97:                 object[] Attributes = Method.GetCustomAttributes(false);
  98:                 foreach (Attribute TempAttribute in Attributes)
  99:                 {
 100:                     if (TempAttribute is TestAttribute)
 101:                     {
 102:                         TestAttribute Attribute = (TestAttribute)TempAttribute;
 103:                         if (!Attribute.Skip)
 104:                         {
 105:                             StopWatch Watch = new StopWatch();
 106:                             object TestClass = Type.Assembly.CreateInstance(Type.FullName);
 107:                             try
 108:                             {
 109:                                 Watch.Start();
 110:                                 Method.Invoke(TestClass, Type.EmptyTypes);
 111:                                 Watch.Stop();
 112:                                 if (Attribute.TimeOut > 0 && Watch.ElapsedTime > Attribute.TimeOut)
 113:                                     throw new TimeOut("Method took longer than expected");
 114:                                 TempOutput.MethodCalled(Method);
 115:                             }
 116:                             catch (Exception e)
 117:                             {
 118:                                 if(e.InnerException!=null)
 119:                                     TempOutput.MethodCalled(Method, e.InnerException);
 120:                                 else
 121:                                     TempOutput.MethodCalled(Method, e);
 122:                             }
 123:                         }
 124:                         else
 125:                         {
 126:                             TempOutput.MethodCalled(Method, new Skipped(Attribute.ReasonForSkipping));
 127:                         }
 128:                     }
 129:                 }
 130:             }
 131:         }
 132:  
 133:         #endregion
 134:     }
 135: }

I'm using code from my utility library and will skip over those parts but you should be able to figure out what they're doing. Anyway this class only has a couple of functions of note. The first couple are simply Test. These functions are called to actually run tests (either sending in a Type, array of Types, or an assembly). These then call TestClass which goes through and sees if any methods on the class are marked as a test. We do this by using the following attribute:

   1: #region Usings
   2: using System;
   3: using System.Collections.Generic;
   4: using System.Linq;
   5: using System.Text;
   6: #endregion
   7:  
   8: namespace MoonUnit.Attributes
   9: {
  10:     /// <summary>
  11:     /// Attribute used to denote a test function
  12:     /// </summary>
  13:     [AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
  14:     public class TestAttribute : Attribute
  15:     {
  16:         #region Constructor
  17:  
  18:         /// <summary>
  19:         /// Constructor
  20:         /// </summary>
  21:         public TestAttribute(long TimeOut=0)
  22:             : base()
  23:         {
  24:             Skip = false;
  25:             this.TimeOut = TimeOut;
  26:         }
  27:  
  28:         /// <summary>
  29:         /// Constructor
  30:         /// </summary>
  31:         /// <param name="ReasonForSkipping">Reason for skipping this test</param>
  32:         public TestAttribute(string ReasonForSkipping,long TimeOut=0)
  33:             : base()
  34:         {
  35:             this.Skip = true;
  36:             this.ReasonForSkipping = ReasonForSkipping;
  37:             this.TimeOut = TimeOut;
  38:         }
  39:  
  40:         #endregion
  41:  
  42:         #region Properties
  43:  
  44:         /// <summary>
  45:         /// Determines if the test should be skipped
  46:         /// </summary>
  47:         public bool Skip { get; private set; }
  48:  
  49:         /// <summary>
  50:         /// The reason for skipping this test
  51:         /// </summary>
  52:         public string ReasonForSkipping { get; private set; }
  53:  
  54:         /// <summary>
  55:         /// If it takes longer than this, consider it a timeout/failed test
  56:         /// </summary>
  57:         public long TimeOut { get; private set; }
  58:  
  59:         #endregion
  60:     }
  61: }

With this attribute, we can mark a method to be run by the system. For instance:

   1: class Example
   2: {
   3:     public void DoesNotRun(){}
   4:     [Test]
   5:     public void Runs(){}
   6: }

In this case Runs would be called by the system and DoesNotRun would not be. You can also tell the system to skip a test by giving an explanation. You can also give it an amount of time for it to wait. If it takes longer than a specified time, the test is considered a failure. The last bit of code that is of interest is when an error is detected (and thus the individual test is considered a failure), we need some way to record this. We do this with the aid of Output:

   1: #region Usings
   2: using System;
   3: using System.Collections.Generic;
   4: using System.Linq;
   5: using System.Text;
   6: using MoonUnit.Exceptions;
   7: using System.Reflection;
   8: #endregion
   9:  
  10: namespace MoonUnit.InternalClasses
  11: {
  12:     /// <summary>
  13:     /// Output class
  14:     /// </summary>
  15:     internal class Output
  16:     {
  17:         #region Constructor
  18:  
  19:         public Output()
  20:         {
  21:             MethodsCalled = new List<TestMethod>();
  22:         }
  23:  
  24:         #endregion
  25:  
  26:         #region Functions
  27:  
  28:         public void MethodCalled(MethodInfo MethodInformation, Exception Exception=null)
  29:         {
  30:             MethodsCalled.Add(new TestMethod(MethodInformation, Exception));
  31:         }
  32:  
  33:         public override string ToString()
  34:         {
  35:             string AssemblyLocation = Assembly.GetAssembly(this.GetType()).Location;
  36:             AssemblyName Name = AssemblyName.GetAssemblyName(AssemblyLocation);
  37:             StringBuilder Builder = new StringBuilder();
  38:             Builder.Append("<?xml version=\"1.0\" encoding=\"UTF-8\" ?>");
  39:             Builder.Append(string.Format("<MoonUnit><Header><FileLocation>{0}</FileLocation><Version>{1}</Version></Header><Tests>", AssemblyLocation, Name.Version));
  40:             foreach (TestMethod Method in MethodsCalled)
  41:             {
  42:                 Builder.Append(Method.ToString());
  43:             }
  44:             Builder.Append("</Tests><Footer></Footer></MoonUnit>");
  45:             return Builder.ToString();
  46:         }
  47:  
  48:         #endregion
  49:  
  50:         #region Properties
  51:  
  52:         public virtual List<TestMethod> MethodsCalled { get; private set; }
  53:  
  54:         #endregion
  55:     }
  56: }

This class simply gets called when a test method is called and the result, it then stores the results in a list and can later export the results into an XML string. The individual test records are stored in the following class:

   1: #region Usings
   2: using System;
   3: using System.Collections.Generic;
   4: using System.Linq;
   5: using System.Text;
   6: using MoonUnit.Exceptions;
   7: using System.Reflection;
   8: #endregion
   9:  
  10: namespace MoonUnit.InternalClasses
  11: {
  12:     /// <summary>
  13:     /// Holds information about test methods
  14:     /// </summary>
  15:     internal class TestMethod
  16:     {
  17:         #region Constructor
  18:  
  19:  
  20:         public TestMethod(MethodInfo MethodInformation,Exception Exception)
  21:         {
  22:             this.MethodInformation = MethodInformation;
  23:             this.Exception = Exception;
  24:         }
  25:  
  26:         #endregion
  27:  
  28:         #region Functions
  29:  
  30:         public override string ToString()
  31:         {
  32:             StringBuilder Builder = new StringBuilder();
  33:             Builder.Append(string.Format("<Test><Class name=\"{0}\" /><Method name=\"{1}\" />",
  34:                                             MethodInformation.DeclaringType.Name,
  35:                                             MethodInformation.Name));
  36:             if (Exception == null)
  37:                 Builder.Append("<Passed />");
  38:             else
  39:                 Builder.Append(string.Format("<Failed>{0}</Failed>", Exception.ToString()));
  40:             Builder.Append("</Test>");
  41:             return Builder.ToString();
  42:         }
  43:  
  44:         #endregion
  45:  
  46:         #region Properties
  47:  
  48:         public virtual MethodInfo MethodInformation { get; private set; }
  49:  
  50:         public virtual Exception Exception { get; set; }
  51:  
  52:         #endregion
  53:     }
  54: }

This class simply stores the method information and the result for an individual test and can export that data it to a string later. That's it. With these four classes, we have our set of classes for finding tests to run, running them, and storing the results.  That just leaves us with assertions and exceptions. Assertions are simply predicates that should be true at a specific position in our code. Which really just means they're a set of functions that determine if something is truly in the state that we're expecting. In some systems like nUnit, there's like 60 of these functions (most are overloaded versions of themselves). In xUnit, there's much fewer (mostly because they use generics, etc. to achieve the same things). In our case we're going to use the xUnit style (in fact many of them are the same name and look about the same, because... Well... There's not much to them really). Anyway, let's take a look at a couple:

   1: #region Usings
   2: using System;
   3: using System.Collections.Generic;
   4: using System.Linq;
   5: using System.Text;
   6: using MoonUnit.Exceptions;
   7: using System.Collections;
   8: using System.Text.RegularExpressions;
   9: #endregion
  10:  
  11: namespace MoonUnit
  12: {
  13:     /// <summary>
  14:     /// Used to make assertions
  15:     /// </summary>
  16:     public static class Assert
  17:     {
  18:         #region Functions
  19:  
  20:         #region DoesNotThrow
  21:  
  22:         /// <summary>
  23:         /// Used when a specific type of exception is not expected
  24:         /// </summary>
  25:         /// <typeparam name="T">Type of the exception</typeparam>
  26:         /// <param name="Delegate">Delegate called to test</param>
  27:         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>
  28:         public static void DoesNotThrow<T>(VoidDelegate Delegate, string UserFailedMessage = "Assert.DoesNotThrow<T>() Failed")
  29:         {
  30:             try
  31:             {
  32:                 Delegate();
  33:             }
  34:             catch (Exception e)
  35:             {
  36:                 if (e is T)
  37:                     throw new DoesNotThrow(typeof(T), e, UserFailedMessage);
  38:             }
  39:         }
  40:  
  41:         /// <summary>
  42:         /// Used when a specific type of exception is not expected and a return value is expected when it does not occur.
  43:         /// </summary>
  44:         /// <typeparam name="T">Exception type</typeparam>
  45:         /// <typeparam name="R">Return type</typeparam>
  46:         /// <param name="Delegate">Delegate that returns a value</param>
  47:         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>
  48:         /// <returns>Returns the value returned by the delegate or thw appropriate exception</returns>
  49:         public static R DoesNotThrow<T, R>(ReturnObjectDelegate<R> Delegate, string UserFailedMessage = "Assert.DoesNotThrow<T,R>() Failed")
  50:         {
  51:             try
  52:             {
  53:                 return Delegate();
  54:             }
  55:             catch (Exception e)
  56:             {
  57:                 if (e is T)
  58:                     throw new DoesNotThrow(typeof(T), e, UserFailedMessage);
  59:             }
  60:             return default(R);
  61:         }
  62:  
  63:         #endregion
  64:  
  65:         #region Empty
  66:  
  67:         /// <summary>
  68:         /// Determines if an IEnumerable is empty or not
  69:         /// </summary>
  70:         /// <param name="Collection">Collection to check</param>
  71:         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>
  72:         public static void Empty(IEnumerable Collection, string UserFailedMessage = "Assert.Empty() Failed")
  73:         {
  74:             if (Collection == null)
  75:                 throw new ArgumentNullException("Collection");
  76:             foreach (object Object in Collection)
  77:                 throw new NotEmpty(Collection,UserFailedMessage);
  78:         }
  79:  
  80:         #endregion
  81:  
  82:         #region Equal
  83:  
  84:         /// <summary>
  85:         /// Determines if two objects are equal
  86:         /// </summary>
  87:         /// <typeparam name="T">Object type</typeparam>
  88:         /// <param name="Expected">Expected result</param>
  89:         /// <param name="Result">Actual result</param>
  90:         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>
  91:         public static void Equal<T>(T Expected, T Result,string UserFailedMessage="Assert.Equal() Failed")
  92:         {
  93:             Equal(Expected, Result, new InternalClasses.EqualityComparer<T>(), UserFailedMessage);
  94:         }
  95:  
  96:         /// <summary>
  97:         /// Determines if two objects are equal
  98:         /// </summary>
  99:         /// <typeparam name="T">Object type</typeparam>
 100:         /// <param name="Expected">Expected result</param>
 101:         /// <param name="Result">Actual result</param>
 102:         /// <param name="Comparer">Comparer used to compare the objects</param>
 103:         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>
 104:         public static void Equal<T>(T Expected, T Result, IEqualityComparer<T> Comparer, string UserFailedMessage = "Assert.Equal() Failed")
 105:         {
 106:             if (!Comparer.Equals(Expected, Result))
 107:                 throw new NotEqual(Expected, Result, UserFailedMessage);
 108:         }
 109:  
 110:         #endregion
 111:  
 112:         #region False
 113:  
 114:         /// <summary>
 115:         /// Determins if something is false
 116:         /// </summary>
 117:         /// <param name="Value">Value</param>
 118:         /// <param name="UserFailedMessage">Message passed to the output in the case of a failed test</param>
 119:         public static void False(bool Value, string UserFailedMessage = "Assert.False() Failed")
 120:         {
 121:             if (Value)
 122:                 throw new NotFalse(UserFailedMessage);
 123:         }
 124:  
 125:         #endregion
 126:  
 127:         #endregion
 128:     }
 129: }

The code above only contains four of the assertions that I've added to the system. Anyway, the basic Assert class is static with a number of static functions. In this case we're looking at DoesNotThrow, Empty, Equal, and False. DoesNotThrow takes in a delegate, which it calls, and sees if it throws a specific error, the Empty function just checks if an IEnumerable is empty (not null, but empty), the equal function checks if two items are equal using an IEqualityComparer, and False just checks if a boolean value is false. Every single assertion is like this, very simple, maybe 5 lines of code, and that's it. The Equals function (and others like it) are a bit more complex as you need to write an IEqualityComparer but I've written one that the system will use by default:

   1: #region Usings
   2: using System;
   3: using System.Collections.Generic;
   4: using System.Linq;
   5: using System.Text;
   6: using MoonUnit.Exceptions;
   7: using System.Reflection;
   8: using System.Collections;
   9: #endregion
  10:  
  11: namespace MoonUnit.InternalClasses
  12: {
  13:     /// <summary>
  14:     /// Internal equality comparer
  15:     /// </summary>
  16:     /// <typeparam name="T">Data type</typeparam>
  17:     internal class EqualityComparer<T>:IEqualityComparer<T>
  18:     {
  19:         #region Functions
  20:  
  21:         public bool Equals(T x, T y)
  22:         {
  23:             if (!typeof(T).IsValueType
  24:                 || (typeof(T).IsGenericType
  25:                 && typeof(T).GetGenericTypeDefinition().IsAssignableFrom(typeof(Nullable<>))))
  26:             {
  27:                 if (Object.Equals(x, default(T)))
  28:                     return Object.Equals(y, default(T));
  29:                 if (Object.Equals(y, default(T)))
  30:                     return false;
  31:             }
  32:             if (x.GetType() != y.GetType())
  33:                 return false;
  34:             if (x is IEnumerable && y is IEnumerable)
  35:             {
  36:                 EqualityComparer<object> Comparer = new EqualityComparer<object>();
  37:                 IEnumerator XEnumerator = ((IEnumerable)x).GetEnumerator();
  38:                 IEnumerator YEnumerator = ((IEnumerable)y).GetEnumerator();
  39:                 while (true)
  40:                 {
  41:                     bool XFinished = !XEnumerator.MoveNext();
  42:                     bool YFinished = !YEnumerator.MoveNext();
  43:                     if (XFinished || YFinished)
  44:                         return XFinished & YFinished;
  45:                     if (!Comparer.Equals(XEnumerator.Current, YEnumerator.Current))
  46:                         return false;
  47:                 }
  48:             }
  49:             if (x is IEquatable<T>)
  50:                 return ((IEquatable<T>)x).Equals(y);
  51:             if (x is IComparable<T>)
  52:                 return ((IComparable<T>)x).CompareTo(y) == 0;
  53:             if (x is IComparable)
  54:                 return ((IComparable)x).CompareTo(y) == 0;
  55:             return x.Equals(y);
  56:         }
  57:  
  58:         public int GetHashCode(T obj)
  59:         {
  60:             throw new NotImplementedException();
  61:         }
  62:  
  63:         #endregion
  64:     }
  65: }

Oh and if you're at all experienced with xUnit's code base, it's pretty much the same as the one they use. Slightly different but there's only so many ways to write these things. Anyway, if you look at the code, if the assertion fails you will notice that it throws an exception. These exceptions use the following base class:

   1: #region Usings
   2: using System;
   3: using System.Collections.Generic;
   4: using System.Linq;
   5: using System.Text;
   6: #endregion
   7:  
   8: namespace MoonUnit.BaseClasses
   9: {
  10:     /// <summary>
  11:     /// Base exception class
  12:     /// </summary>
  13:     public class BaseException:Exception
  14:     {
  15:         #region Constructor
  16:  
  17:         /// <summary>
  18:         /// Constructor
  19:         /// </summary>
  20:         /// <param name="ExceptionText">Exception Text</param>
  21:         /// <param name="Expected">Expected value</param>
  22:         /// <param name="Result">Actual result</param>
  23:         public BaseException(object Expected, object Result,string ExceptionText)
  24:             : base(ExceptionText)
  25:         {
  26:             this.Expected = Expected;
  27:             this.Result = Result;
  28:         }
  29:  
  30:         #endregion
  31:  
  32:         #region Properties
  33:  
  34:         /// <summary>
  35:         /// Expected result
  36:         /// </summary>
  37:         public virtual object Expected { get; private set; }
  38:  
  39:         /// <summary>
  40:         /// Actual result
  41:         /// </summary>
  42:         public virtual object Result { get; private set; }
  43:  
  44:         #endregion
  45:  
  46:         #region Functions
  47:  
  48:         public override string ToString()
  49:         {
  50:             StringBuilder Builder = new StringBuilder();
  51:             Builder.Append(string.Format("<Expected>{0}</Expected><Result>{1}</Result><ErrorText>{2}</ErrorText><Trace>{3}</Trace><ErrorType>{4}</ErrorType>",
  52:                                         Expected, 
  53:                                         Result, 
  54:                                         this.Message, 
  55:                                         this.StackTrace,
  56:                                         this.GetType().Name));
  57:             return Builder.ToString();
  58:         }
  59:  
  60:         #endregion
  61:     }
  62: }

The exception base class just takes in the expected value and the result and spits out the result to text when ToString is called. Fairly simple. An an example of an exception class would look like this:

   1: #region Usings
   2: using System;
   3: using System.Collections.Generic;
   4: using System.Linq;
   5: using System.Text;
   6: using MoonUnit.BaseClasses;
   7: #endregion
   8:  
   9: namespace MoonUnit.Exceptions
  10: {
  11:     /// <summary>
  12:     /// Exception thrown if two items are equal
  13:     /// </summary>
  14:     public class Equal : BaseException
  15:     {
  16:         #region Constructor
  17:  
  18:         /// <summary>
  19:         /// Constructor
  20:         /// </summary>
  21:         /// <param name="ExceptionText">Exception Text</param>
  22:         /// <param name="Expected">Expected value</param>
  23:         /// <param name="Result">Actual result</param>
  24:         public Equal(object Expected, object Result,string ExceptionText)
  25:             : base(Expected, Result, ExceptionText)
  26:         {
  27:         }
  28:  
  29:         #endregion
  30:     }
  31: }
The vast majority of the exception classes just look like that (note that the one above is actually thrown from the NotEqual function that I didn't show you and not the Equal function). So create about 20 assertions, have 20 exceptions that match up, and we're done. With this we can create a class library, have the system load it, look through for our tests and give us results in an XML doc. The only thing left is to write a front end of some description, but I'll leave that for part 2. Oh and if you've paid attention to the namespaces, yes, I have indeed named this thing MoonUnit, and yes I will be releasing this on Codeplex. Anyway, take a look, leave feedback, and happy coding.


Comments