Homemade Lightweight Automapper Part 2

Again, please forgive any code mistakes. I've written this post up largely without the help of a compiler to help catch the bugs. The initial solution I came up with was some sort of case statement but after a few hours of research I came to the realization that C# simply doesn't support a case statements on types. A dictionary object was my next port of call. More specifically a Dictionary of Dictionaries. The nice thing about dictionaries in C# is that you can access them like arrays, using a key in place of the index. So a Dictionary of Dictionaries can be used the same way as a 2 dimensional array. In our case we want to use types as our keys so our dictionary syntax becomes:

dict[typeof(Source)][typeof(Dest)];

Linq syntax uses Func<TSource, TDest> for L2SQL and wraps it into a much less restricted Expression<...> for L2OBJs. It is really easy to pull the Func out of the Expression so our actual Projections are going be of type Expression<Func<TSource, TDest>>

Our key will be the destination type. Given that, our inner dictionary will need to be a dictionary of type:

IDictionary<TDest, Expression<Func<TSource, TDest>>>;

Where TDest is our Key and our Expression is our Value. If that looks ugly, it's because it is.

Our outer dictionary will need to hold our inner dictionary and be accessed by our source type, so it's type will be:

IDictionary<TSource, IDictionary<TDest, Expression<Func<TSource, TDest>>>>;

That is *really* ugly. Not to mention that our code isn't generic at all, and our dictionary object is strongly typed so it won't just accept actual types. It needs the more generic Type type. And for convenience sake, MS have included a non generic Expression class (which is as it turns out more generic.)

IDictionary<Type, IDictionary<Type, Expression>>;

That's about as nice as we're going to get in this case I think. There's not a whole lot we can do to clean that up. So what's the next best thing to getting rid of the dust in the house? Sweeping it under the rug! So what we want to do is abstract the nastiness away from the programmer.

Currently we can only set up our projection dictionary by doing something like the following:

IDictionary<Type, IDictionary<Type, Expression>> projections = new Dictionary<Type, Dictionary<Type, Expression>>() {
   { typeof(Source1), new Dictionary<Type, Expression>() {
      { typeof(Dest1), (Expression)(() => {
         return new Dest1() {
            ...
         }
      }) }
      , { typeof(Dest2), (Expression)(() => {
         return new Dest2() {
            ...
         }
      }) }
   } }
   , { typeof(Source2), new Dictionary<Type, Expression>() {
      { typeof(Dest3), (Expression)(() => {
         return new Dest3() {
            ...
         }
      }) }
      , { typeof(Dest4), (Expression)(() => {
         return new Dest4() {
            ...
         }
      }) }
   } }
};

Which is pretty brutal, especially since in MVC.NET you're going to be working with probably at least 10+ domain models and twice as many view models. The whole point of this exercise is to reduce the amount of code we have to write while still having flexibility. So, lets get to refactoring.

The obvious starting point is to wrap the previous code into an Add function. To use the Source and Dest types we'll have to make it a generic function, which is fine:

void Add<TSource, TDest>(IDictionary<Type, IDictionary<Type, Expression>> Projections, Expression Expression) {
   Type source = typeof(TSource);
   Type dest = typeof(TDest);
   
   if(true == Projections.ContainsKey(source)) {
      Projections[source].Add(Expression);
   } else {
      IDictionary<Type, Expression>> d = new Dictionary<Type, Expression>>();
      d.Add(dest, Expression);

      Projections.Add(source, d);
   }
}

IDictionary<Type, IDictionary<Type, Expression>> projections =
   new Dictionary<Type, IDictionary<Type, Expression>>
;

Expression e = (Expression)(() => {
   return new TDest() {
      ...
   }
});

Add<Source, Dest>(projections, e);

That's not bad, but we can do better. After all, you don't want to have to type all that mess every time. One easy thing we can do to clean the syntax up is move our Expressions to variables rather than methods.

Expression e = (Expression)(o => new TDest() {
      ...
   }
);

But we're not done yet. Since the generic Expression class derives from the Expression class, we can do away with the casting...

Expression e = o => new TDest() {
      ...
   }
;

The Add function could also be refactored more. Instead of having to write Add<Source, Dest>(d, e) for every single projection, wouldn't it be nicer to simply split it out into two? Add<Source>(d) and Add<Dest>(d, e).

Lets do that.

void Add<TSource>(IDictionary<Type, IDictionary<Type, Expression>> Projections, IDictionary<Type, Expression> Dictionary) {
   Projections.Add(typeof(TSource), Dictionary);
}

void Add<TDest>(IDictionary<Type, Expression> Dictionary, Expression Expression) {
   Dictionary.Add(typeof(TDest), Expression);
}

One of the nice features of C# is the ability to make Extension methods. Make your method static and slap it in a static class, add the 'this' keyword to the first parameter and the type of object that it is will allow you to use that method as if it was a part of the original class while abstracting away the parameter. Lets do that here:

static class Mapper {
   static void Add<TSource>(this IDictionary<Type, IDictionary<Type, Expression>> Projections, IDictionary<Type, Expression> Dictionary) {
      Projections(typeof(TSource), Dictionary);
   }

   static void Add<TDest>(this IDictionary<Type, Expression> Dictionary, Expression Expression) {
      Dictionary.Add(typeof(TDest), Expression);
   }
}

Our Add methods can now be called like so:

d2.Add<Dest>(e);
d1.Add<Source>(d2);

Almost done with the Add functions. Rather than calling d1.Add(); many times, wouldn't it be nicer to simply chain them?

static IDictionary<Type, IDictionary<Type, Expression>> Add<TSource>(this IDictionary<Type, IDictionary<Type, Expression>> Projections, IDictionary<Type, Expression> Dictionary) {
   Projections.Add(typeof(TSource), Dictionary);
   return Projections;
}

static void Add<TDest>(this IDictionary<Type, Expression> Dictionary, Expression Expression) {
   Dictionary.Add(typeof(TDest), Expression);
   return Dictionary;
}

Thus we can call them:

d2
   .Add<Dest1>(e1)
   .Add<Dest2>(e2)
   ...
;

d1
   .Add<Source1>(d2)
   .Add<Source2>(d3)
   ...
;

Okay, one last update to the Add function and we'll be close to done. In our haste to refactor, we've committed the cardinal programmer sin. Optimizing at the start. We've made things a little TOO generic. I've always found the generic method syntax (f<t>()) to be a little verbose. The other great thing about extension methods is that if we play our cards right, the 'this' parameter can actually imply the type of the generic, abstracting it away so that it looks like a normal function. Once our projection goes into the dictionary it loses its type and becomes a standard Expression rather than an Expression<> so we can't actually do anything for the outer dictionary Add method, but we can certainly clean up the inner Add.

static Dictionary<Type, Expression> Add<TSource, TDest>(this Dictionary<Type, Expression> Dictionary, Expression<Func<TSource, TDest>> Expression) {
   Dictionary.Add(typeof(TDest), Expression);
   return Dictionary;
}

Now for one final cleanup method:

static Expression GetProjection(Type Source, Type Dest) {
   return Projections[Source][Dest];
}

Altogether our code is looking pretty nice so far:

public static class Mapper {
   private static Dictionary<Type, Expression> Add<TDest>(this Dictionary<Type, Expression> Dictionary, Expression Expression) {
      Dictionary.Add(typeof(TDest), Expression);
      return Dictionary;
   }

   private static Dictionary<Type, Expression> Add<TSource, TDest>(this Dictionary<Type, Expression> Dictionary, Expression<Func<TSource, TDest>> Expression) {
      Dictionary.Add(typeof(TDest), Expression);
      return Dictionary;
   }

   private static Dictionary<Type, Dictionary<Type, Expression>> Add<TSource>(this Dictionary<Type, Dictionary<Type, Expression>> Dictionary, Dictionary<Type, Expression> Projections) {
      Dictionary.Add(typeof(TSource), Projections);
      return Dictionary;
   }

   private static Expression GetProjection(Type Source, Type Dest) {
      return Projections[Source][Dest];
   }

   private static Dictionary<Type, Dictionary<Type, Expression>> Projections {
      get {
         return new Dictionary<Type, Dictionary<Type, Expression>>()
            .Add<Source1>(Source1Projections)
            .Add<Source2>(Source2Projections)
         ;
      }
   }

   private static Dictionary<Type, Expression> Source1Projections{
      get {
         return new Dictionary<Type, Expression>()
            .Add(Dest1.Projection)
            .Add(Dest2.Projection)
         ;
      }
   }

   private static Dictionary<Type, Expression> Source2Projections{
      get {
         return new Dictionary<Type, Expression>()
            .Add(Dest3.Projection)
            .Add(Dest4.Projection)
         ;
      }
   }
}

You can define your actual projections in your view models:

public class Dest1 {
    <properties>
    <constructors>
    <methods>

    public static Expression<Func<Source1, Dest1>> Projection {
       get {
          return o => new Dest1() {
             ...
          };
       }
    }
}

There's probably more I could do to refactor this. There really isn't much reason the projections need to be properties rather than standard variables for example but I'll leave it there. This post is becoming pretty long winded, so I'll save the rest for Part 3. We're 2/3rds of the way there so stay tuned for the Actual implementing the projection methods that will let you use the mapper.

Homemade Lightweight Automapper Part 1

I've been using MVC3.NET for just over a year now in a project for work. I find myself liking the approach, I feel more in control of the code that I write, and while I admire MS efforts, I am glad to be rid of the horrible overhead that is the ViewState.

Using Entity Framework, I've learned a *lot* about linq. I went with a POCO approach for my models. I feel that this approach is much more elegant and lightweight than relying on EF to map everything out, and the less auto-generated nightmarish code the better IMHO.

If there is one thing I could complain about MS's bash at the MVC paradigm, it is how code heavy it is. Classes and classes and more classes. And did I mention classes? Once you get used to the slightly over engineered verboseness of it though, you do begin to appreciate the perks that go with it.

In the project I'm currently working on, we decided early on to start work with the Kendo JS framework, which is pretty shiny, and to try and help reduce code a bit, we started using the AutoMapper 3rd party library fairly early on.

Well, if you've used Kendo, you'll know that for the MVC side of things, all of the 'big' data display controls such as grids or lists are designed fairly heavily around taking their data in IQueryable format. They will take any IEnumerable, but if you have more than a few hundred records being pulled from your database, you don't want to materialize those records into memory until you need to, or things get slooooooooowwww very quickly.

Let me back up and give a brief outline of Entity Framework and Linq. There are two kinds of linq that use the same syntax and both work on most any sort of container object... arrays, dictionaries, lists, etc. Linq to Object works on objects in memory, Linq to SQL works specifically in a monadic way to generate a SQL query that is only run at the last minute.

You generally use L2SQL on IQueryables and L2OBJ on everything else. Problems arise when casting IQueryables to their parent interface IEnumerables because then you can quickly lose track of where your data is and run into performance issues (among other things).

After jumping into the deep end (When I started on this project, I personally had *very* little MVC experience no Kendo experience, no linq or Entity Framework experience and had been working in .NET 2.0 until that time) we quickly discovered that Sorting, Grouping, and Paging, the main 3 things you want to do with data in a Grid, did not play nicely unless you were working with IQueryables... oh they work fine if you only have a few hundred records, but once your DB starts filling up, a few thousand records being sorted, paged and grouped in memory rather than at the DB hurts really badly.

The AutoMapper had worked reasonably well up until that point but there was one hitch: It did not work with IQueryables. I did a bit of research and discovered that it indeed worked with IQueryables if you used a certain syntax... something along the lines of Project().To(), but the line of code would always inexplicably crash, no matter how much I coaxed it. Pretty sure that the particular version of the library that we were using had a bug, and unfortunately at the time it was not an option to upgrade.

This lead me to start looking into ways I could simply do it myself. After all, the Automapper code is fairly ugly to work with anyway, albeit fairly powerful if you work it right. I'll use the next 1-2 blog posts to run through how to create a rudimentary automapper that works fairly well in most complex cases.

Nifty JavaScript Approach to Large Arrays

I figured this method out a while back in regards to storing things fast but also keeping flexibility of indexed arrays. I by no means am an expert in JS, and I learn more every day so if anyone has a better method, definitely let me know!

JavaScript natively has two types of arrays, a standard index array a[0], a[1], a[2], etc., and a key/value dictionary which uses a very similar syntax but with the difference being that you can use any string or number as your index value: a[0], a[5], a['hi'].

The problem is twofold. 1.) Storing large amounts of data in an index array means that you have to run through a loop when doing a look up. If you have less than a few hundred records this isn't so bad, but once you have a few thousand it presents a significant slowdown. And 2.) To loop through a dictionary array you need to use a for...in loop (JS version of a foreach/forall) which is *really* slow compared to a regular for loop.

Neither an index or a dictionary array are very good options for large amounts of data, or I should say specific types of large amounts of data as there are things which both do very well.

Therefore, I came up with a nice little method that has the flexibility of both. The solution is this, and it's a very simple one with the only drawback being a slight memory increase: Use both types of array at once.

In its simplest form it is the following class:

function List() {  
   this.Indexed = [];  
   this.Dictionary = [];
}

List.prototype.KeyValPair = function(Key, Value) {  
   this.Key = Key;  
   this.Value = Value;  
}

List.prototype.GetByKey = function(Key) {
   var i = this.Dictionary[Key];

   return this.Indexed[i].Value;
}

List.prototype.GetAtIndex = function(i) {
   return this.Indexed[i].Value;
}

List.prototype.Add = function(Key, Value) {
   var item = new this.KeyValPair(Key, Value);

   this.AddItem(item);
}

List.prototype.AddItem = function(KeyValPair) {
   this.Indexed.push(KeyValPair);
   this.Dictionary[KeyValPair.Key] = this.Indexed.length - 1;
}

List.prototype.AddRange = function(KeyValPairs) {
   for(var i = 0, len = KeyValPairs.length; i < len; ++i) {
      this.AddItem(KeyValPairs[i]);
   }
}

List.prototype.RemoveAtIndex = function(i) {
   if(undefined === this.Indexed[i]) {
      return;
   }

   var key = this.Indexed[i].Key;

   this.Indexed.splice(i, 1);
   this.Dictionary[key] = undefined;

   this.SyncList(i);
}

List.prototype.RemoveByKey = function(Key) {
   var i = this.Dictionary[Key];
   if(undefined === i) {
      return;
   }

   this.Indexed.splice(i, 1);
   this.Dictionary[Key] = undefined;

   this.SyncList(i);
}

List.prototype.RemoveIndexRange = function(From, To) {
   for(var i = From; i < To; ++i) {
      var key = this.Indexed[i].Key;

      this.Dictionary[key] = undefined;
   }

   this.Indexed.splice(From, To - From);

   this.SyncList(From);
}

List.prototype.RemoveKeyRange = function(Keys) {
   var minI = this.Indexed.length;

   for(var i = 0, len = Keys.length; i < len; ++i) {
      var key = Keys[i];
      var index = this.Dictionary[key];
      
      if(minI > index) {
         minI = index;
      }

      this.Dictionary[key] = undefined;
      this.Indexed.splice(index, 1);
   }

   this.SyncList(minI);
}

List.prototype.SyncList = function(From) {
   if(undefined === From) {
      From = 0;
   }

   for(var i = From, len = this.Indexed.length; i < len; ++i) {
      var keyValPair = this.Indexed[i];

      this.Dictionary[keyValPair.Key] = i;
   }
}

List.prototype.Clear = function() {
   this.Indexed.length = 0;
   this.Dictionary.length = 0;
}

I do have to apologize for any mistakes, I've written this from memory and not run it through any checking for errors. You can of course expand this to have AddRange and RemoveRange functionality fairly easily.

The greatest advantage of this approach is that it abstracts all need for looping to native code when it comes to doing a lookup. The only drawback is having to resync your list every time you remove an item.

I personally have used this method quite often and found it extremely useful for sifting through large amounts of data in JS.

Edit: I fleshed out the class a little bit... simply because I'm a code junky and couldn't help myself, so enjoy.

My thoughts on Always On

Note: This is a semi-rant, but I hope to have presented myself well. Also, I haven't even gotten into the whole 'Always On DRM stops pirates' argument. I may do another blog on that particular gem in the near future.

Having been a major player of World of Warcraft for the last 8 years, having garnered approx. 200+/- days of played time (4800+/- hours) in essentially an Always On environment, you would think that I would be totally fine with the current push towards it as the 'next big thing'. Well here are my thoughts on it.

I come from Australia where the ISPs have no clue what 'unlimited data plan' means. I spent around 5 of those 8 years playing in Australia with 500-2k latency every day constantly balancing the fact that if I downloaded too much that month, my internet would be so throttled that playing anything online would be an exercise in extreme patience.

The next 2 years would be spent in the US under the machinations of Comcast, in an area where apparently our ISP didn't know the meaning of the word stable internet. For 8 months my wife and I would be constantly calling them, wondering why our internet was either bouncing from 1 mb to 11 mb download or disconnected entirely. Needless to say, raiding in WoW was severely hit and miss during this time. At least in Australia I could simply cut back on my YouTube browsing for the month to be able to play.

The last year has been spent with AT&T (and I am currently on hiatus with WoW). Compared to my previous ISPs, AT&T have been like a dream. I think we have only had a few minor disruptions after a year of being with them. However, I have heard stories from people who have used AT&T in other areas that they can be just as hit and miss as my experience with Comcast. Especially after a large storm, which, as I currently live in Alabama, happens quite frequently.

The two major 'always on' games that have been published of late, ActiBlizz's Diablo III and EA's Sim City both had the most horrible launches in gaming history. Bar none.

I actually was a big fan of Diablo II, so when Blizzard announced their annual pass deal that allowed one to receive a free copy of D3 for paying a year's subscription, I was all into it. Why not? I was going to be paying for a year anyway. After installing D3 on launch and getting to play for maybe 30 minutes, I put it down and never picked it up again.

All that to say this: The infrastructure is simply not in place. It isn't. It may be one day, but that day is not now. It won't be now by the time the xBox720 comes out and it probably won't be now until a.) ISPs start swapping to fiber optic cable exclusively and b.) Publishers start making sure their games actually are properly stress tested before release rather than simply shoving them onto the consumers and letting them take the brunt of the fallout.

Even if it (the infrastructure) was ready now at this very moment. The consumer base is clearly not. I have read maybe 1 in 40 (if that) comments actually defending the always on paradigm instead of simply blasting it.

The main gamer demographic is still getting older, as kids who grew up playing Mario get older. It hasn't stabilized yet and probably won't for at  least 20 years. The fact of the matter is that your average gamer now most likely a.) has a job b.) is married and c.) has kids. This cuts the average free time from 16 hours a day to 8 hours, to 2 hours, to maybe 1/2 an hour every 2 days if you're lucky. What these customers don't want is to get to that incredibly rare free time slot and see this: 'Error logging in. Server down.' or this: 'Cannot connect to the internet. Please call your service provider.'

The fallout of publishers jumping on the always on bandwagon may not be instantaneous, but it will be inevitable as more and more consumers get pushed into finding better ways to spend their time than wasting it by staring at a login screen.

I don't know what they think or how they come to the conclusions that they do at major game companies, but statements affirming consumer consent to always on are clearly and unequivocally false. I know that I will probably not be purchasing the next xBox. I know that EA has lost a potential buyer for at least a long time, and many more publishers are heading in the same direction. And finally, I know that there are literally thousands of older games out there that are looking really good to play right now. None of them have always on, or DRM of any kind and to me they look better and better every time someone mentions 'always on' being the way the industry is headed.