The Inquisitive Coder - Davy Brion's Blog » Build Your Own DAL

Build Your Own Data Access Layer: Enabling Bulk Inserts

Davy Brion — Mon, 12 Oct 2009 10:29:43 +0000

Note: This post is part of a series. Be sure to read the introduction here.

I know i wrapped up the series already, but i just had to add the ability to do bulk inserts to this data layer so i figured i'd might as well post about it. Ayende already talked about how to enable the ability to batch inserts (or updates and deletes) here and here so i'm going to skip that part. I used the exact same trick and created a PublicSqlCommandSet class which wraps the hidden SqlCommandSet class. Again, if you have no idea what i'm talking about in that last sentence then you need to read Ayende's 2 posts that i just linked to

After that, adding the bulk insert feature to the DAL was as simple as creating this class:

    public class BulkInsertAction : DatabaseAction
    {
        public BulkInsertAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
            EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
            : base(connection, transaction, metaDataStore, hydrater, sessionLevelCache) {}
 
        public void Insert(IEnumerable entities, int batchSize, int commandTimeOut)
        {
            var tableInfo = MetaDataStore.GetTableInfoFor();
            var insertStatement = tableInfo.GetInsertStatementWithoutReturningTheIdentityValue();
 
            var sqlCommandSet = new PublicSqlCommandSet { CommandTimeout = commandTimeOut, Connection = GetConnection(), Transaction = GetTransaction() };
 
            foreach (var entity in entities)
            {
                var currentCommand = CreateCommand();
                currentCommand.CommandText = insertStatement;
 
                foreach (var parameterInfo in tableInfo.GetParametersForInsert(entity))
                {
                    currentCommand.CreateAndAddInputParameter(parameterInfo.DbType, "@" + parameterInfo.Name, parameterInfo.Value);
                }
 
                sqlCommandSet.Append(currentCommand);
 
                if (sqlCommandSet.CommandCount == batchSize)
                {
                    ExecuteCurrentBatch(sqlCommandSet);
                    sqlCommandSet = new PublicSqlCommandSet { CommandTimeout = commandTimeOut, Connection = GetConnection(), Transaction = GetTransaction() };
                }
            }
 
            if (sqlCommandSet.CommandCount > 0)
            {
                ExecuteCurrentBatch(sqlCommandSet);
            }
        }
 
        private void ExecuteCurrentBatch(PublicSqlCommandSet sqlCommandSet)
        {
            try
            {
                sqlCommandSet.ExecuteNonQuery();
            }
            finally
            {
                sqlCommandSet.Dispose();
            }
        }
    }

And then adding this to the Session class:

        public void BulkInsert(IEnumerable entities)
        {
            CreateAction().Insert(entities, 50, 200);
        }

Obviously, the method signature was also added to the ISession interface. The batch-size and commandtimeout parameters are currently hardcoded but they should come from some kind of configuration file.

All in all, pretty easy stuff

Build Your Own Data Access Layer: Conclusions

Davy Brion — Sat, 29 Aug 2009 13:42:10 +0000

Note: This post is part of a series. Be sure to read the introduction here.

Building your own DAL is almost never a cost-efficient solution. In this case, i wrote this DAL in 24 working hours, but it is limited in scope, power, flexibility and functionality. Having said that, i do think it's better than every single custom DAL i've come across so far. Taking it to the next step however would take a lot more effort, which truly is never worth it. As you try to provide more functionality, overall complexity of your custom DAL will increase heavily and the effort you'll eventually spend on it will more than outweigh any of the downsides that might come with using something that already exists. Building your own DAL is an undertaking that should always be questioned, and shouldn't be considered unless the alternative of doing so is even worse.

If however, you're in a situation where it does make sense, then i hope this series might have been helpful for you. I've shown that you can come up with something relatively decent without having to resort to code generation, without having to spend an insane amount of effort on it, and without having to write repetitive and error-prone code in your application code. Those were the goals i had in mind when i started working on this DAL, and i think i've succeeded at achieving those goals. The final result is an easy-to-use DAL which is far from as powerful as already existing solutions, but it is pretty good for the scenario's where we intend to use this.

The code itself is clear, easy to maintain and in some cases, very easy to extend as well. In total, this DAL is slightly less than 1100 lines of code, and i think the complexity of the code is relatively low so everyone should be able to understand what's going on, how it works, and where things could be modified in order to fix issues or to add new features.

Also, this series of blog posts helps in figuring out how it works since pretty much every aspect of it is now documented pretty extensively

All in all, i had a lot of fun in writing both the DAL and this series of posts (which took another 8 hours in total).

In the introduction of this series, i mentioned the following:

The purpose of this series is basically to:
* Show you that you really don’t need to resort to code generation to build your own custom DAL
* Show you what kind of complexity is involved with the implementation of a good DAL
* Convince you that you typically are better off with simply using something that is already available as a mature, powerful and proven solution

So tell me, did the series succeed in these listed goals? Would you still go for code generation if you had to create a custom DAL? Would you still prefer to use a custom DAL over something that already exists? How would you react to having to use this particular DAL? What would need to be added or modified before you would find it acceptable? What are your thoughts on this in general?

I'd be very interested in hearing about it, so please do post your opinions in the comments

Build Your Own Data Access Layer: Bringing It All Together

Davy Brion — Sat, 29 Aug 2009 13:42:02 +0000

Note: This post is part of a series. Be sure to read the introduction here.

By now we've already covered everything that this DAL has to offer, which admittedly isn't all that much. All of the classes you've seen so far are pretty good at whey they should do, but nobody in their right mind would want to use any of these things directly in application code. Any easy-to-use DAL should offer a simple facade which sits on top of the underlying system and makes it very easy to perform the most typical tasks that you need it to perform for you. You shouldn't need to know about specific classes to be able to use it (that goes for most good frameworks and libraries btw).

So once again, i based my approach on what NHibernate does, and with that the ISession interface was born:

    public interface ISession : IDisposable
    {
        void Commit();
        void Rollback();
 
        IQuery CreateQuery(string sql);
        IQuery CreateQuery(string whereClause);
 
        TEntity Get(object id);
        IEnumerable FindAll();
 
        TEntity Insert(TEntity entity);
        TEntity Update(TEntity entity);
        void Delete(TEntity entity);
 
        TableInfo GetTableInfoFor();
 
        void ClearCache();
        void RemoveFromCache(object entity);
        void RemoveAllInstancesFromCache();
 
        SqlConnection GetConnection();
        SqlTransaction GetTransaction();
    }

Everything that you need to be able to do with this DAL is provided by this single interface. And the implementation of the Session class is very easy as well, since we can simply delegate pretty much everything to each of the classes we've covered in the other posts in the series:

    public class Session : ISession
    {
        private readonly string connectionString;
        private SqlConnection connection;
        private SqlTransaction transaction;
        private readonly MetaDataStore metaDataStore;
        private readonly EntityHydrater hydrater;
        private readonly SessionLevelCache sessionLevelCache;
 
        public Session(string connectionString, MetaDataStore metaDataStore)
        {
            this.connectionString = connectionString;
            this.metaDataStore = metaDataStore;
            sessionLevelCache = new SessionLevelCache();
            hydrater = new EntityHydrater(metaDataStore, this, sessionLevelCache);
        }
 
        private void InitializeConnection()
        {
            connection = new SqlConnection(connectionString);
            connection.Open();
            transaction = connection.BeginTransaction();
        }
 
        public SqlConnection GetConnection()
        {
            if (connection == null)
            {
                InitializeConnection();
            }
 
            return connection;
        }
 
        public SqlTransaction GetTransaction()
        {
            if (transaction == null)
            {
                InitializeConnection();
            }
 
            return transaction;
        }
 
        public IQuery CreateQuery(string sql)
        {
            var command = GetConnection().CreateCommand();
            command.Transaction = GetTransaction();
            command.CommandText = sql;
            return new Query(command, metaDataStore, hydrater);
        }
 
        public IQuery CreateQuery(string whereClause)
        {
            return CreateQuery(metaDataStore.GetTableInfoFor().GetSelectStatementForAllFields() + " " + whereClause);
        }
 
        public TableInfo GetTableInfoFor()
        {
            return metaDataStore.GetTableInfoFor();
        }
 
        public void Commit()
        {
            transaction.Commit();
        }
 
        public void Rollback()
        {
            transaction.Rollback();
        }
 
        public void Dispose()
        {
            if (transaction != null) transaction.Dispose();
            if (connection != null) connection.Dispose();
        }
 
        private TAction CreateAction()
            where TAction : DatabaseAction
        {
            return (TAction)Activator.CreateInstance(typeof(TAction), GetConnection(), GetTransaction(),
                metaDataStore, hydrater, sessionLevelCache);
        }
 
        public TEntity Get(object id)
        {
            return CreateAction().Get(id);
        }
 
        public IEnumerable FindAll()
        {
            return CreateAction().FindAll();
        }
 
        public TEntity Insert(TEntity entity)
        {
            return CreateAction().Insert(entity);
        }
 
        public TEntity Update(TEntity entity)
        {
            return CreateAction().Update(entity);
        }
 
        public void Delete(TEntity entity)
        {
            CreateAction().Delete(entity);
        }
 
        public void InitializeProxy(object proxy, Type targetType)
        {
            CreateAction().InitializeProxy(proxy, targetType);
        }
 
        public void ClearCache()
        {
            sessionLevelCache.ClearAll();
        }
 
        public void RemoveFromCache(object entity)
        {
            sessionLevelCache.Remove(entity);
        }
 
        public void RemoveAllInstancesFromCache()
        {
            sessionLevelCache.RemoveAllInstancesOf(typeof(TEntity));
        }
    }

As you can see, there's nothing special here and it's all very straightforward. Application code can now perform database operations pretty easily once it has a reference to an ISession instance. And obtaining a reference to an ISession instance can be done through the ISessionFactory interface:

    public interface ISessionFactory
    {
        ISession CreateSession();
    }

And its implementation:

    public class SessionFactory : ISessionFactory
    {
        private string connectionString;
        private MetaDataStore metaDataStore;
 
        public static ISessionFactory Create(Assembly assembly, string connectionString)
        {
            var sessionFactory = new SessionFactory {connectionString = connectionString, metaDataStore = new MetaDataStore()};
            sessionFactory.metaDataStore.BuildMetaDataFor(assembly);
            return sessionFactory;
        }
 
        private SessionFactory() {}
 
        public ISession CreateSession()
        {
            return new Session(connectionString, metaDataStore);
        }
    }

The static Create method takes an assembly and a connection string. The assembly (containing your entity types) will be used to to build up the metadata model as covered in the Mapping Classes To Tables post. You would typically call the Create method in your application's startup code, and then you'd have to store a reference to the ISessionFactory somewhere. Your application code can then simply call the ISessionFactory's CreateSession method and that's all there is to it.

Build Your Own Data Access Layer: Executing Custom Queries

Davy Brion — Fri, 28 Aug 2009 14:34:47 +0000

Note: This post is part of a series. Be sure to read the introduction here.

You've already seen that this DAL offers you the ability to query each entity by primary key, or to retrieve a list of all instances of an entity type. Obviously, this isn't even sufficient for the most trivial applications so we need a way to execute custom queries. So i needed to provide something that would enable you to easily execute custom queries and get a list of entity instances, or a single result without having to muck around with transforming the results on your own.

Due to the extremely simplistic nature of this DAL, i can't support much more than queries which either return a single value (being one entity instance, or a scalar value) or a list of entities. As soon as you need to execute queries that return joined results, this DAL won't be able to deal with the results automatically. I'll get back to this part later on in the post so for now, let's focus on queries whose result can be transformed to an entity automatically.

My first version of this DAL hardly had any support for this, and when using the DAL for custom queries, you basically had to deal with SqlCommands and their results manually. Well, you could pass the SqlCommand to the EntityHydrater, but you still had to deal with a lot of ugly code. I had asked Ayende to do a private review before i started with this series, and he quickly pointed out that i needed something better for executing SQL queries and suggested something like (surprise, surprise) what NHibernate offers for SQL queries. So yes, this solution is once again heavily inspired by NHibernate

Let's go over the details... First of all, we have this simple IQuery interface:

    public interface IQuery
    {
        void AddParameter(string name, object value, DbType dbType);
        TResult GetSingleResult();
        IEnumerable GetResults();
        int ExecuteNonQuery();
    }

That is what you should be able to do with a query once you've created it. Creating a query is possible through my session API (which will be covered in the next post):

        IQuery CreateQuery(string sql);
        IQuery CreateQuery(string whereClause);

Through the regular CreateQuery method, you can provide the full SQL string and you have full control over the actual SQL. The CreateQuery overload only requires you to provide the WHERE clause because it will generate a SELECT clause for the given entity which automatically retrieves all of the columns for this entity. This also ensures that the result of CreateQuery can always be cleanly transformed to a list of entities or a single entity instance through the EntityHydrater.

This is the implementation of both CreateQuery methods:

        public IQuery CreateQuery(string sql)
        {
            var command = GetConnection().CreateCommand();
            command.Transaction = GetTransaction();
            command.CommandText = sql;
            return new Query(command, metaDataStore, hydrater);
        }
 
        public IQuery CreateQuery(string whereClause)
        {
            return CreateQuery(metaDataStore.GetTableInfoFor().GetSelectStatementForAllFields() + " " + whereClause);
        }

And here's the actual implementation of the Query class:

    public class Query : IQuery
    {
        private readonly SqlCommand command;
        private readonly MetaDataStore metaDataStore;
        private readonly EntityHydrater hydrater;
 
        public Query(SqlCommand command, MetaDataStore metaDataStore, EntityHydrater hydrater)
        {
            this.command = command;
            this.metaDataStore = metaDataStore;
            this.hydrater = hydrater;
        }
 
        public void AddParameter(string name, object value, DbType dbType)
        {
            command.CreateAndAddInputParameter(dbType, name, value);
        }
 
        public TResult GetSingleResult()
        {
            var tableInfo = metaDataStore.GetTableInfoFor();
 
            if (tableInfo == null)
            {
                var scalar = (TResult)command.ExecuteScalar();
                command.Dispose();
                return scalar;
            }
 
            var result = hydrater.HydrateEntity(command);
            command.Dispose();
            return result;
        }
 
        public IEnumerable GetResults()
        {
            var tableInfo = metaDataStore.GetTableInfoFor();
 
            if (tableInfo == null)
            {
                var listOfValues = GetListOfValues();
                command.Dispose();
                return listOfValues;
            }
 
            var result = hydrater.HydrateEntities(command);
            command.Dispose();
            return result;
        }
 
        private IEnumerable GetListOfValues()
        {
            using (var reader = command.ExecuteReader())
            {
                var list = new List();
                while (reader.Read())
                {
                    list.Add(reader.GetValue(0));
                }
                return list.Cast();
            }
        }
 
        public int ExecuteNonQuery()
        {
            var rowsAffected = command.ExecuteNonQuery();
            command.Dispose();
            return rowsAffected;
        }
    }

Pretty simple, right? The ability to get a strong typed result is something that i find very important when using any DAL, and the Query class makes this very easy to do. If you want to return a list of single value results, you can do that easily. If you want to return a single scalar result, you can do that easily. A single entity instance? No problem, the EntityHydrater takes care of that for us. Same thing goes for a list of entities. Custom delete or update statements? No problem, the ExecuteUpdate method can be used for that and will return the typical number of affected rows as reported by the database.

And now you can do things like this pretty easily:

            var query = session.CreateQuery("where this.NumberOfPurchases > @numberOfPurchases");
            query.AddParameter("@numberOfPurchases", 100, DbType.Int32);
            var goodCustomers = query.GetResults();

            var query = session.CreateQuery("select max(NumberOfPurchases) from Customer");
            int max = query.GetSingleResult();

            var query = session.CreateQuery("where this.NumberOfPurchases = (select max(NumberOfPurchases) from Customer))");
            var bestCustomer = query.GetSingleResult();

I'm sure you get the idea by now

Note: when using the CreateQuery method, the SELECT clause that is generated automatically prefixes each selected column from the entity with the 'this' prefix. This makes it easier to refer to the entity's properties in other clauses while you still have to ability to join on another table in the from clause, though you obviously can't add any columns to the select clause anymore.

Now, this is all pretty good for queries where you only need to return scalar values or specific entities, but what if you want to return values from multiple tables in one query? This DAL doesn't have any support for that, but what you could do instead, is to create a view for your query and map an 'entity' to that view instead. Then you could still get a typed result while querying the view, though the results wouldn't be able to be transformed into your 'real' entities. But for filling grids or just to write typical overview queries, this might already be sufficient.

As you can tell, there's not a lot of power or flexibility behind this approach. But try to think of the complexity involved with trying to deal with results from multiple tables. It makes everything a whole lot more complex. At that time, you really need to consider if it's still worth writing your own DAL because the effort you'll spend on getting it right will be very significant.

Build Your Own Data Access Layer: Lazy Loading

Davy Brion — Thu, 27 Aug 2009 14:51:35 +0000

Note: This post is part of a series. Be sure to read the introduction here.

In the post about entity hydration, i mentioned the following:

If we can’t find the referenced entity instance in the first level cache, what should we do? We obviously can’t load it automatically because that could in turn cause referenced entities’ references to be loaded automatically when they are hydrated. Which in turn could cause their referenced entities… Well, i’m sure you get the point. But those properties obviously can’t be set to a null reference either because the column actually does have a valid foreign key value in the database. Explicitly loading referenced properties leads to seriously ugly (and error-prone) code so that’s not an option i’m willing to consider either. The correct way to deal with this is to use lazy loading. To do that in an automated fashion, we need proxy classes. I’m not going to get into these proxy classes and the whole lazy loading thing just yet, since that will be covered in depth in a future post

It's time to go over the implementation of the lazy loading of this DAL. I honestly expected that this would be the part that would take the most time to get working. It actually turned out to be the easiest and quickest part of the DAL to develop.

As i mentioned, we are going to use proxy classes to achieve our goal of lazy loading.

Consider the following simple entity type:

    [Table("Customer")]
    public class Customer
    {
        [PrimaryKey("ID")]
        public int Id { get; set; }
        [Column("Name")]
        public string Name { get; set; }
        [Reference("AddressID")]
        public Address Address { get; set; }
    }

If we want to avoid having to put any lazy loading logic within this class, we could inherit from it and add the lazy loading logic in the derived class. First, we would have to make the properties of our entity virtual:

    [Table("Customer")]
    public class Customer
    {
        [PrimaryKey("ID")]
        public virtual int Id { get; set; }
        [Column("Name")]
        public virtual string Name { get; set; }
        [Reference("AddressID")]
        public virtual Address Address { get; set; }
    }

Now we could create a proxy class like this:

    public class CustomerProxy : Customer
    {
        private bool needsToBeInitialized = true;
        public Session Session { get; set; }
 
        public override string Name
        {
            get
            {
                InitializeIfNecessary();
                return base.Name;
            }
            set
            {
                InitializeIfNecessary();
                base.Name = value;
            }
        }
 
        public override Address Address
        {
            get
            {
                InitializeIfNecessary();
                return base.Address;
            }
            set
            {
                InitializeIfNecessary();
                base.Address = value;
            }
        }
 
        private void InitializeIfNecessary()
        {
            if (needsToBeInitialized)
            {
                needsToBeInitialized = false;
                Session.InitializeProxy(this, typeof(Customer));
            }
        }
    }

This could definitely work. Whenever we need a proxy to avoid loading an entity, we can simply instantiate a proxy class like the one above, pass it a Session object (again, the Session implementation will be covered in a future post, though i will show the InitializeProxy later on in this post) and once we retrieve any of the overridden properties, the proxy instance will use the Session to initialize itself. The Session would then use a DatabaseAction and the hydration process to make sure the proxy's properties are filled in with the values of its corresponding properties. I didn't override the Id property because accessing the primary key of a proxied object should never result in a database call, so there is no reason to override it.

But we really can't expect people to manually create proxy types like this. For one, it's repetitive and thus, it is error prone. Both are issues that we've aimed to avoid with this DAL from the start. So how can we make this work automagically? The answer is simple: we can use Castle's excellent DynamicProxy library to generate the proxy classes at runtime for us.

DynamicProxy uses a concept known as an Interceptor. The Interceptor basically intercepts method calls on proxied objects and allows you to add custom logic before and after the original method calls. For our lazy loading purposes, we simply need the following LazyLoadingInterceptor class:

    public class LazyLoadingInterceptor : IInterceptor
    {
        private TableInfo tableInfo;
        private readonly Session session;
        private bool needsToBeInitialized = true;
 
        public LazyLoadingInterceptor(TableInfo tableInfo, Session session)
        {
            this.tableInfo = tableInfo;
            this.session = session;
        }
 
        public void Intercept(IInvocation invocation)
        {
            if (invocation.Method.Name.Equals("get_" + tableInfo.PrimaryKey.PropertyInfo.Name) ||
                invocation.Method.Name.Equals("set_" + tableInfo.PrimaryKey.PropertyInfo.Name))
            {
                invocation.Proceed();
                return;
            }
 
            if (needsToBeInitialized)
            {
                needsToBeInitialized = false;
                session.InitializeProxy(invocation.Proxy, invocation.TargetType);
            }
 
            invocation.Proceed();
        }

Ok, so what does this class do? Like i said, an interceptor will intercept all method calls on virtual methods of a proxied object. So basically, if we create a proxy through DynamicProxy, and tell DynamicProxy to add a new instance of this LazyLoadingInterceptor so we can add the behavior of this interceptor to our proxy object, and we then access the properties of our proxy object, it will actually show the same behavior as the manually created CustomerProxy class listed earlier. And we get all of this for free, without having to modify our original Customer class, except for marking the properties as virtual obviously.

Now, when the LazyLoadingInterceptor calls the Session's InitializeProxy method, the session will delegate this call to our new InitializeProxyAction class:

    public class InitializeProxyAction : DatabaseAction
    {
        public InitializeProxyAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
            EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
            : base(connection, transaction, metaDataStore, hydrater, sessionLevelCache)
        {
        }
 
        public void InitializeProxy(object proxy, Type targetType)
        {
            using (var command = CreateCommand())
            {
                var tableInfo = MetaDataStore.GetTableInfoFor(targetType);
                var query = tableInfo.GetSelectStatementForAllFields();
                tableInfo.AddWhereByIdClause(query);
 
                object id = tableInfo.PrimaryKey.PropertyInfo.GetValue(proxy, null);
                command.CommandText = query.ToString();
                command.CreateAndAddInputParameter(tableInfo.PrimaryKey.DbType, tableInfo.GetPrimaryKeyParameterName(), id);
 
                Hydrater.UpdateEntity(targetType, proxy, command);
            }
        }
    }

As you can see, this is extremely similar to the GetByIdAction. In fact, i should probably put the common logic in another base DatabaseAction class that sits between DatabaseAction and GetByIdAction and InitializeProxyAction. Anyways, when the entity has been retrieved, we ask the EntityHydrater to update this entity instance through its newly added UpdateEntity method:

        public void UpdateEntity(Type type, object entity, SqlCommand command)
        {
            using (var reader = command.ExecuteReader())
            {
                reader.Read();
                var tableInfo = metaDataStore.GetTableInfoFor(type);
                Hydrate(tableInfo, entity, GetValuesFromCurrentRow(reader));
            }
        }

The Hydrate method is already shown in the post that covers the process of entity hydration, so there's no need to go over that again.

And now there's only one more missing piece in our lazy loading puzzle, which is the actual creation of the proxy in the EntityHydrater's CreateProxy method:

        private object CreateProxy(TableInfo tableInfo, ReferenceInfo referenceInfo, object foreignKeyValue)
        {
            var proxy = proxyGenerator.CreateClassProxy(referenceInfo.ReferenceType,
                new[] { new LazyLoadingInterceptor(tableInfo, session) });
            var referencePrimaryKey = metaDataStore.GetTableInfoFor(referenceInfo.ReferenceType).PrimaryKey;
            referencePrimaryKey.PropertyInfo.SetValue(proxy, foreignKeyValue, null);
            return proxy;
        }

When using an approach like this, it's best to make everything in your entity classes virtual... NHibernate has a similar restriction and i've tried to explain the reasons behind this in this post.

I'm not sure if i succeeded in explaining this topic in a simple and clear manner, but this technique really is pretty easy. If you have any questions, i'd be glad to anser them in the comments

Build Your Own Data Access Layer: Session Level Cache

Davy Brion — Wed, 26 Aug 2009 15:11:28 +0000

Note: This post is part of a series. Be sure to read the introduction here.

In the previous 2 posts of this series, you may have noticed the usage of the SessionLevelCache class. This class is a simple implementation of the Identity Map pattern. It basically offers us two important benefits:

It ensures that we never load the same entity from the database twice.
It ensures that we can't accidentally have two instances which point to the same database record.

Well actually... there is one situation where you could possibly load the same entity twice (which i'll show momentarily) but even then we can make sure that our users will always have the same instance of the entity.

The SessionLevelCache class is actually very simple:

    public class SessionLevelCache
    {
        private readonly Dictionary> cache = new Dictionary>();
 
        public object TryToFind(Type type, object id)
        {
            if (!cache.ContainsKey(type)) return null;
 
            string idAsString = id.ToString();
            if (!cache[type].ContainsKey(idAsString)) return null;
 
            return cache[type][idAsString];
        }
 
        public void Store(Type type, object id, object entity)
        {
            if (!cache.ContainsKey(type)) cache.Add(type, new Dictionary());
 
            cache[type][id.ToString()] = entity;
        }
 
        public void ClearAll()
        {
            cache.Clear();
        }
 
        public void RemoveAllInstancesOf(Type type)
        {
            if (cache.ContainsKey(type))
            {
                cache.Remove(type);
            }
        }
 
        public void Remove(object entity)
        {
            var type = entity.GetType();
 
            if (!cache.ContainsKey(type)) return;
 
            string keyToRemove = null;
 
            foreach (var pair in cache[type])
            {
                if (pair.Value == entity)
                {
                    keyToRemove = pair.Key;
                }
            }
 
            if (keyToRemove != null)
            {
                cache[type].Remove(keyToRemove);
            }
        }
    }

It uses a nested dictionary where the outer dictionary uses the entity type as its key value, and stores an inner dictionary for each entity type which uses the primary key value as the key for each entity instance. I think the code is simple enough so there's not really any point to go over the implementation details. Instead, let's focus on when and where the SessionLevelCache is used.

First, let's look back at the Get method of the GetByIdAction class:

        public TEntity Get(object id)
        {
            var cachedEntity = SessionLevelCache.TryToFind(typeof(TEntity), id);
            if (cachedEntity != null) return (TEntity)cachedEntity;
 
            using (var command = CreateCommand())
            {
                var tableInfo = MetaDataStore.GetTableInfoFor();
 
                var query = tableInfo.GetSelectStatementForAllFields();
                tableInfo.AddWhereByIdClause(query);
 
                command.CommandText = query.ToString();
                command.CreateAndAddInputParameter(tableInfo.PrimaryKey.DbType, tableInfo.GetPrimaryKeyParameterName(), id);
                return Hydrater.HydrateEntity(command);
            }
        }

As you can see, when we enter this method we first check to see whether the entity instance is already present in the cache. If it is, we obviously don't need to hit the database so we can simply return the cached entity.

But like i said, there is one situation where we could potentially retrieve the entity from the database even though we already have it in the cache, and that is when we are hydrating a list of entities that have been retrieved from the database. You might remember the following method from the EntityHydrater class:

        private TEntity CreateEntityFromValues(IDictionary values)
        {
            var tableInfo = metaDataStore.GetTableInfoFor();
 
            var cachedEntity = sessionLevelCache.TryToFind(typeof(TEntity), values[tableInfo.PrimaryKey.Name]);
            if (cachedEntity != null) return (TEntity)cachedEntity;
 
            var entity = Activator.CreateInstance();
            Hydrate(tableInfo, entity, values);
            sessionLevelCache.Store(typeof(TEntity), values[tableInfo.PrimaryKey.Name], entity);
            return entity;
        }

This method is used to hydrate every single entity that we create. Obviously, when hydrating a single entity we've actually already performed the cache-look-up in the GetByIdAction. However, when a custom query is executed, or when all instances are retrieved, there is no way for us to exclude already cached entity instances from the result of the query. Well, theoretically speaking you could attempt to do this by adding a clause to the WHERE statement of each query that would prevent cached entities from being loaded. But then you might have to add the cached entity instances to the resulting list of entities anyways if they would otherwise satisfy the other query conditions. Obviously, trying to get this right is simply put insane and i don't think there's any DAL or ORM that actually does this (even if there was, i can't really imagine any of them getting this right in every corner case that will pop up).

So a good compromise is to simply check for the existence of a specific instance in the cache before hydrating a new instance. If it is there, we return it from the cache and we skip the hydration for that database record. In this way, we avoid having to modify the original query, and while we could potentially return a few records that we already have in memory, at least we will be sure that our users will always have the same reference for any particular database record.

There is one more scenario that needs to be covered. If an entity holds a reference through a foreign key to another entity instance, and that referenced entity is already present in the cache, we need to make sure that the entity we are hydrating will refer to the already cached referred-to-instance instead of creating a proxy by default. After all, if we were to create a proxy object for an entity instance that is already in our cache, we will have failed to achieve our goal of avoiding the possibility of more than one instance representing the same database record. Therefore, when hydrating the reference properties of an entity, we have the following piece of code:

                        var referencedEntity = sessionLevelCache.TryToFind(referenceInfo.ReferenceType, foreignKeyValue) ??
                                               CreateProxy(tableInfo, referenceInfo, foreignKeyValue);
 
                        referenceInfo.PropertyInfo.SetValue(entity, referencedEntity, null);

This way, we are sure that a proxy object for a referenced entity will only be created if that entity is not already present in the SessionLevelCache.

Now we're pretty much covered when it comes to retrieving entity instances from the database. But we obviously also need to update the SessionLevelCache whenever an entity is inserted, and whenever an entity is deleted. In the InsertAction, you can find the following code at the end of the Insert method:

                SessionLevelCache.Store(typeof(TEntity), id, entity);

And in the DeleteAction we can also spot the following line of code:

                SessionLevelCache.Remove(entity);

There is still one problem however. When a user executes a custom DELETE statement, there is no way for us to know which entities were actually removed from the database. But if any of those deleted entities happen to remain in the SessionLevelCache, this could lead to buggy application code whenever a piece of code tries to retrieve a specific entity which has already been removed from the database, but is still present in the SessionLevelCache. In order to deal with this scenario, the SessionLevelCache has a ClearAll and a RemoveAllInstancesOf method which you can use from your application code to either clear the entire SessionLevelCache, or to remove all instances of a specific entity type from the cache. Calling these methods would obviously be the responsability of the application code, since we can't possibly take care of this automatically in such a simplistic DAL. Actually, even for powerful ORM's this can be pretty difficult to get right.

Another thing i'd like to point out is that the SessionLevelCache is not threadsafe. A session (which will be covered in a later post) is not threadsafe, so within each of the classes that are used by the session i take no care of thread-safety whatsoever.

The SessionLevelCache might not seem like much to you, but i do consider its to be an absolute must for any DAL. Notice that this doesn't even come close to a proper second-level cache like NHibernate offers, but the complexity of implementing such a thing is way beyond the scope of both this series and probably most custom DAL's out there.

Build Your Own Data Access Layer: Hydrating Entities

Davy Brion — Tue, 25 Aug 2009 15:42:31 +0000

Note: This post is part of a series. Be sure to read the introduction here.

In the previous post of this series, you saw that some of the DatabaseActions use the EntityHydrater to umm.. hydrate the entities with their values from the database. In this post, we'll go over how this actually works. First, i'm going to post the code of the entire class and then we'll go over the interesting parts.

So, here's the entire code of the EntityHydrater class:

    public class EntityHydrater
    {
        private readonly MetaDataStore metaDataStore;
        private readonly SessionLevelCache sessionLevelCache;
 
        public EntityHydrater(MetaDataStore metaDataStore, SessionLevelCache sessionLevelCache)
        {
            this.metaDataStore = metaDataStore;
            this.sessionLevelCache = sessionLevelCache;
        }
 
        public TEntity HydrateEntity(SqlCommand command)
        {
            IDictionary values;
 
            using (var reader = command.ExecuteReader())
            {
                if (!reader.HasRows) return default(TEntity);
                reader.Read();
                values = GetValuesFromCurrentRow(reader);
            }
 
            return CreateEntityFromValues(values);
        }
 
        public IEnumerable HydrateEntities(SqlCommand command)
        {
            var rows = new List>();
            var entities = new List();
 
            using (var reader = command.ExecuteReader())
            {
                while (reader.Read())
                {
                    rows.Add(GetValuesFromCurrentRow(reader));
                }
            }
 
            foreach (var row in rows)
            {
                entities.Add(CreateEntityFromValues(row));
            }
 
            return entities;
        }
 
        private IDictionary GetValuesFromCurrentRow(SqlDataReader dataReader)
        {
            var values = new Dictionary();
 
            for (int i = 0; i < dataReader.FieldCount; i++)
            {
                values.Add(dataReader.GetName(i), dataReader.GetValue(i));
            }
 
            return values;
        }
 
        private TEntity CreateEntityFromValues(IDictionary values)
        {
            var tableInfo = metaDataStore.GetTableInfoFor();
 
            var cachedEntity = sessionLevelCache.TryToFind(typeof(TEntity), values[tableInfo.PrimaryKey.Name]);
            if (cachedEntity != null) return (TEntity)cachedEntity;
 
            var entity = Activator.CreateInstance();
            Hydrate(tableInfo, entity, values);
            sessionLevelCache.Store(typeof(TEntity), values[tableInfo.PrimaryKey.Name], entity);
            return entity;
        }
 
        private void Hydrate(TableInfo tableInfo, TEntity entity, IDictionary values)
        {
            tableInfo.PrimaryKey.PropertyInfo.SetValue(entity, values[tableInfo.PrimaryKey.Name], null);
            SetRegularColumns(tableInfo, entity, values);
            SetReferenceProperties(tableInfo, entity, values);
        }
 
        private void SetRegularColumns(TableInfo tableInfo, TEntity entity, IDictionary values)
        {
            foreach (var columnInfo in tableInfo.Columns)
            {
                if (columnInfo.PropertyInfo.CanWrite)
                {
                    object value = values[columnInfo.Name];
                    if (value is DBNull) value = null;
                    columnInfo.PropertyInfo.SetValue(entity, value, null);
                }
            }
        }
 
        private void SetReferenceProperties(TableInfo tableInfo, TEntity entity, IDictionary values)
        {
            foreach (var referenceInfo in tableInfo.References)
            {
                if (referenceInfo.PropertyInfo.CanWrite)
                {
                    object foreignKeyValue = values[referenceInfo.Name];
 
                    if (foreignKeyValue is DBNull)
                    {
                        referenceInfo.PropertyInfo.SetValue(entity, null, null);
                    }
                    else
                    {
                        var referencedEntity = sessionLevelCache.TryToFind(referenceInfo.ReferenceType, foreignKeyValue) ??
                                               CreateProxy(tableInfo, referenceInfo, foreignKeyValue);
 
                        referenceInfo.PropertyInfo.SetValue(entity, referencedEntity, null);
                    }
                }
            }
        }
 
        private object CreateProxy(TableInfo tableInfo, ReferenceInfo referenceInfo, object foreignKeyValue)
        {
            // NOTE: this will be covered in a later post, so i took out the spoiler code 
            return null;
        }
    }

As you can see, there are 2 public methods: HydrateEntity and HydrateEntities. They are both pretty similar, except that the former only hydrates a single entity and the latter a list of entities. They both retrieve the values from the current position in a DataReader and store them in a Dictionary with the name of the column being the key and the value of the column being the value in the dictionary:

        private IDictionary GetValuesFromCurrentRow(SqlDataReader dataReader)
        {
            var values = new Dictionary();
 
            for (int i = 0; i < dataReader.FieldCount; i++)
            {
                values.Add(dataReader.GetName(i), dataReader.GetValue(i));
            }
 
            return values;
        }

This dictionary is then passed to the CreateEntityFromValues method:

        private TEntity CreateEntityFromValues(IDictionary values)
        {
            var tableInfo = metaDataStore.GetTableInfoFor();
 
            var cachedEntity = sessionLevelCache.TryToFind(typeof(TEntity), values[tableInfo.PrimaryKey.Name]);
            if (cachedEntity != null) return (TEntity)cachedEntity;
 
            var entity = Activator.CreateInstance();
            Hydrate(tableInfo, entity, values);
            sessionLevelCache.Store(typeof(TEntity), values[tableInfo.PrimaryKey.Name], entity);
            return entity;
        }

This method will first check to see if an entity instance for the row in the values dictionary already exists. If it exists already, it simply returns the entity instance and ignores the values in the dictionary. I'm not sure yet whether this behavior is correct or not. Theoretically speaking, it's possible when using ReadCommitted isolation level that the values in the dictionary will be more recent than the entity instance in the session level cache. However, since this simple DAL has no change tracking, i also can't deduce whether the instance in the cache has already had one or more of its properties updated by application code. Simply overwriting them with newly retrieved values of the database doesn't seem like the right thing to do here. So the current options are to either ignore this possibility, or to rely on optimistic concurrency. Oh, and i don't have support for optimistic concurrency either. A good DAL should however provide some optimistic concurrency strategies here, or be able to track the changes in the entity and throw an exception if it notices that the database contains more recent values AND the local entity has already been modified. If it hasn't been modified yet, it could overwrite the values of the instances though i'm not sure everyone would agree with this behavior. Either way, this particular problem is definitely interesting enough to think about for a while

Anyways, in this implementation, i'm completely ignoring this situation and i either return the already loaded entity of the session level cache, or i hydrate a new entity instance and return that instead. In the latter case, i also store the entity instance in the session level cache. Note that the actual implementation of the session level cache will be covered in the next post of this series.

The actual Hydrate method looks like this:

        private void Hydrate(TableInfo tableInfo, TEntity entity, IDictionary values)
        {
            tableInfo.PrimaryKey.PropertyInfo.SetValue(entity, values[tableInfo.PrimaryKey.Name], null);
            SetRegularColumns(tableInfo, entity, values);
            SetReferenceProperties(tableInfo, entity, values);
        }

First the primary key value of the record is set in the primary key property of the new instance, and then we proceed with putting the regular column values in their properties, and after that, the reference properties. Filling the regular column properties is very straightforward:

        private void SetRegularColumns(TableInfo tableInfo, TEntity entity, IDictionary values)
        {
            foreach (var columnInfo in tableInfo.Columns)
            {
                if (columnInfo.PropertyInfo.CanWrite)
                {
                    object value = values[columnInfo.Name];
                    if (value is DBNull) value = null;
                    columnInfo.PropertyInfo.SetValue(entity, value, null);
                }
            }
        }

Dealing with the references is a bit more interesting though.

        private void SetReferenceProperties(TableInfo tableInfo, TEntity entity, IDictionary values)
        {
            foreach (var referenceInfo in tableInfo.References)
            {
                if (referenceInfo.PropertyInfo.CanWrite)
                {
                    object foreignKeyValue = values[referenceInfo.Name];
 
                    if (foreignKeyValue is DBNull)
                    {
                        referenceInfo.PropertyInfo.SetValue(entity, null, null);
                    }
                    else
                    {
                        var referencedEntity = sessionLevelCache.TryToFind(referenceInfo.ReferenceType, foreignKeyValue) ??
                                               CreateProxy(tableInfo, referenceInfo, foreignKeyValue);
 
                        referenceInfo.PropertyInfo.SetValue(entity, referencedEntity, null);
                    }
                }
            }
        }

If we can't find the referenced entity instance in the first level cache, what should we do? We obviously can't load it automatically because that could in turn cause referenced entities' references to be loaded automatically when they are hydrated. Which in turn could cause their referenced entities... Well, i'm sure you get the point. But those properties obviously can't be set to a null reference either because the column actually does have a valid foreign key value in the database. Explicitly loading referenced properties leads to seriously ugly (and error-prone) code so that's not an option i'm willing to consider either. The correct way to deal with this is to use lazy loading. To do that in an automated fashion, we need proxy classes. I'm not going to get into these proxy classes and the whole lazy loading thing just yet, since that will be covered in depth in a future post

So that's pretty much it (for now) for our EntityHydrater class. As you can see, it's still relatively simple but then again, the use cases that it supports are extremely simple as well. This current implementation is incapable of hydrating entities based on a SQL statement that selects data from more than just the entity's table. And that is a pretty big shortcoming. For instance, with NHibernate you can execute queries where you can instruct NHibernate to fetch some (or all) of the entity's references (associations in NHibernate) with just one SQL statement, using the join syntax. NHibernate can then hydrate the root entity, and populate its reference properties with the other values that were returned by the sql statement. While it wouldn't be that complex to add this capability to this EntityHydrater class, it wouldn't exactly be completely trivial either. Again, this is a limitation that many (maybe even most?) custom DAL's have. This one probably makes it easy enough to still add this feature though

Build Your Own Data Access Layer: Out Of The Box CRUD Functionality

Davy Brion — Mon, 24 Aug 2009 17:29:19 +0000

Note: This post is part of a series. Be sure to read the introduction here.

One thing that i consider an absolute must-have in any data access layer is the ability to perform CRUD operations out-of-the-box without having to write any code to enable these operations. Once your data access layer knows about your classes and your tables, CRUD operations should 'just work'.

As you've seen in the previous post of this series, the TableInfo class offers a couple of methods to automatically build the required SQL statements for CRUD actions. With these statements, we can easily create SqlCommand instances for all CRUD operations.

First of all, i use the following helper method to easily add a SqlParameter to a SqlCommand:

        public static void CreateAndAddInputParameter(this SqlCommand command, DbType type, string name, object value)
        {
            var parameter = command.CreateParameter();
            parameter.Direction = ParameterDirection.Input;
            parameter.DbType = type;
            parameter.ParameterName = name;
 
            if (value == null)
            {
                parameter.IsNullable = true;
                parameter.Value = DBNull.Value;
            }
            else
            {
                parameter.Value = value;
            }
 
            command.Parameters.Add(parameter);
        }

I also have the following abstract DatabaseAction class which has a few properties that are used by most of the CRUD actions:

    public abstract class DatabaseAction
    {
        private readonly SqlConnection connection;
        private readonly SqlTransaction transaction;
        protected MetaDataStore MetaDataStore { get; private set; }
        protected EntityHydrater Hydrater { get; private set; }
        protected SessionLevelCache SessionLevelCache { get; private set; }
 
        protected DatabaseAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
                                 EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
        {
            this.connection = connection;
            this.transaction = transaction;
            MetaDataStore = metaDataStore;
            Hydrater = hydrater;
            SessionLevelCache = sessionLevelCache;
        }
 
        protected SqlCommand CreateCommand()
        {
            var command = connection.CreateCommand();
            command.Transaction = transaction;
            return command;
        }
    }

Did you notice the EntityHydrater and SessionLevelCache? I'm going to ignore those as much as possible for now, since they will be covered in depth in the following two posts in these series. The important thing to note is that each derived DatabaseAction will have a reference to the MetaDataStore.

And now we can easily start implementing our CRUD actions. Let's start with the GetByIdAction:

    public class GetByIdAction : DatabaseAction
    {
        public GetByIdAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
            EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
            : base(connection, transaction, metaDataStore, hydrater, sessionLevelCache)
        {
        }
 
        public TEntity Get(object id)
        {
            var cachedEntity = SessionLevelCache.TryToFind(typeof(TEntity), id);
            if (cachedEntity != null) return (TEntity)cachedEntity;
 
            using (var command = CreateCommand())
            {
                var tableInfo = MetaDataStore.GetTableInfoFor();
 
                var query = tableInfo.GetSelectStatementForAllFields();
                tableInfo.AddWhereByIdClause(query);
 
                command.CommandText = query.ToString();
                command.CreateAndAddInputParameter(tableInfo.PrimaryKey.DbType, tableInfo.GetPrimaryKeyParameterName(), id);
                return Hydrater.HydrateEntity(command);
            }
        }
    }

Pretty simple stuff, right? This will first check the session level cache to see if this instance has already been retrieved in the current session (i'll discuss the session in a later post) and if so, it will return that instance. If it's not in the cache, it will create a SqlCommand and fill its CommandText property with a SQL string that is provided by the relevant TableInfo class. After that, it passes the SqlCommand to the EntityHydrater so it can return an actual entity instance.

The details of EntityHydration will be fully explored in the next post of this series, so for now you only need to know that it can transform the results from the SqlCommand to an instance of TEntity.

It's always useful to get a collection of all instances of a certain entity class, so we also have this very simple FindAllAction:

    public class FindAllAction : DatabaseAction
    {
        public FindAllAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
            EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
            : base(connection, transaction, metaDataStore, hydrater, sessionLevelCache)
        {
        }
 
        public IEnumerable FindAll()
        {
            using (var command = CreateCommand())
            {
                command.CommandText = MetaDataStore.GetTableInfoFor().GetSelectStatementForAllFields().ToString();
                return Hydrater.HydrateEntities(command);
            }
        }
    }

We also need an InsertAction:

    public class InsertAction : DatabaseAction
    {
        public InsertAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
            EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
            : base(connection, transaction, metaDataStore, hydrater, sessionLevelCache)
        {
        }
 
        public TEntity Insert(TEntity entity)
        {
            using (var command = CreateCommand())
            {
                var tableInfo = MetaDataStore.GetTableInfoFor();
 
                command.CommandText = tableInfo.GetInsertStatement();
 
                foreach (var parameterInfo in tableInfo.GetParametersForInsert(entity))
                {
                    command.CreateAndAddInputParameter(parameterInfo.DbType, parameterInfo.Name, parameterInfo.Value);
                }
 
                object id = Convert.ChangeType(command.ExecuteScalar(), tableInfo.PrimaryKey.DotNetType);
                tableInfo.PrimaryKey.PropertyInfo.SetValue(entity, id, null);
                SessionLevelCache.Store(typeof(TEntity), id, entity);
                return entity;
            }
        }
    }

There's not a lot to this one either... The actual insert statement is once again retrieved through the TableInfo class, as are the parameter values (including their values for this specific entity). You can go back to the previous post to look at the implementation of TableInfo's GetParametersForInsert method

Keep in mind that there is a limitation here that i only support SQL Server's Identity-style generators. Again, if you want to support multiple identifier strategies like NHibernate does, you'll have to deal with a lot more complexity in the InsertAction class.

The UpdateAction is very similar:

    public class UpdateAction : DatabaseAction
    {
        public UpdateAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
            EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
            : base(connection, transaction, metaDataStore, hydrater, sessionLevelCache)
        {
        }
 
        public TEntity Update(TEntity entity)
        {
            using (var command = CreateCommand())
            {
                var tableInfo = MetaDataStore.GetTableInfoFor();
 
                command.CommandText = tableInfo.GetUpdateStatement();
 
                foreach (var parameterInfo in tableInfo.GetParametersForUpdate(entity))
                {
                    command.CreateAndAddInputParameter(parameterInfo.DbType, parameterInfo.Name, parameterInfo.Value);
                }
 
                command.ExecuteNonQuery();
                return entity;
            }
        }
    }

And finally, we have the DeleteAction:

    public class DeleteAction : DatabaseAction
    {
        public DeleteAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
            EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
            : base(connection, transaction, metaDataStore, hydrater, sessionLevelCache)
        {
        }
 
        public void Delete(TEntity entity)
        {
            using (var command = CreateCommand())
            {
                var tableInfo = MetaDataStore.GetTableInfoFor();
                command.CommandText = tableInfo.GetDeleteStatement();
                object id = tableInfo.PrimaryKey.PropertyInfo.GetValue(entity, null);
                command.CreateAndAddInputParameter(tableInfo.PrimaryKey.DbType, tableInfo.GetPrimaryKeyParameterName(), id);
                command.ExecuteNonQuery();
                SessionLevelCache.Remove(entity);
            }
        }
    }

And that's all there is to it. We now have some classes that will give us out-of-the-box CRUD functionality for all of the mapped entity classes. Obviously, you will still need some way of actually accessing this functionality from your application code and you certainly don't want to instantiate and use these DatabaseAction classes directly. All of that will be covered in the "Bringing It All Together" post, so stay tuned

Build Your Own Data Access Layer: Mapping Classes To Tables

Davy Brion — Sun, 23 Aug 2009 15:08:47 +0000

Note: This post is part of a series. Be sure to read the introduction here.

When you need to populate entity instances with data from a database, you need to know which table the data needs to come from, which columns will map to which property on the entity class, and you'll need to deal with a variety of types. The approach that i've chosen to use tries to make this as simple as possible. The idea is basically to place an attribute with the name of the table on top of the entity class, and an attribute on each property with the name of the column it maps to. For foreign keys, i wanted to be able to just use properties of the type of the referenced entity, instead of having foreign keys in my entities. For these references, we will use an attribute with the name of the foreign key column.

First, we'll need to define these attributes:

    [AttributeUsage(AttributeTargets.Class, Inherited = true)]
    public class TableAttribute : Attribute
    {
        public string TableName { get; private set; }
 
        public TableAttribute(string tableName)
        {
            TableName = tableName;
        }
    }

    [AttributeUsage(AttributeTargets.Property, Inherited = true)]
    public class PrimaryKeyAttribute : Attribute
    {
        public string ColumnName { get; private set; }
 
        public PrimaryKeyAttribute(string columnName)
        {
            ColumnName = columnName;
        }
    }

    [AttributeUsage(AttributeTargets.Property, Inherited = true)]
    public class ColumnAttribute : Attribute
    {
        public string ColumnName { get; private set; }
 
        public ColumnAttribute(string columnName)
        {
            ColumnName = columnName;
        }
    }

    [AttributeUsage(AttributeTargets.Property, Inherited = true)]
    public class ReferenceAttribute : Attribute
    {
        public string ColumnName { get; private set; }
 
        public ReferenceAttribute(string columnName)
        {
            ColumnName = columnName;
        }
    }

Notice how none of these properties have any indication of types to use. The .NET type will be inferred automatically, and it will be mapped to a compatible DbType without having to specify these types all over the place.

We will use the following helper class to map .NET types to their respective DbTypes:

    public static class TypeConverter
    {
        private static readonly Dictionary typeToDbType = new Dictionary
        {
            { typeof(string), DbType.String },
            { typeof(DateTime), DbType.DateTime },
            { typeof(DateTime?), DbType.DateTime },
            { typeof(int), DbType.Int32 },
            { typeof(int?), DbType.Int32 },
            { typeof(long), DbType.Int64 },
            { typeof(long?), DbType.Int64 },
            { typeof(bool), DbType.Boolean },
            { typeof(bool?), DbType.Boolean },
            { typeof(byte[]), DbType.Binary },
            { typeof(decimal), DbType.Decimal },
            { typeof(decimal?), DbType.Decimal },
            { typeof(double), DbType.Double },
            { typeof(double?), DbType.Double },
            { typeof(float), DbType.Single },
            { typeof(float?), DbType.Single },
            { typeof(Guid), DbType.Guid },
            { typeof(Guid?), DbType.Guid }
        };
 
        public static DbType ToDbType(Type type)
        {
            if (!typeToDbType.ContainsKey(type))
            {
                throw new InvalidOperationException(string.Format("Type {0} doesn't have a matching DbType configured", type.FullName));
            }
 
            return typeToDbType[type];
        }
    }

Obviously, more type conversions can be added... these are just the ones i've needed so far.

Once you've placed all the attributes on top of your entities and properties, we can start building a model of all this metadata. This will all be stored in a MetaDataStore class that i'll show later on in this post. Having access to the MetaDataStore makes the implementation of some of these metadata types easier, so i have the following abstract class:

    public abstract class MetaData
    {
        protected MetaDataStore MetaDataStore { get; private set; }
 
        protected MetaData(MetaDataStore metaDataStore)
        {
            MetaDataStore = metaDataStore;
        }
    }

Now we can go over each piece of metadata. First, the ColumnInfo class:

    public class ColumnInfo : MetaData
    {
        public string Name { get; private set; }
        public Type DotNetType { get; private set; }
        public DbType DbType { get; private set; }
        public PropertyInfo PropertyInfo { get; private set; }
 
        public ColumnInfo(MetaDataStore store, string name, Type dotNetType, PropertyInfo propertyInfo)
            : this(store, name, dotNetType, TypeConverter.ToDbType(dotNetType), propertyInfo)
        {
        }
 
        public ColumnInfo(MetaDataStore store, string name, Type dotNetType, DbType dbType, PropertyInfo propertyInfo)
            : base(store)
        {
            Name = name;
            DotNetType = dotNetType;
            DbType = dbType;
            PropertyInfo = propertyInfo;
        }
    }

As you can see, we have all the information we need to be able to do something with this column. We have its Name, the .NET type that is used in the mapped class, the DbType and a PropertyInfo reference to its respective property in the mapped class so we can get and set its value.

For references, we need to know something more:

    public class ReferenceInfo : ColumnInfo
    {
        public Type ReferenceType { get; private set; }
 
        public ReferenceInfo(MetaDataStore store, string name, Type referenceType, PropertyInfo propertyInfo)
            : base(store, name, store.GetTableInfoFor(referenceType).PrimaryKey.DotNetType,
                    store.GetTableInfoFor(referenceType).PrimaryKey.DbType, propertyInfo)
        {
            ReferenceType = referenceType;
        }
    }

For a regular column, it's sufficient to know the .NET type of the property and the DbType. But for a reference, you need to know the actual type of the referenced entity, as well as the .NET type of it's primary key column. As you can see in the constructor, we retrieve the TableInfo of the referenced entity, and use the .NET type and the DbType of the primary key of the referenced entity. The PrimaryKey property of a TableInfo class (which i'll show below) is also a ColumnInfo object. We obviously also store the actual type of the referenced entity. And of course, we again store a PropertyInfo so we can get/set the value of the reference.

The TableInfo class can now hold all of the information that we need. We know all about its primary key (through the PrimaryKeyAttribute), its regular properties (through the ColumnAttribute) and its referenced properties (through the ReferenceAttribute). With all of that information, the TableInfo class is able to build your typical default SQL statements for CRUD functionality:

    public class TableInfo : MetaData
    {
        public string Name { get; private set; }
        public Type EntityType { get; private set; }
        public ColumnInfo PrimaryKey { get; set; }
        public IEnumerable References { get { return references.Values; } }
        public IEnumerable Columns { get { return columns.Values; } }
 
        private readonly Dictionary columns = new Dictionary();
        private readonly Dictionary references = new Dictionary();
 
        public TableInfo(MetaDataStore store, string name, Type entityType)
            : base(store)
        {
            Name = name;
            EntityType = entityType;
        }
 
        public void AddColumn(ColumnInfo column)
        {
            if (columns.ContainsKey(column.Name))
            {
                throw new InvalidOperationException(string.Format("An item with key {0} has already been added", column.Name));
            }
 
            columns.Add(column.Name, column);
        }
 
        public void AddReference(ReferenceInfo reference)
        {
            if (references.ContainsKey(reference.Name))
            {
                throw new InvalidOperationException(string.Format("An item with key {0} has already been added", reference.Name));
            }
 
            references.Add(reference.Name, reference);
        }
 
        public ColumnInfo GetColumn(string columnName)
        {
            if (!columns.ContainsKey(columnName))
            {
                throw new InvalidOperationException(string.Format("The table '{0}' does not have a '{1}' column", Name, columnName));
            }
 
            return columns[columnName];
        }
 
        public StringBuilder GetSelectStatementForAllFields()
        {
            StringBuilder builder = new StringBuilder("SELECT " + Escape(PrimaryKey.Name) + ", ");
 
            AddReferenceColumnNames(builder);
            AddRegularColumnNames(builder);
            RemoveLastCommaAndSpaceIfThereAreAnyColumns(builder);
            builder.Append(" FROM " + Escape(Name));
 
            return builder;
        }
 
        public string GetInsertStatement()
        {
            StringBuilder builder = new StringBuilder("INSERT INTO " + Escape(Name) + " (");
 
            AddReferenceColumnNames(builder);
            AddRegularColumnNames(builder);
            RemoveLastCommaAndSpaceIfThereAreAnyColumns(builder);
            builder.Append(") VALUES (");
            AddReferenceColumnParameterNames(builder);
            AddRegularColumnParameterNames(builder);
            RemoveLastCommaAndSpaceIfThereAreAnyColumns(builder);
            builder.Append("); SELECT SCOPE_IDENTITY();");
 
            return builder.ToString();
        }
 
        public string GetUpdateStatement()
        {
            StringBuilder builder = new StringBuilder("UPDATE " + Escape(Name) + " SET ");
 
            AddReferenceColumnsNameWithParameterName(builder);
            AddRegularColumnsNameWithParameterName(builder);
            RemoveLastCommaAndSpaceIfThereAreAnyColumns(builder);
            AddWhereByIdClause(builder);
            builder.Append(";");
 
            return builder.ToString();
        }
 
        public string GetDeleteStatement()
        {
            StringBuilder builder = new StringBuilder("DELETE FROM " + Escape(Name) + " ");
 
            AddWhereByIdClause(builder);
            builder.Append(";");
 
            return builder.ToString();
        }
 
        public IEnumerable GetParametersForInsert(object entity)
        {
            return GetParametersForAllReferenceAndRegularColumns(entity);
        }
 
        public IEnumerable GetParametersForUpdate(object entity)
        {
            var parameters = GetParametersForAllReferenceAndRegularColumns(entity);
            parameters.Add(new AdoParameterInfo(PrimaryKey.Name, PrimaryKey.DbType, PrimaryKey.PropertyInfo.GetValue(entity, null)));
            return parameters;
        }
 
        public StringBuilder AddWhereByIdClause(StringBuilder query)
        {
            query.Append(" WHERE " + Escape(PrimaryKey.Name) + " = " + GetPrimaryKeyParameterName());
            return query;
        }
 
        public string GetPrimaryKeyParameterName()
        {
            return "@" + PrimaryKey.Name;
        }
 
        private List GetParametersForAllReferenceAndRegularColumns(object entity)
        {
            var parameters = new List();
 
            foreach (var referenceInfo in References)
            {
                var referencedEntity = referenceInfo.PropertyInfo.GetValue(entity, null);
                var referencePrimaryKeyProperty = MetaDataStore.GetTableInfoFor(referenceInfo.ReferenceType).PrimaryKey.PropertyInfo;
 
                if (referencedEntity == null)
                {
                    parameters.Add(new AdoParameterInfo(referenceInfo.Name, referenceInfo.DbType, null));
                }
                else
                {
                    parameters.Add(new AdoParameterInfo(referenceInfo.Name, referenceInfo.DbType, referencePrimaryKeyProperty.GetValue(referencedEntity, null)));
                }
            }
 
            foreach (var columnInfo in Columns)
            {
                parameters.Add(new AdoParameterInfo(columnInfo.Name, columnInfo.DbType, columnInfo.PropertyInfo.GetValue(entity, null)));
            }
 
            return parameters;
        }
 
        private void RemoveLastCommaAndSpaceIfThereAreAnyColumns(StringBuilder builder)
        {
            if ((References.Count() + Columns.Count()) > 0)
            {
                RemoveLastCharacters(builder, 2);
            }
        }
 
        private void AddReferenceColumnNames(StringBuilder builder)
        {
            foreach (var referenceInfo in References)
            {
                builder.Append(Escape(referenceInfo.Name) + ", ");
            }
        }
 
        private void AddReferenceColumnParameterNames(StringBuilder builder)
        {
            foreach (var referenceInfo in References)
            {
                builder.Append("@" + referenceInfo.Name + ", ");
            }
        }
 
        private void AddReferenceColumnsNameWithParameterName(StringBuilder builder)
        {
            foreach (var referenceInfo in References)
            {
                builder.Append(Escape(referenceInfo.Name) + " = @" + referenceInfo.Name + ", ");
            }
        }
 
        private void AddRegularColumnNames(StringBuilder builder)
        {
            foreach (var columnInfo in Columns)
            {
                builder.Append(Escape(columnInfo.Name) + ", ");
            }
        }
 
        private void AddRegularColumnParameterNames(StringBuilder builder)
        {
            foreach (var columnInfo in Columns)
            {
                builder.Append("@" + columnInfo.Name + ", ");
            }
        }
 
        private void AddRegularColumnsNameWithParameterName(StringBuilder builder)
        {
            foreach (var columnInfo in Columns)
            {
                builder.Append(Escape(columnInfo.Name) + " = @" + columnInfo.Name + ", ");
            }
        }
 
        private string Escape(string name)
        {
            return "[" + name + "]";
        }
 
        private void RemoveLastCharacters(StringBuilder stringBuilder, int numberOfCharacters)
        {
            stringBuilder.Remove(stringBuilder.Length - numberOfCharacters, numberOfCharacters);
        }
    }

This is actually the biggest class in this DAL. I probably should move the building of the SQL statements and providing parameter info into some kind of helper class because this is a bit of a Single Responsability Principle violation. Speaking of parameter info, i'm using the following helper class to store this information:

    public class AdoParameterInfo
    {
        public DbType DbType { get; private set; }
        public string Name { get; private set; }
        public object Value { get; private set; }
 
        public AdoParameterInfo(string name, DbType dbType, object value)
        {
            Name = name;
            DbType = dbType;
            Value = value;
        }
    }

One thing that you may have noticed is that the generated INSERT statement assumes that SQL Server identity-style generators are being used for primary key values. Not only that, i'm not even trying to target any other database then SQL Server with this DAL. Those are 2 rather significant shortcomings of this DAL. First of all, dealing with multiple identifier strategies can become pretty complex pretty fast. For this DAL, SQL Server Identity primary keys are sufficient but in a lot of cases you will probably want support for assigned identifier strategies, for GUIDs (preferably locally generated with a sequential GUID algorithm), HiLo and maybe even other ones. If you really want to, you can do all of this yourself, but you'll quickly spend an entire week (or more) to properly implement all of these identifier strategies.

As for only targeting SQL Server, that is sufficient in our scenario but a proper DAL should be able to deal with multiple databases. Of course, this has a direct impact on a lot of implementation details. For starters, you'd never be able to just construct a SQL statement directly in your code and you will need something to make sure the correct statements are generated for your specific database. NHibernate does a pretty nice job of this by providing a strategy-like implementation through its Dialect class and its derivatives. Also, some of your identifier strategies will be different for each database that you need to support. If you got a headache just from reading these last 2 paragraphs, just imagine implementing this and getting it all 'right' in a maintainable matter.

Anyways, back to the topic at hand. We now have the classes we need to build up our metadata model of all of the tables we need to provide data access functionality for. Well, we still need something to hold all of this information and to actually build up this model:

    public class MetaDataStore
    {
        private readonly Dictionary typeToTableInfo = new Dictionary();
 
        public TableInfo GetTableInfoFor()
        {
            return GetTableInfoFor(typeof(TEntity));
        }
 
        public TableInfo GetTableInfoFor(Type entityType)
        {
            if (!typeToTableInfo.ContainsKey(entityType))
            {
                return null;
            }
 
            return typeToTableInfo[entityType];
        }
 
        public void BuildMetaDataFor(Assembly assembly)
        {
            BuildMapOfEntityTypesWithTheirTableInfo(assembly);
 
            foreach (KeyValuePair pair in typeToTableInfo)
            {
                // we need this info for each entity before we can deal with references to other entities
                LoopThroughPropertiesWith(pair.Key, pair.Value, SetPrimaryKeyInfo);
            }
 
            foreach (KeyValuePair pair in typeToTableInfo)
            {
                LoopThroughPropertiesWith(pair.Key, pair.Value, AddReferenceInfo);
                LoopThroughPropertiesWith(pair.Key, pair.Value, AddColumnInfo);
            }
        }
 
        private void BuildMapOfEntityTypesWithTheirTableInfo(Assembly assembly)
        {
            foreach (var type in assembly.GetTypes())
            {
                var typeAttributes = Attribute.GetCustomAttributes(type, typeof(TableAttribute));
 
                if (typeAttributes.Length > 0)
                {
                    var tableAttribute = (TableAttribute)typeAttributes[0];
                    var tableInfo = new TableInfo(this, tableAttribute.TableName, type);
                    typeToTableInfo.Add(type, tableInfo);
                }
            }
        }
 
        private void LoopThroughPropertiesWith(Type entityType, TableInfo tableInfo,
            Action andExecuteFollowingCode)
            where TAttribute : Attribute
        {
            foreach (var propertyInfo in entityType.GetProperties())
            {
                var attribute = GetAttribute(propertyInfo);
 
                if (attribute != null)
                {
                    andExecuteFollowingCode(tableInfo, propertyInfo, attribute);
                }
            }
        }
 
        private void SetPrimaryKeyInfo(TableInfo tableInfo, PropertyInfo propertyInfo, PrimaryKeyAttribute primaryKeyAttribute)
        {
            tableInfo.PrimaryKey = new ColumnInfo(this, primaryKeyAttribute.ColumnName, propertyInfo.PropertyType, propertyInfo);
        }
 
        private void AddColumnInfo(TableInfo tableInfo, PropertyInfo propertyInfo, ColumnAttribute columnAttribute)
        {
            tableInfo.AddColumn(new ColumnInfo(this, columnAttribute.ColumnName, propertyInfo.PropertyType, propertyInfo));
        }
 
        private void AddReferenceInfo(TableInfo tableInfo, PropertyInfo propertyInfo, ReferenceAttribute referenceAttribute)
        {
            tableInfo.AddReference(new ReferenceInfo(this, referenceAttribute.ColumnName, propertyInfo.PropertyType, propertyInfo));
        }
 
        private TAttribute GetAttribute(PropertyInfo propertyInfo) where TAttribute : Attribute
        {
            var attributes = Attribute.GetCustomAttributes(propertyInfo, typeof(TAttribute));
            if (attributes.Length == 0) return null;
            return (TAttribute)attributes[0];
        }
    }

This class gives you the ability to retrieve the TableInfo class for a specfic entity type. It also allows you to build the metadata model by passing in an assembly. It will then loop through all of the types in the assembly to discover the types that have a TableAttribute, and it will then build the TableInfo objects with all of the information we need.

And that's all we need to create mappings between tables and our entities. This wasn't hard, but it's not very powerful either. We can't define custom user types that our DAL needs to be able to deal with, nor can we define any database inheritance strategies. Our attributes are all inheritable, so you can use some inheritance with your entities, but you are essentially limited to the Table Per Class inheritance strategy. Implementing support for the other inheritance strategies would obviously introduce a lot more complexity in the whole mapping aspect.

In the next post, i'll show you how this DAL will use TableInfo's methods to create CRUD statements to offer out-of-the-box CRUD functionality for each mapped entity.

Build Your Own Data Access Layer Series

Davy Brion — Sun, 23 Aug 2009 13:38:43 +0000

I'm definitely not a fan of building your own Data Access Layer (DAL), since there are plenty of powerful and mature options already available. However, we have 2 customers at work who simply don't let us use any existing libraries/tools as a DAL and want us to just use straight ADO.NET. I don't want to get into their reasons for this, but the reality of the situation is that whenever we have to develop projects for them, we need to use a custom built DAL. I've never seen a custom built DAL that i found acceptable, let alone one that i actually wanted to use.

A lot of people typically go the code generation route when faced with this situation, which is exactly what we have done in the past. Been there, done that, hated it with a passion for various reasons. One of my coworkers recently started a new project for one of these customers, and he started implementing a new DAL. I had to review this, and while it had some good ideas there was a large amount of repetitive and error-prone code that still needed to be written by developers for every table. So i set out to come up with something better. If we did have to use a custom DAL for these customers, i wanted to make sure that it would at least avoid having us write repetitive, error-prone code for every table that we needed to use. Oh, and without having to resort to code generation. Since we are all NHibernate users (when customers don't have a problem with us using it, that is) i wanted something that was somewhat similar in ease-of-use though it could obviously never match its feature set, power and maturity.

I spent about 24 working hours (in total) on this, and i believe i came up with something that is acceptable for most simple forms-over-data applications. This DAL allows you to write your entity classes as POCO's, offers 'out-of-the-box' CRUD functionality for every mapped table, and has lazy loading for reference properties (so you don't need to pollute your entity classes with foreign key properties). There is also a simple session-level cache, and there is some functionality to ease the pain of using simple, custom queries (with that i mean: every query that is not a select all or select by id and that doesn't join other tables).

Compared to a real ORM, it is missing a lot: there is no Unit Of Work implementation, no automated change tracking of entities, no dirty checks, no collection support, no advanced querying possibilities, no statement batching, no serious caching functionality, no transitive persistence, and a whole host of features that something like NHibernate gives you for free. Each and every one of those features comes with a great cost of complexity and development time to get 'right' so it truly doesn't make a lot of sense to do all of this yourself.

In this series, we're going to go over the entire implementation of this DAL and throughout the series i will point out its shortcomings and try to explain the complexity that would be required to make it truly powerful. The purpose of this series is basically to:

Show you that you really don't need to resort to code generation to build your own custom DAL
Show you what kind of complexity is involved with the implementation of a good DAL
Convince you that you typically are better off with simply using something that is already available as a mature, powerful and proven solution

These are the posts that this series consists of:

I will update the list above with actual links to the posts as soon as they are published.

Note: the code of this series can be found here