Build Your Own DAL

Build Your Own Data Access Layer: Session Level Cache

10 commentsWritten on August 26th, 2009 by
Categories: Build Your Own DAL

Note: This post is part of a series. Be sure to read the introduction here.

In the previous 2 posts of this series, you may have noticed the usage of the SessionLevelCache class. This class is a simple implementation of the Identity Map pattern. It basically offers us two important benefits:

  1. It ensures that we never load the same entity from the database twice.
  2. It ensures that we can't accidentally have two instances which point to the same database record.

Well actually... there is one situation where you could possibly load the same entity twice (which i'll show momentarily) but even then we can make sure that our users will always have the same instance of the entity.

The SessionLevelCache class is actually very simple:

    public class SessionLevelCache
    {
        private readonly Dictionary<Type, Dictionary<string, object>> cache = new Dictionary<Type, Dictionary<string, object>>();
 
        public object TryToFind(Type type, object id)
        {
            if (!cache.ContainsKey(type)) return null;
 
            string idAsString = id.ToString();
            if (!cache[type].ContainsKey(idAsString)) return null;
 
            return cache[type][idAsString];
        }
 
        public void Store(Type type, object id, object entity)
        {
            if (!cache.ContainsKey(type)) cache.Add(type, new Dictionary<string, object>());
 
            cache[type][id.ToString()] = entity;
        }
 
        public void ClearAll()
        {
            cache.Clear();
        }
 
        public void RemoveAllInstancesOf(Type type)
        {
            if (cache.ContainsKey(type))
            {
                cache.Remove(type);
            }
        }
 
        public void Remove(object entity)
        {
            var type = entity.GetType();
 
            if (!cache.ContainsKey(type)) return;
 
            string keyToRemove = null;
 
            foreach (var pair in cache[type])
            {
                if (pair.Value == entity)
                {
                    keyToRemove = pair.Key;
                }
            }
 
            if (keyToRemove != null)
            {
                cache[type].Remove(keyToRemove);
            }
        }
    }

It uses a nested dictionary where the outer dictionary uses the entity type as its key value, and stores an inner dictionary for each entity type which uses the primary key value as the key for each entity instance. I think the code is simple enough so there's not really any point to go over the implementation details. Instead, let's focus on when and where the SessionLevelCache is used.

First, let's look back at the Get method of the GetByIdAction class:

        public TEntity Get<TEntity>(object id)
        {
            var cachedEntity = SessionLevelCache.TryToFind(typeof(TEntity), id);
            if (cachedEntity != null) return (TEntity)cachedEntity;
 
            using (var command = CreateCommand())
            {
                var tableInfo = MetaDataStore.GetTableInfoFor<TEntity>();
 
                var query = tableInfo.GetSelectStatementForAllFields();
                tableInfo.AddWhereByIdClause(query);
 
                command.CommandText = query.ToString();
                command.CreateAndAddInputParameter(tableInfo.PrimaryKey.DbType, tableInfo.GetPrimaryKeyParameterName(), id);
                return Hydrater.HydrateEntity<TEntity>(command);
            }
        }

As you can see, when we enter this method we first check to see whether the entity instance is already present in the cache. If it is, we obviously don't need to hit the database so we can simply return the cached entity.

But like i said, there is one situation where we could potentially retrieve the entity from the database even though we already have it in the cache, and that is when we are hydrating a list of entities that have been retrieved from the database. You might remember the following method from the EntityHydrater class:

        private TEntity CreateEntityFromValues<TEntity>(IDictionary<string, object> values)
        {
            var tableInfo = metaDataStore.GetTableInfoFor<TEntity>();
 
            var cachedEntity = sessionLevelCache.TryToFind(typeof(TEntity), values[tableInfo.PrimaryKey.Name]);
            if (cachedEntity != null) return (TEntity)cachedEntity;
 
            var entity = Activator.CreateInstance<TEntity>();
            Hydrate(tableInfo, entity, values);
            sessionLevelCache.Store(typeof(TEntity), values[tableInfo.PrimaryKey.Name], entity);
            return entity;
        }

This method is used to hydrate every single entity that we create. Obviously, when hydrating a single entity we've actually already performed the cache-look-up in the GetByIdAction. However, when a custom query is executed, or when all instances are retrieved, there is no way for us to exclude already cached entity instances from the result of the query. Well, theoretically speaking you could attempt to do this by adding a clause to the WHERE statement of each query that would prevent cached entities from being loaded. But then you might have to add the cached entity instances to the resulting list of entities anyways if they would otherwise satisfy the other query conditions. Obviously, trying to get this right is simply put insane and i don't think there's any DAL or ORM that actually does this (even if there was, i can't really imagine any of them getting this right in every corner case that will pop up).

So a good compromise is to simply check for the existence of a specific instance in the cache before hydrating a new instance. If it is there, we return it from the cache and we skip the hydration for that database record. In this way, we avoid having to modify the original query, and while we could potentially return a few records that we already have in memory, at least we will be sure that our users will always have the same reference for any particular database record.

There is one more scenario that needs to be covered. If an entity holds a reference through a foreign key to another entity instance, and that referenced entity is already present in the cache, we need to make sure that the entity we are hydrating will refer to the already cached referred-to-instance instead of creating a proxy by default. After all, if we were to create a proxy object for an entity instance that is already in our cache, we will have failed to achieve our goal of avoiding the possibility of more than one instance representing the same database record. Therefore, when hydrating the reference properties of an entity, we have the following piece of code:

                        var referencedEntity = sessionLevelCache.TryToFind(referenceInfo.ReferenceType, foreignKeyValue) ??
                                               CreateProxy(tableInfo, referenceInfo, foreignKeyValue);
 
                        referenceInfo.PropertyInfo.SetValue(entity, referencedEntity, null);

This way, we are sure that a proxy object for a referenced entity will only be created if that entity is not already present in the SessionLevelCache.

Now we're pretty much covered when it comes to retrieving entity instances from the database. But we obviously also need to update the SessionLevelCache whenever an entity is inserted, and whenever an entity is deleted. In the InsertAction, you can find the following code at the end of the Insert method:

                SessionLevelCache.Store(typeof(TEntity), id, entity);

And in the DeleteAction we can also spot the following line of code:

                SessionLevelCache.Remove(entity);

There is still one problem however. When a user executes a custom DELETE statement, there is no way for us to know which entities were actually removed from the database. But if any of those deleted entities happen to remain in the SessionLevelCache, this could lead to buggy application code whenever a piece of code tries to retrieve a specific entity which has already been removed from the database, but is still present in the SessionLevelCache. In order to deal with this scenario, the SessionLevelCache has a ClearAll and a RemoveAllInstancesOf method which you can use from your application code to either clear the entire SessionLevelCache, or to remove all instances of a specific entity type from the cache. Calling these methods would obviously be the responsability of the application code, since we can't possibly take care of this automatically in such a simplistic DAL. Actually, even for powerful ORM's this can be pretty difficult to get right.

Another thing i'd like to point out is that the SessionLevelCache is not threadsafe. A session (which will be covered in a later post) is not threadsafe, so within each of the classes that are used by the session i take no care of thread-safety whatsoever.

The SessionLevelCache might not seem like much to you, but i do consider its to be an absolute must for any DAL. Notice that this doesn't even come close to a proper second-level cache like NHibernate offers, but the complexity of implementing such a thing is way beyond the scope of both this series and probably most custom DAL's out there.

Build Your Own Data Access Layer: Hydrating Entities

15 commentsWritten on August 25th, 2009 by
Categories: Build Your Own DAL

Note: This post is part of a series. Be sure to read the introduction here.

In the previous post of this series, you saw that some of the DatabaseActions use the EntityHydrater to umm.. hydrate the entities with their values from the database. In this post, we'll go over how this actually works. First, i'm going to post the code of the entire class and then we'll go over the interesting parts.

So, here's the entire code of the EntityHydrater class:

    public class EntityHydrater
    {
        private readonly MetaDataStore metaDataStore;
        private readonly SessionLevelCache sessionLevelCache;
 
        public EntityHydrater(MetaDataStore metaDataStore, SessionLevelCache sessionLevelCache)
        {
            this.metaDataStore = metaDataStore;
            this.sessionLevelCache = sessionLevelCache;
        }
 
        public TEntity HydrateEntity<TEntity>(SqlCommand command)
        {
            IDictionary<string, object> values;
 
            using (var reader = command.ExecuteReader())
            {
                if (!reader.HasRows) return default(TEntity);
                reader.Read();
                values = GetValuesFromCurrentRow(reader);
            }
 
            return CreateEntityFromValues<TEntity>(values);
        }
 
        public IEnumerable<TEntity> HydrateEntities<TEntity>(SqlCommand command)
        {
            var rows = new List<IDictionary<string, object>>();
            var entities = new List<TEntity>();
 
            using (var reader = command.ExecuteReader())
            {
                while (reader.Read())
                {
                    rows.Add(GetValuesFromCurrentRow(reader));
                }
            }
 
            foreach (var row in rows)
            {
                entities.Add(CreateEntityFromValues<TEntity>(row));
            }
 
            return entities;
        }
 
        private IDictionary<string, object> GetValuesFromCurrentRow(SqlDataReader dataReader)
        {
            var values = new Dictionary<string, object>();
 
            for (int i = 0; i < dataReader.FieldCount; i++)
            {
                values.Add(dataReader.GetName(i), dataReader.GetValue(i));
            }
 
            return values;
        }
 
        private TEntity CreateEntityFromValues<TEntity>(IDictionary<string, object> values)
        {
            var tableInfo = metaDataStore.GetTableInfoFor<TEntity>();
 
            var cachedEntity = sessionLevelCache.TryToFind(typeof(TEntity), values[tableInfo.PrimaryKey.Name]);
            if (cachedEntity != null) return (TEntity)cachedEntity;
 
            var entity = Activator.CreateInstance<TEntity>();
            Hydrate(tableInfo, entity, values);
            sessionLevelCache.Store(typeof(TEntity), values[tableInfo.PrimaryKey.Name], entity);
            return entity;
        }
 
        private void Hydrate<TEntity>(TableInfo tableInfo, TEntity entity, IDictionary<string, object> values)
        {
            tableInfo.PrimaryKey.PropertyInfo.SetValue(entity, values[tableInfo.PrimaryKey.Name], null);
            SetRegularColumns(tableInfo, entity, values);
            SetReferenceProperties(tableInfo, entity, values);
        }
 
        private void SetRegularColumns<TEntity>(TableInfo tableInfo, TEntity entity, IDictionary<string, object> values)
        {
            foreach (var columnInfo in tableInfo.Columns)
            {
                if (columnInfo.PropertyInfo.CanWrite)
                {
                    object value = values[columnInfo.Name];
                    if (value is DBNull) value = null;
                    columnInfo.PropertyInfo.SetValue(entity, value, null);
                }
            }
        }
 
        private void SetReferenceProperties<TEntity>(TableInfo tableInfo, TEntity entity, IDictionary<string, object> values)
        {
            foreach (var referenceInfo in tableInfo.References)
            {
                if (referenceInfo.PropertyInfo.CanWrite)
                {
                    object foreignKeyValue = values[referenceInfo.Name];
 
                    if (foreignKeyValue is DBNull)
                    {
                        referenceInfo.PropertyInfo.SetValue(entity, null, null);
                    }
                    else
                    {
                        var referencedEntity = sessionLevelCache.TryToFind(referenceInfo.ReferenceType, foreignKeyValue) ??
                                               CreateProxy(tableInfo, referenceInfo, foreignKeyValue);
 
                        referenceInfo.PropertyInfo.SetValue(entity, referencedEntity, null);
                    }
                }
            }
        }
 
        private object CreateProxy(TableInfo tableInfo, ReferenceInfo referenceInfo, object foreignKeyValue)
        {
            // NOTE: this will be covered in a later post, so i took out the spoiler code 
            return null;
        }
    }

As you can see, there are 2 public methods: HydrateEntity and HydrateEntities. They are both pretty similar, except that the former only hydrates a single entity and the latter a list of entities. They both retrieve the values from the current position in a DataReader and store them in a Dictionary with the name of the column being the key and the value of the column being the value in the dictionary:

        private IDictionary<string, object> GetValuesFromCurrentRow(SqlDataReader dataReader)
        {
            var values = new Dictionary<string, object>();
 
            for (int i = 0; i < dataReader.FieldCount; i++)
            {
                values.Add(dataReader.GetName(i), dataReader.GetValue(i));
            }
 
            return values;
        }

This dictionary is then passed to the CreateEntityFromValues method:

        private TEntity CreateEntityFromValues<TEntity>(IDictionary<string, object> values)
        {
            var tableInfo = metaDataStore.GetTableInfoFor<TEntity>();
 
            var cachedEntity = sessionLevelCache.TryToFind(typeof(TEntity), values[tableInfo.PrimaryKey.Name]);
            if (cachedEntity != null) return (TEntity)cachedEntity;
 
            var entity = Activator.CreateInstance<TEntity>();
            Hydrate(tableInfo, entity, values);
            sessionLevelCache.Store(typeof(TEntity), values[tableInfo.PrimaryKey.Name], entity);
            return entity;
        }

This method will first check to see if an entity instance for the row in the values dictionary already exists. If it exists already, it simply returns the entity instance and ignores the values in the dictionary. I'm not sure yet whether this behavior is correct or not. Theoretically speaking, it's possible when using ReadCommitted isolation level that the values in the dictionary will be more recent than the entity instance in the session level cache. However, since this simple DAL has no change tracking, i also can't deduce whether the instance in the cache has already had one or more of its properties updated by application code. Simply overwriting them with newly retrieved values of the database doesn't seem like the right thing to do here. So the current options are to either ignore this possibility, or to rely on optimistic concurrency. Oh, and i don't have support for optimistic concurrency either. A good DAL should however provide some optimistic concurrency strategies here, or be able to track the changes in the entity and throw an exception if it notices that the database contains more recent values AND the local entity has already been modified. If it hasn't been modified yet, it could overwrite the values of the instances though i'm not sure everyone would agree with this behavior. Either way, this particular problem is definitely interesting enough to think about for a while ;)

Anyways, in this implementation, i'm completely ignoring this situation and i either return the already loaded entity of the session level cache, or i hydrate a new entity instance and return that instead. In the latter case, i also store the entity instance in the session level cache. Note that the actual implementation of the session level cache will be covered in the next post of this series.

The actual Hydrate method looks like this:

        private void Hydrate<TEntity>(TableInfo tableInfo, TEntity entity, IDictionary<string, object> values)
        {
            tableInfo.PrimaryKey.PropertyInfo.SetValue(entity, values[tableInfo.PrimaryKey.Name], null);
            SetRegularColumns(tableInfo, entity, values);
            SetReferenceProperties(tableInfo, entity, values);
        }

First the primary key value of the record is set in the primary key property of the new instance, and then we proceed with putting the regular column values in their properties, and after that, the reference properties. Filling the regular column properties is very straightforward:

        private void SetRegularColumns<TEntity>(TableInfo tableInfo, TEntity entity, IDictionary<string, object> values)
        {
            foreach (var columnInfo in tableInfo.Columns)
            {
                if (columnInfo.PropertyInfo.CanWrite)
                {
                    object value = values[columnInfo.Name];
                    if (value is DBNull) value = null;
                    columnInfo.PropertyInfo.SetValue(entity, value, null);
                }
            }
        }

Dealing with the references is a bit more interesting though.

        private void SetReferenceProperties<TEntity>(TableInfo tableInfo, TEntity entity, IDictionary<string, object> values)
        {
            foreach (var referenceInfo in tableInfo.References)
            {
                if (referenceInfo.PropertyInfo.CanWrite)
                {
                    object foreignKeyValue = values[referenceInfo.Name];
 
                    if (foreignKeyValue is DBNull)
                    {
                        referenceInfo.PropertyInfo.SetValue(entity, null, null);
                    }
                    else
                    {
                        var referencedEntity = sessionLevelCache.TryToFind(referenceInfo.ReferenceType, foreignKeyValue) ??
                                               CreateProxy(tableInfo, referenceInfo, foreignKeyValue);
 
                        referenceInfo.PropertyInfo.SetValue(entity, referencedEntity, null);
                    }
                }
            }
        }

If we can't find the referenced entity instance in the first level cache, what should we do? We obviously can't load it automatically because that could in turn cause referenced entities' references to be loaded automatically when they are hydrated. Which in turn could cause their referenced entities... Well, i'm sure you get the point. But those properties obviously can't be set to a null reference either because the column actually does have a valid foreign key value in the database. Explicitly loading referenced properties leads to seriously ugly (and error-prone) code so that's not an option i'm willing to consider either. The correct way to deal with this is to use lazy loading. To do that in an automated fashion, we need proxy classes. I'm not going to get into these proxy classes and the whole lazy loading thing just yet, since that will be covered in depth in a future post ;)

So that's pretty much it (for now) for our EntityHydrater class. As you can see, it's still relatively simple but then again, the use cases that it supports are extremely simple as well. This current implementation is incapable of hydrating entities based on a SQL statement that selects data from more than just the entity's table. And that is a pretty big shortcoming. For instance, with NHibernate you can execute queries where you can instruct NHibernate to fetch some (or all) of the entity's references (associations in NHibernate) with just one SQL statement, using the join syntax. NHibernate can then hydrate the root entity, and populate its reference properties with the other values that were returned by the sql statement. While it wouldn't be that complex to add this capability to this EntityHydrater class, it wouldn't exactly be completely trivial either. Again, this is a limitation that many (maybe even most?) custom DAL's have. This one probably makes it easy enough to still add this feature though ;)

Build Your Own Data Access Layer: Out Of The Box CRUD Functionality

14 commentsWritten on August 24th, 2009 by
Categories: Build Your Own DAL

Note: This post is part of a series. Be sure to read the introduction here.

One thing that i consider an absolute must-have in any data access layer is the ability to perform CRUD operations out-of-the-box without having to write any code to enable these operations. Once your data access layer knows about your classes and your tables, CRUD operations should 'just work'.

As you've seen in the previous post of this series, the TableInfo class offers a couple of methods to automatically build the required SQL statements for CRUD actions. With these statements, we can easily create SqlCommand instances for all CRUD operations.

First of all, i use the following helper method to easily add a SqlParameter to a SqlCommand:

        public static void CreateAndAddInputParameter(this SqlCommand command, DbType type, string name, object value)
        {
            var parameter = command.CreateParameter();
            parameter.Direction = ParameterDirection.Input;
            parameter.DbType = type;
            parameter.ParameterName = name;
 
            if (value == null)
            {
                parameter.IsNullable = true;
                parameter.Value = DBNull.Value;
            }
            else
            {
                parameter.Value = value;
            }
 
            command.Parameters.Add(parameter);
        }

I also have the following abstract DatabaseAction class which has a few properties that are used by most of the CRUD actions:

    public abstract class DatabaseAction
    {
        private readonly SqlConnection connection;
        private readonly SqlTransaction transaction;
        protected MetaDataStore MetaDataStore { get; private set; }
        protected EntityHydrater Hydrater { get; private set; }
        protected SessionLevelCache SessionLevelCache { get; private set; }
 
        protected DatabaseAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
                                 EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
        {
            this.connection = connection;
            this.transaction = transaction;
            MetaDataStore = metaDataStore;
            Hydrater = hydrater;
            SessionLevelCache = sessionLevelCache;
        }
 
        protected SqlCommand CreateCommand()
        {
            var command = connection.CreateCommand();
            command.Transaction = transaction;
            return command;
        }
    }

Did you notice the EntityHydrater and SessionLevelCache? I'm going to ignore those as much as possible for now, since they will be covered in depth in the following two posts in these series. The important thing to note is that each derived DatabaseAction will have a reference to the MetaDataStore.

And now we can easily start implementing our CRUD actions. Let's start with the GetByIdAction:

    public class GetByIdAction : DatabaseAction
    {
        public GetByIdAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
            EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
            : base(connection, transaction, metaDataStore, hydrater, sessionLevelCache)
        {
        }
 
        public TEntity Get<TEntity>(object id)
        {
            var cachedEntity = SessionLevelCache.TryToFind(typeof(TEntity), id);
            if (cachedEntity != null) return (TEntity)cachedEntity;
 
            using (var command = CreateCommand())
            {
                var tableInfo = MetaDataStore.GetTableInfoFor<TEntity>();
 
                var query = tableInfo.GetSelectStatementForAllFields();
                tableInfo.AddWhereByIdClause(query);
 
                command.CommandText = query.ToString();
                command.CreateAndAddInputParameter(tableInfo.PrimaryKey.DbType, tableInfo.GetPrimaryKeyParameterName(), id);
                return Hydrater.HydrateEntity<TEntity>(command);
            }
        }
    }

Pretty simple stuff, right? This will first check the session level cache to see if this instance has already been retrieved in the current session (i'll discuss the session in a later post) and if so, it will return that instance. If it's not in the cache, it will create a SqlCommand and fill its CommandText property with a SQL string that is provided by the relevant TableInfo class. After that, it passes the SqlCommand to the EntityHydrater so it can return an actual entity instance.

The details of EntityHydration will be fully explored in the next post of this series, so for now you only need to know that it can transform the results from the SqlCommand to an instance of TEntity.

It's always useful to get a collection of all instances of a certain entity class, so we also have this very simple FindAllAction:

    public class FindAllAction : DatabaseAction
    {
        public FindAllAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
            EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
            : base(connection, transaction, metaDataStore, hydrater, sessionLevelCache)
        {
        }
 
        public IEnumerable<TEntity> FindAll<TEntity>()
        {
            using (var command = CreateCommand())
            {
                command.CommandText = MetaDataStore.GetTableInfoFor<TEntity>().GetSelectStatementForAllFields().ToString();
                return Hydrater.HydrateEntities<TEntity>(command);
            }
        }
    }

We also need an InsertAction:

    public class InsertAction : DatabaseAction
    {
        public InsertAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
            EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
            : base(connection, transaction, metaDataStore, hydrater, sessionLevelCache)
        {
        }
 
        public TEntity Insert<TEntity>(TEntity entity)
        {
            using (var command = CreateCommand())
            {
                var tableInfo = MetaDataStore.GetTableInfoFor<TEntity>();
 
                command.CommandText = tableInfo.GetInsertStatement();
 
                foreach (var parameterInfo in tableInfo.GetParametersForInsert(entity))
                {
                    command.CreateAndAddInputParameter(parameterInfo.DbType, parameterInfo.Name, parameterInfo.Value);
                }
 
                object id = Convert.ChangeType(command.ExecuteScalar(), tableInfo.PrimaryKey.DotNetType);
                tableInfo.PrimaryKey.PropertyInfo.SetValue(entity, id, null);
                SessionLevelCache.Store(typeof(TEntity), id, entity);
                return entity;
            }
        }
    }

There's not a lot to this one either... The actual insert statement is once again retrieved through the TableInfo class, as are the parameter values (including their values for this specific entity). You can go back to the previous post to look at the implementation of TableInfo's GetParametersForInsert method :)

Keep in mind that there is a limitation here that i only support SQL Server's Identity-style generators. Again, if you want to support multiple identifier strategies like NHibernate does, you'll have to deal with a lot more complexity in the InsertAction class.

The UpdateAction is very similar:

    public class UpdateAction : DatabaseAction
    {
        public UpdateAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
            EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
            : base(connection, transaction, metaDataStore, hydrater, sessionLevelCache)
        {
        }
 
        public TEntity Update<TEntity>(TEntity entity)
        {
            using (var command = CreateCommand())
            {
                var tableInfo = MetaDataStore.GetTableInfoFor<TEntity>();
 
                command.CommandText = tableInfo.GetUpdateStatement();
 
                foreach (var parameterInfo in tableInfo.GetParametersForUpdate(entity))
                {
                    command.CreateAndAddInputParameter(parameterInfo.DbType, parameterInfo.Name, parameterInfo.Value);
                }
 
                command.ExecuteNonQuery();
                return entity;
            }
        }
    }

And finally, we have the DeleteAction:

    public class DeleteAction : DatabaseAction
    {
        public DeleteAction(SqlConnection connection, SqlTransaction transaction, MetaDataStore metaDataStore,
            EntityHydrater hydrater, SessionLevelCache sessionLevelCache)
            : base(connection, transaction, metaDataStore, hydrater, sessionLevelCache)
        {
        }
 
        public void Delete<TEntity>(TEntity entity)
        {
            using (var command = CreateCommand())
            {
                var tableInfo = MetaDataStore.GetTableInfoFor<TEntity>();
                command.CommandText = tableInfo.GetDeleteStatement();
                object id = tableInfo.PrimaryKey.PropertyInfo.GetValue(entity, null);
                command.CreateAndAddInputParameter(tableInfo.PrimaryKey.DbType, tableInfo.GetPrimaryKeyParameterName(), id);
                command.ExecuteNonQuery();
                SessionLevelCache.Remove(entity);
            }
        }
    }

And that's all there is to it. We now have some classes that will give us out-of-the-box CRUD functionality for all of the mapped entity classes. Obviously, you will still need some way of actually accessing this functionality from your application code and you certainly don't want to instantiate and use these DatabaseAction classes directly. All of that will be covered in the "Bringing It All Together" post, so stay tuned ;)

Build Your Own Data Access Layer: Mapping Classes To Tables

11 commentsWritten on August 23rd, 2009 by
Categories: Build Your Own DAL

Note: This post is part of a series. Be sure to read the introduction here.

When you need to populate entity instances with data from a database, you need to know which table the data needs to come from, which columns will map to which property on the entity class, and you'll need to deal with a variety of types. The approach that i've chosen to use tries to make this as simple as possible. The idea is basically to place an attribute with the name of the table on top of the entity class, and an attribute on each property with the name of the column it maps to. For foreign keys, i wanted to be able to just use properties of the type of the referenced entity, instead of having foreign keys in my entities. For these references, we will use an attribute with the name of the foreign key column.

First, we'll need to define these attributes:

    [AttributeUsage(AttributeTargets.Class, Inherited = true)]
    public class TableAttribute : Attribute
    {
        public string TableName { get; private set; }
 
        public TableAttribute(string tableName)
        {
            TableName = tableName;
        }
    }

    [AttributeUsage(AttributeTargets.Property, Inherited = true)]
    public class PrimaryKeyAttribute : Attribute
    {
        public string ColumnName { get; private set; }
 
        public PrimaryKeyAttribute(string columnName)
        {
            ColumnName = columnName;
        }
    }

    [AttributeUsage(AttributeTargets.Property, Inherited = true)]
    public class ColumnAttribute : Attribute
    {
        public string ColumnName { get; private set; }
 
        public ColumnAttribute(string columnName)
        {
            ColumnName = columnName;
        }
    }

    [AttributeUsage(AttributeTargets.Property, Inherited = true)]
    public class ReferenceAttribute : Attribute
    {
        public string ColumnName { get; private set; }
 
        public ReferenceAttribute(string columnName)
        {
            ColumnName = columnName;
        }
    }

Notice how none of these properties have any indication of types to use. The .NET type will be inferred automatically, and it will be mapped to a compatible DbType without having to specify these types all over the place.

We will use the following helper class to map .NET types to their respective DbTypes:

    public static class TypeConverter
    {
        private static readonly Dictionary<Type, DbType> typeToDbType = new Dictionary<Type, DbType>
        {
            { typeof(string), DbType.String },
            { typeof(DateTime), DbType.DateTime },
            { typeof(DateTime?), DbType.DateTime },
            { typeof(int), DbType.Int32 },
            { typeof(int?), DbType.Int32 },
            { typeof(long), DbType.Int64 },
            { typeof(long?), DbType.Int64 },
            { typeof(bool), DbType.Boolean },
            { typeof(bool?), DbType.Boolean },
            { typeof(byte[]), DbType.Binary },
            { typeof(decimal), DbType.Decimal },
            { typeof(decimal?), DbType.Decimal },
            { typeof(double), DbType.Double },
            { typeof(double?), DbType.Double },
            { typeof(float), DbType.Single },
            { typeof(float?), DbType.Single },
            { typeof(Guid), DbType.Guid },
            { typeof(Guid?), DbType.Guid }
        };
 
        public static DbType ToDbType(Type type)
        {
            if (!typeToDbType.ContainsKey(type))
            {
                throw new InvalidOperationException(string.Format("Type {0} doesn't have a matching DbType configured", type.FullName));
            }
 
            return typeToDbType[type];
        }
    }

Obviously, more type conversions can be added... these are just the ones i've needed so far.

Once you've placed all the attributes on top of your entities and properties, we can start building a model of all this metadata. This will all be stored in a MetaDataStore class that i'll show later on in this post. Having access to the MetaDataStore makes the implementation of some of these metadata types easier, so i have the following abstract class:

    public abstract class MetaData
    {
        protected MetaDataStore MetaDataStore { get; private set; }
 
        protected MetaData(MetaDataStore metaDataStore)
        {
            MetaDataStore = metaDataStore;
        }
    }

Now we can go over each piece of metadata. First, the ColumnInfo class:

    public class ColumnInfo : MetaData
    {
        public string Name { get; private set; }
        public Type DotNetType { get; private set; }
        public DbType DbType { get; private set; }
        public PropertyInfo PropertyInfo { get; private set; }
 
        public ColumnInfo(MetaDataStore store, string name, Type dotNetType, PropertyInfo propertyInfo)
            : this(store, name, dotNetType, TypeConverter.ToDbType(dotNetType), propertyInfo)
        {
        }
 
        public ColumnInfo(MetaDataStore store, string name, Type dotNetType, DbType dbType, PropertyInfo propertyInfo)
            : base(store)
        {
            Name = name;
            DotNetType = dotNetType;
            DbType = dbType;
            PropertyInfo = propertyInfo;
        }
    }

As you can see, we have all the information we need to be able to do something with this column. We have its Name, the .NET type that is used in the mapped class, the DbType and a PropertyInfo reference to its respective property in the mapped class so we can get and set its value.

For references, we need to know something more:

    public class ReferenceInfo : ColumnInfo
    {
        public Type ReferenceType { get; private set; }
 
        public ReferenceInfo(MetaDataStore store, string name, Type referenceType, PropertyInfo propertyInfo)
            : base(store, name, store.GetTableInfoFor(referenceType).PrimaryKey.DotNetType,
                    store.GetTableInfoFor(referenceType).PrimaryKey.DbType, propertyInfo)
        {
            ReferenceType = referenceType;
        }
    }

For a regular column, it's sufficient to know the .NET type of the property and the DbType. But for a reference, you need to know the actual type of the referenced entity, as well as the .NET type of it's primary key column. As you can see in the constructor, we retrieve the TableInfo of the referenced entity, and use the .NET type and the DbType of the primary key of the referenced entity. The PrimaryKey property of a TableInfo class (which i'll show below) is also a ColumnInfo object. We obviously also store the actual type of the referenced entity. And of course, we again store a PropertyInfo so we can get/set the value of the reference.

The TableInfo class can now hold all of the information that we need. We know all about its primary key (through the PrimaryKeyAttribute), its regular properties (through the ColumnAttribute) and its referenced properties (through the ReferenceAttribute). With all of that information, the TableInfo class is able to build your typical default SQL statements for CRUD functionality:

    public class TableInfo : MetaData
    {
        public string Name { get; private set; }
        public Type EntityType { get; private set; }
        public ColumnInfo PrimaryKey { get; set; }
        public IEnumerable<ReferenceInfo> References { get { return references.Values; } }
        public IEnumerable<ColumnInfo> Columns { get { return columns.Values; } }
 
        private readonly Dictionary<string, ColumnInfo> columns = new Dictionary<string, ColumnInfo>();
        private readonly Dictionary<string, ReferenceInfo> references = new Dictionary<string, ReferenceInfo>();
 
        public TableInfo(MetaDataStore store, string name, Type entityType)
            : base(store)
        {
            Name = name;
            EntityType = entityType;
        }
 
        public void AddColumn(ColumnInfo column)
        {
            if (columns.ContainsKey(column.Name))
            {
                throw new InvalidOperationException(string.Format("An item with key {0} has already been added", column.Name));
            }
 
            columns.Add(column.Name, column);
        }
 
        public void AddReference(ReferenceInfo reference)
        {
            if (references.ContainsKey(reference.Name))
            {
                throw new InvalidOperationException(string.Format("An item with key {0} has already been added", reference.Name));
            }
 
            references.Add(reference.Name, reference);
        }
 
        public ColumnInfo GetColumn(string columnName)
        {
            if (!columns.ContainsKey(columnName))
            {
                throw new InvalidOperationException(string.Format("The table '{0}' does not have a '{1}' column", Name, columnName));
            }
 
            return columns[columnName];
        }
 
        public StringBuilder GetSelectStatementForAllFields()
        {
            StringBuilder builder = new StringBuilder("SELECT " + Escape(PrimaryKey.Name) + ", ");
 
            AddReferenceColumnNames(builder);
            AddRegularColumnNames(builder);
            RemoveLastCommaAndSpaceIfThereAreAnyColumns(builder);
            builder.Append(" FROM " + Escape(Name));
 
            return builder;
        }
 
        public string GetInsertStatement()
        {
            StringBuilder builder = new StringBuilder("INSERT INTO " + Escape(Name) + " (");
 
            AddReferenceColumnNames(builder);
            AddRegularColumnNames(builder);
            RemoveLastCommaAndSpaceIfThereAreAnyColumns(builder);
            builder.Append(") VALUES (");
            AddReferenceColumnParameterNames(builder);
            AddRegularColumnParameterNames(builder);
            RemoveLastCommaAndSpaceIfThereAreAnyColumns(builder);
            builder.Append("); SELECT SCOPE_IDENTITY();");
 
            return builder.ToString();
        }
 
        public string GetUpdateStatement()
        {
            StringBuilder builder = new StringBuilder("UPDATE " + Escape(Name) + " SET ");
 
            AddReferenceColumnsNameWithParameterName(builder);
            AddRegularColumnsNameWithParameterName(builder);
            RemoveLastCommaAndSpaceIfThereAreAnyColumns(builder);
            AddWhereByIdClause(builder);
            builder.Append(";");
 
            return builder.ToString();
        }
 
        public string GetDeleteStatement()
        {
            StringBuilder builder = new StringBuilder("DELETE FROM " + Escape(Name) + " ");
 
            AddWhereByIdClause(builder);
            builder.Append(";");
 
            return builder.ToString();
        }
 
        public IEnumerable<AdoParameterInfo> GetParametersForInsert(object entity)
        {
            return GetParametersForAllReferenceAndRegularColumns(entity);
        }
 
        public IEnumerable<AdoParameterInfo> GetParametersForUpdate(object entity)
        {
            var parameters = GetParametersForAllReferenceAndRegularColumns(entity);
            parameters.Add(new AdoParameterInfo(PrimaryKey.Name, PrimaryKey.DbType, PrimaryKey.PropertyInfo.GetValue(entity, null)));
            return parameters;
        }
 
        public StringBuilder AddWhereByIdClause(StringBuilder query)
        {
            query.Append(" WHERE " + Escape(PrimaryKey.Name) + " = " + GetPrimaryKeyParameterName());
            return query;
        }
 
        public string GetPrimaryKeyParameterName()
        {
            return "@" + PrimaryKey.Name;
        }
 
        private List<AdoParameterInfo> GetParametersForAllReferenceAndRegularColumns(object entity)
        {
            var parameters = new List<AdoParameterInfo>();
 
            foreach (var referenceInfo in References)
            {
                var referencedEntity = referenceInfo.PropertyInfo.GetValue(entity, null);
                var referencePrimaryKeyProperty = MetaDataStore.GetTableInfoFor(referenceInfo.ReferenceType).PrimaryKey.PropertyInfo;
 
                if (referencedEntity == null)
                {
                    parameters.Add(new AdoParameterInfo(referenceInfo.Name, referenceInfo.DbType, null));
                }
                else
                {
                    parameters.Add(new AdoParameterInfo(referenceInfo.Name, referenceInfo.DbType, referencePrimaryKeyProperty.GetValue(referencedEntity, null)));
                }
            }
 
            foreach (var columnInfo in Columns)
            {
                parameters.Add(new AdoParameterInfo(columnInfo.Name, columnInfo.DbType, columnInfo.PropertyInfo.GetValue(entity, null)));
            }
 
            return parameters;
        }
 
        private void RemoveLastCommaAndSpaceIfThereAreAnyColumns(StringBuilder builder)
        {
            if ((References.Count() + Columns.Count()) > 0)
            {
                RemoveLastCharacters(builder, 2);
            }
        }
 
        private void AddReferenceColumnNames(StringBuilder builder)
        {
            foreach (var referenceInfo in References)
            {
                builder.Append(Escape(referenceInfo.Name) + ", ");
            }
        }
 
        private void AddReferenceColumnParameterNames(StringBuilder builder)
        {
            foreach (var referenceInfo in References)
            {
                builder.Append("@" + referenceInfo.Name + ", ");
            }
        }
 
        private void AddReferenceColumnsNameWithParameterName(StringBuilder builder)
        {
            foreach (var referenceInfo in References)
            {
                builder.Append(Escape(referenceInfo.Name) + " = @" + referenceInfo.Name + ", ");
            }
        }
 
        private void AddRegularColumnNames(StringBuilder builder)
        {
            foreach (var columnInfo in Columns)
            {
                builder.Append(Escape(columnInfo.Name) + ", ");
            }
        }
 
        private void AddRegularColumnParameterNames(StringBuilder builder)
        {
            foreach (var columnInfo in Columns)
            {
                builder.Append("@" + columnInfo.Name + ", ");
            }
        }
 
        private void AddRegularColumnsNameWithParameterName(StringBuilder builder)
        {
            foreach (var columnInfo in Columns)
            {
                builder.Append(Escape(columnInfo.Name) + " = @" + columnInfo.Name + ", ");
            }
        }
 
        private string Escape(string name)
        {
            return "[" + name + "]";
        }
 
        private void RemoveLastCharacters(StringBuilder stringBuilder, int numberOfCharacters)
        {
            stringBuilder.Remove(stringBuilder.Length - numberOfCharacters, numberOfCharacters);
        }
    }

This is actually the biggest class in this DAL. I probably should move the building of the SQL statements and providing parameter info into some kind of helper class because this is a bit of a Single Responsability Principle violation. Speaking of parameter info, i'm using the following helper class to store this information:

    public class AdoParameterInfo
    {
        public DbType DbType { get; private set; }
        public string Name { get; private set; }
        public object Value { get; private set; }
 
        public AdoParameterInfo(string name, DbType dbType, object value)
        {
            Name = name;
            DbType = dbType;
            Value = value;
        }
    }

One thing that you may have noticed is that the generated INSERT statement assumes that SQL Server identity-style generators are being used for primary key values. Not only that, i'm not even trying to target any other database then SQL Server with this DAL. Those are 2 rather significant shortcomings of this DAL. First of all, dealing with multiple identifier strategies can become pretty complex pretty fast. For this DAL, SQL Server Identity primary keys are sufficient but in a lot of cases you will probably want support for assigned identifier strategies, for GUIDs (preferably locally generated with a sequential GUID algorithm), HiLo and maybe even other ones. If you really want to, you can do all of this yourself, but you'll quickly spend an entire week (or more) to properly implement all of these identifier strategies.

As for only targeting SQL Server, that is sufficient in our scenario but a proper DAL should be able to deal with multiple databases. Of course, this has a direct impact on a lot of implementation details. For starters, you'd never be able to just construct a SQL statement directly in your code and you will need something to make sure the correct statements are generated for your specific database. NHibernate does a pretty nice job of this by providing a strategy-like implementation through its Dialect class and its derivatives. Also, some of your identifier strategies will be different for each database that you need to support. If you got a headache just from reading these last 2 paragraphs, just imagine implementing this and getting it all 'right' in a maintainable matter.

Anyways, back to the topic at hand. We now have the classes we need to build up our metadata model of all of the tables we need to provide data access functionality for. Well, we still need something to hold all of this information and to actually build up this model:

    public class MetaDataStore
    {
        private readonly Dictionary<Type, TableInfo> typeToTableInfo = new Dictionary<Type, TableInfo>();
 
        public TableInfo GetTableInfoFor<TEntity>()
        {
            return GetTableInfoFor(typeof(TEntity));
        }
 
        public TableInfo GetTableInfoFor(Type entityType)
        {
            if (!typeToTableInfo.ContainsKey(entityType))
            {
                return null;
            }
 
            return typeToTableInfo[entityType];
        }
 
        public void BuildMetaDataFor(Assembly assembly)
        {
            BuildMapOfEntityTypesWithTheirTableInfo(assembly);
 
            foreach (KeyValuePair<Type, TableInfo> pair in typeToTableInfo)
            {
                // we need this info for each entity before we can deal with references to other entities
                LoopThroughPropertiesWith<PrimaryKeyAttribute>(pair.Key, pair.Value, SetPrimaryKeyInfo);
            }
 
            foreach (KeyValuePair<Type, TableInfo> pair in typeToTableInfo)
            {
                LoopThroughPropertiesWith<ReferenceAttribute>(pair.Key, pair.Value, AddReferenceInfo);
                LoopThroughPropertiesWith<ColumnAttribute>(pair.Key, pair.Value, AddColumnInfo);
            }
        }
 
        private void BuildMapOfEntityTypesWithTheirTableInfo(Assembly assembly)
        {
            foreach (var type in assembly.GetTypes())
            {
                var typeAttributes = Attribute.GetCustomAttributes(type, typeof(TableAttribute));
 
                if (typeAttributes.Length > 0)
                {
                    var tableAttribute = (TableAttribute)typeAttributes[0];
                    var tableInfo = new TableInfo(this, tableAttribute.TableName, type);
                    typeToTableInfo.Add(type, tableInfo);
                }
            }
        }
 
        private void LoopThroughPropertiesWith<TAttribute>(Type entityType, TableInfo tableInfo,
            Action<TableInfo, PropertyInfo, TAttribute> andExecuteFollowingCode)
            where TAttribute : Attribute
        {
            foreach (var propertyInfo in entityType.GetProperties())
            {
                var attribute = GetAttribute<TAttribute>(propertyInfo);
 
                if (attribute != null)
                {
                    andExecuteFollowingCode(tableInfo, propertyInfo, attribute);
                }
            }
        }
 
        private void SetPrimaryKeyInfo(TableInfo tableInfo, PropertyInfo propertyInfo, PrimaryKeyAttribute primaryKeyAttribute)
        {
            tableInfo.PrimaryKey = new ColumnInfo(this, primaryKeyAttribute.ColumnName, propertyInfo.PropertyType, propertyInfo);
        }
 
        private void AddColumnInfo(TableInfo tableInfo, PropertyInfo propertyInfo, ColumnAttribute columnAttribute)
        {
            tableInfo.AddColumn(new ColumnInfo(this, columnAttribute.ColumnName, propertyInfo.PropertyType, propertyInfo));
        }
 
        private void AddReferenceInfo(TableInfo tableInfo, PropertyInfo propertyInfo, ReferenceAttribute referenceAttribute)
        {
            tableInfo.AddReference(new ReferenceInfo(this, referenceAttribute.ColumnName, propertyInfo.PropertyType, propertyInfo));
        }
 
        private TAttribute GetAttribute<TAttribute>(PropertyInfo propertyInfo) where TAttribute : Attribute
        {
            var attributes = Attribute.GetCustomAttributes(propertyInfo, typeof(TAttribute));
            if (attributes.Length == 0) return null;
            return (TAttribute)attributes[0];
        }
    }

This class gives you the ability to retrieve the TableInfo class for a specfic entity type. It also allows you to build the metadata model by passing in an assembly. It will then loop through all of the types in the assembly to discover the types that have a TableAttribute, and it will then build the TableInfo objects with all of the information we need.

And that's all we need to create mappings between tables and our entities. This wasn't hard, but it's not very powerful either. We can't define custom user types that our DAL needs to be able to deal with, nor can we define any database inheritance strategies. Our attributes are all inheritable, so you can use some inheritance with your entities, but you are essentially limited to the Table Per Class inheritance strategy. Implementing support for the other inheritance strategies would obviously introduce a lot more complexity in the whole mapping aspect.

In the next post, i'll show you how this DAL will use TableInfo's methods to create CRUD statements to offer out-of-the-box CRUD functionality for each mapped entity.

Build Your Own Data Access Layer Series

19 commentsWritten on August 23rd, 2009 by
Categories: Build Your Own DAL

I'm definitely not a fan of building your own Data Access Layer (DAL), since there are plenty of powerful and mature options already available. However, we have 2 customers at work who simply don't let us use any existing libraries/tools as a DAL and want us to just use straight ADO.NET. I don't want to get into their reasons for this, but the reality of the situation is that whenever we have to develop projects for them, we need to use a custom built DAL. I've never seen a custom built DAL that i found acceptable, let alone one that i actually wanted to use.

A lot of people typically go the code generation route when faced with this situation, which is exactly what we have done in the past. Been there, done that, hated it with a passion for various reasons. One of my coworkers recently started a new project for one of these customers, and he started implementing a new DAL. I had to review this, and while it had some good ideas there was a large amount of repetitive and error-prone code that still needed to be written by developers for every table. So i set out to come up with something better. If we did have to use a custom DAL for these customers, i wanted to make sure that it would at least avoid having us write repetitive, error-prone code for every table that we needed to use. Oh, and without having to resort to code generation. Since we are all NHibernate users (when customers don't have a problem with us using it, that is) i wanted something that was somewhat similar in ease-of-use though it could obviously never match its feature set, power and maturity.

I spent about 24 working hours (in total) on this, and i believe i came up with something that is acceptable for most simple forms-over-data applications. This DAL allows you to write your entity classes as POCO's, offers 'out-of-the-box' CRUD functionality for every mapped table, and has lazy loading for reference properties (so you don't need to pollute your entity classes with foreign key properties). There is also a simple session-level cache, and there is some functionality to ease the pain of using simple, custom queries (with that i mean: every query that is not a select all or select by id and that doesn't join other tables).

Compared to a real ORM, it is missing a lot: there is no Unit Of Work implementation, no automated change tracking of entities, no dirty checks, no collection support, no advanced querying possibilities, no statement batching, no serious caching functionality, no transitive persistence, and a whole host of features that something like NHibernate gives you for free. Each and every one of those features comes with a great cost of complexity and development time to get 'right' so it truly doesn't make a lot of sense to do all of this yourself.

In this series, we're going to go over the entire implementation of this DAL and throughout the series i will point out its shortcomings and try to explain the complexity that would be required to make it truly powerful. The purpose of this series is basically to:

  • Show you that you really don't need to resort to code generation to build your own custom DAL
  • Show you what kind of complexity is involved with the implementation of a good DAL
  • Convince you that you typically are better off with simply using something that is already available as a mature, powerful and proven solution

These are the posts that this series consists of:

  1. Mapping Classes To Tables
  2. Out Of The Box CRUD Functionality
  3. Hydrating Entities
  4. Session Level Cache
  5. Lazy Loading
  6. Executing Custom Queries
  7. Bringing It All Together
  8. Conclusions
  9. Enabling Bulk Inserts

I will update the list above with actual links to the posts as soon as they are published.

Note: the code of this series can be found here