The Inquisitive Coder - Davy Brion's Blog » Multithreading

Using Copy-On-Write In Multithreaded Code To Reduce Locking Overhead

Davy Brion — Mon, 08 Mar 2010 15:36:35 +0000

I recently posted some code that i asked you to review. When i posted it, the code had never even executed (that’s right, not even through a test) and i only thought it would do what i needed it to do. I consider the actual implementation non-obvious (at least for those who don’t know the copy-on-write approach to avoid traditional locking) so i just wanted to hear some reactions to the code from people who didn’t knew the context. I promised to do a follow-up post to discuss the code in its entirety so here it is.

First, i’ll show the whole class again:

    public class TenantSessionFactoryManager : ITenantSessionFactoryManager
    {
        private readonly ITenantContext tenantContext;
        private readonly ITenantInfoHolder tenantInfoHolder;
        private readonly string mappingAssemblyName;
 
        private readonly object writeLock = new object();
        private Dictionary<Guid, ISessionFactory> sessionFactories;
 
        public TenantSessionFactoryManager(ITenantContext tenantContext, ITenantInfoHolder tenantInfoHolder, string mappingAssemblyName)
        {
            this.tenantContext = tenantContext;
            this.tenantInfoHolder = tenantInfoHolder;
            this.mappingAssemblyName = mappingAssemblyName;
            sessionFactories = new Dictionary<Guid, ISessionFactory>();
        }
 
        public ISession CreateSessionForCurrentTenant()
        {
            var tenantId = tenantContext.CurrentTenantId;
 
            if (!sessionFactories.ContainsKey(tenantId))
            {
                CreateSessionFactoryForCurrentTenant();
            }
 
            return sessionFactories[tenantId].OpenSession();
        }
 
        private void CreateSessionFactoryForCurrentTenant()
        {
            lock (writeLock)
            {
                var tenantId = tenantContext.CurrentTenantId;
 
                if (!sessionFactories.ContainsKey(tenantId))
                {
                    var connectionString = tenantInfoHolder.GetDatabaseConnectionString(tenantId);
 
                    var sessionFactory = new Configuration()
                        .Configure()
                        .AddProperties(new Dictionary<string, string>
                                {
                                    { "connection.connection_string", connectionString },
                                    { "cache.region_prefix", "Tenant_" + tenantId }
                                })
                        .AddAssembly(mappingAssemblyName)
                        .BuildSessionFactory();
 
                    var newDictionary = new Dictionary<Guid, ISessionFactory>(sessionFactories);
                    newDictionary[tenantId] = sessionFactory;
                    sessionFactories = newDictionary;
                }
            }
        }
 
        public void RemoveSessionFactoryForTenant(Guid tenantId)
        {
            if (!sessionFactories.ContainsKey(tenantId))
            {
                return;
            }
 
            lock (writeLock)
            {
                if (!sessionFactories.ContainsKey(tenantId))
                {
                    return;
                }
 
                var sessionFactory = sessionFactories[tenantId];
                var newDictionary = new Dictionary<Guid, ISessionFactory>(sessionFactories);
                newDictionary.Remove(tenantId);
                sessionFactories = newDictionary;
 
                sessionFactory.Dispose();
            }
        }
    }

Basically, the purpose of this class is to hold a set of ISessionFactory instances, each of which belongs to a particular tenant in a multi-tenant application. Tenants can be added on the fly (without restarting the application) and when an ISessionFactory doesn’t exist yet for a particular tenant, it must be created when the first request for an ISession for that tenant comes in. Obviously, access to the sessionFactories dictionary must be thread-safe since multiple threads will be reading from the dictionary as well as occasionally writing to it.

I considered 3 options to make sure access to the dictionary would be thread-safe:

Traditional locking (through the lock statement or the Monitor class)
Using the ReadWriterLockSlim class
Using the copy-on-write pattern

Traditional locking was quickly scratched from the list because that would require me to lock for every read of the dictionary as well as every write. Now, pretty much every single request requires an NHibernate session which means that pretty much every single request results in a lookup in the sessionFactories dictionary. If i need to lock for every read, this significantly hurts overall throughput of the system.

The ReadWriterLockSlim might be a good solution here… after all, the short description of this class in MSDN says this:

Represents a lock that is used to manage access to a resource, allowing multiple threads for reading or exclusive access for writing.

Sounds like what i need, right? But the thing is, i’ve never used the ReadWriterLockSlim class before and it hasn’t really gained my trust yet. I know that’s a terrible excuse for not using it, but here me out. While the ReadWriterLockSlim likely reduces locking overhead over traditional locking substantially, there still has to be some overhead for read operations, even if it is small. In most situations, that small overhead wouldn’t bother me but in this case, that little overhead would be added to pretty much every single request in the system. Now, writing to a dictionary implies that a new tenant has been added to the system. In the context of this system, that’s not even gonna happen on a daily basis. Hell, once a week is probably a best-case estimation and even that is highly optimistic. So i really don’t want any kind of overhead on read operations when the write operation is only going to happen very occasionally.

That leaves the copy-on-write pattern. I’ve used it before with success (though at the time, i didn’t know it was a known pattern) so this approach has already gained my trust. It basically implies that we don’t do any locking on the read operations, but whenever a write operation occurs we copy the original set of objects, perform the write on the newly copied set and then set the reference of the original set to the newly created and modified instance. During this whole time, every single read is safe. Successive reads within the same logical operation however aren’t, so the following code would not be thread-safe:

            if (sessionFactories.ContainsKey(tenantId))
            {
                return sessionFactories[tenantId].OpenSession();
            }

Because there’s no locking on the reads, the code within the if-block could fail because the sessionFactories reference could be pointing to a new dictionary which no longer contains the element for that key.

Of course, if you have frequent writes, the overhead of copying the set of objects every time you need to add/remove one might be bigger than you want, so this isn’t a pattern that you should use whenever you need to protect access to a shared resource. For this situation however, i think it’s ideal… though i’d obviously like to hear about better solutions

Now, let’s take a closer look at the pieces of code that perform the write operations. First, adding a new ISessionFactory to the dictionary:

        private void CreateSessionFactoryForCurrentTenant()
        {
            lock (writeLock)
            {
                var tenantId = tenantContext.CurrentTenantId;
 
                if (!sessionFactories.ContainsKey(tenantId))
                {
                    var connectionString = tenantInfoHolder.GetDatabaseConnectionString(tenantId);
 
                    var sessionFactory = new Configuration()
                        .Configure()
                        .AddProperties(new Dictionary<string, string>
                                {
                                    { "connection.connection_string", connectionString },
                                    { "cache.region_prefix", "Tenant_" + tenantId }
                                })
                        .AddAssembly(mappingAssemblyName)
                        .BuildSessionFactory();
 
                    var newDictionary = new Dictionary<Guid, ISessionFactory>(sessionFactories);
                    newDictionary[tenantId] = sessionFactory;
                    sessionFactories = newDictionary;
                }
            }
        }

As you can see, the entire operation is put between a lock on the writeLock object instance. The downside of this is that creating an ISessionFactory instance is an expensive operation, which means the lock will be held for a long time (could easily be one or more seconds). Then again, i don’t anticipate this happening frequently so it’s not that big of an issue… especially since reads aren’t being blocked by this anyway. This approach also prevents the creation of 2 ISessionFactory instances for the same tenant. Well, unless i missed a bug here :p

Now, once the ISessionFactory instance is created, we create a new Dictionary based on the contents of the old one and then we add the new ISessionFactory instance to it. After that, we replace the sessionFactories references with the new dictionary and from that point on, every read will use the new dictionary instance. During this entire operation, no read operation was impacted negatively.

Now lets take a look at the other write operation, removing an ISessionFactory instance from the dictionary:

        public void RemoveSessionFactoryForTenant(Guid tenantId)
        {
            if (!sessionFactories.ContainsKey(tenantId))
            {
                return;
            }
 
            lock (writeLock)
            {
                if (!sessionFactories.ContainsKey(tenantId))
                {
                    return;
                }
 
                var sessionFactory = sessionFactories[tenantId];
                var newDictionary = new Dictionary<Guid, ISessionFactory>(sessionFactories);
                newDictionary.Remove(tenantId);
                sessionFactories = newDictionary;
 
                sessionFactory.Dispose();
            }

The first if-check, which happens outside of the lock is a bug that i missed but that was pointed out in the comments of the original post. If CreateSessionFactoryForCurrentTenant and RemoveSessionFactoryForTenant would execute concurrently for the same tenant, it’s possible that the ISessionFactory instance of that tenant is never removed from the dictionary (and also never disposed of…) since the check happens outside of the lock and could be executed before the ISessionFactory of the tenant was added to the dictionary. In that case, the ISessionFactory instance would stay in the dictionary as long as the application stays up. This is definitely a race condition that you want to avoid in every other situation though in this case, the odds that we’re simultaneously adding and removing the same tenant are slim to none. Nevertheless, i don’t want to be accused of promoting race conditions so we’ll make the change anyway

        public void RemoveSessionFactoryForTenant(Guid tenantId)
        {
            lock (writeLock)
            {
                if (!sessionFactories.ContainsKey(tenantId))
                {
                    return;
                }
 
                var sessionFactory = sessionFactories[tenantId];
                var newDictionary = new Dictionary<Guid, ISessionFactory>(sessionFactories);
                newDictionary.Remove(tenantId);
                sessionFactories = newDictionary;
 
                sessionFactory.Dispose();
            }
        }

Now, as you can see we once again create a new dictionary based on the previous one, then remove the ISessionFactory instance for the current tenant and then we overwrite the sessionFactories instance once again.

Finally, there’s the read operation that i specifically didn’t want suffering from locking overhead:

        public ISession CreateSessionForCurrentTenant()
        {
            var tenantId = tenantContext.CurrentTenantId;
 
            if (!sessionFactories.ContainsKey(tenantId))
            {
                CreateSessionFactoryForCurrentTenant();
            }
 
            return sessionFactories[tenantId].OpenSession();
        }

The only time this code will block is when a new ISessionFactory for the current tenant needs to be created. Luckily, that only happens once for each tenant. As i mentioned earlier in the post, using this pattern doesn’t guarantee that successive reads within the same logical operation are thread safe, so there is a bug in here. If a tenant already has an ISessionFactory instance, it’s possible that the RemoveSessionFactoryForTenant method has been executed between the if-check and accessing the ISessionFactory based on the tenantId. In that particular scenario, the ISessionFactory instance is no longer in the dictionary which will cause this code to throw an exception.

That’s a bug that i don’t feel like fixing though… Once a tenant has been removed, they are no longer a paying customer. If they are no longer paying for the software, there is no reason whatsoever why i should care about any possible exceptions they could get while running the software

Seriously though, if the RemoveSessionFactoryForTenant method is called, users of that tenant won’t even have access to the system anymore so it’s really a non-issue.

Anyways, i think i’ve covered the implementation in more detail than you probably cared for. So, any thoughts? Are there still issues that i haven’t thought of? Is there another approach that you would use for this specific scenario?

The Multithreaded Task Executor

Davy Brion — Mon, 26 May 2008 15:01:51 +0000

Sometimes, you’ve got a bunch of actions that you need to execute in a loop. The problem is that those actions are all performed synchronously so this could take some time depending on the action. But, if the action itself is thread-safe, and the actions are not dependent on the results of previous actions, you might get much better performance if you spread that workload over a few different threads. Especially if you have multiple CPU cores.

Wouldn’t it be cool if you could do something like this:



            MultiThreadedTaskExecutor taskExecutor = new MultiThreadedTaskExecutor(numberOfThreadsToUse);
 
            foreach (Input input in inputs)
            {
                Input newVariable = input;
                taskExecutor.QueueTask(() => ProcessInput(newVariable));
            }
 
            taskExecutor.RunTasksAndWait();

Note: the reason why you need to use a new variable inside the loop is to avoid that the ‘input’ variable (not the reference to the object, but the actual variable) is captured by the anonymous method. If you don’t use a new variable, each created anonymous method would refer to the ‘input’ loop variable, which by the time all the tasks are executed points to the last Input instance in the inputs collection. The result would be that each task is executed on the same input instance. This is a known issue with variable capturing and anonymous methods.

Anyways… the code above basically spreads the workload over the given number of threads.

The rough, not-quite-production-ready code of the MultiThreadedTaskExecutor class looks like this:



    public class MultiThreadedTaskExecutor
    {
        private readonly List<Thread> threads = new List<Thread>();
        private readonly List<EventWaitHandle> eventWaitHandles = new List<EventWaitHandle>();
        private readonly List<Type> swallowedExceptionTypes;
 
        private readonly Queue<Action> taskQueue;
        private readonly object queueMonitor = new object();
 
        public MultiThreadedTaskExecutor(int numberOfThreads)
        {
            swallowedExceptionTypes = new List<Type>();
            taskQueue = new Queue<Action>();
            CreateThreads(numberOfThreads);
        }
 
        public void QueueTask(Action task)
        {
            taskQueue.Enqueue(task);
        }
 
        public void AddExceptionTypeToSwallow(Type type)
        {
            swallowedExceptionTypes.Add(type);
        }
 
        public void RunTasksAndWait()
        {
            foreach (Thread thread in threads)
            {
                thread.Start();
            }
 
            WaitHandle.WaitAll(eventWaitHandles.ToArray());
        }
 
        private void CreateThreads(int number)
        {
            for (int i = 0; i < number; i++)
            {
                var eventWaitHandle = new EventWaitHandle(false, EventResetMode.ManualReset);
                eventWaitHandles.Add(eventWaitHandle);
                threads.Add(new Thread(() => ProcessTasks(eventWaitHandle)));
            }
        }
 
        private void ProcessTasks(EventWaitHandle eventWaitHandle)
        {
            try
            {
                Action action;
 
                while ((action = GetTask()) != null)
                {
                    try
                    {
                        action();
                    }
                    catch (Exception e)
                    {
                        if (!swallowedExceptionTypes.Contains(e.GetType())) throw;
                    }
                }
            }
            finally
            {
                eventWaitHandle.Set();
            }
        }
 
        private Action GetTask()
        {
            lock (queueMonitor)
            {
                if (taskQueue.Count == 0) return null;
                return taskQueue.Dequeue();
            }
        }
    }

So how does it work? It uses a queue to hold each task that was added by the consumer of the class. It creates the given amount of threads and also creates an EventWaitHandle for each thread. Then when the user starts the execution with a call to RunTasksAndWait, each thread is started and then the call to RunTasksAndWait will wait until each thread is finished. In the meantime, each thread will get the next task off the queue and executes it. If an exception is thrown within the task, it is caught and is either swallowed or rethrown (the consumer can add exception types that can be swallowed). Each thread keeps doing this until it can’t get a new task off the queue. When that happens, the thread signals the EventWaitHandler and then it dies. When all threads are dead, RunTasksAndWait will stop blocking and control is returned to the caller. All of the tasks have been executed and the workload has been spread over the given amount of threads.

Note: due to the call to WaitHandle.WaitAll, this won’t work on STA threads because the implementation of WaitHandle.WaitAll simply throws a NotSupportedException on STA threads.

Keep in mind that this is a rough version of this code… it definitely needs a bit more polish (better exception handling for when the threads are interrupted and stuff like that mostly) but you get the idea

But if you know a better way to do this, or if you spot flaws in this implemenation, i’d love to hear about it

Easy non-blocking locking

Davy Brion — Tue, 13 May 2008 09:08:29 +0000

I’m currently reading Release It and in it, the author really stresses the importance of not having any blocking-calls in your code. The reason is fairly simple, if something goes wrong in a blocking call, the thread executing it might hang forever. If you’re servicing requests, you surely don’t want all of your request-handling threads to hang forever because before you know it, all of the threads will be blocked and no requests will be dealt with.

You may remember my post about thread-safe repositories from a while ago. In the code from that post, i use the lock keyword to make sure instances of the class are thread-safe. The problem with the lock keyword is that it uses the Monitor.Enter method, which blocks indefinitely until it can acquire a lock on the object you pass to it. And since you should avoid blocking calls, you really should use Monitor.TryEnter instead, because it allows you to set a timeout. If the lock can’t be acquired within the timeout period, the method simply returns false and you did not get a lock. This makes it possible to avoid blocking threads and deadlocks.

The problem is that the lock keyword makes everything pretty easy… it guarantees that the lock is released (by calling Monitor.Exit) when leaving the code block, whatever may have possible went wrong. So if you want to use Monitor.TryEnter, you basically have to use a try/finally everytime. Not only does that make the code more ugly, it’s so boring and tedious that it’s easy to screw up once in a while. So i started googling for a different approach, and luckily, Ian Griffith has two excellent posts on the subject. He basically uses a TimedLock struct in a using block… when you request the lock, you specify a timeout value, and when the lock can’t be acquired within the given timeout period, it throws a LockTimeOutException. Here’s a (simplified) version of the TimedLock struct:



    public struct TimedLock : IDisposable
    {
        private readonly object target;
 
        private TimedLock(object o)
        {
            target = o;
        }
 
        public void Dispose()
        {
            Monitor.Exit(target);
        }
 
        public static TimedLock Lock(object o)
        {
            return Lock(o, TimeSpan.FromSeconds(5));
        }
 
        public static TimedLock Lock(object o, TimeSpan timeout)
        {
            return Lock(o, timeout.Milliseconds);
        }
 
        public static TimedLock Lock(object o, int milliSeconds)
        {
            var timedLock = new TimedLock(o);
 
            if (!Monitor.TryEnter(o, milliSeconds))
            {
                throw new LockTimeoutException();
            }
 
            return timedLock;
        }
    }
 
    public class LockTimeoutException : ApplicationException
    {
        public LockTimeoutException() : base("Timeout waiting for lock") {}
    }

The version on his blog also has a way of detecting unreleased locks when you’re in debug mode.

So now you can basically change the following code:



        public virtual T Get(Id id)
        {
            lock(MonitorObject)
            {
                if (!Members.ContainsKey(id))
                {
                    return null;
                }
 
                return Members[id];
            }
        }

To this:



        public virtual T Get(Id id)
        {
            using (TimedLock.Lock(MonitorObject, 250))
            {
                if (!Members.ContainsKey(id))
                {
                    return null;
                }
 
                return Members[id];
            }
        }

So now it tries to acquire a lock on the MonitorObject instance with a timeout of 250 milliseconds. If it can’t acquire the lock, something is wrong and the LockTimeOutException is thrown. Which is a lot better than blocking indefinitely

Thread-safe repositories

Davy Brion — Tue, 25 Mar 2008 13:37:05 +0000

First of all, this is not about true DDD repositories… i just needed an in-memory data container with basic storing/retrieving functionality so i figured i’d call it a Repository.

These repositories will be accessed by multiple concurrent threads so i needed to make sure that all access to the underlying dictionary is properly synchronized. But, i also want them to be highly usable and i especially wanted a thread-safe way of dynamically executing queries on them.

Here’s what i came up with:

    public class Repository where T : Member
    {
        private readonly object _monitor = new object();
        private readonly Dictionary<Id, T> _members = new Dictionary<Id,T>();
 
        protected object Monitor
        {
            get { return _monitor; }
        }
 
        /// 
        /// any derived classes that use this property are responsible for their own locking!
        /// (use the Monitor property for that)
        /// 
        protected Dictionary<Id, T> Members
        {
            get { return _members; }
        }
 
        public virtual void Store(T member)
        {
            lock (Monitor)
            {
                Put(member);
            }
        }
 
        public virtual void StoreRange(IEnumerable members)
        {
            lock (Monitor)
            {
                foreach (T member in members)
                {
                    Put(member);
                }
            }
        }
 
        public virtual void Remove(T member)
        {
            lock (Monitor)
            {
                if (Members.ContainsValue(member))
                {
                    Members.Remove(member.Id);
                }
            }
        }
 
        public virtual T Get(Id id)
        {
            lock (Monitor)
            {
                if (!Members.ContainsKey(id))
                {
                    return null;
                }
 
                return Members[id];
            }
        }
 
        public virtual T FindFirst(Funcbool> expression)
        {
            lock (Monitor)
            {
                return Members.Values.FirstOrDefault(expression);
            }
        }
 
        public virtual IEnumerable FindAll(Funcbool> expression)
        {
            lock (Monitor)
            {
                return Members.Values.Where(expression).ExecuteImmediately();
            }
        }
 
        /// 
        /// this method should ONLY be called when a lock on Monitor has been acquired!
        /// 
        /// 
        private void Put(T member)
        {
            // this overwrites existing entries with the same ID… it’s not a mistake  
            Members[member.Id] = member;
        }
    }

each access to the dictionary is properly synchronized (at least, i think so… if anyone knows of a better way to do the locking, please leave a comment) and i can still execute dynamic queries on them by passing lambda expressions to the FindFirst and FindAll methods, like this:

            User me = userReposority.FindFirst(u => u.FirstName == “Davy” && u.LastName == “Brion”);
            IEnumerable<User> usersWithNoLastName = 
                userReposority.FindAll(u => string.IsNullOrEmpty(u.LastName));