MSDTC

MSDTC And Microsoft Security Essentials: A Killer Combo!

1 Comment »Written on June 21st, 2010 by
Categories: MSDTC

Just when i thought i had experienced more than a lifetime's worth of MSDTC issues, i ran into another one. All of a sudden, my MSDTC transactions started causing problems on my system again. Now, this happens occasionaly for reasons that i've been unable to figure out. I've learned to just accept it, reboot my system when it happens, and once it's back up it all just works again. So this time, i rebooted my system, waited for it come back up, tried running the same code that failed before and much to my surprise i got the same exception as i had gotten before:

System.Transactions.TransactionManagerCommunicationException: Communication with the underlying transaction manager has failed. ---> System.Runtime.InteropServices.COMException: The MSDTC transaction manager was unable to push the transaction to the destination transaction manager due to communication problems. Possible causes are: a firewall is present and it doesn't have an exception for the MSDTC process, the two machines cannot find each other by their NetBIOS names, or the support for network transactions is not enabled for one of the two transaction managers.

I checked my MSDTC security settings to make sure that everything was still configured correctly, and it was. I then checked to see if Windows Firewall is still disabled, and it was. Then i remembered that i've had problems with MSDTC transactions failing after Microsoft Security Essentials (MSE) had installed updates. Again, when that happens i typically just reboot and everything works again.  Just for the hell of it, i turned off MSE's real-time protection and tried again.  And it worked without problems. I turned the real-time protection back on to see if it would fail again but it kept working. I have no idea what the problem is, but at least i'm now armed with yet another thing to try when MSDTC stops working on my machine.  Gotta love options!

Seriously though, with all the problems i (and others) have run into with using MSDTC, i'm starting to wonder if Microsoft has some kind of covert Make-MSDTC-Usage-As-Painful-As-Possible mission going on somewhere...  Then again, maybe their QA is just getting even worse than it already was.  Oh well...

Avoiding Leaking Connections With NHibernate And TransactionScope

31 commentsWritten on May 6th, 2010 by
Categories: MSDTC, NHibernate

One of the applications we’re currently working on wasn’t responding anymore on the development server.  The logfile showed the following message:

System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.

We’ve been using NHibernate in almost all of our projects for about 2 years now and we’ve never had this issue before.  The thing is though, we never used System.Transactions’ TransactionScope class before and always used NHibernate’s transactions directly.  In this project, we’re using TransactionScope for the first time (mostly because we need distributed transactions occasionally) so i immediately suspected that we were either doing something wrong with it, or that there was a bug somewhere.

After a while i managed to reproduce the problem with the following simple test fixture:

    [TestFixture]

    public class ConnectionLeakTest

    {

        private static ISessionFactory sessionFactory;

 

        static ConnectionLeakTest()

        {

            sessionFactory = new Configuration()

                .Configure()

                .AddAssembly("MyAssembly")

                .BuildSessionFactory();

        }

 

        [Test]

        public void LeaksConnection()

        {

            var counter = 11;

            for (int i = 0; i < counter; i++)

            {

                using (var scope = new TransactionScope(TransactionScopeOption.Required))

                {

                    using (var session = sessionFactory.OpenSession())

                    {

                        var blah = session.CreateCriteria<User>().List<User>();

                    }

                }

            }

        }

    }

 

If you assign counter a value that is greater than your ‘Max Pool Size’ value of your connection string, then you will get the following exception once you’re going through the loop for the ‘Max Pool Size’ + 1 time:

System.InvalidOperationException : Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.

Bingo!

In normal situations, you would obviously call the Complete method of the TransactionScope instance before it is disposed to commit the current database transaction.  But in a situation where you can’t call the Complete method (as in: when an exception has occurred and you want to rollback the transaction), your database connection will actually leak until you’ve used up all of the connections in your connection pool. 

So the question is: How can we use a TransactionScope with NHibernate without leaking connections when we throw an exception (which should result in an automatic rollback of the current transaction)? 

I was always under the impression that merely opening an NHibernate session within a transaction scope was actually OK and that the session’s connection would make use of the current ambient transaction (the one created by the outer TransactionScope).  Pretty much every session-related action you perform in NHibernate results in calling the CheckAndUpdateSessionStatus method of the SessionImpl class, which in turn calls the EnlistInAmbientTransactionIfNeeded method.  That (as you can probably guess) will automatically enlist the current NHibernate session in the ambient transaction, which is the one that was created by the TransactionScope.  So, as i understand it, it should indeed be sufficient to open an NHibernate session within a TransactionScope to have every action performed by that session being enlisted in the current ambient transaction.   And it actually behaves as transactional as you’d expect it to be.  Your transaction is properly committed if you call the Complete method of the TransactionScope and it is indeed rolled back if you do not call the Complete method.  The only problem (and it’s a major one obviously) is that you’re connection will leak when the transaction is rolled back. 

Interestingly enough, if we change the test code to this:

        [Test]

        public void LeaksConnection()

        {

            var counter = 11;

            for (int i = 0; i < counter; i++)

            {

                using (var scope = new TransactionScope(TransactionScopeOption.Required))

                {

                    using (var session = sessionFactory.OpenSession())

                    using (var transaction = session.BeginTransaction())

                    {

                        var blah = session.CreateCriteria<User>().List<User>();

                        transaction.Rollback();

                    }

                }

            }

        }

 

Then the problem goes away.  Transactions are correctly rolled back, and none of the connections leak anymore.  It’s too bad that we still need to use NHibernate’s transactions only for the sake of being able to rollback in case of failure without leaking the connection, because again, when you’re not using an NHibernate transaction but are within a TransactionScope, your Session’s connection will indeed be enlisted in the transaction being governed by the TransactionScope.  So there certainly appears to be some kind of bug in NHibernate when it comes to cleaning up the connection of a session that has been enlisted in a TransactionScope’s transaction which is rolled back instead of committed.

Now, since i frequently see people on the intarweb showing examples of using an NHibernate session together with a TransactionScope yet without using an NHibernate transaction, i’d might as well post the way that i believe is the safest, and that shouldn’t leak any connections.

            using (var scope = new TransactionScope(TransactionScopeOption.Required))

            {

                using (var session = sessionFactory.OpenSession())

                using (var transaction = session.BeginTransaction())

                {

                    // do what you need to do with the session

                    transaction.Commit();

                }

                scope.Complete();

            }

 

If you need to rollback, simply throw an exception within the using block of the NHibernate transaction (before the transaction.Commit() call obviously) and everything takes care of itself.

MSDTC Woes With NServiceBus And NHibernate

19 commentsWritten on March 19th, 2010 by
Categories: .NET bugs, MSDTC, NHibernate, nservicebus

I’ve spent about 3 days trying to get something working that should’ve just worked.  I basically wanted some .NET code to use a distributed transaction to update some data in a database, and then publish a message on the service bus.  I want to do this in a distributed transaction because if something goes wrong, i want to roll back both transactions (the database change and the published message).  Normally, this should just work if you have MS DTC configured correctly.  On my machine, i enabled Network DTC Access, and allowed outbound transaction communication.  On the database server, Network DTC Access was already enabled and both outbound and inbound communication was allowed. 

Now the thing is, i’d either expect DTC to fail outright or to just work.  But it shouldn’t fail in one situation, and work in another.  On my machine, it failed in the following situation (which i’ll further refer to as Situation A):

  1. open a transaction scope
  2. open an nhibernate session
  3. hit the db
  4. publish a message through nservicebus
  5. close the nhibernate session
  6. complete and close the transaction scope

Step 4 and 5 could be switched around but it didn’t make a difference.  In Situation A, i always got a TransactionManagerCommunicationException with the following message:

Network access for Distributed Transaction Manager (MSDTC) has been disabled. Please enable DTC for network access in the security configuration for MSDTC using the Component Services Administrative tool.

Everyone who’s worked with MSDTC before probably knows that exception since it usually takes some fiddling with the settings to make things work.  The thing is, i was pretty sure that my settings, as well as the ones on the database server were correct.  Unfortunately, DTCPing didn’t confirm that since that too failed.

However, i also tried the following sequence of events (Situation B):

  1. open a transaction scope
  2. open an nhibernate session
  3. publish a message
  4. hit the db
  5. close the nhibernate session
  6. complete and close the transaction scope

And guess what.  That actually worked.  With full DTC transaction semantics.  The DTC statistics on the server confirmed that it was indeed using a DTC transaction, and if i made the code fail with an exception both the database action and the published message were correctly rolled back.

So the question is: why on earth does it only work when i publish a message before i hit the db?

During my investigation i noticed that in Situation A, the internal transaction that the transaction scope was using was a SqlDelegatedTransaction.  Which, if i’m not mistaken is an LTM transaction.  When trying to send a message to a message queue, the transaction manager tries to promote the current transaction to an OletxCommittableTransaction since the OleTx transaction protocol is required when using MSMQ (it doesn’t support LTM transactions).  For some reason, promoting the SqlDelegatedTransaction to a full DTC (OleTx) transaction fails on my machine.

In Situation B, the internal transaction is promoted to an OletxCommittableTransaction as soon as you try to send the message to a message queue.  Once it’s time to hit the DB, NHibernate nicely works together with the OletxCommittableTransaction and everything just works.

Now, i have no idea on earth why promotion of a SqlDelegatedTransaction fails, but after a long number of attempts and experiments to get it working correctly, i sorta gave up and figured i’d have to resort to a hack.  What i basically needed was for the transaction scope’s internal transaction to automatically be promoted to an OletxCommittableTransaction before i’d hit the database and without having to publish a dummy message at the beginning of the transaction.

I found one way of doing this which, while being a huge hack, is still relatively clean i think.  I wrote the following class:

    public class DummyEnlistmentNotification : IEnlistmentNotification

    {

        public static readonly Guid Id = new Guid("E2D35055-4187-4ff5-82A1-F1F161A008D0");

 

        public void Prepare(PreparingEnlistment preparingEnlistment)

        {

            preparingEnlistment.Prepared();

        }

 

        public void Commit(Enlistment enlistment)

        {

            enlistment.Done();

        }

 

        public void Rollback(Enlistment enlistment)

        {

            enlistment.Done();

        }

 

        public void InDoubt(Enlistment enlistment)

        {

            enlistment.Done();

        }

    }

 

Then, right after opening the transaction scope and before doing anything else, i do this:

    Transaction.Current.EnlistDurable(DummyEnlistmentNotification.Id, new DummyEnlistmentNotification(), EnlistmentOptions.None);

 

This basically tells the System.Transactions infrastructure that we’re adding our own Resource Manager to the current transaction.  And because it’s a durable Resource Manager, it now automatically promotes the internal transaction to an OletxCommittableTransaction and everything just works.  While our Resource Manager participates in the 2-phase-commit process, it doesn’t actually do anything.  It’s sole purpose is to force the creation of an OletxCommittableTransaction.

Like i said, it’s a hack but it’s still relatively clean.  I still have no idea why i needed to resort to this hack though… If anyone can shed some light on this, i’d highly appreciate it :)

Also, if you ever want to learn more about transactions in .NET or distributed transactions in particular, you really need to check out this article.  Without it, i probably wouldn’t have figured out what to do :)

Meta

Recent Posts

Categories

Archives