Archive for July, 2009

One Of My Favorite NHibernate 2.1 Features

4 commentsWritten on July 30th, 2009 by
Categories: NHibernate

Check out the following piece of code:


            Session.CreateQuery(
                @"delete from DocumentTypeAssignment 
                  where Id.DmsDocument in (from DmsDocument where Id = :documentId) and 
                        Id.DocumentType.Id not in (:newDocumentTypeIds)")
                .SetInt64("documentId", dmsDocumentId)
                .SetParameterList("newDocumentTypeIds", newDocumentTypesToAssign.ToList(), NHibernateUtil.Int64)
                .ExecuteUpdate();

(pay no attention to Id.DmsDocument or Id.DocumentType... it's a composite key for a legacy table)

Which results in this SQL statement:


delete 
    from
        DocumentManagement.DocumentTypeAssignment 
    where
        (
            DmsDocumentID in (
                select
                    dmsdocumen1_.ID 
                from
                    DocumentManagement.DmsDocument dmsdocumen1_ 
                where
                    dmsdocumen1_.ID=@p0
            )
        ) 
        and (
            DocumentTypeID not in  (
                @p1 , @p2
            )
        );
    @p0 = 1634, @p1 = 2313, @p2 = 2310

Why I Dislike Classic Or Typical WCF Usage

25 commentsWritten on July 21st, 2009 by
Categories: Opinions, WCF

If you've ever used or read about WCF, you've most likely seen classic or typical WCF usage. With that i mean service contracts with various operations on them, service implementations that either contain logic or always delegate to other classes, generated client proxies that need to be kept in synch every time you add an operation to a service and a lot of repetitive XML in your configuration files. In some cases (usually for very specific services with limited functionality) there is nothing wrong with that. But for a service layer that sits on top of your business layer so it can be used by the front end of your application (be it ASP.NET, Silverlight, a WPF client or whatever) i find typical WCF usage to be far from ideal.

My biggest issue with typical WCF usage is that of service contract design. It's very hard to get this 'right' and most people simply don't. First of all, you need to decide between fine-grained and coarse-grained operations. Fine-grained operations typically lead to chatty communication between the client and the service, which could seriously impact performance and scalability. Then again, fine-grained operations do offer a lot of flexibility when it comes to reusing functionality in various parts in the client. Coarse-grained operations typically don't have the same negative effect on performance, but they can easily introduce implicit coupling between your service and the client which can get pretty annoying when you need to deal with a new kind of client, like a Silverlight application for instance (which tends to have different service usage patterns than typical ASP.NET applications). It can also lead to duplication when parts of functionality offered by a coarse-grained operation are needed somewhere else.

Even if you do get the granularity of your operations right, you have to figure out where to put them. Are you going to create a service for each kind of functional subdomain (billing, invoicing, registrations, etc...), for each entity that is 'known' to the client (there are many people who do this, unfortunately) or on a per-feature basis (story or use case driven)? Every possible variation you can think of is already being used today, and they almost always come with a lot of disadvantages. Get this part wrong and you can end up with clients that might need 2 or more service proxies just to show some related data on a screen (or worse, to complete a business transaction). Whether you get it right or wrong, odds are high that you'll have spent quite a bit of time defining your service contracts. Time that might have been better off being spent on other parts.

The second thing that bothers me a lot about typical WCF usage is the quality of the code in service implementations. In the worst cases, these classes actually contain true business logic to implement the service operations. That's not just breaking the Single Responsibility Principle, it's more like assaulting it. Whether they contain actual business logic or merely delegate to other classes, these service implementations will need to talk to instances of classes or components that they depend on sooner or later. My question is, where do these instances come from? Do you manually resolve them through your IOC container? Do you have them injected in your service instance (if creation of the service instance is controlled by an IOC container, that is)? What if the granularity of the service contract forces the service implementation to depend on components that might not be used together in all operations?

And then there's the thorny issue of cross cutting concerns within service implementations. How many times have you seen the same old tired logging and exception handling code repeated in every service operation? How about transaction handling, auditing, authorization or resource management? True, you can plug into WCF's extensibility model (if you're willing to look for it...) or use AOP tricks to minimize that kind of code duplication but that in turn limits your possibilities when you want to add just a little bit of custom code to some of the cross cutting concern's logic when needed. In most cases, service implementations simply contain a lot of smelly code, the large majority of which is essentially a form of waste.

That's already an important list of problems when designing and implementing typical WCF services. But what about using typical WCF services? This is somewhat problematic as well, IMHO. First of all, you'll need a proxy to access the service. You can either develop these proxies yourself, or you can generate them through visual studio, svcutil, or custom tools that you (or others) developed. Whatever way you choose to use, you can't escape the fact that you constantly have to keep those proxy implementations in synch with their respective service contract. Every single time one of the developers adds an operation to a service (which is pretty often once there are a few people working on a large project), you're going to have to do something to make sure you can access that operation through your proxy. It's usually not a lot of work, but it does kinda disrupt your coding flow. Over and over again. It can also be a pain in the ass when it comes to source control conflicts.

Then there's the actual matter of using the proxy instances. Can you implement your story or use case with just one client proxy type? Great! Do you reuse the channel when making multiple service calls? You do? Great! Do you make sure you handle faulted channels properly? And if you need multiple client proxy types (depending on your service contract granularity), are you aware of the fact that you are using multiple WCF channels (and yes, they are expensive) which are often kept open longer than they should be if you're not careful? Are you generally careful about chatty communication? Obviously, that last one depends on the contract of the service and not really the implementation of the client proxy. There are quite a lot of things you need to keep in mind when using these proxies.

My final thing on the list of 'dislikes' is the configuration that is required to use these services. For every service you expose you'll need to add some XML to your configuration file. Twice even (server and client)! Most of this XML configuration is extremely tedious and very repetitive which just invites minor mistakes to be made. Those minor mistakes could lead to a lot of debugging/problem solving time though.

I've been subjected to each and every one of the issues i've mentioned in projects that made typical use of WCF (or classic ASP.NET Web Services for that matter). This type of service layer implementation is, IMHO, hurtful for productivity, detrimental to code quality and i don't really like the implicit trade-off between reusability and performance/scalability (depending on the granularity of your operations).

About a year ago, i really wanted to figure out a way to implement services which would enable me to keep fine-grained operations on my service layer without having to pay the performance/scalability penalty for that. As some of you already know, that led to the whole WCF call batching thing (more info here and here) which i later on called the Request/Response Service Layer. Some people liked the approach, but i don't think anyone else actually uses it. At least, i've never heard of anyone using it in a real project.

Well obviously, we've been using this at work for about a year now and while i won't claim that it's perfect (hint: nothing is), it has worked very well for us in several projects. More specifically, this approach has offered us the following benefits, which heavily contrast all of the downsides of typical WCF usage that i listed above:

  • Since we only have one service contract with one service operation, we don't need to spend time thinking about how to design and implement our service contracts and our operations. After all, every operation that the service layer must support is a specific request type that can be added, together with its requesthandler.
  • We can keep our operations as fine-grained as we want (which increases reusability and overall flexibility), without having to pay the cost for chatty network communication by batching multiple requests per roundtrip as much as possible in a transparent manner.
  • The actual implementation of our service is very minimal. It's just a small class which resolves the appropriate requesthandler through the IOC container, based on the type of the incoming request. It then delegates to the requesthandler by passing the request to it, and it returns the response to the client. We can simply add 'operations' by adding request types and requesthandlers to our assemblies... everything gets registered automatically when the application starts up
  • We avoid repetitive code for cross cutting concerns by putting it in a base requesthandler class that the other ones inherit from. That kind of code now only occurs once, and we can plug in custom code at any point of the execution by simply using the Template Method pattern.
  • The implementation of our requesthandlers doesn't contain any code that doesn't have to be there. Each requesthandler simply implements the Handle method to handle the incoming request, and can do as it pleases to fulfill the request. All dependencies are injected automatically by the IOC container. It's usually nothing more than using the dependencies to execute the necessary business logic and then returning a response-derived object.
  • Since we only have one service, we only need one client proxy which never needs to be updated (technically, we have 2: one which is entirely asynch and mostly used in Silverlight clients, and one which is strictly synchronous and is mostly used by ASP.NET applications and Windows Services or command line tools.
  • This single client proxy implementation can make sure that underlying WCF resources are utilized as efficiently as possible and cleaned up properly throughout the client application(s).
  • The client proxy is easy to stub during unit tests which increases the testability of our client side code.
  • Very little configuration. We only have to configure one client-side endpoint, and one server-side endpoint. That's it.
  • All of this is very easy to put in some kind of reusable library. Our applications simply reference the library, inherit from the base requesthandler types, make sure everything is registered properly upon application startup, add a couple of lines of XML and we can start the development of our service layer without any friction.

The only downside to this approach that i can think of after using it for a year, is interoperability with other platforms. I haven't used it in that situation yet, and while i do think it can be made to work i don't think it'll be easy nor pretty.

Now, just to be clear: this entire post was not a rant against WCF. I actually do like WCF as a technology, it's just the typical usage patterns of WCF that i really dislike. I love the fact that WCF is very extensible, configurable and flexible, but i merely consider it as a communication technology. Nothing more, nothing less.

When You Absolutely Need To Use A Real Database In Tests

4 commentsWritten on July 20th, 2009 by
Categories: testing

The usage of a real database in tests is a topic that's been debated countless times before. There are probably just as many people who feel that you should always use a real database in tests as there are people who feel that you should avoid it at all costs. Some will use an in-memory database during tests, some will use stubbed data layers which use pure in-memory storage, some will mock the data layer.

No matter which way you prefer, i think there is at least one thing that you really have to test on the real database. With 'real' i obviously don't mean the actual database that your application will use in production, but it should be the exact same database (vendor and version). So what is it? Queries! Practically every query in your application deserves one or more tests. You should test that your query returns results that it should return, and that it doesn't return results that it shouldn't return. And this should preferably be done on the real database, even if you use an in-memory database for your other tests. Suppose you're using NHibernate to use SQLite during your tests... sure, you're testing against a database, but how valuable are those tests when your application will be using SQL Server or Oracle at runtime? Every generated query could potentially (and in many cases quite likely) be different on another database and you really ought to make sure you don't get bitten by that.

If you're using a database with a well known set of data during tests, then this is pretty easy. Just call the method which will execute the query (either through an ORM, or some code that calls a stored procedure, or some plain ADO.NET code with a sql string, whatever...) and verify that the result contains the expected data and that it doesn't contain data that it shouldn't have.

If you're using an empty database for each test, insert some records (and any records they need to exist) that should be retrieved by the query. Also insert some records that should not be retrieved by the query. Call the method which will execute the query and verify your expectations on the results. And don't forget to get rid of the data you just inserted, either by deleting it in a fool-proof manner or by rolling back the transaction.

This does take some effort, but it's usually well worth it. If you do this for most queries in your application, you'll (normally) end up with very good coverage (and i don't mean the code-coverage kind) of your datalayer, which in turn gives you more confidence to mock your datalayer in your other tests, if you'd want to do that.

Protecting Your Application From Remote Problems

10 commentsWritten on July 5th, 2009 by
Categories: Aspect Oriented Programming, Castle Windsor, Patterns

If you have a web application which communicates with a remote service, it's important to protect that web application from any problems the remote service might be dealing with. For instance, if the remote service goes down (for whatever reason) you really don't want your application to keep making calls to this service. These failing calls increase the load on the service, which is already having problems, and will also block your threads which takes away resources from your application to deal with other requests. One pattern which is very suitable to reduce the problems for this situation is the Circuit Breaker (read that unless you're familiar with the circuit breaker).

The biggest issue i have with my previous implementation is that it required you to call it manually to protect potentially risky calls. I don't like having to call my circuit breaker whenever i want to make a service call because as a consumer of a service proxy, i shouldn't even know about the circuit breaker. I also don't want any coupling between my service proxy and the actual circuit breaker. Sounds like a good candidate for some AOP magic, right?

We're going to use Castle Windsor's Interceptors to make this work. First, the implementation of the CircuitBreaker class:

    public class CircuitBreaker : IInterceptor
    {
        private readonly object monitor = new object();
        private CircuitBreakerState state;
        private int failures;
        private int threshold;
        private TimeSpan timeout;
 
        public CircuitBreaker(int threshold, TimeSpan timeout)
        {
            this.threshold = threshold;
            this.timeout = timeout;
            MoveToClosedState();
        }
 
        public void Intercept(IInvocation invocation)
        {
            using (TimedLock.Lock(monitor))
            {
                state.ProtectedCodeIsAboutToBeCalled();
            }
 
            try
            {
                invocation.Proceed();
            }
            catch (Exception e)
            {
                using (TimedLock.Lock(monitor))
                {
                    failures++;
                    state.ActUponException(e);
                }
                throw;
            }
 
            using (TimedLock.Lock(monitor))
            {
                state.ProtectedCodeHasBeenCalled();
            }
        }
 
        private void MoveToClosedState()
        {
            state = new ClosedState(this);
        }
 
        private void MoveToOpenState()
        {
            state = new OpenState(this);
        }
 
        private void MoveToHalfOpenState()
        {
            state = new HalfOpenState(this);
        }
 
        private void ResetFailureCount()
        {
            failures = 0;
        }
 
        private bool ThresholdReached()
        {
            return failures >= threshold;
        }
 
        private abstract class CircuitBreakerState
        {
            protected readonly CircuitBreaker circuitBreaker;
 
            protected CircuitBreakerState(CircuitBreaker circuitBreaker)
            {
                this.circuitBreaker = circuitBreaker;
            }
 
            public virtual void ProtectedCodeIsAboutToBeCalled() { }
            public virtual void ProtectedCodeHasBeenCalled() { }
            public virtual void ActUponException(Exception e) { }
        }
 
        private class ClosedState : CircuitBreakerState
        {
            public ClosedState(CircuitBreaker circuitBreaker)
                : base(circuitBreaker)
            {
                circuitBreaker.ResetFailureCount();
            }
 
            public override void ActUponException(Exception e)
            {
                if (circuitBreaker.ThresholdReached()) circuitBreaker.MoveToOpenState();
            }
        }
 
        private class OpenState : CircuitBreakerState
        {
            private readonly Timer timer;
 
            public OpenState(CircuitBreaker circuitBreaker)
                : base(circuitBreaker)
            {
                timer = new Timer(circuitBreaker.timeout.TotalMilliseconds);
                timer.Elapsed += TimeoutHasBeenReached;
                timer.AutoReset = false;
                timer.Start();
            }
 
            private void TimeoutHasBeenReached(object sender, ElapsedEventArgs e)
            {
                circuitBreaker.MoveToHalfOpenState();
            }
 
            public override void ProtectedCodeIsAboutToBeCalled()
            {
                throw new OpenCircuitException();
            }
        }
 
        private class HalfOpenState : CircuitBreakerState
        {
            public HalfOpenState(CircuitBreaker circuitBreaker) : base(circuitBreaker) { }
 
            public override void ActUponException(Exception e)
            {
                circuitBreaker.MoveToOpenState();
            }
 
            public override void ProtectedCodeHasBeenCalled()
            {
                circuitBreaker.MoveToClosedState();
            }
        }
    }

Notice how the CircuitBreaker class implements Windsor's IInterceptor interface. The Intercept method will be called by Windsor whenever we try to call a method from a protected component. Within the Intercept method we can add the necessary logic to apply the Circuit Breaker pattern to the code that was originally called.

Now we just need to configure the Windsor IOC container to apply this bit of AOP magic for us.

First, we register the CircuitBreaker with the container:

            container.Register(Component.For<CircuitBreaker>().LifeStyle.Singleton
                                   .Named("serviceProxyCircuitBreaker")
                                   .DependsOn(new Hashtable { { "threshold", 5 }, { "timeout", TimeSpan.FromMinutes(5) } }));

Notice that we register the CircuitBreaker implementation with a Singleton lifestyle, a custom name and the required constructor parameters to create an instance of the CircuitBreaker.

Then we register our service proxy:

            container.Register(Component.For<IServiceProxy>().ImplementedBy<ServiceProxy>().LifeStyle.Transient
                                .Interceptors(InterceptorReference.ForKey("serviceProxyCircuitBreaker")).Anywhere);

Notice how we registered the service proxy as a transient component, while referencing the singleton CircuitBreaker interceptor. This means that each resolved instance of our service proxy will be protected by the same CircuitBreaker instance. If you have multiple services that you want to protect, simply register multiple CircuitBreakers with different keys and link each service you want to protect with the correct CircuitBreaker key.

Save Some Time With The Right Testrunner

3 commentsWritten on July 1st, 2009 by
Categories: testing

If you're running your tests multiple times throughout the day, it's in your best interest to keep those testruns as fast as possible. We have a solution with 2 test projects. The first testproject contains 2700 regular tests that don't use any external resources so they should be very fast. The second testproject contains 1200 database tests (you do write tests for your queries, right?) which is a little bit slower, as is to be expected.

The problem is that with Resharper's testrunner, running the database tests actually takes less time than running the 'fast' tests. The 1200 database tests take 1m14 seconds to run. The 2700 in memory tests take 1m20 seconds to run... so that's almost 3 minutes of waiting time whenever you run the entire testsuite.

Just out of curiosity, i ran all of the tests with TestDriven.NET to see how much faster it would be... The 1200 database tests only took about 45 seconds, and the 2700 in memory tests only took 28 seconds to run. Running the entire testsuite with TestDriven.NET actually takes less than half the time it takes with Resharper's testrunner...

Meta

Recent Posts

Categories

Archives