Protecting Your Application From Remote Problems
Posted by Davy Brion on July 5th, 2009
If you have a web application which communicates with a remote service, it’s important to protect that web application from any problems the remote service might be dealing with. For instance, if the remote service goes down (for whatever reason) you really don’t want your application to keep making calls to this service. These failing calls increase the load on the service, which is already having problems, and will also block your threads which takes away resources from your application to deal with other requests. One pattern which is very suitable to reduce the problems for this situation is the Circuit Breaker (read that unless you’re familiar with the circuit breaker).
The biggest issue i have with my previous implementation is that it required you to call it manually to protect potentially risky calls. I don’t like having to call my circuit breaker whenever i want to make a service call because as a consumer of a service proxy, i shouldn’t even know about the circuit breaker. I also don’t want any coupling between my service proxy and the actual circuit breaker. Sounds like a good candidate for some AOP magic, right?
We’re going to use Castle Windsor’s Interceptors to make this work. First, the implementation of the CircuitBreaker class:
public class CircuitBreaker : IInterceptor
{
private readonly object monitor = new object();
private CircuitBreakerState state;
private int failures;
private int threshold;
private TimeSpan timeout;
public CircuitBreaker(int threshold, TimeSpan timeout)
{
this.threshold = threshold;
this.timeout = timeout;
MoveToClosedState();
}
public void Intercept(IInvocation invocation)
{
using (TimedLock.Lock(monitor))
{
state.ProtectedCodeIsAboutToBeCalled();
}
try
{
invocation.Proceed();
}
catch (Exception e)
{
using (TimedLock.Lock(monitor))
{
failures++;
state.ActUponException(e);
}
throw;
}
using (TimedLock.Lock(monitor))
{
state.ProtectedCodeHasBeenCalled();
}
}
private void MoveToClosedState()
{
state = new ClosedState(this);
}
private void MoveToOpenState()
{
state = new OpenState(this);
}
private void MoveToHalfOpenState()
{
state = new HalfOpenState(this);
}
private void ResetFailureCount()
{
failures = 0;
}
private bool ThresholdReached()
{
return failures >= threshold;
}
private abstract class CircuitBreakerState
{
protected readonly CircuitBreaker circuitBreaker;
protected CircuitBreakerState(CircuitBreaker circuitBreaker)
{
this.circuitBreaker = circuitBreaker;
}
public virtual void ProtectedCodeIsAboutToBeCalled() { }
public virtual void ProtectedCodeHasBeenCalled() { }
public virtual void ActUponException(Exception e) { }
}
private class ClosedState : CircuitBreakerState
{
public ClosedState(CircuitBreaker circuitBreaker)
: base(circuitBreaker)
{
circuitBreaker.ResetFailureCount();
}
public override void ActUponException(Exception e)
{
if (circuitBreaker.ThresholdReached()) circuitBreaker.MoveToOpenState();
}
}
private class OpenState : CircuitBreakerState
{
private readonly Timer timer;
public OpenState(CircuitBreaker circuitBreaker)
: base(circuitBreaker)
{
timer = new Timer(circuitBreaker.timeout.TotalMilliseconds);
timer.Elapsed += TimeoutHasBeenReached;
timer.AutoReset = false;
timer.Start();
}
private void TimeoutHasBeenReached(object sender, ElapsedEventArgs e)
{
circuitBreaker.MoveToHalfOpenState();
}
public override void ProtectedCodeIsAboutToBeCalled()
{
throw new OpenCircuitException();
}
}
private class HalfOpenState : CircuitBreakerState
{
public HalfOpenState(CircuitBreaker circuitBreaker) : base(circuitBreaker) { }
public override void ActUponException(Exception e)
{
circuitBreaker.MoveToOpenState();
}
public override void ProtectedCodeHasBeenCalled()
{
circuitBreaker.MoveToClosedState();
}
}
}
Notice how the CircuitBreaker class implements Windsor’s IInterceptor interface. The Intercept method will be called by Windsor whenever we try to call a method from a protected component. Within the Intercept method we can add the necessary logic to apply the Circuit Breaker pattern to the code that was originally called.
Now we just need to configure the Windsor IOC container to apply this bit of AOP magic for us.
First, we register the CircuitBreaker with the container:
container.Register(Component.For<CircuitBreaker>().LifeStyle.Singleton
.Named("serviceProxyCircuitBreaker")
.DependsOn(new Hashtable { { "threshold", 5 }, { "timeout", TimeSpan.FromMinutes(5) } }));
Notice that we register the CircuitBreaker implementation with a Singleton lifestyle, a custom name and the required constructor parameters to create an instance of the CircuitBreaker.
Then we register our service proxy:
container.Register(Component.For<IServiceProxy>().ImplementedBy<ServiceProxy>().LifeStyle.Transient
.Interceptors(InterceptorReference.ForKey("serviceProxyCircuitBreaker")).Anywhere);
Notice how we registered the service proxy as a transient component, while referencing the singleton CircuitBreaker interceptor. This means that each resolved instance of our service proxy will be protected by the same CircuitBreaker instance. If you have multiple services that you want to protect, simply register multiple CircuitBreakers with different keys and link each service you want to protect with the correct CircuitBreaker key.
July 6th, 2009 at 9:30 am
[...] Protecting Your Application From Remote Problems – Davy Brion talks about the importance of insulating your applications from problems in the remote systems they call, and looks at the concept of the Circuit Breaker. [...]
July 6th, 2009 at 11:51 am
Oh, so powerful!
Why Microsoft don’t give us some functions like this?
July 6th, 2009 at 3:02 pm
[...] Protecting Your Application From Remote Problems – Davy Brion ‘ If you have a web application which communicates with a remote service, it’s important to protect that web application from any problems the remote service might be dealing with ‘ [...]
July 7th, 2009 at 1:07 pm
Nice Davy. Some previous implementations:
http://blog.yoot.be/index.php/2008/06/12/circuitbreaker-interceptor/
http://timross.wordpress.com/2008/02/10/implementing-the-circuit-breaker-pattern-in-c/
http://timross.wordpress.com/2008/02/17/implementing-the-circuit-breaker-pattern-in-c-part-2/