Another thing is that your Cache isn’t checking for semantically equal objects.

put object A in cache with key “x” and timeout 10 seconds
wait 9 seconds,
add object B with key “x” and timout 10 seconds.
wait one second,
B gets removed from the cache by the timer event handler of the replaced object A because the timer didn’t get cancelled when A was evicted from the cache.

But I would expect B to be removed after 10 seconds have passed after insertion of B.

This also means that if any other test in a testrun also uses the key “test” while using the singleton cache , but with a much shorter timeout that falls within your testrun duration, you will see some very intermittant failures.

A fix would be to let the timeout handler check if the key is still associated with the same value for which the timeout was set, and to do nothing when the value was updated, or to cancel the timer when a CacheableObject is replaced or removed from the cache.

However, since you have a timer in your objects it would be best to add proper locking anyway.

Another very easy way to maybe check this is to add simple log statements in the CacheProvider that spits out the key name it’s looking for, the key name you added, etc. Then in the Elapsed_time event add a log statement that tells you that it’s removing said key. Match up your console output to what you expect, see if there is any differences.

One problem you could run into in a production system is that if the timer goes to Add said value the key might exist and cause a duplicate key exception or some such because your contains runs in a seperate call so by the time the contains returns true or false it could be invalid.

ie in your Put on CacheProvider it should look like this:

public void Put(CacheableObject value)
{
lock (monitor)
{
if (ContainsKey(value.Key))
{
Remove(value.Key);
}

value.Provider = this;
_cacheDictionary.Add(value.Key, value);
value.StartTicking();
}
}

This will allow you to create a single Atomic transaction. Also, on your remove you might want to add something similar that checks the contains key. As well on your Get command. You could have a problem that the Contains works fine and the get doesn’t.

To verify just add some logging like so to find out:

public CacheableObject Get(object key)
{
Console.WriteLine(“Getting data for key: ” + key.ToString());
if (_cacheDictionary.ContainsKey(key))
{
Console.WriteLine(“Found Key: ” + key.ToString());
return _cacheDictionary[key];
}
return null;
}

Hope that might help some.

but there is indeed one test that stores 2 objects immediately after each other with the same key…

Tapio

Here’s one (hopefully) last theory.

Because the remove does not stop the timer, the timer.Elapsed event will always fire once for each cached object. You probably have tests that cache more than 1 object or tests that verify that the put replaces an existing object in the cache. This will result in the situation where multiple timer.elapsed events will fire very close to each other. Each event is handled by a separate thread. Because the CacheProvider and the Dictionary classes are not thread-safe those events might corrupt the internal state of cache/dictionary.

Tapio

Tapio

What testrunner do you use?

If you run the tests directly from visual studio, there will be a different singleton instance for each run. But if you use for example nunit.exe as your testrunner every testrun will share the same instance of your singleton instance. Therefore the timer_elapsed events from an earlier testrun will interfere with your current testrun if the time between your testruns is about 60 secs.

Tapio

your summary is correct, but the timer doesn’t expire for 60 seconds, and this particular TestFixture never takes that long to run. The CacheProvider is also not used anywhere else.

I think it would’ve been better if the CacheProvider was less permissive when it comes to adding items with keys that already exist. But i’m not convinced that the actual cache implementation (or that of the CacheableObject) has a bug…

Is the following summary correct?

An object ‘A’ instance with the key “test” is inserted in the cache and its timer starts to tick. Another object ‘B’ with the same key “test” replaces ‘A’ in the dictionary. ‘A’ is now dereferenced and a candidate for GC. However, in some test runs the timer for ‘A’ expires and calls Remove on the cache provider for key “test”, for which the value is object ‘B’.

Looks like a bug in the cache to me, not just a test-time issue.

that definitely sounds like the most plausible explanation but when one of these tests fails, it takes about 10ms to do so, while the other tests all complete almost instantly. 6 of the 7 tests use a time to live of 60 seconds, while the other test uses a time to live of 500ms. But that test waits for 1000ms and then verifies that the object removed itself from the cache.

If the test with the short time to live would cause the problem, it should fail as well as any other test whose state got corrupted by it. But it’s always only one failing test.

@Andy

no, this is with NUnit on a buildserver.

There’s a race condition. Yes, it’s multithreaded code. There’s a Timer. Yes, it’s unlikely that it will happen. If that’s what’s causing the problem you probably wouldn’t see it often. Maybe once in a blue moon something unexpected causes the tests to take longer to run*. If it’s not the cause of the problem, the race in the test is still a bug IMO. But I’m an SDET – people pay me to say things like that.

The Elapsed event changes the state of that key in the CacheProvider singleton. Presumably one that other tests are using. It could be that one test’s timer is messing up another test. Possibly, though, it’s something else slowing the main thread while the timer ticks away.

Clearing the timer or somesuch is probably one solution. Not using the singleton might be a good idea, too.

* True story: There was a race in one of my tests once that I couldn’t reproduce, but another tester could repro some of the time. It turned out that sometimes (I don’t remember why) that tester would mouse-click on the cmd console running the test so the main thread of the test wasn’t running but the worker it spawned was. Hit any key in the console and the test continued, determined that its activity took too long, and the test case failed.

1) I have a feeling that test runners in VS re-use app domains/processes which may lead to your singleton-based cache being ‘dirty’. Perhaps a Flush() method that empties the dictionary and stops all the timers of cached objects?
2) Try replacing “test” with new object() to avoid object.GetHashCode() collisions.

In theory….

When you remove an object from the cache, you should also clear the provider and the timer of that object. The object is still alive (until the next GC) and the time.elapsed could cause the object to remove an other object from the cache b/c they share the same key.

Tapio Kulmala