Resharper 5.1’s Test Runner Is Broken
Posted by Davy Brion on July 13th, 2010
I used to be very happy with pretty much everything JetBrains released, but lately they seem to be having some serious quality issues with their Resharper product. The 5.0 release was pretty much unusable for us due to a stupid bug in their Silverlight support. It was fixed in the 5.1 EAP builds, but the newly released 5.1 version has another major (IMO) problem when it comes to running tests:
It no longer runs inherited tests. Which is especially annoying to us because we use inherited tests to make sure that our NHibernate CRUD operations work. We also use the inherited tests pattern for some other things, but that means we’re pretty much screwed now.
The bug has already been reported but since they closed the Early Access Program due to the release of 5.1, i guess we’re gonna have to wait quite a while before we can get our hands on a version that has a usable test runner.
If anyone from the Resharper team reads this: please tell me how on earth something like this could break in a bugfix release? You sell software for continuous integration, for code coverage, for running tests. If you’d actually use these things and wrote correct tests then this wouldn’t have happened…
July 13th, 2010 at 9:20 am
You could actually use testdriven.net.
I use it every day, and can’t be without it. I hate the UI testrunners like resharper and mstest. It gets in my way.
July 13th, 2010 at 9:24 am
@Henning
i know, but then we have to buy licenses for that while we already have licenses for something that _should_ work as well
testdriven.net is nice, it runs tests faster than resharper does, but i very much prefer the way Resharper deals with failed tests… it’s much nicer to deal with the output of a failed test through Resharper’s test runner than it is with testdriven.net IMO
July 13th, 2010 at 9:50 am
Yeah, I too find TestDriven.NET too intrusive in my development flow.
Other options to consider for running tests:
– NUnit Gui Runner, it was awful when i’ve looked at it last time.
– Gallio.org – seems to be dead
– Console runner
Anything else?
July 13th, 2010 at 10:08 am
Yeah, testdriven.net isn’t all that great when you get some weird failures. It can be difficult to see what is the cause and where the offending code is, but all in all I prefer td.net.
We all have our choices though
As for other UI runners, I don’t know of any else.
July 13th, 2010 at 11:16 am
@Davy
I’m looking into why this has slipped through if it’s the case. Also, apart from blogging, please feel free to contact me personally either via Twitter (@hhariri) or Email.
Unfortunately we cannot subscribe or read all blogs daily to find issues posted by our users (this is not the case but in many other instances it is).
Thanks!
July 13th, 2010 at 11:36 am
@Hadi Hariri
It’s in jetbrains’s YouTrack, reported yesterday.
Blogging/twittering works well when you want to bring more attention to some issues and may be get it fixed faster
July 13th, 2010 at 11:47 am
@Hadi
“Unfortunately we cannot subscribe or read all blogs daily to find issues posted by our users”
no but you should keep an eye on your own issue tracking system and then you would’ve known about this issue sooner
@Valeriu
“Blogging/twittering works well when you want to bring more attention to some issues and may be get it fixed faster”
that would be a nice side effect, but the main reason why i posted this was to express my frustration with the quality of the 5.x series of Resharper and more importantly, the point i’m trying to make in the last paragraph of the post
July 13th, 2010 at 11:51 am
@Davy,
Per my previous comment, this is not the case, but other instances there are where people only blog about issues and not report them.
Regarding your last line, I’d comment that it’s better to find out all the facts before passing judgement.
Thanks.
July 13th, 2010 at 11:53 am
@Hadi
the most important fact is that it used to work, and now it doesn’t… there are plenty of practices and tools available to prevent such things from happening
July 13th, 2010 at 12:12 pm
For anyone who stumbles across this, i’ve found R# EAP build 5.1.1713.5 works perfectly fine, assuming you can still get it.
July 13th, 2010 at 12:18 pm
@Xerxes
and that will stop working after a while
July 13th, 2010 at 12:29 pm
@Davy,
I understand your frustration, and apologize for the issue it’s causing you, but please also know that we’re very well aware of the tools and practices that not only do we provide, but apply ourselves. As a software developer that you are yourself, even with a very high percentage of code coverage (and we’re assuming correctly written tests), you know that things can still happen. Therefore I don’t think it’s as simple as assuming that we are not aware of our own tools or not applying them. Rest assured that we do, far beyond you could imagine.
Once again, I do apologize for the issue and I’ll do my best to try and find out what might have happened.
Thanks.
July 13th, 2010 at 12:37 pm
@Hadi
code coverage is pretty much meaningless:
http://davybrion.com/blog/2009/10/theres-only-one-thing-you-can-learn-from-code-coverage/
And yes, it is possible that some bugs can still happen when you’re using the right tools and following the right practices. But this is a pretty major regression IMO… and something that orginally should’ve been covered with correct tests. The fact that it no longer works means that it simply isn’t covered with correct tests.
As for what actually happened, that would be interesting to know but what would be far more helpful is to get an idea on how long we’re gonna have to deal with this until we get a version where this just works… like it used to.
July 13th, 2010 at 1:26 pm
@Davy,
Actually I don’t agree that code coverage is meaningless. I think exclusively focusing on the percentage of coverage is wrong, but coverage reports can point to potential issues in your code. And once again, as with my previous comments, we are assuming correct tests.
And since youtrack is currently down due to a blackout and not being able to investigate further since I don’t have latest sources I can’t assume that a specific test was or was not in place.
July 14th, 2010 at 7:09 am
Same topic http://youtrack.jetbrains.net/issue/RSRP-185948
already assigned to Victor Kropp
July 14th, 2010 at 7:58 am
these bugs just marked as critical!
victory!!! ))
July 14th, 2010 at 8:16 am
Here is what happened. During EAP program we were running with NUnit 2.5.5 integrated. But right before release we found critical issue in NUnit – it was running certain tests twice, in the cases involving inherited test fixtures. So we reverted NUnit to 2.5.3, then to 2.5.2 in order to fix this problem. Unfortunately, we didn’t had the other case in our regression testing, and anyway, we can’t do anything with it. The bug is in NUnit, not ReSharper. You can download NUnit 2.5.5 and install it, then tell ReSharper to run tests using installed version: Options / Tools / Unit Testing, select NUnit, specify different installation.
July 14th, 2010 at 8:21 am
“The bug is in NUnit, not ReSharper”
if it worked with Resharper 5.0, and it doesn’t work with Resharper 5.1 while i didn’t change any NUnit assembly, then clearly, the problem is with Resharper
it’s good to know there’s a workaround, but again, the workaround shouldn’t even be necessary since it worked fine with resharper 5.0
July 14th, 2010 at 8:31 am
actually the workaround doesn’t work… i installed nunit 2.5.5, set the ‘Specified NUnit installation’ field to ‘C:\Program Files\NUnit 2.5.5\bin\net-2.0′ (where it’s installed) and i just get the following exception when i try to run the tests:
Unit Test Runner failed to load test assembly:
JetBrains.ReSharper.TaskRunnerFramework.TaskException: Value cannot be null.
Parameter name: path1
July 14th, 2010 at 10:00 am
Nunit 2.5.5 runs tests from base class, Resharper not
July 14th, 2010 at 10:04 am
another sad thing is that RSRP-185948 was created 5 days ago, with no visible attention paid to it before it was marked as critical by Hadi Hariri a couple of hours ago
July 14th, 2010 at 10:33 am
@Alexey, yes, NUnit 2.5.5 runs tests from base class, but ReSharper is bundled with NUnit 2.5.2. If you install NUnit 2.5.5 and customize NUnit runner as described above ReSharper will use it run your tests.
July 14th, 2010 at 10:34 am
@Ilya
unless R# can’t start the custom version as i mentioned above…
July 14th, 2010 at 10:35 am
@Davy, ReSharper 5.0 was bundled with NUnit 2.5.3, but we had to revert to even earlier version due to it crashing in certain other cases. We are sorry for the confusion created. As for “visible attention”, we were investigating what is happening before “opening” the issue.
July 14th, 2010 at 1:57 pm
@Davy, just verified it again, downloaded 2.5.5 zip file, upacked into separate folder, set custom path to C:\Tools\NUnit\bin\net-2.0 and it works just fine. Also, we don’t have any unit-testing related “path1″ parameter. It looks like either NUnit has another bug, or your testing cases somehow causing exception during setup process.. Can you reproduce it on a separate sample project with simplest possible test fixtures?
July 14th, 2010 at 2:12 pm
@Ilya
it works on a separate sample project with the simplest possible test fixture
unfortunately that doesn’t help me at all since i want it all to work on the projects where it used to work and no longer doesn’t
July 14th, 2010 at 2:18 pm
@Ilya
btw, the fact that you guys don’t have any “path1″ parameter doesn’t mean you can’t look for the reason of the exception… it is throwing a JetBrains.ReSharper.TaskRunnerFramework.TaskException so it is coming from your software _somewhere_
regardless of whether this is an NUnit bug or not, this entire experience (the test runner’s problems in general) is really hurting the confidence that i used to have in you guys and i’m betting i’m not the only one
July 14th, 2010 at 2:56 pm
@Ilya
Suggested workaround worked for me…
I’ve posted this solution also in your bug tracker, so that people that is coming and voting for the issue can continue use Test Runner.
July 14th, 2010 at 3:11 pm
@Davy, TaskException is just a wrapper for real exception occurred in remote AppDomain (in separate process), so that it can be serialized and delivered to Visual Studio. I agree that diagnostics could be a little bit better. I’m here to try to help you have a working environment while we are investigating things on our side. If you can find NUnit 2.5.3 (shipped with ReSharper 5), you could try that and see if it works. If it does, it could give us more info in our work with NUnit guys on solving this issue. Thanks for being with us
July 14th, 2010 at 3:12 pm
@Valeriu great to see it helped you, thanks for posting info in YouTrack!
July 14th, 2010 at 3:15 pm
@Ilya
i get the same exception when trying this with NUnit 2.5.3 :s
July 14th, 2010 at 3:40 pm
@Davy
I’m not sure it’ll make a difference, but I’ve got NUnit by unzipping in my “Tools” folder. Same as it was suggested by Ilya. And you were using the installer and putting it to “Program Files”…
July 14th, 2010 at 3:43 pm
@Valeriu
tried both, same result :s
July 15th, 2010 at 7:16 am
@Davy, could you please try using NUnit GUI of the respective version to run your tests? Does it work without any problem?
July 15th, 2010 at 8:15 am
@Ilya
yeah, that works as expected
July 15th, 2010 at 12:39 pm
Specified Nunit installation does not work for me.
Failed to load test assembly…
VS2010, Nunit 2.5.5 (with alone 2.5.2 same error)
July 18th, 2010 at 12:10 am
For what it’s worth, the NUnit 2.5.5 workaround worked for me. Thanks Davy and Ilya!
July 19th, 2010 at 3:53 am
@Davy
Things worked fine for me when I was using NUnit 2.5.5 with the tests written into the super class. But as soon as I unchecked Tools -> Unit Testing -> Shadow-copy assemblies being tested, the error with path1 started happening.
Let me know if you guys are seeing the same thing/have a fix.
July 19th, 2010 at 8:27 am
@Bill
Yes I checked “Shadow-copy” and Nunit 2.5.5 started
July 19th, 2010 at 8:31 am
@Bill
you’re right… enabling shadow-copying makes it work
i usually turn it off though since we sometimes get bad results because of it
July 20th, 2010 at 8:55 am
for future reference, the path1 thing was indeed a resharper bug and has been fixed:
http://twitter.com/resharper/status/18943045975
July 28th, 2010 at 8:46 am
For the record, I think you were being a tad harsh on Jetbrains on this.
I’m actually even more impressed with them as a result of this. Keep up the great work and support guys!
July 28th, 2010 at 9:21 am
@Martin
if their last 2 releases had major issues for me, both of which i believe should never have made it into a released version then i have every right to react the way i do
it’s not because they’re JetBrains that they can get a free pass