Resharper 5.1’s Test Runner Is Broken

46 commentsWritten on July 13th, 2010 by
Categories: resharper

I used to be very happy with pretty much everything JetBrains released, but lately they seem to be having some serious quality issues with their Resharper product.  The 5.0 release was pretty much unusable for us due to a stupid bug in their Silverlight support.  It was fixed in the 5.1 EAP builds, but the newly released 5.1 version has another major (IMO) problem when it comes to running tests:

resharper_fail

It no longer runs inherited tests.  Which is especially annoying to us because we use inherited tests to make sure that our NHibernate CRUD operations work.  We also use the inherited tests pattern for some other things, but that means we’re pretty much screwed now.

The bug has already been reported but since they closed the Early Access Program due to the release of 5.1, i guess we’re gonna have to wait quite a while before we can get our hands on a version that has a usable test runner.

If anyone from the Resharper team reads this: please tell me how on earth something like this could break in a bugfix release?  You sell software for continuous integration, for code coverage, for running tests.  If you’d actually use these things and wrote correct tests then this wouldn’t have happened…

« Off For A Week500 Posts »

This entry was posted on Tuesday, July 13th, 2010 at 08:59 and is filed under resharper. You can subscribe to any comments to this entry via RSS 2.0. You can leave a response, or trackback from your own site.

  • Henning Anderssen

    You could actually use testdriven.net.

    I use it every day, and can’t be without it. I hate the UI testrunners like resharper and mstest. It gets in my way.

  • http://davybrion.com Davy Brion

    @Henning

    i know, but then we have to buy licenses for that while we already have licenses for something that _should_ work as well

    testdriven.net is nice, it runs tests faster than resharper does, but i very much prefer the way Resharper deals with failed tests… it’s much nicer to deal with the output of a failed test through Resharper’s test runner than it is with testdriven.net IMO

  • Valeriu Caraulean

    Yeah, I too find TestDriven.NET too intrusive in my development flow.

    Other options to consider for running tests:
    – NUnit Gui Runner, it was awful when i’ve looked at it last time.
    – Gallio.org – seems to be dead
    – Console runner

    Anything else?

  • Henning Anderssen

    Yeah, testdriven.net isn’t all that great when you get some weird failures. It can be difficult to see what is the cause and where the offending code is, but all in all I prefer td.net.

    We all have our choices though :)

    As for other UI runners, I don’t know of any else.

  • http://hadihariri.com Hadi Hariri

    @Davy

    I’m looking into why this has slipped through if it’s the case. Also, apart from blogging, please feel free to contact me personally either via Twitter (@hhariri) or Email.

    Unfortunately we cannot subscribe or read all blogs daily to find issues posted by our users (this is not the case but in many other instances it is).

    Thanks!

  • Valeriu Caraulean

    @Hadi Hariri
    It’s in jetbrains’s YouTrack, reported yesterday.

    Blogging/twittering works well when you want to bring more attention to some issues and may be get it fixed faster ;)

  • http://davybrion.com Davy Brion

    @Hadi

    “Unfortunately we cannot subscribe or read all blogs daily to find issues posted by our users”

    no but you should keep an eye on your own issue tracking system and then you would’ve known about this issue sooner

    @Valeriu

    “Blogging/twittering works well when you want to bring more attention to some issues and may be get it fixed faster”

    that would be a nice side effect, but the main reason why i posted this was to express my frustration with the quality of the 5.x series of Resharper and more importantly, the point i’m trying to make in the last paragraph of the post

  • http://hadihariri.com Hadi Hariri

    @Davy,

    Per my previous comment, this is not the case, but other instances there are where people only blog about issues and not report them.

    Regarding your last line, I’d comment that it’s better to find out all the facts before passing judgement.

    Thanks.

  • http://davybrion.com Davy Brion

    @Hadi

    the most important fact is that it used to work, and now it doesn’t… there are plenty of practices and tools available to prevent such things from happening

    • Chris Mccall

      Don’t you think Jetbrains has been punished enough for this bug? Have some courtesy, they are a very productive and reliable vendor, and it’s unlikely you would be as successful in your work without them. We get it: you found a bug and it’s stopping you from running a few of your tests. Relax and let the vendor fix it. Maybe a copy of the developer’s pink slip and a photo of a Christmas tree with no gifts under it will satisfy your thirst for revenge?

  • http://www.xerxesb.com Xerxes

    For anyone who stumbles across this, i’ve found R# EAP build 5.1.1713.5 works perfectly fine, assuming you can still get it.

  • http://davybrion.com Davy Brion

    @Xerxes

    and that will stop working after a while :)

  • http://hadihariri.com Hadi Hariri

    @Davy,

    I understand your frustration, and apologize for the issue it’s causing you, but please also know that we’re very well aware of the tools and practices that not only do we provide, but apply ourselves. As a software developer that you are yourself, even with a very high percentage of code coverage (and we’re assuming correctly written tests), you know that things can still happen. Therefore I don’t think it’s as simple as assuming that we are not aware of our own tools or not applying them. Rest assured that we do, far beyond you could imagine.

    Once again, I do apologize for the issue and I’ll do my best to try and find out what might have happened.

    Thanks.

    • https://www.google.com/accounts/o8/id?id=AItOawk7N2IVbxV-BXVNcYEtxd9dNIBuBukcPP0 SleeperSmith

      “Rest assured that we do, far beyond you could imagine.”

      And it’s exactly people like YOU that made bugs like this appear. As explained later:
      “Unfortunately, we didn’t had the other case in our regression testing” after you guys reverted back to NUnit 2.5.2.

      In plain English. YOU HAD THE TESTS BEFORE NOW YOU DON’T. Don’t come out here bashing people acting all knowledgeable when your process failed right in front of you. Whether it was a human error or a process problem, it doesn’t matter. No one cares. You shipped a product with a bug that it didn’t have before and is a behaviour you guys ALREADY IDENTIFIED.

      Yeah, you know why that Lotus you bought earlier broke down? Oh because the Toyota engine in it has problem. Tell me you would swallow that.

      Oh no wait, NUnit is a closed source software and you can’t do anything about it, am I right?

      Another developer with no business perspective as usual. I love JetBrains. But your attitude just plain stinks.

  • http://davybrion.com Davy Brion

    @Hadi

    code coverage is pretty much meaningless:
    http://davybrion.com/blog/2009/10/theres-only-one-thing-you-can-learn-from-code-coverage/

    And yes, it is possible that some bugs can still happen when you’re using the right tools and following the right practices. But this is a pretty major regression IMO… and something that orginally should’ve been covered with correct tests. The fact that it no longer works means that it simply isn’t covered with correct tests.

    As for what actually happened, that would be interesting to know but what would be far more helpful is to get an idea on how long we’re gonna have to deal with this until we get a version where this just works… like it used to.

  • http://hadihariri.com Hadi Hariri

    @Davy,

    Actually I don’t agree that code coverage is meaningless. I think exclusively focusing on the percentage of coverage is wrong, but coverage reports can point to potential issues in your code. And once again, as with my previous comments, we are assuming correct tests.

    And since youtrack is currently down due to a blackout and not being able to investigate further since I don’t have latest sources I can’t assume that a specific test was or was not in place.

  • Alexey Petriashev

    Same topic http://youtrack.jetbrains.net/issue/RSRP-185948
    already assigned to Victor Kropp

  • Alexey Petriashev

    these bugs just marked as critical!
    victory!!! ))

  • Ilya Ryzhenkov

    Here is what happened. During EAP program we were running with NUnit 2.5.5 integrated. But right before release we found critical issue in NUnit – it was running certain tests twice, in the cases involving inherited test fixtures. So we reverted NUnit to 2.5.3, then to 2.5.2 in order to fix this problem. Unfortunately, we didn’t had the other case in our regression testing, and anyway, we can’t do anything with it. The bug is in NUnit, not ReSharper. You can download NUnit 2.5.5 and install it, then tell ReSharper to run tests using installed version: Options / Tools / Unit Testing, select NUnit, specify different installation.

  • http://davybrion.com Davy Brion

    “The bug is in NUnit, not ReSharper”

    if it worked with Resharper 5.0, and it doesn’t work with Resharper 5.1 while i didn’t change any NUnit assembly, then clearly, the problem is with Resharper

    it’s good to know there’s a workaround, but again, the workaround shouldn’t even be necessary since it worked fine with resharper 5.0

  • http://davybrion.com Davy Brion

    actually the workaround doesn’t work… i installed nunit 2.5.5, set the ‘Specified NUnit installation’ field to ‘C:\Program Files\NUnit 2.5.5\bin\net-2.0′ (where it’s installed) and i just get the following exception when i try to run the tests:

    Unit Test Runner failed to load test assembly:
    JetBrains.ReSharper.TaskRunnerFramework.TaskException: Value cannot be null.
    Parameter name: path1

  • Alexey Petriashev

    Nunit 2.5.5 runs tests from base class, Resharper not

  • http://davybrion.com Davy Brion

    another sad thing is that RSRP-185948 was created 5 days ago, with no visible attention paid to it before it was marked as critical by Hadi Hariri a couple of hours ago

  • Ilya Ryzhenkov

    @Alexey, yes, NUnit 2.5.5 runs tests from base class, but ReSharper is bundled with NUnit 2.5.2. If you install NUnit 2.5.5 and customize NUnit runner as described above ReSharper will use it run your tests.

  • http://davybrion.com Davy Brion

    @Ilya

    unless R# can’t start the custom version as i mentioned above…

  • Ilya Ryzhenkov

    @Davy, ReSharper 5.0 was bundled with NUnit 2.5.3, but we had to revert to even earlier version due to it crashing in certain other cases. We are sorry for the confusion created. As for “visible attention”, we were investigating what is happening before “opening” the issue.

  • Ilya Ryzhenkov

    @Davy, just verified it again, downloaded 2.5.5 zip file, upacked into separate folder, set custom path to C:\Tools\NUnit\bin\net-2.0 and it works just fine. Also, we don’t have any unit-testing related “path1″ parameter. It looks like either NUnit has another bug, or your testing cases somehow causing exception during setup process.. Can you reproduce it on a separate sample project with simplest possible test fixtures?

  • http://davybrion.com Davy Brion

    @Ilya

    it works on a separate sample project with the simplest possible test fixture

    unfortunately that doesn’t help me at all since i want it all to work on the projects where it used to work and no longer doesn’t

  • http://davybrion.com Davy Brion

    @Ilya

    btw, the fact that you guys don’t have any “path1″ parameter doesn’t mean you can’t look for the reason of the exception… it is throwing a JetBrains.ReSharper.TaskRunnerFramework.TaskException so it is coming from your software _somewhere_

    regardless of whether this is an NUnit bug or not, this entire experience (the test runner’s problems in general) is really hurting the confidence that i used to have in you guys and i’m betting i’m not the only one

  • Valeriu Caraulean

    @Ilya
    Suggested workaround worked for me…

    I’ve posted this solution also in your bug tracker, so that people that is coming and voting for the issue can continue use Test Runner.

  • Ilya Ryzhenkov

    @Davy, TaskException is just a wrapper for real exception occurred in remote AppDomain (in separate process), so that it can be serialized and delivered to Visual Studio. I agree that diagnostics could be a little bit better. I’m here to try to help you have a working environment while we are investigating things on our side. If you can find NUnit 2.5.3 (shipped with ReSharper 5), you could try that and see if it works. If it does, it could give us more info in our work with NUnit guys on solving this issue. Thanks for being with us :)

  • Ilya Ryzhenkov

    @Valeriu great to see it helped you, thanks for posting info in YouTrack!

  • http://davybrion.com Davy Brion

    @Ilya

    i get the same exception when trying this with NUnit 2.5.3 :s

  • Valeriu Caraulean

    @Davy
    I’m not sure it’ll make a difference, but I’ve got NUnit by unzipping in my “Tools” folder. Same as it was suggested by Ilya. And you were using the installer and putting it to “Program Files”…

  • http://davybrion.com Davy Brion

    @Valeriu

    tried both, same result :s

  • Ilya Ryzhenkov

    @Davy, could you please try using NUnit GUI of the respective version to run your tests? Does it work without any problem?

  • http://davybrion.com Davy Brion

    @Ilya

    yeah, that works as expected

  • Alexey Petriashev

    Specified Nunit installation does not work for me.
    Failed to load test assembly…
    VS2010, Nunit 2.5.5 (with alone 2.5.2 same error)

  • G-Mac

    For what it’s worth, the NUnit 2.5.5 workaround worked for me. Thanks Davy and Ilya!

  • Bill Blinn

    @Davy

    Things worked fine for me when I was using NUnit 2.5.5 with the tests written into the super class. But as soon as I unchecked Tools -> Unit Testing -> Shadow-copy assemblies being tested, the error with path1 started happening.

    Let me know if you guys are seeing the same thing/have a fix.

  • Alexey Petriashev

    @Bill

    Yes I checked “Shadow-copy” and Nunit 2.5.5 started

  • http://davybrion.com Davy Brion

    @Bill

    you’re right… enabling shadow-copying makes it work

    i usually turn it off though since we sometimes get bad results because of it

  • http://davybrion.com Davy Brion

    for future reference, the path1 thing was indeed a resharper bug and has been fixed:
    http://twitter.com/resharper/status/18943045975

  • http://blog.martin.aatmaa.net Martin Aatmaa

    For the record, I think you were being a tad harsh on Jetbrains on this.

    I’m actually even more impressed with them as a result of this. Keep up the great work and support guys!

  • http://davybrion.com Davy Brion

    @Martin

    if their last 2 releases had major issues for me, both of which i believe should never have made it into a released version then i have every right to react the way i do

    it’s not because they’re JetBrains that they can get a free pass

  • Chris

    I’m getting the same issue on the latest Resharper version (5.1.1753.4) and it is running NUnit 2.5.7 according to the Resharper –> Options –> Unit Test –> NUnit. This completely screws all of our tests.

Meta

Recent Posts

Categories

Archives