More ORM Ignorance
Posted by Davy Brion on September 4th, 2009
Tony Davis from simple-talk wrote a post about the so-called ORM Brouhaha that followed from the recent benchmarking fiasco. I have some problems with some of the statements in his post, and since he loves to hear what we think i decided to share my thoughts on this.
He starts off the post with the following:
A while back, Laila Lotfi wrote an editorial on the need for a standard benchmark for Object-Relational mappers, such as Entity Framework and nHibernate. By how much do they really slow down database applications?
ORM’s slow down database applications? That’s a generalization that i believe many of us can make about stored procedures or classic data access layers as well, after all, they are quite frequently used as incorrectly as ORM’s are often misused. First of all, the purpose of an ORM is not just to ease your development tasks or to keep your code free from repetitive cruft. It is as much about optimizing your usage of the database as it is about ease of development.
Any decent ORM will aim to minimize overhead when it comes to communicating with the database. Generally speaking, an ORM should try to send as few statements to the database as possible, either through usage of batching techniques, being able to generate good queries, and not executing unnecessary statements. How many times have we seen data access code that doesn’t have proper dirty checking and sends update statements for entities that haven’t really been modified? I’ve seen plenty of implementations where doing something like myEntity.SomeProperty = myEntity.SomeProperty still resulted in an unnecessary update statement because a dirty flag was set in a naive manner. A good ORM will not do this, and will try to keep the overhead of communicating with the database as low as possible.
There many complaints to the effect that the benchmark tests are useless, because the tool “should never be used in that way”. If it can be, it almost certainly will be; and it is up to the tool creator to make sure that it stands up as well as its competitors.
Is it the responsibility of a tool creator to make sure it performs as well as it possibly can in usage scenarios that are actively discouraged? Should we hold everyone to this rule? How about we apply that logic to databases or DBA’s? Using the same twisted logic, we could actually blame bad database performance on database implementations or DBA’s because hey, we are able to use them badly, so it is their responsibility to make sure that it performs as well as it possibly can. Seems like a bit of a ridiculous statement, no? This kind of ‘logic’ actually conflicts with the problems that many people incorrectly associate with ORM’s so i think we can safely ignore this.
I imagine the sight of such a brawl sent a chill down the spine of managers who may have been planning to use ORM technology.
Pardon me for being blunt, but having to follow the technical decisions from a manager who holds value to the results of silly, unrealistic benchmarks is a situation that would send chills down my spine as well.
The IT industry is increasingly coming to suspect that the performance and scalability issues that come from use of ORMs outweigh the benefits of ease and the speed of development.
I haven’t really noticed this so called trend. In fact, i’m under the impression that usage of ORM’s is finally becoming more and more accepted, especially in the .NET world. In the Java world, where i think we can objectively state that many large-scale and highly performant applications have been developed over the years, ORM usage is pretty prevalent.
And seriously, how many of us have had to deal with applications or systems that didn’t use an ORM but used either a classic data access layer or stored procedures and still suffered from bad database performance? I think the large majority of the development community has experience with exactly this kind of situation, so i really don’t think ORM’s are the problem here…
We are crying out for objective benchmarks and if the ORM industry itself cannot hope to agree on how to do it, then perhaps benchmarks will have to be imposed on them.
Impose all you want, but don’t be surprised if those benchmarks will be ignored or questioned by those who hold more value to properly educating developers instead of basing their decisions on irrelevant benchmarks.
September 4th, 2009 at 4:53 pm
Davy:
“A good ORM will not do this, and will try to keep the overhead of communicating with the database as low as possible.”
I can assure you that the nHibernate project I work on is the most “chatty” application I’ve ever seen. The amount of data access (much of it unneeded) is truely horrendous.
“Is it the responsibility of a tool creator to make sure it performs as well as it possibly can in usage scenarios that are actively discouraged?”
Do you have any links to those “actively discouraged scenarios” for nHibernate? I’ve never seen any. The fact is if the framework can be used in a particular way its fair to measure its performance against other frameworks that are tested in the same way. If these frameworks are so great why hasn’t anyone shown how to perform the given tests “properly” ?
The more people I talk to the more I find that are becoming very sceptical of ORMs, having had failed projects and problems.
September 4th, 2009 at 5:37 pm
@Mark
i’d love to see the code that is causing the chatty behavior… and the relevant mappings as well. If data is retrieved in a chatty manner then i would first suspect abuse of lazy loading (typically the result of lazy coding) or seriously bad queries (either through HQL or Criteria). In either case, the result of manually written code. If these problems are prevalent in the code base of your application, then i would suspect that you and your teammembers are just not used to using an ORM. And if the unneeded database calls are unneeded update statements, then you are truly doing some weird things in your code, or in your mappings if it tries to do unneeded relationship updates. That last situation is actively discouraged in the official NHibernate documentation btw
you can find plenty of discussion about ‘actively discouraged scenarios’ in the NHibernate in Action book, on Fabio Maulo’s blog, Ayende’s blog, and sometimes on mine as well. Last but not least, be sure to check out the nhusers group on Google Groups where incorrect usage is very actively discouraged
. Oh, and then there’s always the NHibernate reference documentation of course.
“If these frameworks are so great why hasn’t anyone shown how to perform the given tests “properly” ?”
bulk data processing or loading large sets of data are not exactly things that ORM’s aim to be good at. Why should developers of these frameworks spend time trying to test these situations ‘properly’ if said developers realize that ORM’s are just not the right tools for these jobs?
“The more people I talk to the more I find that are becoming very sceptical of ORMs, having had failed projects and problems.”
A large percentage of software projects fail. Perhaps we should all just quit coding, since coding obviously seems to be at the root of the problem somehow…
September 4th, 2009 at 7:46 pm
The best thing we can do, imo, is continue to post and educate on best practices with ORM’s.
Many focus on just query performance… what about the unit of work capability – that itself is a major reason why I use a ORM – tracking changes, optimistic concurrency, etc…
Again though, education on how to set this up is critical.
September 4th, 2009 at 8:42 pm
“The IT industry is increasingly coming to suspect that the performance and scalability issues that come from use of ORMs”. That equals “As ORM-s become widespread and commonly accepted, there are more and more old schoolers creating a whole lot of locomotion as they are subjected to change”. There’s a new gun to shoot yourself into leg with and while your existing arsenal is just as good for blowing your both legs off several times, the things you know doesn’t seem so dangerous…
September 4th, 2009 at 11:24 pm
Davy, it’s enough to visit FAQ: http://ormbattle.net/index.php/faqs.html to get answers to many questions you try to discuss here.
It’s fun to see how weak players are walking around its “bad quality”, “wrong tests” & “framework misuse” further and further.
Take a look at Telerik or BLToolkit. Or read this: http://ormbattle.net/index.php/blog/93-great-quote-from-todays-e-mail.html
I think if you don’t want to be compared to others, may be it’s your time to leave the market.
September 5th, 2009 at 12:02 am
I’m not sure of the point you are trying to make here. Tony’s point, picked up from Laila, seems clear enough. At Teched several statements were made that ‘Entity Framework was about four times slower than LINQ for SQL’. Why? How? Under what circumstances? It all seemed so vague that it seemed obvious that it ought to be put to objective test. Who better to devise the benchmarks than the ORM providers themselves? Obviously we, as consumers, would find this very hard to do, but if the ORM industry can’t do it then someone else will have to.
As I understand it, you are saying that there shouldn’t be ORM benchmarks. You also seem to think that Tony is against ORMs. Hold on. As I read it, He’s saying that benchmarking will actually help acceptance of ORMs as it will put an end to vague statements about performance deficits. He is warning that current language being used in the debate about ORM benchmarking will serve just to put managers off. If so, Yes. It will. Even your comments about managers in the blog made me wince.
I’m not close enough to the issues to join in the debate on what type of benchmarks would be fair, and how they should be administered, but I can’t help but agree that clear and objective benchmarks would help me a great deal, and would surely serve to highlight weaknesses in individual ORMs.
September 5th, 2009 at 7:51 am
> Why? How? Under what circumstances?
Likely, there was a performance test similar to ours, since real life it can’t differ so much. Although maximal possible difference must be close to 3-4 times. Compare e.g. BLToolkit and Entity Framework performance – there is nearly the same case. BLToolkit is much closer to DAL than to ORM (as well as LINQ 2 SQL), but it is really fast.
> As I read it, he’s saying that benchmarking will actually help acceptance of ORMs as it will put an end to vague statements about performance deficits.
++
September 5th, 2009 at 10:13 am
Alex,
i have no problem being compared to others… but you and i think very, very differently about a lot of things. And you’ve already shown on Ayende’s blog that you are impossible to talk to. Not only that, but i think there is something seriously wrong with your thought process in general (i’m basing this on everything i’ve read from you so far), not to mention the manner in which you try to engage people in conversation. Because of this, i have zero intention in even trying to explain the differences in our thoughts and i won’t waste time trying to argue with you. Any future attempts from your side to begin any kind of discussion with me will simply be ignored. So do yourself a favor, and don’t waste time on me
September 5th, 2009 at 10:37 am
@Andrew
i’m not saying that there shouldn’t be ORM benchmarks, only that people shouldn’t base decisions on bad benchmarks. Take the whole entity hydration cost thing… let’s say that ORM A has a lower overhead when it comes to hydrating entities than ORM B. What does it truly matter if, in real world code, ORM A requires more roundtrips to the database to achieve what you need to do than ORM B would? Running tests like that in an environment where the database is present on the same machine as the code is useless because in a real world system, nobody in their right mind would have the DB installed on the same machine. The overhead of network roundtrips definitely changes a lot of things.
Also, there is a lot more to it than simply performance results in silly benchmarks. Why is nobody paying to attention to features, power, flexibility? Current acceptance and user community, not to mention documentation are things that really need to be taken into account as well. If performance in benchmarks is what we should base our decisions on, none of us should be writing code in C# anyway.
As for my comment about managers, i truly feel that they shouldn’t even be involved in deciding between ORMs or any library for that matter, unless there are licensing issues attached to it (either through a non acceptable OSS license, or a license cost)
September 5th, 2009 at 10:46 am
We have used nHibernate on a number of projects which in the past would have been just hand rolled SQL using SP’s and have had positive feedback about how much it cuts down on the simple DAL code we would be normally writing and maintaining, sure there are edge cases were we need some custom queries and in these cases we would create the SQL manually, I think a lot people miss the 80-20 rule and think that by using nHibernate they lose all control over there data access.
Anyway just wanted to share some positives we have had using nHibernate and appreciation for people like yourself who contribute to this project, thank you.
Mike
September 8th, 2009 at 6:15 am
Davy, tactics like this never works.
> And you’ve already shown on Ayende’s blog that you are impossible to talk to.
You tell me I’m the person impossible to talk, but facts are completely different:
- I’m ready to discuss everything. You aren’t: “Because of this, i have zero intention in even trying to explain the differences in our thoughts and i won’t waste time trying to argue with you.”.
- The same about Oren. But I’m glad he answers a bit more carefully now.
“i think there is something seriously wrong with your thought process in general” – so tell me, what is wrong there?
The problem I see is that you’re trying to play with the words instead of proving your position. Great, go further with this. But that’s purely your problem, not my.
I can add that I answered on most of questions forwarded to me (except may be the most ridiculous ones). And I’m still ready to do this. Yes, I don’t promise anyone to agree with him, and that’s what people must accept. I never simply say “no”. I’m ready to waste a lot of my time explaining why something that was offered won’t work. In if there are no strong arguments on your side, I won’t agree with you.
Speaking about Oren, try to count how many times he said “I agree with this” in these discussions (actually I can’t remember even a single point). Compare this to my “I agree with this”. Now compare the numbers and think who’s really impossible to talk here.
> i have no problem being compared to others…
I didn’t mean exactly you. It was about anyone.
> What does it truly matter if, in real world code, ORM A requires more roundtrips to the database to achieve what you need to do than ORM B would?
There are lots of factors in real world, and we will never be able to measure everything. I hope this is clear. On the other hand, we’re trying to measure the most important ones.
Here you’re talking about quality of query, fetch and preload pipeline. Yes, currently we don’t measure its quality in count of queries its send; moreover, our current tests measure pretty basic stuff now. But:
1) We’ll go further with this. Bulk load tests involving future queries and preload pipelines will be there some day.
2) Even what’s was measured now shows the difference pretty well. E.g. NH and Subsonic (that was claiming to be super-fast & super-simple) have exposed their query pipeline is the slowest one. Do you think running a simple query returning 1 page 7 (or 20!) times slower than competitors is ok? I don’t think so.
3) Moreover, I think it’s hardly possible these tools “save” so many queries that they can compete with leaders. I hardly believe that in real applications they run even 2 times less queries.
4) IMHO, 3) makes me think there can be absolutely the same situation with queries (possibly not with NH, but quite likely – with Subsonic): bad raw performance just proves they never been seriously profiled & compared to other products.
So why I must BELIEVE they are better from this point? Because they’re POPULAR? I know this does not work. DO1.X-2.X was pretty popular. But if I’d benchmarked it @ ORMBattle.NET, likely, it would be the worst one.
Popularity is related to acceptance by average developer (likely, it implies simplicity or minimal learning curve), costs and many other factors (documentation, etc.). Moreover, popularity is related to popularity itself: the more tool is known, the faster it gets new fans. But performance does not play any role here, until it is acceptable for this average developer.
And if there are no benchmarks, this acceptance limit can be very low. On the other hand, when benchmarks appears, performance requirements are raising up. The same happened many times in history.
The best thing is that customers win because of this.
P.S. Good luck with analysis of my wrong thought process.
September 8th, 2009 at 9:40 am
@Alex
I think I speak for a lot of people that followed the discussion at Oren’s blog when I say “dude, let it go”.
September 8th, 2009 at 12:09 pm
@Alex
I’m sure that all of us who are in favor of the participants in the ORM industry working together to adopt fair and standard benchmarks want to discourage any public wrangling. It is tedious for us to have to read. If these guys say stupid things about you then we are quite capable of reading what they write, and judging, without your elaborate responses/goadings. It is much better to leave it, let it go, and allow us make our minds up.
The established benchmarking sites in the IT industry are all run by people who have to be able to resist considerable pressure from misguided pressure groups, if ever it happens, without flinching.
With good wishes to everyone in the ORM industry,
Phil
September 8th, 2009 at 3:58 pm
Thanks
I understand that likely I already said everything I wanted… On the other hand, it is my reputation as well, and I prefer to keep it by my own.
September 8th, 2009 at 4:43 pm
Alex, you should go into politics.
BTW your reputation already got a big dent in it I think. Maybe you can fill up that dent with improving your product, instead of saying the same things over and over again about the competition and its people.
Then again, it’s your choice of course. If you want to waste time, argue all you want. I just sit back and enjoy the soap opera.
To Davy:
Some people will always love hand-coded SQL, SP’s etc… Better to focus on the people that want to learn, instead of the unwilling who stick their head into the sand.
Oh and I love your blog!
September 8th, 2009 at 5:53 pm
Alex,
For a benchmark to be valid, it needs to represent real-world scenarios, and in the case of databases, that means the DB is on its own server and any and all accesses to it has network overhead. Sure, you can change the scenario to measure other things, but if the scenario isn’t real-world, then the results are irrelevant in the real-world.
Of course, the point we’re all missing is that if there’s anything in my first paragraph that’s not obvious to someone, then that person is clearly either an entry-level developer or not a developer at all. Therefore, like Davy, I’m going to ignore anything you have to say as well, but because of your lack of expertise rather than your child-like refusal to consider the possibility that a benchmark could be irrelevant.
September 9th, 2009 at 6:04 am
@ Alwin:
> BTW your reputation already got a big dent in it I think.
I don’t think so. But I’m pretty sure the reputation of few persons that were thinking about exclusion of their products from tests will suffer.
> Maybe you can fill up that dent with improving your product, instead of saying the same things over and over again
So you just tied together two independent things and think it’s ok? Or you seriously think we don’t work on DO4?
> If you want to waste time, argue all you want.
Obviously, I don’t want waste the time. Ridiculous statement.
Now read your own post – none of sentences there is related to the subject. They all are about me personally. Have you said something important? I don’t think so. Just flood. But you THOUGHT you said, yes?
So think about politics. Blaming the others, giving obvious advices, providing no proofs – that’s you.
Further I’ll reply to comparison-related questions only.
@ John:
> if the scenario isn’t real-world, then the results are irrelevant in the real-world
They are relevant. Yes, real-world results might be different, but the numbers we show describe real world cases pretty well, you must just clearly know how to apply them.
Let’s take my last post as an example: http://ormbattle.net/index.php/blog/105-a-honest-comparison-of-entity-framework-and-nhibernate.html
Why NH with prefetch is just 2 times faster than EF without prefetch? How can it be true that 1 query is just 2 times faster than 2500 fetching the same data? The answer is obvious: slow materialization + slow query pipeline. That’s what we’ve shown.
So dealing with large amount of queries or data isn’t the case where NH will be the best choice. Is it related to real-world? I think, yes.
See also:
- http://ormbattle.net/index.php/faqs/85-what-results-of-your-benchmark-show.html
- http://ormbattle.net/index.php/faqs/84-when-results-of-these-benchmarks-are-applicable.html
> if there’s anything in my first paragraph that’s not obvious to someone, then that person is clearly either an entry-level developer or not a developer at all
There is actually third option: you’re simply wrong. I’m leaving the opportunity to prove this for you.
September 9th, 2009 at 7:18 am
you know, if nobody responds to his comments, he would just go away…
September 9th, 2009 at 10:47 pm
@Alex,
Aside from the fact that I get put off by your argumentative and abrasive tone, I think the point you are generally missing is that you are [blindly?] obsessed with proving that ORMBattle.Net is the “one true way” to compare ORMs. You write countless essays (er, I mean comments) about how test X is correctly written and measurement Y is the true key to measuring “the best” ORM on the market. Arguing on the internet has got to be the biggest time-sink there is so do yourself and us an favour and stop obsessing. If you truly believe in your site and product then they should not need to be so vigorously defended and the competition need not be attacked. Focus on your site and product and stop “educating” us via huge comment threads, which, honestly, it isn’t doing you any good, public-image wise.
I know I for one will gladly consider any ORM — I’ve used SubSonic and EntitySpaces on a few projects and have recently begun experimenting with NHibernate and Fluent NHibernate. I mean to try LLBLGen Pro too but with it having a similarly steep learning curve to NHibernate I can only choose one to learn, for now. I would probably have tried DO too, but to be honest I was put off by the *appearance* of a possible hidden agenda on the comparison site. I know you’ll say that wasn’t the intent and that you removed DO from the running but the damage was done.
Even that could have been overlooked though, but when I read your posts/diatribes other blogs and sites — like the “leave the industry” one above — it really sealed the deal for me. I’ll pretty much have to exhaust all other options before I turn to your ORM. Those options have grown for me thanks to your site by the way, so thanks for that. Unfortunately it also means I have more tools to play with before I try yours.
Good luck with your product and site, and please don’t take my comments as insults. They are not intended as such. Rather they are meant as insight into how you are unknowingly hurting your cause, in my mind at least.
September 10th, 2009 at 5:25 am
Sean, I agree with most part of your answer.
I know such a web site significantly polarizes people’s opinion, and this isn’t related just to my company and product. You decided you will study our product at last mainly because of my position. But many others decided to study it first. The same is about other products listed there. Some people reading Oren’s blog decided to stay with NH independently of results we’ve shown, others have started to think about alternatives. I think that’s great.
There are few other aspects I could discuss, but for now there is really no time for this.
September 11th, 2009 at 1:39 am
Sean,
> Arguing on the internet has got to be the biggest time-sink there is…
You nailed it. This has been going on for weeks now, and nobody, or at least none of the main players, have changed their view on the matter.
For me it’s just geek entertainment, and I learn something about NH along the way. But even still, it’s a time sink. Can’t imagine how much time wasted for those who are active in responding to everyone about this.
Your comment was what I tried to explain to Alex, but better worded.
Alex,
> So think about politics. Blaming the others, giving obvious advices, providing no proofs – that’s you.
At least I’m not the only one, huh Alex?
It is not my intention to make you look like a fool.
What I find funny is that you advice others to stop wasting time and start improving their product, while you keep discussing with everyone as if your time is in abundance. Maybe it is, I don’t know.
If you agree with most of Seans answer, maybe also with the part “they are meant as insight into how you are unknowingly hurting your cause, in my mind at least.”? That is kind of what I meant with the dent in your reputation and filling it up.
Davy,
> you know, if nobody responds to his comments, he would just go away…
I find this entertaining so I don’t mind. But if you want him to go away, just say so, I’ll stop responding then.
September 11th, 2009 at 9:31 pm
@ Alwin,
I’m 100% sure there is no dent in my reputation. What’s the reason of its appearance? Yes, I’ve made a test suite showing many lacks in ORM products. Taking into account BLToolkit results, even our own product isn’t ideal there. So what kind of dent to you see?
Before I reply to everything else, can you tell me, are you talking just about performance tests? Test suite consists of two parts: LINQ and performance tests. So I’d like to hear something about LINQ tests as well.
September 11th, 2009 at 9:32 pm
“to you see” -> “do you see”.