ORM

Nhibernate vs Entity Framework – a revised performance comparison

Note: before reading this article please see my views on comparing ORM tools. In short: this comparison is not suitable for comparing NHibernate and Entity Framework, due to the complexity of the tools vs the simplicity of this test; treat this post accordingly.

The reason I’m revising previous results is because some members of the NHibernate community looked into the code and discovered that the comparison wasn’t totally fair. While the test cases with Entity Framework used batch operations, the NHibernate implementation had this feature turned off. Tuna Toksoz was kind enough to re-write the critical parts of the code and apply settings enabling batching, you can download the revised source code here.

Note that I’ve made the source open for two reasons. First, I’d like the tests to be reproducible. And more importantly: I have experience with the two frameworks however I’m not an expert at any of them: therefore treat the results with caution. Anyone is free to analyze the source and if there are some issues in the code resulting in major performance drops, please let me know.

After Tuna modified the code NHibernate storage speed rocketed up compared to the previous version:

Framework/ No. of operations 1K 3K 8K 14 K 23K 33K 40K
NHibernate 1,22 3,25 8,98 13,11 28 38 46
Entity Framework 2,11 6,18 18,4 31,3 51,7 77 98

store2

However improving performance does not stop here. Fabio Maulo posted a great article on why the original code was far from optimal and improved runtime results even more.

Conclusion

As I’ve stated before I don’t beleive there is a good way in comparing ORM frameworks and even if there were, this little test certanly is not it.

As the contribution of the NHibernate community members showed the more complex and configurable the tool, the more experience you’ll need to make the best of this. If you’re working on a performance critical product, you’d better know you’re tool well and use it accordingly.

But which one should I use?

From the current results it seems that there is no major performance difference between two tools and even if there is it might be due to misuse. So my advice is to use which ever tool suits you more based on its features.

At first glance Entity Framework seems to have a shorter learning curve as well as some nice integrated VS tools to make development easier. On the other hand there is a constantly active and helping NHibernate community always ready to answer questions and NHibernate seems to be much more customizable at a higher level. Also see this thread on EF vs NH features and learning as well to help decide.

The decision is up to you which one you find more suitable for your needs. However since both are proven and mature solutions you won’t be making a big mistake going with which either.

Thoughts on ranking ORM tools

Posting of the previous performance comparison results (see here) has caused a much greater response I ever expected as well as some confusion about the goal of this test that I would like to respond to.

How real (world) is this?

Most importantly a performance test is useful if it can be used for predicting performance of a real world application. Now the performance test in I did was a really primitive, batch based CRUD test. Unless you are developing an application that does batch actions such as in this test, the results can’t really be used for predicting performance of real world applications. We’re talking about two mature frameworks with lots of additional services – e.g. caching and transaction handling – that are not evaluated in this test even though they can have a huge impact on performance.

Be a little sceptic

I would suggest treating results with caution and checking on the source. There are a lot of people expert on a single ORM framework, some are experts on two but it mostly stops there. Unless stated that the author(s) is/are experts on the topic I would approach these results in a sceptic way.

As for my own case: I do have experience with the tested frameworks however am far from being an expert with any of them – so treat my results with caution. I’ve published the so anyone can analyze it and if there are some issues in the code resulting in major performance drop, hopefully someone will let me know – just as some members of the NHibernate community did after publisihing the first results.

There is no “best”

I believe there is no valid way to decide which framework is ultimately better. There is an ongoing debate about ranking ORM tools and how well a test like this is good for deciding on which framework is better. I think it is not. Just to mention a few reasons:

  • Most real world applications have a far more complex working than this test simulates. What’s more, commonly used additional framework features – e.g. caching, transaction handling – are not tested at all even though these features can have a great impact on real world performance
  • Performance may not be everything. Even if these performance tests would reflect performance in a real world environment (as they don’t), ORM is not used based on performance. When using an ORM tool you make a compromise to have worse performance than using native SQL and gain additional features. It’s usually a tradeoff: the more features you get, the bigger the overhead will be.

There is one case when this kind of test can be useful though. If you find a great performance difference between two frameworks (e.g 50-1000 times), it’s a good indicator that there’s either some issue with the test itself or the framework. If you’re sure that the test is written correctly this comparison could lead to the analysis of why the framework is unacceptably better than the other. If you can find the valid answer, then good. If you can’t, now that’s when things get interesting and the test was worth doing.

The best valid way of comparing an ORM tool in my opinion is based on the project you need it for. Write together your needs, weigh them and evaluate the frameworks accordingly. ORM tools are comparable just as cars are. You can’t decide which one’s better unless you know what to use it for. An F1 car may be great for a speed race but would utterly fail in a desert challenge.

NHibernate vs Entity Framework: a performance test

As part of my thesis I measured the performance of some .NET ORM frameworks including NHibernate and Entity Framework. Measuring was done by implementing two simple applications using the same table structure and doing the same operations on the same data.

Note: before reading this article please see my views on comparing ORM tools. Also see the follow up post to this comparison for revised results.

I measured the time it took for each framework to complete these operations:

  • store
  • read over relations
  • read by ID
  • update
  • delete

The results were somewhat interesting. Here is a short summary of what I’ve found out.

Operation \ Number of operations NHiberante – 4K Entity Framework – 4K NHiberante – 40K Entity Framework- 40K Winner
Store 37,37 9,19 1500 98 Entity Framework
Read over relations 1,01 0,54 10,13 4,18 Entity Framework
Read by ID 3,06 25,22 246 230 NHibernate with smaller amount of objects
Update 6,61 7,34 77 72 Both
Delete 3,35 16,76 58 1824 NHibernate

Read on if you’re interested in the details of the performance measurements.

Update: the source code of the program used to measure is now available for download.
Read More…

NHibernate LINQ 1.0 is here

One of my biggest concerns with NHibernate is that to use it one must become familiar with Criteria and Expression quieries which is a timely process.

However LINQ support has been released for NHibernate meaning you can create complex queries without the need of learning its querying language. You can download and try it from the NHibernate Contrib site and read the official announcement on the creator’s blog.

Querying many-to-many relations in NHibernate

I’ve ran into a querying scenario with NHibernate that was much less obvious to solve with the NHibernate query API than it would have been with SQL – for me at least.

In my model I’ve had a simple many-to-many relation: Entries that had multiple Categories each and Categories that belonged to multiple Entries as well.

Categories and Entries: an Entity-Relationship Diagram

Categories and Entries: a many-to-many relationship

In the underlying SQL model the many-to-many relationship was implemented via a relationship table:

Many-to-many relationship table structure

Categories and Entries table structure

Now I wanted to find all entites within a given category. Pretty simple, right? In SQL it would have been a simple join:

SELECT Entry.* FROM Entry JOIN Category_rel_Entry ON Category_rel_Entry.Entry_ID = Entry.ID WHERE Category_rel_Entry.Category_ID = @CategoryId

In Nhibernate this query is a bit more tricky, let me share how it can be done.

Note: this post is also a good example of how to define a many to many relationship schema in NHibernate.

Read More…

 Scroll to top