NHibernate vs Entity Framework: a performance test

As part of my thesis I measured the performance of some .NET ORM frameworks including NHibernate and Entity Framework. Measuring was done by implementing two simple applications using the same table structure and doing the same operations on the same data.

Note: before reading this article please see my views on comparing ORM tools. Also see the follow up post to this comparison for revised results.

I measured the time it took for each framework to complete these operations:

  • store
  • read over relations
  • read by ID
  • update
  • delete

The results were somewhat interesting. Here is a short summary of what I’ve found out.

Operation \ Number of operations NHiberante – 4K Entity Framework – 4K NHiberante – 40K Entity Framework- 40K Winner
Store 37,37 9,19 1500 98 Entity Framework
Read over relations 1,01 0,54 10,13 4,18 Entity Framework
Read by ID 3,06 25,22 246 230 NHibernate with smaller amount of objects
Update 6,61 7,34 77 72 Both
Delete 3,35 16,76 58 1824 NHibernate

Read on if you’re interested in the details of the performance measurements.

Update: the source code of the program used to measure is now available for download.

How performance was measured

The data model I’ve set up to measure with was pretty simple. I used three entites: a Company entity which had multiple Employees all of which had multiple Reports all with a 1-n relation.

The detailed results for each operation are as follow (note that the graphs use logarithmic scale):

Storage

Framework/ No. of operations 1K 3K 8K 14 K 23K 33K 40K
NHibernate 4,95 20,64 112 280 575 1107 15000
Entity Framework 2,11 6,18 18,4 31,3 51,7 77 98

storage3

Update (2009. 08. 24.):
After NHibernate improvements by Tuna Toksoz (download source) this metric changed the following way:

Framework/ No. of operations 1K 3K 8K 14 K 23K 33K 40K
NHibernate 1,22 3,25 8,98 13,11 28 38 46
Entity Framework 2,11 6,18 18,4 31,3 51,7 77 98

store2

Read over relations

Framework/ No. of operations 1K 3K 8K 14 K 23K 33K 40K
NHibernate 0,21 0,66 1,81 3,31 5,0 8,24 10,13
Entity Framework 0,17 0,4 0,96 1,76 2,47 3,91 4,18

read

Read by ID

Framework/ No. of operations 1K 3K 8K 14 K 23K 33K 40K
NHibernate 0,28 1,31 13,0 37,6 80 175,5 246
Entity Framework 5,56 17,03 48 83 132 195 230

readbyid

Update

Framework/ No. of operations 1K 3K 8K 14 K 23K 33K 40K
NHibernate 1,74 4,54 17,1 33,2 43,47 68 77
Entity Framework 1,99 5,29 14,2 24,2 41,9 59 72

update

Delete

Framework/ No. of operations 1K 3K 8K 14 K 23K 33K 40K
NHibernate 0,78 2,31 8,83 20,6 28 46 58
Entity Framework 1,78 8,19 52,2 133 320 1014 1895

delete

Conclusion

The performance measurements only provided significant differences in two cases.

  • When it comes to storing data, Entity Framework proved to be significantly faster than NHibernate
  • When deleting data, NHibernate was much faster than Entity Framework

My conclusion however after the test was that in those scenarios when the number of the operations is not too high performance does not differentiate that much. I would choose between these two tools based on which ones provides faster, more efficient and more straightforward development with the current project.

Give me the code

As many of the commenters pointed out a comparison is not really valid without publishing the code enabling validation of this measurement. The program I’ve written to do the test can therefore be downloaded: NHibernate vs Entity Framework source code.

To set up and run the project after downloading an unzipping the package follow these steps:

  • Create two MSSQL Databases (e.g. EntityFrameworkTest and NHibernateTest)
  • Run CreateTables.sql on both databases to create the database structure
  • In the App.config section of the MeasurementController project set the connection strings for NHibernate and Entity Framework (replace the !!! marks)
  • Build the solution
  • Run the MeasurementController project
  • Run times are written to the output and also saved in files in the bin/Data folder

Some additional comments for understanding the source:

  • Upon every run test data is created in an XML file. From this XML file Company, Emlpoyee and Report objects are created in-memory as the test structure
  • This test structure is passed to the NHibernate and Entity Framework implementations of the Store, Read, Read by ID, Update and Delete tests (in the source additionally search is also measured) on each test run.
46 Responses to NHibernate vs Entity Framework: a performance test
  1. Trackback: DotNetShoutout
    » NHibernate vs Entity Framework: a performance test - Greg Does IT... Thank you for submit... dotnetshoutout.com/-NHibernate-vs-Entity-Framework-a-performance-test-Greg-Does-IT
  2. Tuna Toksoz

    In order for this benchmark to make sense, you should publish some code.

  3. Gergely Orosz

    You’re right. I’ll be publishing the source code by tomorrow for the test to be repeatable.

  4. Dani Pfulg

    Source code would be very much appreciated indeed.

  5. [...] saw the following post where the performance of NHibernate and Entity Framework is compared for a couple... davybrion.com/blog/2009/08/of-course-nhibernate-is-slow-when-you-use-it-incorrectly
  6. Dmitry Polskoy

    Greg, which versions of EF and NHibernate did you use?

  7. [...] la plus utilisée) EF est plus performant que NHibernate. C’est en tout cas mon interprétation des te... blogs.developpeur.org/matthieu/archive/2009/08/20/ef-plus-performant-que-nhibernate.aspx
  8. minsou

    I’m agree with Tuna !

  9. Thomas

    Hello, I’m interested in your code too.

  10. Mike

    Your graphs are all showing the labels incorrectly – shouldn’t time be the vertical axis and no. of operations be the horizontal?

  11. [...] the launch of the ORMBattle.NET site, a new discussion started on the blogosphere; later on, this post b... ilovenet.com.ar/2009-08-20/orm-performance-comparison
  12. Gergely Orosz

    @Tuna, @Dani, @minsou, @Thomas: the source code is now available for download. Sorry for the delay.

  13. Gergely Orosz

    @Mike: thanks for noticing. They were incorrect, I’ve updated them.

  14. Gergely Orosz

    @Dmitry: the version I’ve used to evaluate (also in the source code as reference) is NHibernate 2.0.1.4 and Entity Framework 3.5.

  15. Travis

    I’d assume your NH version of SearchByID would go faster if you didnt do all the unboxing/typeof crap, and simply used generics.

    NHEmployee foundEmployee = session.Load(employee.Id);

  16. Travis

    WP stripped the type, from the code. But you get the idea…

  17. Gergely Orosz

    It would probably improve performance a bit, you’re right.

    Actually @Tuna improved the NH version by modifying the source enabling batching instead of calling Flush() every time thus dramatically improving NH results. I’ll post that updated version with the new results as well. I’ll also get rid of unnecesarry boxing there.

  18. Alexis Kochetov

    Hi, Gergely. Could you contact ORMBattle.net team and offer them your performance tests. It will be great to have all ORM tests at one place ).

  19. Fabio Maulo

    @Alex sure… but would be grate to have real performance tests and not MDTD as in ORMBattle.net

  20. Alexis Kochetov

    @Fabio, many people talk about real tests, but nobody proposes real scenarios. I would like to have real tests, so feel free to contact me (alexis [dot] kochetov [at] gmail.com) if you have any thoughts about them.

  21. Darius Damalakas

    waiting for your updated version.

  22. Fabio Maulo

    @Alex If you are interested in something you should study what do, and don’t ask to others to do something for you. If you can’t create a real example it is your problem.
    About this post I had published the second part, with the expected behaviour, in my blog:
    http://fabiomaulo.blogspot.com/2009/08/nhibernate-perfomance-analisys.html

  23. [...] The blog of Gergely Orosz, Sense/Net 6.0 developer on IT related topics: web, software design, .NET, PHP... gregdoesit.com/2009/08/thoughts-on-ranking-orm-tools
  24. [...] After Tuna modified the code NHibernate storage speed rocketed up compared to the previous version: [...... gregdoesit.com/2009/08/nhibernate-vs-entity-framework-%e2%80%93-a-revised-performance-comparison
  25. Gergely Orosz

    @Alex I’m afraid this test is not a good example for comparing ORMs. See my post http://gregdoesit.com/2009/08/thoughts-on-ranking-orm-tools/ for details on why.

    @Darius: published an updated version although Fabio probably wrote an even better blog post on this (http://fabiomaulo.blogspot.com/2009/08/nhibernate-perfomance-analisys.html)

  26. [...] términos de rendimiento entre Entity Framework y NHibernate…y los resultados los podéis ver en este ... geeks.ms/blogs/ciin/archive/2009/09/11/ado-net-entity-framework-vs-nhibernate-una-primera-comparativa.aspx
  27. Javier Calvarro Nelson

    You are not deleting items in EF properly. To delete an object you only have to create in memory a new instance of the object, then call context.CreateEntityKey(createdObject) and call context.Attach && context.DeleteObject or context.DeleteObject directly, i’m don’t remember the final step but you haven’t got to query the DB for the objects first in order to delete them later. Do you understand my point (sorry for my english)?

  28. Ray

    Improper tests due to lack of knowledge of how internally works NHibernate and how to optimize it for certain scenarios.

  29. [...] you’re interested in the details of the performance measurements, please go to GregDoesIt.com. Cate... blog.xtramileconsulting.com/2009/09/21/nhibernate-vs-entity-framework-a-performance-test
  30. André

    the code doesn´t work… :(

  31. Gergely Orosz

    Have you looked at the Readme.txt? You have to set up your database in order to run the examples.

  32. Dipen

    The comparison I think is not valid. It all depends on how you have written your nHibernate or EF code to fetch database. NHibernate has lot of performance options like FetchMode->Join, First and Second Level Caching.

    At the end it all depends on how you use the tool and not on the tool itself.

  33. [...] vs. Entity Framework Sapiando em uns blogs por ai encontrei um post bem interessante. Apesar de não mui... hesenger.com/blog/nhibernate-vs-entity-framework
  34. [...] NHibernate, make sure to read this post on InfoQ (I especially recommend the comments section) and this ... softwarearchitectblog.wordpress.com/2010/04/18/entity-framework-4-0-is-finally-useful
  35. Manju

    first of all thanks for the code :)
    After executing the code I got the results under bin\Results. The file name is TestResults_EntityFramework_22_11154_withoutIndex.txt.

    Here what does 22 and 11154 represent? Can you please help me

  36. Eric

    What i would like to know, is the time to write your thow benchmark application.

  37. Muhammad Gouda

    The comparison will vary in case of larger databases. e.g. 200 table or more with more complex relationships just like those in real life. My experience with EF says it will fail in large systems

  38. [...] NHibernate vs Entity Framework: http://foofish.org/blog/?p=57NHibernate vs Entity Framework performance ... robertcorvus.com/2009/12/pros-and-cons-of-nhibernate
  39. [...] year () is almost the same period, and have a the elder brothers on the EF vs NHPerformance test.Similar... unknownerror.net/2011-09/54887-ormbattle-entity-framework-4-vs-nhibernate-3-2-lists-some-points.html
  40. masoud ramezani

    It is not true. I work with both of them in a real environment. If you use NHibenate truely, it is very better than EF.

  41. Gergely Orosz

    Masoud: have you read the follow-up post? http://gregdoesit.com/2009/08/nhibernate-vs-entity-framework-%E2%80%93-a-revised-performance-comparison/ I *never* stated that EF is better/faster than NH or the other way around. Please take the time to read before imagining things that aren’t in the post.

  42. Trackback: Akkuschrauber
    Akkuschrauber... [...]Greg Does IT » Blog Archive » NHibernate vs Entity Framewo... giftsellers.net/mild-is-the-enemy-of-intruders
  43. Jose Joaquín

    http://ayende.com/blog/4351/nhibernate-vs-entity-framework-4-0

    Write batching – NHibernate can be configured to batch all writes to the database so that when you need to write several statements to the database, NHibernate will only make a single round trip, instead of going to the database per each statement.