Performance

World Cup 2010 Real Time Twitter Mashup

This World Cup has been exciting for me not just because of the great games but also because during the group stages I’ve been building a cool real time spectator monitoring application for it: Tweet For Your Team (powered by Kwwika real time data services).

What does it do?

The application streams tweets live from the two teams playing live or playing the next game (using the Kwwika service). It also shows the live score on the top (provided by Opta), so even if you’re not in front of a telly, you can be aware of it. Now, for the fun part…

tweetintensity

The app monitors the tweet intensity, that is the number of tweets per minute for each side. Hours before the game this value is typically around 5-10/minute, right before the game it’s usually at least 300/minute and from there it pretty well reflects how nerve wracking the game is. The highest number I’ve experienced up to the semi finals. was around 3500/minute for Spain, when they shot – and missed – the second penalty against Paraguay in the quarter finals.

tweetmood

Tweet intensity is an interesting indicator, but an even more useful one is the mood meter. The application tries to evaluate the mood of each tweet coming in and decide whether it is more of a positive message (“yaaay!!”) or a negative one (“boooo!”). The tweets are colored according to this: red ones mean negative, green ones positive and yellow is neutral (or at least the application couldn’t decide).

Tweet intensity tends to pretty well reflect on the game: usually when a team is scored a goal it tends to go down… though it’s hard to predict by how much – sometimes the fans just keep sending positive messages to the team even after their team has gotten behind!
Read More…

Thoughts on ranking ORM tools

Posting of the previous performance comparison results (see here) has caused a much greater response I ever expected as well as some confusion about the goal of this test that I would like to respond to.

How real (world) is this?

Most importantly a performance test is useful if it can be used for predicting performance of a real world application. Now the performance test in I did was a really primitive, batch based CRUD test. Unless you are developing an application that does batch actions such as in this test, the results can’t really be used for predicting performance of real world applications. We’re talking about two mature frameworks with lots of additional services – e.g. caching and transaction handling – that are not evaluated in this test even though they can have a huge impact on performance.

Be a little sceptic

I would suggest treating results with caution and checking on the source. There are a lot of people expert on a single ORM framework, some are experts on two but it mostly stops there. Unless stated that the author(s) is/are experts on the topic I would approach these results in a sceptic way.

As for my own case: I do have experience with the tested frameworks however am far from being an expert with any of them – so treat my results with caution. I’ve published the so anyone can analyze it and if there are some issues in the code resulting in major performance drop, hopefully someone will let me know – just as some members of the NHibernate community did after publisihing the first results.

There is no “best”

I believe there is no valid way to decide which framework is ultimately better. There is an ongoing debate about ranking ORM tools and how well a test like this is good for deciding on which framework is better. I think it is not. Just to mention a few reasons:

  • Most real world applications have a far more complex working than this test simulates. What’s more, commonly used additional framework features – e.g. caching, transaction handling – are not tested at all even though these features can have a great impact on real world performance
  • Performance may not be everything. Even if these performance tests would reflect performance in a real world environment (as they don’t), ORM is not used based on performance. When using an ORM tool you make a compromise to have worse performance than using native SQL and gain additional features. It’s usually a tradeoff: the more features you get, the bigger the overhead will be.

There is one case when this kind of test can be useful though. If you find a great performance difference between two frameworks (e.g 50-1000 times), it’s a good indicator that there’s either some issue with the test itself or the framework. If you’re sure that the test is written correctly this comparison could lead to the analysis of why the framework is unacceptably better than the other. If you can find the valid answer, then good. If you can’t, now that’s when things get interesting and the test was worth doing.

The best valid way of comparing an ORM tool in my opinion is based on the project you need it for. Write together your needs, weigh them and evaluate the frameworks accordingly. ORM tools are comparable just as cars are. You can’t decide which one’s better unless you know what to use it for. An F1 car may be great for a speed race but would utterly fail in a desert challenge.

NHibernate vs Entity Framework: a performance test

As part of my thesis I measured the performance of some .NET ORM frameworks including NHibernate and Entity Framework. Measuring was done by implementing two simple applications using the same table structure and doing the same operations on the same data.

Note: before reading this article please see my views on comparing ORM tools. Also see the follow up post to this comparison for revised results.

I measured the time it took for each framework to complete these operations:

  • store
  • read over relations
  • read by ID
  • update
  • delete

The results were somewhat interesting. Here is a short summary of what I’ve found out.

Operation \ Number of operations NHiberante – 4K Entity Framework – 4K NHiberante – 40K Entity Framework- 40K Winner
Store 37,37 9,19 1500 98 Entity Framework
Read over relations 1,01 0,54 10,13 4,18 Entity Framework
Read by ID 3,06 25,22 246 230 NHibernate with smaller amount of objects
Update 6,61 7,34 77 72 Both
Delete 3,35 16,76 58 1824 NHibernate

Read on if you’re interested in the details of the performance measurements.

Update: the source code of the program used to measure is now available for download.
Read More…

Monitoring website load

Over the weekend I’ve been in charge of monitoring one of our bigger portals which consists a database server (running SQL Server 2005) and multiple web servers (running IIS 6.0 and ASP.NET) connecting to it. The architecture of the system is the following:

System architecture diagram

I’ve chosen a really simple – and at first glance efficient – way of monitoring the system with the Windows Performance Monitor (also known as PerfMon): Read More…

 Scroll to top