Redis vs RavenDB – Benchmarks for .NET Client NoSQL Solutions

September 8th, 2010 by mythz Leave a reply »

These Redis vs RavenDB benchmarks have been made into a bar chart for a better visualization.

Seeing that Redis v2.0 has been just been released and Oren Eini (aka @ayende) has just checked in performance optimization improvements that show a 2x speed improvement for raw writes in RavenDB, I thought it was a good time to do a benchmark pitting these 2 popular NoSQL data stores against each other.

Benchmarks Take 2 – Measuring write performance

For the best chance of an Apples to Apples comparison I just copied the RavenDB’s benchmarks solution project and modified it slightly only to slot in the equivalent Redis operations. The modified solution is available here. Redis was also configured to run in its ‘most safest mode’ where it keeps an append only transaction log with the fsync option so the operation does not complete until the transaction log entry is written to disk. This is so we can get Redis to closely match RavenDB’s default behaviour. Enabling this behaviour in Redis is simply a matter of uncommenting the lines below in redis.conf:

appendonly yes
appendfsync always

To use this new configuration simply run ‘redis-server.exe /path/to/redis.conf’ on the command line.
Other changes I made for these new set of benchmarks was to remove batching from the Redis benchmark since its an accidental complexity not required or useful for the Redis Client.

Here are the benchmarks with these new changes in place:

Which for this scenario show that:

Redis is 11.75x faster than RavenDB

Note: The benchmarks here are of Redis running on a Windows Server through the Linux API emulation layer - Cygwin. Expect better results when running Redis on Unix servers where it is actively developed and optimized for. It is understood that the Cygwin version of redis-server is 4-10x slower than the native Linux version so expect results to be much better in production.

I attribute the large discrepancy between Redis and RavenDB due to the fact that Redis doesn’t use batches so only pays the ‘fsync penalty’ once instead of once per batch.

The ‘appendfsync always‘ mode is not an optimal configuration for a single process since Redis has to block to wait for the transaction log entry to be written to disk, a more sane configuration would be ‘appendfsync everysec‘ which writes to the transaction log asynchronously. Running the same benchmark using the default configuration yields the following results:

Which is a 39% improvement over the previous benchmarks where now:

Redis is 16.9x faster than RavenDB

Which unless I hear otherwise? should make this the fastest NoSQL solution available for .NET or MONO clients.

Measuring raw write performance using Redis is a little unfair since it has a batchful operation MSET specifically optimized for this task. But that is just good practice, whenever you cross a process boundary you should be batching requests to minimize the number of your calls minimizing latency and maximizing performance.

Even though performance is important, its not the only metric when deciding which NoSQL database to use. If you have a lot of querying and reporting requirements that you don’t know up front then a document database like RavenDB, MongoDB or CouchDB is a better choice. Likewise if you have minimal querying requirements and performance is important than you would be better suited to using Redis – either way having a healthy array of vibrant choices available benefits everybody.

Notes about these benchmarks

Since these benchmarks just writes entities in large batches to a local Redis or RavenDB instance using a single client, I don’t consider this to be indicative of a *real-world* test rather a measure is raw write performance, i.e. How fast each client can persist 5,163 entities in their respective datastore.

A better *real-world* test would be one that accesses the server over the network using multiple concurrent clients that were benchmarking typical usage of a real-world application rather than just raw writes as done here.

So why is Redis so fast?

Based on the comments below there appears to be some confusion as to what Redis is and how it works. Redis is a high-performance data structures server written in C that operates predominantly in-memory and routinely persists to disk and maintains an Append-only transaction log file for data integrity – both of which are configurable.

For redundancy each instance has built-in support for replication so you can turn any redis instance into a slave of another, which can also be trivially configured at runtime. It also features its own Virtual Machine implementation so if your dataset exceeds your available memory, un-frequented values are swapped out to disk whilst the hot values remain in memory.

Like other high-performance network servers e.g. Nginx (the worlds fastest HTTP server), Node.js (a popular, very efficient web framework for JavaScript), Memcached, etc it achieves maximum efficiency by having each Redis instance run in a single process where all IO is asynchronous and no time is wasted context-switching between threads. To learn more about this architecture, check out Douglas Crockford (of JavaScript and JSON fame) imformative speech comparing event-loops vs threading for simulating concurrency.

It achieves concurrency by being really fast and achieves integrity by having all operations atomic. You are not just limited to the available transactions either as you can compose any combination of Redis commands together and process them atomically in a single transaction.

Effectively if you wanted to create the fastest NoSQL data store possible you would design it just like Redis and Memcached. Big kudos to @antirez for his continued relentless pursuit of optimizations resulting in Redis’s stellar performance.

The Redis Client,  JSON and the Redis Admin UI

Behind the scenes the Redis Client automatically stores the entities as JSON string values in Redis. Thanks to the ubiquitous nature of JSON I was easily able to develop a Redis Admin UI which provides a quick way to navigate and introspect your data in Redis. The Redis Admin UI runs on both .NET and Linux using Mono – A live demo is available here.

Download Benchmarks

The benchmarks (minus the dependencies) are available in ServiceStack’s svn repo.

I also have a complete download with including all dependencies available here:
http://servicestack.googlecode.com/files/NoSqlPerformance.zip (18MB)

Gaining in Popularity

Redis is sponsored by VMWare and has a vibrant pro-community behind it and been gaining a lot of popularity lately. Already with a library for every popular language in active use today, it is gaining momentum outside its Linux roots with twitter now starting to make use of it as well as popular .NET shops like the StackOverflow team taking advantage of it.

Unlike RavenDB and MongoDB which are document-orientated data stores, Redis is a ‘data structures’ server which although lacks some of the native querying functionalities found in Document DBs, encourage you to leverage its comp-sci data structures to maintain your own custom indexes to satisfy all your querying needs.

Try Redis in .NET

If these results have piqued your interest in Redis I invite you to try it out. If you don’t have a linux server handy, you can still get started by trying one of the windows server builds.

Included with ServiceStack is a feature-rich C# client which provides a familiar and easy to use C# API which like the rest of Service Stack runs on .NET and Linux with Mono.

Useful resources for using the C# .NET Client

I also have some useful documentation to help you get started:
- Designing a NoSQL Database using Redis
+ A refactored example showing how to use Redis behind a repository pattern
- Painless data migrations with schema-less NoSQL datastores and Redis
- How to create custom atomic operations in Redis
- Publish/Subscribe messaging pattern in Redis
- Achieving High Performance, Distributed Locking with Redis

Advertisement

8 comments

  1. Jeremy says:

    Of course redis is faster, it is essentially a main memory database whereas raven is essentially not. Horses for courses!

    I appreciate your post and your contribution of code but I would suggest that when benchmarking apples and oranges it would be advisable to declare them as such.

  2. mythz says:

    I already mentioned that Redis and RavenDB/MongoDB differs. Based on your comment I’ve since added the ‘Why is Redis so fast?’ section explaining the difference in more detail.

    Although Redis is obviously going to be faster for you and me – this is not the general consensus in the .NET community and these results are actually provide informative results of the difference.

    They may not be the same product/technology but they both look to serve a lot of the same use-cases. Obviously I choose to use Redis where RavenDB would be an equally good fit. The fact that we have choices benefits everybody.

  3. Ryan Heath says:

    Dennis, you seem to have copyed over the bug Simon Labrecque is talking about.
    http://ayende.com/Blog/archive/2010/08/28/ravendb-performance-optimizations.aspx#41983

    Does it make any difference when you reset the batch counter when 128 is reached?

    // Ryan

  4. mythz says:

    The 128 batch size was present in the original benchmarks so I just left it as is. I’m not sure why Oren selected the 128 batch size but I imagine that since Raven uses a Session/Unit of work client model that it would have a positive impact on performance.

    The 128 batch size is not needed by the Redis client and in fact hurts performance since it results in more network calls to achieve the same result.

    EDIT:
    Actually I see the bug now, I’ll re-run the benchmarks with the fix when I get home.

  5. Demis,
    _Did_ you configure your Redis server to write to disk on every operation (to match more closely what RavenDB is doing)?

  6. mythz says:

    The benchmarks are both using the standard configuration for both servers, so no.

    I will re-run the benchmarks with the bug fix and configure it to write on every operation when I get home tonight.

  7. mythz says:

    New benchmarks are in, I removed some of the overheads imposed on the Redis Client i.e. active entity id tracking and batching then enabled the appendonly transaction log with ‘fsync always’ configuration.

    The new benchmarks show Redis is 11.75x faster than RavenDB with this configuration.

    If you disable the append only file Redis becomes 16.9x faster than RavenDB.

    Note as I’m using the batchful MSET operation behind the scenes, the fsync penalty is only taken once.

Leave a Reply