Great stuff. A couple of quick nit-pick items (I'd happily fix but I don't have wiki edit permissions):

- Would be good to indicate the axis on the graphs, if only to say whether longer bars are better or worse. - Also might be useful to break up the graphs with a horizontal line or something, it's hard to tell whether the text goes with graph above or below it. - "Tree major contributions" => "Three major contributions" - "May very a lot" => "May vary a lot"

One minor comment (which I can help with): perhaps also simple sum of deser + ser, since oftentimes this is the overall time taken, either by one side of conversatio n (server or client) or by both combined.

I have also noticed that results vary a lot between different machines, my older home workstation giving totally different numbers from work machine.

I did some change (for other result)

optimise Xpp to generate compactXml instead of Pretty/indented allow XStream to select the implementation of Stax (use Woodstox by default) fix label of xstream configuration

Would be nice if you compare against the optimize-for-speed version of protobuf, which is not the default.

results from protobuf are with "optimize-for speed".

use the mailing-list/group

I'd love to see how Hessian (http://hessian.caucho.com/) and Ice (http://www.zeroc.com/ice.html) stack up here. Awesome work guys.

Joselo: "optimize-for-speed" is on. Without it, PB is much slower (for me, 3x).

tmnichols: I can add Hessian, should be simple (done it before)

One minor change: I removed 10x multiplier for object creation, otherwise it'll skew results (since you don't create 10 instances, serializer just one). I think it was originally just used to make run time long enough.

Due to change in object creation, here's numbers I get on my work station, for reference (eishay can update 'official' results too):

                        ,   Object create,   Serialization, Deserialization,      Total Time, Serialized Size
protobuf                ,        59.19145,       537.94075,       294.13640,       891.26860,        217
thrift                  ,        52.96210,       494.54925,       522.01160,      1069.52295,        314
java                    ,        48.08995,      1736.61825,      7258.44035,      9043.14855,        845
java (externalizable)   ,        46.87810,       690.94225,      1870.71525,      2608.53560,        315
scala                   ,        45.36270,      4018.60430,     19122.84775,     23186.81475,       1950
stax/woodstox           ,        48.64915,       751.47950,      1018.75335,      1818.88200,        406
stax/aalto              ,        48.75460,       543.86225,       785.45220,      1378.06905,        406
binaryxml/FI            ,        48.73490,      1550.54480,      1537.80335,      3137.08305,        224
json (jackson)          ,        48.67715,       382.99400,       593.11015,      1024.78130,        310
xstream (xpp)           ,        48.57065,      9741.56200,     17444.81985,     27234.95250,        759
xstream (xpp with conv) ,        48.66520,      1704.66610,      5507.66675,      7260.99805,        287
xstream (stax)          ,        48.84445,      9799.39625,     15580.16955,     25428.41025,        797
xstream (stax with conv),        48.62815,      1440.40925,      2615.74715,      4104.78455,        325
javolution xmlformat    ,        48.96825,       535.32845,      1315.22985,      1899.52655,        345
sbinary                 ,        45.05305,       406.14195,       386.88480,       838.07980,        190

Hmmh. I think Xstream(stax) has something funny going on, since serialized size differs from that of Xstream(xpp). Perhaps name aliasing is not done with stax? Otherwise sizes should be the same.

One more thing: it would be great if during warmup roundtrip checking was done: start with object, serialize, deserialize, verify that results match. This to ensure ser/deser are not broken -- while serialized size gives some indication, it's not reliable test.

for Java VM based tests, it'd be helpful if you mention which version of the JDK. There seems to be a big difference between 1.5 and 1.6

Sorry for late response, didn't get notifications on the comments. In the future please comment on the project mailing list.

@darugar thanks! fixed

@tsaloranta isn't it the "Total Time" ?

@joselo we are using optimize for speed in this benchmark. The difference is very large, check out this post for more info: http://www.eishay.com/2008/11/protobuf-with-option-optimize-for-speed.html

@tmnichols contributions are happily accepted, shoot me an email or tweet @eishay if you have some time to work on it

@tsaloranta very interesting, I'll check it out when I'll have some time. protobuf seems to be now faster then json, I know some people who will be delighted :-)

@arnieg0001 we used java6 but note that the results are not absolute but a comparison. I guess (though need to verify) that the jvm version should effect them proportionally.

Please include also memory usage. Very often it is possible to serialize 1 GB of object structures, but deserialization will fail with OutOfMemory? or simular exception.

That's a bit hard to do. Any suggestions?

Memory usage measurementis non-trivial, but more importantly I am not sure I see the point. Why? Because most impls (and all fast ones) are effectively streaming/chunking, so the only thing retained in memory are the business objects (result or input). And those should have about the same size.

That could expose some flaws/suboptimalities in implementations, perhaps that would be the point? And others would work up until point where memory is used by "legitimate" objects.

The number is in 'nanosecond' but I see the following in the code

return iterationTime(delta) / 10d;

So, the numbers are scaled wrong???

Thanks @stinkyminky, you are right, there was a merge clobber which left behind some buggy code. Fixing it and loading the correct data withthe latest results from @tsaloranta additional serializer code.

Eishay, thanks for putting these benchmarks together. I do some work on Thrift and I am looking into the benchmarks to understand some things that don't quite fit.

I posted a bit of a report and some follow-up to the thrift-dev mailing list.

http://mail-archives.apache.org/mod_mbox/incubator-thrift-dev/200905.mbox/%3CC62E9622.97C4C%25cwalter@microsoft.com%3E

You can see more follow-up on the rest of the mailing list archives there too -- just look for posts with the title "Report on thrift-protobuf-compare" in http://mail-archives.apache.org/mod_mbox/incubator-thrift-dev/200905.mbox/thread

The main upshot is that: 1. You should update to a more recent version of Thrift and use TCompactProtocol 2. The API you have chosen around byte puts Thrift at a bit of disadvantage since it doesn't support that directly. Also, the way that you serialize once and then deserialize 10000 times doesn't fit well with Thrift's transport models, which are more designed for real world RPC cases. 3. By making some fairly minor modifications to Thrift and the benchmark program, I was able to get Thrift to perform at roughly the same level as protocol buffers, albeit with some modest room for improvement on deserialization that I am still investigating 4. The biggest issue is that the dynamic serializations are not being tested in a manner that gives a true apples-to-apples comparison with Thrift, protocol buffers, and other statically generated systems. This makes Thrift and protocol buffers look like they perform no better or even worse than, say, JSON, but this is not an accurate reflection of the underlying realities because the JSON serialization and deserialization implementations are not actually correct. You can see this by adding another person to the list in StdMediaSerializer?::create(). I am happy to discuss further.

Let's chat more about this and see if there are ways to get an accurate picture of the benchmarks that puts each implementation in its best possible light without compromising correctness.

Cheers,

Chad

Hi Chad,

That's an awesome feedback, thanks!

There are indeed few things we need to fix, some are obvious like the URLs in the java std serializer and some are less like the way dynamic serializers are handling lists (a good point which I totally agree on). I also agree we should exploit the serializers as much as possible for best performance, but without being unfair to the other serializers. Since you invested so much in it, I hope you could help me a bit more. Could you please start a discussion in the project's group and cross link it with the one at the thrift-dev list? As for fixing / enhancing the code, right now I am a bit overloaded but I promise to work on it when I'll have time. Could you please post appropriate bugs? Or even better, if you have the time and will take the oath of being fair to other serializers then I'll be happy to make you a committer.

As a side note, usage patterns and data set may have a huge effect of any of the serializers performances. Given that, the code samples of how do the APIs looks like and how to use them correctly might be more valuable them the numbers in the wiki.

Eishay, let me look into what is feasible in terms of time commitment and also code contribution. I can certainly file some bugs and propose some enhancements.

I added two issues. Issue #4 is much much more significant. The way things are currently is giving people the incorrect perception that, say, "json performs as well or better than protocol buffers and Thrift". However, the benchmarks only show this because the json implementation is not doing a proper serialization and deserialization. Just because you can write one object into a string and then read that string back to produce the same object does not mean that you have a proper implementation of serialization and deserialization -- you need to be able to do the same for a variety of similar objects and many of these implementations do not stand up under what should be non-material changes to the input object.

Thanks Chad!

FWIW, I improved json and stax/xml serializers to resolve issue #4 regarding these serializers. Same could and should be done for other serializers that have the issue. For me the numbers didn't seem to change a lot, but it'd be good to run 'official' numbers using same setup as with earlier published numbers.

Updated the page with latest results (using Java6), added the new serializers.

Hello,

Where can I download the source for those performance tests?

Thanks Tai

Sure, go to: http://code.google.com/p/thrift-protobuf-compare/source/checkout

Added JsonMarshaller? serializer (by Pascal-Louis Perez) results http://code.google.com/p/jsonmarshaller/

Added Kryo results: http://code.google.com/p/kryo/ Updated the page to latest results, which include some bug fixes important to accurate comparison. Also sorted the charts from smallest to largest, since smaller is always better this makes it easier to compare.