Benchmark

This is a benchmark run on Amazon EC2, performed on September 24th, 2009.

Introduction

Why another lightweight HTTP library in Erlang, when there's such a popular and (may I add) wonderful Mochiweb library out there?

The answer is simple. Misultin is targeted to be more lightweight, and faster. The benchmark here below shows some comparison tests between Mochiweb and Misultin. According to this particular benchmark, Misultin is around 50-60% faster than Mochiweb.

This benchmark is definitely not exhaustive, but seems to be a good start. It takes into consideration only speed and throughput, while it is completely ignoring RAM and CPU usage, or number of supported open connections, for instance.

If you wish to run your own benchmark tests you are most welcome, and please let me know your results if you do.

An independent benchmarking which includes Misultin has been done by Lev Walkin and is available here. A more recent benchmark of a web framework which uses Misultin has been done by Mrinal Wadhwa and can be read here.

Description

The benchmark has been designed for libraries/servers built to provide dynamic content. Therefore, the HTTP servers do the following:

• accept an incoming HTTP request with the parameter value passed in GET;
• respond with an XML echoing this variable.

Method

The benchmark has been run on two large Amazon EC2 instances with Ubuntu 9.04 Jaunty Server 64-Bit. As per Amazon specs, these machines have each 7.5 GB memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB instance storage with high I/O performance.

Apache JMeter has been used to test the performances. It has been run several times, increasing the number of threads with requests totaling each time 500,000. The naming convention is the name of the library/server coupled with the number of parallel threads running to perform the requests. For instance, Misultin 10 stands for results run against the Misultin library with 10 parallel threads sending requests.

Installed on the Client machine:

• JMeter 2.3.4.

Installed on the Target machine:

• Erlang R13B02-1;
• Apache HTTP server 2.2 with PHP5.

The tests have been run against:

• Mochiweb svn r104;
• Misultin 0.3;
• Latest Apache running PHP5.

Note: PHP is not the fastest dynamic engine available for Apache, and Apache itself is not the fastest webserver either. This test is here to provide a reference for control purposes only.

Code

For Misultin

-module(misultin_bench).
-export([start/1, stop/0, handle_http/1]).

start(Port) ->
    misultin:start_link([{port, Port}, {loop, fun(Req) -> handle_http(Req) end}]).

stop() ->
    misultin:stop().

handle_http(Req) ->
    % get value parameter
    Args = Req:parse_qs(),
    Value = proplists:get_value("value", Args),
    case Value of
        undefined ->
            Req:ok([{"Content-Type", "text/xml"}], ["<http_test><error>no value specified</error></http_test>"]);
        _ ->
            Req:ok([{"Content-Type", "text/xml"}], ["<http_test><value>", Value, "</value></http_test>"])
    end.

For Mochiweb

-module(mochi_bench).
-export([start/1, stop/0, handle_http/1]).

start(Port) ->
    mochiweb_http:start([{port, Port}, {loop, fun(Req) -> handle_http(Req) end}]).

stop() ->
    mochiweb_http:stop().

handle_http(Req) ->
    % get value parameter
    Params = Req:parse_qs(),
    Value = proplists:get_value("value", Params),
    case Value of
        undefined ->
            Req:respond({200, [{"Content-Type", "text/xml"}], ["<http_test><error>no value specified</error></http_test>"]});
        _ ->
            Req:respond({200, [{"Content-Type", "text/xml"}], ["<http_test><value>", Value, "</value></http_test>"]})
    end.

For PHP on Apache

<?php
// set header
header("content-type: text/xml");
// get request
if (isset($_GET['value'])) {
	echo "<http_test><value>".$_GET["value"]."</value></http_test>";
} else {
	echo "<http_test><error>no value specified</error></http_test>";
}
?>

As it can be seen, the code is extremely similar for both Misultin and Mochiweb.

Results

Mochiweb

Misultin

Apache with PHP

Conclusions

• The least difference is observed with 1 thread, where Misultin outperforms Mochiweb only by 18%;
• at 100 and 200 parallel threads, Misultin outperforms Mochiweb by 58% (the highest difference being observed with 500 parallel threads, where misultin outperforms Mochiweb by 65%);
• Misultin is an average 50-60% faster than Mochiweb;
• Standard Deviation is considerably lower in Misultin.

To Do

More deep tests:

• different HTTP server funcionalities;
• provide also memory, CPU usage, number of open connections,...

Footnote

Don't get me wrong here. I believe Mochiweb is a wonderful library and it's programmer, Bob Ippolito, has done a great job in enriching it with useful features. As a matter of fact, some of Mochiweb code is also included in Misultin. Though, I needed to get as fast as possible with an HTTP library, thus here it is.

Also, I wish to thank Michele Sciabarrà for his contribution to the realization of this benchmark.

Enjoy.

>-|-|-(°>