My favorites | Sign in
Project Logo
                
Search
for
Updated Jan 12, 2009 by mcculls
Labels: Featured
Benchmarks  
Various benchmarks and other flammable content!

OSGi service proxy microbenchmark

This is a very simple (perhaps simplistic!) microbenchmark that calculates how much overhead on average the given OSGi service proxy adds to each method call. It measures how long the raw implementation takes and subtracts this from however long the proxied service took.

It does not take into account the flexibility/features of each proxy implementation, but merely looks at the overhead cost.

  1. download and unzip the service proxy microbenchmark
  2. import the contents into Eclipse as an existing project
  3. open OSGiServiceProxyTest.target and set it as the target platform:
  4. right-click on OSGiServiceProxyTest.launch and select Run As... -> OSGiServiceProxyTest
  5. you should see results similar to this:
osgi> Testing OSGi service proxy overhead...

Guice-Peaberry:	  649.00 ns/call
Spring-DM:	 3718.00 ns/call

Which shows a lot less overhead when using the peaberry service proxy.

Basic bundle footprint

This is another simple 'benchmark' where we look at the size of the minimum bundle set required to run the above testcase.

Of course Spring-DM provides a lot of additional functionality over Guice-Peaberry, but we're interested in the minimum footprint possible.

   8K com.springsource.org.aopalliance-1.0.0.jar
 632K guice-snapshot-20090105.jar
 176K peaberry-1.0-rc1.jar

816K Guice-Peaberry 1.0-rc1

   8K com.springsource.org.aopalliance-1.0.0.jar
  24K com.springsource.slf4j.api-1.5.6.jar
   8K com.springsource.slf4j.nop-1.5.6.jar
  20K com.springsource.slf4j.org.apache.commons.logging-1.5.6.jar
 320K spring-aop-2.5.6.jar
 480K spring-beans-2.5.6.jar
 468K spring-context-2.5.6.jar
 280K spring-core-2.5.6.jar
 352K spring-osgi-core-1.2.0-m2.jar
 116K spring-osgi-extender-1.2.0-m2.jar
  36K spring-osgi-io-1.2.0-m2.jar

2112K Spring-DM 1.2.0-m2

As you can see Guice-Peaberry takes up less than half the space of the smallest Spring-DM setup.


Comment by chouaieb, Jan 12, 2009

thanks, I am working on a comparison of Peaberry and Spring DM and iPOJO and the service proxy overhead should help.

Comment by Rinsvind, Jan 12, 2009

Hey if this comparison is going to be accessible online please post a message in this projects forum. I myself had to pick between iPojo and Peaberry for a project and did an ad-hoc comparison. Peaberry won :P

Comment by chouaieb, Jan 12, 2009

it's not finished yet and it's not clean. I'll post it while terminated, in my tests too peaberry was better than iPOJO, but I found that Spring DM is the most complete one.

Comment by mcculls, Jan 12, 2009

I'll look forward to reading your comparison when it's ready, in the meantime feel free to raise issues to make peaberry more complete :) Todor's lifecycle issue (10) should also help with this...

Comment by Rinsvind, Jan 13, 2009

Just as a side-note (a rather long one :)

I think the killer-mix that Peaberry offers is

- Total dependency injection: you can start from the core set of singletons and continue to DI the shorter-living generations of objects. Both iPojo and Spring DM seem to concentrate on injecting only the longest living objects - they call them components. As for the others - it is possible but not easy. I think Spring even discourages the practice.

- Seamless dynamics: Wraps the services in proxies so you can pass them around the bundle and have OSGi dynamics support everywhere. iPojo instruments only the implementation class of the component. You have to hand-code proxies if you want to pass services to collaborating objects. Lately (perhaps due to my questions about it :) Clement has extended the Temporal Dependency of iPojo to do seamless proxying but I have not had the time to check it out. Spring DM uses dynamic proxies like Peaberry as the benchmarks above testify :)

These two features allow you to introduce safe services into the bundle space and than inject them directly to the points of use. Having just safe services will make you code fire-brigades to pass them from object to object until they reach the point of use. Having only DI will plainly not work because you can not "outject" the service objects when they become invalid. But put them together the sum seems to be greater than the parts :P

Comment by Rinsvind, Jan 13, 2009

I have some questions about the design of the benchmark. I have a micro-benchmark of my own I use to test some proxying schemes but I never really trusted it. So I want to learn more about yours if you don't mind :)

1) Why use

Math.exp(val Math.cosh(val));

rather than just

val Math.cosh(val);

When I used the simpler expression the percentage differences between the things I test changed by about 5-7%. I suppose you were looking for a piece of code that has the same performance given the same input and is complicated enough to fool the JIT not to optimize it away during warm up. But why this exact expression? :)

2) Why at the end of the benchmark you don't do anything with the "value" dummy variable? The JIT seems to not pick this up and optimize the entire loop away. When I print the value on the screen nothing changes :) Yet once I used integer arithmetic for a benchmark and unless I printed the dummy accumulator on the screen the results were erratic :P

Comment by mcculls, Jan 13, 2009

About the benchmark code:

1) I use nested Math calls for two reasons: one is that it stops the JIT from doing too much optimization, another is that the function takes a reasonable amount of time without making the test take ages for thousands of calls, or create large numbers of objects which can trigger GC. But it's not ideal ;)

2) The JIT doesn't know if the compute() method has any side-effect as we're calling it via an interface, so it cannot optimize away this method call.

The most important thing is that the direct and proxied calls use the same implementation underneath - of course there are still many things that can disturb the timing (GC, background JIT compilation, OS, etc.) but I'm really looking at the order of magnitude rather than the exact numbers.

HTH

Comment by clement.escoffier, Jan 19, 2009

Hi,

I agree that the iPOJO temporal injection using dynamic proxy was a nightmare in term of performance. It was a temporary solution (and was developped as a temporary solution). As iPOJO is based on ASM, they are now using ASM (as in peaberry).

I also quickly extend the bench to add three iPOJO based benchs : - Service Dependency (field injection, not proxy) - Temporal Service Dependency (field injection, not proxy) - Proxied Temporal Dependency (ASM proxy)

Here are the results: Guice-Peaberry: 276.00 ns/call iPOJO Service Dependency: 118.00 ns/call Spring-DM: 2384.00 ns/call iPOJO Temporal Dependency: 159.00 ns/call iPOJO Temporal Dependency with proxy injection: 173.00 ns/call

If somebody is interested by the code I can put it somewhere. I just use the iPOJO API to declare my component types as peaberry does (it is closer to the Dependency Manager than peaberry).

Clement


Sign in to add a comment
Hosted by Google Code