PermGen out-of-memory, EasyMock and JUnit runner interactions #366

johanhaleby · 2015-07-24T14:31:45Z

From dmeibu...@gmail.com on September 07, 2011 14:56:34

What steps will reproduce the problem? 1. Run a large set of mocking unit tests that use @PrepareForTest. What is the expected output? What do you see instead? Result is a PermGen OutOfMemoryException. What version of the product are you using? On what operating system? PowerMock 1.4.10 which has many of the fixes for memory issues, especially Issue 207 . EasyMock 3.0, with a patched ClassProxyFactory as attached to EasyMock issue: Memory leaks in EasyMock - ID: 3145906 (referred to from Issue 207 ).

OS (Windows 32 bit/Linux 64 bit) makes some difference to how large your -XXMaxPermSize needs to be to avoid the OOM. However, with large enough set of tests (like we have) the number becomes infeasible (>1G on 32-bit for example). Please provide any additional information below. I used Eclipse MAT to analyse a number of heap dumps to investigate this issue. There are obvious complications - the tests are run as "plugin tests", hence use OSGi classloading.

After the fixes and patches above, it came down to small set of class references that were preventing the garbage collection of MockClassLoader instances. These class references are the values in the @PrepareForTest, @RunWith annotations themselves! The culprit is the JUnit Description class with contains an array of annotations from the test class. With the PowerMock runner (we're using the Junit44 delegate variant), these annotations are created with the MockClassLoader, hence the class references are from this class loader. The Description instances were then being collected by the Eclipse JDT JUnit runner to track the tests that had been run - presumably other runners similarly keep track of tests this way.

The workaround/fix we've implemented is to use Whitebox to set the fAnnotations field in the Description for the test after the run. We already had subclassed the PowerMockRunner, so it was easy to do this.

@Override
public void run(final RunNotifier notifier) {
    Description description = getDescription();
    try {
        super.run(notifier);
    }
    finally {
        Whitebox.setInternalState(description, "fAnnotations", new Annotation[] {});
    }
}

Note that getDescription() will NPE if called after super.run(). PowerMock already was trying to free references.

A more permanent fix should be possible in PowerMock doing something similar.

Original issue: http://code.google.com/p/powermock/issues/detail?id=346

The text was updated successfully, but these errors were encountered:

johanhaleby · 2015-07-24T14:31:46Z

From johan.ha...@gmail.com on September 10, 2011 11:43:18

Hi,

Thanks A LOT for you're detailed description and the time it must have taken you to analyze this. I'll add the "try-finally" fix to PowerMock soon.

Awesome work! Thanks!

Status: Accepted
Labels: -Type-Defect Type-Enhancement

johanhaleby · 2015-07-24T14:31:52Z

From johan.ha...@gmail.com on September 14, 2011 01:10:25

I've now applied it to trunk. Please verify whether it works. Thanks again for your help!

Status: Fixed

johanhaleby · 2015-07-24T14:31:58Z

From rlag...@gmail.com on September 14, 2011 08:38:28

Hi,

Thanks to the reporter for the good job. May I know in which release (when) this fix will be available for those not working with the trunk version?

Thanks you
Rodrigue

johanhaleby · 2015-07-24T14:32:03Z

From johan.ha...@gmail.com on September 14, 2011 10:46:23

It will be available in the next release for sure but it'll probably take a while before it's released :/

johanhaleby · 2015-07-24T14:32:09Z

From rlag...@gmail.com on September 14, 2011 17:59:17

Hi Johan,

I just tested the quick fix suggested by Sprynter, and it doesn't work. Did you use exact the same solution?.

Thank you

johanhaleby · 2015-07-24T14:32:15Z

From dmeibu...@gmail.com on September 14, 2011 21:38:06

Note that you also need the EasyMock patch mentioned above. It is crucial.

The fix here to PowerMock is highly dependent on the JUnit runner being used. It fixes a reference leak that becomes apparent with the Eclipse JDT runner (specifically, Eclipse 3.5.x - not sure about other versions or other runners). In our specific case, this runner was being used with the Maven Tycho plugin.

johanhaleby · 2015-07-24T14:32:20Z

From johan.ha...@gmail.com on September 14, 2011 23:10:14

Yeah I know. Unfortunately there's not much I can do about EasyMock . We should try to bring attention to the issue by upvoting the issue if possible and talk to the EasyMock guys to patch it.

johanhaleby · 2015-07-24T14:32:26Z

From rlag...@gmail.com on September 15, 2011 04:36:27

Hi Sprynter and Johan,

And where do I find the path?

Thank you

johanhaleby · 2015-07-24T14:32:32Z

From dmeibu...@gmail.com on September 15, 2011 04:53:29

EasyMock patch: http://sourceforge.net/tracker/?func=detail&aid=3145906&group_id=82958&atid=567837

johanhaleby · 2015-07-24T14:32:37Z

From rlag...@gmail.com on September 16, 2011 05:41:01

Hi Sprynter and Johan,

just want to thank you so much for the awesome work on this issue. Your solutions work. I just applied the patch manually on easymock-3.0 source code.

You saved me a lot of time. I'm glad to close this problem (I was fighting on it since 5 days without success). :-)

Thank you.

johanhaleby · 2015-07-24T14:32:43Z

From johan.ha...@gmail.com on September 16, 2011 05:51:25

Glad it works for you. Perhaps you could make the EasyMock community aware of this by sending them an e-mail on their mailing-list.

johanhaleby · 2015-07-24T14:32:49Z

From amanriqu...@gmail.com on January 17, 2012 05:32:47

Hi,

I still have problems with the last version of powermock-mockito-1.4.11-full. In change logs says that this is issue is fixed ( Fixed an OutOfMemory issue in PowerMockRunner ( issue #346 ) ) but I still have the same problem. This is the stack trace:

java.lang.OutOfMemoryError: PermGen space
at org.mockito.internal.stubbing.defaultanswers.ReturnsMoreEmptyValues.(ReturnsMoreEmptyValues.java:47)
at org.mockito.internal.stubbing.defaultanswers.ReturnsSmartNulls.(ReturnsSmartNulls.java:60)
at org.mockito.Answers.(Answers.java:28)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.lang.Class.getEnumConstantsShared(Class.java:2942)
at java.lang.Class.enumConstantDirectory(Class.java:2963)
at java.lang.Enum.valueOf(Enum.java:191)
at sun.reflect.annotation.AnnotationParser.parseEnumValue(AnnotationParser.java:413)
at sun.reflect.annotation.AnnotationParser.parseMemberValue(AnnotationParser.java:278)
at java.lang.reflect.Method.getDefaultValue(Method.java:720)
at sun.reflect.annotation.AnnotationType.(AnnotationType.java:99)
at sun.reflect.annotation.AnnotationType.getInstance(AnnotationType.java:66)
at sun.reflect.annotation.AnnotationParser.parseAnnotation(AnnotationParser.java:202)
at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:69)
at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:52)
at java.lang.reflect.Field.declaredAnnotations(Field.java:1014)
at java.lang.reflect.Field.getAnnotation(Field.java:1000)
at java.lang.reflect.AccessibleObject.isAnnotationPresent(AccessibleObject.java:168)
at org.powermock.reflect.internal.matcherstrategies.FieldAnnotationMatcherStrategy.matches(FieldAnnotationMatcherStrategy.java:37)
at org.powermock.reflect.internal.WhiteboxImpl.findAllFieldsUsingStrategy(WhiteboxImpl.java:535)
at org.powermock.reflect.internal.WhiteboxImpl.getFieldsAnnotatedWith(WhiteboxImpl.java:2343)
at org.powermock.reflect.internal.WhiteboxImpl.getFieldsAnnotatedWith(WhiteboxImpl.java:2326)
at org.powermock.reflect.Whitebox.getFieldsAnnotatedWith(Whitebox.java:568)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit47RunnerDelegateImpl$PowerMockJUnit47MethodRunner.executeTest(PowerMockJUnit47RunnerDelegateImpl.java:70)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runBeforesThenTestThenAfters(PowerMockJUnit44RunnerDelegateImpl.java:284)
at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:84)
at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:49)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.invokeTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:209)
at org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl.runMethods(PowerMockJUnit44RunnerDelegateImpl.java:148)

johanhaleby · 2015-07-24T14:32:55Z

From johan.ha...@gmail.com on January 17, 2012 07:39:50

Some OOM issues have been resolved and EasyMock has accepted one of my patches that solved an OOM issue in EasyMock itself. There may be more. Please help out!

johanhaleby · 2015-07-24T14:33:01Z

From johan.ha...@gmail.com on January 17, 2012 07:40:14

Did you get the OOM issue in version 1.4.10 as well?

johanhaleby · 2015-07-24T14:33:06Z

From amanriqu...@gmail.com on January 17, 2012 09:05:40

Yes, the same issue in 1.4.10 than in 1.4.11. I run 458 unit tests that are mocking a lot of things ... each one of them use @PrepareForTest... If I run this tests in diferent executions there is no problem ... Should I apply some other patch?? or there is still a lot of work to do to fix those memory leaks ?? thanks

johanhaleby · 2015-07-24T14:33:12Z

From johan.ha...@gmail.com on January 17, 2012 12:15:59

Tracking down and fixing memory issues is very hard and takes a lot of time. I've spent a huge amount of time trying to detect and fix memory leaks in PowerMock but I suppose I haven't found all of them. I don't have time to track this down myself so I really would need help from the community to fix the remaining issues.

johanhaleby · 2015-07-24T14:33:18Z

From torbjor...@gmail.com on May 14, 2012 09:18:14

Just a comment:

I was experiencing the same problem (with PowerMock 1.4.12), with PermGenSpace errors when building with Maven(2). I monitored my build with VisualVM, and noticed that Maven was far from running out of PermGenSpace. So, i examined the error message a bit closer, and noticed that it was SureFire that threw the error. After a bit of googleing this error message, I found that the problem was related to Maven's thread management, so I built with "mvn clean install -DforkMode=never". Now, I did not get a PermGenSpace error, but instead some of the tests failed (they passed when running in Intellij IDEA.)

I looked into the surefire report, and saw that something was wrong with the way something was mocked (I will get back to this), so i refactored the test, and ran "mvn clean install -DforkMode=never" again. The project built successfully. Then i tried running "mvn clean install" (without the forkMode parameter). This also failed with PermGenSpace, but after a different testclass than the original error.

So, the structure of one my tests was actually the culprit, combined with Maven/Surefires fork settings.

So, the first error first:

In the project I am working on, there is a structure where domain classes come in two versions, one annotated with JPA-annotations, used with Hibernate, and one for use elsewhere in the system (don't ask me why..). In between there are converter classes, which basically writes the contents of the variables of one class over to the other (the are basically identical). These converters are also static factories of themselves, so one gains an instance by calling _Converter.convertWithAllRelations(). Then, .convert(_ from) is called on this instance (where *** = the class in question).

When testing this, I tried the following (maybe not that thought through):

//set up the testclass with testrunner and preparefortest

    final ***Converter converter = mock(***Converter.class);

   PowerMockito.mockStatic(***Converter.class);
   PowerMockito.when(***Converterer.converWithReducedRelations()).thenReturn(converter);
    when(converter.convert(***from)).thenReturn(***to);

So, basically a static mock of the factory method, telling it to return a mocked instance when called.

This worked in IDEA, but with Maven, not too good.

Now, on to the second testclass that made a PermGenSpace error:

This turned out to be a bit trickier. I first tried commenting out the tests i added, leaving the class as it was, only with the PowerMockRunner testrunner and a preparefortest annotation (which I also added). This too resulted in a PermGenSpace error. So the problem was in one of the existing tests. So, I added @ignore to all tests, and no PermGenSpace error was thrown. I then continued to unignore one test at the time, to try and figure out which test(s) lead to the error. Having any one of the tests unignored spawned the error, so I tried with a dummy test (assertTrue(true)), and the same happened. So it was something in my @Before-method. With the contents of the @before method all commented out, my dummy-test ran without error. So, I narrowed it down line by line, and found that it was one of the mock(Something.class) statements that caused the error.

The class to be mocked differed from most of the other classes in one aspect: It implemented Comparable. So, I made a dummy class that did the same, and tried to mock that, and quite surely, PermGenSpace error (tried using both Mockito and PowerMockito's mock method). The problem seems related to the @PrepareForTest-annotation, it worked with only the @RunWith(PowerMockRunner.class), and failed when PrepareForTest was added (tried both with and without the class to be mocked in the list of classes to be prepared.)

This turned into quite a long comment, but hopefully it will be of some help to others experiencing similar problems. To sum it up:

If your tests run fine in your IDE, but you get memory errors when running with Maven, try running it with -DforkMode=never, as it might uncover the true error. Also, investigate the test being run right before the error occurs, as it is probably the culprit.
Mocking both static and instance parts of the same class is troublesome with PowerMock (or maybe I just did it wrong).
Mocking classes that implement Comparable does not work when using the @PrepareForTest-annotation(this should perhaps be an issue itself).

johanhaleby · 2015-07-24T14:33:24Z

From torbjor...@gmail.com on May 14, 2012 10:15:58

After some more trial and error:

It seems that I was wrong about the problem with Comparable, it gave a PermGenSpace error no matter what i mocked (other than util-classes). I guess this is because it really was low on PermGenSpace.

It seems to me that PowerMock has higher demands for PermGenSpace than Surefire delivers as standard (64m isn't it?). So the solution, that has been working so far at least, was to give the plugin some more memory by adding a line in the pom.xml:

  \<plugin>
    \<artifactId>maven-surefire-plugin</artifactId>
    \<configuration>
      \<argLine>-XX:PermSize=128m -XX:MaxPermSize=512m</argLine>
    \</configuration>
  \</plugin>

(If you have a multi-module project, with redefinition of the plugin, add it to the pom.xml closest to your tests.)

johanhaleby · 2015-07-24T14:33:29Z

From johan.ha...@gmail.com on May 14, 2012 10:18:13

Thanks for sharing your comments. There could be a memory leak lurking around somewhere they can be very difficult to find.

johanhaleby · 2015-07-24T14:33:35Z

From alwyn.sc...@gmail.com on July 15, 2012 07:15:52

I assume that Surefire uses its own runner to execute junit tests?

Would it not make sense to somehow incorporate the fix from sprynter into the surefire setup?

johanhaleby · 2015-07-24T14:33:41Z

From justinba...@gmail.com on July 18, 2013 00:53:27

We've got the same problem with Powermock 1.5, so it doesn't seem to be fixed. Instead of setting MaxPermSize on the surefire plugin, you can set reuseForks=false, then it will start a new JVM for each test (ie with a fresh PermGen). Otherwise you need to keep increasing MaxPermSize as more unit tests are added.

Unfortunately JaCoCo plugin (and Sonar) does not work with reuseForks=false set on the surefire plugin :-(

johanhaleby · 2015-07-24T14:33:47Z

From johan.ha...@gmail.com on July 18, 2013 04:09:04

Have you tried the version in trunk? Please try to build the latest version of powermock from source (skip tests and javadoc generation to make it quicker) and try if it works.

johanhaleby · 2015-07-24T14:33:52Z

From guru.pra...@gmail.com on January 01, 2014 03:48:26

We are using Powermock 1.5.1, the issue still exists.

johanhaleby · 2015-07-24T14:33:58Z

From cbr...@infinio.com on January 08, 2014 14:19:22

We are also having this problem and we are using 1.5.2

johanhaleby · 2015-07-24T14:34:04Z

From cbr...@infinio.com on January 08, 2014 14:35:44

The issue I am experiencing is actually this: https://code.google.com/p/powermock/issues/detail?id=207 Which seems to be fixed in the upcoming 1.6 release.

johanhaleby added enhancement imported Priority:Medium labels Jul 24, 2015

johanhaleby closed this as completed Jul 24, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PermGen out-of-memory, EasyMock and JUnit runner interactions #366

PermGen out-of-memory, EasyMock and JUnit runner interactions #366

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

PermGen out-of-memory, EasyMock and JUnit runner interactions #366

PermGen out-of-memory, EasyMock and JUnit runner interactions #366

Comments

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015

johanhaleby commented Jul 24, 2015