My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 2003: java.lang.OutOfMemoryError: Java heap space
47 people starred this issue and may be notified of changes. Back to list
Status:  Released
Owner:  ----
Closed:  Nov 2013


Sign in to add a comment
 
Reported by nikki.ka...@gmail.com, Jul 9, 2013

Affected Version: 2.6.1

What steps will reproduce the problem?
1. SSH access to gerrit hangs
2.
3.

What is the expected output? What do you see instead?
Error_log shows:
ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user tudosl279 account 43) during git-receive-pack '/psogate-jrules'
com.google.gerrit.sshd.BaseCommand$Failure: fatal: Unpack error, check server log
	at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:159)
	at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
	at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
	at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)
	at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:429)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: Unpack error on project "psogate-jrules":
  AdvertiseRefsHook: org.eclipse.jgit.transport.AdvertiseRefsHookChain@5b541889class org.eclipse.jgit.transport.AdvertiseRefsHookChain

	at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:158)
	... 13 more
Caused by: org.eclipse.jgit.errors.UnpackException: Exception while parsing pack stream
	at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:239)
	at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:160)
	at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:100)
	... 13 more
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.eclipse.jgit.internal.storage.file.PackIndexV2.<init>(PackIndexV2.java:123)
	at org.eclipse.jgit.internal.storage.file.PackIndex.read(PackIndex.java:137)
	at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:96)
	at org.eclipse.jgit.internal.storage.file.PackFile.idx(PackFile.java:167)
	at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:253)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject1(ObjectDirectory.java:371)
	at org.eclipse.jgit.internal.storage.file.CachedObjectDirectory.openObject1(CachedObjectDirectory.java:199)
	at org.eclipse.jgit.internal.storage.file.FileObjectDatabase.openObjectImpl1(FileObjectDatabase.java:173)
	at org.eclipse.jgit.internal.storage.file.CachedObjectDirectory.openObject(CachedObjectDirectory.java:191)
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:145)
	at org.eclipse.jgit.transport.PackParser.verifySafeObject(PackParser.java:1010)
	at org.eclipse.jgit.transport.PackParser.whole(PackParser.java:984)
	at org.eclipse.jgit.transport.PackParser.indexOneObject(PackParser.java:902)
	at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:473)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:179)
	at org.eclipse.jgit.transport.BaseReceivePack.receivePack(BaseReceivePack.java:938)
	at org.eclipse.jgit.transport.BaseReceivePack.receivePackAndCheckConnectivity(BaseReceivePack.java:766)
	at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:191)
	at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:160)
	at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:100)
	at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
	at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
	at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)
	at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:429)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[2013-07-09 07:24:54,192] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user stefana330 account 69) during git-upload-pack '/pepsi-main'
java.lang.OutOfMemoryError: Java heap space
	at org.eclipse.jgit.internal.storage.file.PackIndexV2.<init>(PackIndexV2.java:124)
	at org.eclipse.jgit.internal.storage.file.PackIndex.read(PackIndex.java:137)
	at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:96)
	at org.eclipse.jgit.internal.storage.file.PackFile.idx(PackFile.java:167)
	at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:253)
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject1(ObjectDirectory.java:371)
	at org.eclipse.jgit.internal.storage.file.FileObjectDatabase.openObjectImpl1(FileObjectDatabase.java:184)
	at org.eclipse.jgit.internal.storage.file.FileObjectDatabase.openObject(FileObjectDatabase.java:158)
	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:145)
	at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:229)
	at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:809)
	at org.eclipse.jgit.internal.storage.file.RefDirectory.doPeel(RefDirectory.java:482)
	at org.eclipse.jgit.internal.storage.file.RefDirectory.peel(RefDirectory.java:461)
	at org.eclipse.jgit.lib.Repository.peel(Repository.java:950)
	at org.eclipse.jgit.transport.RefAdvertiser.send(RefAdvertiser.java:173)
	at org.eclipse.jgit.transport.UploadPack.sendAdvertisedRefs(UploadPack.java:691)
	at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:554)
	at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:521)
	at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:57)
	at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
	at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
	at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)
	at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:429)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
[2013-07-09 07:24:08,174] WARN  / : Error in changeDetail
java.lang.OutOfMemoryError: Java heap space
	at com.googlecode.prolog_cafe.lang.InternalDatabase.<init>(InternalDatabase.java:41)
	at com.googlecode.prolog_cafe.lang.Prolog.<init>(Prolog.java:184)
	at com.googlecode.prolog_cafe.lang.PrologControl.<init>(PrologControl.java:31)
	at com.googlecode.prolog_cafe.lang.BufferingPrologControl.<init>(BufferingPrologControl.java:23)
	at com.google.gerrit.rules.PrologEnvironment.<init>(PrologEnvironment.java:65)
	at sun.reflect.GeneratedConstructorAccessor75.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
	at com.google.inject.internal.DefaultConstructionProxyFactory$2.newInstance(DefaultConstructionProxyFactory.java:85)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254)
	at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
	at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
	at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:632)
	at $Proxy41.create(Unknown Source)
	at com.google.gerrit.server.project.ProjectState.newPrologEnvironment(ProjectState.java:175)
	at com.google.gerrit.server.project.SubmitRuleEvaluator.runSubmitFilters(SubmitRuleEvaluator.java:199)
	at com.google.gerrit.server.project.SubmitRuleEvaluator.evaluate(SubmitRuleEvaluator.java:155)
	at com.google.gerrit.server.project.ChangeControl.canSubmit(ChangeControl.java:358)
	at com.google.gerrit.server.project.ChangeControl.getSubmitRecords(ChangeControl.java:320)
	at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory.java:143)
	at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory.java:63)
	at com.google.gerrit.httpd.rpc.Handler.to(Handler.java:65)
	at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailServiceImpl.changeDetail(ChangeDetailServiceImpl.java:47)
	at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91)
	at com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:379)
	at com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:265)
	at com.google.gerrit.httpd.rpc.GerritJsonServlet.service(GerritJsonServlet.java:120)
[2013-07-09 07:23:48,352] WARN  org.eclipse.jetty.servlet.ServletHandler : Error for /changes/8127/revisions/1/files/%2FCOMMIT_MSG/reviewed
java.lang.OutOfMemoryError: Java heap space
	at java.util.ArrayList.<init>(ArrayList.java:132)
	at org.eclipse.jgit.util.TemporaryBuffer.reset(TemporaryBuffer.java:277)
	at org.eclipse.jgit.util.TemporaryBuffer.<init>(TemporaryBuffer.java:93)
	at org.eclipse.jgit.util.TemporaryBuffer$Heap.<init>(TemporaryBuffer.java:499)
	at com.google.gerrit.httpd.restapi.RestApiServlet.heap(RestApiServlet.java:857)
	at com.google.gerrit.httpd.restapi.RestApiServlet.replyJson(RestApiServlet.java:536)
	at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:314)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
	at com.google.gerrit.pgm.http.jetty.GetUserFilter.doFilter(GetUserFilter.java:76)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:70)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)

Please provide any additional information below.


Jul 11, 2013
#1 gavinswa...@gmail.com
Seeing this in 2.7-rc2 as well
Jul 15, 2013
#2 nikki.ka...@gmail.com
This has happened 3-4 times now (in the last 1 week). Is this a bug or am I missing some configuration here?
Jul 15, 2013
#3 sp3...@gmail.com
IMO a bug.. I've had v2.5.2 before and it worked fine. Now I observe OOM two ~ three times a day. This is major IMO not minor.
Jul 15, 2013
#4 gavinswa...@gmail.com
I'd be glad to provide more info on this problem. It's very irritating to restart gerrit 2-3 times a day. I too was running the 2.5 series for a while without this issue. I upgraded to 2.7-rc2 via 2.6 rather than going straight from 2.5 to 2.7.
Jul 17, 2013
#5 manuel.c...@gmail.com
We also see OOME about every 18-24 hours.
Jul 24, 2013
#6 gavinswa...@gmail.com
Has anyone tried 2.7-rc3 maybe this was miraculously fixed?
Jul 29, 2013
#7 jtolds
Yeah this is killing me. I'm nervous about upgrading to a release candidate to try and get my system more stable.
Jul 29, 2013
#8 gavinswa...@gmail.com
Last week I fixed this problem with my Gerrit-Jenkins setup:
https://issues.jenkins-ci.org/browse/JENKINS-18338

I came back into work today and Gerrit is still running strong, with no heap issues.
My theory is that when the Gerrit-Trigger connection from Jenkins was denied due to permissions something wasn't cleaned up properly, and over time this caused the heap overrun issue. This is totally a theory, but I haven't changed anything else related to Gerrit.

2.7-rc2
Jul 29, 2013
#9 jtolds
@#8: I am also running Jenkins. I fixed that issue weeks ago (JENKINS-18338). This bug (#2003) is definitely still happening for me.
Aug 7, 2013
#10 thomas.w...@gmail.com
I'm having the same problem (gerrit 2.6.1). It fails after about 2 days of operation. I end up with 500K of sshd server sessions. 

Seems that the session objcts are lingering in the heap and won't be GC:ed.   

Attaching dominator-tree from heapdump.

 Please, can someone from the gerrit dev-team share some info in this matter? 

Ps.
Noticed the MINA sshd-core was bumped in this gerrit release from 0.5.0 -> 0.6.0 
Ds.
sshd_server_session.txt
3.1 KB   View   Download
Aug 7, 2013
#11 icee...@googlemail.com
I had a similar problem that was caused by the jenkins-gerrit-trigger plugin, as the jenkins user did not have streamevents rights (new in 2.6, see https://code.google.com/p/gerrit/issues/detail?id=1886).
Aug 7, 2013
#12 gavinswa...@gmail.com
I agree the gerrit trigger permission is a work around. But it's not a fix.
Aug 8, 2013
#13 thomb...@gmail.com
the gerrit trigger permission, did not resolve the issue for us.

we've now set the sshd.idleTimeout to a lower value (5 min), which seems to help with the open ssh sessions. (see https://gerrit-review.googlesource.com/#/c/44351/)

so far so good ... memory consumption seems to be more stable now
Aug 9, 2013
#14 thomas.w...@gmail.com
Thanks for the tip! I'm also  running with the idle-timeout (at 5min ) option and it looks quite promising. 5min should be sufficient enough in our setup as well .

Was skeptical at first due to that all TCP-connections was long gone, so I thought that it would be impotent try cleaning up the stale sessions, whithout knowing the rulset for the cleaning-task.

After 6h the dominator is not the 'delayqueue' anymore. Comparing heapdumps reveals no tendency for ramp up new sessions either, it seems quite stable.

Worth to know, currently the default timeout is ~24 days!! or idleTimeoutSeconds = MILLISECONDS.toSeconds(Integer.MAX_VALUE); 
	
Wonder why they didn't mentioned this any clearer in the documentation this was not an option at all in 2.5  so obvious in retrospect....
Oct 2, 2013
Project Member #15 dougk....@gmail.com
Apparently, 75,000 failed logins from Jenkins causes a bit of a problem.  Not sure why it causes such an extreme resource leak, but the 2.7 test box was running stable doing periodic gc, and suddenly, we saw memory usage go off the charts--we had just started testing with a test CI node.

Thanks for the tip, all!
Oct 13, 2013
#16 r6squee...@gmail.com
just wanted to add that I'm experiencing this too.

I was on gerrit 2.6.1 + jenkins and was using 512mb of heap and the system was fine. I just upgraded to gerrit 2.7 and I ran out of heap. I increased my heap to 2gb and it still crashes in about 1 or 2 days time. 

This is a low usage system too.

I don't think this should be considered minor as it causes a complete system failure every 1 or 2 days on very low usage system...

I'm attempting the idletimeout and other suggestions here now but won't know the results for a few days.
Oct 14, 2013
#17 manuel.c...@gmail.com
FYI: setting the sshd.idleTimeout fixed the problem in our environment completely. We set it to "5m".
Oct 17, 2013
#18 mani.cha...@tcs.com
We are running gerrit-2.6 on linux redhat and are also facing this error -  Caused by: java.lang.OutOfMemoryError: Java heap space.

To resolve this we tried to set sshd.idleTimeout to 50m but then we got message like connection timed out after 50m.
After this change, Gerrit is working as expected when it terminates the idle stream-events sessions (this we avoided by setting ServerAliveInternal to less than 50 minutes in ssh_config), but the problem is that after a while stream-events don't work anymore. Users can still start new stream-events sessions, but no events are ever received through them.

When this happens, all stream event handler threads (by default there's
25 of them) can be seen on the show-queue output of Gerrit like this:

Task     State                 Command
------------------------------------------------------------------------------
1ed1d1aa                       com.google.gerrit.sshd.commands.StreamEvents$3@5
478c30bb                       com.google.gerrit.sshd.commands.StreamEvents$3@6
4a5a02c3                       com.google.gerrit.sshd.commands.StreamEvents$3@4


Can somebody please help on -
1- How we can either resolve the memory issue 
or 
2- Fix stream-events sending if ssh.idleTimeout is used to work around the memory leak with ssh sessions.  
Oct 23, 2013
#19 mani.cha...@tcs.com
I am running gerrit 2.6 and to resolve the memory heap issue I have set sshd.idleTimeout to 5m in gerrit config. But after doing this setting I am getting below message after 5 mins when I run stream-events.
Received disconnect from ::1: 2: User idle has timed out after 300000ms.

For example:
[mani@localhost bin]$ ssh -p 29418 gerrit_developer1@localhost gerrit stream-events
{"type":"reviewer-added","change":{"project":"myproject","branch":"master","id":"I264123a59806f4e9a23f05a11a06659a668efe46","number":"1","subject":"My first change12123","owner":{"name":"gerrit developer1","email":"mani.chandel@tcs.com","username":"gerrit_developer1"},"url":"http://localhost:8080/1"},"patchSet":{"number":"3","revision":"0fd6237541fa6616ac20a596e273247a5eff9e46","parents":[],"ref":"refs/changes/01/1/3","uploader":{"name":"gerrit developer1","email":"mani.chandel@tcs.com","username":"gerrit_developer1"},"createdOn":1382352926,"author":{"name":"gerrit developer1","email":"mani.chandel@tcs.com","username":"gerrit_developer1"},"sizeInsertions":1,"sizeDeletions":0},"reviewer":{"name":"gerrit admin1","email":"mani_chandel@yahoo.co.in","username":"gerrit_admin1"}}
Received disconnect from ::1: 2: User idle has timed out after 300000ms.

Oct 24, 2013
#20 kotakaha...@aiming-inc.com
As you know, the point of this issue is providing session idle timeout: the review is here https://gerrit-review.googlesource.com/#/c/44351/ .
Also discussed about this issue in such review, and upgrading Apache SSHD to 0.7.0 will solve (see also: https://issues.apache.org/jira/browse/SSHD-147 ) but upgrading was shelved due to SSHD 0.7.0 introduces new feature which is not good for Gerrit: X11 Forwarding and Port Forwarding.

However, I think to be able to disable such feature with org.apache.sshd.server.ForwardingFilter class and already committed at https://gerrit.googlesource.com/gerrit/+/v2.6.1/gerrit-sshd/src/main/java/com/google/gerrit/sshd/SshDaemon.java (line 478).
Thus, Gerrit with SSHD 0.7.0 may work expected.



Then, I tried ssh stress test (~16 parallels) like following with stable-2.6 branch (1a8ce59 with a patch) and monitor JVM activity.
=======================
while true; do
  ssh -p 29418 user@gerrit.example.com gerrit show-queue >/dev/null 2>&1
done
=======================


1) with SSHD 0.6.0

* "jstat -gcutil ..." shows:
=======================
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT   
... snip ...
 62.51   0.00  54.87  99.86  54.67   1682   13.777     0    0.000   13.777
  0.00  59.38  36.75  99.90  54.67   1683   13.790     0    0.000   13.790
 78.14   0.00  23.81  99.94  54.67   1684   13.798     0    0.000   13.798
  0.00  71.89   0.00  99.98  54.67   1685   13.807     1    0.000   13.807
  0.00   0.00  30.67  64.05  49.84   1685   13.807     1    0.717   14.524
 57.82   0.00  14.55  64.05  49.84   1686   13.815     1    0.717   14.533
=======================

* "jmap -histo ..." (pre FGC) shows:
=======================
num       #instances    #bytes  Class description
--------------------------------------------------------------------------
1:              1388261 99954792        java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
2:              21362   22182096        int[]
3:              31395   17084128        byte[]
... snip ...
=======================

* "jmap -histo ..." (post FGC) shows:
=======================
num       #instances    #bytes  Class description
--------------------------------------------------------------------------
1:              1213477 87370344        java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
2:              60962   8415416 * ConstMethodKlass
3:              60962   7814432 * MethodKlass
... snip ...
=======================


2) with SSHD 0.7.0
Note: I used "ssh ... gerrit show-caches --gc" to occur FGC, because it will take too much times to fill up Old Gen.

* "jstat -gcutil ..." shows:
=======================
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT   
... snip ...
 64.07   0.00  17.20  43.25  54.62   1746    7.724     0    0.000    7.724
  0.00  46.88   0.00  43.26  54.62   1747    7.726     0    0.000    7.726
  0.00  46.88  82.83  43.26  54.62   1747    7.726     0    0.000    7.726
  0.00   0.00  42.96  11.70  44.14   1749    7.736     2    0.333    8.069
 57.81   0.00  35.42  11.70  44.14   1750    7.739     2    0.333    8.072
  0.00  54.70  28.34  11.75  44.14   1751    7.743     2    0.333    8.076
=======================

* "jmap -histo ..." (pre FGC) shows:
=======================
num       #instances    #bytes  Class description
--------------------------------------------------------------------------
1:              11733   19978976        int[]
2:              265860  19141920        java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
3:              32415   17396760        byte[]
... snip ...
=======================

* "jmap -histo ..." (post FGC) shows:
=======================
num       #instances    #bytes  Class description
--------------------------------------------------------------------------
... snip ...
9:              4007    1585984 * MethodDataKlass
10:             18694   1345968 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
11:             5       1310800 org.h2.util.CacheObject[]
... snip ...
=======================


SSHD 0.6.0 based Gerrit seems to leak various scheduled future tasks (a.k.a. idleTimerTask).
However SSHD 0.7.0 based one, due to it doesn't construct idleTimerTask (when idleTimeout=0), # of scheduled future tasks (shown in pre FGC) is few against SSHD 0.6.0 based one, even if same count YGC was occured.



Also I tested disabling {X11,Port} Forwarding and Gerrit ignores those as expected.
=======================
$ ssh -p 29418 -X -L 8080:www.example.com:80 user@gerrit.example.com gerrit stream-events
X11 forwarding request failed on channel 2
channel 3: open failed: administratively prohibited: connect denied
=======================


In conclusion, I propose to upgrade SSHD to 0.7.0 with a attached patch to solve this issue.


P.S.
I think, it is better that send a patch to Apache SSHD project to avoid generating unnecessary Tasks for closed sessions, actually.
patch
2.5 KB   View   Download
Oct 29, 2013
Project Member #21 David.Os...@gmail.com
https://gerrit-review.googlesource.com/51142
Status: ChangeUnderReview
Nov 8, 2013
Project Member #22 David.Os...@gmail.com
(No comment was entered for this change.)
Status: Submitted
Feb 19, 2015
Project Member #23 edwin.ke...@gmail.com
 Issue 2005  has been merged into this issue.
Feb 19, 2015
Project Member #24 jrn@google.com
https://gerrit-review.googlesource.com/51516 was part of gerrit 2.8.
Status: Released
Labels: FixedIn-2.9
Feb 23, 2015
Project Member #25 edwin.ke...@gmail.com
 Issue 2112  has been merged into this issue.
Feb 23, 2015
Project Member #26 edwin.ke...@gmail.com
 Issue 1995  has been merged into this issue.
Sign in to add a comment

Powered by Google Project Hosting