My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 92: Gerrit leaks file descriptors - too many open files
  Back to list
Status:  Released
Owner:  code-rev...@gtempaccount.com
Closed:  Oct 2012


Sign in to add a comment
 
Reported by code-rev...@gtempaccount.com, Sep 24, 2009
Reported by Shawn Pearce <sop@google.com> on Mon Mar 02 10:28:06 PST 2009
Source: JIRA GERRIT-92
Affected Version: 2.0.4

2009-02-28 13:18:01,653::ERROR: com.google.gerrit.server.ssh.Receive  - Error
storing replacement patch for change 338, commit
644455f015ee7ed7cea197eec05e25faaa240d1f
2009-02-28 13:18:31,973::ERROR: com.google.gerrit.server.ssh.Receive  - Error
storing replacement patch for change 338, commit
644455f015ee7ed7cea197eec05e25faaa240d1f
2009-02-28 13:21:48,753::ERROR: com.google.gerrit.server.ssh.Receive  - Error
storing replacement patch for change 338, commit
644455f015ee7ed7cea197eec05e25faaa240d1f
2009-02-28 13:22:34,673::ERROR: com.google.gerrit.server.ssh.Receive  - Error
storing replacement patch for change 338, commit
6af9d6675eb4df46852e9857976337f148676afd
2009-02-28 13:23:26,773::WARN : org.apache.mina.util.DefaultExceptionMonitor
- Unexpected exception.
java.io.IOException: Too many open files
       at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
       at sun.nio.ch.ServerSocketChannelImpl.accept
(ServerSocketChannelImpl.java:205)
       at org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept
(NioSocketAcceptor.java:218)
       at org.apache.mina.transport.socket.nio.NioSocketAcceptor.accept
(NioSocketAcceptor.java:49)
       at org.apache.mina.core.polling.AbstractPollingIoAcceptor
$Acceptor.processHandles(AbstractPollingIoAcceptor.java:472)
       at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run
(AbstractPollingIoAcceptor.java:405)
       at org.apache.mina.util.NamePreservingRunnable.run
(NamePreservingRunnable.java:65)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
(ThreadPoolExecutor.java:665)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:690)
       at java.lang.Thread.run(Thread.java:810)

Is this a Gerrit bug? A Jetty bug (found a thread saying this might be related
to the maxIdleTime value)? Do I need to change the ulimit -n value on my
server?
Sep 24, 2009
#1 code-rev...@gtempaccount.com
Comment by Shawn Pearce <sop@google.com> on Mon Mar 02 11:30:54 PST 2009

I also just saw this on review.source.android.com when I tried to warm-reload
gerrit 2.0.5.

Caused by: java.io.FileNotFoundException: /android/kernel/msm.git/objects/pack/
pack-8b49a958de585325270e1dde8658a7722e10c722.pack (Too many open files)
Sep 24, 2009
#2 code-rev...@gtempaccount.com
Update by Shawn Pearce <sop@google.com> on Mon Mar 16 13:58:44 PDT 2009
Labels: -Priority-Critical Priority-Blocker
Sep 24, 2009
#3 code-rev...@gtempaccount.com
Comment by Shawn Pearce <sop@google.com> on Mon Mar 16 15:46:33 PDT 2009

Quite likely an issue with the pack files.  ls -l /proc/$pid/fd of a running
Jetty on review.source.android.com suggests we have leaked the PackFile handle
and didn't close it.  There's way too many packs open given the expected
content of the WindowCache for the server.
Sep 24, 2009
#4 code-rev...@gtempaccount.com
Comment by Shawn Pearce <sop@google.com> on Mon Mar 16 16:46:17 PDT 2009

I'm not sure its a leak per-se, just a bad configuration.

The default configuration of WindowCache is:

  maxByteCount  = 10 MB
  windowSize = 8 KB

10 MB / 8 KB = 1280 windows in the cache.

Assuming each window is from a unique pack file, which isn't unreasonable
given that each upload goes into its own pack and a server may have many
packs, WindowCache believes it is OK to open 1280 file descriptors before it
reaches its maximum caching configuration.  By default a server probably is
restricted to 1024 file descriptors for the entire process, including
container access to JAR/WAR files, log files, network sockets, pipes to
execute git diff, etc.

JGit's WindowCache needs a hard upper bound on file descriptor usage.
Sep 24, 2009
#5 code-rev...@gtempaccount.com
Comment by Shawn Pearce <sop@google.com> on Mon Mar 16 18:17:00 PDT 2009

Patches posted to git ML http://thread.gmane.org/gmane.comp.version-control.git/113410
Sep 24, 2009
#6 code-rev...@gtempaccount.com
Update by Shawn Pearce <sop@google.com> on Thu Mar 19 15:55:19 PDT 2009
Sep 24, 2009
#7 code-rev...@gtempaccount.com
Comment by Shawn Pearce <sop@google.com> on Thu Mar 19 15:55:26 PDT 2009

Patches accepted; will shipped in gerrit 2.0.8
Sep 24, 2009
#8 code-rev...@gtempaccount.com
Update by Shawn Pearce <sop@google.com> on Sun Mar 22 18:09:26 PDT 2009
Sep 24, 2009
#9 code-rev...@gtempaccount.com
Comment by Shawn Pearce <sop@google.com> on Sun Mar 22 18:11:11 PDT 2009

Fixed by https://review.source.android.com/9360
Sep 24, 2009
#10 code-rev...@gtempaccount.com
Update by Shawn Pearce <sop@google.com> on Sun Mar 22 18:11:11 PDT 2009

Fixed in version 2.0.8.
Status: Fixed
Sep 25, 2009
#11 code-rev...@gtempaccount.com
(No comment was entered for this change.)
Labels: FixedIn-2.0.8
Oct 25, 2012
#12 sop@google.com
(No comment was entered for this change.)
Status: Released
Sign in to add a comment

Powered by Google Project Hosting