My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 460: Internal server error, too many open files
7 people starred this issue and may be notified of changes. Back to list
Status:  Released
Owner:  ----
Closed:  Mar 2012


Sign in to add a comment
 
Reported by jyrki...@gmail.com, Feb 18, 2010
Affected Version: (2.1.1.1)

We are experimenting with Hudson CI server and Gerrit. Hudson polls 
review sets from Gerrit periodically. After couple of weeks, Gerrit started to 
report "Internal server error" and in logs there was a lot of following lines:

[2010-02-19 07:38:12,033] INFO  
org.apache.sshd.common.keyprovider.FileKeyPairProvider : Unable to read 
key /Users/gerrit/review_site/etc/ssh_host_rsa_key: 
java.io.FileNotFoundException: 
/Users/gerrit/review_site/etc/ssh_host_rsa_key (Too many open files)
...
Caused by: java.io.IOException: Cannot run program "git" (in directory 
"/Users/gerrit/review_site/git/...."): error=24, Too many open files

More examples in attached file.

Restarting Gerrit fixed this problem.
gerrit_log.txt
14.8 KB   View   Download
Feb 21, 2010
#1 sop@google.com
Are you starting Gerrit with the gerrit.sh script under bin?
Did you modify core.packedGitOpenFiles?

Folks were suspecting that the current master branch has
a file handle leak, but it looks like the leak might even be
in 2.1.1.1.  So its not a new regression, but an old bug we
really need to track down and fix.
Status: Accepted
Feb 21, 2010
#2 jyrki...@gmail.com
> Are you starting Gerrit with the gerrit.sh script under bin?
Yes.

> Did you modify core.packedGitOpenFiles?
No.
Feb 21, 2010
Project Member #3 lscar...@gmail.com
I have seen this same error in my server numerous times too.
Will try to collect some logs and attache to this later.
Feb 21, 2010
#4 sop@google.com
Server error_log files are less valuable than the output
of ls -l /proc/$pid/fd on a Linux box, along with the
jstack $pid dump for the same process.

Trying to track down leaked file descriptors is a real
pain, but having some idea of what's open and what
threads are active might help narrow down who has
leaked something.
Feb 21, 2010
#5 jyrki...@gmail.com
Here's lsof for gerrit process. We're running gerrit on Mac OS and I couldn't find /proc/ 
there and jstack didn't work (I don't know too much about macs). Nevertheless, projects 
1 and 4 are polled from Hudson regularly and the rest aren't. Refspec for polling is 
+refs/changes/*:refs/remotes/origin/*. Hope this helps.
lsof.txt
47.5 KB   View   Download
Feb 26, 2010
#6 jyrki...@gmail.com
Here's results of lsof -p gerrit_pid from three days. I increased 
core.packedGitOpenFiles to 500, so there hasn't been any errors. But  it seems that 
count of open pack files is increasing. On 24.2. there was about 387 lines in output 
file, 25. line count was around 401 and now it is about 438.

When grepping lines where there's "pack", I get following:
$ grep pack 20100224094400_lsof | wc -l
     137
$ grep pack 20100225094900_lsof | wc -l
     153
$ grep pack 20100226131000_lsof | wc -l
     189

20100224094400_lsof
57.1 KB   View   Download
20100225094900_lsof
60.1 KB   View   Download
20100226131000_lsof
66.6 KB   View   Download
Jun 7, 2010
#7 jjhel...@gmail.com
We're hitting this problem quite often after we upgraded to 2.1.2.5.  Not sure about 
earlier versions, problem reports may have been masked by  issue 390 .

We use the default value (i.e., do no specify) core.packedGitOpenFiles.

One of the callstacks I saw today was:


[2010-06-07 03:13:52,007] ERROR com.google.gerrit.server.cache.SelfPopulatingCache : 
Cannot lookup PatchListKey[3rdpar
ty/atheros BASE..9d3f8d8731236be69dd5123919b9d9b0b1792281 IGNORE_NONE] in "diff"
net.sf.ehcache.CacheException: Could not fetch object for cache entry with key 
"PatchListKey[3rdparty/atheros BASE..9d
3f8d8731236be69dd5123919b9d9b0b1792281 IGNORE_NONE]".
        at 
net.sf.ehcache.constructs.blocking.SelfPopulatingCache.get(SelfPopulatingCache.java:8
8)
        at 
com.google.gerrit.server.cache.SelfPopulatingCache.get(SelfPopulatingCache.java:107)
        at 
com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:115)
        at 
com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:127)
        at 
com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:119)
        at 
com.google.gerrit.httpd.rpc.changedetail.PatchSetDetailFactory.call(PatchSetDetailFac
tory.java:83)
        at 
com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.loadCurrentPatchSet(Chan
geDetailFactory.java:1
88)
        at 
com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory
.java:108)
        at 
com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory
.java:53)
        at com.google.gerrit.httpd.rpc.Handler.to(Handler.java:65)
        at 
com.google.gerrit.httpd.rpc.changedetail.ChangeDetailServiceImpl.changeDetail(ChangeD
etailServiceImpl.java:
42)
        at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91)
        at com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:382)
        at com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:268)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at 
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:216)
        at 
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:141)
        at 
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:
93)
        at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:6
3)
        at 
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)
        at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:5
9)
        at 
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)
        at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:5
9)
        at 
com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:76)
        at 
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129)
        at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:5
9)
        at 
com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:54)
        at 
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129)
        at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:5
9)
        at 
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)
        at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:5
9)
        at 
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:1
22)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:110)
        at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:118
7)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:425)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:931)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:362)
        at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:867)
        at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
        at org.eclipse.jetty.server.Server.handle(Server.java:334)
        at 
org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:559)
        at 
org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:10
07)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:747)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:209)
        at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:406)
        at 
org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:462)
        at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
        at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Cannot run program "git" (in directory 
"/var/gerrit2/review_site/git/3rdparty/atheros.
git"): java.io.IOException: error=24, Too many open files
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
        at java.lang.Runtime.exec(Runtime.java:593)
        at 
com.google.gerrit.server.patch.PatchListCacheImpl.exec(PatchListCacheImpl.java:537)
        at 
com.google.gerrit.server.patch.PatchListCacheImpl.readPatchList(PatchListCacheImpl.ja
va:176)
        at 
com.google.gerrit.server.patch.PatchListCacheImpl.compute(PatchListCacheImpl.java:134
)
        at 
com.google.gerrit.server.patch.PatchListCacheImpl.access$000(PatchListCacheImpl.java:
64)

        at 
com.google.gerrit.server.patch.PatchListCacheImpl$2.createEntry(PatchListCacheImpl.ja
va:102)
        at 
com.google.gerrit.server.patch.PatchListCacheImpl$2.createEntry(PatchListCacheImpl.ja
va:99)
        at 
com.google.gerrit.server.cache.SelfPopulatingCache$1.createEntry(SelfPopulatingCache.
java:60)
        at 
net.sf.ehcache.constructs.blocking.SelfPopulatingCache.get(SelfPopulatingCache.java:7
2)
        ... 51 more
Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
        ... 60 more

Jun 7, 2010
#8 sop@google.com
Unfortunately core.packedGitOpenFiles isn't a hard
limit.  The server can go over it for many reasons.

The default means we are using 256 for the limit on
file descriptors in the JVM.  That is probably too
low for your workload given that we can overshoot
the packedGitOpenFiles limit.

I'd suggest trying to set it higher.  Maybe you can
set it to core.packedGitOpenFiles = 512, having it
double up to what should be your OS default of 1024.
Labels: Component-JGit
Jun 15, 2010
#9 oskari.j...@gmail.com
We have used the following work around for the past 6 days:

--- bin/gerrit.sh.orig      2010-06-03 10:42:39.000000000 -0700
+++ bin/gerrit.sh   2010-06-15 01:24:49.000000000 -0700
@@ -294,7 +294,8 @@
 ulimit -d unlimited    ; # data seg size
 ulimit -f unlimited    ; # file size
 ulimit -m unlimited    ; # max memory size
-ulimit -n $GERRIT_FDS  ; # open files
+ulimit -n 768  ; # a hack for  bug 460 /585
+#ulimit -n $GERRIT_FDS  ; # open files
 ulimit -t unlimited    ; # cpu time
 ulimit -t unlimited    ; # virtual memory

With this, there have been no more "Too many open files" errors.

If I changed core.packedGitOpenFiles in the configuration file, would that not cause an increase in the resource requirements of Gerrit as well?

Jun 16, 2010
#10 sop@google.com
 Issue 586  has been merged into this issue.
Jun 16, 2010
#11 sop@google.com
 Issue 585  has been merged into this issue.
Jun 16, 2010
#12 sop@google.com
Fixed by I45e62a8af9f3763eacf977431c299193f014bd1e

I set the minimum FDs we request to 1024.
Status: Fixed
Labels: FixedIn-2.1.3
Mar 27, 2012
#13 sop@google.com
(No comment was entered for this change.)
Status: Released
Sign in to add a comment

Powered by Google Project Hosting