| Issue 460: | Internal server error, too many open files | |
| 7 people starred this issue and may be notified of changes. | Back to list |
Affected Version: (2.1.1.1) We are experimenting with Hudson CI server and Gerrit. Hudson polls review sets from Gerrit periodically. After couple of weeks, Gerrit started to report "Internal server error" and in logs there was a lot of following lines: [2010-02-19 07:38:12,033] INFO org.apache.sshd.common.keyprovider.FileKeyPairProvider : Unable to read key /Users/gerrit/review_site/etc/ssh_host_rsa_key: java.io.FileNotFoundException: /Users/gerrit/review_site/etc/ssh_host_rsa_key (Too many open files) ... Caused by: java.io.IOException: Cannot run program "git" (in directory "/Users/gerrit/review_site/git/...."): error=24, Too many open files More examples in attached file. Restarting Gerrit fixed this problem.
Feb 21, 2010
#1
sop@google.com
Status:
Accepted
Feb 21, 2010
> Are you starting Gerrit with the gerrit.sh script under bin? Yes. > Did you modify core.packedGitOpenFiles? No.
Feb 21, 2010
I have seen this same error in my server numerous times too. Will try to collect some logs and attache to this later.
Feb 21, 2010
Server error_log files are less valuable than the output of ls -l /proc/$pid/fd on a Linux box, along with the jstack $pid dump for the same process. Trying to track down leaked file descriptors is a real pain, but having some idea of what's open and what threads are active might help narrow down who has leaked something.
Feb 21, 2010
Here's lsof for gerrit process. We're running gerrit on Mac OS and I couldn't find /proc/ there and jstack didn't work (I don't know too much about macs). Nevertheless, projects 1 and 4 are polled from Hudson regularly and the rest aren't. Refspec for polling is +refs/changes/*:refs/remotes/origin/*. Hope this helps.
Feb 26, 2010
Here's results of lsof -p gerrit_pid from three days. I increased
core.packedGitOpenFiles to 500, so there hasn't been any errors. But it seems that
count of open pack files is increasing. On 24.2. there was about 387 lines in output
file, 25. line count was around 401 and now it is about 438.
When grepping lines where there's "pack", I get following:
$ grep pack 20100224094400_lsof | wc -l
137
$ grep pack 20100225094900_lsof | wc -l
153
$ grep pack 20100226131000_lsof | wc -l
189
Jun 7, 2010
We're hitting this problem quite often after we upgraded to 2.1.2.5. Not sure about earlier versions, problem reports may have been masked by issue 390 . We use the default value (i.e., do no specify) core.packedGitOpenFiles. One of the callstacks I saw today was: [2010-06-07 03:13:52,007] ERROR com.google.gerrit.server.cache.SelfPopulatingCache : Cannot lookup PatchListKey[3rdpar ty/atheros BASE..9d3f8d8731236be69dd5123919b9d9b0b1792281 IGNORE_NONE] in "diff" net.sf.ehcache.CacheException: Could not fetch object for cache entry with key "PatchListKey[3rdparty/atheros BASE..9d 3f8d8731236be69dd5123919b9d9b0b1792281 IGNORE_NONE]". at net.sf.ehcache.constructs.blocking.SelfPopulatingCache.get(SelfPopulatingCache.java:8 8) at com.google.gerrit.server.cache.SelfPopulatingCache.get(SelfPopulatingCache.java:107) at com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:115) at com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:127) at com.google.gerrit.server.patch.PatchListCacheImpl.get(PatchListCacheImpl.java:119) at com.google.gerrit.httpd.rpc.changedetail.PatchSetDetailFactory.call(PatchSetDetailFac tory.java:83) at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.loadCurrentPatchSet(Chan geDetailFactory.java:1 88) at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory .java:108) at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailFactory.call(ChangeDetailFactory .java:53) at com.google.gerrit.httpd.rpc.Handler.to(Handler.java:65) at com.google.gerrit.httpd.rpc.changedetail.ChangeDetailServiceImpl.changeDetail(ChangeD etailServiceImpl.java: 42) at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91) at com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:382) at com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:268) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:216) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:141) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java: 93) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:6 3) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:5 9) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:5 9) at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:76) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:5 9) at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:54) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:5 9) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:5 9) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:1 22) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:110) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:118 7) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:425) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:931) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:362) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:867) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113) at org.eclipse.jetty.server.Server.handle(Server.java:334) at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:559) at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:10 07) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:747) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:209) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:406) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:462) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: Cannot run program "git" (in directory "/var/gerrit2/review_site/git/3rdparty/atheros. git"): java.io.IOException: error=24, Too many open files at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) at java.lang.Runtime.exec(Runtime.java:593) at com.google.gerrit.server.patch.PatchListCacheImpl.exec(PatchListCacheImpl.java:537) at com.google.gerrit.server.patch.PatchListCacheImpl.readPatchList(PatchListCacheImpl.ja va:176) at com.google.gerrit.server.patch.PatchListCacheImpl.compute(PatchListCacheImpl.java:134 ) at com.google.gerrit.server.patch.PatchListCacheImpl.access$000(PatchListCacheImpl.java: 64) at com.google.gerrit.server.patch.PatchListCacheImpl$2.createEntry(PatchListCacheImpl.ja va:102) at com.google.gerrit.server.patch.PatchListCacheImpl$2.createEntry(PatchListCacheImpl.ja va:99) at com.google.gerrit.server.cache.SelfPopulatingCache$1.createEntry(SelfPopulatingCache. java:60) at net.sf.ehcache.constructs.blocking.SelfPopulatingCache.get(SelfPopulatingCache.java:7 2) ... 51 more Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) at java.lang.ProcessImpl.start(ProcessImpl.java:65) at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ... 60 more
Jun 7, 2010
Unfortunately core.packedGitOpenFiles isn't a hard limit. The server can go over it for many reasons. The default means we are using 256 for the limit on file descriptors in the JVM. That is probably too low for your workload given that we can overshoot the packedGitOpenFiles limit. I'd suggest trying to set it higher. Maybe you can set it to core.packedGitOpenFiles = 512, having it double up to what should be your OS default of 1024.
Labels:
Component-JGit
Jun 15, 2010
We have used the following work around for the past 6 days: --- bin/gerrit.sh.orig 2010-06-03 10:42:39.000000000 -0700 +++ bin/gerrit.sh 2010-06-15 01:24:49.000000000 -0700 @@ -294,7 +294,8 @@ ulimit -d unlimited ; # data seg size ulimit -f unlimited ; # file size ulimit -m unlimited ; # max memory size -ulimit -n $GERRIT_FDS ; # open files +ulimit -n 768 ; # a hack for bug 460 /585 +#ulimit -n $GERRIT_FDS ; # open files ulimit -t unlimited ; # cpu time ulimit -t unlimited ; # virtual memory With this, there have been no more "Too many open files" errors. If I changed core.packedGitOpenFiles in the configuration file, would that not cause an increase in the resource requirements of Gerrit as well?
Jun 16, 2010
Issue 586 has been merged into this issue.
Jun 16, 2010
Issue 585 has been merged into this issue.
Jun 16, 2010
Fixed by I45e62a8af9f3763eacf977431c299193f014bd1e I set the minimum FDs we request to 1024.
Status:
Fixed
Labels: FixedIn-2.1.3
Mar 27, 2012
(No comment was entered for this change.)
Status:
Released
|
|
| ► Sign in to add a comment |