Issue 2002: Cannot query the database
Status:  New
Owner: ----
Reported by nikki.ka...@gmail.com, Jul 9, 2013
Affected Version: 2.6.1

What steps will reproduce the problem?
1. SSH access to Gerrit hangs or is very slow

What is the expected output? What do you see instead?
Here is the error from the error_log file:
ERROR com.google.gerrit.server.git.ChangeMergeQueue : Merge attempt for soagov-mds,refs/heads/design failed
com.google.gerrit.server.git.MergeException: Cannot query the database
	at com.google.gerrit.server.git.MergeOp.merge(MergeOp.java:321)
	at com.google.gerrit.server.git.ChangeMergeQueue$2.call(ChangeMergeQueue.java:207)
	at com.google.gerrit.server.git.ChangeMergeQueue$2.call(ChangeMergeQueue.java:204)
	at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:222)
	at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:201)
	at com.google.gerrit.server.git.PerThreadRequestScope$Propagator$1.call(PerThreadRequestScope.java:75)
	at com.google.gerrit.server.git.ChangeMergeQueue.mergeImpl(ChangeMergeQueue.java:204)
	at com.google.gerrit.server.git.ChangeMergeQueue.access$200(ChangeMergeQueue.java:53)
	at com.google.gerrit.server.git.ChangeMergeQueue$MergeEntry.run(ChangeMergeQueue.java:247)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection
	at com.google.gwtorm.jdbc.Database.newConnection(Database.java:129)
	at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)
	at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$23.<init>(Unknown Source)
	at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$23_Factory_GwtOrm$$24.open(Unknown Source)
	at com.google.gwtorm.jdbc.Database.open(Database.java:121)
	at com.google.gerrit.server.git.MergeOp.openSchema(MergeOp.java:263)
	at com.google.gerrit.server.git.MergeOp.merge(MergeOp.java:270)
	... 17 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 27,545 milliseconds ago.  The last packet sent successfully to the server was 1 milliseconds ago.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
	at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3567)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3456)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3997)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:919)
	at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4593)
	at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1296)
	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2397)
	at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2430)
	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2215)
	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:813)
	at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
	at sun.reflect.GeneratedConstructorAccessor46.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399)
	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:334)
	at com.google.gwtorm.jdbc.SimpleDataSource.getConnection(SimpleDataSource.java:103)
	at com.google.gwtorm.jdbc.Database.newConnection(Database.java:127)
	... 23 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
	at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3017)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3467)
	... 41 more
[2013-07-09 06:37:56,334] WARN  / : Error in suggestChangeReviewer
com.google.inject.ProvisionException: Guice provision errors:

1) Cannot open ReviewDb
  at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:71)
  while locating com.google.gerrit.reviewdb.server.ReviewDb

1 error
	at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987)
	at com.google.gerrit.httpd.rpc.BaseServiceImplementation.run(BaseServiceImplementation.java:68)
	at com.google.gerrit.httpd.rpc.SuggestServiceImpl.suggestChangeReviewer(SuggestServiceImpl.java:219)
	at sun.reflect.GeneratedMethodAccessor76.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91)
	at com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:379)
	at com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:265)
	at com.google.gerrit.httpd.rpc.GerritJsonServlet.service(GerritJsonServlet.java:120)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
	at com.google.gerrit.pgm.http.jetty.GetUserFilter.doFilter(GetUserFilter.java:76)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:70)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)
	at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:67)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:365)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:856)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:722)
Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection
	at com.google.gwtorm.jdbc.Database.newConnection(Database.java:129)
	at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40)
	at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$23.<init>(Unknown Source)
	at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$23_Factory_GwtOrm$$24.open(Unknown Source)
	at com.google.gwtorm.jdbc.Database.open(Database.java:121)
	at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:45)
	at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:27)
	at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:71)
	at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:104)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40)
	at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024)
	at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974)
	... 60 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 20,061 milliseconds ago.  The last packet sent successfully to the server was 17,138 milliseconds ago.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
	at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3567)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3456)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3997)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:919)
	at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4578)
	at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1296)
	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2397)
	at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2430)
	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2215)
	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:813)
	at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
	at sun.reflect.GeneratedConstructorAccessor46.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399)
	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:334)
	at com.google.gwtorm.jdbc.SimpleDataSource.getConnection(SimpleDataSource.java:103)
	at com.google.gwtorm.jdbc.Database.newConnection(Database.java:127)
	... 75 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
	at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3017)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3467)
	... 93 more

Please provide any additional information below.
The ReviewDB is MYSQL and sits on the same server as gerrit
Aug 12, 2013
#1 yegor...@gmail.com
This seem to be only surfacing on systems with high usage. We have few installations and only one of them has this issue.
Sep 18, 2013
#2 psz...@gmail.com
Same issue in my organization. 2 identical Gerrit instances: the one with heavy load - same issue, the one with the low load - all is fine.
Sep 25, 2013
#3 casta...@motorola.com
We're having this problem too.   Can we get it fixed ASAP?

Thanks.
Oct 11, 2013
#4 daniel.n...@accesso.com
Same issue within my organization as well. Gerrit completely hangs nightly and starts consuming 100% cpu usage; we are forced to restart every morning. 

Error logs are very similar.
Mar 12, 2014
#6 h.hamza...@gmail.com
server hang for a while when processing changes (even http interface is not responding during processing) then display an error:

Counting objects: 11, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.98 KiB | 0 bytes/s, done.
Total 6 (delta 5), reused 0 (delta 0)
remote: Resolving deltas: 100% (5/5)
remote: Processing changes: done    
remote: error: internal error while processing changes com.google.inject.ProvisionException: Guice provision errors:
remote: 
remote: 1) Cannot open ReviewDb
remote:   at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70)
remote:   while locating com.google.gerrit.reviewdb.server.ReviewDb
remote:     for parameter 0 at com.google.gerrit.server.ApprovalsUtil.<init>(ApprovalsUtil.java:51)
remote:   while locating com.google.gerrit.server.ApprovalsUtil
remote:     for parameter 4 at com.google.gerrit.server.change.ChangeInserter.<init>(ChangeInserter.java:90)
remote:   while locating com.google.gerrit.server.change.ChangeInserter annotated with @com.google.inject.internal.UniqueAnnotations$Internal(value=8)
remote: 
remote: 1 error