| Issue 2002: | Cannot query the database | |
| 15 people starred this issue and may be notified of changes. | Back to list |
Affected Version: 2.6.1 What steps will reproduce the problem? 1. SSH access to Gerrit hangs or is very slow What is the expected output? What do you see instead? Here is the error from the error_log file: ERROR com.google.gerrit.server.git.ChangeMergeQueue : Merge attempt for soagov-mds,refs/heads/design failed com.google.gerrit.server.git.MergeException: Cannot query the database at com.google.gerrit.server.git.MergeOp.merge(MergeOp.java:321) at com.google.gerrit.server.git.ChangeMergeQueue$2.call(ChangeMergeQueue.java:207) at com.google.gerrit.server.git.ChangeMergeQueue$2.call(ChangeMergeQueue.java:204) at com.google.gerrit.server.util.RequestScopePropagator$5.call(RequestScopePropagator.java:222) at com.google.gerrit.server.util.RequestScopePropagator$4.call(RequestScopePropagator.java:201) at com.google.gerrit.server.git.PerThreadRequestScope$Propagator$1.call(PerThreadRequestScope.java:75) at com.google.gerrit.server.git.ChangeMergeQueue.mergeImpl(ChangeMergeQueue.java:204) at com.google.gerrit.server.git.ChangeMergeQueue.access$200(ChangeMergeQueue.java:53) at com.google.gerrit.server.git.ChangeMergeQueue$MergeEntry.run(ChangeMergeQueue.java:247) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:337) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection at com.google.gwtorm.jdbc.Database.newConnection(Database.java:129) at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40) at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$23.<init>(Unknown Source) at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$23_Factory_GwtOrm$$24.open(Unknown Source) at com.google.gwtorm.jdbc.Database.open(Database.java:121) at com.google.gerrit.server.git.MergeOp.openSchema(MergeOp.java:263) at com.google.gerrit.server.git.MergeOp.merge(MergeOp.java:270) ... 17 more Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet successfully received from the server was 27,545 milliseconds ago. The last packet sent successfully to the server was 1 milliseconds ago. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3567) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3456) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3997) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:919) at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4593) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1296) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2397) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2430) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2215) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:813) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.GeneratedConstructorAccessor46.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:334) at com.google.gwtorm.jdbc.SimpleDataSource.getConnection(SimpleDataSource.java:103) at com.google.gwtorm.jdbc.Database.newConnection(Database.java:127) ... 23 more Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3017) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3467) ... 41 more [2013-07-09 06:37:56,334] WARN / : Error in suggestChangeReviewer com.google.inject.ProvisionException: Guice provision errors: 1) Cannot open ReviewDb at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:71) while locating com.google.gerrit.reviewdb.server.ReviewDb 1 error at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987) at com.google.gerrit.httpd.rpc.BaseServiceImplementation.run(BaseServiceImplementation.java:68) at com.google.gerrit.httpd.rpc.SuggestServiceImpl.suggestChangeReviewer(SuggestServiceImpl.java:219) at sun.reflect.GeneratedMethodAccessor76.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91) at com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:379) at com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:265) at com.google.gerrit.httpd.rpc.GerritJsonServlet.service(GerritJsonServlet.java:120) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) at com.google.gerrit.pgm.http.jetty.GetUserFilter.doFilter(GetUserFilter.java:76) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:70) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:67) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:365) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:856) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:722) Caused by: com.google.gwtorm.server.OrmException: Cannot open database connection at com.google.gwtorm.jdbc.Database.newConnection(Database.java:129) at com.google.gwtorm.jdbc.JdbcSchema.<init>(JdbcSchema.java:40) at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$23.<init>(Unknown Source) at com.google.gerrit.reviewdb.server.ReviewDb_Schema_GwtOrm$$23_Factory_GwtOrm$$24.open(Unknown Source) at com.google.gwtorm.jdbc.Database.open(Database.java:121) at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:45) at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:27) at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:71) at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:104) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) ... 60 more Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet successfully received from the server was 20,061 milliseconds ago. The last packet sent successfully to the server was 17,138 milliseconds ago. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1117) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3567) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3456) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3997) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:919) at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4578) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1296) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2397) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2430) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2215) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:813) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.GeneratedConstructorAccessor46.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:399) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:334) at com.google.gwtorm.jdbc.SimpleDataSource.getConnection(SimpleDataSource.java:103) at com.google.gwtorm.jdbc.Database.newConnection(Database.java:127) ... 75 more Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost. at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3017) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3467) ... 93 more Please provide any additional information below. The ReviewDB is MYSQL and sits on the same server as gerrit
Aug 12, 2013
#1
yegor...@gmail.com
Sep 18, 2013
Same issue in my organization. 2 identical Gerrit instances: the one with heavy load - same issue, the one with the low load - all is fine.
Sep 25, 2013
We're having this problem too. Can we get it fixed ASAP? Thanks.
Oct 11, 2013
Same issue within my organization as well. Gerrit completely hangs nightly and starts consuming 100% cpu usage; we are forced to restart every morning. Error logs are very similar.
Mar 12, 2014
server hang for a while when processing changes (even http interface is not responding during processing) then display an error: Counting objects: 11, done. Delta compression using up to 8 threads. Compressing objects: 100% (6/6), done. Writing objects: 100% (6/6), 1.98 KiB | 0 bytes/s, done. Total 6 (delta 5), reused 0 (delta 0) remote: Resolving deltas: 100% (5/5) remote: Processing changes: done remote: error: internal error while processing changes com.google.inject.ProvisionException: Guice provision errors: remote: remote: 1) Cannot open ReviewDb remote: at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70) remote: while locating com.google.gerrit.reviewdb.server.ReviewDb remote: for parameter 0 at com.google.gerrit.server.ApprovalsUtil.<init>(ApprovalsUtil.java:51) remote: while locating com.google.gerrit.server.ApprovalsUtil remote: for parameter 4 at com.google.gerrit.server.change.ChangeInserter.<init>(ChangeInserter.java:90) remote: while locating com.google.gerrit.server.change.ChangeInserter annotated with @com.google.inject.internal.UniqueAnnotations$Internal(value=8) remote: remote: 1 error |
|
| ► Sign in to add a comment |