My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 1087: Gerrit doesn't survive database restart cleanly
10 people starred this issue and may be notified of changes. Back to list
Status:  Accepted
Owner:  ----


Sign in to add a comment
 
Reported by hannu.ko...@vincit.fi, Aug 11, 2011
Affected Version: Gerrit 2.2.1

What steps will reproduce the problem?
Happens sometimes for some users, no known way to set up a situation where these occur.


What is the expected output? What do you see instead?
It's expected that
* gerrit web page loads normally
* you can push commits via ssh

Instead:
* git push review HEAD:refs/for/devel
=>
fatal: internal server error
fatal: The remote end hung up unexpectedly

In the error_log:

[2011-08-11 08:58:24,002] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user hidden account 1000001) during git-receive-pack 'hidden.git'
java.lang.IllegalStateException: Can't overwrite cause
        at java.lang.Throwable.initCause(Throwable.java:320)
        at com.google.gwtorm.jdbc.JdbcAccess.convertError(JdbcAccess.java:329)
        at com.google.gwtorm.jdbc.JdbcAccess.queryList(JdbcAccess.java:152)
        at com.google.gerrit.reviewdb.Change_Access_changes_GwtOrm$$0.byProjectOpenNext(Unknown Source)
        at com.google.gerrit.server.git.ReceiveCommits.advertiseHistory(ReceiveCommits.java:253)
        at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:89)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:103)
        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:34)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:69)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:399)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:324)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)


Gerrit web page access attempt resulted to this in error_log:

[2011-08-11 08:58:50,030] WARN  / : Error in allQueryNext
java.lang.IllegalStateException: Can't overwrite cause
        at java.lang.Throwable.initCause(Throwable.java:320)
        at com.google.gwtorm.jdbc.JdbcAccess.convertError(JdbcAccess.java:329)
        at com.google.gwtorm.jdbc.JdbcAccess.queryList(JdbcAccess.java:152)
        at com.google.gerrit.reviewdb.Change_Access_changes_GwtOrm$$0.allOpenNext(Unknown Source)
        at com.google.gerrit.server.query.change.ChangeQueryRewriter$8.scan(ChangeQueryRewriter.java:277)
        at com.google.gerrit.server.query.change.ChangeQueryRewriter$PaginatedSource.scan(ChangeQueryRewriter.java:616)
        at com.google.gerrit.server.query.change.ChangeQueryRewriter$ChangeSource.read(ChangeQueryRewriter.java:579)
        at com.google.gerrit.server.query.change.AndSource.read(AndSource.java:94)
        at com.google.gerrit.httpd.rpc.ChangeListServiceImpl.searchQuery(ChangeListServiceImpl.java:167)
        at com.google.gerrit.httpd.rpc.ChangeListServiceImpl.access$100(ChangeListServiceImpl.java:55)
        at com.google.gerrit.httpd.rpc.ChangeListServiceImpl$6.query(ChangeListServiceImpl.java:137)
        at com.google.gerrit.httpd.rpc.ChangeListServiceImpl$QueryNext.run(ChangeListServiceImpl.java:325)
        at com.google.gerrit.httpd.rpc.ChangeListServiceImpl$QueryNext.run(ChangeListServiceImpl.java:307)
        at com.google.gerrit.httpd.rpc.BaseServiceImplementation.run(BaseServiceImplementation.java:63)
        at com.google.gerrit.httpd.rpc.ChangeListServiceImpl.allQueryNext(ChangeListServiceImpl.java:133)
        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91)
        at com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:382)
        at com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:268)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:216)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:141)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:63)
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
        at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:76)
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
        at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:54)
        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:122)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:110)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:473)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:921)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:856)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114)
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:59)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114)
        at org.eclipse.jetty.server.Server.handle(Server.java:352)
        at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
        at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1069)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:805)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218)
        at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:510)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:450)
        at java.lang.Thread.run(Thread.java:662)
 

Please provide any additional information below.
Gerrit is running in Ubuntu 11.04 x86_64.
Postgresql 8.4.8-0ubuntu0.11.04 is used as the database.
Java is Sun Java 6.26-1natty1
Gerrit is using its built-in servlet container.
Aug 11, 2011
#1 hannu.ko...@vincit.fi
I figured out what triggers this problem: it's a database server restart. That is, the problem can be reproduced by restarting postgresql while Gerrit is running and then attempting to use Gerrit.

A workaround is to to stop gerrit before db restart and start it after but of course it would be better if Gerrit was more robust in this situation. 

I guess the issue title should be changed to "Gerrit doesn't survive database restart cleanly" or something but unfortunately this issue tracking system doesn't allow me to do that.
Aug 11, 2011
#2 sop@google.com
(No comment was entered for this change.)
Summary: Gerrit doesn't survive database restart cleanly
Status: Accepted
Oct 14, 2011
#3 di...@google.com
I got similar errors and indeed I tracked them as started after a database restart (no idea what restarted it, maybe Ubuntu's automated updates or something):

2011-10-14 07:48:56,053] WARN  org.eclipse.jetty.util.log : /g/login/q,15117,n,z
java.lang.IllegalStateException: Can't overwrite cause
	at java.lang.Throwable.initCause(Throwable.java:337)
	at com.google.gwtorm.jdbc.JdbcAccess.convertError(JdbcAccess.java:329)
	at com.google.gwtorm.jdbc.JdbcAccess.queryOne(JdbcAccess.java:128)
	at com.google.gerrit.reviewdb.AccountExternalId_Access_accountExternalIds_GwtOrm$$7.get(Unknown Source)
	at com.google.gerrit.server.account.AccountManager.authenticate(AccountManager.java:111)
	at com.google.gerrit.httpd.auth.container.HttpLoginServlet.doGet(HttpLoginServlet.java:123)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
....


[2011-10-14 07:40:03,803] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user foo account 1000010) during gerrit ls-projects
java.lang.IllegalStateException: Can't overwrite cause
	at java.lang.Throwable.initCause(Throwable.java:337)
	at com.google.gwtorm.jdbc.JdbcAccess.convertError(JdbcAccess.java:329)
	at com.google.gwtorm.jdbc.JdbcAccess.queryList(JdbcAccess.java:152)
	at com.google.gerrit.reviewdb.Project_Access_projects_GwtOrm$$21.all(Unknown Source)
	at com.google.gerrit.sshd.commands.ListProjects.display(ListProjects.java:96)
	at com.google.gerrit.sshd.commands.ListProjects.access$100(ListProjects.java:40)
	at com.google.gerrit.sshd.commands.ListProjects$1.run(ListProjects.java:77)
	at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:399)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)

So it's clear the errors stem from JDBC failures but what's up with the confusing "java.lang.IllegalStateException: Can't overwrite cause" exception? Can't that be improved?

Also I'm seeing that the error just stopped a few minutes after the restart, seems a Gerrit restart is not needed to fix this issue (but I do wonder if Gerrit lost any data while this happened, is that a possibility?)

As a proper fix, Gerrit's startup script should depend on the pgsql startup script so Ubuntu (or your other distro) knows to restart Gerrit before restarting pgsql.
Jan 27, 2012
#4 m...@zenrobotics.com
The code in com.google.gwtorm.jdbc.JdbcAccess.convertError is wrong. It does:
    if (err.getCause() == null && err.getNextException() != null) {
      err.initCause(err.getNextException());
    }

getCause() can return null either if no cause has ever been set (Throwable's cause field == this) or if cause has been set to null (Throwable's cause field == null). However, calling initCause is legal only if cause has never been set (cause == this). So the code in convertError doesn't match the contract for initCause.
Sign in to add a comment

Powered by Google Project Hosting