Issue 3412: Create Project is blocking on replication plugin
Status:  Duplicate
Merged:  issue 3625
Owner: ----
Closed:  Oct 21
Project Member Reported by dougk....@gmail.com, Jun 5, 2015
Affected Version: 2.10.2

What steps will reproduce the problem?
1. Have a large replication queue to any Gerrit slave
2. Attempt to create a project over the REST API

What is the expected output? What do you see instead?
Expect project to create normally, but instead, the request times out at the reverse proxy.

Please provide any additional information below.
I captured the following jstack output:
"HTTP-1366923" #1366923 prio=5 os_prio=0 tid=0x00007fbd82e40000 nid=0x42ea waiting for monitor entry [0x00007fbd4bffc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:93)
        - waiting to lock <0x00007fc1d5e45300> (a com.google.gerrit.server.git.LocalDiskRepositoryManager$Lifecycle$1)
        at com.googlesource.gerrit.plugins.replication.ReplicationQueue.connect(ReplicationQueue.java:416)
        at com.googlesource.gerrit.plugins.replication.ReplicationQueue.executeRemoteSsh(ReplicationQueue.java:396)
        at com.googlesource.gerrit.plugins.replication.ReplicationQueue.createRemoteSsh(ReplicationQueue.java:277)
        at com.googlesource.gerrit.plugins.replication.ReplicationQueue.createProject(ReplicationQueue.java:238)
        at com.googlesource.gerrit.plugins.replication.ReplicationQueue.onNewProjectCreated(ReplicationQueue.java:151)
        at com.google.gerrit.server.project.PerformCreateProject.createProject(PerformCreateProject.java:130)
        at com.google.gerrit.server.project.CreateProject.apply(CreateProject.java:142)
        at com.google.gerrit.server.project.CreateProject.apply(CreateProject.java:51)
        at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:304)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:279)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:269)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at com.google.gerrit.httpd.RequireIdentifiedUserFilter.doFilter(RequireIdentifiedUserFilter.java:55)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:70)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:113)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.ProjectDigestFilter.doFilter(ProjectDigestFilter.java:107)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)
        at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
        - locked <0x00007fc2db8a9718> (a com.google.inject.servlet.GuiceFilter$Context)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1636)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:564)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:219)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:498)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:98)
        at org.eclipse.jetty.server.Server.handle(Server.java:461)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:284)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
        at java.lang.Thread.run(Thread.java:745)

Since there's a long replication queue ahead of the creation, the creation times out.  This would better be handled if projects were created asynchronously/queued for replication on the mirrors, instead of requiring synchronous creation.

Jun 5, 2015
Project Member #1 dougk....@gmail.com
Actually, it appears the original assumption of the replication queue was incorrect.  The issue was most likely caused by a slave going offline during replication (really, became unresponsive and required a restart).  Since the connection was held open and never reset, the only way to resolve was restart Gerrit.
Oct 21, 2015
Project Member #2 dougk....@gmail.com
(No comment was entered for this change.)
Status: Duplicate
Mergedinto: 3625
Oct 21, 2015
Project Member #3 dougk....@gmail.com
Picking 3625, since it has slightly more detail on this issue. :)