My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
for
  Advanced search   Search tips   Subscriptions
Issue 1850: Unable to remove replication plugin while replication jobs exist in Gerrit's queue
4 people starred this issue and may be notified of changes. Back to list
Status:  New
Owner:  ----


Sign in to add a comment
 
Reported by earl.gr...@gmail.com, Mar 31, 2013
************************************************************
***** NOTE: THIS BUG TRACKER IS FOR GERRIT CODE REVIEW *****
***** DO NOT SUBMIT BUGS FOR CHROME, ANDROID, INTERNAL *****
***** ISSUES WITH YOUR COMPANY'S GERRIT SETUP, ETC.    *****
***** THOSE ISSUE BELONG IN DIFFERENT ISSUE TRACKERS!  *****
************************************************************

Affected Version: 2.5

What steps will reproduce the problem?
1. Add a sufficiently large replication delay to replication.config so that it's possible to have a replication job in Gerrit's queue before it's processed by Gerrit ( replicationDelay = 35 )
2. Create a change and upload it to Gerrit
3. Remove the replication plugin while the job is queued by Gerrit waiting to run

What is the expected output? What do you see instead?
The replication plugin is never removed by Gerrit. It's necessary to restart Gerrit but before restarting, the replication.jar.disabled file needs to be renamed to replication.jar. If not, the plugin does not start on a Gerrit restart.

Please provide any additional information below.

Replication config:

[remote "slave"]
    url = git://slave:9419/${name}.git
    adminUrl = slaveuser@slave:/git/repositories/${name}.git
    push = +refs/*:refs/*
    replicationDelay = 35
    threads = 10
    mirror = true

Confirm queue is empty:

Task     State                 Command
------------------------------------------------------------------------------
36fdcf6f 12:11:10.675          Log File Compressor
------------------------------------------------------------------------------
  1  tasks

Plugin filename:

ls plugins/
replication.jar

Plugin is enabled:
ssh -p 29418 master gerrit plugin ls
Name                           Version    Status
-------------------------------------------------------------------------------
replication                    1.0        ENABLED

Push a change to Gerrit
git push origin HEAD:refs/for/master

Task     State                 Command
------------------------------------------------------------------------------
d7db1a63 11:20:58.880          push git://slave:9419/myproject
36fdcf6f 12:11:10.676          Log File Compressor
------------------------------------------------------------------------------
  2 tasks

Remove replication plugin:
ssh -p 29418 master gerrit plugin remove replication

ls plugins/
replication.jar.disabled

Gerrit's LOG while trying to remove plugin

[2013-03-31 11:20:44,621] INFO  com.google.gerrit.server.plugins.PluginLoader : Disabling plugin replication
[2013-03-31 11:20:44,621] WARN  com.googlesource.gerrit.plugins.replication.ReplicationQueue : Cancelled 1 replication events during shutdown
[2013-03-31 11:21:48,589] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1120_4820055441694090468.jar
[2013-03-31 11:21:48,590] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1111_3769375487048786266.jar
[2013-03-31 11:21:48,590] WARN  com.google.gerrit.server.plugins.PluginLoader : 1 plugins still waiting to be reclaimed after 1 minutes
[2013-03-31 11:22:26,100] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1121_1666127951484955268.jar
[2013-03-31 11:22:26,101] WARN  com.google.gerrit.server.plugins.PluginLoader : 1 plugins still waiting to be reclaimed after 1 minutes
[2013-03-31 11:25:36,219] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1122_1424127429921268974.jar
[2013-03-31 11:25:36,220] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1124_253066937408235522.jar
[2013-03-31 11:25:36,220] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1123_4188205777389701287.jar
[2013-03-31 11:25:36,220] WARN  com.google.gerrit.server.plugins.PluginLoader : 1 plugins still waiting to be reclaimed after 4 minutes
[2013-03-31 11:29:50,172] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1128_1462254843060851799.jar
[2013-03-31 11:29:50,172] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1127_8797027389254032869.jar
[2013-03-31 11:29:50,172] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1125_209452049010666226.jar
[2013-03-31 11:29:50,172] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1126_6693391358698355539.jar
[2013-03-31 11:29:50,172] WARN  com.google.gerrit.server.plugins.PluginLoader : 1 plugins still waiting to be reclaimed after 9 minutes

The number of Plugin Cleaner jobs increases until it reaches a limit, then starts over.

Task     State                 Command
------------------------------------------------------------------------------
57196a36 11:21:44.623          Plugin Cleaner (waiting for 2 plugins)
36fdcf6f 12:11:10.676          Log File Compressor
------------------------------------------------------------------------------
  2 tasks

Task     State                 Command
------------------------------------------------------------------------------
57196a36 11:21:44.622          Plugin Cleaner (waiting for 3 plugins)
36fdcf6f 12:11:10.675          Log File Compressor
------------------------------------------------------------------------------
  2 tasks

Task     State                 Command
------------------------------------------------------------------------------
7724aea1 11:22:18.589          Plugin Cleaner (waiting for 1 plugins)
36fdcf6f 12:11:10.675          Log File Compressor
------------------------------------------------------------------------------
  2 tasks

Task     State                 Command
------------------------------------------------------------------------------
7724aea1 11:22:18.589          Plugin Cleaner (waiting for 2 plugins)
36fdcf6f 12:11:10.675          Log File Compressor
------------------------------------------------------------------------------
  2 tasks

STOP GERRIT, then RESTART

ls plugins/
replication.jar.disabled

ssh -p 29418 master gerrit plugin ls
Name                           Version    Status
-------------------------------------------------------------------------------


Task     State                 Command
------------------------------------------------------------------------------
37845e2c 11:33:48.957          Plugin Cleaner (waiting for 1 plugins)
d7862234 12:32:48.950          Log File Compressor
------------------------------------------------------------------------------
  2 tasks

Task     State                 Command
------------------------------------------------------------------------------
5730f2d1 11:37:20.295          Plugin Cleaner (waiting for 3 plugins)
d7862234 12:32:48.951          Log File Compressor
------------------------------------------------------------------------------
  2 tasks


[2013-03-31 11:33:49,421] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1132_5939560954018629418.jar
[2013-03-31 11:33:49,421] WARN  com.google.gerrit.server.plugins.PluginLoader : 1 plugins still waiting to be reclaimed after 1 minutes
[2013-03-31 11:34:20,296] WARN  com.google.gerrit.server.plugins.PluginLoader : 1 plugins still waiting to be reclaimed after 1 minutes
[2013-03-31 11:37:21,572] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1133_5538450256879605994.jar
[2013-03-31 11:37:21,572] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1135_1955457217262328990.jar
[2013-03-31 11:37:21,572] INFO  com.google.gerrit.server.plugins.PluginLoader : Cleaned plugin plugin_replication_130331_1134_2102375682649291206.jar
[2013-03-31 11:37:21,572] WARN  com.google.gerrit.server.plugins.PluginLoader : 1 plugins still waiting to be reclaimed after 4 minutes


Sign in to add a comment

Powered by Google Project Hosting